Mine does that too, it's most likely buildup. I personally think I have something else going on, when I spin my rotor with the wheel off I can actually feel it having more friction in certain places and less in others as I spin it, and I see no buildup anywhere. If you have cementite that has gotten on there, you'll need to resurface them, but you need to be doing some really hard braking.
You can resurface the rotors and use old pads, you just need to bed them in again. The rotor is harder than the pads. It may take more than a few bed-ins, but it'll all settle out.
I don't know about balljoints, but bushings can affect braking performance as well.
Courtesy of cowmoo32