I ground off the rivets when I did mine. It wasn't a bad job at all. The axle pivot bushings are a real PIA to get out if you decide to do them also.
I've owned several Ford trucks that liked to wander including a brand new E-150 Van and every time it wasn't something mechanical it was just a poor alignment job. Most shops just set the Toe on Ford Trucks and don't want to get involved with setting the Caster/Camber unless it's way off. My new Van was "in spec" but was wandering all over the road. I had taken it back to the dealer and they kept telling me it was OK. I took it to a shop a friend recommended that was good with Fords to have it aligned and it drove perfect after they put a shim in it. That Van now has 180K on it and hasn't needed a wheel alignment since.

Try going to an alignment shop and explain your wandering problem if they are any good they should know what to look for.

Sorry about the long post hope this info helps someone.