ya i just replaced the alternator. but when ever i pull back on the tensioner its really easy to pull on it. I usually have to use two hands and walk around to the side of the truck. Lately i can do it with one hand.
A new tensioner is a good start but Check Your pulleys as well.
I had a issue like this on my F-150 (4.9 I-6).When I replaced the tensioner my belt was still coming off due to the water pump trying to freeze up.