I'd wager the previous owner filled up with well water, and it had been sitting like that for a while. I'd detox with bleach solution, flush a few times, and then use standard chlorinated municipal water for a while. Replacing the tank sounds like overkill unless you've got other objectives too.
I'll also throw this out there. Flushing, bleaching, etc. we still never drink from it, strictly hand washing and such. We carry separate gallon jugs for drinking, cooking, coffee.
Im the same way. I use it for washing a dish or pan, shower, hosing off the dogs, etc... for drinking i carry two 5 gallon jugs of fresh water.