I have pretty much changed bath towels after one use my whole life. It was what I grew up with. And one of my grandmothers was insistent on that policy when I visited.
One issue that has concerned me is athlete foot fungus. I have never had problems with athlete's foot, but if I should happen to pick it up one day, I don't particularly want to risk spreading to other parts of me via a bath towel used multiple times.
In order to minimize wash, I currently use a policy of using hand towels for bathing. They are not ideal, but they work adequately for me. The towels probably work better than the thin, small towels we got in junior high PE. Plus I can get double use out of hand towels--a day for drying hands, then used for bathing.
Washing towels, I use the hottest water possible. This time of year, I machine dry. Summer I air dry, and the line has exposure to the sun.