I have the same problem
Granted, I don't have the world's best towels, but they are Costco/Kirkland brand Egyptian cotton, the kind they sell for $7-8 for a large bath towel. After about two years they become stiff and scratchy, but each towel is washed maybe 3-4 times/month, so you are looking at 100 washings by the time two years rolls around.
I use HE powder, very low amounts (no suds residue on inner door) at the end of cycle. We have very hard water here and I don't have a softener. I use 20 Mule Team Borax as a water conditioner, but I am adding it to the wash compartment section. Should I be adding it to the conditioner section instead? By using it in the wash section, it's probably helping to soften/condition the water for wash cycle, but not helping for rinse cycles. Another option would be to add some to both wash and conditioner compartments, so that water is conditioned at both times.
What I did notice after starting to use Borax is that detergent rinses out of the wash more cleanly, and there is less (or no) suds residue on the door at end of cycle. Even though I presently use Borax only in the wash cycle compartment. (no pre-wash on my FL).
As per Whirlcool's advice, I generally use Medium and not High heat setting on my gas dryer. Towels are washed on Heavy cycle with max spin speeds and extra rinse (three or four rinses in all). Wash temp is "hot" or sometimes on "warm", never "cold" for towels. Machine does not have Auto Temp Control.