Maybe it's just me
But there is no method I have found of leaving clothes softer than using a good quality liquid fabric softener (Lenor).
I use the recommended dose in every load, no more, and I have never had the problems of mould or waxy build-ups that people discuss on here.
Our water is as soft as you can get, never seen any limescale deposits or mineral build ups anywhere here, and clothes are still softer when fabric softener is used.
It is especially beneficial when line drying (which I do as much as possible). I want my clothes to feel as soft and fluffy as possible.
My whites are as white as can be, so unless you don't wash them properly it doesn't make them dull or yellow.
As for using less detergent meaning you won't require softner, I have never found this to make any difference (unless using more detergent means rinse performance has been reduced).
If I use less shampoo in the shower and don't use conditioner, my hair feels greasy and is less manageable, washing it more thoroughly and using conditioner leaves it smooth and not greasy at all.
So no, using fabric softener will not harm your machine or make it dirty, and it will not dull your clothes or damage them, as long as you don't use more than is necessary and keep your machine clean.
It does help that last time I checked Lenor (Downy) sold here is entirely synthetic, and not plant based, perhaps this isn't the case in the U.S.? That might explain the difference.
Matt