Yes and no.... yes, it can build up if you don't have good laundry habits, don't use a good detergent, don't use hot water, LCB, etc..
I have my hand VERY heavy on fabric softener and my washers have ZERO buildup.
My water heater is set to "almost trigger the safety valve", my hot washers are...