Yes and no.... yes, it can build up if you don't have good laundry habits, don't use a good detergent, don't use hot water, LCB, etc..
I have my hand VERY heavy on fabric softener and my washers have ZERO buildup.
My water heater is set to "almost trigger the safety valve", my hot washers are really hot, I use plenty of detergent (I confess my hand is also a bit heavy), rinse properly, repeat rinses depending on the washer, etc.
Now, if you have a heavy hand on FS, use only liquids or pods, cold water, quick cycles and overloaded machines, your washer will soon look like a Pantone scale with 50 shades of blue.