Black's are finicky
On pure blacks, no, they shouldn't make much of a difference.
The lighter the color the more impact the same amount of added reflectivity should have.
Thing is that as soon as black begins to fade - which it will, no matter what - optical brighteners can pull that fading into odd directions.
Black's can begin to look brown or blueish.
Many assume the greyish look like a white film on Black's is fading but never owned a black item that faded to grey.
I assume that's just poor rinsing.
I found that most pastel colours don't bother much about detergent.
My mum washers here greys and off-whites with a detergent without optical brighteners while my more colored shirts go through the whites load.
One would think either would be better but they look both pretty well.
What I found is that even many color detergents somehow contain optical brighteners when in powder form.
Still others do not.
A nice thing about German labeling practice is that besides all the chemical names, most if not any detergents have a section that lists tensides, enzymes, optical brighteners and bleaching agents just as that.
So it's really just a plain text statement. Helps a lot when looking for a fitting detergent.