That is based on the color of the illumination and this of course affects the perceived color. It's the difference between emitted and reflected light, but in the case of a comparison with musical notes it would be fair to only use emitted light.
There is no exact equivalent to reflected light with its own color illuminating a colored drawing. Though it would be interesting to see if such a thing could be constructed somehow artificially using a device that receives sounds and then somehow frequency shifts them before emitting them again. That would be a fun experiment!
When we perceive emitted light color we’re also perceiving RGB emissions that are blended. I love giant LED panels that when you get close you can clearly see the individual colors. It’s a trip.
Humans are horrible (incapable?) at evaluating absolute color. It’s entirely relative.
Ah ok, that's simply an optical illusion. The brain is full of pre-processing that you can mess with in order to trick it to see things that aren't there and to shift colors around as well as to play with figure-background. But that is a case of 'bad faith', you could do the same for audio illusions, it wouldn't help to draw any further equivalence between the visual and the auditory system.
Both work on the perception of waves with a certain periodic repetition but there the equivalence ends, there is no such thing as 'timbre' in vision, we simply don't work with harmonics there and the shape of the wave in sound is very important and non existent in vision (you can see a single photon in sufficiently dark adapted conditions, your eye as a fundamental particle detector!).
You may want to amend your original statement "Of course we can. Everybody that isn't somehow colorblind can reliably distinguish between a basic number of colors [...]" then, because clearly that's not the case as you state yourself.
I think a Ring Modulator might have some equivalence. Depending on the frequency you set it to the ability to accurately detect the frequency of the input notes can diminish quite drastically.
There is no exact equivalent to reflected light with its own color illuminating a colored drawing. Though it would be interesting to see if such a thing could be constructed somehow artificially using a device that receives sounds and then somehow frequency shifts them before emitting them again. That would be a fun experiment!