Ah ok, that's simply an optical illusion. The brain is full of pre-processing that you can mess with in order to trick it to see things that aren't there and to shift colors around as well as to play with figure-background. But that is a case of 'bad faith', you could do the same for audio illusions, it wouldn't help to draw any further equivalence between the visual and the auditory system.
Both work on the perception of waves with a certain periodic repetition but there the equivalence ends, there is no such thing as 'timbre' in vision, we simply don't work with harmonics there and the shape of the wave in sound is very important and non existent in vision (you can see a single photon in sufficiently dark adapted conditions, your eye as a fundamental particle detector!).
You may want to amend your original statement "Of course we can. Everybody that isn't somehow colorblind can reliably distinguish between a basic number of colors [...]" then, because clearly that's not the case as you state yourself.
Both work on the perception of waves with a certain periodic repetition but there the equivalence ends, there is no such thing as 'timbre' in vision, we simply don't work with harmonics there and the shape of the wave in sound is very important and non existent in vision (you can see a single photon in sufficiently dark adapted conditions, your eye as a fundamental particle detector!).