I think its a fallacy to believe that a color is interpreted exactly the same by one person than the other. I'm not even talking about color blindness.
Some have more cones/rods (changes intensity), bigger eyes, cornea changes with age, deep set eyes (if the sun hits the eye, the color contrast will be lost), lens that are incorrect (astigmatism when high enough, subtly changes some colors, that what I have). That's just some of the changes to color caused by the eye apparatus.
Then, you've got the whole mind thing. Its obvious that some people have an uncanny ability to differentiate colors. Do they see the same thing as others. The mind interprets the color inputs from the eyes and puts a tag on it, I'd be surprised that this tag is the same for all.
When I see the color match that many make, I wonder if I'm the one who's brain misinteprets their colors or they have a different color pallette in their head