Color is an extremely fascinating and useful topic. It's also much deeper and more complex than it might appear on the surface. The subject of color spans many fields including physics, biology, psychology, philosophy, and art. Several months ago I started studying color in more depth from a few angles, including colorimetry: the measurement of color.
There are lots of color spaces and color appearance models that try to quantify the human perception of color. One of the newer and better ones is called CIE L*a*b*. CIE L*a*b* was designed to be perceptually uniform, that is, the distance between two colors in the space is proportional to the perceptual difference between them. Like all color spaces, it's based on three dimensions. In this case, L* is the perceptual lightness, while a* and b* represent the two chromatic channels of the opponent process model of color vision.
Recently, I used Photorealizer's sampling and image processing abilities to render a top-down view of the CIE L*a*b* (CIELAB for short) color solid showing only the sRGB gamut—the colors that are displayable on a typical computer screen. Here's the result:
Notice that the primary (red, green, and blue) and secondary colors (cyan, magenta, yellow) of your computer screen are located at the corners, and white is right in the center.
The colors in that image are not all at the same position on the lightness axis. Here's a version showing the lightness everywhere on that image:
Perceptual lightness is not just an average of the RGB values. Nor is it the same thing as relative luminance (Y), however it is derived from it. I won't go into too much detail here, but I'm planning on posting more color stuff in the future (and maybe I'll even extend this post).
If I just take a cross-section of the CIELAB color space in the a*–b* plane, all of the values have the same perceptual lightness. It looks like this at L* = 75:
And this at L* = 50:
If you were to look at the gamut of human color vision in CIELAB space, it would be much larger at any given lightness (and wouldn't have sharp corners). sRGB can only represent a fraction of the colors that typical humans can see.