The answer is no...let me tell you why.
YOu do realize you need ambient light to see colour right? Do you also realize that visible light and IR are essentially the same thing but different frequencies? That means you can't use IR to tell if something is red, green, or blue. YOu need to use visible light, by definition. Ambient light isn't even a problem- you want it. As a result, you don't need "immune to ambient light". That has no meaning when you are talking about a colour detector. You can't detect light unless it reflects off of something- which means you can't see it in mid-air (assuming the air is free of particulates).
Just get three photo sensors with a R, G, and B filters in front of each one...easy enough, simple. All you need is a 3-channel ADC. The only concern is that if the ambient light is unbalanced (ie. if there is only red light around, all objects will appear red- except pure black, blue, and green objects will all appear black). Is this what you mean by immunity to ambient light? All if you have to do is have white LEDs or another source of white light shining whever the colour sensor is looking to provide all the wavelengths of light so the true colour can be accurately reflected.
And yes, if you can read the data from your camera you can deduce the colour. You can also buy one from
www.picaxe.com in camera-form where it has been simplified to be just a colour sensor (no imaging), just in case this was part of a larger project and you only wanted to build one because you couldn't find one.