Yep, that's pretty much what I was thinking. Actually, I was assuming that, depending on the LED (size, brightness, speed and color) I'd probably end up using a mix of transistors and IC comparators. The comparator especially for trying to capture the light from the bi-color LEDs with filters.
If you've still got a bug in your ear, you might like to try some different colors with cellophane filters. It's been awhile since I saw the color response curve for a CdS cell but it's likely that there are some conbinations that would be tough.