I think you are thinking about the problem the wrong way. You are thinking about what the camera does rather than how it will do it. Don't make the camera identify one, or 3 people and follow them. That's really advanced machine vision stuff. Try thinking colour tracking. You could try motion tracking, but it's too much to explain on the forum so I won't- basically comparing two images taken in the same position spaced apart on a human time scale (0.5ms maybe, since that's the speed that people move at rather than us), and then compare what parts of the image have changed and find some way and somehow figure out which direction the changed part of the image has moved in, and center the camera on that. The camera movement will be jerky since it has to do the comparisons half a second apart to be able to see the movements that people make (assuming they are large movements), but then it doesnt depend on colour.
When you speak about the motion, what do you mean by XYZ coordinates? Is the camera moving linearily in space? Or rotating to follow the user? If it's just rotating it's just as easy to do 360 degrees as 180 degrees...just change the software limits of rotation and use a better motor.
Sounds hard...good luck! I'd go colour tracking. Give it a tracking priority order for red, green, and blue so that when someone tries it out it will *look* like it's picked a random person to follow when it's actually just following a colour.