For FM in the US an F connector is used. For some parts in Europe, A PAL connector (At least that's what I've heard it called). These should be for 75 ohm coax. BNC is TYPICALLY 50 ohms, but can be had in 75 ohm Z.
You may also see screw terminals for 300 ohm balanced transmission line. I've seen one or both. There is a clothespin like device that can be used to easily disconnect twinlead. I haven't seen them in a long time.
For AM, I have seen nothing and I've seen screw terminals. One is a ground. On some sets, there is a link that can be removed to disconnect the internal antenna.
My GUESS is that an FM transmitter would have a 50 ohm output and be an N connector for high power. Can't say with 100% certainty.