Usually the problem is that some of the output sound frequency energy is being detected by the input port of the circuit. In this case, the input is the speaker utilized as a microphone.
One possibility is to reduce the gain of the system, but this is contrary to your use of the amplifier.
You might try to attenuate the higher frequencies at the speaker/microphone input.
You only need 2k or 3k ... maybe less .... of audio bandwidth frequency in order to have intelligible, normal speech recognition.
Try, placing a 1µF capacitor in series with a 100 Ω resistor at the intercom speaker terminal. .... Not absolutely sure about the values, but you should get some effect. .....
Do the wires going to the speaker/microphone have any sort of polarity?
Does one wire connect to the intercom ground?
If so, a polarized capacitor should go + to + when installing. However, a non-polarized cap can be installed without regard to terminal polarity.