If your system is using 4 wires, 2 for Tx and 2 for Rx, it is probably more accurate to call it RS422.
I assume that the master transmits to 3 slaves all set to receive and that is OK.
When the slaves transmit, if you only have one slave connected to the (master receive) line, all is OK.
But when you add more slaves, the received voltage at the master is reduced until it no linger works.
It sounds to me that the problem is that your slaves have their transmitters permanently enabled.
What you must do is to ensure that the slaves only turn on their transmitters when they have data to send, (there may be a setup to put them into half duplex).
The rest of the time they must be switched off to their high impedance state so that they do not load the line and reduce the voltage of the slave which is trying to transmit.
JimB