If I’ve understood you correctly, then there is a square wave where polarity is shifting from red to black wire. that is: first current flow from black to red then from red to black. If this is so then it doesn't matter which way the wires are connected since it is an AC signal and not a DC signal. The Diode will be ON half the time no matter what way it is wired. There might be a different problem though: if you use a standard optocoupler like 4N25 it can only take 6v reverse voltage, since 5 volt is quite close to 6 I might include another small signal diode (any kind of diode really) in series after the coupler to cut the reverse.
When the diode is not turned on at all (no signal for the buzzer) no current will flow through the transistor on the other side of the optocoupler, which in turn means that there is no current flowing through the second transistor. This means that C1 is being pulled up (charged) to +5 volt, and the output is HIGH. HIGH = NO SMOKE
When the transistor turns ON (the buzzer would be on) the capacitor is being uncharged through R4 and the transistor and the output goes LOW. LOW = SMOKE
We want the charging to be slow (keeps the signal LOW for a long time and gives a lot of time for the microcontroller to read the port) but the uncharging should be fast.
Since the charging of a capacitor is determined by the formula Vc = V * (1 – e^(-T/R*C) and since we want to keep the voltage under 0,8 V (this is the upper limit for a logical Low for the PIC) for some time (say a couple of hundred ms). We could try for instance with a 100 mikrofarad capacitor and calculate a resistor value: 0,8 = 5V * (1-e^(-0,2 sec /R * 0,0001)) R=11K5 ohm, so R3 + R4 should be over 11k5, R4 should be small in order to make discharge quick, so you could try R4 = 560 ohm and R3 = 12k.
The rest of the resistors are more or less arbitrary: There should be about 10 mA on the diode, so try another 560 ohm for R1. R2 should let enough current pass for the transistor to be saturated good, try 1000 ohm. The pull down R5 should be much bigger than R2 but I don’t think it can be too big, so try 50k
Good luck