It is more likely that the designer would encode TX/RX pins as end user interface, if it were uart the pin would more likely be along with/like ICSP bus , so we would have 5 pin coming out, clr,vcc,gnd,clk,dat
serial implies we are looking at 9 wire, and inversion is already thought of, on both encoder and decoder:
**broken link removed**
of course we are still speculating here, my camera outputs 9600b serial and has 4 wire. our box could just as well be outputting alieneese encryption,
FOR SAFETY: check that we are not using the +10 - -10 v encryption, I have read about these so I know what others are talking about, but this is mostly associated with old modems and long wires(im sure other things as well *or something like that), none of the serial devices in my bsmt here drive at anything other than 5v, and i have never seen one that does, BUT check to be sure!!! if only 5v on our power and data pins , sounds like green light to me!
I think it would be best to describe/image how many wires are coming out and what shape they are in,.
Our best tool in this scenario is a scope, if no scope than logic analyzer on pickit,
if not either of those then just plug in, why not, by doing so we will not be able to see exactly what is coming through but you can see noise get decoded as long as you ensure to have the start/stop bits on the correct edge of the "noise signal" to allow the dataflow,
or what i mean is:
1)setup terminal so there is no stop bit and no parity
2)terminal now acts as sloppy scope , by changing the baud we will see your data resolve from more than FF's and 00's
3) after we are getting any input maybe turn stop bit back on , it will be trial and error on our exact: baud,parity,stop,handshake setting. If we don.t have the papers.
4)It is this point we will know if Eric's inverter is required(if we need to translate by inversion).
Certain serial handshakes require RTS and other pins as well, i think default is: XON/XOFF
Also if we are splicing our own plugs don't get confused that ones TX is the others RX
The second electrical incompatibility could be this: if your PC outputs logic1, and device outputs logic0 connecting these two TX wires would/could be hazard for system. Same thing if we get vcc/gnd backwards. We should be able to presume that this is a completed functioning device, but should go through steps where doubt is in order about design flaws.
BTW! Br@y Terminal ROCKS!