break it down into more manageable pieces and check each of them individually.
have you done any serial communication before? if not i seriously suggest getting two ports and null modem (crossover) cable.
ports don't have to be on same computer. but each need to be open by some terminal program like HyperTerminal, RealTerm etc.
both ports must be set exactly the same (baud rate, data bits, stop bit, parity and handshaking - normally you would use no handshaking).
then whatever you type in one terminal should come out in other (and the other way too).
simplest nul modem cable is just two DB9 female connectors and three wires:
DB9F1 DB9F2
2-----------3
3-----------2
5-----------5
note that 5 goes to 5 (GND is always straight through) but 2 and 3 are crossed (send from one goes to receive of other port and vice versa).
devices that use handshaking lines (DRT, DTS...) use additional wires. on microcontroller it is common to just use TX/RX so
setting should be "no handshaking".
now you can replace one of the terminals with matlab. if the matlab opens port and connects correctly,
you should see in other terminal exactly what matlab is sending and if there is expected reply,
you can type it in by hand. once you get this working, you do the same with other side (pic).
you connect your PC to PIC and on the PC open terminal to talk to PIC. again same port settings are must
be used or there will be no talking. you can type in whatever you like and PIC should handle it.
if it does not or if gives incorrect reply, you need to modify PIC code.
finally when the bugs are removed, you can try matlab talking to PIC.