Thank you guys for you response. I have played with it a little bit more. It appears that the XTAL should be the actual clock at the clock pin and not the one after the PLL. I figured out that the clock division was the limiting factor, so I will be stack with it for a while.
I finally figured out how the BRGH works, to summarized:
There are two models for calculating SPBRG, one is based on low speed transmission (BRGH=0) and the other is based on high speed transmission (BRGH=1). Those high/low speed transmission settings can be derived from looking at the results of those two models.
Low speed transmission: SPBRG = (XTAL*d'1000000')/(d'64'*BAUD_RATE) - 1
High speed transmission: SPBRG = (XTAL*d'1000000')/(d'16'*BAUD_RATE) - 1
* the SPBRG should be rounded to the nearest integer
*BAUD_RATE is the desired baud rate (19200, 57600, etc)
Since the above SPBRG equations round the results, we want to calculate the actual transmission rate, and the percentage error due to this rounding.
low speed: ACTUAL_BAUD = XTAL*d'1000000' / (d'64'*(SPBRG+1))
high speed: ACTUAL_BAUD = XTAL*d'1000000' / (d'16'*(SPBRG+1))
The percentage error is calculated as (same eqn for both models):
% error = ((ACTUAL_BAUD-BAUD_RATE)/BAUD_RATE)*100%
Example: we want to get transmission rate of 19200 bits/s w/ crystal of 10M
=> XTAL=10, BAUD_RATE=19200
low speed: SPBRG = (10*1000000) / (64*19200) - 1 = 8.138 - 1 => 7
ACTUAL_BAUD = 10*1000000 / (64*(7+1)) = 19531.25
% error = ((19531.25-19200)/19200) * 100% = 1.73%
hish speed: SPBRG = (10*1000000) / (16*19200) -1 = 32.55 - 1 => 31
ACTUAL_BAUD = 10*1000000 / (16*(31+1)) = 19531.25
%error = 1.73% as for low speed, but if we were to round the SPBRG, we would get SPBRG= 32.55-1 = 32,
then ACTUAL_BAUD = 10*1000000 / (16*(32+1)) = 18939.4
and %error = ((18939.4-19200)/19200) * 100% = -1.14%
Results: using high speed model is always better cause it provides smaller % error. The only situation where it cannot be used, is when SPBRG for it is greater than 255. This will typically occur at high clock frequencies and very slow transmission rates. In this cases low speed transmission model should be used.
Hope this will help someone