Yes, it's quite tricky to get the oscillator speeds set correctly - particularly for 32MHz, as it's not just a matter of the config fuses, you have to set registers as well. Here's the C routine for setting mine to 32MHz, generated by MCC.
C:
void OSCILLATOR_Initialize(void)
{
// NOSC HFINTOSC with 2x PLL; NDIV 1;
OSCCON1 = 0x10;
// CSWHOLD may proceed; SOSCPWR Low power;
OSCCON3 = 0x00;
// MFOEN disabled; LFOEN disabled; ADOEN disabled; SOSCEN enabled; EXTOEN disabled; HFOEN disabled;
OSCEN = 0x08;
// HFFRQ 16_MHz;
OSCFRQ = 0x05;
// HFTUN 0;
OSCTUNE = 0x00;
// Wait for PLL to stabilise
while(PLLR == 0)
{
}
}