I would have expected exactly what you describe. You should be very, very careful in designing anything that has input or output voltages higher than the positive supply or lower than the negative supply at any time.
From the data sheet:-
Voltage on any pin with respect to VSS (except VDD and MCLR) .............. -0.3V to (VDD + 0.3V)
There are clamping diodes from every pin (except MCLR and the power pins) to Vss and Vdd, so the input pin cannot rise much above Vdd. In your circuit, the diodes stop Vdd falling below the input voltage.
Use a continuity tester and you can see that those diodes are there. You can power a PIC like that from any pin, not that you should. Current flows from the pin, through the clamping diode and you will see about 4.4 V on the power rails. The decoupling capacitors will help the PIC run under those conditions.
There are several things you can do to stop that happening.
1. Use a transistor as a buffer, with a pull-up to Vdd
2. Put a big resistor in series. The input current on a digital input is tiny, so you can use a 100kΩ or so, which won't provide enough current to run much.
3. Use a PIC with high voltage tolerant inputs. Most are 3.3 V devices. In the data sheet they say:-
Voltage on any digital only pin with respect to VSS ........................................ -0.3V to +6.0V
Now that has no reference to Vdd, so you can feed it with 5V at any time. They have different clamping systems. You have to chose your pins because analogue capable pins generally don't like higher voltages.
4. Use a gate like a 74AHC1G04 as a buffer. It can run from 5V and can stand 5V input even with no supply voltage, so if you wire the supply to the gate to the PIC supply, both can fall to 0V while the input (USB) is still at 5V
5. Power everything off the USB anyhow.