Yes, you cannot have both the 3.3V and the 5V on at the same time.
Here's a circuit that uses two P-MOSFETs in series back-to-back to block the 5V from feeding back through the transistors, without any diode drop when they are ON.
That works because MOSFETs conduct equally well in both directions when ON.
This configuration requires an additional transistor to control the MOSFETs, since the MOSFET gate-source voltage must be near zero to turn them off (which here is provided by R1 when the control transistor is OFF).
The connection of the Q1 transistor emitter to the +5V prevents the 3.3V from being turned on when the 5V is present.
LTspice sim of example circuit below:
The output (yellow trace) is 3.3V when the Ctl input (green trace) is high, and the 5V is not present.
Otherwise the output is 5V minus the D1 diode drop when the 5V is supplied (red trace), independent of the 3.3V Ctl input voltage.
The MOSFETs and diode can be the ones you chose, of course.