As Nigel implied, the diode only works if there is a return path to the other end of the capacitor (GND) after you remove power. If you disconnect power with a switch, you can use the circuit below. It requires a DPST switch. The 100 ohm resistor limits the peak current in the diode and the switch.
Alternately, if you can afford to waste some power, you can eliminate the 100 ohm resistor and the switch pole to ground. Add a resistor of a few kohms from +12V to ground instead.
If a recovery time of a couple of seconds is OK, you can omit the 100 ohm resistor, and just connect the diode across the pot. The 4.7k resistor becomes your discharge current limiter.