As long as you're only using one analogue input then there's no real concern about the source impedance - the problem isn't the input impedance of the pin, but the time it takes to discharge or charge the sample and hold capacitor via the source impedance. This only really causes problems if you're switching between different inputs, you have to allow substantial time delays for the charging/discharging to fully take place.
But as you're just using a pot, why not use a sensible value one anyway? (4.7K/5K would be good).