The notion of the earth as a zero potential reference is something that I never really considered independently. Out of curiosity, I found that the earth, as a whole, including the atmosphere has a neutral charge. So, in that sense, the earth as a whole is not a reference point at all since there can be no current flow from a charged body (a voltage node) to a neutrally charge body, thus no potential difference between them.
The solid earth, alone, is negatively charged (electrified). This is balanced out by the positively charged atmosphere concentrated mostly in the ionosphere, so the two "plates", combined with the dielectric air between the two, can be considered a poor capacitor across which electrons from the solid earth continually leak back into the ionosphere. Lightning strikes are the electrical generators that keep pumping the leaked electrons back to the solid earth, maintaining a huge charged capacitor with about 300,000 volts across the "plates".
So, I would think, the solid earth, by itself, having a huge accumulation of electrons (negative charge of about 500,000 coulombs) should exhibit a potential difference (voltage) against a man made body with a lesser negative charge, or positive charge. I would think. Theoretically. I cannot say that it is a zero voltage reference, though. Hmmmm.