An easy way to think of it, presume you're doing building work in a multi-storey building, fitting light switches.
When you're on the ground floor, you measure from the 'ground' up to get the switch at the correct height.
On the higher floors you measure from the floor of each level, you 'could' measure each one from the ground - but it would be FAR more difficult. So effectively, 'ground' is different for each level, as it's referenced to a different place.
As others have said, it's mainly just a 'point of reference', nothing else.