I agree with you, we must not leave any floating input in logic circuits.
But TTL logic (LS/S/plain) is has some immunity to floating inputs, observe that the input section reminds us a common-base amplifier, which means that it has a low impedance input -> "High current" to drive it. (from 20uA to 400 uA against the 1uA from a HC family).
Those are maximum figures. In practice, TTL pulls high, and HCMOS has zero input current. If you charge an HCMOS input in dry conditions it will stay at the voltage you left it for hours.
But to my surprise, i saw that the light used to glow even if there was No Gate Input (I1), actually the output did not change if i removed the Input Connection(I-1) of Not Gate.
EDIT:
I can't see a series resistor so unless the LED has one built-in the only thing limiting the current is the gate's output impedance and batteries. It also appears to be running off two AAs which shouldn't work if it's real TTL but I wouldn't rule it out. It could be CMOS but the behaviour is too much like TTL.
I believe he is describing a chronological happening of the facts:
1st -> He powers up the circuit with no input and the LED is turned off.
2nd -> He grounds the input, and the LED powers up, as it should be.
3rd -> He removes the input, and the LED still glowing.
4th -> He pulls the input high, and the LED turns off, as it should be.
You could be right, you might be wrong, the picture isn't clear enough to see the marking on the IC so we'll never know until the original poster returns, if (s)he ever does.