Yes, the 4 Ohm load will cause a higher current flow which drains the battery faster with many type of audio amplifiers. It's only the switching type amplifier (such as class D) that would not show this behavior, as they would have an inductor which allows for an actual power conversion similar to a buck circuit. The inductor stores energy part of the time and provides energy at other times which allows a (theoretically) ideal power conversion. If you put 1 watt in at whatever voltage is available, you get 1 watt out at whatever voltage is required. That's almost never the case with a regular amplifier of lower class which might have to take in 2 watts to produce just 1 watt output.
I think we are talking about the non-class D (and up) types in this thread so we would see this problem.