I'll argue that, I've built and fixed small amps using a 9v battery and LM386 and they are capable of being "annoyingly loud" especially if driven into clipping.
If it is too quiet then you have either too small a source signal or too small a gain in the amp, or possibly the amp is faulty or wired incorrectly.
You said it sounds distorted so start with the source sound, this is probably too smal and/or distorted. You could connect that into another amp like your HiFi to test it.