Placing high frequency low ESR capacitors- ceramic for non audio paths and pollycarb for audio paths- makes the source impedance of a large electrolytic capacitor appear to be lower at high frequencies. This has two major benefits:
(1) Presents the audio amp with a lower impedance on the supply rails. Supply rail modulation is one of the biggest sources of distortion, after xover distortion, in class AB amplifiers. This is partly why a common upgrade on some commercial audio power amplifiers of a better auxiliary PSU brings such huge improvements in performance. On the Mission Cyrus range of amplifiers the addition of an external PSX unit transformed an already good amp.
I did an experiment once of placing large LiIon batteries across the supply rails with the bridge rectifier disconnected. This improved the sound noticeably (LiIon batteries have a very low source impedance and the existing capacitors also helped, especially at high frequencies).
I had plans at one time to build an amp where batteries were part of the PSU, but for some reason never got around to it. I did design a headphone amplifier powered by large LiIon batteries but that project is still on the back burner, like many projects.
(2) For the same reason as above, HF capacitors sprinkled around a physical circuit can also suppress interference both in and out of a unit.
An important point to note is that it is practically impossible to make a low distortion, high EMC equipment without a good physical layout, especially earth reference points. By the way, there is no such thing as a resistor, capacitor, inductor or conductor in the real word. Also, the nice clean schematic that you work to is an illusion. That is why sims can be an illusion too if not deployed wisely.
You can see evidence of this by looking at the waveforms inside a classic Difference/VAS/VtoI audio amp reproducing a sine wave. Often you will not see a sine wave but a saw tooth like waveform as the colossal amount of overall negative feedback wrestles with the parasitics etc to make the output voltage sine wave simply a bigger version of the input sine wave. On audio amps which heavily use local feedback, as was the fashion at one time, you will see sine waves though.
And don't even bother poking around the output stage as the drivers and output transistors switch on and off around the center of the input sine wave. By the way I am talking in hifi terms here where very small distortion artifacts can totally wreck the sound of an audio amp.