I've just tested up a lashup version of the new design I dreamed up after seeing the problems you were having & parts costs etc.
It is using the PAM8403 I mentioned, plus some other common bits.
It works!
Two independent channels, or stereo. Most of the interconnections are on the underside, made with wire wrap wire. Thee are a couple more 10K resistors under each opamp IC as well.
Everything is battery powered for the test, except the focusrite interface, which has the audio fed via an isolating transformer, as the PC has a lot of high frequency noise on the USB cables.
The small board is just a simple mic preamp to get a suitable audio level in to it; I was using an SM58.
Provisional schematics attached.
The prototype only has a second order filter and the caps are far too small, there is RF carrier leakage on the audio, seen on on a scope. The one in the schematic should be better.
I'll be doing a proper article on if for my youtube channel sometime soon.
Audio of the initial test here, as MP3 to keep the size down. The levels are a bit off and it's clipping somewhere, but for a first test and no proper adjustments, I don't think it is too bad!