Hello there,
The Arduino is capable of putting out a variety of frequencies that would be considered noise, not just the 16MHz or whatever.
That is because as the different code is executed there will be sub harmonics based on how long those codes take to execute and what they are actually doing. As they draw differing power levels from the power source, the harmonics present on the 5v line will be varied.
It may be hard to filter them by filtering the supply itself, although you could try. Obviously a ceramic and electrolytic would be something to try.
A trick from RF though would be to isolate the noise source so that it alone can be filtered. This would work if there is only one source of noise. Unfortunately there may also be RF noise getting to the audio front end which could get demodulated into audio frequency noise.
Assuming the former alone, you might try using the logic output from the port to drive an NPN transistor. The pullup resistor can come from a highly filtered voltage source. To get that filtered source, a small inductor in series with a resistor and two filter caps (one ceramic and one electro) and that should be able to clean up the power supply noise.
The RF noise, if present, would be harder to deal with as you may have to shield the Arduino board.
Try the transistor trick first and see what happens. You need a base resistor, collector pullup, and inductor and resistor in series with the 5v line and two caps for filtering. If you dont have an inductor try the resistor in series alone first, and if that does not work good enough try winding your own air core inductor with inductance say 10uH or better.
Interesting idea BTW. A Due board may have been a better choice, but yes that is more expensive.