That sounds like ultrasonic or RF oscillation / squegging, due to the layout and/or lack of sufficient decoupling.
It could also be affected by the proximity of input and output connecting wires to other parts of the circuit, hence the request for photos.
I've built diode noise sources so I'm aware of the gain needed, and also I've never experienced noise due to LEDs.
But I've never used one in a high gain audio circuit with no decoupling, either, hence the question of if that could have any effect.