I agree that the insulation won’t be rated at higher voltage, but it’s difficult to know at what voltage it will break down.
The magnetic core of any mains transformer will be running near saturation when running at its rated voltage and frequency. The manufacturer wouldn’t waste material to have more core than needed.
The result is that applying too much voltage to one of the windings will saturate the core and the inductance will be reduced, so the current will be large. However, at higher frequency it will take more voltage to saturate the core. The relationship is linear, so at twice the frequency, it will take twice the voltage to saturate the core.
Increasing the frequency will increase the eddy current heating in the core, but most toroidal transformers have ferrite cores, so the heating at high frequency wouldn’t be as much as with a laminated core.
The insulation is likely to be the limit if you can increase the frequency.