You don't say what the design current is, but you may be limited by the size of the input capacitor.
If you are taking 1 A at 5 V, that will mean that the average current is around 200 mA at 24 V. I'll assume 60 Hz mains, and with half-wave rectification, the capacitor has to keep the supply going for around 16 ms each cycle, so it needs to produce around 0.0033 Coulombs of charge. A 100 μF capacitor will drop by 33 V if you take that much charge from it, so basically your circuit will be starved of power each mains cycle.
Switch-mode regulators take more current as the input voltage decreases, so you should have a good reserve of energy in the capacitor. I would aim for around a 1 V drop. If you change to a full-wave rectifier, and the output current is 1 A, then you need around 0.0016 Coulombs of charge. I would suggest that you use something like a 1500 μF capacitor, although you might get away with less.
Layout and good connections are important for power supplies like that. I suggest soldering, not the breadboard.