Collect old wall-warts, if you don't have a power supply and don't want to keep spending money on batteries. Get some 12V, 9V, 5V, etc.
I like having some 7805s and 7809s around for when I actually build the final project, and it can't run off the bench supply anymore.
On a related note, old electronics are a great place to find cheap parts which are still perfectly usable. Recently I pulled an old UPS out of my neighbour's garbage pile and opened it up; there are lots of goodies in there like MOSFETs, regulators, various other ICs, and of course LEDs, resistors, caps, MOVs, etc. I have already used the internal wiring and spade terminals in another project (along with the relay from a burned-out RV furnace ignitor board). And I got a pair of 6V 4.5Ah SLA batteries out of the UPS, which is always nice.
Regards,
Torben