When I was a kid, my dad made me something very much like this. It was really cool and I spent many hours on it. It's probably the single biggest thing that made me want to get into electronics.
If I recall, my electronics box (or whatever you wanted to call it) was enclosed in a wooden frame with a plexiglass top; this see-through top is imperative to make the child wonder about the stuff inside. The top would also simply slide out when one end of the wooden frame was removed (ie, there were grooves on the top edges of the wood). This is also a good thing to do because as the child gets older, it means that he/she can easily open up the box to make modifications to the stuff inside.
My box was split into two main sections. One section was a low voltage side that ran from a 6V wall wart, the other was a 12V side that ran off of another wall wart. The 12V side was fused. On the 6V side, there were a few different things to play with, including some LEDs, a basic audio amplifier, and a basic frequency generator.
The frequency generator was just a 555 timer with a bunch of different resistors hooked up in parallel with push buttons. There was also a potentiometer in parallel. The output of the 555 would go directly to a speaker that was mounted under the plexiglass (with holes drilled to let the sound through). The result is that when you push the buttons and play with the pot you can get different tones out of the speaker.
The audio amplifier was hooked up to the same speaker through a spdt switch, and it has just a simple pot to control volume. It would be neat to also add some active filters/amps that could be used to "morph" the sound from the mic.
The 12 volt side basically had one main feature; there were a number of RCA jacks that various things could be plugged in to. above each RCA jack was also a screw terminal (in case the item to be plugged in didnt have a male plug to fit the RCA jack). Each of these jacks had a rotary switch that was setup to switch the outputs on or off, in either polarity. There were also LEDs hooked up in each direction on the output so that one LED would light when the polarity is one way, and the other would light when its in the other direction. The purpose of these jacks was that you could get interesting electric machines to plug in to the board. Fun examples of stuff to plug in would be a vibrating motor (as someone mentioned earlier), a normal motor, maybe a small electric car with geared back wheels, an electric bell, an electromagnet, various incandescent lamps, etc..