Hi Dan,
Speaker cones are supposed to swing forward and rearward. Your transistor drives it in only one direction with the other direction missing. It will have severe distortion. Real amplifiers have a transistor like yours to pull the speaker in the positive direction plus a PNP transistor to push it negative.
Your transistor doesn't have any voltage gain. The speaker's output level will be very low when the transistor is driven by the 100mV signal from a tuner, tape deck, CD player or MP3 player.
The minimum number of parts for a speaker amplifier is 3 transistors, a few resistors and a couple of capacitors. The 3rd transistor provides voltage gain.
I would show an amp like that to you but it sounds like crap because it has such a few number of parts.
You could make a pretty good, cheap amp with a power amplifier IC. They use about 30 transistors but cost the same as two.