The pot resistance you are using is too high. Likely, you effective load resistance is a few Ohms, and you are using 10K or higher pots.
Determine effective load resistance you are trying to drive. Make the rheostat (pot) resistance about 2 to 5 times higher than that.
Suppose you have a 1.5V motor that draws 0.25A when it is running. That means its effective resistance is E/I = 1.5/0.25 = 6Ω. To vary the speed of this motor, it would take rheostat of ~20Ωmax. When the rheostat is set to ~6Ω, the motor would run at half speed.