Since the count button, driving the transistor, serves as clock for the 4510, the display should not change but once for each press of the button. The way I read your post, you are indicating the counter is continuously counting either up or down by simply pressing the count button.
If this is so, and you are getting more than one count per button press, either the debounce is not working properly, or there is a mis-wiring somewhere in the circuit to pin 5 of the 4510.
Double check this portion of your circuit.