...And I really only need it to count from 1-4 so 3 or 4 bit would work.
Actually, if it only needs to count 1,2,3,4 it has only four total states, so it only needs to be a 2bit counter. 2Λ2=4.
So, it needs two inputs; Dir (0=count down, 1=count up) and clock (count on every rising edge).
Next question, what happens if you keep counting up? e.g. does it go 1,2,3,4,4,4,... forever until the Dir is changed? Or does it roll over, as in 1,2,3,4,1,2,3,4,1,...
Similarly, what happens if you keep counting down? roll or repeat at 1?
Also, does it need to have an asynchronous Clear input that sets the counter to 1 independent of the Clock?
Do you need to have a separate decoded output to signify which state the counter is in?
Maybe you should describe in more detail what you are trying to do with the counter.
This can be done with a dual flip-flop and a few gates. Answer my questions and I can help you with it.