The panel you have probably does a good job for the four batteries, if you can keep it pointing directly at the sun. Unfortunately, it takes hours to charge batteries properly, so you would need to keep moving the panel. A better way would be to use a much larger panel, 2-3 times the voltage you need to charge the batteries and regulate it down.
The panels from garden lights are usually 3-4 volts, 20-50 mA, and charge low capacity batteries (280 mAh), and take a full day of sun. You would need to add quite a few.