32kHz watch crystals are designed to be driven with very low drive power. They are usually driven by an oscillator that is powered on a 1.5V battery, biased so that the oscillator barely starts, and consumes uW of power.
If driven hard, as they would be in a CMOS or TTL biased gate oscillator operated off 5V, the crystal self-heats, and stability goes to hell.
If you really want long-term stability, you will need to start with a crystal in the 4MHz range and follow that up with a suitable counting chain. Even better, put the 4MHz in an oven. Even better, use a "GPS-disiplined" oscillator.