All aircraft autopilot gyros I have played with have a heading bug with an analog bipolar differential output, meaning something like +100mV when 30deg right of course, zero when on course, and -100mV when left of course.
Two LM339 IC comparators configured as a "window", where neither is high while the signal remains between say -5mV and +5mV would keep the boat on a heading within a couple of degrees of where the heading bug is set. To turn the boat on the reciprocal track, you would just have to rotate the heading bug to the reciprocal course on the compass card.
The only problem is the
cost of a new or overhauled gyro. I bought an overhaulable core off ebay for about $200, and it cost another $400 to have it overhauled and recertified. You might be able to pick up a working used one for a few hundred $.
To do this on the cheap, I would just use the NEMA output from an inexpensive GPS (even a blind GPS that can be had for $35). Feed that into a USB port on a laptop. Write some code which parses the NEMA and extracts the previous track info. When you push a key, the program begins comparing the current track to the track that existed when the key was pushed. Based on the difference, you use a couple of pins on the serial port to control your steering...
Another way is to use a GPS with NEMA output on RS232. Interface that to a PIC which is programmed to parse NEMA sentence to extract the track. Then do what is described above, where two port pins steer the boat...
This product is available for homebuilt aircraft.
For what you want to do, I think it is better if you use "track" vs "heading".