OK, to better understand this lets first look at the relay contacts. You have a 3 pole double throw relay (3PDT).
Pin 1 is the Common for pin 3 Normally Open and Pin 4 Normally Closed. Pin 6 is the Common for Pin 7 Normally Open and Pin 5 Normally Closed. Pin 11 is the Common for Pin 9 Normally Open and Pin 8 Normally Closed. Three sets of double throw contacts, N/O and N/C and each with their own Common.
You may want to print the relay drawing.
With the relay de-energized:
The positive motor power flows through pins 5 and 6 (Normally Closed) contacts of RY1. This applies a Positive to Connection 2 of the motor. At the same time, the negative motor power flows through pins 4 and 1 (Normally Closed) contacts of RY1. This applies a Negative to Connection 1 of the motor.
With the relay energized (12 volt power applied to pins 2 and 10):
The Positive motor power flown through pins 3 and 1 (Normally Open now closed) contacts of RY1. This applies a Positive to Connection 1 of the motor. At the same time, the Negative motor power flows through pins 7 and 6 (Normally Open now closed) contacts of RY1. This applies a Negative to Connection 2 of the motor.
The remaining relay contact set is unused, however, could be used if one wanted lamps or LEDs to indicate motor rotation such as Forward / Reverse or if a linear actuator was shimming In / Out.
The relay configuration is very, very common and used frequently for reversing a motor. Since I can't see your layout and connections I can't guess why it will not work. So here you have a detailed explanation of how it works.
<EDIT> I see KISS is up and about.
</EDIT>
Ron