It seems that I didn’t read far enough about the 74154. It says in the description: Each 4-bit address drives one output low. All others stay high. Enable inputs E1 (18) & E2 (19) must be low. If one or both are high, all outputs go low. Down the page they show an example referring to pin 18 (E1) as DATA IN. With the description: Selected output is low when DATA IN is low. If DATA IN is high, selected output is high.
So, wire pin 18 high and forget the inverters. Also it looks like if you send it a 0000, pin 1, output #0 will be active. If you want all relays off when you send it a 0000, you should start with pin 2, output #1.