VA is simply the transformer RMS secondary current multiplied by the RMS secondary voltage, without regard to phase shift. Think of it like Watts.
The maximum RMS secondary current that the transformer will deliver is 24VA/110V=0.218A.
The problem with your relay is that the coil is rated for 100VDC, but your transformer puts out AC. If you just use a single diode as a half-wave rectifier, you will deliver unfiltered half-cycle pulses of DC current to the coil of the relay, and it will buzz loudly. If you put a several uF capacitor (rated at ~150VDC) across the relay coil, it will probably quiet the buzzing, but it is likely to raise the DC voltage across the coil much above 100V.
Measure the DC resistance of the coil with your Ohmmeter, and post the value here. With that information, we can devise a rectifier-filter-resistor network which will operate the relay coil at the correct 100VDC.