No, the AMX232 will blow it up, for sure. You cannot use it directly.
USE ONLY a USB to 3.0 V TTL converter. Not sure if 3.3 is a problem, but I suspect it is. 3 V or greater is DEFINATELY a problem.
But, you are correct that Tx gets connected to Rx and vice versa unless you can change the type of port DTE/DCE.
On a bad note, the user manual link for the TTL-USB converter doesn't work for me. On a good note, the chip that it's based on
https://www.electro-tech-online.com/custompdfs/2012/08/PL2303-1.pdf is OK for 3, 3.3 and 5V levels.
And, it's entirely POSSIBLE that if 3.3V was used, the modem is toast, but as I said, RS232 or TTL RS232 is tough to get to work.
Baud has to be right. Handshaking has to be right etc. Parity, data, stop etc all have to be right.
You can disable handshaking by playing with CTS and RTS and DSR and DTR. Looping CTS to RTS usually works. I have to look at the datasheet again to figure out if DTR or DSR is used and if it's an input or an output. Since this is a modem, it should be DSR and it would be an output.
So, connecting CTS and RTS looped at the modem and Rx connected to Tx and Tx connected to Rx and handshaking disabled with Hyperterminal, you should be able to find the baud rate by trial and error. Don't forget SG or signal ground.
Then you have to get handshaking to work for real data transmission to occur reliably.