Turned out not to be quite as straightforward as the various videos showed.
I took out the two bottom screws, and then used the suction cup to try to lift the display off of the case. However, the adhesive that holds the display together had let go, and only the top part of the display came loose. The rear housing of the display was still firmly stuck into the case. Took me a while to figure out how to get that out. I did some gentle poking, prodding and prying, and finally got that part out.
Changing the battery was easy enough once that was done. Then I had to fix the display. I glued it back together with superglue, putting a dab of glue in the same places as the original adhesive. Reassembled everything and crossed my fingers, hoping that the display would still work, and that the new battery would be okay. The display was okay, but the phone went through several power up cycles after I connected the charger. I thought maybe the battery connector wasn't making good contact. I gave the phone a few good shakes, and taps on the table, and the power hasn't cycled since then. Hopefully, it'll be okay now. I'll have to keep an eye on it for a while though.