But it occurred to me that a potentiometer at the output stage would have challenges because of impedance mismatch problems and also the high current.
Impedance mismatch isn't a problem, as matching is rarely required, and must not be matched in this kind of case.
The high current IS a problem though, hence the need for a big pot.