It sounds like you want to setup your 16x16 matrix a little like the circuit below? If so, you can probably use 2.7-amp PFET column drivers but I'm not sure I've ever seen "standard" 20-ma LEDs with a "peak" or "pulsed" current spec' as high as 160-ma...
Have you considered driving your 16x16 matrix as four individual 8x8 matrices? Perhaps using four '5821 row drivers and 32 PNP column drivers? The advantages would be an overall 12.5% display duty cycle and using cheapie "off-the-shelf" 600-ma 2N4403 or 800-ma PN2907A PNP column driver transistors... And it would only require a single 8-bit Port to drive the columns and three pins to drive the '5821's...
Food for thought... Have fun... Regards, Mike