The problem is that the 35 LEDs in each matrix are too many to have a wire for each one. That is why they are wired in rows and columns so that there are only 12 wires. They are designed to be lit one row or one column at at time, very quickly, and rely on persistence of vision to make it look as though all the lights are on.
LEDs are supposed to be lit with controlled currents. It's normal to put a resistor in series with an LED to control the current, and there are various LED calculators on the web to tell you what resistor you need. With the 5x7 matrix you can't separate out all 35 LED currents, so you can control all of the currents at the same time. That problem goes away if you light each column in turn, as you are only lighting 7 LEDS, with 7 connections, and each can have a controlled current while it is lit.
However, I suggest that you try resistors on both the columns and rows. For 12 V, and allowing about 15 mA per LED, you would get 75 mA for each of the rows and 105 mA for each of the columns. You've got about 2 V drop on the LEDs, leaving about 10 V. If you split that evenly, you have 5 V for the row resistors and also 5 V for the column resistors. Then you need 66.666 ohms (use 68 ohms as it is a standard value) producing 0.36 W (use a 1 W resistor) for each row. On the columns you want 47 ohms, also rated at 1 W.
That will ensure that each row and each column takes the same current, but there could still be variations between LEDs. I don't know how much that will be, so you'll have to try it. You must make sure that all the rows and columns are connected before turning on the power if you use this scheme, because the 75 mA or 105 mA is too much for one LED.