When you have two dimensional coordinates, say (x, y), you can plot that point using some basis vectors. Usually the coordinate point is plotted using the x and y axes of the cartesian coordinate system (the basis vectors being i and j).
You can also define your own basis vectors. If you look at the matrix:
[ cosθ sinθ]
[-sinθ cosθ]
You notice that the column vectors are unit vectors (their length is one) and that the vectors are also orthogonal. When you multiply this matrix by a vector containing your coordinates x and y:
[ cosθ sinθ] [x]
[-sinθ cosθ] [y]
You notice that the x coordinate scales the first column vector and the y coordinate scales the second column vector.. and finally the two vectors are added.
So, the matrix basically defines new basis vectors to represent your coordinate system (relative to some "world" coordinate system).
Draw the column vectors [cosθ -sinθ]' and [sinθ cosθ]' on a paper (with some fixed angle θ) and you will easily see how the system works.