The Spice simulator does a "initial dc bias solution" before allowing all the circuit nodes to be determined as a function of time. During the bias solution, the input voltage source is set to -1mV. Since the opamp + input is grounded, the - input must be essentially at ground. Since during a DC solution there can be no steady-state current through the input coupling capacitor, the current in the input resistor, and the feedback resistor must be zero as well. That means that the input capacitor must be charged so that its left end is at -1mV, while its right end is at 0V. As the simulation begins, the input voltage source varies from -1mV to +1mV, with an average value of 0V. So to go from a net -1mV to 0V across the capacitor, the capacitor must "discharge" through the input resistor with a time-constant of R1*C. That "discharge" curve is super-imposed on the normal square-wave that comes from the input signal.
This behavior matches the real world. Any amplifier with a DC blocking capacitor at its input will follow a step input, and then that will revert back to its previous level with a time-constant of RC.
A simple solution to making the turn on transient disappear from your sim is to use the .IC (initial conditions) at the opamp - input node to -0.5mV. Another is to just simulate for a longer time, and then only plot the output after the initial transient has died out.