Hi,
There could be several reasons why this might happen.
As the core is moved into the primary coil, it is subject to more and more ampere turns. If there is any DC current present in the coil it could saturate the core once it gets far enough in, or even the AC part could be affecting the saturation of the core. Also, as the core enters the primary center it is excited more and more, which could mean more core loss and so attenuation of the signal.
The 'core' does not look like it is made for magnetic applications, and this means it will function very poorly anyway even if it did work at all. Cores made for magnetic applications are made from special materials that are known to have good magnetic properties. A steel nail for example has some 100 to 1000 times LESS effectiveness as a core than a real transformer core made from real transformer metal.
You could try other materials, but you probably dont have anything worthy of a magnetic application laying around.
You could also try reducing the AC input amplitude to the primary and see if you get better results. Less excitation could mean better performance.