After level shifting and/or inverting, you can have the level on the input of the op amp (the inverter/level shifter) such that with zero volts, the oscillator will be at its design center frequency value. With a positive voltage, the frequency will go above that center frequency and with a negative voltage, the frequency will go below that center frequency. With inversion on the VCF input, a positive will drive the frequency down from center and a negative up from center.
A simple op amp circuit will perform inversion, level shifting and gain modification (i.e., VCF input voltage-to-output frequency ratio) all at once.
Dean