Ok, I see the problem. You cannot ground the non-inverting input! It has to be biased somewhere near Vcc/2, or you can leave it grounded if you operate the opamp on split-supplies.
The gain from each pot wiper to the output is -Rfb/Rin. So that the Rin doesn't load the pot, make it ~10 times the Pot resistance, i.e. I would use 2K pots, and make Rin about 20K. If you want a gain of -1, then make Rfb 20K too.
Make a voltage divider consisting of two 10K resistors tied to Vcc and Gnd. Tie the center tap to the non-inverting input, and then the output and the two inputs will be relative to Vcc/2. If you set the two pots exactly halfway (2.5V?), the output will be at 2.5V. Offsetting either pot from the mid point will move the output in the opposite direction from 2.5V. Moving them both a like amount will move the output twice as much...
Better yet, make the gain only -0.5, then each input individually can swing from 0 to 5V without the output of the opamp being overdriven. Look at this.