You are just looking to switch them so you don't have to unplug the wire each time right ? Not trying to use 2 active hosts or anything like that right ?
USB can drive upto 450mA if I remember right (I'm probably wrong on that, but its pretty high up there).
So that rules out a few digital SPST switches. Maybe mechanical switches or solid state relays ?
Maybe MOSFETS ? I just learned all about low On Resistance mosfets, you could get tricky with them,
Having just reaseched digital switches myself, and if you are going to stay in spec, 450mA is going to be a pain.
Thats about all I can help with,
good luck