Soldering USB switch... speed considerations?

I am building a simple two-way USB switch to be used as a KVM switch. My wireless mouse and keyboard pair to a single RF dongle that I want to switch between 2 computers on my desk.

The connection is basically 2 male USB plugs and 1 female USB socket for the dongle to go in; I am keeping all GNDs tied together, and the VCC, D+ and D- lines will each be switched by a 3P2T switch (i.e., connect to computer A or computer B)

Is the communication speed on USB fast enough that I would need a fancy PCB layout, or would it be possible to do on perfboard?

also is it OK to keep the GNDs shorted together? I know PCs use switching power supplies, but I think both sides are referenced to earth ground, right? I dont want to create any crazy ground loops. I could find a 4p2t switch if that makes more sense.

Thanks!

Perf board should be fine.

However, as such units are freely available why build one?.