The direct line connection is ok if you don't have any exposed metal, serial port, etc or any connections or grounds to any parts which do. Some types of buttons and switches would probably be inadvisable too.
The other prob is you still need to power the device. A 5v device taking even 10mA will make 1.7W of heat in a linear regulator running on rectified-filtered 120VAC! Also difficult to find 5v regs that run off of 160VAC. So you need a fancier solution at that point.