That happens to me all the time.
Here's the problem: All PCs have a configurable boot order list. They try to boot from the first device on the list, and if there's nothing there to boot from, they check device number two. Your PC's boot order lists the USB ports before the hard drive. When you boot the PC, it checks to see if there is a storage device plugged into a USB port. If there is, it tries to boot from that device. If the device lacks an operating system, the PC hangs.
It would be smarter, of course, if the PC went to the next boot device instead, as it would if it found an unbootable disc in the CD/DVD drive. But the PC isn't that smart.
You can get around this problem by changing the boot order. I can't tell you exactly how because it depends on your BIOS. When you turn on your PC, watch for a message (one of the first things to display) that tells you to press a particular key (say, F2) for Setup. Press that key immediately for the setup menu. Search the menu for boot or boot order options, and when you find the right option, change it.