PICs are NOT capable of interfacing directly with USB devices; they can act as USB devices to a computer, but they cannot act as a USB host to another device. It could be done using an external USB host interface chip, however those are also very complicated devices and it would NOT be a reasonable project for someone who is a beginner (or even intermediate) with PICs.
SD memory cards are a simpler option, they can be interfaced via the SPI serial protocol which a PIC can easily handle, and they're about the same price as USB thumb drives anyway. If you add a card reader (inexpensive) to your computer, and implement the FAT16 file system (rather than reading/writing raw data on the memory card), you can easily access the card directly from your computer as well.