Most USB ports on most machines are configured for Safe Removal rather than Performance thus no caching and slower reads and writes but minimal rise to the files and structure on the USB drive. But not all machines are configured that way, and one way to increase performance is to maintain the File Allocation Table in RAM. Quick removal with the current FAT in RAM will destroy your USB drive.
Basically you can yank the flash drive out whenever you want and see whether your lucky or not. So long as you have a backup, or two, of your files the damage is nothing more than an inconvenience.
