Probably a problem with your code... Some suggestions:
- removing the power causes a dip in power that is detected as a brown-out, and you have a handler
- there is a USB interrupt dfor device removal that is not being handled
- there is a fault being generated
Whatever the problem, you will need to use a debugger to find the cause