OK, after a bit of digging I've answered my own question. Seems I had the MISCCR register configured to supply internal clock to the USB core, but with our clock speed the PLL can't supply 60MHz. I need to supply an external clock source via the USB_CLKIN GPIO.