I've now found the cause of the problem (the patch from @PeterChen isn't relevant).
It is not a kernel panic, it's a kernel lock up in the ChipIdea udc driver.
The function hw_ep_flush never returns. It is waiting for the hardware to complete endpoint flush that never happens.
https://elixir.bootlin.com/linux/latest/source/drivers/usb/chipidea/udc.c#L101
I've fixed this lock up by simply adding a timeout to that loop. If the flush doesn't complete we just continue as if it had. The next connection still works.
Further testing shows that there are other lock conditions that can occur (approx 1 in 200 connections) during connection/disconnection. There are a number of other loops in that code that will not exit unless the hardware complies.
I don't know if this hardware problem exists in other implementations using the ChipIdea driver but I can recreate it on multiple boards.