After talking with some FAEs from Avnet, we discovered that the Config Tools do not generate the code to wait for the XO_READY bit. After adding this line in clock_config.c, it is working fine.
CLOCK_SetupExtClocking(24000000U); /* Enable XTALHF clock */
while(!(ANACTRL->XO32M_STATUS & ANACTRL_XO32M_STATUS_XO_READY_MASK)); /* Wait for XO ready */