Your code seems ok to me. Only thing i can imagine is that the order of register writes could be important. Some of the registers like MOD have "write buffers" that only take effect later.
When I tested quadrature encoder mode the following code worked for me:
It is counting pulses on Phase A with the abbility to count up or down according to Phase B
// Flextimer 1 Setup / Quadrature Decoder Mode for counting pulses
SIM->SCGC6 |= (1UL << 25);/* Enable FTM1 Clock */
SIM->SCGC5 |= (1UL << 10);/* Enable Port B Clock */
PORTB->PCR[0] = 0x00000600; /* setup pin for FTM1 PHA */
PORTB->PCR[0] = 0x00000600; /* setup pin for FTM1 PHB */
FTM1->MODE = 0x00000005; /* enable advanced FTM registers */
FTM1->CNTIN = 0x00000000; /* initial counter value after overflow */
FTM1->CNT = 0x00000000; /* current count, writing reloads CNTIN */
FTM1->MOD = 0x0000FFFF; /* overflow count -> free running counter */
FTM1->SC = 0x00000048; /* FTM clock = 48MHz / 8 = 6.0 MHz, TOF interrupt enabled */
FTM1->FILTER = 0x000000044; /* input filter on PHA and PHB, adjusted for counting up to 1MHz signals */
FTM1->QDCTRL = 0x000000D9; /* quadrature decoder ensabled*/
NVIC_EnableIRQ(FTM1_IRQn); /* Enable FTM1 Interrupt */
So try changing the order of your register writes. Hope this helps.