Rizwan,
First I would recommend to arrange your data in the same endianess, it will save some struggling, but I am guessing is something you already looked at... What is this data you are reading? is general data or Graphics? Is there something you could let me know to understand more about the target application?
Did you looked at the QSPI_MCR[END_CFG] I understand from the manual you can change the endianess of what you are reading.
My initial guess is that you can't have byte read access to the RX buffers, have you tried this using the CPU? to see how it affects byte accesses on the RX buffers?
If QSPI_MCR[END_CFG] does not work on your specific case, maybe is better if you use the mode: AHB Buffer data read via memory mapped access. That mode is exactly as if you were reading OCRAM or DDR, can have byte/word read accesses because the controller maps the memory to certain memory address range. If your eDMA configuration re arranges bytes correctly from RAM to RAM it will work from QuadSPI to RAM.
Regards,
Ioseph