It is not going to serial download mode when the eMMC is blank, it is loading from the SD card. That is not expected.
I cannot verify I can boot from the eMMC on my custom board without burning fuses, because it is connected to USDHC1, not USDHC3. And once I burn the fuses, I lose the option to boot from the SD card, as described in my original post.
As previously mentioned, I have already verified that the boot mode pins are being read correctly.
I only have two prototype boards, so I am not going to risk burning the fuses on the second one until I have a solution to this. Do you have access to the source code of the boot ROM program? Reading that would seem to be the only way to answer this question.