Mike,
The Serial Download Protocol can just write a memory location, that is memory mapped. A SD card requires a driver, and is not a memory directly accessible by the core. Therefore, you can't write directly to a SD, to a NAND flash, NOR flash, ...
This means that the write file must be done typically to a RAM location where the code will be executed: SDRAM, SRAM, internal RAM, NAND flash buffer,...
Actually, the file should be loaded at the same place than if it was loaded by the ROM code during boot. This address is the one given in the "Boot Data Structure" defined in the manual (7.6.1.2): "start: Absolute address of the image".
This is typically somewhere in the SDRAM, and hopefully the address where the code was linked.