NOTE
Flash accesses (reads/writes) by a bus master other than the core, DMA
controller, or writes to flash by the core during programming must use the
backdoor flash address of IPSBAR plus an offset of 0x0400_0000. For
example, for a DMA transfer from the first location of flash when IPSBAR
is at its default location of 0x4000_0000, the source register would be
loaded with 0x4400_0000. Backdoor access to flash for reads can be made
by the bus master, but it takes two cycles longer than a direct read of the
flash if using its FLASHBAR address.
So I think you want to leave that unchanged.
Did you try it unchanged?
-- Rich