Kyle
It may be that the frames are not arriving at all to your board (if possible, repeat the tests using a hub connected to the board and sniffer (like Ethereal) to see whether the frames are really there.
Switches in the network often use some form of store and forward mechanism and if you are sending lots of UDP frames there are often buffer overflows in them. This shows that short fast UDP frames without a reliable protocol on top can be quite unreliable even in local networks!!
Also take a look at the uTasker project since it allows you to run the M5223X code on a PC in a simulator where over flows are very unlikely - it is useful for detecting problems releated to the hardware or of purely software nature.
Regards
Mark
www.uTasker.com