Please read the "User Manual" :smileyalert: of LPC1768 more carefully. There are a lot hints inside, otherwise it will not work.
Also search Internet/NXP forum for questions about IAP, Bootloader, ...
ISP is mainly used for downloading from PC via serial port, e.g. via Flash Magic. Regular firmware (flashed before) is not running and is most of the time replaced by a new one. Is over serial port wants uuencoded data.
IAP is used, when you want to write own data into flash. Data can also be a new application or a new bootloader into flash. With IAP functions you use binary data (not hex nor uuencoded data). Pay attention that a certain SRAM area must be kept free to be used by IAP. There are other things to pay attention to like disabling interrupts or possibly moving executable code to RAM before deleting/writing new pgm data, otherwise you are cutting the tree you are sitting on...
Also look for bootloader examples, not limited to Ethernet, but also other peripherals.
Search for AN10866 ( http://www.nxp.com/documents/application_note/AN10866.pdf )