Hi, Suraj,
Of course, it is okay not to use the LPC4322 for the lpc54628 target board, you can connect the SWD device to download and debug code.
you can connect the SWD signals to a 10 pins mail connector, you can use LPC-link2, or J-Link devices to download code and debug with Keil or MCUXpresso, IAR tools.
I copy the main connection from data sheet of LPC54628.
Hope it can help you
This is lpc-link2 board link:
https://www.nxp.com/design/microcontrollers-developer-resources/lpc-microcontroller-utilities/lpc-li...
The J-Link device is from third party segger.
BR
XiangJun Rong
xiangjun_rong_0-1608618615713.png