I've since figured this out.
Just use the examples for the 54608 that come in the SDK and go to properties in the project and set
the MCU setting to 54606. It will prompt you saying your changing the MCU.
Everything works fine after then rebuilding the project.
You can then download thru SWD just fine.
I was getting wrong part number in the beginning.
It wasn't obvious that this is what your supposed to do,
to reuse the 54608 SDK examples with the other 546XX MCU's.