Hi eungsoo Lee,
Regarding your questions:
1. when you connect using VN1610, Is Comm Mode set to Vector CANCardXLE?
ANS: CANCardXLE option is used for Vector boards that are connected over the PCI with the Host PC.
For Vector devices that are connected via USB you should use the "CaseXL" option.
2. Is there any other way I can verify that CAN communication is connected?
ANS: One simple way - is to create a small program that broadcast a CAN message from the processor. This way you will validate that the CAN msgs will go out.
Now, coming back your error:

It looks like you have not configured the setup correctly because the RAppID is not able to connect with the Vector Devices.
In the RAppIDBL_UserManual.pdf there is a chapter called "Enabling Vector Hardware" where you need to open the Vector Hardware Config application and assign RAppID_BL to one of the CAN channels. Have you configured this step ?

To conclude, i would approach this problem as:
- step#1: build an application that will transmit a counter value over CAN all the time (infinite loop).
- step#2: then load this applicaiton via JTAG
- step#3: using any CAN device i would check that i can receive those messages and decode them correctly. This will ensure the CAN baud rate and CAN network is OK (perhaps you need to lower the CAN speed and/or add a 120ohm termination resistor)
- step4#: configure the RAppID Bootloader download the code via Serial (this will validate that your code will fit into S12ZVM flash.
- step5#: configure the RAppID Bootloader to use the Vector device and then download the code via CAN interface
Hope this helps!
Best regards,
Daniel