CAN unresponsive after securing S32K14x

[webfp46]

Short question: Can anybody tell me why our S32K14x no longer responds to CAN commands after the flash is secured? And how it can be unsecured when the CSEc stores keys.

Complete question:

We are using an S32K144 and S32K146 in the product we are developing, both chips in different versions of the product, however running roughly the same source code. For this product it is important that the software is secured and cannot be read from the chip using a debugger to protect the IP that is part of this software. Therefore, we are making use of the flash securing features of the S32K14x.

Also the software should be updateable over CAN and transmitted securely so that the software cannot be read in the process. So we have created a bootloader that is able to receive the encrypted firmware and decrypt this firmware using the keys which are securely stored in the CSEc. The flash configuration of the MCU is set by the bootloader program which programs the first flash sector: 0x00000 – 0x10000.

The problem we are experiencing is that when we secure the flash by setting:

    .long 0xFFFF7FFB     /* FDPROT:FEPROT:FOPT:FSEC */

The device (in the application and in the bootloader) no longer responds to any commands over CAN and is no longer programmable. We figured that the flash could not be unsecured by a mass erase as the CSEc is enabled (keys are stored) as mentioned here:  https://community.nxp.com/t5/S32-Design-Studio/S32K146-MCU-quot-Device-is-secure-Erase-to-unsecure-q...

However, running firmware (which was already loaded in the application image) that erases the first sector and then overwrites it with the following flash configuration does not unsecure the image as well. We did verify that the application code is run by the bootloader.

       .long 0xFFFF7FFE     /* FDPROT:FEPROT:FOPT:FSEC(0xFE = unsecured) */

Does anybody know why the device becomes unresponsive over CAN after securing the flash? And is there any way to unsecure the device when we have keys stored in the CSEc? Can the CMD_DBG_CHAL and CMD_DBG_AUTH commands also be run by the applications code or only by an external debugger?

Thanks,

Frank

[lukaszadrapa]

Hi Frank,

If your code does not contain a functionality which would run CMD_DBG_CHAL and CMD_DBG_AUTH commands based on some external event (like command from UART/CAN, pushing a button…) AND if the device is secured AND no Backdoor Access is implemented, there’s no way to recover because the mass erase is disabled. The only option is to replace the chip.

It doesn’t matter if CMD_DBG_CHAL and CMD_DBG_AUTH commands are initiated by code running on the device or by JTAG/SWD interface. But, of course, the device can’t be secured or you need to have implemented Backdoor Access to be able to do that via JTAG/SWD.

https://community.nxp.com/t5/S32K-Knowledge-Base/Example-S32K144-Verify-Backdoor-Access-Key-S32DS1-3...

Securing of the flash does not affect the application. It affects only debugging capabilities / SWD access. If changing of flash configuration field changed the behavior of your application, it’s probably just some side effect and it’s caused by something else.

To check this, it will necessary to use new device. For development purposes, my recommendation is to implement some features which will allow you to do simple recovery. For example, use some external pin(s) to generate an interrupt which will run Verify Backdoor Access Key to temporarily unsecure the device to be able to debug such situation. Or you can directly reprogram first sector – change flash configuration field back to unsecured state. Or you can run CMD_DBG_CHAL and CMD_DBG_AUTH commands to disable the CSEc, so it would be possible to execute mass erase again.

Regards,

Lukas

[webfp46]

Hi Lukas,

Thank you for your quick reply. This is a viable option to work with for us. I assumed the CMD_DBG_CHAL and CMD_DBG_AUTH commands should be run by a debugger.

I tried to implement both options you mentioned on another device:

- Running CMD_DBG_CHAL and CMD_DBG_AUTH commands to reset the CSEc so a Mass Erase can unsecure the device.

- Running the verify backdoor access command and thus unsecuring the device and then erasing the first sector and overwriting it with an unsecured flash configuration.

Everything worked perfectly when the device was unsecured and the CSEc was reset (mainly visible by the keys being programmable with counter 1 and the FlexNVM partitioning being reset) and the FSEC register was rewritting.
After having build some confidence in the solution I secured the flash with the same code still running (the unsecuring was done in the application, the flash config is part of the bootloader). However now the device is still secured and does not unsecure itself. Also the Device did become unresponsive again on the CAN bus so I assume something might get messed up when securing the device and therefore it no longer runs the code to unsecure it correctly. 

Do you maybe have any idea what might cause the device to no longer function when being secured?

Thanks,

Frank

EDIT:
In an attempt to isolate the issue I started running examples 1 and 5 from AN5401SW. Using example 1 to configure and load keys into the CSEc and using example 5 to reset the CSEc to factory default conditions. This works well when I build example 1 to run from RAM and example 5 to run from flash. (When the CSEc is configured from flash the device crashes, as expected).

I can secure the device and recover it using a mass erase using the following steps:

1. Load example 1 into RAM (unsecured) to configure the CSEc and load keys
2. Load example 5 into FLASH (unsecured) to reset the CSEc
3. Load example 5 into FLASH (secured: FSEC set to 0xFF).
4. Perform a mass erase to unsecure the device.
5. I am able to upload any code and debug as normal now.

However when I perform skip step 2 the device is bricked:

1. Load example 1 into RAM (unsecured) to configure the CSEc and load keys
2. Load example 5 into FLASH (secured: FSEC set to 0xFF) to secure the device and then also reset the CSEc.
3. Perform a mass erase to unsecure the device, however this fails
4. The device is now bricked and I can no longer attach a debugger.

So it seems that the device is unable to reset the CSEc when the device is secured. When I have reset the CSEc beforehand the mass erase does work. Should I run the command to reset the CSEc from RAM when the device is secured or perform any additional steps?

The software that was used for this test is attached.

The flashconfig:

    .section .FlashConfig, "a"
    .long 0x76543210     // Backdoor comparison key
    .long 0xFEDCBA98     // Backdoor comparison key                                            
    .long 0xFFFFFFFF     // FPROT0-3
    .long 0xFFFF7FBB     /* FDPROT:FEPROT:FOPT:FSEC */

The code I am using:

int main(void) {  
	  uint16_t __attribute__((unused)) csec_error = 0; //1 means No Error
	  csec_error = INIT_RNG(); /* Initialize the Random Number Generator before generating challenge */
	  // Load Master ECU key
	  csec_error = loadMasterECUKey();
	  // Erase all keys and reset the part to the factory state
	  uint32_t MASTER_ECU_KEY_VALUE[4] ={0xa2435381, 0x2c4b929f, 0x4c044a8a,0xfbf4b729};
	  csec_error = resetCSEc(MASTER_ECU_KEY_VALUE);

  // Initialize program flash memory controller
  PROGRAMFLASH_memoryInit();
  /* Check the Verify backdoor access key command */
  PROGRAMFLASH_unsecureDevice();
}

// Reset the CSEc to delete the keys and reconfigure partition
uint16_t resetCSEc(uint32_t *master_ecu_key_value){
	  uint16_t  csec_error = 0; //1 means No Error

	  // Factory reset the CSEc
	  uint32_t dbg_challenge_out[4] = {0,0,0,0};
	  csec_error = DBG_CHAL(dbg_challenge_out);
	  csec_error = DBG_AUTH(dbg_challenge_out,master_ecu_key_value);

	  return csec_error;
}

/* Issue Debug challenge command */
uint16_t DBG_CHAL(uint32_t *dbg_challenge_out)
{
	while((FTFC->FSTAT & FTFC_FSTAT_CCIF_MASK) != FTFC_FSTAT_CCIF_MASK); //Check for the ongoing FLASH command

	FTFC->FSTAT = (FTFC_FSTAT_FPVIOL_MASK | FTFC_FSTAT_ACCERR_MASK);// Write 1 to clear error flags */

	/* Start command by wring Header */
	command_header= (CMD_DBG_CHAL << 24) | (CMD_FORMAT_COPY << 16) | (CALL_SEQ_FIRST <<  | 0x00; //Write to Page0 Word0, Value = 0x12000000
	Start_Command(command_header);

	csec_error_bits = CSE_PRAM->RAMn[1].DATA_32 >> 16; //Read Page0 Word1, Error Bits

	for(i=4,j=0;i<8;i++,j++)
		dbg_challenge_out[j] = CSE_PRAM->RAMn[i].DATA_32;

	return csec_error_bits;
}

/* Issue Debug Authorization command */
uint16_t DBG_AUTH(uint32_t *dbg_challenge_out,uint32_t *master_ecu_key_value)
{
	uint32_t K_out[4];

	uint32_t UID[4] = {0,0,0,0};
	uint32_t UID_MAC[4] = {0,0,0,0};
	uint32_t authorization[4];

	//First Calculate the Authorization
    csec_error_bits = GET_UID(UID, UID_MAC);

	csec_error_bits = KDF(K_out, master_ecu_key_value, DEBUG_KEY_C);

	csec_error_bits = LOAD_RAM_KEY(K_out, RAM_KEY);

	uint32_t DATA[8];
	for(i=0; i<4; i++)
		DATA[i] = dbg_challenge_out[i];
	for(;i<8;i++)
		DATA[i] = UID[i-4];

	csec_error_bits = CMAC(authorization, DATA, RAM_KEY, 248);

	//Now actually issue authorization command
	while((FTFC->FSTAT & FTFC_FSTAT_CCIF_MASK) != FTFC_FSTAT_CCIF_MASK); //Check for the ongoing FLASH command

	FTFC->FSTAT = (FTFC_FSTAT_FPVIOL_MASK | FTFC_FSTAT_ACCERR_MASK);  // Write 1 to clear error flags

	for(i=4,j=0; i<8; i++,j++) //Write to Page1
		CSE_PRAM->RAMn[i].DATA_32 = authorization[j];

	/* Start command by wring Header */
	command_header= (CMD_DBG_AUTH << 24) | (CMD_FORMAT_COPY << 16) | (CALL_SEQ_FIRST <<  | 0x00; // Write to Page0 Word0, Value = 0x13000000
	Start_Command(command_header);

	csec_error_bits = CSE_PRAM->RAMn[1].DATA_32 >> 16; //Read Page0 Word1, Error Bits

	return csec_error_bits;
}

// Unsecure device
void PROGRAMFLASH_unsecureDevice() {
    flash_drv_status_t ret;        /* Store the driver api return code */
    const uint8_t backdoorAccessKey[8] = {0x10, 0x32, 0x54, 0x76, 0x98, 0xBA, 0xDC, 0xFE};

	// Disable interrupts during flash commands
	INT_SYS_DisableIRQGlobal();
	ret = FlashSecurityBypass(&flashSSDConfig, backdoorAccessKey,pCmdSequence);
	INT_SYS_EnableIRQGlobal();
}

flash_drv_status_t FlashSecurityBypass(const flash_ssd_config_t * pSSDConfig, \
                                       const uint8_t* keyBuffer, \
                                       flash_command_sequence_t pFlashCommandSequence)
{
#ifdef DEV_ERROR_DETECT
    DEV_ASSERT(pSSDConfig != NULL);
    DEV_ASSERT(keyBuffer != NULL);
    DEV_ASSERT(pFlashCommandSequence != NULL);
#endif
    flash_drv_status_t ret;    /* Return code variable */
    uint32_t temp;             /* Temporary variable */
    uint8_t i;

    /* Clear RDCOLERR & ACCERR & FPVIOL flag in flash status register. Write 1 to clear */
    CLEAR_FTFx_FSTAT_ERROR_BITS;

    /* Passing parameter to the command */
    FTFx_FCCOB0 = FTFx_SECURITY_BY_PASS;
    for (i = 0U; i < 8U; i++)
    {
        temp = FTFx_BASE + i + 0x08U;
        *(uint8_t *)temp = keyBuffer[i];
    }
    ret = pFlashCommandSequence(pSSDConfig);

    return (ret);
}

[lukaszadrapa]

Hi Frank,

that's interesting, I need to find some time to test it on my side. In the meantime, could you tell me which debugger you use? Which version of S32DS? And once the device is bricked, is reset asserted or is it toggling or is it deasserted? What kind of message did you get in the debugger?

I will get back to you as soon as possible. I'm quite overloaded now, so expect about 2-3 days.

Regards,

Lukas

[webfp46]

Hi Lukas,

I am using the S32DS version 2.2 (S32DS.ARM.2.2) with the 'S32KxxRTM SDK 3.0.0' SDK (with some modifications to run the SDK as PIC), GCC6.3 (arm-none-eabi-gcc) as build toolchain and the PEmicro Multilink Universal rev D as debugger while using the SWD protocol.

When I say the device is 'bricked', it does not mean that it does no longer function, only that we cannot communicate with it over CAN and we can no longer attach a debugger to the device. The devices do however boot, run the bootloader and end up in the application code. The LEDs show us that the device is running code and is not in an odd state, although CAN communication does not work.

The error that is given by the debugger is that the GDB server cannot establish a connection to the target processor (see screenshot). The first time the board is secured and program it asked "Device is secure. Erase to unsecure?". I have 'Emergency Kinetis Device Recovery by Full Chip Erase' enabled in my debug configuration. When I disable the full chip erase option the question "Device is secure. Erase to unsecure?"  pops back up when trying to debug. Clicking yes again gives the error in the screenshot

The reset pin is in all cases, secured and unsecured 5V just as the supply voltage of the Chip. When trying to program the device the reset line get asserted (pulled low). See the oscilloscope picture below. During normal operation the line stays a steady 5V.

The reset line when trying to program a secured device (emergency device recovery enabled)

The reset line when programming an unsecured device (for comparison):

 

Kind regards,

Frank

[lukaszadrapa]

Hi Frank,

finally I'm back with some results.

I built my own test project to test CSEc and security at the same time. I used UART interface to print some status messages and to send commands for required operations.

The most important result - I was able to successfully run CMD_DBG_CHAL and CMD_DBG_AUTH commands from flash memory even if the device was secured. After that, I'm able to run mass erase command via debug interface. So, there are no hardware limitations, it works as expected on my side. 

I tried to ask if security can affect some other functionality but I don't think so, I'm not aware of anything. I will let you know later.

Regards,

Lukas

[webfp46]

Hi Lukas,

Thank you for testing. And good to hear that it should be possible in hardware. 

Then still I am riddled why it does not work in our case. Can you share the software you used to test this so I can compare it with our software to maybe spot the difference or gradually change your solution over to ours to figure out what change causes it to work or not?

Thanks,

Frank

[lukaszadrapa]

Hi Frank,

the project is attached. It's definitely not a nice one, I just put more examples together in a hurry.

For CSEc initialization, I used original project 1_Configure_part_and_Load_keys from AN5401 without modification. Then I loaded this one to the flash.

I used S32K144EVB, S32DS for ARM 2.2, serial port LPUART1 via USB/OpenSDA on pins PTC6/PTC7, 9600 speed and Tera Term on PC.

Regards,

Lukas

[webfp46]

Hi Lukas,

Thanks. I will try to reproduce your results and trace the problems.

Frank

[webfp46]

Hi Lukas,

It took some time but we were finally succesfull to create working software that is able to successfully run the DBG_CHAL and DBG_AUTH commands when called via the CAN bus. The software does reset the CSEc and then we can perform a mass erase to unsecure the flash via a debugger.

I still don't know why it didn't work before but I got your example working and then gradually integrated all the changes into our software. For other people with a comparable issue: one of the key changes that  made this work was to run all of the CSEc commands from RAM (I moved the full CSEc_functions.o into the .code_ram section).

So now we have a functional way to unsecure our product when the keys are set.

However the issue we run into now is that we require a reset line to be connected between the debugger and the MCU to be able to perform the mass erase when the device is secured (as outlined here: https://community.nxp.com/t5/Kinetis-Microcontrollers/Mass-Erase-through-SWD-without-access-to-reset...). The problem is that our debug connector does not have a reset line so this will be impossible in the field.

A possible solution we are thinking of is to execute the 'Erase All Blocks Unsecure' Flash operation to erase all flash and unsecure the device. Then we should be able to program the MCU normally over SWD without the reset line. It this will work is still to be tested.

Thanks you very much already for all your work and support. I really helped us get to this point.

Kind regards,

Frank

[webfp46]

Hi Lukas,

Eventually we got everything working and stable. This includes unsecuring the flash from the device without needing to mass erase using a programmer as the devices don't have their debug pins exposed. So now everything works, we can secure our firmware and unsecure it over CAN.

Thanks a lot for your help it really helped us forward.

For anybody struggeling with the same or similar problems of trying to get the device to run the 'Erase All Blocks Unsecure' Flash operation after the DBG_CHAL and DBG_AUTH commands and running into stability issues: the following 2 things solved this for us:

1. Adding a small delay between the DBG_AUTH command and the Erase All Blocks Unsecure operation. (All with code in RAM)
2. Disabeling interrupts before running the DBG_CHAL command (and leaving them disabled for the rest of the process).

Kind regards to all,

Frank