LittleFS is a file system used for microcontroller internal flash and external NOR flash. Since it is more suitable for small embedded systems than traditional FAT file systems, more and more people are using it in their projects. So in addition to NOR/NAND flash type storage devices, can LittleFS be used in SD cards? It seems that it is okay too. This article will use the littlefs_shell and sdcard_fatfs demo project in the i.mxRT1050 SDK to make a new littefs_shell project for reading and writing SD cards. This experiment uses MCUXpresso IDE v11.7, and the SDK uses version 2.13. The littleFS file system has only 4 files, of which the current version shown in lfs.h is littleFS 2.5. The first step, of course, is to add SD-related code to the littlefs_shell project. The easiest way is to import another sdcard_fatfs project and copy all of the sdmmc directories into our project. Then copy sdmmc_config.c and sdmmc_config.h in the /board directory, and fsl_usdhc.c and fsl_usdhc.h in the /drivers directory. The second step is to modify the program to include SD card detection and initialization, adding a bridge from LittleFS to SD drivers. Add the following code to littlefs_shell.c. extern sd_card_t m_sdCard; status_t sdcardWaitCardInsert(void) { BOARD_SD_Config(&m_sdCard, NULL, BOARD_SDMMC_SD_HOST_IRQ_PRIORITY, NULL); /* SD host init function */ if (SD_HostInit(&m_sdCard) != kStatus_Success) { PRINTF("\r\nSD host init fail\r\n"); return kStatus_Fail; } /* wait card insert */ if (SD_PollingCardInsert(&m_sdCard, kSD_Inserted) == kStatus_Success) { PRINTF("\r\nCard inserted.\r\n"); /* power off card */ SD_SetCardPower(&m_sdCard, false); /* power on the card */ SD_SetCardPower(&m_sdCard, true); // SdMmc_Init(); } else { PRINTF("\r\nCard detect fail.\r\n"); return kStatus_Fail; } return kStatus_Success; } status_t sd_disk_initialize() { static bool isCardInitialized = false; /* demostrate the normal flow of card re-initialization. If re-initialization is not neccessary, return RES_OK directly will be fine */ if(isCardInitialized) { SD_Deinit(&m_sdCard); } if (kStatus_Success != SD_Init(&m_sdCard)) { SD_Deinit(&m_sdCard); memset(&m_sdCard, 0U, sizeof(m_sdCard)); return kStatus_Fail; } isCardInitialized = true; return kStatus_Success; } In main(), add these code if (sdcardWaitCardInsert() != kStatus_Success) { return -1; } status = sd_disk_initialize(); Next, create two new c files, lfs_sdmmc.c and lfs_sdmmc_bridge.c. The call order is littlefs->lfs_sdmmc.c->lfs_sdmmc_bridge.c->fsl_sd.c. lfs_sdmmc.c and lfs_sdmmc_bridge.c acting as intermediate layers that can connect the LITTLEFS and SD upper layer drivers. One of the things that must be noted is the mapping of addresses. The address given by littleFS is the block address + offset address. See figure below. This is a read command issued by the ‘mount’ command. The block address refers to the address of the erased sector address in SD. The read and write operation uses the smallest read-write block address (BLOCK) of SD, as described below. Therefore, in lfs_sdmmc.c, the address given by littleFS is first converted to the byte address. Then change the SD card read-write address to the BLOCK address in lfs_sdmmc_bridge.c. Since most SD cards today exceed 4GB, the byte address requires a 64-bit variable. Finally, the most important step is littleFS parameter configuration. There is a structure LittlsFS_config in peripherals.c, which contains not only the operation functions of the SD card, but also the read and write sectors and cache size. The setup of this structure is critical. If the setting is not good, it will not only affect the performance, but also cause errors in operation. Before setting it up, let's introduce some of the general ideal of SD card and littleFS. The storage unit of the SD card is BLOCK, and both reading and writing can be carried out according to BLOCK. The size of each block can be different for different cards. For standard SD cards, the length of the block command can be set with CMD16, and the block command length is fixed at 512 bytes for SDHC cards. The SD card is erased sector by sector. The size of each sector needs to be checked in the CSD register of the SD card. If the CSD register ERASE_BLK_EN = 0, Sector is the smallest erase unit, and its unit is "block". The value of sector size is equal to the value of the SECTOR_SIZE field in the CSD register plus 1. For example, if SECTOR_SIZE is 127, then the minimum erase unit is 512*(127+1)=65536 bytes. In addition, sometimes there are doubts, many of the current SD cards actually have wear functions to reduce the loss caused by frequent erasing and writing, and extend the service life. So in fact, delete operations or read and write operations are not necessarily real physical addresses. Instead, it is mapped by the SD controller. But for the user, this mapping is transparent. So don't worry about this affecting normal operation. LittleFS is a lightweight file system that has power loss recovery and dynamic wear leveling compared to FAT systems. Once mounted, littleFS provides a complete set of POSIX-like file and directory functions, so it can be operated like a common file system. LittleFS has only 4 files in total, and it basically does not need to be modified when used. Since the NOR/NAND flash to be operated by LittleFS is essentially a block device, in order to facilitate use, LittleFS is read and written in blocks, and the underlying NOR/NAND Flash interface drivers are carried out in blocks. Let's take a look at the specific content of LittleFS configuration parameters. const struct lfs_config LittleFS_config = { .context = (void*)0, .read = lfs_sdmmc_read, .prog = lfs_sdmmc_prog, .erase = lfs_sdmmc_erase, .sync = lfs_sdmmc_sync, .read_size = 512, .prog_size = 512, .block_size = 65536, .block_count = 128, .block_cycles = 100, .cache_size = 512, .lookahead_size = LITTLEFS_LOOKAHEAD_SIZE }; Among them, the first item (.context) is not used in this project, and is used in the original project to save the offset of the file system stored in Flash. Items two (.read) through five (.sync) point to the handlers for each operation. The sixth item (.read_size) is the smallest unit of read operation. This value is roughly equal to the BLOCK size of the SD card. In the SD card driver, this size has been fixed to 512. So for convenience, it is also set to 512. The seventh item (.prog_size) is the number of bytes written each time, which is 512 bytes like .read_size. The eighth item is .block_size. This can be considered to be the smallest erase block supported by the SD card when performing an erase operation. Here the default value is not important, you need to set it in the program according to the actual value after the SD card is initialized. The card used in this experiment is 64k bytes as an erase block, so 65536 is used directly here. Item 9 (.block_count) is used to indicate how many erasable blocks there are. Multiply the .block_size to get the size of the card. If the card is replaceable, it needs to be determined according to the parameters after the SD card is initialized. The tenth item (.block_cycles) is the number of erase cycles per block. Item 11 (.cache_size) is about the cache buffer. It feels like bigger is better, but actually modifies this value won't work. So still 512. Item 12 (lookahead_size), littleFS uses a lookahead buffer to manage and allocate blocks. A lookahead buffer is a fixed-size bitmap that records information about block allocations within an area. The lookahead buffer only records the information of block allocations in one area, and when you need to know the allocation of other regions, you need to scan the file system to find allocated blocks. If there are no free blocks in the lookahead buffer, you need to move the lookahead buffer to find other free blocks in the file system. The lookahead buffer position shifts one lookahead_size at a time. Use the original value here.  That’s all for the porting work. We can test the project now. You can see it works fine. The littleFS-SD project can read/write/create folder and erase. And it also support append to an exist file. But after more testing, a problem was found, if you repeatedly add->-close->-add-> close a file, the file will open more and more slowly, even taking a few seconds. This is what should be added and is not written directly in the last block of the file, but will apply for a new block, regardless of whether the previous block is full or not. See figure below. The figure above prints out all the read, write, and erase operations used in each write command. You can see that each time in the lfs_file_open there is one more read than the last write operation. In this way, after dozens or hundreds of cycles, a file will involve many blocks. It is very time-consuming to read these blocks in turn. Tests found that more than 100 read took longer than seconds. To speed things up, it is recommended to copy the contents of one file to another file after adding it dozens of times. In this way, the scattered content will be consolidated to write a small number of blocks. This can greatly speed up reading and writing.

RT10XX RT-UFL modification for QSPI QE and DQS factor 1. Abstract Recently, a customer used a QSPI flash (Puya simi P25Q16H) as XIP memory in the RT1050 project, but always encountered the phenomenon that the first time download failed, the download succeeded again after powering on again, and the app could run. To the program algorithm, they use the RT-UFL. After analysis, this situation is usually related to the fact that the QE of the new QSPI flash is not enabled. Therefore, based on the QE position of the QSPI flash used by the customer, the author specially enabled the corresponding QE in the SDK flexspi_nor_polling_transfer code, let the customer try to run it in RAM to check whether still have the program issues after enabling QE in the new QSPI flash. However, the customer even can’t run flexspi_nor_polling_transfer project. According to the customer's previous description, the hardware can run RAM code, and the first flash download does not work, but it can run after re-downloading, so the hardware works. Based on the phenomenon, it is initially speculated that the new problem may be related to FlexSPI DQS being occupied. Under normal circumstances, it is recommended to leave FlexSPI DQS floating. Because the project flexSPI frequency given to customers is 120Mhz, if DQS is used, the internal sampling clock source of FlexSPI read data is: Read strobe provided by memory device and input from DQS pad. This method will have problems. So asked the customer to confirm the hardware again. The result is DQS is used as a control pin for other circuits on the customer's board. Usually there are two points to note in this situation: First, the FlexSPI clock is controlled within 60MHz. Second, the internal sampling clock source configuration of FlexSPI read data is: Dummy read strobe generated by FlexSPI controller and looped back internally (FlexSPIn_MCR0[RXCLKSRC] = 0x0)      Therefore, this article focuses on how to prepare the test code for the corresponding QE position based on the QSPI flash used by the customer, consider the operation when DQS is enabled, modify and test the RT-UFL downloading algorithm. 2. Hardware and software prepare To reproduce the customer issues, need to prepare the related software, hardware, and the flash programming flashdriver, and the code for testing the QE situation. 2.1 Hardware prepare MIMXRT1050-EVKB, modify the on board resistor, from the default hyperflash to QSPI flash. The modification points: USE QSPI FLASH(Mount R153~R158, DNP R356,R361~R366)。 Remove the on board U33 ISSI QSPI flash, burn the new QSPI flash with customer used Puya simi P25Q16H. Customer is using JLINK, so prepare JLINK plus for downloading. 2.2 flexspi_nor_polling_transfer software prepare SDK2.14.0 code：flexspi_nor_polling_transfer, used to test the QE situation. App project: led_blinky RT-UFL program algorithm code: https://github.com/JayHeng/RT-UFL JLINK driver: used JLINKV768B, higher version is also OK. 2.2.1 P25Q16H QE position Fig 1    We can see, it is still the typical Status register bit 9. The related LUT write and read commander is: Fig 2 We can see that for writing, it is command 0X01, and 2 consecutive bytes need to be written. But for the read command, the commands for the two status register bytes are separate. So you need to pay attention to this when operating the QE bit. 2.2.2 flexspi_nor_polling_transfer code prepare This code is mainly used to test the QE enablement and disabling, and the erase, write and read functions of external flash. The code modification points include: modifying the LUT command to comply with P25Q16H; adding QE read, write and erase functions; modifying the frequency of flexSPI and the situation of DQS loopback internal. The relevant code is as follows: LUT related commander: flexspi_nor_polling_transfer.c const uint32_t customLUT[CUSTOM_LUT_LENGTH] = { /* Normal read mode -SDR */ [4 * NOR_CMD_LUT_SEQ_IDX_READ_NORMAL] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x03, kFLEXSPI_Command_RADDR_SDR, kFLEXSPI_1PAD, 0x18), [4 * NOR_CMD_LUT_SEQ_IDX_READ_NORMAL + 1] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_READ_SDR, kFLEXSPI_1PAD, 0x04, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0), /* Fast read mode - SDR */ [4 * NOR_CMD_LUT_SEQ_IDX_READ_FAST] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x0B, kFLEXSPI_Command_RADDR_SDR, kFLEXSPI_1PAD, 0x18), [4 * NOR_CMD_LUT_SEQ_IDX_READ_FAST + 1] = FLEXSPI_LUT_SEQ( kFLEXSPI_Command_DUMMY_SDR, kFLEXSPI_1PAD, 0x08, kFLEXSPI_Command_READ_SDR, kFLEXSPI_1PAD, 0x04), /* Fast read quad mode - SDR */ [4 * NOR_CMD_LUT_SEQ_IDX_READ_FAST_QUAD] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0xEB, kFLEXSPI_Command_RADDR_SDR, kFLEXSPI_4PAD, 0x18), [4 * NOR_CMD_LUT_SEQ_IDX_READ_FAST_QUAD + 1] = FLEXSPI_LUT_SEQ( kFLEXSPI_Command_DUMMY_SDR, kFLEXSPI_4PAD, 0x06, kFLEXSPI_Command_READ_SDR, kFLEXSPI_4PAD, 0x04), /* Read extend parameters */ [4 * NOR_CMD_LUT_SEQ_IDX_READSTATUS] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x81, kFLEXSPI_Command_READ_SDR, kFLEXSPI_1PAD, 0x04), /* Write Enable */ [4 * NOR_CMD_LUT_SEQ_IDX_WRITEENABLE] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x06, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0), /* Erase Sector */ [4 * NOR_CMD_LUT_SEQ_IDX_ERASESECTOR] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x20, kFLEXSPI_Command_RADDR_SDR, kFLEXSPI_1PAD, 0x18),//0xD7 /* Page Program - single mode */ [4 * NOR_CMD_LUT_SEQ_IDX_PAGEPROGRAM_SINGLE] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x02, kFLEXSPI_Command_RADDR_SDR, kFLEXSPI_1PAD, 0x18), [4 * NOR_CMD_LUT_SEQ_IDX_PAGEPROGRAM_SINGLE + 1] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_WRITE_SDR, kFLEXSPI_1PAD, 0x04, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0), /* Page Program - quad mode */ [4 * NOR_CMD_LUT_SEQ_IDX_PAGEPROGRAM_QUAD] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x32, kFLEXSPI_Command_RADDR_SDR, kFLEXSPI_1PAD, 0x18), [4 * NOR_CMD_LUT_SEQ_IDX_PAGEPROGRAM_QUAD + 1] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_WRITE_SDR, kFLEXSPI_4PAD, 0x04, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0), /* Read ID */ [4 * NOR_CMD_LUT_SEQ_IDX_READID] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x9F, kFLEXSPI_Command_READ_SDR, kFLEXSPI_1PAD, 0x04), /* Enable Quad mode */ [4 * NOR_CMD_LUT_SEQ_IDX_WRITESTATUSREG] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x01, kFLEXSPI_Command_WRITE_SDR, kFLEXSPI_1PAD, 0x04), /* Enter QPI mode */ [4 * NOR_CMD_LUT_SEQ_IDX_ENTERQPI] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x35, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0), /* Exit QPI mode */ [4 * NOR_CMD_LUT_SEQ_IDX_EXITQPI] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_4PAD, 0xF5, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0), /* Read status register */ [4 * NOR_CMD_LUT_SEQ_IDX_READSTATUSREG] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x05, kFLEXSPI_Command_READ_SDR, kFLEXSPI_1PAD, 0x04), /* Read status register */ [4 * NOR_CMD_LUT_SEQ_IDX_READSTATUSREG1] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0x35, kFLEXSPI_Command_READ_SDR, kFLEXSPI_1PAD, 0x04), /* Erase whole chip */ [4 * NOR_CMD_LUT_SEQ_IDX_ERASECHIP] = FLEXSPI_LUT_SEQ(kFLEXSPI_Command_SDR, kFLEXSPI_1PAD, 0xC7, kFLEXSPI_Command_STOP, kFLEXSPI_1PAD, 0),//0xC7 }; flexspi_nor_flash_ops.c: QE read and write status_t flexspi_nor_enable_quad_mode(FLEXSPI_Type *base) { flexspi_transfer_t flashXfer; status_t status; uint32_t writeValue = FLASH_QUAD_ENABLE; #if defined(CACHE_MAINTAIN) && CACHE_MAINTAIN flexspi_cache_status_t cacheStatus; flexspi_nor_disable_cache(&cacheStatus); #endif /* Write enable */ status = flexspi_nor_write_enable(base, 0); if (status != kStatus_Success) { return status; } /* Enable quad mode. */ flashXfer.deviceAddress = 0; flashXfer.port = FLASH_PORT; flashXfer.cmdType = kFLEXSPI_Write; flashXfer.SeqNumber = 1; flashXfer.seqIndex = NOR_CMD_LUT_SEQ_IDX_WRITESTATUSREG; flashXfer.data = &writeValue; flashXfer.dataSize = writeValue <= 0xFFU ? 1 : 2; status = FLEXSPI_TransferBlocking(base, &flashXfer); if (status != kStatus_Success) { return status; } status = flexspi_nor_wait_bus_busy(base); /* Do software reset. */ FLEXSPI_SoftwareReset(base); #if defined(CACHE_MAINTAIN) && CACHE_MAINTAIN flexspi_nor_enable_cache(cacheStatus); #endif return status; } status_t flexspi_nor_disable_quad_mode(FLEXSPI_Type *base) { flexspi_transfer_t flashXfer; status_t status; uint32_t writeValue = 0x0;//FLASH_QUAD_ENABLE; #if defined(CACHE_MAINTAIN) && CACHE_MAINTAIN flexspi_cache_status_t cacheStatus; flexspi_nor_disable_cache(&cacheStatus); #endif /* Write enable */ status = flexspi_nor_write_enable(base, 0); if (status != kStatus_Success) { return status; } /* Enable quad mode. */ flashXfer.deviceAddress = 0; flashXfer.port = FLASH_PORT; flashXfer.cmdType = kFLEXSPI_Write; flashXfer.SeqNumber = 1; flashXfer.seqIndex = NOR_CMD_LUT_SEQ_IDX_WRITESTATUSREG; flashXfer.data = &writeValue; flashXfer.dataSize = 2; status = FLEXSPI_TransferBlocking(base, &flashXfer); if (status != kStatus_Success) { return status; } status = flexspi_nor_wait_bus_busy(base); /* Do software reset. */ FLEXSPI_SoftwareReset(base); #if defined(CACHE_MAINTAIN) && CACHE_MAINTAIN flexspi_nor_enable_cache(cacheStatus); #endif return status; } status_t flexspi_nor_QE_register(FLEXSPI_Type *base, uint32_t *QEvalue) { /* Wait status ready. */ bool isBusy; uint32_t readValue; status_t status; flexspi_transfer_t flashXfer; flashXfer.deviceAddress = 0; flashXfer.port = FLASH_PORT; flashXfer.cmdType = kFLEXSPI_Read; flashXfer.SeqNumber = 1; flashXfer.seqIndex = NOR_CMD_LUT_SEQ_IDX_READSTATUSREG1; flashXfer.data = &readValue; flashXfer.dataSize = 1; do { status = FLEXSPI_TransferBlocking(base, &flashXfer); if (status != kStatus_Success) { return status; } if (FLASH_BUSY_STATUS_POL) { if (readValue & (1U << FLASH_BUSY_STATUS_OFFSET)) { isBusy = true; } else { isBusy = false; } } else { if (readValue & (1U << FLASH_BUSY_STATUS_OFFSET)) { isBusy = false; } else { isBusy = true; } } *QEvalue = readValue; } while (isBusy); return status; } QE position：App.h #define FLASH_QUAD_ENABLE 0X0200 QE operation：flexspi_nor_polling_transfer.c PRINTF("Get the QE bit value before QE enable!\r\n"); uint32_t QEvalue=0; status = flexspi_nor_QE_register(EXAMPLE_FLEXSPI, &QEvalue); if (status != kStatus_Success) { return status; } PRINTF("QE=%X!\r\n",(uint8_t)QEvalue); #if 1 status = flexspi_nor_disable_quad_mode(EXAMPLE_FLEXSPI); if (status != kStatus_Success) { return status; } PRINTF("Get the QE bit value after QE disable!\r\n"); status = flexspi_nor_QE_register(EXAMPLE_FLEXSPI, &QEvalue); if (status != kStatus_Success) { return status; } PRINTF("QE=%X!\r\n",(uint8_t)QEvalue); #endif PRINTF("Enable the QE bit value !\r\n"); /* Enter quad mode. */ status = flexspi_nor_enable_quad_mode(EXAMPLE_FLEXSPI); if (status != kStatus_Success) { return status; } status = flexspi_nor_QE_register(EXAMPLE_FLEXSPI, &QEvalue); if (status != kStatus_Success) { return status; } PRINTF("QE=%X!\r\n",(uint8_t)QEvalue); FlexSPI frequency modification：flexspi_nor_polling_transfer.c，app.h flexspi_device_config_t deviceconfig = { .flexspiRootClk = 60000000, .flashSize = FLASH_SIZE, .CSIntervalUnit = kFLEXSPI_CsIntervalUnit1SckCycle, .CSInterval = 2, .CSHoldTime = 3, .CSSetupTime = 3, .dataValidTime = 0, .columnspace = 0, .enableWordAddress = 0, .AWRSeqIndex = 0, .AWRSeqNumber = 0, .ARDSeqIndex = NOR_CMD_LUT_SEQ_IDX_READ_FAST_QUAD, .ARDSeqNumber = 1, .AHBWriteWaitUnit = kFLEXSPI_AhbWriteWaitUnit2AhbCycle, .AHBWriteWaitInterval = 0, }; static inline void flexspi_clock_init(void) { const clock_usb_pll_config_t g_ccmConfigUsbPll = {.loopDivider = 0U}; CLOCK_InitUsb1Pll(&g_ccmConfigUsbPll); CLOCK_InitUsb1Pfd(kCLOCK_Pfd0, 24); /* Set PLL3 PFD0 clock 360MHZ. */ CLOCK_SetMux(kCLOCK_FlexspiMux, 0x3); /* Choose PLL3 PFD0 clock as flexspi source clock. */ CLOCK_SetDiv(kCLOCK_FlexspiDiv, 5); /* flexspi clock 60M. */ } Loop back internally：app.h #define EXAMPLE_FLEXSPI_RX_SAMPLE_CLOCK kFLEXSPI_ReadSampleClkLoopbackInternally 2.2.3 flexspi_nor_polling_transfer testing after modification Download the modified code to RT1050 RAM and run the results as follows:    Fig 3 From the figure, we can see, QE can normally implement the functions of reading, writing, erasing, and reading. It is read as 2 here which QE is enabled for the first time, because the QSPI in this article has been operated previously. If it is a new chip, it will read 0 by default, which means that QE is not enabled. And it can be seen that after modification, it can accurately erase, program, and read external flash, indicating that the current code modification is successful. LUT, QE position, DQS consideration (60Mhz+loopback internal) are all working. 2.3 APP prepare Use the led_blinky code in the SDK to mainly modify the frequency of FCB and readSampleClkSrc. evkbimxrt1050_flexspi_nor_config.c is modified as follows:    const flexspi_nor_config_t qspiflash_config = { .memConfig = { .tag = FLEXSPI_CFG_BLK_TAG, .version = FLEXSPI_CFG_BLK_VERSION, .readSampleClksrc=kFlexSPIReadSampleClk_LoopbackInternally, .csHoldTime = 3u, .csSetupTime = 3u, .controllerMiscOption = (1u << kFlexSpiMiscOffset_SafeConfigFreqEnable), .deviceType = kFlexSpiDeviceType_SerialNOR, .sflashPadType = kSerialFlash_4Pads, .serialClkFreq = kFlexSpiSerialClk_60MHz, .sflashA1Size = 8u * 1024u * 1024u, … }   This code will be used to test the new modified program flashdriver in debug mode, compile the project, generate the .srec, used for the JFLASH method flash program.   3. RT-UFL JLINK flash algorithm modification After downloading the super download algorithm RT-UFL, you need to modify the super download algorithm based on the two factors mentioned above: First, QE is enabled; second, DQS is used. For the solution in this article, RT-UFL still uses option ROM to initialize flexSPI. According to the options description, choose: OPTION 0: 0xc0000201 OPTION 1:0x0 Just like this situation: Fig 4 3.1 RT-UFL code modification Here, use the keil project: \RT-UFL-1.0\build\mdk The code modification is as follows: Ufl_main.c: ufl_set_target_property1 case kChipId_RT105x: uflTargetDesc->flexspiInstance = MIMXRT105X_1st_FLEXSPI_INSTANCE; uflTargetDesc->flexspiBaseAddr = MIMXRT105X_1st_FLEXSPI_BASE; uflTargetDesc->flashBaseAddr = MIMXRT105X_1st_FLEXSPI_AMBA_BASE; //p25q16h QESet bit 1 in Status Register 2 {.option0.U = 0xc0000201, .option1.U = 0x00000000}, uflTargetDesc->configOption.option0.U = 0xc0000201; uflTargetDesc->configOption.option1.U = 0x0; Ufl_romapi.c: readSampleClkSrc configuration status_t flexspi_nor_auto_config(uint32_t instance, flexspi_nor_config_t *config, serial_nor_config_option_t *option) { // Wait until the FLEXSPI is idle register uint32_t delaycnt = 10000u; while(delaycnt--) { } status_t status = flexspi_nor_get_config(instance, config, option); if (status != kStatus_Success) { return status; } config->memConfig.readSampleClksrc=kFlexSPIReadSampleClk_LoopbackInternally; //For DQS is used by other circuit return flexspi_nor_flash_init(instance, config); } FlashDev.c struct FlashDevice const FlashDevice = { FLASH_DRV_VERS, // Driver Version, do not modify! "MIMXRT_FLEXSPI", // Device Name EXTSPI, // Device Type 0x60000000, // Device Start Address 0x00800000, // Device Size in Bytes (8mB) 256, // Programming Page Size 0, // Reserved, must be 0 0xFF, // Initial Content of Erased Memory 100, // Program Page Timeout 100 mSec 5000, // Erase Sector Timeout 5000 mSec // Specify Size and Address of Sectors 0x1000, 0x00000000, // Sector Size 4kB (256 Sectors) SECTOR_END }; FlashOS.h: it will generate the UFL_L0 type, to define the flash page, sector size #define FLASH_DRV_SIZE_OPT (0) #if (FLASH_DRV_SIZE_OPT == 0) #define FLASH_DRV_PAGE_SIZE (0x100) #define FLASH_DRV_SECTOR_SIZE (0x1000) #elif (FLASH_DRV_SIZE_OPT == 1) #define FLASH_DRV_PAGE_SIZE (0x200) #define FLASH_DRV_SECTOR_SIZE (0x1000) #elif (FLASH_DRV_SIZE_OPT == 2) #define FLASH_DRV_PAGE_SIZE (0x200) #define FLASH_DRV_SECTOR_SIZE (0x10000) #endif Compile the code, it will get flashdriver firmware:MIMXRT_FLEXSPI_UV5_UFL.FLM Rename it to:MIMXRT_FLEXSPI_UV5_UFL_P25Q16H.FLM 3.2 JLINK driver flashdriver update After installing the JLINK driver,  modify it to use the RT-UFL algorithm.   According to this article, the driving algorithm of JLINK is modified to RT-UFL algorithm: https://www.cnblogs.com/henjay724/p/14942574.html   In fact, just copy: RT-UFL-1.0\RT-UFL-1.0\algo\SEGGER\JLink_Vxxx To the installed JLINK path: C:\Program Files\SEGGER\JLINKV768B   But this article need to based on this to add the modified flash algorithm for P25Q16H, the modification points are: （1）Copy attached file RT1050_P25Q16H_JLINK\program\ JLinkDevices.xml to： C:\Program Files\SEGGER\JLINKV768B Fig 5 The .xml modification is as follows, add the P25Q16H item and it’s algorithm： Fig 6 Note: device name is MIMXRT1050_UFL_P25Q16H （2） CopyRT1050_P25Q16H_JLINK\program\ IMXRT_FLEXSPI_UV5_UFL_P25Q16H.FLM to：C:\Program Files\SEGGER\JLINKV768B\Devices\NXP\iMXRT_UFL Fig 7 This MIMXRT_FLEXSPI_UV5_UFL_P25Q16H.FLM is the modified flashdriver algorithm in the above. （3）run C:\Program Files\SEGGER\JLINKV768B\JLinkDLLUpdater.exe, update the modified driver to the IDE IAR 3.3 Flashdriver algorithm downloading test For MIMXRT1050-EVKB, to use external JLINK, you need to disconnect J33 on the EVKB board and plug JTAG into J21. 3.3.1 Use JFLASH downloading test First, use the previously modified EVKB-IMXRT1050-flexspi_nor_polling_transfer, disable the QE bit, to simulate the new QSPI flash chip. The test is as follows:   Fig 8 JFlash test result is: Fig 9 We can see, use the Jflash with new flashdriver, can program the flash successfully. 3.3.2 led_blinky app debug test Disable the QE bit, to simulate the new QSPI flash chip, the test as Fig 8. APP demo use the IAR project(customer use it), option select JLINK： Fig 10 Fig 11 It should be noted here that the device is selected as the modified super download algorithm device name. The method is as follows. The settings->xxx.jlink generated by IAR debug is modified as follows: Fig 12 Two points: override =1, and device is the new modified algorithm device name. Debug test result: Fig 13 We can see that the algorithm can be successfully debugged and the algorithm is also modified by UFL. Run it at full speed and you can see the on board LED is flashing. It means that all flash driver algorithms, hardware, and codes already support the new P25Q16H QSPI flash. 4. Summary When using a new QSPI flash, first need to pay attention to the position of QE and whether DQS is used, and then prepare the corresponding RT-UFL programming algorithm. The UFL algorithm can usually support most flash chips by default. When QE and DQS are used, they only need to fine-tune the algorithm to support the new QSPI flash. Therefore, this article has successfully solved the problem of burning customer projects after modifying the algorithm. For other QSPI flash, you can also use the method in this article to modify the burning algorithm accordingly to ensure that it meets your own project needs.  

RT1170 AVB fresh tasting 1 Abstract AVB (Audio Video Bridging) is the audio and video bridging technology. AVB is also a time-sensitive network. It is mainly used to solve audio and video transmission problems within the local area network: delay problems and synchronization problems. AVB consists of a series of IEEE standards aimed at efficiently transmitting audio and video data in a local area network. The protocol of AVB is as follows: Fig 1 AVB is mainly a link layer protocol and coexists with the traditional TCP/IP protocol.     AVB related protocols include: gPTP (IEEE 802.1AS-2020): Precision time synchronization protocol AVTP (IEEE 1722-2016): Audio and video transmission protocol FQTSS (IEEE 802.1Q-2018, section 34): Credit Based Shaper protocol SRP (IEEE 802.1Q-2018, section 35) :Stream Reservation Protocol AVDECC (IEEE 1722.1-2013): Audio and video management protocol EST (IEEE 802.1Qbv-2015) FP (IEEE 802.3br-2016/IEEE 802.1Qbu-2016) The topology diagram of AVB is as follows: Fig 2 End Station：Listener andTalker Listener: Node that accepts audio and video data Talker: Node that outputs audio and video data AV Bridge：AVB bridge The purpose of this article is not to talk about the AVB protocol. As a first-time experience of AVB with RT1170, it mainly explains how to use the officially provided AVB/TSN protocol stack to implement the AVB audio data transmission function on the NXP MIMXRT1170-EVK board.。 2 .RT1170 AVB testing 2.1 Hardware prepare    2*MIMXRT1170-EVK REV C4，one is Talker，another is Listener.    Pin modification: remove R228, R234, R232, R229 resistor    Board default configuration:      J27:1-2      J5,J6,J7,J8:short connect      J38:5-6      SW1: 1-OFF,2-OFF,3-ON,4-OFF      SW2: 1-OFF,2-OFF,3-OFF,4-OFF,5-OFF,6-OFF,7-OFF,8-OFF,9-OFF,10-OFF      J11: code downloading and vcom port      J4  : 1G ENET, used for AVB communication Fig 3   2.2 Software tool prepare 2.2.1 Related code SDK_2_13_0_MIMXRT1170-EVK: https://mcuxpresso.nxp.com/en/builder?hw=MIMXRT1170-EVK&rel=667 Download the corresponding SDK according to the system which you need to use (windows/linux). For example, download the Linux version for Linux, and download the windows version for windows. This article takes the Linux compilation system as an example, so download the SDK Linux version. genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0.zip: https://mcuxpresso.nxp.com/en/dashboard?download=84124a72b3f5916f99168a06ef287f2f genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0 file includes: Fig 4 Binaries：include RT1170 AVB/TSN bin file, RT1050 AVB bin file. Doc：AVB/TSN stack related document, very important, recommend to read it at first. Genavb-apps-freertos-5_6_0:GenAVB/TSN app examples genavb-sdk –5_6_0.tar.gz GenAVB/TSN SDK supported configuration and device .patch:GenAVB/TSN patch to RT1170,RT1050 SDK   2.2.2 Related software tool Linux platform CMake (>= 3.10.2, tested with 3.16.3) make unzip patch Windows platform CMake (>= 3.10, tested with 3.22.1) minGW-w64 (tested with 4.3.5) 7zip Git with git bash (necessary for patch utility) This document use the linux ubuntu platform, the software version is: Fig 5 A small experience sharing, as the used Ubuntu version is lower, then the installed cmake version is also low, when install the cmake 3.16.3, meet some issues, here share the experience: At first, download cmake-3.16.3-Linux-x86_64.tar.gz： https://github.com/Kitware/CMake/releases/download/v3.16.3/cmake-3.16.3-Linux-x86_64.tar.gz unzip cmake-3.16.3-Linux-x86_64.tar.gz to get cmake-3.16.3-Linux-x86_64： tar -zxvf cmake-3.16.3-Linux-x86_64.tar.gz Add the linker to /usr/bin/cmake： sudo ln -s /home/nxa07323/TSN_GENAVB/cmake-3.16.3-Linux-x86_64/bin/cmake /usr/bin/cmake In /usr/bin/cmake path, use:ls -al, we can see the link information: Fig 6 After the above operation, use : cmake –version We can see the cmake version is cmake 3.16.3, which Fig 5 shows. 2.2.2 Code configuration   To build the AVB code, it also needs to use the RT1170 SDK, the steps are:   Unzip genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0.zip and SDK_2_13_0_MIMXRT1170-EVK_linux.zip Fig 7   Enter SDK_2_13_0_MIMXRT1170-EVK_linux folder path, add genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/ mcuxpresso-sdk-SDK_2_13_0_MIMXRT1170-EVK-5_6_0.patch  Patch add commander is: $ patch -p1 < path/to/mcuxpresso-sdk-SDK_2_13_0_MIMXRT1170-EVK-5_6_0.patch   The real operation command is: patch -p1 < /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/mcuxpresso-sdk-SDK_2_13_0_MIMXRT1170-EVK-5_6_0.patch Fig 8    Add the linker for the SDK and AVB SDK, the structure is: genavb-apps-freertos-5_6_0 ├── boards │ ├── evkbimxrt1050 │ │ ├── demo_apps │ │ └── mcu-sdk -> /path/to/SDK_2_13_0_EVKB-IMXRT1050 (required for RT1052) │ ├── evkmimxrt1170 │ │ ├── demo_apps │ │ └── mcu-sdk -> /path/to/SDK_2_13_0_MIMXRT1170-EVK (required for RT1176) │ └── src │ └── demo_apps └── gen_avb -> /path/to/genavb-sdk-5_6_0 To the RT1170, it contains 2 link: 1)Add the SDK path for the avb sdk board level $ cd path/to/genavb-apps-freertos-5_6_0/boards/evkmimxrt1170 $ ln -s path/to/SDK_2_13_0_MIMXRT1170-EVK mcu-sdk The used situation: For path /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0/boards/evkmimxrt1170 It needs to add the SDK path link: /home/nxa07323/avbdoc/SDK_2_13_0_MIMXRT1170-EVK_linux Firstly, unzip the genavb-apps-freertos-5_6_0.tar.gz： tar -zxvf genavb-apps-freertos-5_6_0.tar.gz Fig 9 The commander is: cd /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0/boards/evkmimxrt1170 ln -s /home/nxa07323/avbdoc/SDK_2_13_0_MIMXRT1170-EVK_linux mcu-sdk Fig 10   We can see, in the path genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0/boards/evkmimxrt1170, already add the SDK path with link, the name is: mcu-sdk. 2）Add the GENAVB/TSN SDK path link to AVB app top level $ cd path/to/genavb-apps-freertos-5_6_0/ $ ln -s path/to/genavb-sdk-5_6_0 gen_avb To /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0/ Add the AVB SDK path link:/home/nxa07323/avbdoc/SDK_2_13_0_MIMXRT1170-EVK_linux Unzip genavb-sdk-5_6_0.tar.gz: tar -zxvf genavb-sdk-5_6_0.tar.gz Fig 11 Link commander: cd /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0 ln -s /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-sdk-5_6_0 gen_avb Fig 12 We can see, /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0 path link also added, the name is gen_avb. 2.3 code build   The build file path: genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0/boards/evkmimxrt1170/demo_apps/avb_tsn/avb_audio_app/armgcc use build_release.sh file for the linux version. Commander: ./build_release.sh Fig 13 Result: Fig 14 We can see, the avb_app.bin already be generated. File path: /home/nxa07323/avbdoc/genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0/genavb-apps-freertos-5_6_0/boards/evkmimxrt1170/demo_apps/avb_tsn/avb_audio_app/armgcc/release Open avb_app.bin file： Fig 15     We can see, the generated bin file’s RT1170 QSPI FCB. 2.4 Code programming 2.4.1 MSD downloading method     By referring the doc: genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0\doc\NXP_GenAVB_TSN_Stack_FreeRTOS_Eval_User_s_Guide-5_6_rev0.pdf. It is recommended to use the MSD method and directly copy app_avb.bin to the MSD of EVK. However, in the actual test, since there is no information for successful burning progress after copying, it is easy to cause problems. For example, after copying the bin file to the MSD disk (it seems that the copy has been completed), reset or power off the board, but the burning has not actually been completed. At this time, there will be a problem that the code has not been successfully burned, and AVB cannot be successfully run. Therefore, if you use the MSD method, after copying to the MSD, it is best to wait for a period of time to ensure that the code is successfully burned, such as 30s. Fig 16 However, after many tests, the MSD method cannot perform chip mass erase. For example, some filesystem configurations of AVB have been made, but the code needs to be re-downloaded for reconfiguration. It is found that even if app_avb.bin is burned again, the previous AVB configuration is still exists, so if you need a new system file configuration, it is recommended to perform a full chip erase first and then use this method to burn, or directly use the serial download mode to burn.     In addition, some customers may update EVK's opensda version, and sometimes the updated opensda may not have MSD. In this way, it is also recommended to use the following serial download MCUBootutility method to burn.    2.4.2 MCUBootutility downloading method At first, EVK enter the serial download mode: SW1:1-OFF,2-OFF,3-OFF,4-ON Use two USB cables to connect the J11 and J20 SDP, then power off and power on the board or reset the board to enter the serial download mode. MCUBootutility tool download link: https://github.com/JayHeng/NXP-MCUBootUtility/releases/tag/v5.3.0 Tool related document: https://github.com/JayHeng/NXP-MCUBootUtility   Use the generated avb_app.bin file, and burn it to the MIMXRT1170-EVK board: Fig 17 It should be noted here that from Figure 15 you can see that the bin file stores the FCB starting from 0, but for the MIMXRT1170 chip, the FCB storage is offset by 0X400, so the burning position of avb_app.bin needs to start from 0X30000400. Follow the 7-steps in Figure 17 to burn the bin file. After burning in this way, change SW1 of EVK to: 1-OFF, 2-OFF, 3-ON, 4-OFF, which is the internal boot mode.   AVB testing requires two MIMXRT1170-EVK development boards. The codes burned on the two boards are the same, avb_app.bin is burned in both.   After restarting and testing, you will find that the old filesystem has also been cleared. At this time, you can configure a new system file again.     2.5 talker listener configuration   Two MIMXRT1170-EVK board, one as AVB Talker, another as AVB Listener. Because the programmed app is the same, then just need to use the filesystem to configure which EVK is talker, which is listener.   This document use 2 EVK, and with the back-to-back method, the connection is: Fig 18 Talker: collect the Mic audio data, then transfer to the Listener through the AVB network Listener：receive the talker audio data, play the talker mic audio data through Audio Out J33 2.5.1 Talker filesystem configuration Talker related configuration commander is: ---------------------------------------- cd .. ls mkdir avb_app write avb_app/mclock_role 0 mkdir avdecc write avdecc/btb_mode 0 mkdir fgptp write fgptp/gmCapable 1 mkdir port0 write port0/hw_addr 00:22:33:44:55:66 ----------------------------------------------- The description for the talker configuration: avb_app/mclock_role=0, Media Clock Master. avdecc/btb_mode=0, avdecc back-to-back mode fgptp/gmCapable=1，gPTP grand master port0/hw_addr=00:22:33:44:55:66, configure the hardware address For hw_addr, it should be noted that the first byte must be 00. The author previously configured it as 11, but the communication was always unsuccessful. Through EVK's J11 serial port configuration, after powering on, you can see that the terminal prints a lot of data. First, you need to enter the shell and press the ISERT key. At this time, the terminal will appear >>. For specific shell commands, you can view the documentation: NXP_GenAVB_TSN_Stack_FreeRTOS_Eval_User_s_Guide-5_6_rev0.pdf, Chapter 6.1.1 Filesystem commands. The mainly used commander is: write: write a file with a given string cat: print the content of a file ls: list all files and directories in the current directory rm: remove a file or a directory (if it is empty) cd: change directory pwd: print working directory mkdir: create a directory The following picture is for the shell entry and the file system configuration: Fig 19  According to the Talker command mentioned above, configure the talker file system: Fig 20   Until now, the talker filesystem is configured. 2.5.2 Listener filesystem configuration   Listener related commander is: ------------------------------- cd .. ls mkdir avb_app write avb_app/mclock_role 1 mkdir avdecc write avdecc/btb_mode 1 write avdecc/talker_id 0x00049f4455660000 ------------------------------------------ The description for the listener configuration: avb_app/mclock_role =1, Media Clock Slave. avdecc/btb_mode=1, avdecc fast-connect back-to-back mode avdecc/talker_id =0x00049f4455660000，for fast connect mode configure the talker entity id。 Entity id configuration rule is: eui[0] = 0x00; eui[1] = 0x04; eui[2] = 0x9f; eui[3] = mac_addr[3]; eui[4] = mac_addr[4]; eui[5] = mac_addr[5]; eui[6] = 0x00; eui[7] = 0x00; mac_addr is determined by the hw_addr, eg, talker configure the hw_addr to: 00:22:33:44:55:66, this is the mac_addr[0-5] Then talker_id： eui[0] = 0x00; eui[1] = 0x04; eui[2] = 0x9f; eui[3] = mac_addr[3]=44; eui[4] = mac_addr[4]=55; eui[5] = mac_addr[5]=66; eui[6] = 0x00; eui[7] = 0x00; talker_id =0x00049f4455660000。 Next, configure the Listener file system: Press the computer's insert button to enter shell mode, and then do the filesystem configuration commander input: Fig 21   Until now, the Listener filesystem configuration is finished. 2.6 Test Result Let's start the connection test. Find a network cable to connect two MIMXRT1170-EVK boards, which is the 1G network port J4 of the Talker and the listener. Listener's J33 is inserted into the earphone to listen to the microphone audio data sent from the talker board. Connection picture is as follows: Fig 22 The test result is, when two board power on, after the short time sync, the audio sound collected from the Talker microphone can be heard in the Listener's headphones, indicating that the RT1170 AVB communication is working. Partial log diagrams are given below. The complete logs of talker and listener can be viewed in the attachment. Fig 23 Fig 24 Fig 25 Fig 26 3.Summarization After several testing, we can realize the RT1170 AVB audio transfer function. If the customer don’t want to build the project, just want to do the simple testing, they also can use the AVB/TSN stack’s generated bin file, the path is: \genavb_tsn-mcuxpresso-SDK_2_13_0-5_6_0\binaries\genavb-avb_audio_app-evaluation-freertos_rt1176-5_6_0.tar\genavb-avb_audio_app-evaluation-freertos_rt1176-5_6_0\release\ avb_app.bin This document is just the AVB tasting, for the deeper knowledge, will learn and share it later. Meet issues during the testing: 1). The cmake install during the linux build, which can be found from chapter 2.2.2 Related software tool 2). The printf log is messing, in Tera Term, use the 115200 baudrate to printf, the default printf log is miss order, it is difficult to the detail content, just like this: Fig 27 Solution: Fig 28 Fig 29 After the above configuration, we can see the log is in order. 3). Board programming, at first use the MSD to download the code, but didn’t wait the 30s, then do the power off or the reset, it always causes MSD burning failure, and the generated avb_app.bin , didn’t find the FCB location is not do the 0x400 offset when use the mcubootutility, It also meet issues. So when use the mcubootutility, need to burn from 0x30000400, please refer to chapter 2.4 Code programming 4). Talker filesystem configuration for port0/hw_addr, the first bytes should be 0X00, if none 0, it will have the AVB communication issues, after modify the first byte to the 0X00 in hw_addr, the issue is solved.  

Note: for similar EVKs, see: Using J-Link with MIMXRT1060-EVKB or MIMXRT1040-EVK Using J-Link with MIMXRT1060-EVK or MIMXRT1064-EVK Using J-Link with MIMXRT1160-EVK or MIMXRT1170-EVK This article provides details using a J-Link debug probe with this EVK.  There are two options: the onboard MCU-Link debug probe can be updated with Segger J-Link firmware, or an external J-Link debug probe can be attached to the EVK.  Using the onboard debug circuit is helpful as no other debug probe is required.  This article details the steps to use either J-Link option. MIMXRT1170-EVKB jumper locations   Using external J-Link debug probe Segger offers several J-Link probe options.  To use one of these probes with these EVKs, configure the EVK with these settings: Install a jumper on JP5, to disconnect the SWD signals from the onboard debug circuit.  This jumper is open by default. Power the EVK: the default option is connecting the power supply to barrel jack J43, and setting power switch SW5 to On position (3-6).  The green LED D16 next to SW5 will be lit when the EVK is properly powered. Connect the J-Link probe to J1, 20-pin dual-row 0.1" header.   Using onboard MCU-Link with J-Link firmware Install the MCU-Link Installer for the drivers and firmware update tool Disconnect any USB cables from the EVK Power the EVK: the default option is connecting the power supply to barrel jack J43, and setting power switch SW5 to On position (3-6).  The green LED D16 next to SW5 will be lit when the EVK is properly powered. Install a jumper at JP3 to force the MCU-Link in ISP mode Connect a USB cable to J86, to the MCU-Link debugger Go to the scripts directory in the MCU-Link software package installation and run the program_JLINK.cmd (Windows) or program_JLINK (Linux/MacOS) script by double-clicking it. Follow the onscreen instructions.  In Windows, this script is typically installed at C:\nxp\MCU-LINK_installer_3.122\scripts\program_JLINK.cmd Unplug the USB cable at J86 Remove the jumper at JP3 Plug the USB cable back in to J86.  Now the MCU-Link debugger should boot as a JLink. Remove jumper JP5, to connect the SWD signals from the MCU-Link debugger.  This jumper is open by default.  

RT Linux SDK build based on Ubuntu 1. Abstract The SDK of NXP MIMXRT products can support three operation systems: windows, Linux, and macOS. Usually, the vast majority of users use the windows version combined with IDE compilation, and the documentation is relatively complete. However, for the Linux version, although the SDK is downloaded, it also contains documents, but the documents are the same as those of windows, not for Linux. Therefore, when a small number of customers use Ubuntu Linux to compile, they suffer from no documentation reference, especially for novices, it is difficult to use.      This article will implement the build of RT1060 linux version SDK based on Ubuntu. 2. Tool preparation You need to prepare a computer with Ubuntu system. Windows can install a virtual machine with Ubuntu system. This article uses the Ubuntu system of the web server. Tools required for testing: Ubuntu system cmake ARMGCC: ARGCC for ARM Cortex M core SDK: SDK_2_13_1_EVK-MIMXRT1060_linux.zip EVK: MIMXRT1060-EVK This article takes MIMXRT1060-EVK SDK as an example, and the situation of other RT development boards with Linux SDK is the same. 2.1 SDK downloading     Download link: https://mcuxpresso.nxp.com/en/builder?hw=EVK-MIMXRT1060 Fig 1 Download the SDK code, named as: SDK_2_13_1_EVK-MIMXRT1060_linux.zip If you download it under Windows, you need to copy the SDK to the Ubuntu system. Here you can use FileZilla or MobaXterm to transfer the file. Because I use the web server Ubuntu, it is based on MobaXterm. This software is free to use, and the download link is: https://mobaxterm.mobatek.net/ Put the downloaded SDK into the Ubuntu folder, in MobaXterm, drag the file can realize the file transfer from Windows to Ubuntu: Fig 2 Unzip SDK, the commander is： unzip SDK_2_13_1_EVK-MIMXRT1060_linux.zip -d ./SDK_2_13_1_EVK-MIMXRT1060_linux Fig 3 Fig 4 It can be seen that the SDK has been successfully unzipped to the SDK_2_13_1_EVK-MIMXRT1060_linux folder. At this point, the Linux SDK is ready to use. 2.2 ARMGCC install and configuration Download ARMGCC, as you can see from the release note of the SDK, the supported GCC Arm Embedded version: GCC Arm Embedded, version is 10.3-2021.10   Download link:https://developer.arm.com/downloads/-/gnu-rm Download the file: gcc-arm-none-eabi-10.3-2021.10-x86_64-linux.tar.bz2  Copy it to the Ubuntu, and unzip it, the unzip commander is: tar -xjvf gcc-arm-none-eabi-10.3-2021.10-x86_64-linux.tar.bz2 Fig 5 Fig 6 You can see that ARMGCC has been decompressed. Configure the environment variables below and add ARMGCC_DIR to /etc/profile: Add the path at the end of the profile to save and exit: export ARMGCC_DIR=/home/nxa07323/rtdoc/gcc-arm-none-eabi-10.3-2021.10/ export PATH=$PATH:/home/nxa07323/rtdoc/gcc-arm-none-eabi-10.3-2021.10/bin/ Fig 7 Valid profile, and check the ARMGCC_DIR is really valid. source /etc/profile echo $ARMGCC_DIR Fig 8 Until now, ARMGCC is ready to use! 2.3 cmake download and install Build also need the cmake tool, so use the following command to install cmake and check whether the installation is successful: sudo apt-get install cmake cmake –version Fig 9 Cmake is also ready! 3. Testing All the tools are ready, let’s start compiling the code, here we take hello_world as an example to compile an executable file downloaded to Flash. 3.1 Executable file Compilation Enter the hello_world gcc path of the SDK: Fig10 It can be seen that there are many files under the armgcc folder, which are compilable files that generate different images: build_debug，build_release：the linker file is RAM linker, where text and data section is put in internal TCM. build_flexspi_nor_debug, build_flexspi_nor_release: The linker file is flexspi_nor linker, where text is put in flash and data put in TCM. build_flexspi_nor_sdram_debug, build_flexspi_nor_sdram_release: The linker file is flexspi_nor_sdram linker, where text is put in flash and data put in SDRAM. build_sdram_debug, build_sdram_release: The linker file is SDRAM linker, where text is put in internal TCM and data put in SDRAM. build_sdram_txt_debug, build_sdram_txt_release: The linker file is SDRAM_txt linker, where text is put in SDRAM and data put in OCRAM. Now, compile build_flexspi_nor_debug.sh, this script will generate flash .elf file, the command is: ./build_flexspi_nor_debug.sh Fig 11 The compiled .elf is placed in the flexspi_nor_debug folder: Fig 12 Convert the hello_world.elf file to hex and bin for the RT board burning, conversion command is: arm-none-eabi-objcopy -O ihex hello_world.elf hello_world.hex arm-none-eabi-objcopy -O binary hello_world.elf hello_world.bin Fig 13 3.2 Code Downloading Test The generated files hello_world.hex and hello_world.bin are the executable files, which can be downloaded to the EVK board through MSD, serial downloader, or debugger software. Open the bin file to view: Fig 14 As you can see, this file is an app executable file with FCB. Here use the MCUbootUtility tool to download, and the EVK board enters the serial download mode: SW7 1-OFF, 2-OFF, 3-OFF, 4-ON Fig 15 After the downloading is finished, EVK board enter the internal boot mode: SW7 1-OFF,2-OFF,3-ON,4-OFF Fig 16 We can see, the printf works, it means the Ubuntu Linux build the file works OK. 3.3 Code configuration Some customers may think that the executable files to be loaded by some of our tools do not need FCB, so how to realize to generate the app without FCB Linux, here we need to modify the flags.cmake file, the path is:     /home/nxa07323/rtdoc/SDK_2_13_1_EVK-MIMXRT1060_linux/boards/evkmimxrt1060/demo_apps/hello_world/armgcc Configure BOOT_HEADER_ENABLE=0： Default is BOOT_HEADER_ENABLE=1(Fig 17), modified to Fig 18: Fig 17                              Fig18 Build again, to generate the .bin, check the .bin file: Fig 19 We can see that this file is a pure app file that does not contain FCB+IVT. It can be used in occasions that do not require FCB. Until now, the RT1060 Linux version of the SDK can be compiled to generate an executable file under Ubuntu, and the function is normal after the function test.            

1.Introduction Recently, some customers need the RT1170 LWIP socket client, so this post is mainly share the socket client code which is based on the RT1170 SDK, it is just a simple demo, which also give the test result based on the NXP official EVKB board. 2. Code modification Platform: MIMXRT1170-EVKB SDK_2_13_1_MIMXRT1170-EVKB MCUXpresso IDE v11.7.1 Code is based on the SDK project : lwip_ping_freertos_cm7. This project already add the socket related file, so the modification is simple, just need to add the socket related header file and the app function. The modification is: Add socket server IP address, port, and the message which want to sendout. #define INIT_THREAD_STACKSIZE 1024 /*! @brief Priority of the temporary lwIP initialization thread. */ #define INIT_THREAD_PRIO DEFAULT_THREAD_PRIO #define HOST_NAME "192.168.0.100" #define BUF_LEN 100 uint8_t senddata[]= "Socket client test"; #define PORT 54321 #define IP_ADDR "192.168.0.100" Comment the ping code calling in stack_init API. //  ping_init(&netif_gw); Add the socket client thread: sys_thread_new("socketclient", socketclient_thread, NULL, DEFAULT_THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); The thread code is: static void socketclient_thread(void *arg) { int sock = -1,rece; struct sockaddr_in client_addr; char* host_ip; ip4_addr_t dns_ip; err_t err; uint32_t *pSDRAM= pvPortMalloc(BUF_LEN);// host_ip = HOST_NAME ; PRINTF("host name : %s , host_ip : %s\r\n",HOST_NAME,host_ip); // while(1) // { PRINTF("Start server Connect !\r\n"); // create connection sock = socket(AF_INET, SOCK_STREAM, 0); if (sock < 0) { PRINTF("Socket error\n"); vTaskDelay(10); // continue; } client_addr.sin_family = AF_INET; client_addr.sin_port = htons(PORT); client_addr.sin_addr.s_addr = inet_addr(host_ip); memset(&(client_addr.sin_zero), 0, sizeof(client_addr.sin_zero)); if (connect(sock, (struct sockaddr *)&client_addr, sizeof(struct sockaddr)) == -1) { PRINTF("Connect failed!\r\n"); closesocket(sock); vTaskDelay(10); // continue; } PRINTF("Connect to server successful!\r\n"); // PRINTF("\r\n************************************************************\n\r"); // PRINTF("\r\n Begin write\n\r"); write(sock,senddata,sizeof(senddata)); while (1) { //receive data rece = recv(sock, (uint8_t*)pSDRAM, BUF_LEN, 0);//BUF_LEN if (rece <= 0) break; PRINTF("recv %d len data\r\n",rece); PRINTF("%.*s\r\n",rece,(uint8_t*)pSDRAM); write(sock,pSDRAM,rece); } //rec data process memset(pSDRAM,0,BUF_LEN); closesocket(sock); vTaskDelay(10000);//about 10s //10000 // } }   3. Test Result Firstly, use the PC to configure the ENET IP for the server:     192.168.0.100   After configuration, customer can use the TCP test tool, eg:USR-TCP232-Test, which is configured to the TCP server, local IP is：192.168.0.100, host port is：54321, then enter the listen mode:   After the code download to the MIMXRT1170-EVKB, and run it, we can see, the server can detect the connected client IP:192.168/0.102, after the client connect to the server, it will send out the message: “socket client test”, then the server can send out the message to the client, the client will use the UART printf it, also loop back to the server again. This is the test result video: Code attached：evkbmimxrt1170_lwip_socket_client_freertos_cm7.7z  

NXP Updated the Hardware Development Guide for the MIMXRT1160/1170 Processor (MIMXRT1170HDUG) The main difference is： Updated capacitance value of VDDA_1P8_IN in Table 1 and Table 2 From 0.1uF to 1uF     This will help improve the robustness of the circuit at low temperatures.

RT106X secure JTAG test and IDE debug 1 Introduction     Regarding the usage of RT10XX Secure JTAG, the nxp.com has already released a very good application note AN12419 Secure JTAG for i.MXRT10xx: https://www.nxp.com/docs/en/application-note/AN12419.pdf This application note talks about the principle of Secure JTAG, how to modify the fuse to implement the Secure JTAG function, and the content of the related JLINKscript file, and then gives the use of JLINK commander to realize the identification of the ARM core. Usually, if the ARM core can be identified, it indicates that Secure JTAG connection has been passed. But in practical usage, I found many customers encounter the different issues, for example, the Secure JTAG could not find the ARM core directly, or the core identify is not stable, and some customers asked how to use common IDEs, such as MCUXPresso, IAR , MDK to add this Secure JTAG function to realize  Secure JTAG debugging.   For the test of secure JTAG, it also needs the cost, because the fuse needs to be modified. If the position of the fuse is accidentally modified, it may cause irreversible problems. Due to the different situations of customers, I also done more tests, borrowing boards with chip socket which can replace the different RT chip, I have tested RT1050, RT1060, RT1064, but in practical usage, there are still some customers mentioned that it will be reproduced on the EVK, so I also tested the secure JTAG function on the RT1060 and RT1064 EVK     This article will share all the previous relevant experience, so that latecomers can have a reference when encountering similar problems, and avoid unnecessary minefields. This document used the platform: MIMXRT1064-EVK revA: RT1060-EVK, RT1050-EVKB is similar SDK_2_13_0_EVK-MIMXRT1064 MCUXpresso IDE v11.7.1_9221 MDK V5.36: higher reversion is the same IAR 9.30.1: higher reversion is the same Segger JLINK plus JLINK driver version:V788D NXP-MCUBootUtility-5.1.0 2 RT1064 secure JTAG modification Under normal circumstances, it is not recommended for customers to burn all the related fuses directly and then test it directly. I usually proceeds step by step, hardware layout, to ensure that it can support JTAG, and then save the original read of the fuse, burn JTAG, test JTAG, and finally Burn and test other fuses for secure JTAG.    2.1 MIMXRT1064-EVK Hardware modification For RT10XX EVK, the board default situation is the same as the chip situation, which supports SWD. The JTAG pin is connected to other hardware modules from the hardware, so it will affect JTAG function. When it is determined to use JTAG function, the circuit needs to be modified, just like MIMXRT105060HDUG has said:    (1). Burn fuse DAP_SJC_SWD_SEL from ‘0’ to ‘1’ to choose JTAG. (2). DNP R323,R309,R152 to isolate JTAG multiplexed signals. (3). Keep off J47 to J50 to isolate board level debugger.     So, to the MIMXRT1064-EVK board, just need to remove R323, R309, R152, disconnect J47,J48,J49,J50, which is used to disconnect the on board debugger, then use the external Segger JLINK JTAG interface to connect the MIMXRT1064-EVK on board J21. 2.2 Original fuse map read First, the MIMXRT1064-EVK board enters the serial download mode, SW7: 1-OFF, 2-OFF, 3-OFF, 4-ON. Use MCUBootUtility tool to connect EVK, and read the initial fuse map, the situation is as follows:     Fig 1 2.3 JTAG Modification and test    Modify fuse to realize SWD to JTAG: 0X460[19] DAP_SJC_SWD_SEL=1   Fig 2     Use the JLINK commander, JTAG method to connect the board, to find the ARM CM7 core: Fig 3     If the ARM CM7 core can’t be identified, it means the hardware still have issues, or the fuse modified bit is not correct, just do the double check, make sure the ARM core can be found, then go to the next steps. 2.4 Secure JTAG Modification     Modify fuse bit to realize Secure JTAG:     0X460[23:22]:JTAG_SMODE =1     0X460[26]: KTE_FUSE=1     0X610,0X600 burn key: 0xedcba987654321, user also can burn with other custom keys, but need to record it, as the JLINKScript needs to use it.   Fig 4 In the above picture, the secure JTAG fuse and key fuse is finished, at last, to burn fuse 0X400[6]: SJC_RESP_LOCK=1, which is used to close the write and read to secret response key: Fig 5 Here, we can see, the 0X600,0X610 key area is shadow. Now, record the UUID0, UUID1, it will use the script to read out to check the UUID correction or not. 2.5 Secure JTAG JLINK commander test Because during the secure JTAG connection process, the JTAG_MOD pin needs to be pulled low and high, so a wire needs to be connected to pull JTAG_MOD low and high. MIMXRT1064-EVK can use J25_4, which is 3.3V, and JTAG_MOD signal point can use TP11 test point. By default, JTAG_MOD is pulled low. When it needs to be pulled high, it can be connected to J25_4.         During the test, it will need to use the JLINKScript, the content is as follows, also can check  the attached NXP_RT1064_SecureJTAG.JlinkScript file： int InitTarget(void) { int r; int v; int Key0; int Key1; JLINK_SYS_Report("***********************************************"); JLINK_SYS_Report("J-Link script: InitTarget() *"); JLINK_SYS_Report("NXP iMXRT, Enable Secure JTAG *"); JLINK_SYS_Report("***********************************************"); JLINK_SYS_MessageBox("Set pin JTAG_MOD => 1 and press any key to continue..."); // Secure response stored @ 0x600, 0x610 in eFUSE region (OTP memory) Key0 = 0x87654321; Key1 = 0xedcba9; JLINK_CORESIGHT_Configure("IRPre=0;DRPre=0;IRPost=0;DRPost=0;IRLenDevice=5"); CPU = CORTEX_M7; JLINK_SYS_Sleep(100); JLINK_JTAG_WriteIR(0xC); // Output Challenge instruction // Readback Challenge, Shift 64 dummy bits on TDI, TODO: receive Challenge bits on TDO JLINK_JTAG_StartDR(); JLINK_SYS_Report("Reading Challenge ID...."); JLINK_JTAG_WriteDRCont(0xffffffff, 32); // 32-bit dummy write on TDI / read 32 bits on TDO v = JLINK_JTAG_GetU32(0); JLINK_SYS_Report1("Challenge UUID0:", v); JLINK_JTAG_WriteDREnd(0xffffffff, 32); v = JLINK_JTAG_GetU32(0); JLINK_SYS_Report1("Challenge UUID1:", v); JLINK_JTAG_WriteIR(0xD); // Output Response instruction JLINK_JTAG_StartDR(); JLINK_JTAG_WriteDRCont(Key0, 32); JLINK_JTAG_WriteDREnd(Key1, 24); JLINK_SYS_MessageBox("Change pin JTAG_MOD => 0, press any key to continue..."); return 0; }   SecJtag.bat file content is: jlink.exe -JLinkScriptFile NXP_RT1064_SecureJTAG.JlinkScript -device MIMXRT1064XXX6A -if JTAG -speed 4000 -autoconnect 1 -JTAGConf -1,-1 This command is mainly used the JLINK commander and JLINKScript to realize the Secure JTAG connection. When test it, put the SecJtag.bat, JLink.exe, and NXP_RT1064_SecureJTAG.JlinkScript 3 files in the same folder. For testing, can change the board mode to the internal boot mode, SW7:1-OFF,2-OFF, 3-ON, 4-OFF. Run SecJtag.bat, the test situation is: It indicates to connect JTAG_MOD to higher level   Fig 6 Here, use the wire to connect the J25_4 and TP11, which is connect the JTAG_MOD=1, then click OK, go to the next step:   Fig 7 It can be seen here that the correct UUID has been recognized, which is consistent with the UUID read by MCUBootutility above. Many customers cannot read the correct UUID here, indicating that there is a problem with hardware modification, or fuse modification, or another. Or in the case, the JTAG pin in the app is not enabled, which will be described in detail later. Here disconnect the connection between TP11 and J25_4, the default is JTAG_MOD=0, click OK to continue Fig 8 Here, we can see, the ARM CM7 core is found, it means this hardware platform already realize the Secure JTAG connection. Now, can use the IDEs to do the debugging. 3. Secure JTAG debug function in 3 IDEs This chapter aims at how to use secure JTAG function in RT10XX three commonly used IDEs: MCUXpresso, IAR, MDK,  to implement secure JTAG code debug operation.    3.1 Software code prepare This article selects the SDK hello_world project as the test demo: SDK_2_13_0_EVK-MIMXRT1064\boards\evkmimxrt1064\demo_apps\hello_world     Two points should be noted here:  Do not use led_blinky directly, because the led control pin GPIO_AD_B0_09 used by the code is JTAG_TDI, which will cause the Secure JTAG connection to fail after downloading this code, because the pin function of JTAG has been changed. Add the pin configuration for JTAG in app code pinmux.c, otherwise there will be a phenomenon due to the lack of JTAG pin configuration, to the empty RT1064, which the chip that has not burned the code can use Secure JTAG connection, but once the code is burned, the connection will be failed. Add the following code to Pinmux.c: IOMUXC_SetPinMux(IOMUXC_GPIO_AD_B0_11_JTAG_TRSTB, 0U); IOMUXC_SetPinMux(IOMUXC_GPIO_AD_B0_06_JTAG_TMS, 0U); IOMUXC_SetPinMux(IOMUXC_GPIO_AD_B0_07_JTAG_TCK, 0U); IOMUXC_SetPinMux(IOMUXC_GPIO_AD_B0_09_JTAG_TDI, 0U); IOMUXC_SetPinMux(IOMUXC_GPIO_AD_B0_10_JTAG_TDO, 0U); 3.2 MCUXpresso Secure JTAG debug    Use MCUXpresso IDE to import the SDK hello world demo, modify the pinmux.c, which add the JTAG pin function configuration.    Configure MCUXPresso ID’s debugger JLinkGDBServerCL.exe version as your used JLINK driver version, Window->preferences Fig 9 Run->Debug configurations, configure to JTAG, choose device as MIMXRT1064xxx6A, add the JLINKscriptfile   Fig10   Fig 11 Connect JTAG_MOD=1, which is connect TP11 to J25_4, connect OK.   Fig 12 We can see, it already gets the correct UUID, it also requires connect JTAG_MOD=0, here just leave the TP11 floating, then connect OK:   Fig 13 It can be seen that at this time, it has successfully entered the debug mode and can do debugging. For details, you can check the MCUXpresso11_7_1_MIMXRT1064_SJTAG.mp4 file in the attachment. The test experience here is that MCUXpresso V11.7.1 is found to be a bit unstable and needs to be tried a few more times, but the download of the higher version V11.8.0 version is very stable. If you can get a version higher than V11.7.1, it is recommended to use a higher version of MCUXpresso IDE . 3.3 IAR Secure JTAG debug Some customers need to use the IAR IDE to debug Secure JTAG function, you can use the hello world in the SDK demo, modify pinmux.c to add the JTAG pin configuration code.     The difference is:   (1) Run JLINK driver:JLinkDLLUpdater.exe   Fig 14 Just to refresh the JLINK driver to the IAR,MDK IDE. (2) Modify the file name of JLINKscript to be consistent with the name of the demo, and put it under the settings folder of the project folder. For example, the routine here is hello_world_flexspi_nor_debug, and the file name of JlinkScript is required: hello_world_flexspi_nor_debug.JlinkScript, so that IAR will automatically call the corresponding JlinkScript file   Fig15 (3) Configure IAR debugger as JLINK JTAG   Fig 16                                          Fig 17 Click debug button to enter debug mode:   Fig 18 It needs to set JTAG_MOD=1, just to connect TP11 to J25_4.   Fig 19 It needs to set JTAG_MOD=0, just leave the TP11 floating, click OK to continue.   Fig 20 We can see, the IAR already can do the secure JTAG debugging. 3.4 MDK Secure JTAG debug   For the MDK secure JTAG configuration, the basic requirement is:     (1) Modify pinmux.c code to enable the JTAG pin function     (2) Run JLINK driver, JLinkDLLUpdater.exe，refresh the driver to MDK     (3) JlinkScript file name changed to JLinkSettings.JlinkScript, copy it to the folder in the mdk project, then the MDK will call the JLINKscript file automatically   Fig 21       (4) Modify debugger to JLINK, then modify the interface to JTAG   Fig 22   Fig 23 So far, the Secure JTAG related configuration of MDK has been completed. From theory, it can be directly debugged to run. But I found some problems after many tests. For the code of RAM (hello_world debug), it is no problem to be able to perform secure JTAG debug, but for the code of flash (hello_world_flexspi_nor_debug), there is no problem through secure jtag download, but the debug will run the program abnormal, check the memory data in the flash, also get the wrong data     Fig 24 We can see, UUID also correct, normally, this issue is related to the flashloader during downloading, however, the flashloader of JLINK has not been directly accessed, so I tried to use RT-UFL as the flashloader, and the debugger was successful. If customers encounter similar problems when want to use the MDK to do the secure JTAG debugging, they can use RT-UFL as the flashloader. The reference document is: https://www.cnblogs.com/henjay724/p/13951686.html https://www.cnblogs.com/henjay724/p/15465655.html To summarize it here, copy the iMXRT_UFL file to the JLINK driver folder: C:\Program Files\SEGGER\JLINK\Devices\NXP Copy JLinkDevices.xml to folder: C:\Program Files\SEGGER\JLINK The Jlinkscript file add is the same as the Figure 21. Modify the JlinkSettings.ini file, device is MIMXRT1064_UFL, override =1.   Fig 25 Delete the program algorithm, will use the RT-UFL algorithm   Fig 26 Uncheck update target before Debugging   Fig 27 Enter debug mode:   Fig 28 Configure JTAG_MOD=1, connect TP11 to J25_4, click OK to continue:   Fig 29 Leave the TP11 as floating, click OK to enter the debug mode, the result is:   Fig 30 We can see, after changing the flashloader to the RT-UFL, MDK project Secure JTAG debug also works OK, the attachment also share the RT-UFL related files.  4. Summary For Secure JTAG, you need to modify the hardware to support JTAG function, modify the fuse to support secure JTAG, and modify the code pins to enable the JTAG function. For the IDE debug, you need to configure the relevant interface as JTAG and add the correct JlinkScriptfile, so that the secure JTAG function can be successfully run , and perform IDE code debugging. Attachments: evkmimxrt1064_hello_world_SJTAG.zip：MCUXpresso project EVK-MIMXRT1064-hello_world_iar.7z：IAR project EVK-MIMXRT1064-hello_world_mdk.7z：MDK project File\ NXP_RT1064_SecureJTAG.JlinkScript, JLINK script File\ SecJtag.bat, associate with JLink.exe and NXP_RT1064_SecureJTAG.JlinkScript to realize JLINK Commander connection, which will find the ARM core. File\ RT-UFL: RT ultra flashloader algorithm, source：https://github.com/JayHeng/RT-UFL   Here, really thanks so much for our expert @juying_zhong 's help with the Secure JTAG patient guide during my testing road!

Wireless module combinations from Tables 2 and 3 are not updated with the latest SDK 2.12.1 in the user manual UM11441. Major updates in Table 2 and Table 3: u-blox modules are supported only on rt1060 Murata modules are only tested with i.MX RT1060 EVKB and i.MX RT1040 EVK platforms Modified Murata modules names Renamed i.MX RT685S EVK to IMXRT685-AUD-EVK Please refer to the attached PDF for updated information.

Note: for similar EVKs, see: Using J-Link with MIMXRT1170-EVKB Using J-Link with MIMXRT1060-EVKB or MIMXRT1040-EVK Using J-Link with MIMXRT1060-EVK or MIMXRT1064-EVK This article provides details using a J-Link debug probe with either of these EVKs.  There are two options: the onboard debug circuit can be updated with Segger J-Link firmware, or an external J-Link debug probe can be attached to the EVK.  Using the onboard debug circuit is helpful as no other debug probe is required.  Appnote AN13206 has more details on this, and the comparison of the firmware options for the debug circuit.  This article details the steps to use either J-Link option.   Using external J-Link debug probe Segger offers several J-Link probe options.  To use one of these probes with these EVKs, configure the EVK with these settings: Remove jumpers J6 and J7, to disconnect the SWD signals from the onboard debug circuit.  These jumpers or installed by default. Power the EVK: the default option is connecting the power supply to barrel jack J43, and setting power switch SW5 to On position (3-6).  The green LED D16 next to SW5 will be lit when the EVK is properly powered. Connect the J-Link probe to J1, 20-pin dual-row 0.1" header.   Using onboard debug circuit with J-Link firmware Disconnect any USB cables from the EVK Power the EVK: the default option is connecting the power supply to barrel jack J43, and setting power switch SW5 to On position (3-6).  The green LED D16 next to SW5 will be lit when the EVK is properly powered. Short jumper J22 to force the debug circuit in DFU mode Connect a USB cable to J11, to the on-board debugger Follow Appnote AN13206 to program the J-Link firmware to the EVK Unplug the USB cable at J11 Remove the jumper at J22 Plug the USB cable back in to J11.  Now the on-board debugger should boot as a JLink. Install jumpers J6 and J7, to connect the SWD signals from onboard debug circuit.  These jumpers or installed by default. Note: with the JLink firmware loaded, USB J11 is no longer an option to power the EVK.  Another option is connecting the power supply to barrel jack J43, and setting power switch SW5 to On position (3-6).  For this power option, jumper J38 needs to short 1-2, which is the default setting. 

Abstract Today I'd like to discuss a scenario that I will encounter in practical application. In the design phase, the Serial Nor flash is usually used as a code storage device for RT series MCUs, such as QSPI, HyperFlash, etc, as these devices all support XIP features. The flash usually is not only occupied by the code but also used to store data, such as device parameters, and log information, and even used as a file system. So we need to evaluate Flash's size. Is there any constraint to the manipulation of data space in the secure boot mode? And how to keep the data confidential? We'll talk about it below and let's get started. Secure boot mode HAB boot The bootable image is plaintext, and it may be changed after the writing operation of the FlexSPI module. If the digest algorithm obtains the different values will lead to the verification process fails, as the writing operation destroys the integrity of the data, regarding confidentiality, data is stored in plaintext in Serial Nor flash under HAB boot.   Fig1 Signature and verification process Encrypted XIP boot mode After enabling the Encrypted XIP boot mode, what is the impact on FlexSPI's read and write data operations? The first point is that the read data will be treated as encrypted data and decrypted by the BEE or OTFAD module. However, if a write operation is performed, the BEE or OTFAD module will be bypassed, in another word, the data will be written directly to the Serial Nor flash. in a short, it is not affected by the Encrypted XIP boot mode. 1) Read operation As shown in Fig 2, the encrypted code and data stored in Serial Nor flash need to be decrypted before they are sent to the CPU for execution. This is also the implementation mechanism of the Encrypted XIP boot mode. To enable the encrypted XIP boot mode, it needs to burn the keys to eFuse, after that, eFuse is impossible to restore, so the test cost seems a bit high, so I recommend you refer to the source code of the 《How to Enable the On-the-fly Decryption》application note to dynamically configure the key of the BEE module and read the encrypted array by DCP from Flash, then compare to plaintext array to verify BEE module participle the decryption flow. Fig 2 2) Write operation Modify the source code of the above application note, define s_nor_program_buffer [256], then set the values through the following code, and burn them to the 20th sector, the offset address is 0x14000. for (i = 0; i < 0xFFU; i++) { s_nor_program_buffer[i] = i; } status = flexspi_nor_flash_page_program(EXAMPLE_FLEXSPI, EXAMPLE_SECTOR * SECTOR_SIZE, (void *)s_nor_program_buffer); if (status != kStatus_Success) { PRINTF("Page program failure !\r\n"); return -1; } DCACHE_InvalidateByRange(EXAMPLE_FLEXSPI_AMBA_BASE + EXAMPLE_SECTOR * SECTOR_SIZE, FLASH_PAGE_SIZE); memcpy(s_nor_read_buffer, (void *)(EXAMPLE_FLEXSPI_AMBA_BASE + EXAMPLE_SECTOR * SECTOR_SIZE), sizeof(s_nor_read_buffer)); for(uint32_t i = 0; i < 256; i++) { PRINTF("The %d data in the second region is 0x%x\r\n", i, s_nor_read_buffer[i]); } After the programming finishes, connect to Jlink and use J-flash to check whether the burned array is correct. The results prove that the write operation will bypass the BEE or OTFAD module and write the data directly to the Serial Nor flash, which is consistent with Fig 2. Fig3 Sensitive data preservation As mentioned at the beginning, in the real project, we may need to use Flash to store data, such as device parameters, log information, or even as a file system, and the saved data is usually a bit sensitive and should prevent being easily obtained by others. For example, in SLN-VIZNAS-IoT, there is a dedicated area for storing facial feature data. Fig 4 Although the purely facial feature data is only meaningful for specific recognition algorithms, in another word, even if a third-party application obtains the original data, it is useless for hackers. In the development of real face recognization projects, it is usually to declare other data items associated with facial feature data, such as name, age, preferences, etc, as shown below： typedef union { struct { /*put char/unsigned char together to avoid padding*/ unsigned char magic; char name[FEATUREDATA_NAME_MAX_LEN]; int index; // this id identify a feature uniquely,we should use it as a handler for feature add/del/update/rename uint16_t id; uint16_t pad; // Add a new component uint16_t coffee_taste; /*put feature in the last so, we can take it as dynamic, size limitation: * (FEATUREDATA_FLASH_PAGE_SIZE * 2 - 1 - FEATUREDATA_NAME_MAX_LEN - 4 - 4 -2)/4*/ float feature[0]; }; unsigned char raw[FEATUREDATA_FLASH_PAGE_SIZE * 2]; } FeatureItem; // 1kB After enabling the Encrypted XIP boot mode, the above write operation test has proved that FlexSPI's write operation will program the data into Serial Nor flash directly, but during the reading process, the data will be decrypted by BEE or OTFAD, so we'd better use DCP or other modules to encrypt the data prior to writing, otherwise, the read operation will get the values that the plaintext goes through the decryption calculation. The risk of leakage Assume XIP encrypted boot is enabled, and whether it's okay to send the encrypted bootable image sent to the OEM for mass production. Moreover, is it able to allow the customers to access the encrypted bootable image without worrying about application image leakage? In order to verify the above guesses, I do the following testing on MIMXRT1060-EVK. Select the XIP encrypted mode in the MCUXpresso Secure Provisioning tool to generate and burn the bootable image of the Blink LED; Fig5 Observe the burned image through NXP-MCUBootUtility, you can find that the ciphertext image is very messy when compared to the plaintext image on the right border, so it seems like the NXP-MCUBootUtility can't obtain the plaintext image; Fig 6 Let's observe the ciphertext image in another way, read them through the pyocd command, as shown below; Fig 7 Open then 9_21_readback.Bin and compare it with the plain text image on the right border, they are the same actually, in other words, the plaintext image was leaked. Fig 8 Explanation As the above Fig 2 shows, the encrypted code and data stored in Serial Nor flash need to be decrypted before they are sent to the CPU for execution. When Jlink connects to the target MCU, it will load the corresponding flash driver algorithm to run in the FlexRAM. If the flash driver algorithm detects the boot type of the MCU just like the following code, then configures the BEE or OTFAD module according to the detecting result, after that, when reading the ciphertext in the Nor Flash, the data will be automatically decrypted. status = SLN_AUTH_check_context(SLN_CRYPTO_CTX_1); configPRINTF(("Context check status %d\r\n", status)); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a crash if (SLN_AUTH_NO_CONTEXT == status) { configPRINTF(("Ensuring context...\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a crash // Load crypto contexts and make sure they are valid (our own context should be good to get to this point!) status = bl_nor_encrypt_ensure_context(); if (kStatus_Fail == status) { configPRINTF(("Failed to load crypto context...\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a crash // Double check if encrypted XIP is enabled if (!bl_nor_encrypt_is_enabled()) { configPRINTF(("Not running in encrypted XIP mode, ignore error.\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a // crash // No encrypted XIP enabled, we can ignore the bad status status = kStatus_Success; } } else if (kStatus_ReadOnly == status) // Using this status from standard status to indicate that we need to split PRDB { volatile uint32_t delay = 1000000; // Set up context as needed for this application status = bl_nor_encrypt_split_prdb(); configPRINTF(("Restarting BOOTLOADER...\r\n")); while (delay--) ; // Restart DbgConsole_Deinit(); NVIC_DisableIRQ(LPUART6_IRQn); NVIC_SystemReset(); } } else if (SLN_AUTH_OK == status) { configPRINTF(("Ensuring context...\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a crash // We will check to see if we need to update the backup to the reduced scope PRDB0 for bootloader space status = bl_nor_encrypt_ensure_context(); if (kStatus_Fail == status) { configPRINTF(("Failed to load crypto context...\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a crash // Double check if encrypted XIP is enabled if (!bl_nor_encrypt_is_enabled()) { configPRINTF(("Not running in encrypted XIP mode, ignore error.\r\n")); // No encrypted XIP enabled, we can ignore the bad status status = kStatus_Success; } } else if (kStatus_Success == status) // We have good PRDBs so we can update the backup { bool isMatch = false; bool isOriginal = false; configPRINTF(("Checking backup context...\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before a crash // Check if we have identical KIBs and initial CTR status = bl_nor_crypto_ctx_compare_backup(&isMatch, &isOriginal, SLN_CRYPTO_CTX_0); if (kStatus_Success == status) { if (isMatch && isOriginal) { configPRINTF(("Updating backup context with valid address space...\r\n")); // DEBUG_LOG_DELAY_MS(1000); // Optional delay, enable for debugging to ensure log is printed before // a crash // Update backup PRDB0 status = SLN_AUTH_backup_context(SLN_CRYPTO_CTX_0); } } } } How to handle Now we already understand the cause of the leak, we must prohibit external tools from loading flashloader or flash driver algorithms into the FlexRAM to run, so in addition to disabling the Debug port, we also need to disable the Serial download method to prevent the hackers take advantage of the Serial Downloader method to make the ROM code load a special flashloader to run in RAM, then configure the BEE or OTFAD module prior to reading the image. However, compared to simply prohibiting the debug port, I'd highly recommend you select the Secure Debug method, because the debug feature requirement is important to return/filed testing, Secure Debug just is like adding a sturdy lock to the debug port, and only the authorized one can open this lock to enter the debugging mode successfully. Reference AN12852:How to Enable the On-the-fly Decryption 《The trust chain of HAB boot》

Introduction NXP i.MX RT1xxx series provide the High Assurance Boot (HAB) feature which makes the hardware to have a mechanism to ensure that the software can be trusted, as the HAB feature enables the ROM to authenticate the program image by using digital signatures, which can assure the application image's integrity, authenticated and undeniable. So the OEM can utilize it to make their product reject any system image which is not authorized to run. However, what's the trust chain of HAB for implementing the purpose? How the key and certificate generate In the installation directory of MCUXpresso Secure Provisioning:  ~\nxp\MCUX_Provi_v3.1\bin\tools_scripts\keys , there are scripts for generating keys: hab4_pki_tree.sh and hab4_pki_tree.bat (both are applicable to Linux and Windows systems respectively), running any of the above scripts will generate 13 pairs of public and private keys in sequence through OpenSSL, which constitute the below tree structure. Fig1 Key Tree structure The public key and private key generated by OpenSSL are paired one by one, saving the private key and publishing the corresponding public key to the outside world can easily implement asymmetric encryption applications. But how to ensure that the obtained public key is correct and has not been tampered with? At this time, the intervention of authoritative departments is required. Just like everyone can print their resume and say who they are, but if they have the seal of the Public Security Bureau, only the household registration book can prove you are you. This issued by the authority is called a certificate. What's in the certificate? Of course, it should contain a public key, which is the most important; there is also the owner of the certificate, just like the household registration book with your name and ID number, indicating that the book is yours; in addition, there is the issuer of the certificate and the validity period of the ID card is a bit like the issuer institution on the ID card, and how many years of the validity period. If someone fakes a certificate issued by an authority, it's like having fake ID cards and fake household registration books. To generate a certificate, you need to initiate a certificate request, and then send the request to an authority for certification, which is called a CA(Certificate Authority). After sending this request to the authority, the authority will give the certificate a signature. Another question arises, how can the signature be guaranteed to be signed by a genuine authority? Of course, it can only be signed with something that is only in the hands of the authority, which is the CA's private key. The signature algorithm probably works like this: a Hash calculation is performed on the target information to obtain a Hash value. And this process is irreversible, that is to say, the original information content cannot be obtained through the Hash value. When the information is sent out, the hash value is encrypted and sent together with the information as a signature. The process is as follows. Fig2 Signature and verification process Looking at the content of the certificate (as shown below), we will find that there is an Issuer, that is, who issued the certificate; The subject is to who the certificate is issued; Validity is the certificate period; Public-key is the content of the public key, and related signature algorithm. You will find that in order to verify the certificate, the public key of the CA is required. Then a new question arises. How can we be sure that the public key of the CA is correct? This requires a superior CA to sign the CA's public key, and then form the CA's certificate. If you want to know whether a CA's certificate is reliable, you need to see if the public key of the CA's superior certificate can unlock the CA's signature. Just likes if you don’t trust the District Public Security Bureau, you can call the Municipal Public Security Bureau and ask the Municipal Public Security Bureau to confirm its legitimacy of the District Public Security Bureau. This goes up layer by layer until the root CA makes the final endorsement. Through this layer-by-layer credit endorsement method, the normal operation of the asymmetric encryption mode is guaranteed. How does the Root CA prove itself? At this time, Root CA will issue another certificate (as shown below), called the Self-Signed Certificate, which is to sign itself with its own private key, giving people a feeling of "I am me, whether you believe it or not", Therefore, its format content is slightly different from the above CA certificate. Its Issuer and Subject are the same, and its own public key can be used for authentication. So the certificate authentication process will also end here. In this way, in addition to generating the public key and private key through running the script, the OpenSSL will also generate the certificate chain shown below.  Fig3 certificates Boot flow of the HAB mode Figure 4 shows the boot flow of the HAB mode. And steps 1, 2, and 3 are essentially the signature verification process. Fig4 Boot flow of the HAB mode The verification process (as shown in Figure 2) can be used to detect data integrity, identity authentication, and non-repudiation when the public key is trusted, so hab4_pki_tree.sh and hab4_pki_tree.bat scripts can ensure the generated public key and private key pair and the certificate are trusted, it's the "perfectly closed loop". However, the Application image in Figure 4 is plaintext, and the confidentiality of the data is not implemented, so the encrypted boot is always a combination of the HAB boot and the encrypted boot is an advanced usage of an authenticated boot. Reference AN4581: i.MX Secure Boot on HABv4 Supported Devices AN12681: How to use HAB secure boot in i.MX RT10xx  

1.  Abstract NXP EdgeReady solution can use RT106/5 S/L/A/F to achieve speech recognition, but the relevant support software libraries for the RT4-bit series are limited to the S/L/A/F series, if you want to use normal RT chips, how to achieve speech recognition functions? NXP officially launched the VIT software package in the SDK, which can support RT1060, RT1160, RT1170, RT600, RT500 to achieve SDK-based speech recognition functions. For the acquisition of weather information, usually customer can connect with a third-party platform or the cloud weather API, using http client method to access directly, the current weather API platforms, you can register it, then call the API directly, so you can use the RT SDK lwip socket client method to call the corresponding weather API, to achieve real-time specific geographical location weather forecast data.     This article will use MIMXRT1060-EVK to implement customer-defined wake-up word(WW) and voice recognition word recognition(VC) based on SDK VIT lib, and LWIP socket client to achieve real-time weather information acquisition in Shanghai, then print it to the terminal, this article mainly use the print to share the weather information, for the sound broadcasts, it also add the simple method to broadcast the fixed sound with mp3 audio data, but for the freely sound broadcast, it may need to use real-time TTS function, which is not added now.     The system block diagram of this document is as follows:   Fig 1 System Block diagram The VIT custom wake-up word of this system is "小恩小恩", and after waking up, one of the following recognition words can be recognized: ”开灯”("Turn on the lights")，“关灯”("Turn off the lights")，”今天天气”("Today's weather")，“明天天气”("Tomorrow's Weather")，“后天天气”("The day after tomorrow's weather"). Turn on the light or Turn off the lights , that is to control  the external LED red light on the EVK board. ”今天天气” gets today’s weather forecast, it is in the following format:                     "date": "2022-05-27",                     "week": "5",                     "dayweather": "阴",                     "nightweather": "阴",                     "daytemp": "28",                     "nighttemp": "21",                     "daywind": "东南",                     "nightwind": "东南",                     "daypower": "≤3",                     "nightpower": "≤3" “明天天气”，“后天天气” are the same format, but it is 1-2 days after the date of today. To get the weather data, the MIMXRT1060-EVK board needs to connect the network to achieve the acquisition of the Gaode Map(restapi.amap.com) Weather API data. 2.  Related preparations 2.1 Weather API Platform     At present, there are many third-party platforms that can obtain weather on the Internet for Chinese, such as: Baidu Intelligent Cloud, Baidu Map API, Huawei cloud platform, Juhe weather, Gaode Map API, and so on. This article tried several platform, the test results found: Baidu intelligent cloud, the number of daily free calls is small, the need for real-time synthesis of AK, SK, cumbersome to call; Baidu Map API needs to upload ID card information; Several others have a similar situation. In the end, the Gaode Map API with convenient registration, many daily calls and relatively full feedback weather data information was selected.     Here, we mainly talk about the Gaode Map API usage, the link is: https://lbs.amap.com/api/webservice/guide/api/weatherinfo Create the account and the API key, then add the relevant parameters to implement the call of the weather API, the application for API Key is as follows: Fig 2 Gaode map API key The following diagram shows the call volume:   Fig 3 Gaode Map API call volume This is the API calling format:   Fig 4 Weather API calling parameters So, the full Gaode Map API link should like this: https://restapi.amap.com/v3/weather/weatherInfo?key=xxxxxxx&city=xxx&extensions=all&output=JSON If need to test the Shanghai weather, city code is 310000. 2.2 Postman test weather API     Postman is an interface testing tool, when doing interface testing, Postman is equivalent to a client, it can simulate various HTTP requests initiated by users, send the request data to the server, obtain the corresponding response results, and verify whether the result data in the response matches the expected value. Postman download link: https://www.postman.com/   After finding the proper weather API platform and the calling link, use the postman do the http GET operation to capture the weather data, refer to the Fig 4, fill the related parameters to the postman: Fig 5 Postman call weather API Send Get command, we can find the weather information in the position 7, the complete all information is: {     "status": "1",     "count": "1",     "info": "OK",     "infocode": "10000",     "forecasts": [         {             "city": "上海市",             "adcode": "310000",             "province": "上海",             "reporttime": "2022-05-27 17:34:12",             "casts": [                 {                     "date": "2022-05-27",                     "week": "5",                     "dayweather": "阴",                     "nightweather": "阴",                     "daytemp": "28",                     "nighttemp": "21",                     "daywind": "东南",                     "nightwind": "东南",                     "daypower": "≤3",                     "nightpower": "≤3"                 },                 {                     "date": "2022-05-28",                     "week": "6",                     "dayweather": "小雨",                     "nightweather": "中雨",                     "daytemp": "24",                     "nighttemp": "20",                     "daywind": "东南",                     "nightwind": "东南",                     "daypower": "≤3",                     "nightpower": "≤3"                 },                 {                     "date": "2022-05-29",                     "week": "7",                     "dayweather": "大雨",                     "nightweather": "小雨",                     "daytemp": "23",                     "nighttemp": "20",                     "daywind": "南",                     "nightwind": "南",                     "daypower": "≤3",                     "nightpower": "≤3"                 },                 {                     "date": "2022-05-30",                     "week": "1",                     "dayweather": "小雨",                     "nightweather": "晴",                     "daytemp": "27",                     "nighttemp": "20",                     "daywind": "北",                     "nightwind": "北",                     "daypower": "≤3",                     "nightpower": "≤3"                 }             ]         }     ] }   We can see, it can capture the continuous 4 days information, with this information, we can get the weather information easily. From the postman, we also can see the Get code, like this: Fig 6 postman API HTTP code     With this API which already passed the testing, it can capture the complete weather information, here, we can consider adding the working http API to the MIMXRT1060-EVK code.    2.3 VIT custom commands     From the maestro code of the RT1060 SDK, we can know that the SDK already supports the VIT library, what is VIT?     VIT's full name: Voice Intelligent Technology, the library provides voice recognition services designed to wake up and recognize specific commands, control IOT, and the smart home. Fig 7 VIT system block diagram     In NXP RT1060 SDK code, the generated wake word and command word have been provided and placed in the VIT_Model.h file. If in the customer's project, how to customize the wake word and command word? With the NXP's efforts, we have made a web page form for customers to choose their own command, and then generate the corresponding VIT_Model.h file for code to call. VIT command word generation web page is: https://vit.nxp.com/#/home     Login the NXP account, choose the RT chip partn umber, wakeup word, voice command. Please note, the current supported RT chip is: RT1060,RT1160,RT1170,RT600,RT500 The following is the example for generating wakeup word and voice command:   Fig 8 Custom VIT configuration Fig 9 generated result Download the generated model, you can get VIT_Model_cn.h, open to see the command word information and related model data stored in the const PL_MEM_ALIGN (PL_UINT8 VIT_Model_cn[], VIT_MODEL_ALIGN_BYTES) array, the command word information is as follows: WakeWord supported : " 小恩 小恩 " Voice Commands supported     Cmd_Id : Cmd_Name       0    : UNKNOWN       1    : 开灯       2    : 关灯       3    : 今天 天气       4    : 明天 天气       5    : 后天 天气 Use the RT1060 SDK maestro_record demo to test this custom command result:   Fig 10 Custom Wakeup word and voice command test From the test result, we can see, both the wakeup word and voice command is detected. 3 Software code 3.1 LWIP socket client code capture weather API From chapter 2.2, we have been able to obtain the weather API and through testing, we can successfully achieve weather acquisition, so we need to add relevant commands in combination with the needs of our own system. For the acquisition of the weather API, the lwip code based on the RT1060 SDK is in the form of socket client. The relevant code is as follows: #define PORT 80 #define IP_ADDR "59.82.9.133" uint8_t get_weather[]= "GET /v3/weather/weatherInfo?key=xxx&city=310000&extensions=all&output=JSON HTTP/1.1\r\nHost: restapi.amap.com\r\n\r\n\r\n\r\n"; if (sys_thread_new("weather_main", weathermain_thread, NULL, HTTPD_STACKSIZE, HTTPD_PRIORITY) == NULL) LWIP_ASSERT("main(): Task creation failed.", 0); static void weathermain_thread(void *arg) { static struct netif netif; ip4_addr_t netif_ipaddr, netif_netmask, netif_gw; ethernetif_config_t enet_config = { .phyHandle = &phyHandle, .macAddress = configMAC_ADDR, }; LWIP_UNUSED_ARG(arg); mdioHandle.resource.csrClock_Hz = EXAMPLE_CLOCK_FREQ; IP4_ADDR(&netif_ipaddr, configIP_ADDR0, configIP_ADDR1, configIP_ADDR2, configIP_ADDR3); IP4_ADDR(&netif_netmask, configNET_MASK0, configNET_MASK1, configNET_MASK2, configNET_MASK3); IP4_ADDR(&netif_gw, configGW_ADDR0, configGW_ADDR1, configGW_ADDR2, configGW_ADDR3); tcpip_init(NULL, NULL); netifapi_netif_add(&netif, &netif_ipaddr, &netif_netmask, &netif_gw, &enet_config, EXAMPLE_NETIF_INIT_FN, tcpip_input); netifapi_netif_set_default(&netif); netifapi_netif_set_up(&netif); PRINTF("\r\n************************************************\r\n"); PRINTF(" TCP client example\r\n"); PRINTF("************************************************\r\n"); PRINTF(" IPv4 Address : %u.%u.%u.%u\r\n", ((u8_t *)&netif_ipaddr)[0], ((u8_t *)&netif_ipaddr)[1], ((u8_t *)&netif_ipaddr)[2], ((u8_t *)&netif_ipaddr)[3]); PRINTF(" IPv4 Subnet mask : %u.%u.%u.%u\r\n", ((u8_t *)&netif_netmask)[0], ((u8_t *)&netif_netmask)[1], ((u8_t *)&netif_netmask)[2], ((u8_t *)&netif_netmask)[3]); PRINTF(" IPv4 Gateway : %u.%u.%u.%u\r\n", ((u8_t *)&netif_gw)[0], ((u8_t *)&netif_gw)[1], ((u8_t *)&netif_gw)[2], ((u8_t *)&netif_gw)[3]); PRINTF("************************************************\r\n"); sys_thread_new("weather", weather_thread, NULL, DEFAULT_THREAD_STACKSIZE, DEFAULT_THREAD_PRIO); vTaskDelete(NULL); } static void weather_thread(void *arg) { int sock = -1,rece; struct sockaddr_in client_addr; char* host_ip; ip4_addr_t dns_ip; err_t err; uint32_t *pSDRAM= pvPortMalloc(BUF_LEN);// host_ip = HOST_NAME; PRINTF("host name : %s , host_ip : %s\r\n",HOST_NAME,host_ip); while(1) { sock = socket(AF_INET, SOCK_STREAM, 0); if (sock < 0) { PRINTF("Socket error\n"); vTaskDelay(10); continue; } client_addr.sin_family = AF_INET; client_addr.sin_port = htons(PORT); client_addr.sin_addr.s_addr = inet_addr(host_ip); memset(&(client_addr.sin_zero), 0, sizeof(client_addr.sin_zero)); if (connect(sock, (struct sockaddr *)&client_addr, sizeof(struct sockaddr)) == -1) { PRINTF("Connect failed!\n"); closesocket(sock); vTaskDelay(10); continue; } PRINTF("Connect to server successful!\r\n"); write(sock,get_weather,sizeof(get_weather)); while (1) { rece = recv(sock, (uint8_t*)pSDRAM, BUF_LEN, 0);//BUF_LEN if (rece <= 0) break; memcpy(weather_data.weather_info, pSDRAM,1500);//max 1457 } Weather_process(); memset(pSDRAM,0,BUF_LEN); closesocket(sock); vTaskDelay(10000); } }  3.2 VIT detect customer command code    Put the generated VIT_Model_cn.h to the maestro_record folder path:   vit\RT1060_CortexM7\Lib    The specific wake word and voice command related code can be viewed from the code vit_pro.c, mainly involving function is: int VIT_Execute(void *arg, void *inputBuffer, int size) The code is modified as follows, mainly to record the wake and wake word number, for specific function control, the command directly controlled here is the local "开灯:turn on the light", "关灯:turn off the light" command, as for the weather command needs to call the socket client API, so in the main lwip call area combined with the command word recognition number to call: if (VIT_DetectionResults == VIT_WW_DETECTED) { PRINTF(" - WakeWord detected \r\n"); weather_data.ww_flag = 1; //kerry } else if (VIT_DetectionResults == VIT_VC_DETECTED) { // Retrieve id of the Voice Command detected // String of the Command can also be retrieved (when WW and CMDs strings are integrated in Model) VIT_Status = VIT_GetVoiceCommandFound(VITHandle, &VoiceCommand); if (VIT_Status != VIT_SUCCESS) { PRINTF("VIT_GetVoiceCommandFound error: %d\r\n", VIT_Status); return VIT_Status; // will stop processing VIT and go directly to MEM free } else { PRINTF(" - Voice Command detected %d", VoiceCommand.Cmd_Id); weather_data.vc_index = VoiceCommand.Cmd_Id;//kerry 1:ledon 2:ledoff 3:today weather 4:tomorrow weather 5:aftertomorrow weather if(weather_data.vc_index == 1)//1 { GPIO_PinWrite(GPIO1, 3, 1U); //pull high PRINTF(" led on!\r\n"); } else if(weather_data.vc_index == 2)//2 { GPIO_PinWrite(GPIO1, 3, 0U); //pull low PRINTF(" led off!\r\n"); } // Retrieve CMD Name: OPTIONAL // Check first if CMD string is present if (VoiceCommand.pCmd_Name != PL_NULL) { PRINTF(" %s\r\n", VoiceCommand.pCmd_Name); } else { PRINTF("\r\n"); } } }  3.3 Voice recognize weather information    In the weather_thread while, check the wakeup word and voice command, if meet the requirement, then create the socket connection, write the API and capture the weather data.   The related code is: while(1) { //add the command request, only cmd == weather flag, then call it. if((weather_data.ww_flag == 1)) { if(weather_data.vc_index >= 3) { // create connection //write API and read API Weather_process(); } memset(weather_data.weather_info, 0, sizeof(weather_data.weather_info)); weather_data.ww_flag = 0; weather_data.vc_index = 0; } vTaskDelay(10000); } void Weather_process(void) { char * datap, *datap1; datap = strstr((char*)weather_data.weather_info,"date"); if(datap != NULL) { memcpy(today_weather, datap,184);//max 1457 if(weather_data.vc_index == 3) { PRINTF("\r\n*******************today weather***********************************\n\r"); PRINTF("%s\r\n",today_weather); return; } } else return; datap1 = strstr(datap+4,"date"); if(datap1 != NULL) { memcpy(tomorr_weather, datap1,184);//max 1457 if(weather_data.vc_index == 4) { PRINTF("\r\n*******************tomorrow weather*******************************\n\r"); PRINTF("%s\r\n",tomorr_weather); return; } } else return; datap = strstr(datap1+4,"date"); if(datap != NULL) { memcpy(aftertom_weather, datap,184);//max 1457 if(weather_data.vc_index == 5) { PRINTF("\r\n*******************after tomorrow weather**************************\n\r"); PRINTF("%s\r\n",aftertom_weather); } } else return; }   Function Weather_process is used to refer to the voice recognized weather number to get the related date’s weather, and printf it. 4 Test result  the test result video: Print the log results as shown in Figure 11, after testing, you can see that the wakeup word and voice command can be successfully recognized, in the recognition of word sequence numbers 3, 4, 5 is the weather acquisition, you can successfully call the lwip socket client API, successfully obtain weather information and printf it.   Fig 11 system test print result  evkmimxrt1060_maestro_weather_backup.zip is the project without sound playback, weather information will print to the terminal! 5 Meet issues conclusion 5.1 LWIP failed to get weather    When creating the code, call the postman provided http code: GET /v3/weather/weatherInfo?key=8f777fc7d867908eebbad7f96a13af10&amp; city=310000&amp; extensions=all&amp; output=JSON HTTP/1.1 Host: restapi.amap.com    Add it to the socket API function: uint8_t get_weather[]= "GET /v3/weather/weatherInfo?key=xxx&amp;city=310000&amp;extensions=all&amp;output=JSON HTTP/1.1\r\nHost: restapi.amap.com\r\n\r\n\r\n\r\n";    The test result is:   Fig 12 socket weather API return issues     We can see, server connection is OK, http also return back the data, but it report the parameter issues, after checking, we use the postman C code, and put it to the get_weather: uint8_t get_weather[]= "GET /v3/weather/weatherInfo?key=xxx&city=310000&extensions=all&output=JSON HTTP/1.1\r\nHost: restapi.amap.com\r\n\r\n\r\n\r\n"; Then, it can capture the weather data, the same as postman test result. 5.2 VIT LWIP merger memory is not enough     After combining the maestro_record and lwip socket code together, compile it, it will meet the DTCM memory overflow issues. Fig 13 memory overflow After optimize, still meet the DTCM overflow issues, so, at last, choose to reconfigure the FlexRAM: OCRAM 192K, DTCM 256K, ITCM 64K Compile it, and the memory overflow issues disappear:   Fig 14 FlexRAM recofiguration 5.3 Print Chinese word in tera    Directly use teraterm, when the weather API returns the Chinese word, the print out information is the garbled code, and then after the following configuration, to achieve Chinese printing: Setup  ->  Terminal Locale    : american->chinese Codepage : 65001 ->936 Fig 15 Tera Term Chinese word print In summary, after various data collection and problem solving, in MIMXRT1060-EVK board  combined with the official SDK complete the function of customizing VIT voice commands to obtain real-time weather and local control.So, even if the ordinary RT series which is not S/L/A/F series, you also can use VIT to implement speech recognition functions. 6 Add the sound broadcast    This chapter mainly gives the method how to add the sound broadcast with the mp3 video data which is stored in the memory, but to the realtime weather data playback, it is not very freely, it needs to check the weather data, and use the video mp3 data lib get the correct mp3 data, as it is not the online TTS method.     So, here, just share one example add the sound broadcast, eg: WW : “小恩小恩”    ->   “小恩来了，请吩咐！” VC  ：“今天天气”   ->   “温度32.1度” VC playback is fixed now, if need to play real data, it needs to generate the mp3 voice data lib, then according to the feedback weather information, to generate the correct weather mp3 data array, and play it, as this is a little complicated, but not difficult, so here, just use one fixed sound give an example of it. 6.1 MP3 playback audio data preparation     For audio broadcasting which need to convert the Chinese word into MP3 files, you can use some online speech synthesis software, here use Baidu online speech synthesis function, you can view the previous article, chapter 2.2.2 online speech synthesis: https://community.nxp.com/t5/i-MX-RT-Knowledge-Base/RT106L-S-voice-control-system-based-on-the-Baidu-cloud/ta-p/1363295     If use the Baidu online speech synthesis generated mp3 file to convert to the c array directly, it will meet the first audio play issues, so, here we use the Audacity to convert the mp3 file, the convert configuration is like this:  Fig 16 Audacity convert configuration     After the regeneration of mp3, you can use xxd .exe to convert the mp3 file to an array of C files, and then put it into RT-related memory or external flash , xxd .exe can be found at the following link: https://github.com/baldram/ESP_VS1053_Library/issues/18 The convert command like this: xxd -i your-sound.mp3 ready-to-use-header.c Convert the xiaoencoming.mp3 and temptest.mp3 file to the C array, then modify the data to the C file, save file as: xiaoencoming.h and temptest.h. Here, take xiaoencoming.c as an example: #define XIAOEN_MP3_SIZE  6847 unsigned char xiaoencoming_mp3[XIAOEN_MP3_SIZE] = {   0x49, 0x44, 0x33, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x21, 0x54, 0x58, …   0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55 }; unsigned int xiaoencoming1_mp3_len = XIAOEN_MP3_SIZE;//6847; Until now, the playback audio data is finished.     Copy xiaoencoming.h and temptest.h to project path: evkmimxrt1060_maestro_weather_mp3\source 6.2 Play the MP3 data from memory    Here, share the related code. 6.2.1 app_streamer.c added code    #include "xiaoencoming.h" #include "temptest.h" void *voice_inBuf = NULL; void *voice_outBuf = NULL; status_t STREAMER_file_Create(streamer_handle_t *handle, char *filename, int eap_par) { STREAMER_CREATE_PARAM params; OsaThreadAttr thread_attr; int ret; ELEMENT_PROPERTY_T prop; MEMSRC_SET_BUFFER_T inBufInfo = {0}; SET_BUFFER_DESC_T outBufInfo = {0}; PRINTF("Kerry test begin!\r\n"); if(filename == "temptest.mp3") inBufInfo = (MEMSRC_SET_BUFFER_T){.location = (int8_t *)temptest_mp3, .size = TEMPtest_MP3_SIZE}; else if(filename == "xiaoencoming.mp3") inBufInfo = (MEMSRC_SET_BUFFER_T){.location = (int8_t *)xiaoencoming_mp3, .size = XIAOEN_MP3_SIZE}; /* Create message process thread */ osa_thread_attr_init(&thread_attr); osa_thread_attr_set_name(&thread_attr, STREAMER_MESSAGE_TASK_NAME); osa_thread_attr_set_stack_size(&thread_attr, STREAMER_MESSAGE_TASK_STACK_SIZE); ret = osa_thread_create(&msg_thread, &thread_attr, STREAMER_MessageTask, (void *)handle); osa_thread_attr_destroy(&thread_attr); if (ERRCODE_NO_ERROR != ret) { return kStatus_Fail; } /* Create streamer */ strcpy(params.out_mq_name, APP_STREAMER_MSG_QUEUE); params.stack_size = STREAMER_TASK_STACK_SIZE; params.pipeline_type = STREAM_PIPELINE_MEM; params.task_name = STREAMER_TASK_NAME; params.in_dev_name = "buffer"; params.out_dev_name = "speaker"; handle->streamer = streamer_create(&params); if (!handle->streamer) { return kStatus_Fail; } prop.prop = PROP_DECODER_DECODER_TYPE; prop.val = (uintptr_t)DECODER_TYPE_MP3; ret = streamer_set_property(handle->streamer, prop, true); if (ret != STREAM_OK) { streamer_destroy(handle->streamer); handle->streamer = NULL; return kStatus_Fail; } prop.prop = PROP_MEMSRC_SET_BUFF; prop.val = (uintptr_t)&inBufInfo; ret = streamer_set_property(handle->streamer, prop, true); if (ret != STREAM_OK) { streamer_destroy(handle->streamer); handle->streamer = NULL; return kStatus_Fail; } handle->audioPlaying = false; error: PRINTF("End STREAMER_file_Create\r\n"); PRINTF("Kerry test end!\r\n"); return kStatus_Success; }   The code implements the thread build, creates a streamer, defines it as playing from memory, decodes the properties for MP3, and specifies an array of MP3 files in memory. Specify a different array of mp3 files in memory based on the calling file name. 6.2.2 cmd.c added code void play_file(char *filename, int eap_par) { STREAMER_Init(); int ret = STREAMER_file_Create(&streamerHandle, filename, eap_par); if (ret != kStatus_Success) { PRINTF("STREAMER_file_Create failed\r\n"); goto file_error; } STREAMER_Start(&streamerHandle); PRINTF("Starting playback\r\n"); file_playing = true; while (streamerHandle.audioPlaying) { osa_time_delay(100); } file_playing = false; file_error: PRINTF("[play_file] Cleanup\r\n"); STREAMER_Destroy(&streamerHandle); osa_time_delay(100); }   Play file, it calls the STREAMER_file_Create API function, start play, and wait the play finished, then release the STREAMER. shellRecMIC API function add the VIT recorded flag, which is used to play feedback audio file. static shell_status_t shellRecMIC(shell_handle_t shellHandle, int32_t argc, char **argv) { … //kerry PRINTF("Kerry MP3 stream data test!\r\n"); PRINTF("---weather_data.ww_flag =%d--\r\n ", weather_data.ww_flag); PRINTF("---weather_data.vc_inde =%d--\r\n ", weather_data.vc_index); PRINTF("---weather_data.mp3_flag =%d--\r\n ", weather_data.mp3_flag); if(weather_data.ww_flag == 1) { play_file("xiaoencoming.mp3", 0); } if(weather_data.vc_index == 3) { play_file("temptest.mp3", 0); } if(weather_data.mp3_flag != 0) { weather_data.ww_flag = 0; weather_data.vc_index = 0; } weather_data.mp3_flag = 0; /* Delay for cleanup */ osa_time_delay(100); return kStatus_SHELL_Success; } If detect the Wakeup Word: “小恩小恩”, play feedback audio: “小恩来了请吩咐”. If detect the voice command: “今天天气”， play feedback audio: “温度32.1度”, please note, this playback just an example, it is the fixed audio, you also can create audio word lib, then according to the received weather information, combine the related word audio together, then playback it. This is a little complicated, but not difficult. So, if need to play the free audio, also can consider the online TTS method in real time. 6.2.3 VIT WW and VC flag VIT_Execute function int VIT_Execute(void *arg, void *inputBuffer, int size) { … if (VIT_DetectionResults == VIT_WW_DETECTED) { PRINTF(" - WakeWord detected \r\n"); weather_data.ww_flag = 1; //kerry weather_data.mp3_flag = 1; } else if (VIT_DetectionResults == VIT_VC_DETECTED) { // Retrieve id of the Voice Command detected // String of the Command can also be retrieved (when WW and CMDs strings are integrated in Model) VIT_Status = VIT_GetVoiceCommandFound(VITHandle, &VoiceCommand); if (VIT_Status != VIT_SUCCESS) { PRINTF("VIT_GetVoiceCommandFound error: %d\r\n", VIT_Status); return VIT_Status; // will stop processing VIT and go directly to MEM free } else { PRINTF(" - Voice Command detected %d", VoiceCommand.Cmd_Id); weather_data.vc_index = VoiceCommand.Cmd_Id;//kerry 1:ledon 2:ledoff 3:today weather 4:tomorrow weather 5:aftertomorrow weather weather_data.mp3_flag = 2; if(weather_data.vc_index == 1)//1 { GPIO_PinWrite(GPIO1, 3, 1U); //pull high PRINTF(" led on!\r\n"); } else if(weather_data.vc_index == 2)//2 { GPIO_PinWrite(GPIO1, 3, 0U); //pull low PRINTF(" led off!\r\n"); } // Retrieve CMD Name: OPTIONAL // Check first if CMD string is present if (VoiceCommand.pCmd_Name != PL_NULL) { PRINTF(" %s\r\n", VoiceCommand.pCmd_Name); } else { PRINTF("\r\n"); } } } return VIT_Status; }   Until now, all the code is added. 6.2.4  playback audio test result     This is the audio playback test result:   Fig 17 playback audio log   From the test result, we can see, we also can use the mp3 data which is stored in the memory and play it as audio playback.   The code project is: evkmimxrt1060_maestro_weather_mp3.zip.  

RT10xx image reserve the APP FCB methods 1. Abstract     Regarding RT10XX programming, it is mainly divided into two categories: 1) Serial download mode with blhost proramming     To this method, we can use the MCUBootUtility tool, or blhost+elftosb+sdphost cmd method, we also can use the NXP SPT(MCUXpresso secure provisional Tool). This programming need to enter the serial download mode, then use the flashloader supported UART or the USB HID interface. 2) Use Programmer or debugger with flashdriver programming This method is usually through the SWD/JTAG download interface combined with the debugger + IDE, or directly software burning, the chip mode can be in the internal boot, or in the serial download mode, with the help of the flashloader to generate the flash burning algorithm file. Method 2, The burning method using the debugger tool usually ensures that the burning code is consistent with the original APP.     Method 1, Uses the blhost method to download, usually blhost will regenerate an FCB with a full-featured LUT to burn to the external flash, and then burn the app code with IVT, that is, without the FCB header of the original APP, and re-assemble a blhost generated FCB header and burn it separately. However, for some customers who need to read out the flash image and compare with the original APP image to check the difference after burning, the commonly used blhost method will have the problem of inconsistent FCB area matching. If the customer needs to use the blhost burning method in serial download mode, how to ensure that the flash image after burning is consistent with the original burning file? This article will take the MIMXRT1060-EVK development board as an example, and give specific methods for the command mode and SPT tool mode. 2 Blhost programming reserve APP FCB     From the old RT1060 SDK FCB file (below SDK2.12.0), evkmimxrt1060_flexspi_nor_config.c, we can see:   const flexspi_nor_config_t qspiflash_config = { .memConfig = { .tag = FLEXSPI_CFG_BLK_TAG, .version = FLEXSPI_CFG_BLK_VERSION, .readSampleClksrc=kFlexSPIReadSampleClk_LoopbackFromDqsPad, .csHoldTime = 3u, .csSetupTime = 3u, .sflashPadType = kSerialFlash_4Pads, .serialClkFreq = kFlexSpiSerialClk_100MHz, .sflashA1Size = 8u * 1024u * 1024u, .lookupTable = { // Read LUTs FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0xEB, RADDR_SDR, FLEXSPI_4PAD, 0x18), FLEXSPI_LUT_SEQ(DUMMY_SDR, FLEXSPI_4PAD, 0x06, READ_SDR, FLEXSPI_4PAD, 0x04), }, }, .pageSize = 256u, .sectorSize = 4u * 1024u, .blockSize = 64u * 1024u, .isUniformBlockSize = false, };   This FCB LUT just contains the basic read command, normally, to the app booting, the FCB just need to provide the read command to the ROM, then it can boot normally.     But what happens to the memory downloaded by blhost? Based on the MIMXRT1060-EVK development board, the following shows how to use the command line mode corresponding to blhost to burn the SDK led_blinky project app, and read out the corresponding flash burning code to analysis. 2.1 Normal blhost download command line    This command line also the same as MCUBootUtility download log, source code is attached rt1060 cmd.bat. elftosb.exe -f imx -V -c imx_application_gen.bd -o ivt_evkmimxrt1060_iled_blinky_FCB.bin evkmimxrt1060_iled_blinky.s19 sdphost.exe -t 50000 -u 0x1FC9,0x0135 -j -- write-file 0x20208200 ivt_flashloader.bin sdphost.exe -t 50000 -u 0x1FC9,0x0135 -j -- jump-address 0x20208200 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- get-property 1 0 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- get-property 24 0 blhost.exe -t 5242000 -u 0x15A2,0x0073 -j -- fill-memory 0x20202000 4 0xc0000007 word  //option 0 blhost.exe -t 5242000 -u 0x15A2,0x0073 -j -- fill-memory 0x20202004 4 0 word                 //option1 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- configure-memory 9 0x20202000                    blhost -t 2048000 -u 0x15A2,0x0073 -j -- flash-erase-region 0x60000000 0x8000 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- fill-memory 0x20203000 4 0XF000000F word  blhost -t 50000 -u 0x15A2,0x0073 -j -- configure-memory 9 0x20203000                    blhost -t 5242000 -u 0x15A2,0x0073 -j -- write-memory 0x60001000 ivt_evkmimxrt1060_iled_blinky_FCB_nopadding.bin 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 0x8000 flexspiNorCfg.dat 9 The normal blhost programming is to use the cmd line method, and provide an app which is without the FCB header(Even app with the FCB, will exclude the FCB header at first), then use the elftosb.exe generate the app with IVT, eg ivt_evkmimxrt1060_iled_blinky_FCB_nopadding.bin, download the flashloader file ivt_flashloader to internal RAM, and jump to the flashloader, then use the fill-memory to fill option0, option1 to choose the proper external flash, and use the configure-memory to configure the flexSPI module, with the SFDP table which is got from get configure command, then fill the flexSPI LUT internal buffer. Next, fill-memory 0x20203000 4 0XF000000F associate with configure-memory will generate the full FCB header, burn it from flash address 0x60000000. At last, burn the app which contains IVT from flash address 0X60001000, until now, realize the whole app image programming. Pic 1 shows the comparison between the data read after programming and the original app data. It can be seen that the LUT of the FCB actually programmed on the left is not only contains read, but also contains read status, write enable, program and erase commands. The one on the right is the original app with FCB. The LUT of FCB only contains read commands for boot. So, if you want to keep the FCB header of the original APP instead of the header generated and burned by option0,1 configure-memory, how to do it? The method is that you can also use Option0, 1 to generate and fill in the LUT for flexSPI for communication use, but do not burn the corresponding generated FCB, just burn the FCB that comes with the original APP. pic1 2.2 Reuse option0 and option1 to program the original APP LUT The following command gives reuse option0 and option1, generates LUT and fills in flexSPI LUT for connection with external flash interface, but does not call:  fill-memory 0x20203000 4 0XF000000F and configure-memory 9 0x20203000, so that the generated FCB will not be burned to external memory.    Source file is attached rt1060 cmd_option01.bat. elftosb.exe -f imx -V -c imx_application_gen.bd -o ivt_evkmimxrt1060_iled_blinky_FCB.bin evkmimxrt1060_iled_blinky.s19 sdphost.exe -t 50000 -u 0x1FC9,0x0135 -j -- write-file 0x20208200 ivt_flashloader.bin sdphost.exe -t 50000 -u 0x1FC9,0x0135 -j -- jump-address 0x20208200 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- get-property 1 0 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- get-property 24 0 blhost.exe -t 5242000 -u 0x15A2,0x0073 -j -- fill-memory 0x20202000 4 0xc0000007 word blhost.exe -t 5242000 -u 0x15A2,0x0073 -j -- fill-memory 0x20202004 4 0 word blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- configure-memory 9 0x20202000 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 1024 flexspiNorCfg.dat 9 blhost -t 2048000 -u 0x15A2,0x0073 -j -- flash-erase-region 0x60000000 0x8000 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 1024 flexspiNorCfg.dat 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- write-memory 0x60000000 evkmimxrt1060_iled_blinky_FCB.bin 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 0x8000 flexspiNorCfg.dat 9 Pic 2 is the comparison between the read data after programming and the original programming data. It can be seen that the FCB programmed at this time is exactly the same as the original code FCB. Pic 2 2.3 use 1bit FCB file to configure LUT    The used file cfg_fdcb_RTxxx_1bit_sdr_flashA.bin is copied from MCUBOOTUtility: \NXP-MCUBootUtility-3.4.0\src\targets\fdcb_model . The configuration of Option0 and Option1 is usually for chips that can support SFDP table, but some flash chips cannot support SFDP table. At this time, you need to fill in the flexSPI LUT for the full LUT manually. The so-called full LUT command is not only read commands, but also supports erasing, program, etc. In this way, the flexSPI interface can be successfully connected to the external FLASH, and the corresponding functions of reading, erasing, and writing can be realized. Therefore, the method in this chapter is to use a single-line command, which is also a command supported by general chips, to enable the corresponding function of flexSPI, so it can complete the subsequent APP code programming.   Pic 3     We can see: 03H is read, 05H is read status register, 06H is write enable, D8H is the block 64K erase, 02H is the page program, 60H is the chip erase. This is the 1bit SPI method full function LUT command, which can realize the chip read, write and erase function.     The command line is, source file is attached rt1060 cmd_fdcb_1bit_sdr_flashA.bat: elftosb.exe -f imx -V -c imx_application_gen.bd -o ivt_evkmimxrt1060_iled_blinky_FCB.bin evkmimxrt1060_iled_blinky.s19 sdphost.exe -t 50000 -u 0x1FC9,0x0135 -j -- write-file 0x20208200 ivt_flashloader.bin sdphost.exe -t 50000 -u 0x1FC9,0x0135 -j -- jump-address 0x20208200 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- get-property 1 0 blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- get-property 24 0 blhost -t 5242000 -u 0x15A2,0x0073 -j -- write-memory 0x20202000 cfg_fdcb_RTxxx_1bit_sdr_flashA.bin blhost.exe -t 50000 -u 0x15A2,0x0073 -j -- configure-memory 9 0x20202000 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 1024 flexspiNorCfg.dat 9 blhost -t 2048000 -u 0x15A2,0x0073 -j -- flash-erase-region 0x60000000 0x8000 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 1024 flexspiNorCfg.dat 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- write-memory 0x60000000 evkmimxrt1060_iled_blinky_FCB.bin 9 blhost -t 5242000 -u 0x15A2,0x0073 -j -- read-memory 0x60000000 0x8000 flexspiNorCfg.dat 9 In the command line, where option0,1 was previously filled in, instead of filling in the data of option0,1, the 512-byte Bin file of the complete FCB LUT command is directly given, and then the configure-memory command is used to configure the flashloader’s FlexSPI LUT with the FCB file. so that it can support read and write erase commands, etc. The comparison between the flash data and the original APP data when burning and reading is in the Pic 4, we can see, the readout data from the flash is totally the same as the original APP FCB. Pic 4 3，SPT program reserve APP FCB The NXP officially released MCUXPresso Secure Provisional Tool can support the function of retaining the customer's FCB, but the SPT tool currently uses the APP FCB to fill in the flashloader FlexSPI FCB. Therefore, if the customer directly uses the old SDK demo which just contains the read command in the LUT to generate an APP with FCB, then use the SPT tool to burn the flash, and choose to keep the customer FCB in the tool, you will encounter the problem of erasing failure. In this case, analyze the reason, we can know the FCB on the customer APP side needs to fill in the full FCB LUT command, that is, including reading, writing, erasing, etc. The following shows how the old original SDK led_blinky generates an image with an FCB header and writes it in the SPT tool. As you can see in Pic 5, the tool has information that if you use APP FCB, you need to ensure that the FCB LUT contains the read, erase, program commands. Pic 6 shows the programming situation of APP FCB LUT only including read. It has failed when doing erase. The reason is that there is no erase, program and other commands in the FlexSPI LUT command, so it will fail when doing the corresponding erasing or programming.   Pic 5 Pic 6 Pic 7 If you look at the specific command, as shown in Pic 7, you can find that the SPT tool directly uses the FCB header extracted from the APP image to flash the LUT of the flashloader FlexSPI, so there will be no erase and write commands, and it will fail when erasing. The following is how to fill in the LUT in the FCB of the SDK, open evkmimxrt1060_flexspi_nor_config.c, and modify the FCB as follows: const flexspi_nor_config_t qspiflash_config = {     .memConfig =         {             .tag              = FLEXSPI_CFG_BLK_TAG,             .version          = FLEXSPI_CFG_BLK_VERSION,             .readSampleClksrc=kFlexSPIReadSampleClk_LoopbackFromDqsPad,             .csHoldTime       = 3u,             .csSetupTime      = 3u,             .sflashPadType    = kSerialFlash_4Pads,             .serialClkFreq    = kFlexSpiSerialClk_100MHz,             .sflashA1Size     = 8u * 1024u * 1024u,             .lookupTable =                 {                   // Read LUTs                   FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0xEB, RADDR_SDR, FLEXSPI_4PAD, 0x18),                   FLEXSPI_LUT_SEQ(DUMMY_SDR, FLEXSPI_4PAD, 0x06, READ_SDR, FLEXSPI_4PAD, 0x04),                   // Read status                   [4*1] = FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0x05, READ_SDR, FLEXSPI_1PAD, 0x04),                   //write Enable                   [4*3] = FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0x06, STOP, FLEXSPI_1PAD, 0),                   // Sector Erase byte LUTs                   [4*5] = FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0x20, RADDR_SDR, FLEXSPI_1PAD, 0x18),                   // Block Erase 64Kbyte LUTs                   [4*8] = FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0xD8, RADDR_SDR, FLEXSPI_1PAD, 0x18),                    //Page Program - single mode                   [4*9] = FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0x02, RADDR_SDR, FLEXSPI_1PAD, 0x18),                   [4*9+1] = FLEXSPI_LUT_SEQ(WRITE_SDR, FLEXSPI_1PAD, 0x04, STOP, FLEXSPI_1PAD, 0x0),                   //Erase whole chip                   [4*11] =FLEXSPI_LUT_SEQ(CMD_SDR, FLEXSPI_1PAD, 0x60, STOP, FLEXSPI_1PAD, 0),                                       },         },     .pageSize           = 256u,     .sectorSize         = 4u * 1024u,     .blockSize          = 64u * 1024u,     .isUniformBlockSize = false, }; Please note, after the internal SDK team modification, from SDK_2_12_0_EVK-MIMXRT1060, the evkmimxrt1060_flexspi_nor_config.c already add LUT cmd to the full FCB LUT function. Use the above FCB to generate the APP, then use the SPT tool to burn the app with customer FCB again, we can see, the programming is working now. Pic 8 In summary, if you need to reserve the customer FCB, you can use the above method, but if you use the SPT tool, you need to add read, write, and erase commands to the LUT of the code FCB to ensure that flexSPI successfully operates the external flash.

RT1170 Octal flash enablement 1. Abstract The MIMXRT1170-EVK hardware can directly support two types of flash: QSPI flash IS25WP128 and Octal flash MX25UM51345GXDI00. QSPI flash is used by default, so the FCB of the related SDK and some IDE debugger download flash drivers are also default to the QSPI flash. In practical usage, some customers need to use Octal flash with RT1170, but after modifying the EVK hardware to Octal flash MX25UM51345GXDI00, they may encounter various issues, such as download issues, boot issues, debug issues, and when the flash is empty or there is a valid boot code in the flash, the download result is different and so on. This article will be based on NXP's official MIMXRT1170-EVK REV C1, modify the hardware onboard flash from QSPI flash to Octal flash MX25UM51345GXDI00, and modify the SDK project FCB, do ROM API test, do related tool download, do the debugger related flashloader modification and test with different IDE, so that customers who need it can refer to it. This article is not limited to EVK, but also applies to RT1170+Octal flash MX25UM51345GXDI00 with customer-defined board Pic 1 2. Hardware modification MIMXRT1170-EVK board modify flash to the octal flash, it needs to disconnect the QSPI flash pins, and connect the octal flash pins, the related modification points are: OPTION2: USE Octal Flash( Mount R381/R378/R382/R389/R402/R377/R388/R391, DNP R380/R399/R386/R390/R392/R385) Pic 2 The octal flash BOOT_CFG pin configuration are as follows: Pic 3 Pic 4 After modifying the hardware related resistors to choose the octal flash, SW1, SW2 should be configured to internal boot, and boot from octal flash, then it is controlled by the software part. 3. Octal flash APP FCB 3.1 Test SDK ROM API run from RAM To verify the board hardware can run from the octal flash, we can test the SDK attached fsl_romapi project: \SDK_2_12_0_MIMXRT1170-EVK\boards\evkmimxrt1170\driver_examples\fsl_romapi\cm7 Please note, can’t test the default project directly, as this project configuration is used by the QSPI flash, not the octal flash, if need to use the octal flash, customer need to modify the code at first, and add the GPIO to control the Flash_RST pin, which is used to reset the octal flash. The option0,1 value need to be modified to the octal flash values. 3.1.1 Octal flash option value     Pic 5 We can use the following option0 for testing, don’t need to configure the option1, as the octal flash is connected to the primary pin. option0= 0Xc0400007, option1 = 0 //query_pad =1, cmd_pad=1,133MHz option0= 0Xc0433007, option1 = 0 //query_pad =8, cmd_pad=8,133MHz option0= 0Xc0403007, option1 = 0 //query_pad =1, cmd_pad=8,133MHz 3.1.2 fsl_romapi project testing    The modified code for the ROM API project are as follows, before call ROM_FLEXSPI_NorFlash_GetConfig API, it needs to use the GPIO to reset the external octal flash at first, so, here add the Flash_RST=GPIO_AD_03=GPIO09_IO02 pin GPIO configuration. Pinmux.c related code：     gpio_pin_config_t gpio9_pinP15_config = { .direction = kGPIO_DigitalOutput, .outputLogic = 1U, .interruptMode = kGPIO_NoIntmode }; GPIO_PinInit(GPIO9, 02U, &gpio9_pinP15_config); IOMUXC_SetPinMux( IOMUXC_GPIO_AD_03_GPIO9_IO02, 0U); IOMUXC_SetPinConfig( IOMUXC_GPIO_AD_03_GPIO9_IO02,//OD，it is a workaround for EVK HW bug 0X12);     Flexspi_romapi.c added code:     static serial_nor_config_option_t option_1bit = { .option0.U = 0xc0400007U, .option1.U = 0U, }; static serial_nor_config_option_t option_8bit = { .option0.U = 0xc0433007, .option1.U = 0U, }; static serial_nor_config_option_t option_1_8bit = { .option0.U = 0xC0403007, .option1.U = 0U, }; GPIO_PinWrite(GPIO9, 02U, 0U); // Delay some time to reset external flash with Flash_RST pin for (uint32_t i = 0; i < 60000; i++) __asm volatile ("nop"); GPIO_PinWrite(GPIO9, 02U, 1U); status = ROM_FLEXSPI_NorFlash_GetConfig(FlexSpiInstance, &norConfig, &option_8bit); if (status == kStatus_Success) { PRINTF("\r\n Successfully get FLEXSPI NOR configuration block with option_8bit\r\n "); } else { status = ROM_FLEXSPI_NorFlash_GetConfig(FlexSpiInstance, &norConfig, &option_1_8bit); if (status == kStatus_Success) { PRINTF("\r\n Successfully get FLEXSPI NOR configuration block with option_1_8bit\r\n "); } else { status = ROM_FLEXSPI_NorFlash_GetConfig(FlexSpiInstance, &norConfig, &option_1bit); if (status == kStatus_Success) { PRINTF("\r\n Successfully get FLEXSPI NOR configuration block with option_1bit\r\n "); } else { PRINTF("\r\n Get FLEXSPI NOR configuration block failure!\r\n"); error_trap(); } } }     The test result is: Pic 6 Pic 7 From the test result, we can see, after using the GPIO to reset the external flash, the practical used option value is: option0= 0xc0403007, option1 = 0 //query_pad =1, cmd_pad=8,133MHz It can read the SFDP successfully, get the flash config data to the norConfig, then use these values to configure the flexSPI module, at last, it realizes the octal flash related address area erase-write-read-erase operation. From the above test result, it means, the modified hardware is totally working. Please note, as we can’t confirm the external flash hardware is working, so this project FCB is still the default for the QSPI flash, not the octal flash, then when testing this fsl_romapi project, it’s better to run it from RAM not the external flash. The following picture is how to run the project from the internal RAM, not download the flash directly: Pic 8    Here, for the configuration, we also have one point that needs to note, why configure the GPIO pin Flash_RST: GPIO_AD_03 as OD(open drain)?    It is caused by the current MIMXRT1170-EVK hardware having a bug, so this OD configuration is used as the workaround. Octal flash MX25UM51345GXDI00 power supply voltage is 1.8V, but the GPIO_AD_03 bank voltage is 3.3V, between these two modules, no voltage convert hardware, so it will have the following situation: a) Default GPIO_AD_03 is input mode, this pin has the internal PD(pull down) 35K resistor,  and the external PU(pull up) 10K resistor to the 1.8V, Flash_RST=1.8V*35K/(10K+35K)=1.4V, this voltage also can be used as the enable signal. To the normal boot, as the ROM didn’t control the octal flash Flash_RST pin, then this pin can be freely chose by any pin in the practical usage, so, the reset pin default is input, the voltage is 1.4V, it can enable the octal flash, no influence. b) When need to control the octal flash reset, then it needs to use the GPIO control GPIO_AD_03 pin, if want to output 0, it is OK. But if want to output high level, use the internal PULL resistor, it will output 3.3V to the Flash_RST pin, this voltage already higher than the octal flash 1.8V, and the octal flash datasheet defines the max voltage is 2V, although the flash chip has the voltage buffer, input 3.3V in short time, won’t damage the chip, but after long time working, the chip normal working can’t be guaranteed, this will cause the risk. So, to solve this issues, we can do some workaround in the flashloader, rom API, when output the Flash_RST pin to higher, we can use OD mode, then the detail pin voltage is totally determined by the external pull circuit, and the EVK also have the external 10K PU to 1.8V, so it can output 1.8V not the 3.3V. Pic 9   In the NXP new MIMXRT1170-EVKB board, the hardware adds the voltage convert chip to solve these issues, it can realize the 3.3V and 1.8V conversion: Pic 10 3.2 MCUBootUtility program octal flash We can use the MCUBootUtility to test MIMXRT1170-EVK+octal flash in serial download mode, normally for the chip connect, memory erase, read and program. This method also can use the tool to generated the correct FCB header for the octal flash, then can test the boot situation in the internal boot mode. Serial download-> SW1:1-OFF,2-OFF,3-OFF,4-ON Internal boot-> SW1:1-OFF,2-OFF,3-ON,4-OFF The MCUBootUtility configuration is: Pic 11 This configuration is: option0=0Xc0403007，query 1wire, cmd 8 wire. Then app image can use the MCUBootUtility attached code: NXP-MCUBootUtility-3.5.0\apps\NXP_MIMXRT1170-EVK_Rev.A\cm7\led_blinky_0x3000a000.srec Pic 12   We can see, the code downloading is finished, after this, reset device, configure the board as internal boot mode, then do the POR or the HW reset, we can see the on board LED D34 is blinking, it means the hardware also can boot from the octal flash. 3.3 SDK APP FDCB modification If you need to debug the SDK APP demo from octal flash, you must ensure that the app correct FDCB is provided, then how to modify the app FDCB to octal flash? You can refer to the FDCB which is burned by the MCUBootUtility tool and the datasheet of the octal flash. Here is the FDCB that has been tested for many times. The key point is to provide the correct LUT table. Taking the RT1170 SDK led_blinky project as an example, the evkmimxrt1170_flexspi_nor_config.c file in the project xip file is modified as follows:     const flexspi_nor_config_t octalflash_config = { .memConfig = { .tag = FLEXSPI_CFG_BLK_TAG, .version = FLEXSPI_CFG_BLK_VERSION, .readSampleClksrc=kFlexSPIReadSampleClk_ExternalInputFromDqsPad, .csHoldTime = 3, .csSetupTime = 3, .deviceModeCfgEnable = 1, .deviceModeType = kDeviceConfigCmdType_Spi2Xpi, .waitTimeCfgCommands = 1, .deviceModeSeq = { .seqNum = 1, .seqId = 6, /* See Lookup table for more details */ .reserved = 0, }, .deviceModeArg = 2, /* Enable OPI DDR mode */ .controllerMiscOption = (1u << kFlexSpiMiscOffset_SafeConfigFreqEnable) | (1u << kFlexSpiMiscOffset_DdrModeEnable), .deviceType = kFlexSpiDeviceType_SerialNOR, .sflashPadType = kSerialFlash_8Pads, .serialClkFreq = kFlexSpiSerialClk_133MHz, .sflashA1Size = 64ul * 1024u * 1024u, .dataValidTime = { [0] = {.time_100ps = 16}, }, .busyOffset = 0u, .busyBitPolarity = 0u, .lookupTable = { /* Read */// EEH+11H+32bit addr+20dummy cycles+ 4Bytes read data //133Mhz 20 dummy=10+10 [0 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0xEE, CMD_DDR, FLEXSPI_8PAD, 0x11),//0x871187ee, [0 + 1] = FLEXSPI_LUT_SEQ(RADDR_DDR, FLEXSPI_8PAD, 0x20, DUMMY_DDR, FLEXSPI_8PAD, 0x0A),//0xb30a8b20, [0 + 2] = FLEXSPI_LUT_SEQ(DUMMY_DDR, FLEXSPI_8PAD, 0x0A, READ_DDR, FLEXSPI_8PAD, 0x04),//0xa704b30a, /* Read Status SPI */// SPI 05h+ status data 0X24 maybe 0X04 [4*1 + 0] = FLEXSPI_LUT_SEQ(CMD_SDR , FLEXSPI_1PAD, 0x05, READ_SDR, FLEXSPI_1PAD, 0x24),//0x24040405, /* Read Status OPI *///05H+FAH+ 4byte 00H(addr)+4Byte read [4*2 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x05, CMD_DDR, FLEXSPI_8PAD, 0xFA),//0x87fa8705, [4*2 + 1] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x00, CMD_DDR, FLEXSPI_8PAD, 0x00),//0x87008700, [4*2 + 2] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x00, CMD_DDR, FLEXSPI_8PAD, 0x00),//0x87008700, [4*2 + 3] = FLEXSPI_LUT_SEQ(READ_DDR , FLEXSPI_8PAD, 0x04, STOP_EXE, FLEXSPI_1PAD, 0x00),//0x0000a704, /* Write enable SPI *///06h [4*3 + 0] = FLEXSPI_LUT_SEQ(CMD_SDR , FLEXSPI_1PAD, 0x06, STOP_EXE, FLEXSPI_1PAD, 0x00),//0x00000406, /* Write enable OPI *///06h+F9H [4*4 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x06, CMD_DDR, FLEXSPI_8PAD, 0xF9),//0x87f98706, /* Erase sector */ //21H+DEH + 32bit address [4*5 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x21, CMD_DDR, FLEXSPI_8PAD, 0xDE),//0x87de8721, [4*5 + 1] = FLEXSPI_LUT_SEQ(RADDR_DDR , FLEXSPI_8PAD, 0x20, STOP_EXE, FLEXSPI_1PAD, 0x00),//0x00008b20, /*Write Configuration Register 2 =01, Enable OPI DDR mode*/ //72H +32bit address + CR20x00000000 = 0x01 [4*6 + 0] = FLEXSPI_LUT_SEQ(CMD_SDR , FLEXSPI_1PAD, 0x72, CMD_SDR, FLEXSPI_1PAD, 0x00),//0x04000472, [4*6 + 1] = FLEXSPI_LUT_SEQ(CMD_SDR , FLEXSPI_1PAD, 0x00, CMD_SDR, FLEXSPI_1PAD, 0x00),//0x04000400, [4*6 + 2] = FLEXSPI_LUT_SEQ(CMD_SDR , FLEXSPI_1PAD, 0x00, WRITE_SDR, FLEXSPI_1PAD, 0x01),//0x20010400, /*block erase*/ //DCH+23H+32bit address [4*8 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0xDC, CMD_DDR, FLEXSPI_8PAD, 0x23),//0x872387dc, [4*8 + 1] = FLEXSPI_LUT_SEQ(RADDR_DDR, FLEXSPI_8PAD, 0x20, STOP_EXE, FLEXSPI_1PAD, 0x00),//0x00008b20, /*page program*/ //12H+EDH+32bit address+ write data 4bytes [4*9 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x12, CMD_DDR, FLEXSPI_8PAD, 0xED),//0x87ed8712, [4*9 + 1] = FLEXSPI_LUT_SEQ(RADDR_DDR, FLEXSPI_8PAD, 0x20, WRITE_DDR, FLEXSPI_8PAD, 0x04),//0xa3048b20, /* Chip Erase (CE) Sequence *///60H+9FH [4*11 + 0] = FLEXSPI_LUT_SEQ(CMD_DDR , FLEXSPI_8PAD, 0x60, CMD_DDR, FLEXSPI_8PAD, 0x9F),//0x879f8760, }, }, .pageSize = 256u, .sectorSize = 4u * 1024u, .blockSize = 64u * 1024u, .isUniformBlockSize = false, };     The LUT here is the full function LUT table, it includes: 8 wire read, 1/8 wire status read, 1/8 wire write enable, 8 wire sector erase, 1 wire write configuration enable for OPI DDR mode, 8 wire block erase, 8 wire page program, 8 wire chip erase. The detail command is the same as the MX25UM51345GXDI00 datasheet, and also the same as the MCUBootUtility tool generated FCB. 4. CMSIS DAP Flashloader MIMXRT1170-EVK default use the CMSIS DAP debugger, take the MCUXPresso IDE as an example, it will call the flashloader named as .cfx file, flashloader source code can be found from this path: C:\nxp\MCUXpressoIDE_11.6.0_8187\ide\Examples\Flashdrivers\NXP\iMXRT\ iMXRT117x_FlexSPI_SFDP.zip The exist .cfx file which can be used directly, the path is: C:\nxp\MCUXpressoIDE_11.6.0_8187\ide\binaries\Flash\ MIMXRT1170_SFDP_QSPI.cfx:  QSPI flash MIMXRT1170_SFDP_MXIC_OPI.cfx： octal flash Pic 13 When use flashloader + CMSIS DAP + MCUXpresso debug result is: Pic 14 One point need to note, mcuxpresso IDE flashloader source code iMXRT117x_FlexSPI_SFDP.zip, the project configured to the octal flash, the related Flash_RST control code need to be modified, otherwise, when it output high, the voltage will be 3.3V. Pic 15 In the above picture, add code: MEM_WriteU32(0x400E835CU, 0x12); Which is SW_PAD_CTL_PAD_GPIO_AD_03 register = 0x12, OD mode, then generate the mxic MX25UM51345GXDI00 octal flash .cfx file again, which is used for the mcuxpresso+CMSIS DAP debugger. 5. JLINK Flashloader with RT-UFL In actual use, many customers not only use the on-board CMSIS-DAP debugger, but also like to use external JLINK/JLINK plus, or use the on-board JLINK firmware (use LPCScrypt modify firmware, pay attention to update the JINK firmware according to the instructions on the webpage), or use the external JLINK firmware (need to disconnect the onboard debugger jumper). But if the JLINK flash driver is called directly, it will be QSPI flash. Here is how to use the flash driver of the octal flash in JLINK. Now the JLINK driver uses JLinkARM.dll to define the flash used by different chips, unlike the old JLINK driver. The firmware of the flash is called by JLinkDevices.xml. The .dll file cannot allow users to directly modify the corresponding flash of the device, so it is necessary to provide a flash driver file that supports RT1170 octal flash, and add a calling command in JLinkDevices.xml to override the default QSPI definition in JLinkARM.dll. NXP FAE has developed a very useful full-function flash driver called RT-UFL, which can support general QSPI, hyperflash, octaflash, etc. Users can use JLINK to call RT-UFL flash driver, and then use JFLASH, JLINK commander, or IDE (MCUXpresso, IAR, MDK) to realize the debugging and downloading of RT chips combined with different flashes RT-UFL download link: https://github.com/JayHeng/RT-UFL More detail usage of RT-UFL, please check this blog link:   https://www.cnblogs.com/henjay724/ After downloading, install the RT-UFL to the JLINK driver, copy the following folder file: RT-UFL-1.0\algo\SEGGER\JLink_Vxxx to the JLINK driver install path: C:\Program Files\SEGGER\JLINKV768B The JLINK driver link is: https://www.segger.com/downloads/jlink/JLink_Windows_x86_64.exe Now, use the original RT-UFL firmware combined with JFLASH to test the octal flash in RT1170 directly, and the debugger is JLINK plus, just to check whether it can be run, device selection: MIMXRT1170_UFL_L0. _L0 suffix algorithm is suitable for QSPI Flash and Octal Flash (Page size is 256 Bytes, Sector size is 4KB), _L1/2 suffix algorithm is suitable for Hyper Flash (Page size is 512 Bytes, Sector size is 4KB/64KB). 5.1 RT-UFL JFlash Test Generate a .srec file for the led_blinky project which FCB has been modified to octal flash before, it will be used by JFLASH or JLINK commander later. Use JFLASH combined with JLINK plus to create a new JFLASH project. The chip is selected as MIMXRT1170_UFL_L0 which can support octal flash. The test situation is as follows, you can see that the connect can be successful, and the ARM CortexM7 core can be found, but the programming, reading, and erasing functions will fail. It can be said that the connection to the external octal flash is not successful at all: Pic 16 If the used JLINK is not the JLINK plus, due to the license can’t support the JFLASH, then customer can use the JLINK commander to test it, but here, the test result with the original RT-UFL is totally the same as the JFLASH: Pic 17 It can still meet the issues of “Failed to initialize RAMCode”, even use the mem32 readout the address data, sometimes, the data is not correct, not the real flash data, customer can compare the memory which the data which is readout from the MCUBootUtility. So, the RT-UFL flashdriver code need to be modified to the octal flash, to let the octal flash works. 5.2 RT-UFL Flashloader source code modification     From the analysis of the original RT-UFL combined with the octal flash test, after many modifications and tests, the modification points are listed one by one, the new flashloader file generated by the modified RT-UFL is tested using JFlash, JLINK commander, IDE, etc. JLINK tools are divided into external JLINK plus and onboard JLINK firmware (JFLASH is not supported). 5.2.1 Flashloader modification points 1) RAMCode errors In the JLinkDevices.xml file, define the RAM location to the OCRAM address:     <Device> <ChipInfo Vendor="NXP" Name="MIMXRT1170_UFL_octalFlash" WorkRAMAddr="0x20240000" WorkRAMSize="0x00040000" Core="JLINK_CORE_CORTEX_M7" JLinkScriptFile="Devices/NXP/iMXRT_UFL/iMXRT117x_CortexM7.JLinkScript" Aliases="MIMXRT1176xxx8_M7; MIMXRT1176xxxA_M7" /> <FlashBankInfo Name="Octal Flash" BaseAddr="0x30000000" MaxSize="0x01000000" Loader="Devices/NXP/iMXRT_UFL/MIMXRT_FLEXSPI_UV5_UFL.FLM" LoaderType="FLASH_ALGO_TYPE_OPEN" /> </Device>     2) Add erase sector ROM API The JLINK calls the erase sector, so add the related sector erase API: ufl_main.c     static void ufl_fill_flash_api(void) { ... case kChipId_RT117x: uflTargetDesc->flashDriver.init = g_bootloaderTree_imxrt117x->flexspiNorDriver->init; uflTargetDesc->flashDriver.page_program= g_bootloaderTree_imxrt117x->flexspiNorDriver->page_program; uflTargetDesc->isFlashPageProgram = true; uflTargetDesc->flashDriver.erase_all = g_bootloaderTree_imxrt117x->flexspiNorDriver->erase_all; uflTargetDesc->flashDriver.erase = g_bootloaderTree_imxrt117x->flexspiNorDriver->erase; uflTargetDesc->flashDriver.erase_sector= g_bootloaderTree_imxrt117x->flexspiNorDriver->erase_sector;//kerry add uflTargetDesc->flashDriver.read = g_bootloaderTree_imxrt117x->flexspiNorDriver->read; uflTargetDesc->flashDriver.set_clock_source = NULL; uflTargetDesc->flashDriver.get_config= g_bootloaderTree_imxrt117x->flexspiNorDriver->get_config; uflTargetDesc->iarCfg.enablePageSizeOverride = true; break; … }     ufl_flexspi_nor_flash_imxrt117x.h     typedef struct _flexspi_nor_flash_driver_imxrt117x { uint32_t version; status_t (*init)(uint32_t instance, flexspi_nor_config_t *config); status_t (*page_program)(uint32_t instance, flexspi_nor_config_t *config, uint32_t dst_addr, const uint32_t *src); status_t (*erase_all)(uint32_t instance, flexspi_nor_config_t *config); status_t (*erase)(uint32_t instance, flexspi_nor_config_t *config, uint32_t start, uint32_t lengthInBytes); status_t (*read)(uint32_t instance, flexspi_nor_config_t *config, uint32_t *dst, uint32_t addr, uint32_t lengthInBytes); void (*clear_cache)(uint32_t instance); status_t (*xfer)(uint32_t instance, flexspi_xfer_t *xfer); status_t (*update_lut)(uint32_t instance, uint32_t seqIndex, const uint32_t *lutBase, uint32_t seqNumber); status_t (*get_config)(uint32_t instance, flexspi_nor_config_t *config, serial_nor_config_option_t *option); status_t (*erase_sector)(uint32_t instance, flexspi_nor_config_t *config, uint32_t address);//kerry add status_t (*erase_block)(uint32_t instance, flexspi_nor_config_t *config, uint32_t address); const uint32_t reserved0; status_t (*wait_busy)(uint32_t instance, flexspi_nor_config_t *config, bool isParallelMode, uint32_t address); const uint32_t reserved1[2]; } flexspi_nor_flash_driver_imxrt117x_t;     3) Speed up the erase and program speed FlashDev.c     struct FlashDevice const FlashDevice = { FLASH_DRV_VERS, // Driver Version, do not modify! "MIMXRT_FLEXSPI_RT1170", // Device Name EXTSPI, // Device Type 0x30000000, // Device Start Address 0x01000000, // Device Size in Bytes (16mB) 256*4, // Programming Page Size 0, // Reserved, must be 0 0xFF, // Initial Content of Erased Memory 100, // Program Page Timeout 100 mSec 15000, // Erase Sector Timeout 15000 mSec // Specify Size and Address of Sectors 4096*8, 0x00000000, // Sector Size 4kB (256 Sectors) SECTOR_END };       FlashPrg.c：     int EraseSector (unsigned long adr) { uint32_t instance = g_uflTargetDesc.flexspiInstance; uint32_t baseAddr = g_uflTargetDesc.flashBaseAddr; /*Erase Sector*/ status_t status = flexspi_nor_flash_erase(instance, (void *)&flashConfig, adr - baseAddr, FLASH_DRV_SECTOR_SIZE*8);//kerry *8, 4096 if (status != kStatus_Success) { return (1); } else { return (0); } } int ProgramPage (unsigned long adr, unsigned long sz, unsigned char *buf) { status_t status = kStatus_Success; uint32_t instance = g_uflTargetDesc.flexspiInstance; uint32_t baseAddr = g_uflTargetDesc.flashBaseAddr; uint32_t loadaddr = adr - baseAddr; unsigned char *buffer; buffer = buf; int i; for(i = 0; i < 4; i ++) // kerry add 256*4 { status = flexspi_nor_flash_page_program(instance, (void *)&flashConfig, loadaddr, (uint32_t *)buffer); if (status != kStatus_Success) { return (1); } buffer+=256; loadaddr+=256; } return (0); }     The principle is to reduce the times it takes for the PC to send commands to JLINK and then send to the board, now, one command can directly erase and program multiple blocks. 4) Octal Flash option value polling It should be noted that after Flash Reset, or the first programming, it is necessary to read SFDP in a 1-wire manner. If it is not reset and there is a valid FCB in the flash, the flash is initialized to OPT mode and needs to be read in 8-wire mode. At this time, the valid SFDP table of the flash chip cannot be read in the 1-wire mode. RT-UFL is a flashloader that supports multiple chips, and it does not use specific GPIO as the chip flash_RST. For RT-UFL, GPIO is not added to control flash_RST pin, and a large degree of freedom is reserved. Otherwise, once the customer uses the RST pins that are different from EVK still have problems, and even require customers to spend time modifying the flashloader source code, increasing development time. Therefore, in view of the above considerations, modify the code here, do not add RESET signal control, do not fix option value for 1-wire or 8-wires, but polling the option with 1-wire and 8-wires to read SPDF. ufl_main.c     static void ufl_set_target_property(void) case kChipId_RT117x: uflTargetDesc->flexspiInstance = MIMXRT117X_1st_FLEXSPI_INSTANCE; uflTargetDesc->flexspiBaseAddr = MIMXRT117X_1st_FLEXSPI_BASE; uflTargetDesc->flashBaseAddr = MIMXRT117X_1st_FLEXSPI_AMBA_BASE; // uflTargetDesc->configOption.option0.U = 0xc0433007; // uflTargetDesc->configOption.option1.U = 0x0; break;     Make sure that the above option are not configured, otherwise the option configuration will be fixed and the polling option function will no longer be enabled. ufl_auto_probe_flash.c     static const serial_nor_config_option_t s_flashConfigOpt[] = { // 1st Pinmux, PortA for octal 1 bit SFDP for no FCB in flash eg. MX25UM51345G {.option0.U = 0xc0403007, .option1.U = 0x00000000}, // 1st Pinmux, PortA for octal 8 bit SFDP for FCB in flash eg. MX25UM51345G {.option0.U = 0xc0433007, .option1.U = 0x00000000}, // 1st Pinmux, PortA for octal 1 bit SFDP &1 bit CMD for no FCB in flash eg. MX25UM51345G {.option0.U = 0xc0400007, .option1.U = 0x00000000}, ..}     After the above modifications, compile the source code project of RT-UFL: RT-UFL-1.0\build\mdk\MIMXRT_FLEXSPI_UV5.uvprojx, Generate a new MIMXRT_FLEXSPI_UV5_UFL.FLM programming algorithm and copy it to the following path: C:\Program Files\SEGGER\JLINKV768B\Devices\NXP\iMXRT_UFL The device name defined in JLinkDevices.xml is: MIMXRT1170_UFL_octalFlash. JLinkDevices.xml path: C:\Program Files\SEGGER\JLINKV768B modify the .xml, and add the above-mentioned "RAMCode error" section. After the modification is completed, refresh the JLinkDLLUpdater.exe file, so that several IDEs can synchronize the firmware defined by the updated xml file. 5.2.2 JFlash Test As can be seen from the test results of Jflash below, it can be successfully erased, and the app image can be downloaded to octal flash. After the download is successful, reset it, and you can see the onboard led is flashing. It shows that the modified RT-UFL flashloader has worked. It should be noted here that the use of JFLASH requires the use of an external Segger JLINK Plus. Some customers use normal JLINK or RT EVK onboard JLINK firmware, then they can try the JLINK commander method. Pic 18 5.2.3 JLINK command Test Use the JLINK plus associated with JLINK commander test result is: Pic 19 You can see that the firmware can be downloaded successfully. Now uses the EVK onboard JLINK firmware to test as follows, it can be seen that the programming is normal, so if users don’t have external JLINK can also directly use the RT EVK onboard JLINK firmware. One thing need to pay attention to is when using the onboard JLINK firmware, the firmware will cause the debugger port USB to not directly supply power to the RT chip, so it needs to be configured that J38 is connected to 3-4, and then find another USB cable to connect J20 to supply power to the board. Pic 20 5.3 MCUXpresso + JLINK +Modified RT-UFL Testing At first, select the MCUXPresso IDE JLINK debug path, add the JLINK driver path which is already added RT-UFL firmware, window->preferences, then the path should like this: Pic 21 When downloading, still need to configure RUN->Debugger configurations, the JLINK configuration should be: Pic 22 Please note, don’t check the item “Reset before running” in the debug configuration, as this item is checked in default. Pic 23 If checked “Reset before running”, when debug it, the code will be blocked after downloading the code, and can’t jump to the main function. If customer click halt button, it will meet the issues that the code is stop at 0x223104, but after exit the debug mode, and do the POR again, you will find the flash already downloaded with the app successfully. Pic 24 This problem is related to the impact of RT1170 security policy on JLINK. For details, please refer to the following link: https://www.cnblogs.com/henjay724/p/15725966.html When JLink is connected to the chip, as long as the reset command is executed, it will directly enter the safe debugging mode (the PC stops at 0x223104). Therefore, make sure that "Reset before running" is not checked, so that you can enter the debug and main functions normally, and the test results are as follows: Pic 25 5.4  MDK+JLINK+Modified RT-UFLTesting Next, use the MCUXPresso CFG tool combined with the SDK package to export an MDK project, also based on the led_blinky project, and modify the FCB code for Octal flash, and then call the modified RT-UFL flashloader, the device name is: MIMXRT1170_UFL_octalFlash. Note, be sure to refresh: C:\Program Files\SEGGER\JLINKV768B\JLinkDLLUpdater.exe Make sure the IDE is using the latest firmware link. The following is a specific MDK project related configuration pictures: Pic 26 The project is selected as flexspi_nor_debug, and after compiling, the debug tool is configured as JLINK, the JLINK simulation sequence can be found, and the updated target before debugging in utilities should not be checked: Pic 27 Pic 28 Pic 29 Modify JlinkSettings.ini，configure override =1, device as the RT-UFL modified firmware device name: MIMXRT1170_UFL_octalFlash. Pic 30 Pic 31 As can be seen from the above picture, the modified RT-UFL Octal flash has been successfully used to run the MDK project. It should be noted that do not use the download button, because this button cannot use the overwritten and modified RT-UFL programming algorithm, it will still call the programming algorithm of the deleted area of ​​the interface, and an error will be reported if it cannot be found. 5.5 IAR + JLINK + Modified RT-UFL Testing Here is the use of IAR combined with JLINK to debug RT1170 octal flash, using the modified RT-UFL programming algorithm. The specific configuration is as follows: Pic 32 Don’t check “use flash loaders”, it means don’t use the IAR attached .out download algorithm, and use the JLINK driver’s modified RT-UFL octal flash flashdriver. Pic 33 Reset can choose core, if it is normal, debug will meet the same issues with the MCUXPresso which checked reset item, PC will stop to 0x223104. Pic 34 Here, modify project settings folder ->.jlink file, use the RT-UFL device name: MIMXRT1170_UFL_octalFlash， And override =1, then it will call the modified RT-UFL flashloader. If you can’t find the .jlink in the new created project, just select the JLINK debugger, click download at first, then it will generate the .jlink file, then modify it and debug it again. Pic 35 After modify the RT-UFL program algorithm to the octal flash, click “download and debug” button, it will enter the debug mode, in the following picture, the code is running successfully: Pic 36 6. Conclusion After the above details, the code download of the MIMXRT1170-EVK on-board octalflash can be realized, and the fsl_romspi project is used to run in RAM, which can verify the normal reading and writing of the hardware octalflash, and MCUBootutility to verify the programming and boot conditions. The modification of APP FCB is matched octal flash. The debugger uses JLINK or CMSIS DAP, and the correct flashdriver programming algorithm needs to be used. In particular, JLINK needs to use the RT-UFL as flashloader, but the source code needs to be modified. Finally, you can see that the combination of RT-UFL download algorithm and JLINK, successfully realize the download and operation of octal flash code on MCUXpresso, IAR, MDK three IDEs. This article is not limited to MIMXRT1170-EVK, but also applies to customer design boards using RT1170+ MX25UM51345GXDI00 octal flash. The attachment adds the modified RT-UFL programming algorithm and the octal flash project of the three major IDEs.  

Background: The CAAM manufacturing protection feature provides a mechanism to authenticate the chip to the OEM's server. The manufacturing protection feature can be used to ensure that the chip:  Is a genuine NXP SoC  Is the correct device type and part number  Has been properly configured by means of fuses  Is running authenticated OEM software  Is currently in the secure or trusted mode The CAAM manufacturing protection feature is based on an ECC private key generated by the High Assurance Boot (HAB) code on every boot cycle. The Manufacturing Protection (MP) private key generation takes as input several fixed secrets and the MANUFACTURE_PROTECTION_KEY[255:0] being one of them in SoC fuses.   Issue Description: On certain i.MX RT117x and RT116x devices the MANUFACTURE_PROTECTION_KEY[255:0] fuses were incorrectly programmed at the NXP factory. During the MP private key generation, the CAAM block validates the inputs provided and fails as the MANUFACTURE_PROTECTION_KEY[255:0] provided is not a valid one. As the MPPubK-generation and MPSign CAAM functions depends on the result of MPPrivK-generation function the CAAM manufacturing protection feature cannot be used on the impacted devices. Details regarding manufacturing protection functions can be found in the section "Manufacturing-protection chip-authentication process" in the security reference manuals (SRM).  Please note that in closed mode the CAAM MPPrivK-generation function can be only executed once in the same power-on session. Running a second time returns a CAAM error (0x40000481) undefined protocol command which is not related to the issue described in this document.   Checking if your device is impacted: Customers can check if their device is impacted by following the 3 steps below: Checking the date code: Devices from datecodes prior to 2213 are impacted. Checking HAB events: The HAB code logs a warning event in the HAB persistent memory region after detecting a failure in the MP private key generation. This warning is logged independently regardless of whether HAB is enabled (SEC_CONFIG =1) or not. Customers can parse the HAB persistent memory region at 0x20242000 in order to get the warning events.  Impacted devices should report the event below: Event    | 0xdb | 0x0024 | 0x45 |  SRCE Field: 69 30 e1 1d             |         |             |         |             STS = HAB_WARNING (0x69)             |         |             |         |             RSN = HAB_ENG_FAIL (0x30)             |         |             |         |            CTX = HAB_CTX_ENTRY (0xE1)             |         |             |         |            ENG = HAB_ENG_CAAM (0x1d)             |         |             |         |  Evt Data (hex):             |         |             |         |   00 01 00 02 40 00 04 cc 00 00 00 0f 00 00 00 00             |         |             |         |   00 00 00 00 00 00 00 00 00 00 00 01 3. Checking the CAAM SCFGR register: After running the MPPrivK-generation function the CAAM block stores in the CAAM SCFGR register the elliptic curve that was selected when the MPPrivK generation protocol was executed. Users can check the MPCURVE field [31:28] in the CAAM SCFGR register and on impacted devices this field will be 0.    List of impacted devices:  All i.MX RT117x and RT116x devices prior to 2213 datecode are impacted.   Workaround: No Software Workaround can be implemented. Customers planning to use the Manufacturing Protection feature should request for SoC's that have the correct fuse programming. Please Note: This issue does not impact the Secure Boot flow and does not compromise security.

Issue: 802.11 IEEE station Power Save mode is not working as expected with the latest SDK 2.11.1, supporting NXP wireless solutions 88W8987/88W8977/IW416.   Solution: Modify the structure in file : middleware/wifi/wifidriver/incl/mlan_fw.h, Replace  “ENH_PS_MODES action” to “uint16_t action”.    Note: This fix will officially be part of SDK: 2.12.0

A vulnerability (CVE-2022-22819) has been identified on select NXP processors by which a malformed SB2 file header sent to the device as part of an update or recovery boot can be used to create a buffer overflow. The buffer overflow can then be used to launch various exploits. Refer to the attached bulletin for more information.   09/26/2022 - Bulletin updated to include fix datecode information. 11/01/2022 - Bulletin updated with clarification that mixed datecodes are RT600 only.    

RT106L_S voice control system based on the Baidu cloud 1 Introduction     The NXP RT106L and RT106S are voice recognition chip which is used for offline local voice control, SLN-LOCAL-IOT is based on RT106L, SLN-LOCAL2-IOT is a new local speech recognition board based on RT106S. The board includes the murata 1DX wifi/BLE module, the AFE voice analog front end, the ASR recognition system, the external flash, 2 microphones, and the analog voice amplifier and speakers. The voice recognition process for SLN-LOCAL-IOT and SLN-LOCAL2-IOT is different and the new SLN-LOCAL2-IOT is recommended.     This article is based on the voice control board SLN-LOCAL/2-IOT to implement the following block diagram functions: Pic 1 Use the PC-side speed model tool (Cyberon DSMT) to generate WW(wake word) and VC(voice command) Command related voice engine binary files , which will be used by the demo code. This system is mainly used for the Chinese word recognition, when the user says Chinese word: "小恩小恩", it wakes up SLN-LOCAL/2-IOT, and the board gives feedback "小恩来了，请吩咐". Then system enter the voice recognition stage, the user can say the voice recognition command: “开红灯”，“关红灯”，“开绿灯”，“关绿灯”，“灯闪烁”，“开远程灯”，“关远程灯”， after recognition, the board gives feedback "好的". Among them, “开红灯”，“关红灯”，“开绿灯”，“关绿灯”，“灯闪烁”，the five commands are used for the local light switch, while the 开远程灯”，“关远程灯“two commands can through network communication Baidu cloud control the additional MIMXRT1060-EVK development board light switch. SLN-LOCAL/2-IOT through the WIFI module access to the Internet with MQTT protocol to achieve communication with Baidu cloud, when dectect the remote control command, publish the json packets to Baidu cloud, while MIMRT1060-EVK subscribe Baidu cloud data, will receive data from the IOT board and analyze the EVK board led control. PC side can use MQTT.fx software to subscribe the Baidu cloud data, it also can send data to the device to achieve remote control function directly.  Now, will give the detail content about how to use the SLN-LOCAL/2-IOT SDK demo realize the customized Chinese wake command and voice command, and remote control the MIMXRT1060-EVK through the Baidu Cloud.     2 Platform establish 2.1 Used platform SLN-LOCAL-IOT/SLN-LOCAL2-IOT MIMXRT1060-EVK MQTT.fx SDK_2_8_0_SLN-LOCAL2-IOT MCUXPresso IDE Segger JLINK Baidu Smart Cloud: Baidu cloud control+ TTS Audacity:audio file format convert tool WAVToCode：wav convert to the c array code, which used for the demo tilte play MCUBootUtility: used to burn the feedback audio file to the filesystem Cyberon DSMT: wake word and voice detect command generation tool DSMT is the very important tool to realize the wake word and voice dection, the apply follow is: Pic 2 2.2 Baidu Smart cloud 2.2.1 Baidu cloud IOT control system Enter the IoT Hub: https://cloud.baidu.com/product/iot.html     Click used now. 2.2.1.1 Create device project Create a project, select the device type, and enter the project name. Device types can use shadows as images of devices in the cloud to see directly how data is changing. Once created, an endpoint is generated, along with the corresponding address: Pic 3 2.2.1.2 Create Thing model The Thing model is mainly to establish various properties needed in the shadow, such as temperature, humidity, other variables, and the type of value given, in fact, it is also the json item in the actual MQTT communication.    Click the newly created device-type project where you can create a new thing model or shadow: Pic 4    Here create 3 attributes：LEDstatus，humid，temp It is used to represent the led status, humidity, temperature and so on, which is convenient for communication and control between the cloud and RT board. Once created, you get the following picture:   Pic 5   2.2.1.3 Create Thing shadow In the device-type project, you can select the shadow, build your own shadow platform, enter the name, and select the object model as the newly created Thing model containing three properties, after the create, we can get the details of the shadow:   Pic 6 At the same time will also generate the shadow-related address, names and keys, my test platform situation is as follows: TCP Address: tcp://rndrjc9.mqtt.iot.gz.baidubce.com:1883 SSL Address: ssl://rndrjc9.mqtt.iot.gz.baidubce.com:1884 WSS Address: wss://rndrjc9.mqtt.iot.gz.baidubce.com:443 name: rndrjc9/RT1060BTCDShadow key: y92ewvgjz23nzhgn Port 1883, does not support transmission data encryption Port 1884, supports SSL/TLS encrypted transmission Port 8884, which supports wesockets-style connections, also contains SSL encryption. This article uses a 1883 port with no transmission data encryption for easy testing. So far, Baidu cloud device-type cloud shadow has been completed, the following can use MQTTfx tools to connect and test. In practice, it is recommended that customers build their own Baidu cloud connection, the above user key is for reference only.   2.2.2 Online TTS    SLN-LOCAL/2-IOT board recognizes wake-up words, recognition words, or when powering on, you need to add corresponding demo audio, such as: "百度云端语音测试demo ", "小恩来啦！请吩咐“，"好的". These words need to do a text-to-wav audio file synthesis, here is Baidu Smart Cloud's online TTS function, the specific operation can refer to the following documents: https://ai.baidu.com/ai-doc/SPEECH/jk38y8gno   Once the base audio library is opened, use the main.py provided in the link above and modify it to add the Chinese field you want to convert to the file "TEXT" and add the audio file to be converted in "save_file" such as xxx .wav, using the command: python main.py to complete the conversion, and generate the audio format corresponding to the text, such as .mp3, .wav. Pic 7   After getting the wav file, it can’t be used directly, we need to note that for SLN-LOCAL/2-IOT board, you need to identify the audio source of the 48K sample rate with 16bit, so we need to use the Audioacity Audio tool to convert the audio file format to 48K16bit wav. Import 16K16bit wav files generated by Baidu TTS into the Audioacity tool, select project rate of 48Khz, file->export->export as WAV, select encoding as signed 16bit PCM, and regenerate 48Khz16bit wav for use. Pic 8 “百度云端语音测试demo“：Used for power-on broadcasting, demo name broadcasting, it is stored in RT demo code, so you need to convert it to a 16bit C code array and add it to the project. "小恩来啦！请吩咐"，“好的“：voice detect feedback, it is saved in the filesystem ZH01,ZH02 area. 2.3 playback audio data prepare and burn   There are two playback audio file, it is "小恩来啦！请吩咐"，“好的“，it is saved in the filesystem ZH01,ZH02 area. Filesystem memory map like this: Pic 9 So, we need to convert the 48K16bit wav file to the filesystem needed format, we need to use the official tool:：Ivaldi_sln_local2_iot Reference document：SLN-LOCAL2-IOT-DG chapter 10.1 Generating filesystem-compatible files Use bash input the commands like the following picture: Pic10 Use the convert command to get the playback bin file: python file_format.py -if xiaoencoming_48k16bit.wav -of xiaoencoming_48k16bit.bin -ft H At last, it will generate the file: "小恩来啦！请吩咐"->xiaoencoming_48k16bit.bin,burn to flash address 0x6184_0000 “好的”->OK_48k16bit.bin, burn to flash address 0x6180_0000 Then, use MCUBootUtility tool burn the above two file to the related images. Here, take OK_48k16bit.bin as an example, demo enter the serial download mode(J27-0), power off and power on. Flash chip need to select hyper flash IS26KSXXS, use the boot device memory windows, write button to burn the .bin file to the related address, length is 0X40000 Pic11 Pic12 xiaoencoming_48k16bit.bin can use the same method to download to 0x6184_0000，Length is 0X40000.   2.4 Demo audio prepare and add The prepared baiduclouddemo_48K16bit.wav（“百度云端语音测试demo “） need to convert to the 16bit C array code, and put to the project code, calls by the code, this is used for the demo mode play. The convert need to use the WAVToCode, the operation like this: Pic 13 The generated baiducloulddemo_48K16bit.c，add it to the demo project C files: sln_local_iot_local_demo->audio->demos->smart_home.c。 2.5 WW and VC prepare Wake-up word are generated through the cyberon DSMT tool, which supports a wide range of language, customers can request the tool through Figure 2. The Chinese wake-up words and voice command words in this article are also generated through DSMT. DSMT can have multiple groups, group1 as a wake-up word configuration, CmdMapID s 1. Other groups act as voice command words, such as CMD-IOT in this article, cmdMapID=2. Pic 14   Pic 15 Wake word continuously detects the input audio stream, uses group1, and if successfully wakes up, will do the voice command detection uses group2, or other identifying groups as well as custom groups. The wake-up words using the DSMT tool, the configuration are as follows: Pic 16 The WW can support more words, customer can add the needed one in the group 1. Use the DSMT configure VC like this: Pic 17 Then, save the file, code used file are: _witMapID.bin, CMD_IOT.xml,WW.xml. In the generated files, CYBase.mod is the base model, WW.mod is the WW model, CMD_IOT.mod is the VC model. After Pic 16，17， it finishes the WW and VC command prepare, we can put the DSMT project to the RT106S demo project folder: sln_local2_iot_local_demo\local_voice\oob_demo_zh 3 Code prepare Based on the official SLN-LOCAL2-IOT SDK local_demo, the code in this article modifies the Chinese wake-up words and recognition words (or you can build a new customer custom group directly), add local voice detect the led status operations, Then feedback Chinese audio, demo Chinese audio, Wifi network communication MQTT protocol code, and Baidu cloud shadow connection publish. Source reference code SDK path: SDK_2_8_0_SLN-LOCAL2-IOT\boards\sln_local2_iot\sln_voice_examples\local_demo   SDK_2_8_0_SLN-LOCAL2-IOT\boards\sln_local2_iot\sln_boot_apps SLN-LOCAL2-IOT and SLN-LOCAL-IOT code are nearly the same, the only difference is that the ASR library file is different, for RT106S (SLN-LOCAL2-IOT) using SDK it’s own libsln_asr.a library, for RT106L (SLN-LOCAL-IOT) need to use the corresponding libsln_asr_eval.a library.    Importing code requires three projects: local_demo, bootloader, bootstrap. The three projects store in different spaces. See SLN-LOCAL2-IOT-DG .pdf, chapter 3.3 Device memory map    This is the 3 chip project boot process: Pic 18 This document is for demo testing and requires debug, so this article turns off the encryption mechanism, configures bootloader, bootstrap engineering macro definition: DISABLE_IMAGE_VERIFICATION = 1, and uses JLINK to connect SLN-LOCAL/2-IOT's SWD interface to burn code. The following is to add modification code for app local_demo projects. 3.1 sln-local/2-iot code Sln-local-iot, sln-local2-iot platform, the following modification are the same for the two platform. 3.1.1 Voice recognition related code 1）Demo audio play Play content:“百度云端语音测试demo“ sln_local2_iot_local_demo_xe_ledwifi\audio\demos\ smart_home.c content is replaced by the previously generated baiducloulddemo_48K16bit.C. audio_samples.h，modify： #define SMART_HOME_DEMO_CLIP_SIZE 110733 This code is used for the main.c announce_demo API play:         case ASR_CMD_IOT:             ret = demo_play_clip((uint8_t *)smart_home_demo_clip, sizeof(smart_home_demo_clip));   2）command print information #define NUMBER_OF_IOT_CMDS      7 IndexCommands.h static char *cmd_iot_en[] = {"Red led on", "Red led off", "Green led on", "Green led off",                              "cycle led",        "remote led on",         "remote led off"}; static char *cmd_iot_zh[] = {"开红灯", "关红灯", "开绿灯", "关绿灯", "灯闪烁", "开远程灯", "关远程灯"}; Here is the source code modification using IOT, you can actually add your own speech recognition group directly, and add the relevant command identification.   3）sln_local_voice.c Line757 , add led-related notification information in ASR_CMD_IOT mode. oob_demo_control.ledCmd = g_asrControl.result.keywordID[1];     The code is used to obtain the recognized VC command data, and the value of keywordID[1] represents the number. This number can let the code know which detail voice is detected. so that you can do specific things in the app based on the value of ledcmd. The value of keywordID[1] corresponds to Command List in Figure 17. For example, “开远程灯“, if woke up, and recognized "开远程灯", then keywordID[1] is 5, and will transfer to oob_demo_control.ledCmd, which will be used in the appTask API to realize the detail control. 4) main.c void appTask(void *arg) Under case kCommandGeneric: if the language is Chinese, then add the recognition related control code, at first, it will play the feedback as “好的”. Then, it will check the voice detect value, give the related local led control. else if (oob_demo_control.language == ASR_CHINESE) { // play audio "OK" in Chinese #if defined(SLN_LOCAL2_RD) ret = audio_play_clip((uint8_t *)AUDIO_ZH_01_FILE_ADDR, AUDIO_ZH_01_FILE_SIZE); #elif defined(SLN_LOCAL2_IOT) ret = audio_play_clip(AUDIO_ZH_01_FILE); #endif //kerry add operation code==================================================begin RGB_LED_SetColor(LED_COLOR_OFF); if (oob_demo_control.ledCmd == LED_RED_ON) { RGB_LED_SetColor(LED_COLOR_RED); vTaskDelay(5000); } else if (oob_demo_control.ledCmd == LED_RED_OFF) { RGB_LED_SetColor(LED_COLOR_OFF); vTaskDelay(5000); } else if (oob_demo_control.ledCmd == LED_BLUE_ON) { RGB_LED_SetColor(LED_COLOR_BLUE); vTaskDelay(5000); } else if (oob_demo_control.ledCmd == LED_BLUE_OFF) { RGB_LED_SetColor(LED_COLOR_OFF); vTaskDelay(5000); } else if (oob_demo_control.ledCmd == CYCLE_SLOW) { for (int i = 0; i < 3; i++) { RGB_LED_SetColor(LED_COLOR_RED); vTaskDelay(400); RGB_LED_SetColor(LED_COLOR_OFF); RGB_LED_SetColor(LED_COLOR_GREEN); vTaskDelay(400); RGB_LED_SetColor(LED_COLOR_OFF); RGB_LED_SetColor(LED_COLOR_BLUE); vTaskDelay(400); } } … } In addition to local voice recognition control, this article also add remote control functions, mainly through wifi connection, use the mqtt protocol to connect Baidu cloud server, when local speech recognition get the remote control command, it publish the corresponding control message to Baidu cloud, and then the cloud send the message to the client which subscribe this message,  after the client get the message, it will refer to the message content do the related control.   3.1.3 Network connection code 1）sln_local2_iot_local_demo_xe_ledwifi\lwip\src\apps\mqtt     Add mqtt.c 2）sln_local2_iot_local_demo_xe_ledwifi\lwip\src\include\lwip\apps Add mqtt.h, mqtt_opts.h,mqtt_prv.h The related mqtt driver is from the RT1060 sdk, which already added in the attachment project. 3）sln_tcp_server.c   Add MQTT application layer API function code, client ID, server host, MQTT server port number, user name, password, subscription topic, publishing topic and data, etc., more details, check the attachment code.    The MQTT application code is ported from the mqtt project of the RT1060 SDK and added to the sln_tcp_server.c. TCP_OTA_Server function is used to initialize the wifi network, realize wifi connection, connect to the network, resolve Baidu cloud server URL to get IP, and then connect Baidu cloud server through mqtt, after the successful connection, publish the message at first, so that after power-up through mqttfx to see whether the power on network publishing message is successful. TCP_OTA_Server function code is as follows: static void TCP_OTA_Server(void *param) //kerry consider add mqtt related code { err_t err = ERR_OK; uint8_t status = kCommon_Failed; #if USE_WIFI_CONNECTION /* Start the WiFi and connect to the network */ APP_NETWORK_Init(); while (status != kCommon_Success) { status_t statusConnect; statusConnect = APP_NETWORK_Wifi_Connect(true, true); if (WIFI_CONNECT_SUCCESS == statusConnect) { status = kCommon_Success; } else if (WIFI_CONNECT_NO_CRED == statusConnect) { APP_NETWORK_Uninit(); /* If there are no credential in flash delete the TPC server task */ vTaskDelete(NULL); } else { status = kCommon_Failed; } } #endif #if USE_ETHERNET_CONNECTION APP_NETWORK_Init(true); #endif /* Wait for wifi/eth to connect */ while (0 == get_connect_state()) { /* Give time to the network task to connect */ vTaskDelay(1000); } configPRINTF(("TCP server start\r\n")); configPRINTF(("MQTT connection start\r\n")); mqtt_client = mqtt_client_new(); if (mqtt_client == NULL) { configPRINTF(("mqtt_client_new() failed.\r\n");) while (1) { } } if (ipaddr_aton(EXAMPLE_MQTT_SERVER_HOST, &mqtt_addr) && IP_IS_V4(&mqtt_addr)) { /* Already an IP address */ err = ERR_OK; } else { /* Resolve MQTT broker's host name to an IP address */ configPRINTF(("Resolving \"%s\"...\r\n", EXAMPLE_MQTT_SERVER_HOST)); err = netconn_gethostbyname(EXAMPLE_MQTT_SERVER_HOST, &mqtt_addr); configPRINTF(("Resolving status: %d.\r\n", err)); } if (err == ERR_OK) { configPRINTF(("connect to mqtt\r\n")); /* Start connecting to MQTT broker from tcpip_thread */ err = tcpip_callback(connect_to_mqtt, NULL); configPRINTF(("connect status: %d.\r\n", err)); if (err != ERR_OK) { configPRINTF(("Failed to invoke broker connection on the tcpip_thread: %d.\r\n", err)); } } else { configPRINTF(("Failed to obtain IP address: %d.\r\n", err)); } int i=0; /* Publish some messages */ for (i = 0; i < 5;) { configPRINTF(("connect status enter: %d.\r\n", connected)); if (connected) { err = tcpip_callback(publish_message_start, NULL); if (err != ERR_OK) { configPRINTF(("Failed to invoke publishing of a message on the tcpip_thread: %d.\r\n", err)); } i++; } sys_msleep(1000U); } vTaskDelete(NULL); } Please note the following published json data, it can’t be publish directly in the code. {   "reported": {     "LEDstatus": false,     "humid": 88,     "temp": 22   } } Which need to use this web https://www.bejson.com/ realize the json data compression and convert: {\"reported\" : {     \"LEDstatus\" : true,     \"humid\" : 88,     \"temp\" : 11    } }   4）main appTask Under case kCommandGeneric: ， if the language is Chinese, then add the corresponding voice recognition control code. "开远程灯": turn on the local yellow light, publish the “remote led on” mqtt message to Baidu cloud, control remote 1060EVK board lights on. "关远程灯": turn on the local white light, publish the “remote led off” mqtt message to Baidu cloud, control the remote 1060EVK board light off. Related operation code: else if (oob_demo_control.ledCmd == LED_REMOTE_ON) { RGB_LED_SetColor(LED_COLOR_YELLOW); vTaskDelay(5000); err_t err = ERR_OK; err = tcpip_callback(publish_message_on, NULL); if (err != ERR_OK) { configPRINTF(("Failed to invoke publishing of a message on the tcpip_thread: %d.\r\n", err)); } } else if (oob_demo_control.ledCmd == LED_REMOTE_OFF) { RGB_LED_SetColor(LED_COLOR_WHITE); vTaskDelay(5000); err_t err = ERR_OK; err = tcpip_callback(publish_message_off, NULL); if (err != ERR_OK) { configPRINTF(("Failed to invoke publishing of a message on the tcpip_thread: %d.\r\n", err)); } } 3.2 MIMXRT1060-EVK code The main function of the MIMXRT1060-EVK code is to configure another client in the cloud, subscribe to the message published by SLN-LOCAL/2-IOT which detect the remote command, and then the LED on the control board is used to test the voice recognition remote control function, this code is based on Ethernet, through the Ethernet port on the board, to achieve network communication, and then use mqtt to connect baidu cloud, and subscribe the message from local2, This enables the reception and execution of the Local2 command. the network code part is similar to SLN-LOCAL2-IOT board network code, the servers, cloud account passwords, etc. are all the same, the main function is to subscribe messages. See the code from attachment RT1060, lwip_mqtt_freertos.c file. When receives data published by the server, it needs to do a data analysis to get the status of the led light and then control it. Normal data from Baidu cloud shadow sent as follows Received 253 bytes from the topic "$baidu/iot/shadow/RT1060BTCDShadow/update/accepted": "{"requestId":"2fc0ca29-63c0-4200-843f-e279e0f019d3","reported":{"LEDstatus":false,"humid":44,"temp":33},"desired":{},"lastUpdatedTime":{"reported":{"LEDstatus":1635240225296,"humid":1635240225296,"temp":1635240225296},"desired":{}},"profileVersion":159}" Then you need to parse the data of LEDstatus from the received data, whether it is false or true. Because the amount of data is small, there is no json-driven parsing here, just pure data parsing, adding the following parsing code to the mqtt_incoming_data_cb function: mqtt_rec_data.mqttindex = mqtt_rec_data.mqttindex + len; if(mqtt_rec_data.mqttindex >= 250) { PRINTF("kerry test \r\n"); PRINTF("idex= %d", mqtt_rec_data.mqttindex); datap = strstr((char*)mqtt_rec_data.mqttrecdata,"LEDstatus"); if(datap != NULL) { if(!strncmp(datap+11,strtrue,4))//char strtrue[]="true"; { GPIO_PinWrite(GPIO1, 3, 1U); //pull high PRINTF("\r\ntrue"); } else if(!strncmp(datap+11,strfalse,5))//char strfalse[]="false"; { GPIO_PinWrite(GPIO1, 3, 0U); //pull low PRINTF("\r\nfalse"); } } mqtt_rec_data.mqttindex =0; It use the strstr search the “LEDstatus“ in the received data, and get the pointer position, then add the fixed length to get the LED status is true or flash. If it is true, turn on the led, if it is false, turn off the led. 4 Test Result    This section gives the test results and video of the system. Before testing the voice function, first use MQTTfx to test baidu cloud connection, release, subscription is no problem, and then test sln-local2-iot combined with mimxrt1060-evk voice wake-up recognition and remote control functions.    For SLN-LOCAL2-IOT wifi hotspot join, enter the command in the print terminal: setup AWS kerry123456   4.1 MQTT.fx test baidu cloud connection MQTT.fx is an EclipsePaho-based MQTT client tool written in the Java language that supports subscription and publishing of messages through Topic.    4.1.1 MQTT fx configuration     Download and install the tool, then open it, at first, need to do the configuration, click edit connection： Pic19 Profile name：connect name Profile type: MQTT broker Broker address: It is the baidu could generated broker address, with 1883 no encryption transfer. Broker port:1883 No encryption Client ID: RT1060BTCDShadow, here need to note, this name should be the same as the could shadow name, otherwise, on the baidu webpage, the connection is not be detected. If this Client ID name is the same as the shadow name, then when the MQTT fx connect, the online side also can see the connection is OK. User credentials: add the thing User name and password from the baidu cloud. After the configuration, click connect, and refresh the website. Before conection： Pic 20 After connection： Pic 21 4.1.2 MQTT fx subscribe When it comes to subscription publishing, what is the topic of publishing subscriptions?  Here you can open your thing shadow, select the interaction, and see that the page has given the corresponding topic situation: Pic 22 Subscribe topic is： $baidu/iot/shadow/RT1060BTCDShadow/update/accepted  Publish topic is： $baidu/iot/shadow/RT1060BTCDShadow/update Pic 23 Click subscribe, we can see it already can used to receive the data.   4.1.3 MQTT fx publish Publish need to input the topic： $baidu/iot/shadow/RT1060BTCDShadow/update It also need to input the content, it will use the json content data. Pic 24 Here, we can use this json data: {   "reported" : {     "LEDstatus" : true,     "humid" : 88,     "temp" : 11    } } The json data also can use the website to check the data: https://www.bejson.com/jsonviewernew/ Pic 25 Input the publish data, and click pubish button: Pic 26 4.1.4 Publish data test result   Before publish, clean the website thing data： Pic 27 MQTT fx publish data, then check the subscribe data and the website situation: Pic 28 We can see, the published data also can be see in the website and the mqttfx subscribe area. Until now, the connection, data transfer test is OK.   4.2 Voice recognition and remote control test This is the device connection picture: Pic 29 4.2.1 voice recognition local control Pic 30 This is the SLN-LOCAL2-IOT print information after recognize the voice WW and VC. Red led on: led cycle: 4.2.2 voice recognition remote control   Following test, wakeup + remote on, wakeup+remote off, and also give the print result and the video. Pic 31 remote control:  

Face recognition Actually, face recognition technology is used in many scenes in our daily life, for instance, when taking pictures with the mobile phone, the camera software will automatically recognize the faces in the lens and focus, scan face for real-name verification when registering the App and scan face for pay, etc. The basic steps of face recognition are shown in the below figure. Firstly, the camera captures image data, then through preprocessing such as noise elimination and image format conversion, the image data will be transmitted to the processor for face detection and recognition calculations. After recognizing the face successful, continue to do the follow-up operations. Fig1 The basic steps of face recognition i.MX RT106F MCU based solution for face recognition The below figure is the block diagram of i.MX RT106F MCU-based solution for face recognition provided by the NXP. Comparing with the general processor (CPU) solution, it has comparative advantages in cost and power consumption. Further, the PCB size will be smaller too and the MCU usually can boot up within a few hundred milliseconds even with RTOS, versus to the boot-up speed of the processor (CPU) equipped with a Linux system that is about 10 seconds, it will give customers a better user experience. Fig2 i.MX RT106F MCU based solution for face recognition Of course, the i.MX RT106F MCU-based solution face recognition solution is not intended to replace the solution based on the processor (CPU). As aforementioned, face recognition technology has a lot of application cases, and it will definitely be used in more fields in the future, so the MCU-based face recognition solution provides customers and the market with another choice. i.MX RT106F MCU The i.MX RT106F face recognition crossover processor is an EdgeReady™ solution-specific variant of the i.MX RT1060 family of crossover processors, targeting face recognition applications. It features NXP’s advanced implementation of the Arm Cortex®-M7 core, which operates at speeds up to 600 MHz to provide high CPU performance and the best real-time response. i.MX RT106F based solutions enable system designers to easily and inexpensively add face recognition capabilities to a wide variety of smart appliances, smart homes, smart retail, and smart industrial devices. The i.MX RT106F is licensed to run the OASIS Lite library for face recognition (as the below figure shows) which include: Face detection Anti-spoofing Face tracking Face alignment Glass detection Face recognition Confidence measure Face recognition quantified results, etc Fig3 OASIS Recognition Software Pipeline sln_viznas_iot_elock_oobe The sln_viznas_iot_elock_oobe project is the application on the SLN-VIZNAS-IOT (as the below figure shows, regarding the Bootstrap and Bootloader in the software flowchart, I will introduce them in the future). The following development work is based on the sln_viznas_iot_elock_oobe project, however, I need to sketch the basic workflow of it prior to starting real development work. Fig4 SLN-VIZNAS-IOT software flowchart sln_viznas_iot_elock_oobe's workflow flow In the Camera_Start() function, the task (Camera_Init_Task) completes the initialization of the RGB and IR cameras, then creates a task (Camera_Task); In the Display_Start() function, after the task (Display_Init_Task) completes the initialization of the display medium (USB or LCD), it immediately creates the task (Display_Task) and sends the message queue s_DisplayReqMsg.id = QMSG_DISPLAY_FRAME_REQ to the task (Camera_Task), then the pDispData will point to the s_BufferLcd[0] array for storing the image data to be displayed; In the Oasis_Start() function, firstly, OASISLT_init() completes the initialization of the OAISIT library, then creates a task (Oasis_Task) to send the message queues gFaceDetReqMsg.id = QMSG_FACEREC_FRAME_REQ and gFaceInfoMsg.id = QMSG_FACEREC_INFO_UPDATE to the task (Camera_Task) to make the pDetIR and pDetRGB point to the face block diagram captured by the RGB and IR cameras, and update the content pointed by infoMsgIn. After the camera is initialized, the RGB camera works at first. After the image data is captured, an interrupt is triggered and the callback function Camera_Callback() sends the message queue DQMsg.id = QMSG_CAMERA_DQ to the task (Camera_Task), and DQIndex++; CAMERA_RECEIVER_GetFullBuffer() extracts the image data captured by the RGB camera, and sends the message queue DPxpMsg.id = QMSG_PXP_DISPLAY to the task (PXP_Task) created in the APP_PXP_Start() function and EQIndex++, meanwhile switch the camera from RGB to IR. After the APP_PXPStartCamera2Display() function in the task (PXP_Task) completes processing, it sends the message queue s_DResMsg.id = QMSG_PXP_DISPLAY to the task (Camera_Task), and the task (Camera_Task) sends the message queue DresMsg.id = QMSG_DISPLAY_FRAME_RES to the task (Display_Task) after receiving the above message queue. The task (Display_Task) completes display, then it sends the message queue s_DisplayReqMsg.id = QMSG_DISPLAY_FRAME_REQ to the task (Camera_Task) to make pDispData point to the s_BufferLcd[1] array; After the IR camera completes capturing work, CAMERA_RECEIVER_GetFullBuffer() extracts the image data and sends the message queue DPxpMsg.id = QMSG_PXP_DISPLAY to the (PXP_Task) task created in the APP_PXP_Start() function, continue to execute EQIndex++ and switch to RGB camera again, and repeat the steps 5. Finally, send the message queue FPxpMsg.id = QMSG_PXP_FACEREC to the task (PXP_Task) and set irReady = true. After the task (PXP_Task) receives the above message queue, it calls APP_PXPStartCamera2DetBuf() and after completes the processing, sends the message queue s_FResMsg.id = QMSG_PXP_FACEREC to the task (Camera_Task); CAMERA_RECEIVER_GetFullBuffer() extracts the image data collected by the RGB camera, repeat step 5, when (pDetRGB && irReady) condition is met, send the message queue FPxpMsg.id = QMSG_PXP_FACEREC to the task (PXP_Task) and set irReady = false, pDetRGB = NULL, pDetIR = NULL. After the task (PXP_Task) receives the above message queue, it calls APP_PXPStartCamera2DetBuf() and after completes the processing, sends the message queue s_FResMsg.id = QMSG_PXP_FACEREC to the task (Camera_Task). At this time, the (!pDetIR && !pDetRGB) condition is met and the Queue message FResMsg.id = QMSG_FACEREC_FRAME_RES is sent to the task (Oasis_Task), run OASISLT_run_extend to perform face recognition calculation, and send the message queue gFaceDetReqMsg.id = QMSG_FACEREC_FRAME_REQ to the task (Camera_Task) to make the pDetIR and pDetRGB point to the face block diagram captured by the RGB and IR cameras again. keep repeat steps 6 and 7; Fig5 sln_viznas_iot_elock_oobe's workflow flow Smart Coffee machine Fig 6 is the workflow of the smart coffee machine that I want to develop for, as there is no LCD board on hand, in the below development process, I will select Win10's camera (as the below figure shows) to output the captured image, further, take advantage of the Shell command to simulate the LCD's touch feature to interact with the board.   Fig6 workflow of the smart coffee machine Fig7 Camera Code modification In the commondef.h, add a new member variable 'uint16_t coffee_taste' in Union FeatureItem to stand for the favorite coffee taste; typedef union { struct { /*put char/unsigned char together to avoid padding*/ unsigned char magic; char name[FEATUREDATA_NAME_MAX_LEN]; int index; // this id identify a feature uniquely,we should use it as a handler for feature add/del/update/rename uint16_t id; uint16_t pad; // Add a new component uint16_t coffee_taste; /*put feature in the last so, we can take it as dynamic, size limitation: * (FEATUREDATA_FLASH_PAGE_SIZE * 2 - 1 - FEATUREDATA_NAME_MAX_LEN - 4 - 4 -2)/4*/ float feature[0]; }; unsigned char raw[FEATUREDATA_FLASH_PAGE_SIZE * 2]; } FeatureItem; // 1kB   In featuredb.h, add two member functions into class FeatureDB:  set_taste()  and  get_taste() , and add the definition of the above two member functions in featuredb.cpp; class FeatureDB { public: FeatureDB(); ~FeatureDB(); int add_feature(uint16_t id, const std::string name, float *feature); int update_feature(uint16_t id, const std::string name, float *feature); int del_feature(uint16_t id, std::string name); int del_feature(const std::string name); int del_feature_all(); std::vector<std::string> get_names(); int get_name(uint16_t id, std::string &name); std::vector<uint16_t> get_ids(); int ren_name(const std::string oldname, const std::string newname); int feature_count(); int get_free(int &index); int database_save(int count); int get_feature(uint16_t id, float *feature); void set_autosave(bool auto_save); bool get_autosave(); //Add two customize member functions int set_taste(const std::string username, uint16_t taste_number); int get_taste(const std::string username); private: bool auto_save; int load_feature(); int erase_feature(int index); int save_feature(int index = 0); int reassign_feature(); int get_free_mapmagic(); int get_remain_map(); }; int FeatureDB::set_taste(const std::string username, uint16_t taste_number) { int index = FEATUREDATA_MAX_COUNT; for (int i = 0; i < FEATUREDATA_MAX_COUNT; i++) { if (s_FeatureData.item[i].magic == FEATUREDATA_MAGIC_VALID) { if (!strcmp(username.c_str(), s_FeatureData.item[i].name)) { index = i; } } } if (index != FEATUREDATA_MAX_COUNT) { s_FeatureData.item[index].coffee_taste = taste_number; return 0; } else { return -1; } } int FeatureDB::get_taste(const std::string username) { int index = FEATUREDATA_MAX_COUNT; int taste_number; for (int i = 0; i < FEATUREDATA_MAX_COUNT; i++) { if (s_FeatureData.item[i].magic == FEATUREDATA_MAGIC_VALID) { if (!strcmp(username.c_str(), s_FeatureData.item[i].name)) { index = i; } } } if (index != FEATUREDATA_MAX_COUNT) { taste_number = s_FeatureData.item[index].coffee_taste; return taste_number; } else { return -1; } }   In database.h, add the declarations of  DB_Set_Taste()  and  DB_Get_Taste()  functions, and in database.cpp, add the related codes of the above two functions. These two functions are equivalent to encapsulating the newly added member functions set_taste() and get_taste() of the FeatureDB class; int DB_Del(uint16_t id, std::string name); int DB_Del(string name); int DB_DelAll(); int DB_Ren(const std::string oldname, const std::string newname); int DB_GetFree(int &index); int DB_GetNames(std::vector<std::string> *names); int DB_Count(int *count); int DB_Save(int count); int DB_GetFeature(uint16_t id, float *feature); int DB_Add(uint16_t id, float *feature); int DB_Add(uint16_t id, std::string name, float *feature); int DB_Update(uint16_t id, float *feature); int DB_GetIDs(std::vector<uint16_t> &ids); int DB_GetName(uint16_t id, std::string &names); int DB_GenID(uint16_t *id); int DB_SetAutoSave(bool auto_save); // Add two customize functions int DB_Set_Taste(const std::string username, const uint16_t taste); int DB_Get_Taste(const std::string username); int DB_Set_Taste(const std::string username, const uint16_t taste) { int ret = DB_MGMT_FAILED; ret = DB_Lock(); if (DB_MGMT_OK == ret) { ret = s_DB->set_taste(username, taste); DB_UnLock(); } return ret; } int DB_Get_Taste(const std::string username) { int ret = DB_MGMT_FAILED; ret = DB_Lock(); if (DB_MGMT_OK == ret) { ret = s_DB->get_taste(username); DB_UnLock(); } return ret; } In sln_api.h, add the declarations of the functions  VIZN_SetTaste() ,  VIZN_GetTaste()  and  VIZN_Is_Rec_User() , and add the codes of the above three functions in sln_api.cpp. The VIZN_SetTaste() and VIZN_GetTaste() functions are equivalent to the encapsulation of the DB_Set_Taste() and DB_Get_Taste() functions. Why is it so complicated? To follow the code layering mechanism of the elock_oobe project and reduce the difficulty of code implementation through code layered encapsulation. /** * @brief Set user's favorite coffee taste. * * @Param clientHandle The client handler which required this action * @Param userName Pointer to a buffer which contains the name of the new user. * @Param taste Coffee taste */ vizn_api_status_t VIZN_SetTaste(VIZN_api_client_t *clientHandle, char *UserName, cfg_Coffee_taste taste); /** * @brief Set user's favorite coffee taste. * * @Param clientHandle The client handler which required this action * @Param userName Pointer to a buffer which contains the name of the new user. * @Param taste Pointer to the Coffee taste */ vizn_api_status_t VIZN_GetTaste(VIZN_api_client_t *clientHandle, char *UserName, int *taste); vizn_api_status_t VIZN_Is_Rec_User(VIZN_api_client_t *clientHandle, char *UserName); ~~~~~~~~~ vizn_api_status_t VIZN_SetTaste(VIZN_api_client_t *clientHandle, char *UserName, cfg_Coffee_taste taste) { int32_t status; if (!IsValidUserName(UserName)) { return kStatus_API_Layer_RenameUser_InvalidUserName; } status = DB_Set_Taste(std::string(UserName), (uint16_t)taste); if (status == 0) { return kStatus_API_Layer_Success; } else if (status == -1) { return kStatus_API_Layer_SetTaste_Failed; } } vizn_api_status_t VIZN_GetTaste(VIZN_api_client_t *clientHandle, char *UserName, int *taste) { int32_t status; if (!IsValidUserName(UserName)) { return kStatus_API_Layer_RenameUser_InvalidUserName; } *taste = DB_Get_Taste(std::string(UserName)); if (*taste != -1) { return kStatus_API_Layer_Success; } else { return kStatus_API_Layer_GetTaste_Failed; } } vizn_api_status_t VIZN_Is_Rec_User(VIZN_api_client_t *clientHandle, char *UserName) { if (!IsValidUserName(UserName)) { return kStatus_API_Layer_RenameUser_InvalidUserName; } return kStatus_API_Layer_Success; } In sln_api_init.cpp, declare the variable:  std::string Current_User = "" ; which is used to store the name corresponding to the face after recognition, and add the processing function  Coffee_Rec()  after successful face recognition in the structure variable ops2; std::string Current_User = " "; //Add customize function int Coffee_Rec(VIZN_api_client_t *pClient, face_info_t face_info); client_operations_t ops2 = { .detect = NULL, .recognize = Coffee_Rec,//NULL, .enrolment = NULL, }; //Add customize function int Coffee_Rec(VIZN_api_client_t *pClient, face_info_t face_info) { Current_User = face_info.name; return 1; } In sln_timers.h, increase MS_SYSTEM_LOCKED to extend the locked status time to 25 seconds; ~~~~~~~~ #define MS_SYSTEM_LOCKED 25000 //2000 // MS in which the board is in a locked state after a reg/rec. ~~~~~~~~ In sln_cli.cpp, add three Shell commands: order, set_taste, get_taste to stand for the operations of brewing coffee, setting coffee taste, and checking coffee taste; SHELL_COMMAND_DEFINE(set_taste, (char *)"\r\n\"set_taste username <0|1|2|3|~>\": set user's favorite taste\r\n" "0 - Cappuccino\r\n" "1 - Black Coffee\r\n" "2 - Coffee latte\r\n" "3 - Flat White\r\n" "4 - Cortado\r\n" "5 - Mocha\r\n" "6 - Con Panna\r\n" "7 - Lungo\r\n" "8 - Ristretto\r\n" "9 - Others \r\n", FFI_CLI_SetTasteCommand, SHELL_IGNORE_PARAMETER_COUNT); SHELL_COMMAND_DEFINE(get_taste, (char *)"\r\n\"get_taste username\": return user's favorite taste \r\n", FFI_CLI_GetTasteCommand, SHELL_IGNORE_PARAMETER_COUNT); SHELL_COMMAND_DEFINE(order, (char *)"\r\n\"order <0|1|2|3|~>\": order a favorite taste \r\n", FFI_CLI_OrderCommand, SHELL_IGNORE_PARAMETER_COUNT); ~~~~~~ static shell_status_t FFI_CLI_SetTasteCommand(shell_handle_t shellContextHandle, int32_t argc, char **argv) { if (argc != 3) { SHELL_Printf(shellContextHandle, "Wrong parameters\r\n"); return kStatus_SHELL_Error; } return UsbShell_QueueSendFromISR(shellContextHandle, argc, argv, SHELL_EV_FFI_CLI_SET_TASTE); } static shell_status_t FFI_CLI_GetTasteCommand(shell_handle_t shellContextHandle, int32_t argc, char **argv) { if (argc != 2) { SHELL_Printf(shellContextHandle, "Wrong parameters\r\n"); return kStatus_SHELL_Error; } return UsbShell_QueueSendFromISR(shellContextHandle, argc, argv, SHELL_EV_FFI_CLI_GET_TASTE); } shell_status_t FFI_CLI_OrderCommand(shell_handle_t shellContextHandle, int32_t argc, char **argv) { if (argc > 2) { SHELL_Printf(shellContextHandle, "Wrong parameters\r\n"); return kStatus_SHELL_Error; } return UsbShell_QueueSendFromISR(shellContextHandle, argc, argv, SHELL_EV_FFI_CLI_ORDER); } ~~~~~~ shell_status_t RegisterFFICmds(shell_handle_t shellContextHandle) { SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(list)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(add)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(del)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(rename)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(verbose)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(camera)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(version)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(save)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(updateotw)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(reset)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(emotion)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(liveness)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(detection)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(display)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(wifi)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(app_type)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(low_power)); // Add three Shell commands SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(order)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(set_taste)); SHELL_RegisterCommand(shellContextHandle, SHELL_COMMAND(get_taste)); return kStatus_SHELL_Success; } In sln_cli.cpp, it needs to add corresponding codes for handle order, set_taste, get_taste instructions in task UsbShell_CmdProcess_Task else if (queueMsg.shellCommand == SHELL_EV_FFI_CLI_SET_TASTE) { int coffee_taste = atoi(queueMsg.argv[2]); if (coffee_taste >= Cappuccino && coffee_taste <= Others) { status = VIZN_SetTaste(&VIZN_API_CLIENT(Shell),(char *)queueMsg.argv[1], (cfg_Coffee_taste)coffee_taste); if (status == kStatus_API_Layer_Success) { SHELL_Printf(shellContextHandle, "User: %s like coffee taste: %s \r\n", queueMsg.argv[1], Coffee_type[coffee_taste]); } else { SHELL_Printf(shellContextHandle, "Cannot set coffee taste\r\n"); } } else { SHELL_Printf(shellContextHandle, "Unsupported coffee taste\r\n"); } } else if (queueMsg.shellCommand == SHELL_EV_FFI_CLI_GET_TASTE) { int get_taste_num = 0; status = VIZN_GetTaste(&VIZN_API_CLIENT(Shell),(char *)queueMsg.argv[1], &get_taste_num); if (status == kStatus_API_Layer_Success) { SHELL_Printf(shellContextHandle, "User: %s like coffee taste: %s \r\n", queueMsg.argv[1], Coffee_type[(cfg_Coffee_taste)(get_taste_num)]); } else { SHELL_Printf(shellContextHandle, "Cannot get coffee taste\r\n"); } } else if (queueMsg.shellCommand == SHELL_EV_FFI_CLI_ORDER) { status = VIZN_Is_Rec_User(&VIZN_API_CLIENT(Shell),(char *)Current_User.c_str()); if (status == kStatus_API_Layer_Success) { if (queueMsg.argc == 1) { int get_taste_num = 0; status = VIZN_GetTaste(&VIZN_API_CLIENT(Shell),(char*)Current_User.c_str(), &get_taste_num); if (status == kStatus_API_Layer_Success) { SHELL_Printf(shellContextHandle, "User: %s order the a cup of %s \r\n", Current_User.c_str(), Coffee_type[(cfg_Coffee_taste)(get_taste_num)]); } else { SHELL_Printf(shellContextHandle, "Sorry, please order again, Current user is %s\r\n",Current_User.c_str()); } } else if(queueMsg.argc == 2) { int coffee_taste = atoi(queueMsg.argv[1]); if (coffee_taste >= Cappuccino && coffee_taste <= Others) { status = VIZN_SetTaste(&VIZN_API_CLIENT(Shell),(char*)Current_User.c_str(), (cfg_Coffee_taste)coffee_taste); if (status == kStatus_API_Layer_Success) { SHELL_Printf(shellContextHandle, "User: %s order a cup of %s \r\n", Current_User.c_str(), Coffee_type[coffee_taste]); } else { SHELL_Printf(shellContextHandle, "Cannot set coffee taste, Current user is %s\r\n",Current_User.c_str()); } } else { SHELL_Printf(shellContextHandle, "Unsupported coffee taste\r\n"); } } } } Use the cafe logo of《Friends》to replace the original Welcome_home picture, use the BmpCvt tool to convert the picture into the corresponding array, and add it to welcomehome_320x122.h. static const unsigned short Coffee_shop_320_122[] = { 0x59E6, 0x6227, 0x6247, 0x59C5, 0x59C5, 0x59A5, 0x4103, 0x6A67, 0x6A47, 0x6227, 0x6A47, 0x6A68, 0x7268, 0x6A67, 0x6A67, 0x6A47, 0x72A9, 0x6A68, 0x7268, 0x6A48, 0x5A06, 0x6A88, 0x6A68, 0x6247, 0x6A47, 0x7289, 0x7289, 0x6A47, 0x6A47, 0x6A47, 0x6227, 0x6A68, 0x6206, 0x6A47, 0x5A26, 0x6247, 0x6227, 0x6A27, 0x4924, 0x836D, 0x5207, 0x7BAC, 0x5247, 0x83ED, 0x4A47, 0x2923, 0x7B8C, 0x49E5, 0x49E5, 0x4A05, 0x28C1, 0x5226, 0x6267, 0x6A87, 0x72E9, 0x6267, 0x6AA9, 0x5A27, 0x6AA9, 0x6AA9, 0x5A47, 0x6A88, 0x5A06, 0x5A47, 0x6AA9, 0x5A47, 0x62A9, 0x5206, 0x6288, 0x6268, 0x5A47, 0x5A27, 0x5A47, 0x5A27, 0x49E6, 0x4A07, 0x4A07, 0x5A89, 0x49C6, 0x5A48, 0x5A28, 0x5A47, 0x5226, 0x49E6, 0x49C6, 0x41A6, 0x5208, 0x2082, 0x52A8, 0x6B6B, 0x39A5, 0x39A5, 0x3964, 0x49E7, 0x3104, 0x49C7, 0x3945, 0x41A6, 0x28A2, 0x2061, 0x3965, 0x28E3, 0x1881, 0x3944, 0x3103, 0x3103, 0x3903, 0x4145, 0x51A6, 0x51C6, 0x4985, 0x51E6, 0x51E6, 0x61E7, 0x6A48, 0x6A28, 0x6A28, 0x6A27, 0x61E6, 0x6207, 0x6A68, 0x59E7, 0x4185, 0x51E6, 0x51A6, 0x6228, 0x5A07, 0x6228, 0x5A08, 0x4184, 0x41A5, 0x4164, 0x3944, 0x3944, 0x736B, 0x83ED, 0x41A5, 0x83ED, 0x6288, 0x8BAB, 0x836A, 0x6287, 0x6B2A, 0x5267, 0x83CD, 0x5A68, 0x5228, 0x3986, 0x3985, 0x7B0A, 0x6A67, 0x7267, 0x832B, 0x49A5, 0x6206, 0x8AC9, 0x72A8, 0x82C9, 0x82E9, 0x8309, 0x6A46, 0x8B2B, 0x3860, 0x8329, 0x6A67, 0x7288, 0x7268, 0x61E6, 0x7267, 0x6A67, 0x59C5, 0x51A4, 0x6A46, 0x7AA8, 0x6A26, 0x7287, 0x7AA8, 0x72A8, 0x72A9, 0x51C5, 0x5A27, 0x5A27, 0x3923, 0x ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ 0x7B8C, 0x734B, 0x6B0A, 0x83CD, 0x83ED, 0x8C0E, 0x7B8C, 0x7B6C, 0x20C2, 0x5227, 0x83ED, 0x6AE9, 0x734B, 0x62A9, 0x7B6B, 0x7B8C, 0x62E9, 0x7BAC, 0x7B6B, 0x732A, 0x940D, 0x83AC, 0x732A, 0x7309, 0x8BCC, 0x7309, 0x8BCD, 0x83AC, 0x7B6B, 0x940D, 0x3943, 0x942E, 0x7B6B, 0x734A, 0x7B8B, 0x62C8, 0x7B8B, 0x7B6A, 0x7BAB, 0x732A, 0x7B6B, 0x7B6B, 0x83CC, 0x6B09, 0x6AA9, 0x6AE9, 0x7B6B, 0x7B8B, 0x83AC, 0x734B, 0x6AC9, 0x6B0A, 0x734B, 0x734A, 0x62A8, 0x732A, 0x8C0E, 0x8BCD, 0x944F, 0x734B, 0x7B8B, 0x732A, 0x942E, 0x8BCD, 0x83AD, 0x732B, 0x6B0A, 0x6AEA, 0x62C9, 0x9C90, 0x28C2, 0x8BEE, 0x93EE, 0x8BCD, 0x4183, 0x838B, 0x7B6A, 0x6287, 0x8BCB }; Programming the new project After saving the modified code and recompile the sln_viznas_iot_elock_oobe project (as shown in the figure below), then connect the MCU-LINK to J6 on the SLN-VIZNAS-IOT, just like Fig9 shows. Fig8 Recompile code Fig9 MCU-LINK (Note: it needs to reselect the Flash driver, as the below figure shows.) Fig10 Flash driver After that, it's able to program the code project to the on-board Hyperflash. Test & Summary When the new code project boot-up, please refer to Get Started with the SLN-VIZNAS-IOT to use the serial terminal to test the newly added three Shell commands: orders, set_taste, and get_taste. Once a face is successfully recognized, the cafe logo will appear up (as shown in Fig11). Fig11 Cafe logo Definitely, this smart coffee machine seems like a 'toy' demo, and there is a lot of work to improve it. Below is the list of my future work plans, Use the LCD panel instead of USB to display; Connect an external amplifier to enable voice prompt feature; Enable the Wifi feature to connect to the App; Use the GUI library to enhance UI experience; Add a voice recognition feature to control; And I'll be glad to hear any comments from you.