Hi! I have been able to solve the error. Actually, in my case, uboot was modifying and updating with signature keys. Actual error was happening at this stage of code in u-boot
rsa-verify.c
#if !defined(USE_HOSTCC)
ret = uclass_get_device(UCLASS_MOD_EXP, 0, &mod_exp_dev);
if (ret) {
printf("RSA: Can't find Modular Exp implementation\n");
return -EINVAL;
}
ret = rsa_mod_exp(mod_exp_dev, sig, sig_len, prop, buf);
#else
ret = rsa_mod_exp_sw(sig, sig_len, prop, buf);
#endif
So, I went towards the software implementation of code this way
#if !defined(USE_HOSTCC)
ret = uclass_get_device(UCLASS_MOD_EXP, 0, &mod_exp_dev);
if (ret) {
printf("RSA: Can't find Modular Exp implementation\n");
return -EINVAL;
}
ret = rsa_mod_exp(mod_exp_dev, sig, sig_len, prop, buf);
#else
ret = rsa_mod_exp_sw(sig, sig_len, prop, buf);
#endif
if (ret) {
debug("Error in Modular exponentation\n");
ret = rsa_mod_exp_sw(sig, sig_len, prop, buf);
if (ret) {
printf("%s: attempting rsa_mod_exp_sw instead \n", __func__);
}
}
and adding this additional configuration in u-boot
CONFIG_RSA_SOFTWARE_EXP=y
Here the code for rsa_mod_exp_sw was little broken and giving the linking error undefined reference to 'rsa_mod_exp_sw', so I did the following change in the rsa-mod-exp.h file
--- a/include/u-boot/rsa-mod-exp.h
+++ b/include/u-boot/rsa-mod-exp.h
@@ -1,3 +1,4 @@
+
/* SPDX-License-Identifier: GPL-2.0+ */
/*
* Copyright 2014 Freescale Semiconductor, Inc.
@@ -62,7 +63,7 @@ void rsa_free_key_prop(struct key_prop *prop);
* @out: Result in form of byte array of len equal to sig_len
*/
int rsa_mod_exp_sw(const uint8_t *sig, uint32_t sig_len,
- struct key_prop *node, uint8_t *out);
+ struct key_prop *prop, uint8_t *out);
int rsa_mod_exp(struct udevice *dev, const uint8_t *sig, uint32_t sig_len,
struct key_prop *node, uint8_t *out);
After that, it started working. However, I am still searching why 'rsa_verify_key' is not working fine in my iMX8MM board and rsa_verify_key_sw working
Here are few references
https://u-boot.denx.narkive.com/BpvBKeLY/hangs-after-enabling-secured-boot-gumstix-overo
https://community.nxp.com/t5/i-MX-Processors/U-boot-FIT-image-verification-failed-when-HAB-is-enable...