cst creates invalid RSA-PSS signatures with old OpenSSL

[danielgloeckner]

When using cst with OpenSSL < 3.1 the salt used in the PSS signatures is too long. Processors like the i.MX91 expect the salt to have the same length as the digest, but before OpenSSL 3.1 the default was to make the salt as big as possible.

But even with newer OpenSSL versions using the default is not good as it will accept shorter salts during signature verification.

The fix is to call EVP_PKEY_CTX_set_rsa_pss_saltlen with second parameter set to RSA_PSS_SALTLEN_DIGEST.

Note that OpenSSL 3.0 is EOL since last month.

[AldoG · Accepted solution]

Hello,

Thank you for reaching out, I will report this to the appropiate team.

Best regards/Salduos,
Aldo.