OpenSSL engine hangs on deinitialisation when ciphers are supported

[fvjdxe]

I am working with version 3.0.3 of the plug & trust middle-ware.

I have extended the OpenSSL engine to support ciphers (AES-256-CBC). Now the engine hangs in a deadlock on a mutex when being deinitialised - even when using elliptic curves and not ciphers.

What happens is that OpenSSL's global_engine_lock, which is of type pthread_rwlock_t has already been locked once in engine_table_cleanup() (OpenSSL crypto/engine/eng_table.c) when EmbSe_Finish() is called. When sss_session_close() is called inside EmbSe_Finish(), this involves encryption again. And since the engine now support ciphers, OpenSSL locks the global_engine_lock again. I can see that pthread_rwlock_wrlock() returns EDEADLK at this point, indicating that the mutex cannot be locked again. But OpenSSL doesn't check this error code. So it ends up calling pthread_rwlock_unlock() twice which - according to the documentation - leads to undefined behavior.

I can't tell exactly where the second locking happens, but I saw that CMAX_Init() was active at this point.

Are you aware of this problem? Do you have any proposal on how to solve it? Would you say it is a bug in OpenSSL to lock the mutex twice or are we not allowed to use OpenSSL's encryption from EmbSe_Finish()?

Best regards

[Kan_Li]

Hi @fvjdxe ,

Actually an extension of the OpenSSL engine with AES we have so far not looked into.

Looks like you have an user session opened from the openssl engine, right? As when no user session is open then sss_session_close() does not send an APDU and so needs no crypto functionality.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[fvjdxe]

Hi @Kan_Li,

what do you mean with user session?

I use the OpenSSL command line tool like this:
openssl enc -aes-256-cbc -in test.bin -out test-encrypted.bin -iv $(hexdump -v -e '/1 "%02X"' < iv.bin) -K $(hexdump -v -e '/1 "%02X"' < key-reference-00000003.bin)

Or like this:
openssl dgst -sha256 -sign private-reference-00000002.der -keyform der -out signature.bin test.bin

Best regards

[Kan_Li]

Hi @fvjdxe ,

The user session means the applet session which is an authenticated communication channel between the owner of an Authentication Object and the SE050 IoT applet.

Please refer to section 3.6 in AN12413 for more details on this topic.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[fvjdxe]

Yes, the engine opens a session in EmbSe_Init() and closes it in EmbSe_Finish().

As soon as you add AES to the engine (add call to ENGINE_set_ciphers() in bind_helper() and change default_algorithms to "EC, CIPHERS" in the config file) closing the session hangs, because OpenSSL's global engine lock is locked twice.

I have worked around the issue. I don't open the session in EmbSe_Init(), but just in time when I need it. For example the session is opened and closed inside EmbSe_ECDSA_Do_Sign().

I still wonder if something is wrong conceptually. If OpenSSL's global engine lock must not be locked twice and it is already locked when reaching EmbSe_Finish(), the engine should not be allowed to do things that involve a second lock attempt. But I can't find any documentation on this. Maybe you know more.

[Kan_Li]

Hi @fvjdxe ,

If the session which gets created by the openSSL engine needs some encryption, this is possibly the cause for entering the engine twice. In case the session is a PlatformSCP session, the encryption shall be done using the AcessManager in between the engine and the secure element.

In case the user session is a AES or ECKey session you might have to explicitly configure the openssl AES calls from the MW to not enter the openssl engine.

Hope that makes sense,

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[fvjdxe]

Hello Kan,

the session is an EC key session.

I played around with explicitly using the software engine inside the middle-ware. I think I did something like this:

CMAC_Init(..., ENGINE_by_id("openssl"));

This should tell OpenSSL to use the default software engine for the CMAC operations. Sadly it didn't find the default engine. Instead it was searching for some library file named "openssl".

Do you, by chance, have any experience with engine selection?

Best regards

[Kan_Li]

Hi @fvjdxe ,

Not yet, but as far as I know, it is possible to add OpenSSL engine specific extensions to the OpenSSL configuration file. Using these extensions one can control whether the supported crypto functionality is delegated to the Secure Element or whether it is handled by the OpenSSL SW implementation. Please kindly refer to "se05x_mw_v04.01.00/simw-top/doc/sss/plugin/openssl/scripts/readme.html" for more details.

Have a great day,
Kan

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!
- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------