CST HSM 3.2.0 compiled against Openssl 1.1.1f leaks memory and segfaults

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

CST HSM 3.2.0 compiled against Openssl 1.1.1f leaks memory and segfaults

1,775 Views
atte
Contributor II

Hi,

I'm attempting to compile CST against OpenSSL 1.1.1f. To achieve this I've recompiled libfrontend (as the packaged one seems to be compiled against older openssl) and used this binary to compile the complete tool. The tool out of my compile efforts segfaults, under valgrind the execution completes successfully, but valgrind reports memory leaks and an invalid read (resulting in the segmentation fault under normal run). The same behaviour manifests with both pkcs11-proxy and softhsm.

OS: Ubuntu 20.04 x64

CST ver: 3.2.0

OpenSSL ver: 1.1.1f

PKCS11 lib: pkcs11-proxy (libpkcs11-proxy)

Compilation script: 

#!/bin/bash


export OSTYPE=linux64
export ARCH=64

pushd code/cst/
make clean
make build
popd


pushd code/back_end-hsm/src/

cp ../../cst/code/obj.linux64/libfrontend.a .
make clean
make
make all
popd

Valgrind output (relevant parts):

=9601== Invalid read of size 8
==9601== at 0x4842AD6: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==9601== by 0x5174B36: gck_rpc_mechanism_list_purge (in /usr/local/lib/libpkcs11-proxy.so.0.1)
==9601== by 0x516C6F3: rpc_C_GetMechanismList (in /usr/local/lib/libpkcs11-proxy.so.0.1)
==9601== by 0x1209D1: hsm_register_token (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x121447: pre_init_hsm (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x121E97: hsm_ctrl (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x49AE49A: ENGINE_ctrl_cmd_string (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==9601== by 0x11DC84: gen_sig_data (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x110B60: create_sig_file (cst.c:1389)
==9601== by 0x119300: cmd_handler_authenticatecsf (csf_cmd_aut_dat.c:410)
==9601== by 0x11041B: handle_command (cst.c:928)
==9601== by 0x114310: yyparse (cst_parser.y:109)
==9601== Address 0x4dce440 is 0 bytes after a block of size 544 alloc'd
==9601== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==9601== by 0x12092A: hsm_register_token (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x121447: pre_init_hsm (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x121E97: hsm_ctrl (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x49AE49A: ENGINE_ctrl_cmd_string (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==9601== by 0x11DC84: gen_sig_data (in /home/xzr/work/cst/release/code/back_end-hsm/src/cst)
==9601== by 0x110B60: create_sig_file (cst.c:1389)
==9601== by 0x119300: cmd_handler_authenticatecsf (csf_cmd_aut_dat.c:410)
==9601== by 0x11041B: handle_command (cst.c:928)
==9601== by 0x114310: yyparse (cst_parser.y:109)
==9601== by 0x1111AB: main (cst.c:1676)
==9601==

<snip--> ... <--snip>

CSF Processed successfully and signed data available in out.bin
==9601==
==9601== HEAP SUMMARY:
==9601== in use at exit: 101,165 bytes in 321 blocks
==9601== total heap usage: 4,690 allocs, 4,369 frees, 583,740 bytes allocated
==9601==
==9601== LEAK SUMMARY:
==9601== definitely lost: 59,185 bytes in 127 blocks
==9601== indirectly lost: 11,255 bytes in 103 blocks
==9601== possibly lost: 0 bytes in 0 blocks
==9601== still reachable: 30,725 bytes in 91 blocks
==9601== suppressed: 0 bytes in 0 blocks
==9601== Rerun with --leak-check=full to see details of leaked memory
==9601==
==9601== For lists of detected and suppressed errors, rerun with: -s
==9601== ERROR SUMMARY: 12 errors from 1 contexts (suppressed: 0 from 0)

Any ideas to sort this out? Should I just compile against 1.1.0 or so? I did try to compile against Openssl 1.0.2 but ran into some issues there too (though might have been pebkac).

5 Replies

1,691 Views
Yuri
NXP Employee
NXP Employee

Hello,

   please try to use the latest CST (3.3.0)  with OpenSSL 1.0.2.

Regards,

Yuri.

0 Kudos

1,691 Views
atte
Contributor II

Hi Yuri,

Thanks for the reply!

Apparently I am using CST 3.3.0 (Downloaded cst 3.3.0 tar.gz from the nxp site). However the tool in this package reports it's version as 3.2.0 (with cst -v), which is a bit odd I guess.

I did try compiling against Openssl 1.0.2u by setting OPENSSL_PATH environment variable (pointing to /home/xzr/openssl which has the compiled and installed 1.0.2u), this still results in a segfault. Also linking srktool with this setup results in an error (undefined references for example for EVP_MD_CTX_destroy)

Is there something I'm missing here?

The compilation seems to work on systems that have 1.0.2 as system openssl (like centos7), but I would like to get it working on other platforms as well.

0 Kudos

1,691 Views
atte
Contributor II

Okay. I got it to run without segfaulting but I'm not completely sure why my fix works.

First I utilized the fix in  this thread https://community.nxp.com/thread/512437 

In addition to that I added an extra byte to the mechanism list malloc on line 677 of e_hsm.c

(so now it reads "sizeof (CK_MECHANISM_TYPE) + 1" instead of "sizeof(CK_MECHANISM_TYPE)")

Setup:

CST 3.3.0

OpenSSL 1.0.2u (installed locally)

I'm not sure if this is a desired fix, but the tool seems to work properly now in my environment at least...

BR,

-Atte 

1,671 Views
cheuschkel
Contributor III

@atte How were you able to compile the libfrontend.a library? I get a lot of linker problems after setting my OPENSSL_PATH and OS_TYPE=linux64. As seen in this linked post I created, I can seem to build the libfrontend.a library with an OpenSSL system version of v1.1.1f, but everywhere I read the NXP rep says you have to use 1.0.2.

Thanks,

Cory

0 Kudos

1,612 Views
atte
Contributor II

Hey @cheuschkel

To be fair it's been long enough that I forgot about how to get it working. However I'm currently integrating cst 3.3.1 which is built against openssl 1.1.1 and had no problems building the frontend for that atleast.

0 Kudos