I have a similar problem.
I am trying to perform a mod_inv execution with pkeu´s math funtions.
I convert a k bignum to montgomery format and pass p a mont(k), to security engine and the result doesn't work.
The descryptor has the next fields.
sd_header -> 0x50f00081
ptr[0].len= 66
ptr[0].ptr= P -> prime of nist 521 bits curve
ptr[2].len= 3
ptr[2].ptr= [80 00 00] it is the montgomery representation of u= 1
ptr[4].len= 66
ptr[4].ptr= address of result