JuroV is right, you cannot simply convert an EDEADLK into success, since the concept of recursive mutexes is that the mutex remains locked until all recursive calls have freed the mutex. If you convert the EDEADLK into success, the first unlock will unlock the mutex proper, not what you intended.
> You even cannot mimic this with a new internal wrapper counter (or global variable counter)
> as this counter must be synchronized inside critical section of mutex code.
I beg to differ.
What you can do is to add a counter variable to each mutex (i.e. create your own recursive mutex structure containing one MUTEX_STRUCT and one _mqx_uint counter) and implement your lock as follows:
typedef struct remutex_struct
{
_mqx_uint counter;
MUTEX_STRUCT mutex;
} REMUTEX_STRUCT, _PTR_ REMUTEX_STRUCT_PTR;
_mqx_uint _remutex_lock(REMUTEX_STRUCT_PTR remutex_ptr){ _INT_DISABLE(); if ((remutex_ptr->counter == 0) || (&(remutex_ptr->mutex)->OWNER_TD == td_ptr)) { _mutex_lock(&(remutex_ptr->mutex)); } remutex_ptr->counter++; _INT_ENABLE(); return (MQX_OK);}
Be aware, this is untested, but I hope the principle comes through. In addition, no error checking is done here, that's left as an exercise 
Unlock could be done as follows:
_mqx_uint _remutex_unlock(REMUTEX_STRUCT_PTR remutex_ptr){ _INT_DISABLE(); remutex_ptr->counter--; if (remutex_ptr->counter == 0) { _mutex_unlock(&(remutex_ptr->mutex)); } _INT_ENABLE(); return (MQX_OK);} Again, without error checking, etc.
Since the _INT_ENABLE()/_INT_DISABLE() are recursive (at least in my MQX sources, see DISABLED_LEVEL), this should work IMHO.
HTH,
Johan