Hi Tomas Vanek
Could you share the code that you are using, I would like to see in more details your process. In my understanding of your case, your endpoint is stall by the host; when device doesn't respond to the host, host stall the endpoint and the host have to Unstall these endpoints. This is why you have to make a USB reset to can recover your endpoint. You as device cannot cancel a transfer without inform the host, device has not that kind of control in the transfer, host need to send a request of cancelling the transfer.
I recommend you to check the example code of usb_device_msc, more specific, usb_device_msc.c file, it uses in different times USB_DeviceCancel function, it could help you to understand the usage of this API.
Hope this helps.
Have a great day,
Jorge Alcala