imxrt1064 USB video bit rate

[tamir_michael]

Hello,

I have been experimenting with USB video lately and I would like to know if a imxrt 1064 can stream video over USB at 30 FPS for an uncompressed image with the resolution 640 x 480 (16 bit per pixel). If not - can it be achieved using another data format (say, MJPEG)?

To demonstrate: Using YUYV, an image as specified above yields 0.8 FPS. However, if I set the resolution to 320 x 240 instead the FPS jumps to 3.24.

Can this device do better?

[victorjimenez]

Hi Tamir, 

I apologize for the delayed response, with the migration of the community I lost tracked of this case. 

Could you please confirm if you are still facing the same issue? 

Regards, 

Victor 

[mikestir_tdi]

Hi,

I would like to add that I believe we are having the exact same problem - on an RT1062 in our case. I have a UVC streaming application derived from the virtual camera SDK example but set to stream uncompressed 640x480 UYVY. I have patched USB_DeviceSetSpeed to set the endpoint descriptor correctly for buffer sizes > 1024, and the device enumerates correctly. If I set the maximum packet size to 1024 (for a single transfer per microframe) then the stream works perfectly at about 10 fps, but if I enable multiple transfers then the stream stops altogether and the host reports USB errors as seen in Linux Wireshark (usbmon). The endpoint is shown by lsusb with the expected 3x1024 size.

I notice that the MULT field appears to be dealt with correctly in usb_device_ehci.c, as the OP already pointed out, but it is not clear what happens regarding the buffer DMA. Is it okay to send the whole 3x1024 in a single buffer, or does it need to be sent as three separate transfers of 1024 bytes by the application? If the latter, how could we patch the USB stack to improve on this and reduce interrupt load?

There is a comment in usb_device_dci.c:

* Currently, only one transfer request can be supported for one specific endpoint.
* If there is a specific requirement to support multiple transfer requests for one specific endpoint, the application
* should implement a queue in the application level.
* The subsequent transfer could begin only when the previous transfer is done (get notification through the endpoint
* callback).

What counts as a "transfer" here? In the isochronous case the additional transfers occur within the same microframe (so no extra SOF) - does the driver count this as the same transfer or three separate ones?

Mike

[mikestir_tdi]

I have this working now and perhaps my solution will be of benefit to @tamir_michael as well.

To answer my own question, yes the driver does appear to handle splitting the three transfers. It is therefore okay to send 3072 bytes to USB_DeviceSendRequest. This means that increasing the max transfer size to 3072 and ensuring that the descriptor contains the correct 0x1400 value for wMaxPacketSize should be all that is required to modify the SDK example.

The transfer errors we were seeing appeared to occur at the end of a frame, when the packet is shorter than the maximum size. In particular, when the amount of data to send requires less than the full 3 transfers. I believe this is because the underlying EHCI driver (usb_device_ehci.c) does not set the value in the dTD multiplierOverride field. My solution was to add the following code in USB_DeviceEhciTransfer at the point where the dTD is populated:

        if (ehciState->qh[index].capabilttiesCharacteristicsUnion.capabilttiesCharacteristicsBitmap.mult) {
        	// For isochronous endpoints with additional transfers enabled, override the number of transfers
        	// if we don't need them all for this frame
        	uint16_t mult = (sendLength + ehciState->qh[index].capabilttiesCharacteristicsUnion.capabilttiesCharacteristicsBitmap.maxPacketSize - 1) /
        		ehciState->qh[index].capabilttiesCharacteristicsUnion.capabilttiesCharacteristicsBitmap.maxPacketSize;
        	if (mult == 0) {
        		// RM states "when Total Bytes = 0; then MultO should be 1"
        		mult = 1;
        	}
        	dtd->dtdTokenUnion.dtdTokenBitmap.multiplierOverride = mult;
        } else {
        	dtd->dtdTokenUnion.dtdTokenBitmap.multiplierOverride = 0;
        }

This calculates the required number of transfers for the given block size and sets MultO to a non-zero value if it is less than the full amount. It is conditional on "mult" being non-zero in the queue head, which should hopefully prevent it from being set on non isochronous IN endpoints.

It would be great if NXP could comment on the validity of this fix. The reference manual kind of implies that setting MultO is merely a performance thing, but perhaps the UVC spec requires it? I can confirm that with this added I can now achieve 30fps at 640x480 uncompressed YCbCr.

Mike

[bing_wen]

Mike: @mikestir_tdi 

I am sending 640X480X2 at 13.3fps right now. I am desperately trying to implement your fix, but limited by my knowledge I am not sure where to insert your code.  May I ask several questions?

 1. where in USB_DeviceEhciTransfer you code is inserted? 2. where did you change wMaxPacketSize, can I change HS_STREAM_IN_PACKET_SIZE? 3. do you use a 3072 bytes array when call USB_DeviceVideoSend function?

I appreciate your help.

Bing

[mikestir_tdi]

Hi Bing,

I can't remember exactly which SDK version this is against, but in my code it's at line 986 in usb_device_ehci.c. This is in between the sections that start "Set the dtd field" and the one that starts "Save the data length needed to be transferred", immediately after the line "dtd->dtdTokenUnion.dtdTokenBitmap.totalBytes = sendLength;".

I think you should be able to set HS_STREAM_IN_PACKET_SIZE to 3072, yes. Check that the descriptor then shows 0x1400 using Linux's lsusb or something.

The way I handle the video buffers is quite different to the SDK example, but the simple answer is that yes you can pass a 3072 byte buffer directly to USB_DeviceSendRequest (and so USB_DeviceVideoSend) and it should now work.

Hope this helps.

Mike

[mikestir_tdi]

I think you may also need to patch USB_DeviceSetSpeed (in usb_device_descriptor.c) to fix the wMaxPacketSize value. Add this to the beginning of that function:

uint16_t hs_size = HS_STREAM_IN_PACKET_SIZE;

// Determine number of transfers for specified per-microframe packet size (isochronous)
if (hs_size > 2048) {
  hs_size = 0x1000 | (hs_size / 3);
} else if (hs_size > 1024) {
  hs_size = 0x800 | (hs_size / 2);
}

Then a few lines below that, inside the while loop, replace the single occurrence of "HS_STREAM_IN_PACKET_SIZE" with "hs_size".

[bing_wen]

I changed the data packet format from (12bytes header + 1012 payload ) X 3  to 12 bytes header + 3060 payload. Now the frames are received by the pc but fps is still limited to 13fps.   Very disappointed.

[tamir_michael]

Try the attachment. V4L2 test bench reports 100+ FPS.

[mikestir_tdi]

Looking at my code, this line in USB_DeviceSetSpeed should NOT be changed to use hs_size:

g_UsbDeviceVideoStreamEndpoints[0].maxPacketSize =
HS_STREAM_IN_PACKET_SIZE;

I know you already said you tried reverting that one, but did you try it at the same time as changing the packet format to send a full 3060 bytes plus header?

[bing_wen]

Yes, I realized that the correct format is 12 bytes header +3060 payload so I changed my code to that. g_UsbDeviceVideoStreamEndpoints[0].maxPacketSize  is set to 3072. 

Inside USB_DeviceEhciTransfer, dtd->dtdTokenUnion.dtdTokenBitmap.multiplierOverride is always 3, sendLength is 3072, except at the last packet of a frame it is 2412. 

Everything looks right except windows camera app record at only 13.3fps. Graphedit also display 13.3fps.  I wonder what is limiting the speed.

[bing_wen]

Mike, 

I still can't make it work. These are things I did.

In usb_device_descriptor.h, I changed the HS_STREAM_IN_PACKET_SIZE to 3072U.

In usb_device_ehci.c I inserted your code mult related code to USB_DeviceEhciTransfer, which happen to be after line 986 also. 

in usb_device_descriptor.c, I insert the hs_size code in USB_DeviceSetSpeed.  I replace HS_STREAM_IN_PACKET_SIZE with hs_size in two places. One is in the while loop, it becomes   USB_SHORT_TO_LITTLE_ENDIAN_ADDRESS(hs_size,descriptorHead->endpoint.wMaxPacketSize);    Another one is after the while loop   g_UsbDeviceVideoStreamEndpoints[0].maxPacketSize = hs_size;  The second one I tried to leave it as 3072U, but didn't make a difference. 

After above changes, on PC side I didn't receive a single frame. If I change HS_STREAM_IN_PACKET_SIZE  back to 1024, I would receive the frames at 13 fps. Or if I change USB_DEVICE_ECHI_DTD_TOTAL_BYTES from 0x4000 to 0x400, I can also receive frames at ~ 7fps.  In the latter case, USB_DeviceEhciTransfer chops the 3072 bytes into chunks of 1024 bytes. 

Also I would like to describe the 3072 bytes I send to USB_DeviceEhciTransfer function. s_ImageBuffer  is a 3072 bytes array, starts with a 12 bytes header followed by a 1012 bytes payload. Then 12 bytes header again followed by a 1012 bytes payload. Then 12 bytes header and 1012 bytes payload again. This pattern is different only if it is at the end of a frame.

I am using i.MiMXRT1052 board, with SDK 2.8.6, on MCUxpresso 11.2.0.

Is there anything you can spot that is wrong?  Or can  you share a working example that doesn't contain any proprietary  information? 

I appreciate your great help! 

Bing 

[bing_wen]

Mike,

I will give it a try. Thank you for the help.

Bing

[victorjimenez]

Hello Tamir, 

Regarding your question, the RT1064 is capable of sending a 640 x 480 image through USB at 30 fps.

Best Regards, 

Victor 

[tamir_michael]

Hi and thanks for your reply,

Could you elaborate on how this can be done?

Obviously I'm doing something wrong, but I have no idea how to boost data transfer. Can you help me with this? Let's take your video over USB demo program as an example - it sends out a MJPEG with the resolution of 176x144 at 30 FPS. Can you change this program to send out an MJPEG at 640x480 at 30 FPS? I tried, but then the FPS falls dramatically (in addition to visual distortions which are obvious because I did not change the image size, but that is not important for this experiment).

I have attached a very similar mcuXpresso program that sends out video over USB at 320x240 using YUYV, but it manages only 3.2 FPS. It is very similar to the demonstration program NXP provides.

Can you point out how to boost the FPS of this program or any other?

[victorjimenez]

Hi Tamir, 

I checked with the applications team and a while ago they developed a UVC demo working at 30fps. This demo is for the RT1050-EVKB and it's was developed using IAR IDE. Although you are using a different RT and probably a different IDE, you can use this project to see all the configurations that they need to made to work at 30fps. I already sent the project to your distributor so he can send it to you directly. Let me know if you don't receive it and I will send it to you. 

Best Regards, 

Victor 

[徐芳泽]

您好，请问您是否可以将demo也发送给我一份呢？十分感谢，我也正在进行相关的尝试。

[tamir_michael]

上面提到的例子只建立在 IAR 上，我从来没有让它与 mcuxpresso 一起工作。 我附上了一个流式传输 30 FPS + MPEG 的示例 - 更改设备配置描述符以切换到 UVC。

[徐芳泽]

十分感谢您的帮助，我研究研究这个demo。感谢您！

[bing_wen]

Victor,

Would you please point out how I can get your working project? We have board made using rt1050 but can't send data fast enough. 

Thanks

Bing

[bing_wen]

I have exactly the same problem to ask. Would you please send a copy of the example also? Or tell me what it takes for it to be available. Thanks.