iMX RT1064 streaming over USB at 30 FPS

[tamir_michael]

Hello (NXP support in particular),

I am trying to use the latest (to date) SDK samples provided by NXP to stream video out of a MIMXRT1064 EVAL over USB at 30 FPS (see attachment for working example which is a combination of the examples "video virtual camera" and "csi_rgb565", in other words the image streamed out is no longer a static MJPEG but a UVC frame. All that you need for it is the development board + MT9M114 camera which is normally provided with it, I believe). However, the attached program have 2 main issues:
1. FPS reported by V4L2 is around 15.8 instead of 30. This is my primary concern.
2. Streaming using VLC reports problems so I use V4L2 test bench instead (image is rather corrupt but I don't really care about that right now).

Can someone please have a look and tell me why I cannot reach 30 FPS and what needs to happen to get there?

Thanks in advance!

[徐芳泽]

想请教您，如果用RT1064来移植UVC，按照1050的教程合并virtual camera和CSI时，除了教程中的操作外，还需要进行什么额外操作吗，为何我按照教程移植后，不能正常实现功能

Migrating AN12103 to EVKB-MIMXRT1050 with MCUXpres... - NXP Community

[tamir_michael]

https://community.nxp.com/t5/i-MX-RT/imxrt1064-USB-video-bit-rate/m-p/1098562

[tamir_michael]

您需要更改设备配置描述符以确保设备被枚举为 UVC 帧的来源而不是 MJPEG。 您还必须确保 MJPEG 编码被“USB_DeviceVideoPrepareVideoData”中的 UVC 标头（后跟视频数据）替换。 将我附加的最新 .zip 文件合并到这个文件中，您应该拥有所有需要的资源。 我的中文很基础，很抱歉有很多错误:-)

[kef2]

dwMaxPayloadTransferSize, which should be negotiated via probe-commit, determines your bandwidth. Looks like you limit it to 1024, which is far to the top of isochronous USB2 stream. Fastest would be 3 microframes, 1024 bytes each per "USB2 tick", or 3072 bytes in total for dwMaxPayloadTransferSize. 

With 3072 you can transfer in one second N = (3072-12)*8000 bytes.  Which for uncompressed format and VGA frame size could top at N / 640/ 480 /2 (uncompressed format, 2 bytes per pixel) = 39.8Hz. 

With dwMaxPayloadTransferSize = 1024 you can top at 13.1Hz with uncompressed format

Update: forgot about payload header

[tamir_michael]

Thanks for your reply.
I tried changing

#define HS_STREAM_IN_PACKET_SIZE 512

into

#define HS_STREAM_IN_PACKET_SIZE (1024)

but there is no change in the FPS - it's still at less than 16. Did I change the right parameter?
Programming the value (3 * 1024) yields no video signal at all.

[kef2]

You already had HS_STREAM_IN_PACKET_SIZE = 1024. I meant your probe-commit process seems limiting dwMaxPayloadTransferSize. 

As well check your streaming endpoints descriptors with corresponding Standard VS Interface Descriptor and other structs and functions. Host finds in USB desctiptors one or more set(s) of Standard VS Interface Descriptor (with unique Index) + endpoint descriptor. Each defines available bandwidth. In probe-commit host tells you which format host wants. You use this information to detemine how many bytes you send at once. 

RT1064 clearly supports Transaction length of 3072, but perhaps RT1064 driver doesn't support it. 

You should clearly study USB UVC documentation Video Class v1.5 document set | USB-IF

As a first step for > 1024 transfer you could try fixing in descriptor /* Max Packet Size */. It is not low byte followed by high byte. I don't remember where USB standards specify format of this field, actually it is two fields value, see Remarks from MS here _USBD_PIPE_INFORMATION (usb.h) - Windows drivers | Microsoft Docs

[tamir_michael]

I don't think the firmware supports extra transfers, which must be root cause for all of this:

usb_status_t USB_DeviceSetSpeed(usb_device_handle handle, uint8_t speed)
{
    usb_descriptor_union_t *descriptorHead;
    usb_descriptor_union_t *descriptorTail;

    descriptorHead = (usb_descriptor_union_t *)&g_UsbDeviceConfigurationDescriptor[0];
    descriptorTail =
        (usb_descriptor_union_t *)(&g_UsbDeviceConfigurationDescriptor[USB_DESCRIPTOR_LENGTH_CONFIGURATION_ALL - 1U]);

    while (descriptorHead < descriptorTail)
    {
        if (descriptorHead->common.bDescriptorType == USB_DESCRIPTOR_TYPE_ENDPOINT)
        {
            if (USB_SPEED_HIGH == speed)
            {
                if ((USB_VIDEO_VIRTUAL_CAMERA_STREAM_ENDPOINT_IN ==
                     (descriptorHead->endpoint.bEndpointAddress & USB_ENDPOINT_NUMBER_MASK)) &&
                    ((descriptorHead->endpoint.bEndpointAddress & USB_DESCRIPTOR_ENDPOINT_ADDRESS_DIRECTION_MASK) ==
                     USB_DESCRIPTOR_ENDPOINT_ADDRESS_DIRECTION_IN))
                {
                    descriptorHead->endpoint.bInterval = HS_STREAM_IN_INTERVAL;
                    USB_SHORT_TO_LITTLE_ENDIAN_ADDRESS(HS_STREAM_IN_PACKET_SIZE,
                                                       descriptorHead->endpoint.wMaxPacketSize);

////////////////////////////////////////////////////////////////////
Extra transaction should be programmed into wMaxPacketSize here but they are not
                }

[kef2]

I was doing project like you years ago on Vybrid family:

Solved: Freescale USB Stack and Vybrid - NXP Community

Unmodified old Freescale USB Stack as well had only MJPEG sample (from Terminator movie, isn't it?). That stack code seems quite similar with your codes, some parts seem being improved. IIRC initially it as well didn't support 3072 bytes payload, perhaps it did, I'm not sure. Please compare my modified code at the bottom of that thread with your code. In code look for "mult", check in your RT1064 documentation what that mult means. You should take a look at all routines where they check for transfer size limits. When transfer is >1024, right mult settings should end in USB Td structure. I guess you may succeed modifying your project to support faster frame rate. Unfortunately I'm not going to try RTxxxx soon, chip shortage isn't going to end soon, not a lot of sense to try new parts :-).

Good luck

[tamir_michael]

Thanks you for the useful comments.