imxrt1064 USB video bit rate

[tamir_michael]

Hello,

I have been experimenting with USB video lately and I would like to know if a imxrt 1064 can stream video over USB at 30 FPS for an uncompressed image with the resolution 640 x 480 (16 bit per pixel). If not - can it be achieved using another data format (say, MJPEG)?

To demonstrate: Using YUYV, an image as specified above yields 0.8 FPS. However, if I set the resolution to 320 x 240 instead the FPS jumps to 3.24.

Can this device do better?

[tamir_michael]

Hi Victor,

Yes this is correct.

Thanks for your support,

Tamir

[victorjimenez]

Hi Tamir, 

I made a couple of tests on my side and in MCUXpresso IDE as long as I don't change the value of the macro USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE to 1 the PC does recognize the RT without problems. Is there anything I'm missing to reproduce the behavior that you mentioned? 

Regards, 

Victor 

[tamir_michael]

Hi Victor,

I cannot explain that, but I am not entirely sure it is a fundamental problem: The primary factor affecting data rate is the constant

HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION

Does the example I shared with you still work if you set this to 1 ? Does the example I shared with you still work if you set this to 2 ?

This is the constant that determines the number of additional packet placed on bus and this is the setting that transforms a working sample that delivers 13.28[FPS] to none.

Please change this value with or without changing USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE. What happens?

[victorjimenez]

Hi Tamir, 

I downloaded the example project that you provided and the PC does recognize the RT1064 correctly when I set 

HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION either to 0, 1 or 2. I always kept USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE set to 0. 

Regards, 

Victor 

[tamir_michael]

Hi Victor,

Using the example I posted above, if I set 

HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION

to 1 (and change nothing else, then program), the device enumerates correctly, but I lose video signal.

I go from 13.28 FPS for bus errors! It does not matter if I send more data in "USB_DeviceVideoPrepareVideoData". I always get bus errors and the PC/host will refuse the play the video because of lost frames.

So, in order to see it going wrong, you need to play the video. Again, no camera needed. On a linux machine you probably want to play /dev/videoX (X is an integer) using, just as an example, "V4L2 Test Bench" (freely available). It is even easier on a Windows machine as device is identified by players like VLC, Skype, Zoom etc. as well (Using VLC, you can play the video by opening a capture device. You may to wait several seconds until you actually see the random pattern).

Can you reproduce this behavior?

[victorjimenez]

Hi Tamir, 

If you set HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 2, the same as the example I provided before, do you still see this behavior? 

Regards,

Victor 

[tamir_michael]

Hi Victor,

Yes - absolutely. If I make that change I lose the video signal both on Windows and Linux machines.

Regards,

Tamir

[victorjimenez]

Hi Tamir, 

Since this is a very custom application I'm struggling to reproduce the behavior that you mentioned. Could you please provide a detailed step-by-step guide on how to reproduce the behavior on my end? My test environment is Windows 10 and VLC. 

Regards, 

Victor 

[tamir_michael]

Hi Victor,

After you program a MIMXRT1064 with the sample I provided above, connect the evaluation board to your PC using the USB port. In VLC, go to "Media->Open Capture Device". Under "Device Selection" use the drop-down box "Video device name" to select the new video source. Hit play. You should see a random color pattern.

Once you try to boost the data rate using the change we discussed above you can still select the device but the video will not show.

Regards,

Tamir

[victorjimenez]

Hi Tamir, 

I was able to verify the behavior that you mentioned. I have a couple of questions.

If you enable the camera within your application, are you still able to reproduce this behavior? I ask this because I also made some tests with the RT1050-EVKB demo project that your distributor sent you a while ago. With this project, if I set HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 2 everything works fine, I'm able to see the image that the camera is getting in VLC without problems. If I set HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 1, everything goes fine but the image is getting stuck really often. When I set HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 0, I start to see random color pattern in the image, same as what you reported. So it seems that this random color pattern occurs when the packet additional transaction is set to an incorrect value. 

I noticed that you are managing the linkerscript of your project, is there a particular reason for this? To discard that this is the cause of the behavior you are facing I would recommend that you let MCUXpresso to manage the linkerscript. 

Regards,

Victor 

[tamir_michael]

Hi Victor,

Thanks for looking into this.

The 1050 project is built using IAR. I used an evaluation version of IAR to build, program and use it on the evaluation board having HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION set to 2 without issues. Note however, that the sources differ slightly from the ones delivered with the mcuXpresson 1064 example. Unfortunately I was never able to pinpoint the root cause of why setting HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to anything but 0 on the 1064 program causes USB errors. Admittedly I not try to set HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 1 or 0 on the 1050 so I never saw the phenomenon what you report.

Note, also, that the noise pattern I mentioned above was generated by me (you can find it in the sources) in order to obviate a camera. It would seem that this is more a USB issue that a camera problerm; my noise pattern is not delivered properly to the PC with anything but HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION equal to 2.

I no longer manage the linker script. I learn not to setup what I need from within the ide.

[mjbcswitzerland]

Hi

I don't have access to the example project that you have referred to but I do know the HS USB video example that is included in SDK and is the same for all processors (generally whether running on an i.MX RT 1050 or i.MX RT 106x should not be any different since the USB controllers and thus the drivers are identical).
The example in the SDK use an interrupt on each micro-frame, which is OK when this is the only operation and the work to prepare the data is quick. But in general cases, especially when the camera's data needs to be pre-processed and clock synchronisation between camera and USB host clock domains are needed it a more efficient solution based on using chained USB descriptors is advisable - maybe the example project uses this (?)

In cases of difficulties the cause could be at various points including possibly not keeping up with the USB ISO IN rate as more data is to be prepared (higher frame rate or larger images) so generally saying what needs to be fixed is not possible.

I ave attached a slide showing the handling of isochronous audio or video data in the uTasker project, which allows simple 192MBit/s throughput with very little processor loading (based on USB circular buffer descriptor chaining) which gives a reference.

Regards

Mark

[uTasker project developer for Kinetis and i.MX RT]

[tamir_michael]

Hi Victor,

I received the sample and I'm studying it now. Thank you.

[tamir_michael]

Thank you, that's very kind. I'm looking forward to receiving it.