imxrt1064 USB video bit rate

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

imxrt1064 USB video bit rate

8,447 Views
tamir_michael
Contributor IV

Hello,

I have been experimenting with USB video lately and I would like to know if a imxrt 1064 can stream video over USB at 30 FPS for an uncompressed image with the resolution 640 x 480 (16 bit per pixel). If not - can it be achieved using another data format (say, MJPEG)?

To demonstrate: Using YUYV, an image as specified above yields 0.8 FPS. However, if I set the resolution to 320 x 240 instead the FPS jumps to 3.24.

Can this device do better?

Labels (2)
55 Replies

3,171 Views
tamir_michael
Contributor IV

Hi Victor,

The program you provided indeed delivers 30FPS. 

However, I do have some comments and questions remaining:

1. I have noted that the USB interrupt is blocked until a full frame has been received from CSI. This behavior makes the code as it is unsuitable for production purposes when the main loop / OS task must do timely work. Is it possible to postpone the work of the USB ISR so that it can be executed in the main loop / task (i.e. if the main loop is expected to send out the data and not the ISR, this will happen _after_ the ISR processing by the CPU has finish - which may be unacceptable).

2. I tried to replicate this blocking behavior but somehow the CSI peripheral stopped generating interrupts. The buffer queue to CSI filled up after only a few insertions into it, because somehow the CSI stopped reporting that an frame was available. However, the same blocking code, if executed outside the ISR, does not stop the CSI and it keeps on generating interrupts. CSI interrupt priority is 0 which is higher than USB's 3.

Any comments will be very much appreciated!

0 Kudos

3,169 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

I'm glad to hear that the example project was useful for you. I will reach to the applications team to see if they have any additional information regarding this example project that might help answer your questions. I will give you an update as soon as possible. Thanks a lot for your patience. 

Regards, 

Victor 

0 Kudos

3,160 Views
tamir_michael
Contributor IV

Hi Victor,

I managed to get my controller to deliver video data (640x480x2) at 13.28 FPS. Trying to increase this by enabling 1 or 2 extra 1024 byte transactions (as your demonstration program does) always yields errors on the USB bus (so, I can deliver up to 1024 bytes per 125[us]). In order to get this to work, I programmed "wMaxPacksize" with the right bit pattern, and delivered 3072 bytes via the USB ISR. The device enumerates and video delivery is commenced, but now and then there is a piece of the image that is corrupt which breaks everything. Did I miss a step in adjusting my program to deliver more data?

0 Kudos

2,901 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

In your previous reply, you mentioned the following: The program you provided indeed delivers 30FPS. But now you are saying that the example project doesn't work properly, could you please clarify this? 

Also, regarding your requirement to postpone the work of the USB ISR so that it can be executed in the main loop. I discussed this with the applications team and they modify the example that your distributor sent you before to match this new requirement. I already send this example to your distributor so he can send it to you. 

Regards, 

Victor 

0 Kudos

2,568 Views
tamir_michael
Contributor IV

Hi Victor,

Thanks for the effort - I very much appreciate this.

Regarding your request for clarifications - my apologies, I should have explained the situation better.

Your demonstration program works fine on a MIMXRT1050 and I understand how it works. It takes advantage of 2 extra 1024 byte transfers available to USB high speed connections. It does so by setting the right values in the wMaxPacketSize field, as the specification requires. That allows this program to deliver 3072 bytes per 125us which translates to 30 FPS.

I'm working with the MIMXRT1064, so I simply replicated what I hoped are equivalent settings in my program with the hope to achieve the same bit rate. Alas, I cannot deliver more than 1024 bytes per 125us without bus errors, using either our hardware or NXP's evaluation board. I wonder if there is a detail I forgot - I though setting the maximum packet size of 1024 bytes, enabling the proper bits in wMaxPacketSize and actually delivering the data would be sufficient to reach the data rates I need but alas... I guess the question can be formulated simply as follows: If I would depart from the video demo provided with the SDK for the 1064, what do I need to do to make it capable of high speed transfers (read: delivering 3072 bytes per 125us / USB interrupt)?

0 Kudos

2,565 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

Thank you so much for the detailed information! I will check this with the engineer who created this demo project. I will get back to you as soon as possible. Thank you for your patience. 

Regards, 

Victor 

0 Kudos

2,560 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

I received the following response from the engineer who created this demo project: "The example code I shared with you can run on RT1064-EVK without any modification. I have validated that."

Regards, 

Victor 

0 Kudos

2,554 Views
tamir_michael
Contributor IV

Hi Victor,

Thanks for your response.

I understand that, but what I'm really interested in is why, if I replicate the elements from the sample _I think_ should guarantee up to 3072 bytes worth of video data per 125 us, I get errors on the host side if I exceed 1024 bytes.

To be pin-point precise:

Using your own video sample (SDK v2.7.0), in the file "usb_device_descriptor.h", why does the host generate errors and display no video if I set the value 

#define HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION (0U)  /* MAX Value is 2U*/

to 1 or 2,

and in addition

#define HS_STREAM_IN_PACKET_SIZE (1024U) // instead of 512

and change nothing else at all in the sample?

Can you please ask your engineer to do those changes on the standard SDK (v2.7.0) video example, and tell you what additional changes he had to do to get them to work with a host (i.e. the host displaying the video)?

The example we got from you was built using IAR so there may be something I'm missing here (we only use mcuXpresso). Can you please ask the engineer you mentioned _how_ he would go about transforming one of the standard SDK examples to deliver more data per unit time (3072 bytes per 125us) ?

Thank you.

0 Kudos

2,505 Views
mjbcswitzerland
Specialist V

Hi Tamir

To use more that one data packets per micro-frame the endpoint descriptor must advertise this (I think that you identified that with the HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION() setting of 2 (meaning send 2 additional data packets).

In the HS USB header setting you also need the MULT field set to match:

pastedImage_1.png

Overall the data rate and the ISO data settings must match otherwise the host may refuse to do anything (the same can be see with audio class settings that define a certain bandwidth and endpoint settings that then advertise values that would request much more than needed for the advertised payload - such details are not visible but hosts probably do some rough sanity checks and if the payload settings advertise 1MB and the endpoints request 24MB of bandwidth it presumably refuses to work (with or without error messages).

Regards

Mark

[uTasker project developer for Kinetis and i.MX RT]

0 Kudos

2,502 Views
tamir_michael
Contributor IV

Hi Mark and thanks for your reply,

I believe the value for the "mult" field is taken care of in "usb_device_ehci.c", here:

USB_DeviceEhciEndpointInit()

......

if (USB_ENDPOINT_ISOCHRONOUS == transferType)
{
if (maxPacketSize > USB_DEVICE_MAX_HS_ISO_MAX_PACKET_SIZE)
{
maxPacketSize = USB_DEVICE_MAX_HS_ISO_MAX_PACKET_SIZE;
}
ehciState->qh[index].capabilttiesCharacteristicsUnion.capabilttiesCharacteristicsBitmap.mult =
1U + ((epInit->maxPacketSize & USB_DESCRIPTOR_ENDPOINT_MAXPACKETSIZE_MULT_TRANSACTIONS_MASK) >>
USB_DESCRIPTOR_ENDPOINT_MAXPACKETSIZE_MULT_TRANSACTIONS_SHFIT);
}

where 

USB_DESCRIPTOR_ENDPOINT_MAXPACKETSIZE_MULT_TRANSACTIONS_MASK = 0x1800

and USB_DESCRIPTOR_ENDPOINT_MAXPACKETSIZE_MULT_TRANSACTIONS_SHFIT = 11

Here is my fundamental problem:

I have a program the streams over USB at 13.28 FPS with a packet size of 1024 bytes. I can slow it down by sending less data per 125 us without issues. However, once I dare to set the value of "HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION" to 1 or 2 the video signal is lost, even if I send out up to 3072 bytes per interrupt. Is this a pure timing issue? What I see happening in practice is no video frames are detected _at all_ by the host.

It is allowed to "throttle" the FPS by adjusting the amount of data delivered per microframe? For example, assuming an image dimension of 640x480x2, delivering up to 2340 bytes per 125us should yield 30FPS. However, an IAR example program created for the 1050 delivers up to the maximum amount of 3072 bytes per 125us only to block in the ISR waiting for the next image to come in from the camera. But one might as well simply "spread" the video data over the entire 33ms needed to reach 30FPS, rather then the 25ms the example employs, followed by a wait period (the block I mentioned above). Is such a policy of "speading" the data principally acceptable?

If you have access to a MIMXRT1064-EVK you can very easily reproduce this problem as I pointed out in a post above using NXP SDK video over USB examples (only a very limited number of changes are needed, also demonstrated above. You could even use example code our contact person should have delivered to NXP some days ago - a program that does not even need a camera to demonstrate the problem). Could you or somebody else please help me understand why this is happening? It is perfectly possible that there is no firmware problem at all and that the SDK's video-over-usb sample blocking in USB interrupt waiting for an image is the "solution", but then I'd expect that similar artifical delays I tried (just to get an image) would yield a similar result but they don't.

I would like to point out that even if my controller sends out a frame per 33ms I see no image on the host side. I have verified this with a scope, using the "data throtteling" method I mentioned above.

The fact of the matter is that the product will have to be redesigned unless I get this to work soon...

Thanks in advance.

0 Kudos

2,500 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

I have a couple of questions just to be sure that I understood correctly the changes that you are making to the SDK example. 

  • The only two changes that you are making is setting HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 2 and HS_STREAM_IN_PACKET_SIZE to 1024, correct? 
  • When you make these two changes the SDK example project stops working at all, correct? 
  • Are there any other changes that you have made to the example to achieve the 30fps?  
  • Just to confirm, are you using the dev_video_virtual_camera_bm example? 

Regards, 

Victor 

0 Kudos

2,485 Views
tamir_michael
Contributor IV

Hi Victor,

- "The only two changes that you are making is setting HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION to 2 and HS_STREAM_IN_PACKET_SIZE to 1024, correct? "

Yes.

- "When you make these two changes the SDK example project stops working at all, correct? "

Yes.

- "Are there any other changes that you have made to the example to achieve the 30fps?"

1. Replaced the blockage of the USB ISR (as it waits for the camera ISR) with a balanced data payload delivery per interrupt (without blocking; see (*) below) to allow me to reach 30 FPS (in my case that is 2340 bytes:

2340 bytes/125us = 18720 bytes/ms, or 18720000 bytes/second ~ 640*480*2*30 = 18432000 bytes/second = payload 30 FPS).

2. I've set HS_STREAM_IN_INTERVAL to 1.

- "Just to confirm, are you using the dev_video_virtual_camera_bm example?"

Yes.

Thanks again for your assistance.

Tamir

(*) By only adjusting the value of HS_STREAM_IN_INTERVAL to 1 (i.e. leaving HS_STREAM_IN_PACKET_ADDITIONAL_TRANSACTION set to 0)of I can deliver 1024 bytes/interrupt without blocking the USB ISR yielding 13.28 FPS:

1024 bytes/125us = 8192 bytes/ms = 8192000 bytes/sec

8192000/(640*480*2) = 13.33 FPS

0 Kudos

2,483 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

Thank you for answering my questions! I'm currently checking this with the applications team and I will give you an update as soon as possible. 

Regards, 

Victor 

0 Kudos

2,167 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

I checked this with the applications team and besides the changes that you already made, please make the below modifications:

  • Set USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE to 1.
  • Change all of the MJPEG in the descriptor to UNCOMPRESSED.
  • Specifies characteristics of the RGB565 format:

0x7B, 0xEB, 0x36, 0xE4, 0x4F, 0x52, 0xCE, 0x11, 0x9F, 0x53, 0x00, 0x20, 0xAF, 0x0B, 0xA7,
0x70, /* GUID RGB565 E436EB7B-524F-11CE-9F53-0020AF0BA770 */
USB_VIDEO_CAMERA_UNCOMPRESSED_FRAME_DATA_BITS *
8U, /* Number of bits per pixel used to specify color in the decoded video frame */

  • Set HS_INTERRUPT_IN_INTERVAL to be 1.
  • Set HS_STREAM_IN_INTERVAL to be 1.
  • Modify the descriptors' length macros according to the actual length.
  • Modify the frame/pixel related macros according to the actual values.

For detailed information, please refer to the example project.

Regards,

Victor 

0 Kudos

2,145 Views
tamir_michael
Contributor IV

Hi Victor,

Thanks for your reply.

I have already noticed that the macro "USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE" is set to 1 in the IAR sample, but if I set it to 1 in the mcuXpresso video-over-usb sample for the 1064, the device does not even enumerate properly (try it yourself!). A quick glance at "usb_device_dci.c" (version provided with SDK 2.7.0) reveals why:

#if defined __CORTEX_M && (__CORTEX_M == 7U)
#if (defined(USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE) && (USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE > 0U))


#warning USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE is not supported.

#endif
#endif


Somehow, the IAR sample uses a feature that should not be supported on the 1050 either - but it works. This is because this restriction is not imposed on the USB library the IAR sample uses - it seems to have been introduced later.

All the other changes I have made already. The one above is probably the main reason why the sample does not work with higher data rates.

Is there a difference in the hardware between the 1064 and the 1050 that can account for this behavior change?

0 Kudos

2,136 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

The RT1064 and RT1050 use the same USB IP. Could you please share with me the SDK example project with the modifications that you made to make it work at 30 FPS? This way I can made some tests on my side.

Regards, 

Victor 

0 Kudos

2,136 Views
tamir_michael
Contributor IV

Hi Victor,

See attachment. You can program it as-is onto a MIMXRT1064. If you then play the video, you will see a random pattern of 640x480 at 13.28 FPS. No camera is needed!

Tamir

0 Kudos

2,136 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

Thanks for sharing the example project with me! I will make some tests on my side. I will give you an update as soon as possible. 

Regards, 

Victor 

0 Kudos

2,136 Views
tamir_michael
Contributor IV

Hi Victor,

Did you manage to book some progress...?

Regards,

Tamir

0 Kudos

2,136 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

Sorry for the late response. I was able to reproduce the behavior that you mentioned, with MCUXpresso IDE the project stops working once I set USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE to 1. However, on IAR IDE everything works fine. I'm currently checking this with the applications team and the MCUXpresso IDE team. I will give you an update as soon as possible. Thanks for your patience. 

Regards, 

Victor 

0 Kudos

2,136 Views
victorjimenez
NXP TechSupport
NXP TechSupport

Hi Tamir, 

Just to confirm, if you make all the changes that I mentioned before but you keep the macro USB_DEVICE_CONFIG_BUFFER_PROPERTY_CACHEABLE set to 0 the RT doesn't enumerate, correct? 

Regards, 

Victor 

0 Kudos