i.MX6DL Display lockyp with IPU_INT_STAT_10, and IPU_INT_STAT_5 errors.

[tdefeo23]

Hi,

We are developing a commercial product using an i.MX6 duallite with 1 Gig of RAM, based on the Wandboard design. The os is built from the Yocto Fido release. We are displaying LVDS video at 1360x768@60Hz, rendering openGL, and decoding video content. Overall, the system works well.

However, we occasionally see IPU_INT_STAT_10, and IPU_INT_STAT_5 errors in the kernel log (anywhere from 0 to ~10 occurrences over a three day period). Furthermore, on rare occasion, these errors will be followed by the screen going black, and continuous time out errors during buffer swap, requiring a reboot to recover.

I modified the kernel error handler interrupt to allow the error to be reported multiple times instead of just once. Here is a sample log output:

[ 2164.178070] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[ 9199.588010] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[ 9199.588062] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_5 = 0x00800000
[10596.327461] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[12653.397027] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[13128.513804] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[30185.915102] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[52121.793364] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[56157.722802] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
[76091.073558] imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000

Any help in isolating & fixing the root cause of this would be appreciated.

Thanks,

Tony

[tw22]

In kernel 6.6.3 we were able to get rid of the problem using:

echo 0 > /sys/bus/platform/drivers/imx_busfreq/soc\:busfreq/enable

see IMXLXRM i.MX Linux Reference Manual Rev. LF6.6.3_1.0.0 — 29 March 2024 2.5.4 Dynamic Bus Frequency

[andreapicaro]

Hello Anthony, have you ever solved this? I have exactly same problem. Thankyou

[markguidarelli]

Hi Andrea.  Tony has moved on and I am working on this platform now.  In our case the issue causing the black screen was the drivers/video/mxc/mxc_ipuv3_fb.c: "wait_for_completion_timeout(&mxc_fbi->flip_complete, ..." not completing in time.  We had worked around this issue by resetting the channels (performing a mxcfb_blank(FB_BLANK_POWERDOWN) followed by a mxcfb_blank(FB_BLANK_UNBLANK).  This appeared to work well until recently.  I am observing some cases where this recover mechanism no longer works.

Have you made any progress with your issue?

Mark

[allan_dai]

Hello Mark

Have you got the root cause and fix solution for this issue, I met the same issue here...any suggestion?

[igorpadykov]

Hi Tony

limitation is from IPU bandwidth, so MMDC bus load  may affect it but

indirectly. Are you using IPU IC, try to avoid its usage.

In general one can apply to NXP Professional Services for more extended support:

http://www.nxp.com/support/nxp-professional-services:PROFESSIONAL-SERVICE

Best regards
igor

[markguidarelli]

Hi Igor.  

Tony has moved on and I am working on this platform now.  In our case the issue causing the black screen was the drivers/video/mxc/mxc_ipuv3_fb.c: "wait_for_completion_timeout(&mxc_fbi->flip_complete, ..." not completing in time.  We had worked around this issue by resetting the channels (performing a mxcfb_blank(FB_BLANK_POWERDOWN) followed by a mxcfb_blank(FB_BLANK_UNBLANK).  This appeared to work well until recently.  I am observing some cases where this recover mechanism no longer works.

What could cause flip_complete to continuously timeout?  Is there a way to reset the IPU short of a power cycle?  I am observing this issue on some board set but not others.

Mark 

[igorpadykov]

Hi Tony

bit IPU_INT_STAT_10 is set when there's data starvation, so
data that should be sent to display is not available in IPUs internal FIFO.

One can try the 'dmfc=3' kernel boot-up command option. This would increase
display FIFO size.

Most probable cause for this issue is huge load on the DDR memory bus.
One can adjust this by configuring IPU priorities in IOMUXC_GPR6

or NIC-301 (Chapter 46 Network Interconnect Bus System (NIC-301) i.MX6DL RM) 

or by optimizing custom application.

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

[tdefeo23]

Hi Igor,

Thanks for the quick reply.

I checked IOMUXC_GPR4, IOMUXC_GPR6, IOMUXC_GPR7, they are set to:

IOMUXC_GPR4: 0xf00000cf

IOMUXC_GPR6: 0x007f007f

IOMUXC_GPR7: 0x007f007f

This appears to be correct according to this post:

https://community.nxp.com/thread/340455 

I'll try the dmfc=3 option, although in looking at the code all this does is change the size of the fifo for channel 27 from 128*4 to 256*4. I believe channel 27 is the auxiliary plane, which, if I'm understanding correctly, we're not using. Can you elaborate on the difference between channel 23 (Main plane) and channel 27 (auxiliary plane)?

Our application is in a repeating sequence, so I would think the DDR bandwidth usage would be fairly consistent, while these errors are relatively rare!

Is there any way for me to measure the DDR bus utilization to see if/when it is high?

Thanks,

Tony

[igorpadykov]

Hi Tony

please check memory profile tool mmdc2 mentioned on
https://community.nxp.com/message/829055?commentID=829055#comment-829055

Best regards
igor

[tdefeo23]

Hi Igor,

Thanks for the info, the mmdc2 tool is very useful. I was able to determine that our overall bus load peaks at close to 90%, and that the GPU3D is responsible for the majority of it:

MMDC GPU3D                                                                      
                                                                                
MMDC new Profiling results:                                                     
***********************                                                         
Measure time: 1002ms                                                            
Total cycles count: 396659724                                                   
Busy cycles count: 345300411                                                    
Read accesses count: 19142496                                                   
Write accesses count: 11052216                                                  
Read bytes count: 686131512                                                     
Write bytes count: 560745984                                                    
Avg. Read burst size: 35                                                        
Avg. Write burst size: 50                                                       
Read: 653.04 MB/s /  Write: 533.70 MB/s  Total: 1186.74 MB/s                    
Utilization: 22%                                                                
Overall Bus Load: 87%                                                           
Bytes Access: 41                  

Here's what's interesting though: As a test, I constructed a test scene with a lot of alpha to render. With this scene, I was able to push the overall bus load to a solid 99%. I expected to see a lot of the IPU_INT_STAT_10  = 0x00080000 errors with the bus pegged like this, but I didn't see any! Any ideas why this would be?

FYI, setting dmfc to 3 didn't help, I am still sporadically getting the IPU_INT_STAT_10 & IPU_INT_STAT_5 errors. I will certainly try to optimize our GPU usage to reduce the bus load, but I'm sceptical, since pushing it to the limit did not reproduce the issue. Any insight you might have would be greatly appreciated!

Thanks,

Tony

MMDC GPU3D                                                                      
                                                                                
MMDC new Profiling results:                                                     
***********************                                                         
Measure time: 1000ms                                                            
Total cycles count: 396066594                                                   
Busy cycles count: 395130146                                                    
Read accesses count: 28864881                                                   
Write accesses count: 11611819                                                  
Read bytes count: 969271552                                                     
Write bytes count: 691106496                                                    
Avg. Read burst size: 33                                                        
Avg. Write burst size: 59                                                       
Read: 924.37 MB/s /  Write: 659.09 MB/s  Total: 1583.46 MB/s                    
Utilization: 26%                                                                
Overall Bus Load: 99%                                                           
Bytes Access: 41             

[tdefeo23]

Hi Igor,

I have significantly reduced our application's MMDC bus load from a peak of about 87% down to a peak of about 60%. This was done by lowering our gpu rendering framerate down to 30fps. Note that the ipu video refresh rate is still 60hz.

Unfortunately I am still getting the warnings:

imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_10 = 0x00080000
imx-ipuv3 2400000.ipu: IPU Warning - IPU_INT_STAT_5 = 0x00800000

As noted earlier, as a test, I adjusted our application to utilize 99% of the MMDC bus, and it did not seem to affect the frequency of the IPU warnings. This has led me to believe that there might be something else going on, or at least something that I'm not understanding about the bus utilization and it's affect on the IPU.

Any further help or ideas you may have would be greatly appreciated! This is a show stopper for us, and we need to find some kind of a resolution to this problem.

Thanks!

Tony