GPU hang on sabre running Android 5.1.1

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

GPU hang on sabre running Android 5.1.1

3,596 Views
neerajsharma
Contributor III

Hi,

 

We have a custom platform based on SabreSD imx6q reference running Lollipop L5.1.1_2.0.0 GA (LMY47V) [kernel:3.14.38-138123]

We have been seeing a problem wherein the gpu and the display would just hang randomly.

System remains in the same state unless rebooted.

 

i have figured out that for some reason using chrome, with random events is the fastest way to reproduce this problem.

[Running "monkey --throttle 100 -p com.android.chrome 1000000" infinitely]

 

Tried with Android 5.1.1 2.0.0-ga-rc4 on sabre and i am able to reproduce the problem.

It takes some time before the screen is unresponsive. Android shell is responsive underneath.

Attached are the video and logcat for the problem happening on sabre board 

 

Is this a known issue or could you help us in fixing this ?

 

Have this posted here already. Was advised to post on this forum.

 

HISTORY:

Already tried sabresd_6dq-eng 5.1.1 2.1.0-ga-rc3 and problem happens on this build also.

Tried  galcore.powerManagement=0, causes the problem every time on system resume after suspend.

Tried NXP provided patch to disable GPU's power management feature. Issue still happens

 

We have seen the problem randomly also, while not using chrome.

Testing it on chrome package with monkey is helping us to reproduce this more often.

[Running "monkey --throttle 100 -p com.android.chrome 1000000" infinitely]

 

The version of chrome that is reproducing the problem for us is version 57.0.2987.132.

This version does not have the "Merge Tab" setting.

 

Other source to get it from: The Open GApps Project

On our platform we have seen signatures like:

 

03-20 13:49:40.478 156 156 I kernel : <6>[36394.351361] fence timeout on [d01dda00] after 3000ms
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] objs:
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] --------------
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] viv timeline viv_sync
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt signaled@2623.897486
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt signaled@33633.696669
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt signaled@33644.294139
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt signaled@33644.889483
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt active
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt active
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] pt active
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552]
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] viv timeline viv_sync
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552]
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] viv timeline viv_sync
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552]
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] fences:
03-20 13:49:40.478 156 156 W kernel : <4>[36394.351552] --------------
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-62573: signaled
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt signaled@2623.897486
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-64567: signaled
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt signaled@33633.696669
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-64577: signaled
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt signaled@33644.294139
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-64622: signaled
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt signaled@33644.889483
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-64624: active
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt active
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-64625: active
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt active
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] [00000000] viv sync_fence-64626: active
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645] viv timeline_pt active
03-20 13:49:40.479 156 156 W kernel : <4>[36394.351645]
03-20 13:49:40.480 163 176 E Fence : Throttling EGL Production: fence 36 didn't signal in 3000 ms

 

03-20 13:50:29.885 156 156 W kernel : <4>[36443.761373] [galcore]: GPU[0] hang, automatic recovery.
03-20 13:50:29.885 156 156 W kernel : <4>[36443.761450] [galcore]: recovery done
03-20 13:50:34.880 163 933 W SurfaceFlinger: setTransactionState timed out!
03-20 13:50:34.884 163 176 W SurfaceFlinger: setTransactionState timed out!
Looks similar to: IMX AXI BUS ERROR, GPU hang 

Original Attachment has been moved to: sabreLog.txt.zip

Labels (3)
Tags (3)
0 Kudos
10 Replies

2,115 Views
neerajsharma
Contributor III

Hi Igor,

I tested this on "Freescale/sabresd_6dq/sabresd_6dq:7.1.1/1.0.0-ga-rc10/20170227:user/release-keys" android N release.

And it fails on Android N release also.

This is critical as device never recovers from the problem

Bugreport attached.

0 Kudos

2,115 Views
igorpadykov
NXP Employee
NXP Employee
0 Kudos

2,115 Views
neerajsharma
Contributor III

Hi Igor,

If you missed it in the earlier comment, and also could not determine it from the bugreport

 - We tried the demo bsp for Android N - 7.1.1/1.0.0-ga-rc10 on SABRE SD board and it hangs.

Is NXP not able to reproduce the problem ?

0 Kudos

2,115 Views
igorpadykov
NXP Employee
NXP Employee

Hi neeraj

could you try other board, we did not have other reports that it not work.

Best regards
igor

0 Kudos

2,115 Views
neerajsharma
Contributor III

We have been seeing the same problem on different sabre boards and each of our device based on Sabre platform also. 

Also, it would be helpful to mention again here that, it was already posted here and was being actively worked by NXP support team. So the related NXP team has already seen the problem on their end.

 

Thanks

0 Kudos

2,115 Views
john_smith
Contributor II

Hello!

Did you find any solution to this problem? We have similar problems with GPU after resuming from suspend. We've also asked NXP with no result (you've already posted our thread link - IMX AXI BUS ERROR, GPU hang )

0 Kudos

2,115 Views
igorpadykov
NXP Employee
NXP Employee

Hi neeraj

 

seems there is some misunderstanding.

If you already submitted issue on that link, this means

that imx application team is already working on that issue

(mentioned link is internal confidential and closed for external customers).

I would suggest to continue work on that issue with contacts

from that link. Sorry I have not access to them.

 

Best regards
igor

0 Kudos

2,115 Views
igorpadykov
NXP Employee
NXP Employee

Hi neeraj

L3.14.38 is very old kernel and since then lot of gpu errata were fixed,

please try latest described on

i.MX 6 / i.MX 7 Series Software and Development Tool|NXP 

Fixed errata are described in Release Notes document from documentation package

which can be found on that link

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------

0 Kudos

2,115 Views
neerajsharma
Contributor III

Hi Igor,

Moving to newer versions of lollipop is not an option for our platform.

Its a huge effort. We need to fix this on Lollipop 5.1.1.

Do you have any updates or patches for Lollipop 5.1.1 ?

Thanks

0 Kudos

2,115 Views
igorpadykov
NXP Employee
NXP Employee

Hi neeraj

please check available patches on

i.MX 6 / i.MX 7 Series Software and Development Tool|NXP 

Best regards
igor

0 Kudos