i.MX 8M Min randomly loses GPIO (sysfs) functionality with "Resource temporarily unavailable"

[VincasD]

We got some "panel PC" from China what has NXP i.MX 8M Mini, with system image running:

"Linux imx8mmevk 5.10.72-lts-5.10.y+g2a23c0cdbb9b #1 SMP PREEMPT Tue Oct 25 12:05:04 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux"

with toolchain named:

"fsl-imx-wayland-glibc-x86_64-core-image-base-cortexa53-crypto-imx8mmevk-toolchain-5.10-hardknott"

GPIO is accessed via sysfs, but if I run this test script:

while true;  
do  
   for I in {04..11};
   do  
       cat "/sys/class/gpio/gpio5${I}/value";
   done;
done

After some (random) time I am starting to get errors, losing ability to read GPIO values:

cat: /sys/class/gpio/gpio504/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio505/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio506/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio507/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio508/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio509/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio510/value: Resource temporarily unavailable
cat: /sys/class/gpio/gpio511/value: Resource temporarily unavailable

It seems that to fix this issue I have to turn off device completely, hot system reboot does not help.

Not sure what to thing about it, could it be some buggy custom Chinese kernel code, bug in NXP kernel, or just bad board (electronics) design so that GPIO goes haywire after some random glitch?

Thanks!

[smithcoba]

I would like to use the "ulimit -a" to check the system. 

[VincasD]

# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 5218
max locked memory           (kbytes, -l) 64
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 5218
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

[smithcoba]

ulimit -d unlimited
ulimit -u unlimited

[VincasD]

Well, -d was already unlimited. Set -u ulimited, but did not helped. Unable to read sysfs gpio values after some time of running.

[Sanket_Parekh]

Hi @VincasD 

I hope you are doing well

My sincere apology for the delayed response.

Could you please share the device tree file you are using for further debugging the issue?

Also please confirm that if you were able to read these gpio values from the terminal without using the for loop in the script or not.

Thanks & Regards

Sanket Parekh

[VincasD]

Could you please check that attached .dtb file Sanket?

Thanks!

[Sanket_Parekh]

Hi @VincasD,

I hope you are doing well

Please look into the below information.

The "resource temporarily unavailable" error can be caused due to several reasons.

->It may mean some exact resource or process tries to access that pin. Please check the device tree if the same IO expander pin has been assigned to any other resource or not.

->The fault may be caused due to the bandwidth, power, or a buggy driver. Please check the power/voltage sufficiently configured.  Please try with the latest kernel as well.

->Any dependency i.e. module, driver, library, etc. has been removed.

->The "resource temporarily unavailable" error is related to error -11(EAGAIN) which can occur if attempted for a non-blocking read. As this error is coming randomly, to debug the cause of the error one can use Strace.

I hope this information helps you.

Thanks & Regards,
Sanket Parekh

[VincasD]

I've attached .dtb I've found in /run/media/mmcblk2p5/imx8mm-evk.dtb

[Sanket_Parekh]

Hi @VincasD,

I hope you are doing well

To read/write the gpio using this sysfs interface is being deprecated. We recommend you use the 'gpiod' method. One can read the value of gpio using the command gpioget.

I have tested the reading and writing the gpio in the loop for a long time using the script you are implementing by using the above command. Here we were unable to reproduce the error. It worked fine for us.
In our case, we used board IMX8M mini, BSP version - 6.1.1.

Thanks & Regards,
Sanket Parekh

[VincasD]

I've found code that enables sysfs - removed id, and now we are using gpiod (VTGLinuxIO is our daemon):

# ./gpioinfo gpiochip5
gpiochip5 - 16 lines:
       line   0:      unnamed       unused   input  active-high  
       line   1:      unnamed       unused   input  active-high  
       line   2:      unnamed       unused   input  active-high  
       line   3:      unnamed       unused   input  active-high  
       line   4:      unnamed       unused   input  active-high  
       line   5:      unnamed       unused   input  active-high  
       line   6:      unnamed       unused   input  active-high  
       line   7:      unnamed       unused   input  active-high  
       line   8:      unnamed "VTGLinuxIO"   input  active-high [used]
       line   9:      unnamed "VTGLinuxIO"   input  active-high [used]
       line  10:      unnamed "VTGLinuxIO"   input  active-high [used]
       line  11:      unnamed "VTGLinuxIO"   input  active-high [used]
       line  12:      unnamed "VTGLinuxIO"   input  active-high [used]
       line  13:      unnamed "VTGLinuxIO"   input  active-high [used]
       line  14:      unnamed "VTGLinuxIO"   input  active-high [used]
       line  15:      unnamed "VTGLinuxIO"   input  active-high [used]

But even with gpiod we have issues: time to time line gets unreadable (and we assert and intentionaly abort), and we see "kernel: pca953x 2-0020: failed reading register" error printed in journal:

Sep 05 13:40:40 imx8mmevk VTGGPIOService[6007]: Failed to open LinuxIO device: "Resource temporarily unavailable (system:11)"
Sep 05 13:40:40 imx8mmevk kernel: pca953x 2-0020: failed reading register
Sep 05 13:40:40 imx8mmevk kernel[276]: [ 8292.357291] pca953x 2-0020: failed reading register
Sep 05 13:40:40 imx8mmevk systemd[1]: VTGGPIOService.service: Main process exited, code=exited, status=1/FAILURE

Maybe it's wrong driver, or too old driver, etc? Any ideas?

[Sanket_Parekh]

Hi @VincasD,

I hope you are doing well

The cause of this issue could be the long-running of the particular task. 

The maximum user processes (nproc) limit on Linux counts the number of threads within all processes that can exist for a given user. 

When this limit is reached, no new process can be created, which causes the "Resource temporarily unavailable" error. It is usually caused by hanging processes that have been running for days. The solution is to kill unnecessary processes. 

Please use the "$ulimit -a" command for details related to nproc limit.

I hope this information helps you.

Thanks & Regards,
Sanket Parekh

[VincasD]

Main issue I believe it somethig's wrong with GPIO exntender:

pca953x 2-0020: failed reading register

[VincasD]

No it's really not the processes issue, same POSIX error code might be use AND for process limits, yes, but I get these errors randomy before GPIO fully hangs. There's less than 200 threads on the system, while:

max user processes                  (-u) 1312

After night I can run new processes, etc, all works, bug GPIO is completely stuck until cold restart:

# for PIN in {8..15}; do echo "Reading gpiochip5 pin $PIN"; ./gpioget gpiochip5 $PIN; done
Reading gpiochip5 pin 8
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 9
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 10
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 11
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 12
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 13
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 14
./gpioget: error reading GPIO values: Resource temporarily unavailable
Reading gpiochip5 pin 15
./gpioget: error reading GPIO values: Resource temporarily unavailable

[VincasD]

Could you please share the device tree file you are using for further debugging the issue?

Where do I find it?

Also please confirm that if you were able to read these gpio values from the terminal without using the for loop in the script or not.

We do read values successfully in our "gpio service" application (using QFile), and I do see values changing in /sys/kernel/debug/gpio. Simply after some time of running I can no longer read values - does not matter is it our Qt application or `cat` in terminal.

[Sanket_Parekh]

Hi @VincasD,

I hope you are doing well

This error may occur when the resources we are trying to access are busy. 

We suspect that some other drivers might access these GPIOs when you are running your test script.

Please verify the same in the device tree also.

Thanks & Regards,

Sanket Parekh

[VincasD]

I've built gpiod utilities and I see this:

# ./gpioinfo gpiochip5
gpiochip5 - 16 lines:
       line   0:      unnamed       unused   input  active-high  
       line   1:      unnamed       unused   input  active-high  
       line   2:      unnamed       unused   input  active-high  
       line   3:      unnamed       unused   input  active-high  
       line   4:      unnamed       unused   input  active-high  
       line   5:      unnamed       unused   input  active-high  
       line   6:      unnamed       unused   input  active-high  
       line   7:      unnamed       unused   input  active-high  
       line   8:      unnamed      "sysfs"   input  active-high [used]
       line   9:      unnamed      "sysfs"   input  active-high [used]
       line  10:      unnamed      "sysfs"   input  active-high [used]
       line  11:      unnamed      "sysfs"   input  active-high [used]
       line  12:      unnamed      "sysfs"   input  active-high [used]
       line  13:      unnamed      "sysfs"   input  active-high [used]
       line  14:      unnamed      "sysfs"   input  active-high [used]
       line  15:      unnamed      "sysfs"   input  active-high [used]

I see [used] inputs market as sysfs, maybe that's the ones I am using. But it's [used] even while there is no my script, nor my application running that would access GPIO, so could this mean as you said - that something else are using them?

Not sure about "checking device tree" though..?