AnsweredAssumed Answered

IMX6 & BSP 4.1.15: Memory leak?

Question asked by grim on Mar 27, 2019
Latest reply on Apr 8, 2019 by grim

We have a product that uses the i.mx6 solo.

We've recently needed to upgrade from the 3.14.28 BSP to 4.1.15.

 

When our product is active and idle, it just plays videos until somebody touches the screen.

Initially, with 3.14.28, we experienced memory losses and were able to fix them by

  1. Switching from gstreamer .10 to gstreamer 1.0
  2. Periodically clearing the cache buffers (echo 3 > /proc/sys/vm/drop_caches)

Once we made these changes, the system would run indefinitely without running out of memory.

 

We made the switch to 4.1.15.

We're still using gstreamer 1.0 and periodically clearing the cache buffers but the system eventually runs out of memory and our application crashes and burns.

 

On one of the systems that crashed (after 5 days) I captured a couple of things that I thought were unusual...

I ran the free command and observed the output...

 

  free


  total        used        free      shared  buff/cache   available
  Mem:        1026448      222500      220688        1916      583260      186664
  Swap:             0           0           0

 

Then I cleared the cache...

 

echo 3 > /proc/sys/vm/drop_caches

 

Run the free command again...

 

         free


         total        used        free      shared  buff/cache   available
         Mem:        1026448      222472      221680         488      582296      187884
         Swap:             0           0           0

 

"Clearing" the cache freed up VERY little memory.

 

 

 

That system was eventually rebooted and is running again, waiting for it to crash again.

 

On another system, I made some observations while the system was running.

I went and captured "/proc/meminfo" periodically and observed it.

 

Here's the output from a machine that's only been running for a few hours...

cat /proc/meminfo


MemTotal:        1026404 kB
MemFree:          684864 kB
MemAvailable:     739740 kB
Buffers:            1268 kB
Cached:            63176 kB
SwapCached:            0 kB
Active:            63948 kB
Inactive:          46632 kB
Active(anon):      46120 kB
Inactive(anon):      468 kB
Active(file):      17828 kB
Inactive(file):    46164 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:        1026404 kB
LowFree:          684864 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         46116 kB
Mapped:            53976 kB
Shmem:               472 kB
Slab:              42312 kB
SReclaimable:       3032 kB
SUnreclaim:        39280 kB
KernelStack:         752 kB
PageTables:         1152 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      513200 kB
Committed_AS:     308276 kB
VmallocTotal:    1024000 kB
VmallocUsed:        8784 kB
VmallocChunk:     824796 kB
CmaTotal:         327680 kB
CmaFree:          155088 kB

 

Observing /proc/meminfo an hour or so later...

 

 

cat /proc/meminfo


MemTotal:        1026404 kB
MemFree:          644440 kB
MemAvailable:     729676 kB
Buffers:            2376 kB
Cached:            92420 kB
SwapCached:            0 kB
Active:           104140 kB
Inactive:          37532 kB
Active(anon):      46868 kB
Inactive(anon):      460 kB
Active(file):      57272 kB
Inactive(file):    37072 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:        1026404 kB
LowFree:          644440 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                80 kB
Writeback:             0 kB
AnonPages:         46856 kB
Mapped:            53976 kB
Shmem:               472 kB
Slab:              48476 kB
SReclaimable:       3048 kB
SUnreclaim:        45428 kB
KernelStack:         752 kB
PageTables:         1156 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      513200 kB
Committed_AS:     308276 kB
VmallocTotal:    1024000 kB
VmallocUsed:        8784 kB
VmallocChunk:     826636 kB
CmaTotal:         327680 kB
CmaFree:          155076 kB

 

What I've noticed is that SUnreclaim continues to grow. I have yet to observe how high it gets before the system crashes but it is definitely growing.

 

Is this a known issue? Are there any known workarounds/patches/fixes? I see that kernel 4.9 is now available.... will I experience the same issue with 4.9?

 

I'm in the process of adding "slabtop" support to my system so that I can try to get a better idea of what's going on.

 

Any tips or pointers would be greatly appreciated.

Outcomes