DMA related crash when capturing hi-res video while storing to SSD

[ofer_livny]

Hi all

I'm using an iMX6 processor in a setup that includes Sabrelite board, a custom MIPI-CSI 5MP camera, and an SSD SATA hard drive.

I'm running the latest linux 3.0.35 downloaded from the extranet.

I have my own custom camera driver which sits on the existing V4L2 infrastructure.

I am able to successfully run my own tests that use the V4L2 interface for capturing frames, and store them to disk or send them via TCP/IP.

My problems begin when I'm trying to capture a 5MP RGB888 7fps video and store over 1GB of data directly to an SSD (ext4 fs) HD.

The details:

- I am able to capture 5MP video and save it to an SD card. Then I get low frame rate since SD card is obviously much slower than an SSD.

- However when attempting to store the video to an SSD HD I get the following results:

  * If I capture over 1GB using my test, stop capturing, and then attempt to capture again, I get a "page allocation failure" error (attached).

  * In different attempts, the backtrace of the error is sometimes different, however usually V4L2 functions are somewhere in the stack.

  * Sometimes the error appears while storing over 1GB during the first attempt to capture.

  * When I capture less than 1GB, and then restart the test - all goes well.

- My machine has 1GB RAM. This makes me wonder if the 1GB "virtual threshold" of my crashes might be related to the memory caching linux does automatically for files using free RAM.

- I tried more than one SSD drive, tried replacing Sabrelite board, camera module, etc... It doesn't appear to be hardware related.

As I'm not so familiar with the DMA allocation mechanism, and I'm not so familiar with driver differences between SSD and SD card.

My next steps will be to take the simplest v4l capture test and add recording to it. Perhaps it'll work and I'll be able to understand what went wrong by the code differences. However, since I know my code is already very similar to the mxc_v4l2_capture unit test and since capturing work well when I'm NOT attempting to record more than 1GB HD - I'm not so sure it'll help.

Can anyone help me understand what goes wrong?

Original Attachment has been moved to: crash.log.zip

[agust]

Hi,

from my understanding the problem you observe is the memory fragmentation. The mxc_v4l2_capture

kernel driver dynamically allocates contiguous DMA buffers for image data and frees them quite often.

The driver allocates 10 buffers for its queue and also a dummy buffer. You seem to capture 2592x1944

frames, so the buffers are quite big. The kernel log contains some useful info, you can see that when the

problem appeared, the driver tried to allocate 15118336 Bytes of contiguous space, but due to the

memory fragmentation the allocator cannot satisfy this request, there is no such big block of contiguous

memory any more:

DMA: 127*4kB 93*8kB 58*16kB 55*32kB 49*64kB 45*128kB 45*256kB 38*512kB 23*1024kB 16*2048kB 3*4096kB 2*8192kB 0*16384kB 0*32768kB

The biggest contiguous block available is 8192kB which is not enough.

You do not see this problem when SATA driver is not in use, but the problem may be still there. If the memory

becomes fragmented enough, the issue can pop up. SATA driver allocates memory too, this may result in

quicker fragmentation.

To solve this issue the kernel capture driver will need to be fixed to pre-allocate the capture buffers at

driver's init time and work with pre-allocated buffers, I'm afraid.

[ofer_livny]

Thanks agust,

Helpful information there :smileyhappy:

I'll try and pre-allocate those buffers inside the v4l module, however I prefer a different solution since the video4linux streaming  is not the only component using DMA. After my test failed with the DMA page allocation error, I tried to play sound using ALSA (aplay) - and got a similar error.

I prefer finding a robust solution that allows me to continue using my machine after capturing the video to HD.

Can you suggest ways to release DMA buffers? I tried removing the V4L2 module, disconnecting the SSD HD, and the error message still appears. Aren't these buffers supposed to be freed automatically ?

Like I mentioned - when I tried the same test, but with an SD card instead of SSD HD, everything went ok. I suppose the only two differences are A. speed, B. different handling by the kernel.  So doesn't SD card writes use DMA as well?

[agust]

Hi,

the capture driver releases its DMA buffers, so they will be freed automatically when the last user closes the

v4l video device. There maybe enough free memory but it doesn't help since this memory becomes very fragmented

after some time. In other words there are not enough adjacent memory pages that form big blocks of contiguous memory.

SD card driver uses DMA too, but it seems this driver uses scatter gather buffers and doesn't participate to the

memory fragmentation very much.

[ofer_livny]

Hi again,

While I'm occupied with re-writing the video capture driver for pre-allocating the DMA memory, I keep feeling that I'm missing something here...

I tested a simple HD write access (just wrote random data into a file, just after kernel done loading, without any video capture driver loaded), and after that I couldn't start capturing because of the same reason - not enough continuous memory for DMA allocation.

The question that bothers me is that - will I have the same problem if I run the same tests on an x86 desktop machine? (writing a large file on an SSD HD, and then trying to allocate DMA by trying to capture video)

I understand more or less how this DMA allocation works in the video capture drivers, however I'm not so familiar with the disk access drivers who are responsible for the fragmentation of the free memory.

I keep looking the net for a way to release all the memory that is being allocated during my simple file-writing test, and can't find any.

Sounds like a problematic behavior of the HD/SATA/DMA modules to me, and not the video modules, unless I am really missing the point here...

Another note, just to let you know that unlike what I mentioned in my previous message, the same problem occurs after writing large amount of data to SD card as well - even then memory is too fragmented for new 15MB buffer allocations.

Any thoughts?

[agust · Accepted solution]

Hi,

on x86 the be problem may appear too, but x86 is really not comparable here. It is different architecture with tons of different hardware/controllers/drivers.

Try the attached patch for 3.0.35 kernel. It is supposed to pre-allocate 128 MiB pool on boot time and to assign it exclusively to the mxc capture driver. When this driver will allocate its DMA buffers, the allocation should happen from the reserved pool. The memory fragmentation shouldn't affect this area since it is exclusively assigned to the capture device and only mxc driver can allocate from this pool. Note that this patch is only for capture driver and only for testing, it may be incomplete. Other IPU submodules (frame buffer, overlay, viewfinder, etc.) may be affected by fragmentation problem, too.

[ofer_livny]

Thanks Agust!

Seems to be working just out of the box :smileyhappy:

I'll give it some more testing, but hopefully this is it...

Ofer