I am writing a pcie driver for LS1021a to get data from the pcie card.
In driver, memory is allocated by:
virt_mem = dma_alloc_coherent(&dev, size, &dma_mem, GFP_KERNEL);
Everything works fine, I get virt_mem for kernel and dma_mem for DMA.
This memory is mmapped to user application by:
remap_pfn_range(vma, vma->vm_start, virt_to_phys(dma_mem)>>PAGE_SHIFT, vma->vm_end-vma->vm_start, vma->vm_page_prot);
The mmap is successful in user application.
During self test, the driver (instead of DMA) writes value to this memory, and user application reads it out. Driver writes 1 -> being read, writes 2 ->being read, ...... user application is supposed to get data in right sequence, 1 2 3 4 ...
Issue: User application doesn't get the right sequence of data. Sometimes it gets right number, sometimes gets old value.
The memory should be non-cacheable or coherent. The driver written data should be visible to user application. I am thinking it might be due to something like write-buffer somewhere. I tried using mb() or __asm("dsb") after each data write. But no help, application still gets wrong data sequence.
I then tried allocating the memory using kmalloc (instead of dma_alloc_coherent), and use remap_pfn_range again for mmap. Everything works fine this time. I believe because it is cacheable memory, the driver-write data is correctly read by application.
Can't figure out what goes wrong. Is remap_pfn_range the right choice for dma_alloc_coherent memory ? why mb() seems no help at all ?