rte_eth_rx_burst -> returning same mbuff pointer twice consecutively

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

rte_eth_rx_burst -> returning same mbuff pointer twice consecutively

277 Views
Chidananda22
Contributor III

From my application calling rte_eth_rx_burst 

checking sometimes same mbuff coming twice . Checked code buffer not freeing . With out mbuff free what could be the scenario that same mbuff pointer received twice .

Labels (1)
0 Kudos
Reply
5 Replies

270 Views
yipingwang
NXP TechSupport
NXP TechSupport

Without an explicit mbuf free in your code, the most likely scenarios are that the mbuf is still being effectively returned/reused through another path in the application or shared unsafely across threads/cores, or that a special driver/path-specific buffer handling issue is corrupting lifecycle state.

  • Cross-core/shared mbuf ownership issue. One documented case used one CPU for RX and many worker CPUs consuming mbufs from a ring; the mbuf address was then seen reused/overwritten while still referenced elsewhere. NXP specifically asked whether RTE_MBUF_REFCNT_ATOMIC was enabled for multicore access, and the customer said yes, but the issue was still treated as application-level ownership/lifetime handling.,
  • Duplicate release or hidden return path in the application pipeline. NXP’s guidance in that case was to instrument buffer allocate/release/packet-send actions to see whether the same address was being released back to the buffer pool more than once.
  • PMD/driver returning malformed mbufs to pool in special paths. There is a separate DPAA2 PMD case involving chained mbufs where mbufs returned to the mempool appeared corrupted ( next not cleared), causing sanity-check failures on later allocation. That is not the same symptom as your report, but it does show that special buffer handling paths can lead to later reuse/corruption symptoms.
  • Same source and destination mbuf in crypto path. In a DPAA security/IPsec case, NXP noted that if sym->m_dst is NULL , then m_src and m_dst use the same mbuf. That can make packet contents appear rewritten “in place” under load, even though the pointer identity did not change.

 

0 Kudos
Reply

261 Views
Chidananda22
Contributor III

thanks @yipingwang  for your quick reply and suggestions. from code i am not to figure out more . Any DPDK debug flag can help ? 

0 Kudos
Reply

235 Views
yipingwang
NXP TechSupport
NXP TechSupport
  1. Enable DPDK log levels (first thing to try)

Run your app with:

--log-level=8

What you’ll see:

  • RX descriptor handling
  • buffer allocation/recycling
  • PMD behavior
  1. Enable mbuf debug checking (very useful)

Rebuild DPDK with:

CONFIG_RTE_LIBRTE_MBUF_DEBUG=y

This will:

  • Validate mbuf consistency
  • Catch:
    • corrupted metadata
    • reused mbuf incorrectly
    • invalid refcnt

 

  1. Enable mempool debug

      CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG=y

Helps detect:

  • double allocation
  • cache corruption
  • same buffer returned twice from mempool

 

  1. Poison/free tracking (EXTREMELY useful)

You can enable memory overwrite checking:

CONFIG_RTE_MALLOC_DEBUG=y

and

--mp-alloc=debug

This helps detect:

  • overwrite before/after mbuf
  • hidden memory corruption

 

  1. Quick isolation trick (VERY effective)

Run

testpmd -- -i

Then:

set fwd rxonly

start

 

If duplication disappears:

  • your application bug

If still happens:

  • PMD / HW / queue config issue
0 Kudos
Reply

225 Views
Chidananda22
Contributor III

Thanks @yipingwang  for Quick reply . I will try and update you .

One observation every time multiple mbuf ( same address coming ) getting below prints in screen

 

fslmc: dpaa2_get_qbman_swp(): New Portal 0x17ffd61c0 (5) affined thread - 1953
fslmc: dpaa2_configure_stashing(): Portal= 5 CPU= 10 SDEST= 5
fslmc: DPAA Portal=0x17ffd61c0 (5) is affined to thread 1953

0 Kudos
Reply

169 Views
yipingwang
NXP TechSupport
NXP TechSupport
Those fslmc / dpaa2 portal affined thread logs strongly indicate portal (QBMan SWP) is being re-attached to a different thread/core, which can break ownership model → leading to mbuf reuse / duplication symptoms.
0 Kudos
Reply
%3CLINGO-SUB%20id%3D%22lingo-sub-2379408%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3Erte_eth_rx_burst%20-%26gt%3B%20returning%20same%20mbuff%20pointer%20twice%20consecutively%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2379408%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3CP%3EFrom%20my%20application%20calling%20rte_eth_rx_burst%26nbsp%3B%3C%2FP%3E%3CP%3Echecking%20sometimes%20same%20mbuff%20coming%20twice%20.%20Checked%20code%20buffer%20not%20freeing%20.%20With%20out%20mbuff%20free%20what%20could%20be%20the%20scenario%20that%20same%20mbuff%20pointer%20received%20twice%20.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2379408%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3CLINGO-LABEL%3EQorIQ%20LS1%20Devices%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2379796%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%20translate%3D%22no%22%3ERe%3A%20rte_eth_rx_burst%20-%26gt%3B%20returning%20same%20mbuff%20pointer%20twice%20consecutively%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2379796%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3COL%3E%0A%3CLI%3E%3CSTRONG%3EEnable%20DPDK%20log%20levels%20(first%20thing%20to%20try)%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3ERun%20your%20app%20with%3A%3C%2FP%3E%0A%3CP%3E--log-level%3D8%3C%2FP%3E%0A%3CP%3EWhat%20you%E2%80%99ll%20see%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3ERX%20descriptor%20handling%3C%2FLI%3E%0A%3CLI%3Ebuffer%20allocation%2Frecycling%3C%2FLI%3E%0A%3CLI%3EPMD%20behavior%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3COL%20start%3D%222%22%3E%0A%3CLI%3E%3CSTRONG%3EEnable%20mbuf%20debug%20checking%20(very%20useful)%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3ERebuild%20DPDK%20with%3A%3C%2FP%3E%0A%3CP%3ECONFIG_RTE_LIBRTE_MBUF_DEBUG%3Dy%3C%2FP%3E%0A%3CP%3EThis%20will%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EValidate%20mbuf%20consistency%3C%2FLI%3E%0A%3CLI%3ECatch%3A%0A%3CUL%3E%0A%3CLI%3Ecorrupted%20metadata%3C%2FLI%3E%0A%3CLI%3Ereused%20mbuf%20incorrectly%3C%2FLI%3E%0A%3CLI%3Einvalid%20refcnt%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CBR%20%2F%3E%0A%3COL%20start%3D%223%22%3E%0A%3CLI%3E%3CSTRONG%3EEnable%20mempool%20debug%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%26nbsp%3B%20CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG%3Dy%3C%2FP%3E%0A%3CP%3EHelps%20detect%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3Edouble%20allocation%3C%2FLI%3E%0A%3CLI%3Ecache%20corruption%3C%2FLI%3E%0A%3CLI%3Esame%20buffer%20returned%20twice%20from%20mempool%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CBR%20%2F%3E%0A%3COL%20start%3D%224%22%3E%0A%3CLI%3E%3CSTRONG%3EPoison%2Ffree%20tracking%20(EXTREMELY%20useful)%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EYou%20can%20enable%20memory%20overwrite%20checking%3A%3C%2FP%3E%0A%3CP%3ECONFIG_RTE_MALLOC_DEBUG%3Dy%3C%2FP%3E%0A%3CP%3Eand%3C%2FP%3E%0A%3CP%3E--mp-alloc%3Ddebug%3C%2FP%3E%0A%3CP%3EThis%20helps%20detect%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3Eoverwrite%20before%2Fafter%20mbuf%3C%2FLI%3E%0A%3CLI%3Ehidden%20memory%20corruption%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CBR%20%2F%3E%0A%3COL%20start%3D%225%22%3E%0A%3CLI%3E%3CSTRONG%3EQuick%20isolation%20trick%20(VERY%20effective)%3C%2FSTRONG%3E%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3ERun%3C%2FP%3E%0A%3CP%3Etestpmd%20--%20-i%3C%2FP%3E%0A%3CP%3EThen%3A%3C%2FP%3E%0A%3CP%3Eset%20fwd%20rxonly%3C%2FP%3E%0A%3CP%3Estart%3C%2FP%3E%0A%3CBR%20%2F%3E%0A%3CP%3EIf%20duplication%20disappears%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3Eyour%20application%20bug%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EIf%20still%20happens%3A%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EPMD%20%2F%20HW%20%2F%20queue%20config%20issue%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2379465%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%20translate%3D%22no%22%3ERe%3A%20rte_eth_rx_burst%20-%26gt%3B%20returning%20same%20mbuff%20pointer%20twice%20consecutively%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2379465%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3CP%3Ethanks%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fcommunity.nxp.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F52411%22%20target%3D%22_blank%22%3E%40yipingwang%3C%2FA%3E%26nbsp%3B%20for%20your%20quick%20reply%20and%20suggestions.%20from%20code%20i%20am%20not%20to%20figure%20out%20more%20.%20Any%20DPDK%20debug%20flag%20can%20help%20%3F%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2379422%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%20translate%3D%22no%22%3ERe%3A%20rte_eth_rx_burst%20-%26gt%3B%20returning%20same%20mbuff%20pointer%20twice%20consecutively%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2379422%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3CP%3EWithout%20an%20explicit%20mbuf%20free%20in%20your%20code%2C%20the%20most%20likely%20scenarios%20are%20that%20the%20mbuf%20is%20still%20being%20effectively%20returned%2Freused%20through%20another%20path%20in%20the%20application%20or%20shared%20unsafely%20across%20threads%2Fcores%2C%20or%20that%20a%20special%20driver%2Fpath-specific%20buffer%20handling%20issue%20is%20corrupting%20lifecycle%20state.%3C%2FP%3E%0A%3CUL%20class%3D%22%22%3E%0A%3CLI%20class%3D%22%22%3E%3CSTRONG%20class%3D%22%22%3ECross-core%2Fshared%20mbuf%20ownership%20issue.%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3EOne%20documented%20case%20used%20one%20CPU%20for%20RX%20and%20many%20worker%20CPUs%20consuming%20mbufs%20from%20a%20ring%3B%20the%20mbuf%20address%20was%20then%20seen%20reused%2Foverwritten%20while%20still%20referenced%20elsewhere.%20NXP%20specifically%20asked%20whether%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CCODE%20class%3D%22%22%3ERTE_MBUF_REFCNT_ATOMIC%3C%2FCODE%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Ewas%20enabled%20for%20multicore%20access%2C%20and%20the%20customer%20said%20yes%2C%20but%20the%20issue%20was%20still%20treated%20as%20application-level%20ownership%2Flifetime%20handling.%2C%3C%2FLI%3E%0A%3CLI%20class%3D%22%22%3E%3CSTRONG%20class%3D%22%22%3EDuplicate%20release%20or%20hidden%20return%20path%20in%20the%20application%20pipeline.%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3ENXP%E2%80%99s%20guidance%20in%20that%20case%20was%20to%20instrument%20buffer%20allocate%2Frelease%2Fpacket-send%20actions%20to%20see%20whether%20the%20same%20address%20was%20being%20released%20back%20to%20the%20buffer%20pool%20more%20than%20once.%3C%2FLI%3E%0A%3CLI%20class%3D%22%22%3E%3CSTRONG%20class%3D%22%22%3EPMD%2Fdriver%20returning%20malformed%20mbufs%20to%20pool%20in%20special%20paths.%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3EThere%20is%20a%20separate%20DPAA2%20PMD%20case%20involving%20chained%20mbufs%20where%20mbufs%20returned%20to%20the%20mempool%20appeared%20corrupted%20(%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CCODE%20class%3D%22%22%3Enext%3C%2FCODE%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Enot%20cleared)%2C%20causing%20sanity-check%20failures%20on%20later%20allocation.%20That%20is%20not%20the%20same%20symptom%20as%20your%20report%2C%20but%20it%20does%20show%20that%20special%20buffer%20handling%20paths%20can%20lead%20to%20later%20reuse%2Fcorruption%20symptoms.%3C%2FLI%3E%0A%3CLI%20class%3D%22%22%3E%3CSTRONG%20class%3D%22%22%3ESame%20source%20and%20destination%20mbuf%20in%20crypto%20path.%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3EIn%20a%20DPAA%20security%2FIPsec%20case%2C%20NXP%20noted%20that%20if%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CCODE%20class%3D%22%22%3Esym-%26gt%3Bm_dst%3C%2FCODE%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eis%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CCODE%20class%3D%22%22%3ENULL%3C%2FCODE%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%2C%20then%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CCODE%20class%3D%22%22%3Em_src%3C%2FCODE%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Eand%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3CCODE%20class%3D%22%22%3Em_dst%3C%2FCODE%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3Euse%20the%20same%20mbuf.%20That%20can%20make%20packet%20contents%20appear%20rewritten%20%E2%80%9Cin%20place%E2%80%9D%20under%20load%2C%20even%20though%20the%20pointer%20identity%20did%20not%20change.%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CBR%20%2F%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2380197%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%20translate%3D%22no%22%3ERe%3A%20rte_eth_rx_burst%20-%26gt%3B%20returning%20same%20mbuff%20pointer%20twice%20consecutively%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2380197%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3CP%3EThanks%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fcommunity.nxp.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F52411%22%20target%3D%22_blank%22%3E%40yipingwang%3C%2FA%3E%26nbsp%3B%20for%20Quick%20reply%20.%20I%20will%20try%20and%20update%20you%20.%3C%2FP%3E%3CP%3EOne%20observation%20every%20time%20multiple%20mbuf%20(%20same%20address%20coming%20)%20getting%20below%20prints%20in%20screen%3C%2FP%3E%3CBR%20%2F%3E%3CP%3Efslmc%3A%20dpaa2_get_qbman_swp()%3A%20New%20Portal%200x17ffd61c0%20(5)%20affined%20thread%20-%201953%3CBR%20%2F%3Efslmc%3A%20dpaa2_configure_stashing()%3A%20Portal%3D%205%20CPU%3D%2010%20SDEST%3D%205%3CBR%20%2F%3Efslmc%3A%20DPAA%20Portal%3D0x17ffd61c0%20(5)%20is%20affined%20to%20thread%201953%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-2380657%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%20translate%3D%22no%22%3ERe%3A%20rte_eth_rx_burst%20-%26gt%3B%20returning%20same%20mbuff%20pointer%20twice%20consecutively%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2380657%22%20slang%3D%22en-US%22%20mode%3D%22CREATE%22%3E%3CDIV%3EThose%20%3CCODE%3Efslmc%20%2F%20dpaa2%20portal%20affined%20thread%3C%2FCODE%3E%20logs%20strongly%20indicate%20portal%20(QBMan%20SWP)%20is%20being%20re-attached%20to%20a%20different%20thread%2Fcore%2C%20which%20can%20break%20ownership%20model%20%E2%86%92%20leading%20to%20mbuf%20reuse%20%2F%20duplication%20symptoms.%3C%2FDIV%3E%3C%2FLINGO-BODY%3E