PacketBuffer allocation from PBUF_POOL to PBUF_RAM in Matter All-clusters-app for FRDM-RW612

[farazahmed8]

Hello,
I am currently working with the Matter all-clusters-app example on the FRDM-RW612.
I debugged PacketBufferHandle::New(), I noticed that by default, the function allocated buffers with
pbuf_alloc(PBUF_RAM, allocSize, PBUF_POOL);
as by default CHIP_SYSTEM_CONFIG_USE_LWIP macro is set
This results in chained pbufs instead of a single contiguous buffer. For my case, I noticed a chain of total length 1962 bytes, split into one buffer with len = 1424 bytes and the next buffer with len = 426. In the New() function implementation, tot_len and len are forcibly set to zero, which override the values returned by pbuf_alloc(). Leading, to MaxDataLength() returning the length of first buffer 1424 bytes only.

My question is, what is the correct way to configure the macros so that PacketBufferHandle::New uses PBUF_RAM instead of PBUF_POOL, ensuring all relevant functions return the correct length. I want to have heap-allocated contiguous buffers instead of chained pool buffers, I guess the default pool allocation is limited because of MTU size 1500 bytes.

Here is the code for calling PacketBufferHandle::New

// Construct Sigma2 Msg
    size_t size_of_local_session_id = sizeof(uint16_t);
    size_t data_len =
        TLV::EstimateStructOverhead(kSigmaParamRandomNumberSize, size_of_local_session_id, kP256_PublicKey_Length, PQCLEAN_MLKEM768_CLEAN_CRYPTO_CIPHERTEXTBYTES,
                                    msg_r2_signed_enc_len, CHIP_CRYPTO_AEAD_MIC_LENGTH_BYTES, SessionParameters::kEstimatedTLVSize);

    System::PacketBufferHandle msg_R2 = System::PacketBufferHandle::New(data_len);
    VerifyOrReturnError(!msg_R2.IsNull(), CHIP_ERROR_NO_MEMORY);

    System::PacketBufferTLVWriter tlvWriterMsg2;
    outerContainerType = TLV::kTLVType_NotSpecified;

    ChipLogProgress(SecureChannel, "DataLength: %u", msg_R2->DataLength());
    ChipLogProgress(SecureChannel, "TotalLength: %u", msg_R2->TotalLength());
    ChipLogProgress(SecureChannel, "MaxDataLength: %u", msg_R2->MaxDataLength());
    ChipLogProgress(SecureChannel, "AvailableDataLength: %u", msg_R2->AvailableDataLength());
    ChipLogProgress(SecureChannel, "ReservedSize: %u", msg_R2->ReservedSize());
    ChipLogProgress(SecureChannel, "HasChainedBuffer: %s", msg_R2->HasChainedBuffer() ? "true" : "false");

    tlvWriterMsg2.Init(std::move(msg_R2));

    size_t usedLength = tlvWriterMsg2.GetLengthWritten();
    size_t remainLength = tlvWriterMsg2.GetRemainingFreeLength();

    ChipLogProgress(SecureChannel," UsedLength:%u, RemainingLength:%u",usedLength,remainLength);

    ReturnErrorOnFailure(tlvWriterMsg2.StartContainer(TLV::AnonymousTag(), TLV::kTLVType_Structure, outerContainerType));
    usedLength = tlvWriterMsg2.GetLengthWritten();
    remainLength = tlvWriterMsg2.GetRemainingFreeLength();

    ChipLogProgress(SecureChannel," UsedLength:%u, RemainingLength:%u",usedLength,remainLength);
    ReturnErrorOnFailure(tlvWriterMsg2.PutBytes(TLV::ContextTag(1), &msg_rand[0], sizeof(msg_rand)));
    usedLength = tlvWriterMsg2.GetLengthWritten();
    remainLength = tlvWriterMsg2.GetRemainingFreeLength();

    ChipLogProgress(SecureChannel," UsedLength:%u, RemainingLength:%u",usedLength,remainLength);
    ReturnErrorOnFailure(tlvWriterMsg2.Put(TLV::ContextTag(2), GetLocalSessionId().Value()));
    usedLength = tlvWriterMsg2.GetLengthWritten();
    remainLength = tlvWriterMsg2.GetRemainingFreeLength();

    ChipLogProgress(SecureChannel," UsedLength:%u, RemainingLength:%u",usedLength,remainLength);

    ReturnErrorOnFailure(tlvWriterMsg2.PutBytes(TLV::ContextTag(3), mEphemeralKey->Pubkey(),
                                                static_cast<uint32_t>(mEphemeralKey->Pubkey().Length())));

    usedLength = tlvWriterMsg2.GetLengthWritten();
    remainLength = tlvWriterMsg2.GetRemainingFreeLength();

    ChipLogProgress(SecureChannel," UsedLength:%u, RemainingLength:%u",usedLength,remainLength);
                                            
    ReturnErrorOnFailure(tlvWriterMsg2.PutBytes(TLV::ContextTag(4), mlkem768_ciphtxt,static_cast<uint32_t>(PQCLEAN_MLKEM768_CLEAN_CRYPTO_CIPHERTEXTBYTES)));

    usedLength = tlvWriterMsg2.GetLengthWritten();
    remainLength = tlvWriterMsg2.GetRemainingFreeLength();

    ChipLogProgress(SecureChannel," UsedLength:%u, RemainingLength:%u",usedLength,remainLength);

    Platform::MemoryFree(mlkem768_ciphtxt);
    mlkem768_ciphtxt=nullptr;
    ChipLogProgress(SecureChannel, "ML-KEM768 mlkem768_ciphtxt release!");   

    ChipLogProgress(SecureChannel, "Putting msg into container!");                                                  
    ReturnErrorOnFailure(tlvWriterMsg2.PutBytes(TLV::ContextTag(5), msg_R2_Encrypted.Get(),
                                                static_cast<uint32_t>(msg_r2_signed_enc_len + CHIP_CRYPTO_AEAD_MIC_LENGTH_BYTES)));
    ChipLogProgress(SecureChannel,"SendSigma2 after contextTag 5");  

    VerifyOrReturnError(mLocalMRPConfig.HasValue(), CHIP_ERROR_INCORRECT_STATE);
    ReturnErrorOnFailure(EncodeSessionParameters(TLV::ContextTag(6), mLocalMRPConfig.Value(), tlvWriterMsg2));
    ChipLogProgress(SecureChannel,"SendSigma2 after contextTag 6");
    ReturnErrorOnFailure(tlvWriterMsg2.EndContainer(outerContainerType));
    ReturnErrorOnFailure(tlvWriterMsg2.Finalize(&msg_R2));
    ChipLogProgress(SecureChannel,"SendSigma2 after ending outercontainertype");
    ReturnErrorOnFailure(mCommissioningHash.AddData(ByteSpan{ msg_R2->Start(), msg_R2->DataLength() }));
  
    ChipLogProgress(SecureChannel,"Before Calling Sending Sigma2");
    // Call delegate to send the msg to peer
    ReturnErrorOnFailure(mExchangeCtxt.Value()->SendMessage(Protocols::SecureChannel::MsgType::CASE_Sigma2, std::move(msg_R2),
                                                            SendFlags(SendMessageFlags::kExpectResponse)));
    ChipLogProgress(SecureChannel,"After Calling Sending Sigma2");
    mState = State::kSentSigma2;

This is the PacketBufferHandle::New implementation

PacketBufferHandle PacketBufferHandle::New(size_t aAvailableSize, uint16_t aReservedSize)
{
    // Sanity check for kStructureSize to ensure that it matches the PacketBuffer size.
    static_assert(PacketBuffer::kStructureSize == sizeof(PacketBuffer), "PacketBuffer size mismatch");
    // Setting a static upper bound on kStructureSize to ensure the summation of all the sizes does not overflow.
    static_assert(PacketBuffer::kStructureSize <= UINT16_MAX, "kStructureSize should not exceed UINT16_MAX.");
    // Setting a static upper bound on the maximum buffer size allocation for regular sized messages (not large).
    static_assert(PacketBuffer::kMaxSizeWithoutReserve <= UINT16_MAX, "kMaxSizeWithoutReserve should not exceed UINT16_MAX.");

#if INET_CONFIG_ENABLE_TCP_ENDPOINT
    // Setting a static upper bound on the maximum buffer size allocation for
    // large messages.
#if CHIP_SYSTEM_CONFIG_USE_LWIP
    // LwIP based systems are internally limited to using a u16_t type as the size of a buffer.
    static_assert(PacketBuffer::kLargeBufMaxSizeWithoutReserve <= UINT16_MAX,
                  "In LwIP, max size for Large payload buffers cannot exceed UINT16_MAX!");
#else
    // Messages over TCP are framed using a length field that is 32 bits in
    // length.
    static_assert(PacketBuffer::kLargeBufMaxSizeWithoutReserve <= UINT32_MAX,
                  "Max size for Large payload buffers cannot exceed UINT32_MAX");
#endif // CHIP_SYSTEM_CONFIG_USE_LWIP
#endif // INET_CONFIG_ENABLE_TCP_ENDPOINT

    // Ensure that aAvailableSize is bound within a max and is not big enough to cause overflow during
    // subsequent addition of all the sizes.
    if (aAvailableSize > UINT32_MAX)
    {
        ChipLogError(chipSystemLayer,
                     "PacketBuffer: AvailableSize of a buffer cannot exceed UINT32_MAX. aAvailableSize = 0x" ChipLogFormatX64,
                     ChipLogValueX64(static_cast<uint64_t>(aAvailableSize)));
        return PacketBufferHandle();
    }

    // Cast all to uint64_t and add. This cannot overflow because we have
    // ensured that the maximal value of the summation is
    // UINT32_MAX + UINT16_MAX + UINT16_MAX, which should always fit in
    // a uint64_t variable.
    uint64_t sumOfSizes = static_cast<uint64_t>(aAvailableSize) + static_cast<uint64_t>(aReservedSize) +
        static_cast<uint64_t>(PacketBuffer::kStructureSize);
    uint64_t sumOfAvailAndReserved = static_cast<uint64_t>(aAvailableSize) + static_cast<uint64_t>(aReservedSize);

    // Ensure that the sum fits in a size_t so that casting into size_t variables,
    // viz., lBlockSize and lAllocSize, is safe.
    if (!CanCastTo<size_t>(sumOfSizes))
    {
        ChipLogError(chipSystemLayer,
                     "PacketBuffer: Sizes of allocation request are invalid. (aAvailableSize = " ChipLogFormatX64
                     ", aReservedSize = " ChipLogFormatX64 ")",
                     ChipLogValueX64(static_cast<uint64_t>(aAvailableSize)), ChipLogValueX64(static_cast<uint64_t>(aReservedSize)));
        return PacketBufferHandle();
    }

#if CHIP_SYSTEM_CONFIG_USE_LWIP
    // LwIP based APIs have a maximum buffer size of UINT16_MAX. Ensure that
    // limit is met during allocation.
    if (sumOfAvailAndReserved > UINT16_MAX)
    {
        ChipLogError(chipSystemLayer,
                     "LwIP based systems require total buffer size to be less than UINT16_MAX!"
                     "Attempted allocation size = " ChipLogFormatX64,
                     ChipLogValueX64(sumOfAvailAndReserved));
        return PacketBufferHandle();
    }
#endif // CHIP_SYSTEM_CONFIG_USE_LWIP

    // sumOfAvailAndReserved is no larger than sumOfSizes, which we checked can be cast to
    // size_t.
    const size_t lAllocSize = static_cast<size_t>(sumOfAvailAndReserved);
    PacketBuffer * lPacket;

    CHIP_SYSTEM_FAULT_INJECT(FaultInjection::kFault_PacketBufferNew, return PacketBufferHandle());

    if (lAllocSize > PacketBuffer::kMaxAllocSize)
    {
        ChipLogError(chipSystemLayer, "PacketBuffer: allocation exceeding buffer capacity limits: %lu > %lu",
                     static_cast<unsigned long>(lAllocSize), static_cast<unsigned long>(PacketBuffer::kMaxAllocSize));
        return PacketBufferHandle();
    }

#if CHIP_SYSTEM_CONFIG_USE_LWIP
    // This cast is safe because lAllocSize is no larger than
    // kMaxSizeWithoutReserve, which fits in uint16_t.
    lPacket = static_cast<PacketBuffer *>(
        pbuf_alloc(PBUF_RAW, static_cast<uint16_t>(lAllocSize), CHIP_SYSTEM_PACKETBUFFER_LWIP_PBUF_TYPE));

    SYSTEM_STATS_UPDATE_LWIP_PBUF_COUNTS();
    
    /*/
    if(lAllocSize==1962)//for debugging purpose
    {
        ChipLogProgress(chipSystemLayer,"lPacket->tot_len:%u",lPacket->tot_len);
        ChipLogProgress(chipSystemLayer,"lPacket->next->len:%u",lPacket->len);
    }
    /*/

    ChipLogProgress(chipSystemLayer,"LWIP:PacketBuffer::New Request size=%u, aAvailable=%u, aReserved=%u, MaxSize=%u",lAllocSize,aAvailableSize,aReservedSize,PacketBuffer::kMaxAllocSize);

#elif CHIP_SYSTEM_PACKETBUFFER_FROM_CHIP_POOL

#if !CHIP_SYSTEM_CONFIG_NO_LOCKING && CHIP_SYSTEM_CONFIG_FREERTOS_LOCKING
    if (!sBufferPoolMutex.isInitialized())
    {
        Mutex::Init(sBufferPoolMutex);
    }
#endif
    LOCK_BUF_POOL();

    lPacket = PacketBuffer::sFreeList;
    if (lPacket != nullptr)
    {
        PacketBuffer::sFreeList = lPacket->ChainedBuffer();
        SYSTEM_STATS_INCREMENT(chip::System::Stats::kSystemLayer_NumPacketBufs);
    }

    UNLOCK_BUF_POOL();

    ChipLogProgress(chipSystemLayer,"CHIP POOL:PacketBuffer::New Request size=%u",lAllocSize);

#elif CHIP_SYSTEM_PACKETBUFFER_FROM_CHIP_HEAP
    // sumOfSizes is essentially (kStructureSize + lAllocSize) which we already
    // checked to fit in a size_t.
    const size_t lBlockSize = static_cast<size_t>(sumOfSizes);
    lPacket                 = reinterpret_cast<PacketBuffer *>(chip::Platform::MemoryAlloc(lBlockSize));
    SYSTEM_STATS_INCREMENT(chip::System::Stats::kSystemLayer_NumPacketBufs);

    ChipLogProgress(chipSystemLayer,"CHIP HEAP:PacketBuffer::New Request size=%u",lAllocSize);
#else
#error "Unimplemented PacketBuffer storage case"
#endif

    if (lPacket == nullptr)
    {
        ChipLogError(chipSystemLayer, "PacketBuffer: pool EMPTY.");
        return PacketBufferHandle();
    }

    lPacket->payload = lPacket->ReserveStart() + aReservedSize;
    lPacket->len = lPacket->tot_len = 0;
    lPacket->next                   = nullptr;
    lPacket->ref                    = 1;
#if CHIP_SYSTEM_PACKETBUFFER_FROM_CHIP_HEAP
    lPacket->alloc_size = lAllocSize;
#endif

    return PacketBufferHandle(lPacket);
}

The log of FRDM-RW612

[35927] [INFO] [SC] CASE matched destination ID: fabricIndex 1, NodeID 0x00000000000022B8
[35936] [INFO] [SC] ML-KEM768 Generating shared secret!
[35941] [INFO] [SC] mlkem768_pubkey[0..4]: 8C 7C 9D A5 52
[35953] [INFO] [SC] ML-KEM768 Pub Key Release!
[36141] [INFO] [CSL] lPacket->tot_len:1962
[36145] [INFO] [CSL] lPacket->next->len:426
[36149] [INFO] [CSL] LWIP:PacketBuffer::New Request size=1962, aAvailable=1850, aReserved=112, MaxSize=10000
[36159] [INFO] [SC] DataLength: 0
[36162] [INFO] [SC] TotalLength: 0
[36165] [INFO] [SC] MaxDataLength: 1424
[36169] [INFO] [SC] AvailableDataLength: 1424
[36173] [INFO] [SC] ReservedSize: 112
[36177] [INFO] [SC] HasChainedBuffer: false
[36232] [INFO] [SC]  UsedLength:0, RemainingLength:1424
[36237] [INFO] [SC]  UsedLength:1, RemainingLength:1423
[36242] [INFO] [SC]  UsedLength:36, RemainingLength:1388
[36248] [INFO] [SC]  UsedLength:40, RemainingLength:1384
[36253] [INFO] [SC]  UsedLength:108, RemainingLength:1316
[36258] [INFO] [SC]  UsedLength:1200, RemainingLength:224
[36264] [INFO] [SC] ML-KEM768 mlkem768_ciphtxt release!
[36269] [INFO] [SC] Putting msg into container!
[36273] [INFO] [SC] Source of Error CASESession.cpp line 1105
[36279] [TRACE] [SC] Sending status report. Protocol code 2, exchange 33878
[36290] [INFO] [CSL] LWIP:PacketBuffer::New Request size=120, aAvailable=8, aReserved=112, MaxSize=10000
[36300] [INFO] [EM] <<< [E:33878r S:0 M:237126316] (U) Msg TX from 0000000000000000 to 0:1B9439909583B3DD [0000] [TCP:[FE80::FE84:A7FF:FE51:F90A]:38778] --- Type 0000:40 (SecureChannel:StatusReport) (B:30)
[36319] [ERR] [IN] CASE Session establishment failed: b

[farazahmed8]

I am on the default branch release/v1.4.0. I set LWIP_NETIF_TX_SINGLE_PBUF to 1, however I still get a chained buffer, which triggers the assertion.

VerifyOrReturnError(!msgBuf->HasChainedBuffer(), CHIP_ERROR_INVALID_MESSAGE_LENGTH);

inside SessionManager.cpp, as discussed above.

Below is the log from the FRDM-RW612:

[29801] [INFO] [IN] Incoming connection established with peer at FE80::FE84:A7FF:FE51:F90A.
[29810] [INFO] [IN] Received TCP connection request from TCP:[FE80::FE84:A7FF:FE51:F90A]:53948.
[29819] [INFO] [EM] >>> [E:9949r S:0 M:255536127] (U) Msg RX from 0:71DFAE6725F91483 [0000] to 0000000000000000 --- Type 0000:30 (SecureChannel:CASE_Sigma1) (B:1384)
[29834] [TRACE] [EM] Handling via exchange: 9949r, Delegate: 0x2005bd58
[29840] [INFO] [IN] CASE Server received Sigma1 message . Starting handshake. EC 0x2005cde0
[29849] [INFO] [SC] Received Sigma1 msg
[29853] [TRACE] [SC] Found MRP parameters in the message
[29858] [TRACE] [SC] Peer assigned session key ID 30689
[29863] [INFO] [DL] KVS, get key id:: f/1/g
[29867] [INFO] [DL] KVS, get key id:: f/1/k/0
[29873] [INFO] [SC] CASE matched destination ID: fabricIndex 1, NodeID 0x00000000000022B8
[29881] [INFO] [SC] ML-KEM768 Generating shared secret!
[29886] [INFO] [SC] mlkem768_pubkey[0..4]: E5 24 97 78 51
[29898] [INFO] [SC] ML-KEM768 Pub Key Release!
[30077] [INFO] [CSL] lPacket->tot_len:1962
[30081] [INFO] [CSL] lPacket->len:1536
[30085] [INFO] [CSL] LWIP:PacketBuffer::New Request size=1962, aAvailable=1850, aReserved=112, MaxSize=10000
[30095] [INFO] [SC] DataLength: 0
[30098] [INFO] [SC] TotalLength: 0
[30101] [INFO] [SC] MaxDataLength: 1424
[30105] [INFO] [SC] AvailableDataLength: 1424
[30109] [INFO] [SC] ReservedSize: 112
[30113] [INFO] [SC] HasChainedBuffer: false
[30117] [INFO] [SC]  UsedLength:0, RemainingLength:1424
[30122] [INFO] [SC]  UsedLength:1, RemainingLength:1423
[30127] [INFO] [SC]  UsedLength:36, RemainingLength:1388
[30132] [INFO] [SC]  UsedLength:40, RemainingLength:1384
[30137] [INFO] [SC]  UsedLength:108, RemainingLength:1316
[30143] [INFO] [SC]  UsedLength:1200, RemainingLength:224
[30148] [INFO] [SC] ML-KEM768 mlkem768_ciphtxt release!
[30153] [INFO] [SC] Putting msg into container!
[30158] [INFO] [CSL] LWIP:PacketBuffer::New Request size=1536, aAvailable=1536, aReserved=0, MaxSize=10000
[30167] [INFO] [SC] SendSigma2 after contextTag 5
[30172] [INFO] [SC]  UsedLength:1787, RemainingLength:1173
[30177] [INFO] [SC] SendSigma2 after contextTag 6
[30182] [INFO] [SC] SendSigma2 after ending outercontainertype
[30188] [INFO] [SC] Before Calling Sending Sigma2
[30192] [INFO] [SC] HasChainedBuffer: true
[30197] [INFO] [EM] <<< [E:9949r S:0 M:193763833] (U) Msg TX from 0000000000000000 to 0:71DFAE6725F91483 [0000] [TCP:[FE80::FE84:A7FF:FE51:F90A]:53948] --- Type 0000:31 (SecureChannel:CASE_Sigma2) (B:1840)
[30215] [INFO] [SC] Source of Error CASESession.cpp line 1105
[30221] [TRACE] [SC] Sending status report. Protocol code 2, exchange 9949
[30228] [INFO] [CSL] LWIP:PacketBuffer::New Request size=120, aAvailable=8, aReserved=112, MaxSize=10000
[30238] [INFO] [EM] <<< [E:9949r S:0 M:193763834] (U) Msg TX from 0000000000000000 to 0:71DFAE6725F91483 [0000] [TCP:[FE80::FE84:A7FF:FE51:F90A]:53948] --- Type 0000:40 (SecureChannel:StatusReport) (B:30)
[30257] [ERR] [IN] CASE Session establishment failed: 18

The error code 0x18 corresponds to CHIP_ERROR_INVALID_MESSAGE_LENGTH. Looking at the implementation in PacketBufferHandle.cpp, when CHIP_SYSTEM_CONFIG_USE_LWIP is enabled, there are only two options for packet allocation: PBUF_POOL and PBUF_RAM.

With PBUF_RAM, the PacketBufferHandle implementation seems not fully supported.

With PBUF_POOL, is it possible to configure the size of each pbuf to be larger than 1500 bytes, so that the entire payload fits into a single pbuf instead of being chained? Would lwIP be able to handle payloads larger than 1500 bytes, given that the default MTU size is 1500?

[farazahmed8]

The TLVWriter wrapper by PacketBufferHandle has to be passed true to Init(std::move(msg_R2), true) in the case of a chained buffer. However, at the lower layer (SessionManager.cpp), because of the check within SendPreparedMessage

VerifyOrReturnError(!msgBuf->HasChainedBuffer(), CHIP_ERROR_INVALID_MESSAGE_LENGTH);

this throws an error. If I disable this check, only the first pbuf is sent (1446 bytes). The only way to make it work is by increasing PBUF_POOL_BUFSIZE large enough so the entire payload fits into a single pbuf.

Your help would be greatly appreciated in this regard.

[farazahmed8]

Is it possible on the FRDM-RW612 to increase PBUF_POOL_BUFSIZE to more than 2000 bytes, so that the entire payload can fit within a single pbuf instead of being chained?

[RomanVR]

Hello @farazahmed8 

Please confirm if the application works properly by having the default configurations, including the use chained buffers. Also share with me the specific matter version you are working with, for this last request, please send me a screenshot of the log that appears when you write "git log" on your matter path, this will help me to better analyze your issue.

[farazahmed8]

Yes, the application was working fine with the default configuration.
After that, I just appended the CASE_Sigma2 message, which is now larger than 1500 bytes (the MTU limit), and the issue occurred after that.

[RomanVR]

Hello @farazahmed8, hope you are doing well.

Could you please confirm if you are using the latest matter release (v1.4.0.2.1) and if the application works properly with the default configurations that you mention in your first post? (CHIP_SYSTEM_CONFIG_USE_LWIP macro set and allocated chained pbufs)

Additionally, please try by setting the value of the macro "LWIP_NETIF_TX_SINGLE_PBUF" to 1, you can found this at the path "<path-to-matter>/examples/all-clusters-app/nxp/rt/rw61x/third_party/connectedhomeip/src/lwip/standalone" and inside the file "lwipopts.h". This macro will try to make lwip to put all data into one single pbuf.

Please let me know if this works.