Why are BMan hardware buffers disappearing under high load on 10G with fragmentation enabled?

[mihaiciric]

Why are BMan hardware buffers disappearing under high load? I am using a P4080DS and under high load on the 10G ports, with ip fragmentation enabled on the TX port, BMan buffers seem to never be released by the hardware.

The issue appears at really high load and only when I have fragmentation on the TX port. No error interrupts seem to be triggered anywhere and the enqueue to hardware doesn't report anything.

Thanks,

Mihai

[florentinbourge]

Hello Mihai,

We encounter a very close problem on T1040.

under high load on the "1G" ports, with ip fragmentation enabled on the TX port, "some" BMan buffers seem to never be released by the hardware.

In our linux driver we release 8000 free buffers in the pool 5 dedicated to the Tx port 0.

After high load frame transmission followed by a subsential period of inactivity, we observe from the sysfs pool counter that we never return to the initial state :

root@t1040rdb:/sys#         cat /sys/devices/ffe000000.soc/ffe31a000.bman/pool_count/8

7556

If we repeat that operation we quickly decrease the number of free buffers until none. The number of leaked buffers is quite random between 1 and 200.

Thanks for any help.

Florentin

[yipingwang]

We suggest to put the IP Fragmentation on an Offline Parsing Port just before the TX-Port.

The IP fragmentation acceleration uses by default buffers from the same buffer pool with the one of the original frame.  The FMAN driver supports a custom configuration of the IP fragmentation process where the user can specify force a specific buffer as source for the resulted frame fragments.

typedef struct t_FmPcdManipFragIpParams {
uint16_t sizeForFragmentation;
#if (DPAA_VERSION == 10)
uint8_t scratchBpid;
#endif /* (DPAA_VERSION == 10) */
bool sgBpidEn;
uint8_t sgBpid;
e_FmPcdManipDontFragAction dontFragAction;
} t_FmPcdManipFragIpParams;

This configuration option is not propagated in the DPAA offloading driver API and by default it is disabled. It is needed to propagate this configuration option in "dpa_cls_hm_ip_frag_params" parameters structure in fsl_dpa_classifier.h.

struct dpa_cls_hm_ip_frag_params {
uint16_t mtu;
uint8_t scratch_bpid;
enum dpa_cls_hm_frag_df_action df_action;
};

This will allow the application to specify a buffer pool for frame fragments instead of allowing the IP fragmentation to drain buffers from the original frame descriptor's buffer pool.

As normal scratch_bpid is necessary for the IP fragmentation on FMan v2 devices only. On FMan v3 or newer devices this parameter is ignored.

Probably these two problems are not caused by the same reason.

Florentin, would you please provide detailed procedure to help us to reproduce your problem on the demo board?

Thanks,

Yiping