Delayed ethernet receive interrupt on RT1064

[Daan]

Hi, hopefully someone knows what could be wrong with my setup. I run my own application on an RT1064 with FreeRTOS and lwip / MQTT. I publish some small amount of data sequentially (when successful callback of the publish has been called, I start the next publish).

I notice that the time between publish and callback of the publish takes about 200ms, while I see the TCP acknowledgement 50ms after publish (traced with Wireshark), so the RT1064 is doing something for 150ms until it decides to call the MQTT-callback. This delay starts right on the interrupt, so it's not FreeRTOS or lwip who's blocking.

I see the same effect with the evkmimxrt1064_lwip_mqtt_freertos example on an evaluation board, I also tested this setup without a router, so the board was directly connected to the industrial PC (Beijer) running the broker with a cross cable.

The strange thing here is that if I run the (same) broker on a regular PC, the delay for the publish callback is reduced to about 50ms, exactly the delay of the TCP acknowledgement, so that makes sense.

What could cause the interrupt to be called with such a delay?

Any ideas appreciated!

[Daan]

As mentioned, the code project is an example project of NXP: evkmimxrt1064_lwip_mqtt_freertos, I run it on the evaluation kit MIMXRT1064-EVK Rev.A1. The broker I use is GnatMQ: https://github.com/gnatmq/gnatmq

[jeremyzhou]

Hi,
Thanks for your reply.
I'm working on it now and I'll appreciate it if you can introduce how to use the GnatMQ to deploy a broker on the PC, as I don't find the user guide to do that.
Further, I'd like to suggest you upload the captured data of the WireShark.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan]

Please find GnatMQ attached, you just need to run the exe, no installation, no configuration.

Attachment 103.txt contains the Wireshark log of the MQTT communication with the industrial PC. But no odd  things can be found in here, as the delay appears in the RT1064 MCU.

Thank you for your support.

Danny Con

[jeremyzhou]

Hi,
Thanks for your reply.
According to your description, run the exe to start the local broker in the PC which uses the static IP address, the MIMXRT1064 board as the MQTT client to publish the topic and the board is configured with a static IP address whose netmask is as same as the PC's netmask.
Is my understanding is right? If yes, I was wondering if you can share your modified demo, as I have no idea about the IP address of the broker, is it the PC's IP address?
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan]

The demo runs on DHCP, so it expects an IP-address on startup, you'll need to connect a router to the evaluation kit running the demo in order to receive an IP-address. The broker's IP-address is the IP-address of the PC it is running on. Please change EXAMPLE_MQTT_SERVER_HOST in file lwip_mqtt_freertos.c to your broker's IP-address and compile the code.

In my previously shared logs, the broker has IP-address 192.168.0.103, the demo runs on 192.168.0.101.

[jeremyzhou]

Hi,
Thanks for your reply.
Below is my testing result, in general, the board receives the puback in 50 ms from the broker after publishing the message data.
However, I don't find the Publish Ack in the 103.txt, in another word, the mqtt_message_published_cb won't be called.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan]

Frame 14 is an Ack to frame 13, which is a regular publish message. The log contains lots of Acks like frame 14. Please find a new attachment (103.pcapng) which is the same log as 103.txt, but this file can be opened in Wireshark, you'll see the Ack messages.

Great to see that you have the same results, the publish callback is called within 50ms, that's what I see too, except when using the industrial PC, the time to callback is than increased to 200ms, that's what worries me, how could such a delay occur?

[jeremyzhou]

Hi,
After having a brief review of the attachment, I don't find the publish type whose initiator is board, it can explain that the attachment misses the publish ack type response.
However, it's a bit weird.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan]

I don't really understand what you mean, all publish messages and acknowledges can be seen in the log, I illustrated one publish and Ack in the attached screenshot.

[jeremyzhou]

Hi,
In the 103.pcapng, the board doesn't publish the message to the PC: 1883 port.

Further, according to the MQTT protocol, the client or broker receives the message, it will respond the Publish ACK (like the below figure shows), it also doesn't exist in the 103.pcapng too.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan]

If you look in the detail window, you'll see that port 1883 is used by the board, Wireshark also recognizes this and marks it as an MQTT message:

[jeremyzhou]

Hi，
I think you'd better read my reply carefully, I see the Wireshark recognize the Publish message whose direction is from the broker to client, however, 103.pcapng doesn't have the Publish message whose direction is from the client to broker, that's the key point.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan]

Do you perhaps have some filter applied in Wireshark? As you can see in my screenshot, which is from file 103.pcapng, messages are sent from 192.168.0.101 (the board) to 192.168.0.103 (broker) and vice versa.

[jeremyzhou]

Hi，
Thanks for your reply.
1) As you can see in my screenshot, which is from file 103.pcapng, messages are sent from 192.168.0.101 (the board) to 192.168.0.103 (broker) and vice versa?
-- Yes, I see it, however, these frames are usually sending an ACK, which is used to respond to the previous MQTT publish message frame.

Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[Daan · Accepted solution]

Hi,

MQTT with QoS = 0 is a fire and forget protocol, so on MQTT-level, no Acks are returned. However, it is based on TCP, all messages are delivered guaranteed, so on TCP-level, all publishes are Ack'ed, that's what you see in the logs.

Anyway, I figured it all out, the log 103 I've sent is actually misleading, you only see the traffic between the broker (running on 192.168.0.103) and the PC (192.168.0.101), traffic to the board (192.168.0.102) is not shown, since it is filtered out by the router, so Wireshark (running on the PC) can not detect it.

But to test where the delay is coming from, I logged timestamps in the boards ethernet Tx and Rx interrupt, and I noticed it takes 200ms from publish to receiving the Ack, so the delay is fully caused by the industrial PC (Beijer) and not by the board. I know how I can implement a workaround for that, so I close this issue now.

Thanks for your support!
Cheers,
Danny

[Daan]

Hi Jeremy, thank you for your reply. Yes, your understanding of my situation is correct.

To determine the time the system takes from publish to callback, I just check the current time using xTaskGetTickCount(). Like so:

TickType_t t0;

static void publish_message(void *ctx)
{
  static const char *topic = "MyPublish";
  const char* message = "some_data";
  err_t err;

  LWIP_UNUSED_ARG(ctx);

  err = mqtt_publish(mqtt_client, topic, message, strlen(message), 0, 0, mqtt_message_published_cb, (void *)topic);
  t0 = xTaskGetTickCount();
  if (err != ERR_OK)
  {
    PRINTF("Publish returned %d\r\n", err);
  }
}

static void mqtt_message_published_cb(void *arg, err_t err)
{
  const char *topic = (const char *)arg;
  TickType_t now = xTaskGetTickCount();

  PRINTF("Callback %d, diff: %d\r\n", now, now - t0);
  if (err != ERR_OK)
  {
    PRINTF("Failed to publish to the topic \"%s\": %d.\r\n", topic, err);
  }
}

[jeremyzhou]

Hi,
Thanks for your reply.
I was wondering if you can introduce your testing environment, such as, deploy local broker, hardware wire connection, code project, etc, then then I can replicate the above phenomenon on my side.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------

[jeremyzhou]

Hi,
Thank you for your interest in NXP Semiconductor products and for the opportunity to serve you.
According to your description, the period of time between publish and callback of the publish takes about 200ms when the broker resides on an industrial PC (Beijer), which is about four times the regular PC, is my understanding right?
If yes, I was wondering if you can list the testing steps of calculating the time between publishing and callback of the publish, then I can replicate the phenomenon on my side.
Looking forward to your reply.
Have a great day,
TIC

-------------------------------------------------------------------------------
Note:
- If this post answers your question, please click the "Mark Correct" button. Thank you!

- We are following threads for 7 weeks after the last post, later replies are ignored
Please open a new thread and refer to the closed one, if you have a related question at a later point in time.
-------------------------------------------------------------------------------