IMX8MP ISP - isp_media_server Source Code (for Debbuging HDR)

[tschulze]

Hello,

for a customer we are currently working on a project using the i.MX 8M PLUS and the Onsemi AR0521 sensor. The customer wants to use the sensor in HDR mode (L+S exposure) which is generally working but enabling HDR introduces some ugly image artifacts on the video stream (ghosting, double edges between long and short exposure areas etc.)

We have access to the source code of the AR0521 sensor driver but we could not find any source code related to the ISP functionality (isp_media_server). For a better understanding of the HDR functionality of the iMX8MP and a possible solution to our bad video quality it would be good to have an insight into the ISP functionality.

Is it possible to get access to the isp_media_server code? If yes, where can we find it?

The following website suggests that you can compile the isp_media_server yourself, so there should be at least some source code available:

 ISP_Architecture_i.MX8M_Plus_:_V4L2_Sensor_Drivers_Porting_Guide 

Thanks!

[JorgeCas]

Hello, 

To get access to ISP tuning tool, you need to put in contact with a Sales-person since the tool also requires the lab setup and of course the engineers, the marketing and software teams evaluate if they are needing help from our side that can accelerate the new sensor enablement.

For now you can use the API to tune it by trial and error:

https://www.nxp.com/docs/en/user-guide/UG10168.pdf

Could you please share the image artifacts you're seeing?

Best regards.

[tschulze]

If I understand correctly, the ISP Tuning tools are used to determine the actual ISP tuning parameters (AEC, AWB, WDR3, Filter values etc.) which are then stored (XML file) and loaded to the ISP on ECU start. We already have (somewhat) functioning ISP parameter files and we were already able to change them either by VVEXT tool or changing the values of the XML files directly.

What I actually meant with the question is the code of the ISP logic which is running constantly on the board to set ISP and sensor params/registers dynamically with changing light conditions. In our build we can only see a binary called "isp_media_server" which is downloaded from a NXP fileserver during the project build process, copied to the board and then automatically started.

I attached an image of the "ghosting" artifact that we encounter. The image is taken from a 30 FPS video stream with HDR ratio = 8. If we move an object through the frame we see double outlines of the moving object. We assume this has something to do with the time difference of the short and long exposure on the AR0521 sensor.

We want to remove or at least reduce this effect as much as possible. But unfortunately the HDR related options for the ISP are pretty limited - basically there is only the HDR Ratio that we can change.

Thanks for you help!

30 fps video stream with HDR ratio 8

[JorgeCas]

Hello,

Customers can't get source codes for ISP. But with the SDK, you should be able to bring up the sensors. 

You can refer to this ISP New Camera Sensor Porting and Integration Guide for i.MX 8M Plus EVK application note, you driver is working since you are able to capture image from the sensor, you should try to tune the parameters and check if image improves.

You also can consider put in contact with local DFAE to contact NXP marketing since the tools are offered by the partner. 

Best regards.

[tschulze]

Hello,

you are right: the sensor driver is working correctly (in linear mode). It was actually added to the Linux kernel some time ago. We only added some additions to make it run in HDR mode.

We already tried some different ISP settings but there is basically only one parameter that is related to HDR (and our ghosting issue): the "HDR Ratio". Of course we can turn down this value and decrease the ghosting, but this would also decrease the general HDR strength which we would like to leave at a reasonable level to be useful.

Since we already adjusted some ISP parameters manually we have the impression that the ISP tuning tools can't really help us with this specific ghosting issue.

Regards

[JorgeCas]

Hello,

Usually these kind of issues in the image are solved by tuning.

Since the issues appears by configuring the HDR makes me think that could be the camera itself.

It seems that Toradex have integrated this camera in their BSP. Did you try to use their documentation as a reference for your implementation?

https://community.toradex.com/t/ar0521-with-verdin-imx8mp-dahlia/18012

Best regards.

[tschulze]

Hello,

I was looking through the Toradex forum, but that didn't really help unfortunately. Your linked post is about integrating the AR0521 sensor into the BSP. This is already working on our side (the sensor was already added by NXP and can be used succesfully).

However, we made a little progress. We found the alleged source code for the isp_media_server - so it seems it actually does exist. During build of the isp-imx repo, we download a binary from an NXP server. This is the address: http://www.nxp.com/lgfiles/NMG/MAD/YOCTO/isp-imx-4.2.2.20.0.bin 

This binary is then unpacked and contains the sources (located in mediacontrol - see below). These sources are then build into the isp_media_server binary file.

However, it looks like the isp_media_server is only a message handler inside the ISP chain. Do you have any information on where we can find the actual logic that handles the ISP algorithms (HDR merging, exposure calculations etc.)? Who created it (NXP or Verisilicon?) and which SW component is responsible for that?

Regards

[tschulze]

Here's a video of the behavior