How to support RAW12 output on i.MX8M Plus, Linux system

[chendd14]

In spite of the powerfull ISP module on 8MP, I want to get the bayer raw12 data from the ISP module, so I can process the image with our own algorithms to get a better quality.

Can anyone give me some advice?

Many thanks.

[malik_cisse]

As promised, for completeness:

One can get actually Raw bayer Images if ISP only is enabled but configured in bypass mode (ISI must be disabled in the device tree).
One needs to do:
v4l2-ctl -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat=RG12 --stream-mmap --stream-count=1 --stream-to=test_12bit_bayer.raw

Result is RGGB 12bit raw bayer image in big-endian

When using ISI, RGGB 12bit raw bayer image is in litle endian

The option pixelformat=RG12 in v4l2-ctl command tells the camera engine to "bypass" debayering/demociacing

The size of this bayer image is: (3840x2160x16)/8 == 16588800 Bytes. Each 12bit Bayer Pixel is packed in 16Bits.

This image must be debayered and Color Correction must be applied prior to saving (e.g as TIFF).

12bpp RGB TIFF image is quite big. (3840x2160x3)x16/8 == 49766400. This has lots of overhead because each 12bit pixel is packed in 16bit. Not sure if there is a compact lossless way to pack this.

Doing the same as above using gstreamer:
gst-launch-1.0 v4l2src num-buffers=1 device=/dev/video0 ! video/x-raw,format=rgrg,depth=12,width=3840,height=2160 ! filesink location=raw_image1_RG12_rgrg.raw

[chendd14]

hi, malik_cisse,

I followed your advise and captured a 12bit image successfully with gstreamer.

Since the image is in big-endian, I performed 'BSWAP' first, then debayering with `cv::cvtColor(img_bayer, img_result, cv::COLOR_BayerRG2BGR)`, a debayering function in OpenCV. 

#define BSWAP_16(x) (uint16_t)((((uint16_t)(x)&0x00ff) <<  | (((uint16_t)(x)&0xff00) >> 8))

void Bayer2ImageNXPTest() {
    std::string file_name = "test_12bit_bayer.raw";
    FILE       *fp1       = fopen(file_name.c_str(), "rb");

    int size = GetFileSize(file_name);

    char *buf = new char[size];
    for (int i = 0; i < size; i++) {
        buf[i] = getc(fp1);
    }

    fclose(fp1);

    int width  = 4000;
    int height = 3000;
    std::cout << "file size is " << size << std::endl;
    cv::Mat img_bayer = cv::Mat(height, width, CV_16UC1);
    memcpy(img_bayer.data, buf, size);
    delete buf;

    for (int i = 0; i < height; i++) {
        for (int j = 0; j < width; j++) {
            img_bayer.at<uint16_t>(i, j) = BSWAP_16(img_bayer.at<uint16_t>(i, j)) - 256;
        }
    }

    cv::Mat img_result;
    cv::cvtColor(img_bayer, img_result, cv::COLOR_BayerRG2BGR);
    cv::Mat img_show;
    img_result.convertTo(img_show, CV_8UC3, 1.0f / 256.0f);

    cv::resize(img_show, img_show, cv::Size(800, 600));
    cv::imshow("show", img_show);
    cv::waitKey(0);
}

But I found the image is gray. The debayering process had been proved functional well in another program. 

Would you please help me find out the problem, is there some thing special with this debyering process? I confirm that my sensor is a colorful sensor.

[chendd14]

Thanks for your help, It works.

[malik_cisse · Accepted solution]

Hi Chendd14
There are basically two solutions for the imx8mp.
1-imx8mp ISP does support 12bit RAW bayer output without any debayering. When applying debayered, only 8bit YUYV is supported.
If you want, I can provide the gstreamer command to get Raw.

2-Second solution (in case you need ISP in parallel) would be to connect ISI to same CSI input and get Raw 12bpp bayer from that.
I am using this scenario to stream ISP output while making high quality snapshots using ISI and software debayering in OpenCV.

[Egor-G]

Hello! I'm trying to get RAW12 or RAW10 using ISI on IMX8M PLUS from IMX385 sensor, but v4l2-ctl don't output any data, I'd tryed to modify imx8-sis-cap.c to add RAW formats description in out and src description structures, but nothing helps. Sometimes v4l2-ctl output some data but too slowly ( 0.02 fps ). Could you share, please, your modification of isi driver to get RAW working?

[malik_cisse]

Hi Egor,

If you are using recent BSP (e.g Kirkstone with Kernel 5.15.*) you should not need to change anything in ISI driver itself. Everything is up-and-running there.

The only things you change is implement sensor kernel driver and adapt device tree. Obviously you have done that.
After that Raw 12 Bit Bayer Images can be fetched with these commands (among others):
yavta /dev/video0 -c8 --skip 7 -f SRGGB12 -s 3840x2160 -Fimx477_PixDepth_SRGGB12_1X12_v2.raw
v4l2-ctl --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=1

Make sure ISI sits on /dev/video0

[_angelo_]

maybe you can find something interesting here

https://community.nxp.com/docs/DOC-345149?sr=search&searchId=fe81c77e-1aa9-48de-bc49-f30d8fc50ebe&se...

or 

https://community.nxp.com/t5/i-MX-Processors/iMX8QM-RAW12-Camera-AR0231/m-p/998482