How to capture raw bayer data.

[korneliuszosmen]

I have a problem with passing raw data from sensor.

Sensor sends 10bit packed data, I try to get 8bit MSB but I get probably 8bit LSB.

Also I get low framerate.

Attached kernel patches, and sample image.

Commands used in testing:

GST_DEBUG=3 gst-launch-1.0 -v v4l2src device=/dev/video3 ! "video/x-raw,width=1440,height=1088,format=GRAY8" ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! queue ! rtpjpegpay ! udpsink host=192.168.1.164 port=18888 sync=false

GST_DEBUG=3 gst-launch-1.0 -v v4l2src device=/dev/video3 num-buffers=1 ! "video/x-raw,width=1440,height=1088,format=GRAY8" ! pngenc ! filesink location=a.png

Kernel dump with registers:

[ 1236.732055] Custom device pixelformat 0x30535641 'AVS Encoded Stream'
[ 1236.732086] Custom device pixelformat 0x58564944 'DIVX Encoded Stream'
[ 1236.732102] Custom device pixelformat 0x30305652 'RV Encoded Stream'
[ 1236.732110] Custom device pixelformat 0x30365056 'VP6 Encoded Stream'
[ 1236.732117] Custom device pixelformat 0x304b5053 'SPK Encoded Stream'
[ 1236.732138] Custom device pixelformat 0x43564548 'H265 HEVC Encoded Stream'
[ 1236.732147] Custom device pixelformat 0x4f474f4c 'Logo'
[ 1236.732260] Custom device pixelformat 0x3231544e '4:2:0 2 Planes Y/CbCr'
[ 1236.737350] mxc-jpeg 58400000.jpegdec: Descriptor for dec/enc: ffff000008e57000 0x96002000
[ 1236.737359] mxc-jpeg 58400000.jpegdec: Descriptor for config phase: ffff000008e58000 0x96003000
[ 1236.737365] mxc-jpeg 58400000.jpegdec: Configuration stream: ffff000008e59000 0x96004000
[ 1236.738114] mxc-jpeg 58450000.jpegenc: Descriptor for dec/enc: ffff000008e57000 0x96002000
[ 1236.738120] mxc-jpeg 58450000.jpegenc: Descriptor for config phase: ffff000008e58000 0x96003000
[ 1236.738126] mxc-jpeg 58450000.jpegenc: Configuration stream: ffff000008e59000 0x96004000
[ 1236.739424] mxc-isi 58140000.isi: mxc_isi_capture_open, ISI4
[ 1236.770964] mxc-isi 58140000.isi: mxc_isi_capture_release
[ 1236.776120] mxc-isi 58100000.isi: mxc_isi_m2m_open, ISI0
[ 1236.781384] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_out
[ 1236.781426] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_out
[ 1236.781455] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_out
[ 1236.781466] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_out
[ 1236.781494] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_out
[ 1236.781520] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_out
[ 1236.781619] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781648] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781675] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781702] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781728] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781739] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781769] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781780] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781807] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.781832] mxc-isi 58100000.isi: mxc_isi_m2m_enum_fmt_vid_cap
[ 1236.782504] mxc-isi 58100000.isi: mxc_isi_m2m_release
[ 1236.802959] mxc-isi 58140000.isi: mxc_isi_capture_open, ISI4
[ 1236.835723] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835741] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835763] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835774] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835785] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835795] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835805] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835817] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835827] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835837] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835848] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835869] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835882] mxc-isi 58140000.isi: mxc_isi_cap_enum_fmt_mplane
[ 1236.835896] mxc-isi 58140000.isi: mxc_isi_cap_g_selection
[ 1236.836717] mxc-isi 58140000.isi: mxc_isi_cap_s_fmt_mplane, fmt=0x59455247
[ 1236.838605] mxc-isi 58140000.isi: cap_vb2_queue_setup, buf_n=2, size=1566720
[ 1236.843381] mxc-isi 58140000.isi: cap_vb2_buffer_prepare
[ 1236.843422] mxc-isi 58140000.isi: cap_vb2_buffer_prepare
[ 1236.843460] mxc-isi 58140000.isi: mxc_isi_cap_streamon
[ 1236.843484] mxc-mipi-csi2 58247000.csi: width=1440, height=1088, fmt.code=0x300f
[ 1236.848509] bypass csc
[ 1236.849568] input fmt RG10
[ 1236.850971] output fmt GREY
[ 1236.852491] mxc-isi 58140000.isi: mxc_isi_channel_set_scaling: no scale
[ 1236.852506] mxc-isi 58140000.isi: cap_vb2_start_streaming
[ 1236.855521] mxc-isi 58140000.isi: cap_vb2_start_streaming: num_plane=0 discard_size=1566720 discard_buffer=ffff000025181000
[ 1237.159955] mxc-isi 58140000.isi: ISI CHNLC register dump, isi4
[ 1237.159964] mxc-isi 58140000.isi: CHNL_CTRL 0x0h = 0xe0ff0003
[ 1237.159972] mxc-isi 58140000.isi: CHNL_IMG_CTRL 0x4h = 0x b000001
[ 1237.159979] mxc-isi 58140000.isi: CHNL_OUT_BUF_CTRL 0x8h = 0x 8000
[ 1237.159986] mxc-isi 58140000.isi: CHNL_IMG_CFG 0xCh = 0x 44005a0
[ 1237.159993] mxc-isi 58140000.isi: CHNL_IER 0x10h = 0x3dff0000
[ 1237.160000] mxc-isi 58140000.isi: CHNL_STS 0x14h = 0x 200
[ 1237.160007] mxc-isi 58140000.isi: CHNL_SCALE_FACTOR 0x18h = 0x10001000
[ 1237.160013] mxc-isi 58140000.isi: CHNL_SCALE_OFFSET 0x1Ch = 0x 0
[ 1237.160020] mxc-isi 58140000.isi: CHNL_CROP_ULC 0x20h = 0x 0
[ 1237.160026] mxc-isi 58140000.isi: CHNL_CROP_LRC 0x24h = 0x 0
[ 1237.160032] mxc-isi 58140000.isi: CHNL_CSC_COEFF0 0x28h = 0x 0
[ 1237.160038] mxc-isi 58140000.isi: CHNL_CSC_COEFF1 0x2Ch = 0x 0
[ 1237.160044] mxc-isi 58140000.isi: CHNL_CSC_COEFF2 0x30h = 0x 0
[ 1237.160051] mxc-isi 58140000.isi: CHNL_CSC_COEFF3 0x34h = 0x 0
[ 1237.160057] mxc-isi 58140000.isi: CHNL_CSC_COEFF4 0x38h = 0x 0
[ 1237.160063] mxc-isi 58140000.isi: CHNL_CSC_COEFF5 0x3Ch = 0x 0
[ 1237.160069] mxc-isi 58140000.isi: CHNL_ROI_0_ALPHA 0x40h = 0x 0
[ 1237.160076] mxc-isi 58140000.isi: CHNL_ROI_0_ULC 0x44h = 0x 0
[ 1237.160082] mxc-isi 58140000.isi: CHNL_ROI_0_LRC 0x48h = 0x 0
[ 1237.160088] mxc-isi 58140000.isi: CHNL_ROI_1_ALPHA 0x4Ch = 0x 0
[ 1237.160094] mxc-isi 58140000.isi: CHNL_ROI_1_ULC 0x50h = 0x 0
[ 1237.160101] mxc-isi 58140000.isi: CHNL_ROI_1_LRC 0x54h = 0x 0
[ 1237.160107] mxc-isi 58140000.isi: CHNL_ROI_2_ALPHA 0x58h = 0x 0
[ 1237.160114] mxc-isi 58140000.isi: CHNL_ROI_2_ULC 0x5Ch = 0x 0
[ 1237.160120] mxc-isi 58140000.isi: CHNL_ROI_2_LRC 0x60h = 0x 0
[ 1237.160126] mxc-isi 58140000.isi: CHNL_ROI_3_ALPHA 0x64h = 0x 0
[ 1237.160132] mxc-isi 58140000.isi: CHNL_ROI_3_ULC 0x68h = 0x 0
[ 1237.160139] mxc-isi 58140000.isi: CHNL_ROI_3_LRC 0x6Ch = 0x 0
[ 1237.160145] mxc-isi 58140000.isi: CHNL_OUT_BUF1_ADDR_Y 0x70h = 0xa6800000
[ 1237.160151] mxc-isi 58140000.isi: CHNL_OUT_BUF1_ADDR_U 0x74h = 0x 0
[ 1237.160158] mxc-isi 58140000.isi: CHNL_OUT_BUF1_ADDR_V 0x78h = 0x 0
[ 1237.160164] mxc-isi 58140000.isi: CHNL_OUT_BUF_PITCH 0x7Ch = 0x 5a0
[ 1237.160170] mxc-isi 58140000.isi: CHNL_IN_BUF_ADDR 0x80h = 0x 0
[ 1237.160176] mxc-isi 58140000.isi: CHNL_IN_BUF_PITCH 0x84h = 0x 0
[ 1237.160182] mxc-isi 58140000.isi: CHNL_MEM_RD_CTRL 0x88h = 0x 0
[ 1237.160189] mxc-isi 58140000.isi: CHNL_OUT_BUF2_ADDR_Y 0x8Ch = 0xa6400000
[ 1237.160195] mxc-isi 58140000.isi: CHNL_OUT_BUF2_ADDR_U 0x90h = 0x 0
[ 1237.160201] mxc-isi 58140000.isi: CHNL_OUT_BUF2_ADDR_V 0x94h = 0x 0
[ 1237.160213] mxc-md camera: begin graph walk at 'mxc_isi.4.capture'
[ 1237.160221] mxc-md camera: walk: pushing 'mxc_isi.4' on stack
[ 1237.160228] mxc-md camera: walk: skipping entity 'mxc_isi.4.capture' (already seen)
[ 1237.160235] mxc-md camera: walk: pushing 'mxc-mipi-csi2.1' on stack
[ 1237.160242] mxc-md camera: walk: skipping entity 'mxc_isi.4' (already seen)
[ 1237.160248] mxc-md camera: walk: pushing 'imx296 6-001a' on stack
[ 1237.160255] mxc-md camera: walk: skipping entity 'mxc-mipi-csi2.1' (already seen)
[ 1237.160261] mxc-md camera: walk: returning entity 'imx296 6-001a'
[ 1237.252430] mxc-md camera: walk: returning entity 'mxc-mipi-csi2.1'
[ 1237.252442] mxc-mipi-csi2 58247000.csi: mipi_csi2_s_stream: 1, csi2dev: 0x0
[ 1237.252490] mxc-mipi-csi2 58247000.csi: width=1440, height=1088, fmt.code=0x300f
[ 1237.252513] MIPI CSI2 HC register dump, mipi csi1
[ 1237.255957] MIPI CSI2 HC num of lanes 0x100 = 0x0
[ 1237.259710] MIPI CSI2 HC dis lanes 0x104 = 0xe
[ 1237.263489] MIPI CSI2 HC BIT ERR 0x108 = 0x0
[ 1237.267254] MIPI CSI2 HC IRQ STATUS 0x10C = 0x8
[ 1237.271094] MIPI CSI2 HC IRQ MASK 0x110 = 0x1ff
[ 1237.275033] MIPI CSI2 HC ULPS STATUS 0x114 = 0x0
[ 1237.278804] MIPI CSI2 HC DPHY ErrSotHS 0x118 = 0x0
[ 1237.282569] MIPI CSI2 HC DPHY ErrSotSync 0x11c = 0x0
[ 1237.286342] MIPI CSI2 HC DPHY ErrEsc 0x120 = 0x0
[ 1237.290101] MIPI CSI2 HC DPHY ErrSyncEsc 0x124 = 0x0
[ 1237.293873] MIPI CSI2 HC DPHY ErrControl 0x128 = 0x0
[ 1237.297632] MIPI CSI2 HC DISABLE_PAYLOAD 0x12C = 0x0
[ 1237.301405] MIPI CSI2 HC DISABLE_PAYLOAD 0x130 = 0x0
[ 1237.305167] MIPI CSI2 HC IGNORE_VC 0x180 = 0x0
[ 1237.308936] MIPI CSI2 HC VID_VC 0x184 = 0x0
[ 1237.312700] MIPI CSI2 HC FIFO_SEND_LEVEL 0x188 = 0x0
[ 1237.316468] MIPI CSI2 HC VID_VSYNC 0x18C = 0x0
[ 1237.320253] MIPI CSI2 HC VID_SYNC_FP 0x190 = 0x0
[ 1237.324026] MIPI CSI2 HC VID_HSYNC 0x194 = 0x0
[ 1237.327777] MIPI CSI2 HC VID_HSYNC_BP 0x198 = 0x0
[ 1237.332315] MIPI CSI2 CSR register dump
[ 1237.334847] MIPI CSI2 CSR PLM_CTRL 0x000 = 0x801
[ 1237.338784] MIPI CSI2 CSR PHY_CTRL 0x004 = 0x2000af
[ 1237.343703] MIPI CSI2 CSR PHY_Status 0x008 = 0x1
[ 1237.347468] MIPI CSI2 CSR PHY_Test_Status 0x010 = 0x0
[ 1237.351218] MIPI CSI2 CSR PHY_Test_Status 0x014 = 0x0
[ 1237.355726] MIPI CSI2 CSR PHY_Test_Status 0x018 = 0x0
[ 1237.359482] MIPI CSI2 CSR PHY_Test_Status 0x01C = 0x0
[ 1237.363238] MIPI CSI2 CSR PHY_Test_Status 0x020 = 0x0
[ 1237.366985] MIPI CSI2 CSR VC Interlaced 0x030 = 0x0
[ 1237.370733] MIPI CSI2 CSR Data Type Dis 0x038 = 0x0
[ 1237.374482] MIPI CSI2 CSR 420 1st type 0x040 = 0x0
[ 1237.378232] MIPI CSI2 CSR Ctr_Ck_Rst_Ctr 0x044 = 0x1
[ 1237.381984] MIPI CSI2 CSR Stream Fencing 0x048 = 0x0
[ 1237.385736] MIPI CSI2 CSR Stream Fencing 0x04C = 0x0
[ 1237.389489] mxc-md camera: walk: returning entity 'mxc_isi.4'
[ 1237.389492] mxc-md camera: walk: returning entity 'mxc_isi.4.capture'
[ 1237.389495] mxc-isi 58140000.isi: mxc_isi_pipeline_enable ,entity is no v4l2, mxc_isi.4.capture
[ 1237.389738] vb2_common_vm_open: ffff8008781c4d48, refcount: 2, vma: ffff843a4000-ffff84523000
[ 1237.389742] vb2_dc_mmap: mapped dma addr 0xa6400000 at 0xffff843a4000, size 1568768
[ 1238.347293] mxc-isi 58140000.isi: cap_vb2_buffer_prepare
[ 1238.351766] mxc-isi 58140000.isi: mxc_isi_cap_streamoff
[ 1238.351776] mxc-md camera: begin graph walk at 'mxc_isi.4.capture'
[ 1238.351782] mxc-md camera: walk: pushing 'mxc_isi.4' on stack
[ 1238.351787] mxc-md camera: walk: skipping entity 'mxc_isi.4.capture' (already seen)
[ 1238.351791] mxc-md camera: walk: pushing 'mxc-mipi-csi2.1' on stack
[ 1238.351795] mxc-md camera: walk: skipping entity 'mxc_isi.4' (already seen)
[ 1238.351800] mxc-md camera: walk: pushing 'imx296 6-001a' on stack
[ 1238.351804] mxc-md camera: walk: skipping entity 'mxc-mipi-csi2.1' (already seen)
[ 1238.351808] mxc-md camera: walk: returning entity 'imx296 6-001a'
[ 1238.392256] mxc-md camera: walk: returning entity 'mxc-mipi-csi2.1'
[ 1238.392268] mxc-mipi-csi2 58247000.csi: mipi_csi2_s_stream: 0, csi2dev: 0x0
[ 1238.392355] mxc-md camera: walk: returning entity 'mxc_isi.4'
[ 1238.392363] mxc-md camera: walk: returning entity 'mxc_isi.4.capture'
[ 1238.392371] mxc-isi 58140000.isi: mxc_isi_pipeline_enable ,entity is no v4l2, mxc_isi.4.capture
[ 1238.392382] mxc-isi 58140000.isi: cap_vb2_stop_streaming
[ 1238.393247] vb2_common_vm_close: ffff8008781c4d48, refcount: 3, vma: ffff843a4000-ffff84523000
[ 1238.396300] mxc-isi 58140000.isi: mxc_isi_capture_release

[korneliuszosmen]

I'm afraid I cannot. It uses proprietary libraries, and not optimalized enough (720p <15fps)

It just reads uint16_t from buffer and writes it to second buffer after shifting.

[joanxie]

for capturing raw 10bits data, you need to  set RAW16 in ISI , and right shift 4bits to get the correct data.

[wanbenzhou]

"for capturing raw 10bits data, you need to  set RAW16 in ISI , and right shift 4bits to get the correct data."

How to do it,  could you tell me  which file  and which function?

[korneliuszosmen]

I've done in custom gstreamer block.

[wanbenzhou]

How to do it in gstreamer? Could you send me?

[korneliuszosmen]

Isn't it possible to make ISI to generate data in proper format?

[joanxie]

currently, as I known, customer needs to shift it by themselves

[carolyn_zhang]

Hi, Joan,

So far as I know, it is not support RAW in early BSP. Which version started to support RAW?

Do we need to add RAW in mxc_isi_src_formats and mxc_isi_out_formats?

[korneliuszosmen]

I was thinking if CSC block can be active when output is RAW