Can only get imx*compositor to work with videotestsrc in application

[tyounger]

I have been having some challenges creating a fully functioning Gstreamer application.

I have prototyped all the functions using gst-launch-1.0, and have had success, but my final solution needs to be compiled into an application.

I have been able to get every single element in my test pipelines to function in a compiled application, the issue is that I can't get all of the elements to work together.

For example, I have written the following pipeline into my application:

gst-launch-1.0 rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxeglvivsink borderless-window=true sync=false

After sorting out sometimes pads, my application works great.

I have also compiled this into my application:

gst-launch-1.0 -v imxipucompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 \
sink_1::xpos=512 sink_1::ypos=396 \
! video/x-raw,width=1024,height=768 ! imxeglvivsink native-display=:0 borderless-window=true sync=false \
videotestsrc pattern=0 ! video/x-raw,width=512,height=384 ! queue ! comp.sink_0 \
videotestsrc pattern=1 ! video/x-raw,width=512,height=384 ! queue ! comp.sink_1

After sorting out request pads, my application works great.

I have also compiled this into my application:

gst-launch-1.0 -v $MIXER name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=512 sink_0::height=288 \
sink_1::xpos=512 sink_1::ypos=0 sink_1::width=512 sink_1::height=288 \
sink_2::xpos=0 sink_2::ypos=288 sink_2::width=512 sink_2::height=288 \
sink_3::xpos=512 sink_3::ypos=288 sink_3::width=512 sink_3::height=288 \
! video/x-raw,width=1024,height=576 ! imxipuvideotransform ! imxeglvivsink borderless-window=true sync=false \
rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_0
rtspsrc location=rtsp://10.1.0.94:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_1
rtspsrc location=rtsp://10.1.0.94:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_2
rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_3

When $MIXER is set to "videomixer", my application functions correctly, however CPU usage is high. When $MIXER is set to imxg2dcompositor or imxipucompositor, the pipeline does not work in my application.

What do I need to do differently with the imx compositor than I do with the videomixer?

If I can get the imx compositor to work with h264 live video, I would expect I could have a simpler pipeline:

gst-launch-1.0 -v imxg2dcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=512 sink_0::height=288 \
sink_1::xpos=512 sink_1::ypos=0 sink_1::width=512 sink_1::height=288 \
sink_2::xpos=0 sink_2::ypos=384 sink_2::width=512 sink_2::height=288 sink_2::keep-aspect-ratio=false sink_2::rotation=2 \
sink_3::xpos=512 sink_3::ypos=384 sink_3::width=512 sink_3::height=288 sink_3::keep-aspect-ratio=false sink_3::rotation=2 \
! "video/x-raw, width=1024, height=768" ! imxeglvivsink borderless-window=true sync=false \
rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_0 \
rtspsrc location=rtsp://10.1.0.94:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_1 \
rtspsrc location=rtsp://10.1.0.92:554/SubStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_2 \
rtspsrc location=rtsp://10.1.0.94:554/SubStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_3

This does not work with the videomixer element though as the videomixer element won't scale.

Does anyone have any pointers on how the videomixer and imx*compositor differ, especially when it comes to the difference between an RTSP live video feed and a videotestsrc?

[sjoseph]

Hi all,

I am also working on a similar pipeline.

@tyounger

Could you tell me how you dealt with the request pad for videomixer/compositor element in the c application? I have to set the xpos and ypos of videomixer element (for sink_0 to sink_3). 

Anything would be helpful. 

[tyounger]

    // Connect the multiqueue to the compositor
    for (size_t i = 0; i < NUM_CAMERA; i++)
    {
        char mq_pad_name[10];
        const unsigned int xpositions[NUM_CAMERA] = {0, m_screen_width / 2U, 0, m_screen_width / 2U};
        const unsigned int ypositions[NUM_CAMERA] = {0, 0, m_video_height / 2U, m_video_height / 2U};
        GstPad *mix_pad = gst_element_get_request_pad(mixer, "sink_%u");
        snprintf(mq_pad_name, sizeof(mq_pad_name), "src_%d", i);
        // multique does not have source request pads, the multiqueue source pads are sometimes pads 
        GstPad *mq_pad = gst_element_get_static_pad(multiqueue, mq_pad_name);
        g_object_set(mix_pad,
            "xpos", xpositions[i],
            "ypos", ypositions[i],
            "width", m_screen_width / 2U, /* Software compositor doesn't like width */
            "height", m_video_height / 2U, /* Software compositor doesn't like height */
            "keep-aspect-ratio", FALSE, /* Software compositor doesn't like aspect ratio */
            NULL);
        if (GST_PAD_LINK_OK != gst_pad_link(mq_pad, mix_pad))
        {
            LogError("Error linking mixer");
            gst_object_unref(m_gst_pipeline);
        }

        // cleanup pads
        gst_object_unref(mix_pad);
        gst_object_unref(mq_pad);
        mix_pad = NULL;
        mq_pad = NULL;
    }
 

[tyounger]

My pipeline seems to be working right now.

I believe the issue I was having was due to duplicating inputs. My requirements are to display 4 cameras simultaneously. Due to the fact I have only two cameras of the correct type for testing, I was duplicating the cameras.

Although this worked with the software compositor, it was not working with imxg2dcompositor.

I added an additional feature to my application where I substitute a .PNG in place of a missing camera. After this feature was in place, I tried swapping imxg2dcompositor in for the software compositor, and under these conditions it works when I have the two real cameras selected, and two non-existant cameras (which are dynamically swapped to filesrc ! pngdec ! imagefreeze).

The remaining issue I have though is of reliability. Sometimes it draws to the screen correctly, other times the screen is blank. I have yet to be able to determine programatically if it is displaying correctly. The build of gstreamer I have been given to use issues some errors when the library is loaded; I am hoping that once those errors are fixed, the remaining issue of reliability goes away. The new build of gstreamer was made in order to add the imxg2dtextoverlay element, which I am using to display the camera name over the image.

[igorpadykov]

Hi Tom

seems gstreamer-imx plugins used in the case (like imxvpudec, imxeglvivsink),

issue can be posted on Issues · Freescale/gstreamer-imx · GitHub 

Alternatively one can try with nxp gst1.0-plugins

imx-gst1.0-plugin - i.MX Gstreamer 1.0 Plugin 

Usage is described in Linux Guide included in Linux L4.14.98_2.0.0 Documentation

Best regards
igor
-----------------------------------------------------------------------------------------------------------------------
Note: If this post answers your question, please click the Correct Answer button. Thank you!
-----------------------------------------------------------------------------------------------------------------------