Courtesy: Qualcomm

Qualcomm Linux sample apps – building blocks for AI inference and video in IoT applications (Part 1 of 2)

In this post we’ll explore the first two building-block applications.

1. Multi-camera streaming

The command-line application gst-multi-camera-example demonstrates streaming from two camera sensors simultaneously. It can apply side-by-side composition of the video streams to show on a display device, or it can encode and store the streams to files.

The application pipeline looks like this:

The application supports two configurations:

1. Composition and display – The qtimmfsrc plugin on camera 0 and camera 1 capture the data from the two camera sensors. qtivcomposer performs the composition, then waylandsink displays the streams side by side on the screen.
2. Video encoding – The qtimmfsrc plugin on camera 0 and camera 1 captures the data from the two camera sensors and passes it to the v4l2h264enc plugin. The plugin encodes and compresses the camera streams to H.264 format, then hands them off for parsing and multiplexing using the h264parse and mp4mux plugins, respectively. Finally, the streams are handed off to the filesink plugin, which saves them as files.

Here’s an example of the output from the first configuration: Right side image is monochrome, as second camera sensor on development kit is monochrome.

When would you use this application?

gst-multi-camera-example is a building block for capturing data from two camera sensors, with options for either composing and displaying the video streams or encoding and storing the streams to files. You can use this sample app as the basis for your own camera capture/encoding applications, including dashcams and stereo cameras.

2. Video wall – Multi-channel video decode and display

The command-line application gst-concurrent-videoplay-composition facilitates concurrent video decode and playback for AVC-coded videos. The app performs composition on multiple video streams coming from files or the network (e.g., IP cameras) for display as a video wall.

The application can take multiple (such as 4 or 8) video files as input, decode all the compressed videos, scale them and compose them as a video wall. The application requires at least one input video file, in MP4 format with an AVC codec.

The application pipeline looks like this for 4 channels:

Each channel uses plugins to perform the following processing:

1. Reads compressed video data from a file using filesrc.
2. Demultiplexes the file with qtdemux.
3. Parses H.264 video streams using h264parse.
4. Decodes the streams using v4l2h264dec.
5. The decoded streams from all channels are then composed together using qtivcomposer and displayed using waylandsink.

Here’s an example of using the app gst-concurrent-videoplay-composition on 4 video streams:

When would you use this application?

With gst-concurrent-videoplay-composition you can decode multiple compressed video streams, then compose them into a video wall; for example, in retail spaces and digital signage. As an edge box for video surveillance, you can capture input from multiple IP cameras and display it in a single screen. In a video conferencing application, you can process and display feeds from multiple people on the call, with each participant streaming a video.