Using Gstreamer

Philippe Coval

Created: 2023-09-22 Fri 20:23

1. From theory …
2. … to practice
3. Exercices
4. References
5. Notes

1. From theory …

1.1. What is GStreamer ?

Framework for creating multimedia applications
Gst Handles: Audio, Video, Image, data.
Libre Software : Licensed on LGPL since 2001
Crossplatform : Linux, Windows, OSX, Android?
Competitors: Microsoft's DirectShow, Apple QuickTime, VideoLan …

1.2. What does gst provide ?

Libraries and tools for developing and creating Multimedia applications
player, processing, transport etc
API : Internal and External
Built in C using GLib
Binding to many languages : C++, Python, Ruby, Java, C# …
Integration with Toolkits, browsers etc
Extensible with plugins

1.3. Architecture

1.4. How does it work ?

It's a pipeline of different elements (modules) such as :
- input (capture, files..)
- protocols/transport
- codecs / containers
- filters
- output (render, files…)

 _____________     ____________     ____________
|             |   |            |   |            |
| File source |->-| WAV Parser |->-| Audio Sink |
|_____________|   |____________|   |____________|

1.5. Pipeline of connected elements

Elements do one task once and do it well
see it as black box with input and output (has its own buffer)
provide pads to be connected together :
- input is called sink
- output is called source
transform and processing
examples : jpeg decoder, camera, display

1.6. Pipeline

Chain : linked elements (through their pad)
data flow from sources to sinks
distributed load
bus for communication and synchronization (clocks)
states machine : construction, pause, playing
listens to external events

1.7. connections

Pad has some capabilities
negotiation / priorities
if not connectable
- maybe an extra element can do the adaptation

1.8. smartness

*bin are group of elements (guess flow)
manage priorities
open decode display using : playbin
other bins : de/en/codebin camerabin …
example : audio/video player (without controls) :

gst-launch playbin uri="${url}"

1.9. plugins

base : minimal functionnal behaviour, used to create plugins :
- GstElement, GstBaseSrc, GstBaseSink, GstBaseTransform, GstBin
good : good-quality plug-ins under the LGPL license.
bad : not (yet) endorsed
ugly : legal issues

1.10. Hardware acceleration

Isolated into backend plugins
Using generic API for HW adaptation (drivers)
- libva (intel and some ARM)
- openmax (broadcom)
Or specific per VPU (ARM) or GPU

2. … to practice

2.1. usage : on Debian based

sudo apt-get install gstreamer1.0-tools \
   gstreamer1.0-plugins-base \
   gstreamer1.0-plugins-good \
   gstreamer1.0-plugins-bad \
   gstreamer1.0-plugins-ugly

man gst-launch-1.0

# extra plugins (hardware dependents?)
apt-get install apt-file && apt-file update
apt-file search libgst*.so

2.2. usage : on OpenEmbedded based

ie : yocto for edison
rebuild or use opkg

cat /etc/opkg/base-feeds.conf
opkg list | grep gst
opkg install gstreamer1.0-plugins-good-video4linux2

2.3. usage : check tests

gst-inspect-1.0

# demo pipeline that displays nothing
gst-launch-1.0 videotestsrc  num-buffers=25 ! fakesink

# Display demo video
gst-launch-1.0 videotestsrc num-buffers=25 ! autovideosink
gst-launch-1.0 videotestsrc num-buffers=25 ! fbdevsink
# display window
gst-launch-1.0 videotestsrc ! autovideosink

# tell type
gst-typefind-1.0 /usr/share/sounds/alsa/Noise.wav
/usr/share/sounds/alsa/Noise.wav - audio/x-wav

2.4. tool : gstlaunch

parse pipe line command line
each elements are separated by '!' character
contruct and start playing it
example : dump pipeline : send video stream to null device

gst-launch-1.0 videotestsrc  num-buffers=10000 ! fakesink

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
...

2.5. example : audio output

  _____________     ____________     ____________
 |             |   |            |   |            |
 | File source |->-| WAV Parser |->-| Audio Sink |
 |_____________|   |____________|   |____________|

gst-launch-1.0 \
  filesrc location=/usr/share/sounds/alsa/Noise.wav \
  ! wavparse \
  ! alsasink

2.6. gst-inspect

list all elements

gst-inspect-1.0 | grep 'src:'

alsa:  alsasrc: Audio source (ALSA)
(...)
rtsp:  rtspsrc: RTSP packet receiver
(...)
autodetect:  autovideosrc: Auto video source

2.7. gst-inspect

element introspection
list all pads per element and caps

gst-inspect-1.0 v4l2src
(...)
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
  video/x-raw
    format: RGB15
     width: [ 1, 32768 ]
     height: [ 1, 32768 ]
     framerate: [ 0/1, 100/1 ]
(...)
Element Properties:
(...)
device : Device location
      flags: readable, writable
      String. Default: "/dev/video0"

2.8. specify argument

gst-launch-1.0 -v v4l2src device=/dev/video0 ! autovideosink
Setting pipeline to PAUSED ...
(...)
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstVaapiSink:autovideosink0-actual-sink-vaapi.GstPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
(...)

2.9. specify capabilities

insert desired data specification between elements

gst-launch-1.0 -v \
v4l2src device=/dev/video0 \
! video/x-raw,width=320 \
! autovideosink

source here is video4linux v2 (1st detected webcam)
selected size is 320 width (320*240 native)
outoput display will be selected by system

2.10. sync / A/V

tee duplicate the named stream
queues are to prevent underrun or deadlocking
one queue start a new thread with its own buffer
This pipeline saves displayed frames from webcam (in raw)

gst-launch-1.0 -v \
v4l2src \
! tee name=src \
  src. ! queue ! autovideosink \
  src. ! queue ! filesink location="out.tmp"

2.11. debugging

GST_DEBUG=2 \
gst-launch-1.0 \
-v --gst-debug=v4l2:5 \
  v4l2src ! fakesink num-buffers=1
(...)
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, (...)
(...)
INFO v4l2 gstv4l2object.c:1247:gst_v4l2_object_fill_format_list:<v4l2src0> got 5 format(s):
(...)

-v : verbose list pads
–gst-debug=${plugin}
"GST _ DEBUG" : global env

2.12. trace

export GST_DEBUG='10'
export GST_DEBUG_DUMP_DOT_DIR="."
export GST_DEBUG_OPTIONS=pretty-tags
export GST_TRACE=all

gst-launch-1.0 -v audiotestsrc num-buffers=16 \
  ! vorbisenc ! oggmux ! filesink location="tmp.oga"

# *-gst-launch.*.dot

which dot || apt-get install graphviz
for file in *.dot ; do dot -Tsvg "$file" > "$file.svg" ; done
xdg-open *PAUSED_PLAYING*.svg
# states " NULL_READY > READY_PAUSED > PAUSED_PLAYING > PLAYING_PAUSED > PAUSED_READY"

2.13.

3. Exercices

Grab online MSS stream to file
capture webcam to ogv file
record microphone to OGG/vorbis
play DVD vob files
save each frame of video in separate images
stream webcam input to rtp stream
transcode video to Theora

3.1. example : capture

gst-launch \
   v4l2src ! "video/x-raw-yuv",width=320,height=240  \
   ! theoraenc ! oggmux \
   ! filesink  location="video.ogv"

3.2. example : grab MSS stream to file

gst-launch-1.0 \
  mmssrc location="mms://url/stream.asf" \
  ! filesink location="mss.wmv"

3.3. example : save images from video


gst-launch-1.0 \
    filesrc location="video.avi" \
    ! decodebin ! queue ! autovideoconvert ! pngenc \
    ! multifilesink location="%08d.png"

3.4. example : mic ogg vorbis

gst-launch-1.0 \
  autoaudiosrc ! audioconvert ! vorbisenc ! oggmux \
  ! filesink location="vorbis.oga"

3.5. example : play DVD vobs

gst-launch-1.0 \
  filesrc location="dvd.vob" \
  ! mpegpsdemux name=demux \
  ! mpegvideoparse \
  ! queue \
  ! mpeg2dec \
  ! autovideosink \
    demux. \
    ! queue \
    ! mad \
    ! audioconvert ! audioresample \
    ! alsasink

3.6. example : stream webcam

gst-launch-1.0 -v v4l2src \
! video/x-raw,width=320,height=240 \
! rtpvrawpay \
! udpsink host=127.0.0.1 port=5004

gst-launch-1.0 -v udpsrc port=5004 \
! application/x-rtp, media=video, clock_rate=9000, \
  encoding-name=RAW, sampling=YCbCr-4:2:0, \
  depth='(string)8',width='(string)320',height='(string)240',
  payload=96 \
! rtpvrawdepay ! decodebin ! autovideosink

3.7. example : transcode to theora

gst-launch-1.0 filesrc location="in.wmv" \
  ! decodebin name=d \
  { oggmux name=mux max-delay=500000000 max-page-delay=500000000 \
  ! filesink  location="out.theora.ogv" } \
  { d. ! queue ! ffmpegcolorspace ! theoraenc bitrate=300  \
  ! queue ! mux. }

4. References

5. Notes

Created with emacs, org-mode, reveal.js, ox-reveal
Thanks: #GraphikLabor, EGL, SOSG
License: CC-BY-SA @RzrFreeFr 2016

Using Gstreamer

Philippe Coval

Table of Contents

1. From theory …

1.1. What is GStreamer ?

1.2. What does gst provide ?

1.3. Architecture

1.4. How does it work ?

1.5. Pipeline of connected elements

1.6. Pipeline

1.7. connections

1.8. smartness

1.9. plugins

1.10. Hardware acceleration

2. … to practice

2.1. usage : on Debian based

2.2. usage : on OpenEmbedded based

2.3. usage : check tests

2.4. tool : gstlaunch

2.5. example : audio output

2.6. gst-inspect

2.7. gst-inspect

2.8. specify argument

2.9. specify capabilities

2.10. sync / A/V

2.11. debugging

2.12. trace

2.13.

3. Exercices

3.1. example : capture

3.2. example : grab MSS stream to file

3.3. example : save images from video

3.4. example : mic ogg vorbis

3.5. example : play DVD vobs

3.6. example : stream webcam

3.7. example : transcode to theora

4. References

5. Notes