Using Gstreamer

Philippe Coval

Created: 2020-10-16 Fri 01:33

1 From theory …

1.1 What is GStreamer ?

  • Framework for creating multimedia applications
  • Gst Handles: Audio, Video, Image, data.
  • Libre Software : Licensed on LGPL since 2001
  • Crossplatform : Linux, Windows, OSX, Android?
  • Competitors: Microsoft's DirectShow, Apple QuickTime, VideoLan …

1.2 What does gst provide ?

  • Libraries and tools for developing and creating Multimedia applications
  • player, processing, transport etc
  • API : Internal and External
  • Built in C using GLib
  • Binding to many languages : C++, Python, Ruby, Java, C# …
  • Integration with Toolkits, browsers etc
  • Extensible with plugins

1.3 Architecture

675px-GStreamer_overview.svg.png

1.4 How does it work ?

  • It's a pipeline of different elements (modules) such as :
    • input (capture, files..)
    • protocols/transport
    • codecs / containers
    • filters
    • output (render, files…)
 _____________     ____________     ____________
|             |   |            |   |            |
| File source |->-| WAV Parser |->-| Audio Sink |
|_____________|   |____________|   |____________|

1.5 Pipeline of connected elements

  • Elements do one task once and do it well
  • see it as black box with input and output (has its own buffer)
  • provide pads to be connected together :
    • input is called sink
    • output is called source
  • transform and processing
  • examples : jpeg decoder, camera, display

1.6 Pipeline

  • Chain : linked elements (through their pad)
  • data flow from sources to sinks
  • distributed load
  • bus for communication and synchronization (clocks)
  • states machine : construction, pause, playing
  • listens to external events

Sorry, your browser does not support SVG.

1.7 connections

  • Pad has some capabilities
  • negotiation / priorities
  • if not connectable
    • maybe an extra element can do the adaptation

1.8 smartness

  • *bin are group of elements (guess flow)
  • manage priorities
  • open decode display using : playbin
  • other bins : de/en/codebin camerabin …
  • example : audio/video player (without controls) :
gst-launch playbin uri="${url}"

1.9 plugins

  • base : minimal functionnal behaviour, used to create plugins :
    • GstElement, GstBaseSrc, GstBaseSink, GstBaseTransform, GstBin
  • good : good-quality plug-ins under the LGPL license.
  • bad : not (yet) endorsed
  • ugly : legal issues

1.10 Hardware acceleration

  • Isolated into backend plugins
  • Using generic API for HW adaptation (drivers)
    • libva (intel and some ARM)
    • openmax (broadcom)
  • Or specific per VPU (ARM) or GPU

2 … to practice

2.1 usage : on Debian based

sudo apt-get install gstreamer1.0-tools \
   gstreamer1.0-plugins-base \
   gstreamer1.0-plugins-good \
   gstreamer1.0-plugins-bad \
   gstreamer1.0-plugins-ugly

man gst-launch-1.0

# extra plugins (hardware dependents?)
apt-get install apt-file && apt-file update
apt-file search libgst*.so

2.2 usage : on OpenEmbedded based

  • ie : yocto for edison
  • rebuild or use opkg
cat /etc/opkg/base-feeds.conf
opkg list | grep gst
opkg install gstreamer1.0-plugins-good-video4linux2

2.3 usage : check tests

gst-inspect-1.0

# demo pipeline that displays nothing
gst-launch-1.0 videotestsrc  num-buffers=25 ! fakesink

# Display demo video
gst-launch-1.0 videotestsrc num-buffers=25 ! autovideosink
gst-launch-1.0 videotestsrc num-buffers=25 ! fbdevsink
# display window
gst-launch-1.0 videotestsrc ! autovideosink

# tell type
gst-typefind-1.0 /usr/share/sounds/alsa/Noise.wav
/usr/share/sounds/alsa/Noise.wav - audio/x-wav

2.4 tool : gstlaunch

  • parse pipe line command line
  • each elements are separated by '!' character
  • contruct and start playing it
  • example : dump pipeline : send video stream to null device
gst-launch-1.0 videotestsrc  num-buffers=10000 ! fakesink

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
...

2.5 example : audio output

  _____________     ____________     ____________
 |             |   |            |   |            |
 | File source |->-| WAV Parser |->-| Audio Sink |
 |_____________|   |____________|   |____________|

gst-launch-1.0 \
  filesrc location=/usr/share/sounds/alsa/Noise.wav \
  ! wavparse \
  ! alsasink

2.6 gst-inspect

  • list all elements
gst-inspect-1.0 | grep 'src:'

alsa:  alsasrc: Audio source (ALSA)
(...)
rtsp:  rtspsrc: RTSP packet receiver
(...)
autodetect:  autovideosrc: Auto video source

2.7 gst-inspect

  • element introspection
  • list all pads per element and caps
gst-inspect-1.0 v4l2src
(...)
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
  video/x-raw
    format: RGB15
     width: [ 1, 32768 ]
     height: [ 1, 32768 ]
     framerate: [ 0/1, 100/1 ]
(...)
Element Properties:
(...)
device : Device location
      flags: readable, writable
      String. Default: "/dev/video0"

2.8 specify argument

gst-launch-1.0 -v v4l2src device=/dev/video0 ! autovideosink
Setting pipeline to PAUSED ...
(...)
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstVaapiSink:autovideosink0-actual-sink-vaapi.GstPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
/GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0.GstGhostPad:sink: caps = video/x-raw, format=(string)YV12, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, framerate=(fraction)10/1
(...)

2.9 specify capabilities

  • insert desired data specification between elements
gst-launch-1.0 -v \
v4l2src device=/dev/video0 \
! video/x-raw,width=320 \
! autovideosink
  • source here is video4linux v2 (1st detected webcam)
  • selected size is 320 width (320*240 native)
  • outoput display will be selected by system

2.10 sync / A/V

  • tee duplicate the named stream
  • queues are to prevent underrun or deadlocking
  • one queue start a new thread with its own buffer
  • This pipeline saves displayed frames from webcam (in raw)
gst-launch-1.0 -v \
v4l2src \
! tee name=src \
  src. ! queue ! autovideosink \
  src. ! queue ! filesink location="out.tmp"

2.11 debugging

GST_DEBUG=2 \
gst-launch-1.0 \
-v --gst-debug=v4l2:5 \
  v4l2src ! fakesink num-buffers=1
(...)
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)YUY2, (...)
(...)
INFO v4l2 gstv4l2object.c:1247:gst_v4l2_object_fill_format_list:<v4l2src0> got 5 format(s):
(...)
  • -v : verbose list pads
  • –gst-debug=${plugin}
  • "GST _ DEBUG" : global env

2.12 trace

export GST_DEBUG='10'
export GST_DEBUG_DUMP_DOT_DIR="."
export GST_DEBUG_OPTIONS=pretty-tags
export GST_TRACE=all

gst-launch-1.0 -v audiotestsrc num-buffers=16 \
  ! vorbisenc ! oggmux ! filesink location="tmp.oga"

# *-gst-launch.*.dot

which dot || apt-get install graphviz
for file in *.dot ; do dot -Tsvg "$file" > "$file.svg" ; done
xdg-open *PAUSED_PLAYING*.svg
# states " NULL_READY > READY_PAUSED > PAUSED_PLAYING > PLAYING_PAUSED > PAUSED_READY"

2.13

3 Exercices

  • Grab online MSS stream to file
  • capture webcam to ogv file
  • record microphone to OGG/vorbis
  • play DVD vob files
  • save each frame of video in separate images
  • stream webcam input to rtp stream
  • transcode video to Theora

3.1 example : capture

gst-launch \
   v4l2src ! "video/x-raw-yuv",width=320,height=240  \
   ! theoraenc ! oggmux \
   ! filesink  location="video.ogv"

3.2 example : grab MSS stream to file

gst-launch-1.0 \
  mmssrc location="mms://url/stream.asf" \
  ! filesink location="mss.wmv"

3.3 example : save images from video


gst-launch-1.0 \
    filesrc location="video.avi" \
    ! decodebin ! queue ! autovideoconvert ! pngenc \
    ! multifilesink location="%08d.png"

3.4 example : mic ogg vorbis

gst-launch-1.0 \
  autoaudiosrc ! audioconvert ! vorbisenc ! oggmux \
  ! filesink location="vorbis.oga"

3.5 example : play DVD vobs

gst-launch-1.0 \
  filesrc location="dvd.vob" \
  ! mpegpsdemux name=demux \
  ! mpegvideoparse \
  ! queue \
  ! mpeg2dec \
  ! autovideosink \
    demux. \
    ! queue \
    ! mad \
    ! audioconvert ! audioresample \
    ! alsasink

3.6 example : stream webcam

gst-launch-1.0 -v v4l2src \
! video/x-raw,width=320,height=240 \
! rtpvrawpay \
! udpsink host=127.0.0.1 port=5004

gst-launch-1.0 -v udpsrc port=5004 \
! application/x-rtp, media=video, clock_rate=9000, \
  encoding-name=RAW, sampling=YCbCr-4:2:0, \
  depth='(string)8',width='(string)320',height='(string)240',
  payload=96 \
! rtpvrawdepay ! decodebin ! autovideosink


3.7 example : transcode to theora

gst-launch-1.0 filesrc location="in.wmv" \
  ! decodebin name=d \
  { oggmux name=mux max-delay=500000000 max-page-delay=500000000 \
  ! filesink  location="out.theora.ogv" } \
  { d. ! queue ! ffmpegcolorspace ! theoraenc bitrate=300  \
  ! queue ! mux. }

4 References

5 Notes

  • Created with emacs, org-mode, reveal.js, ox-reveal
  • Thanks: #GraphikLabor, EGL, SOSG
  • License: CC-BY-SA @RzrFreeFr 2016