Song and Dance and TWD using GStreamer Thomas

Song and Dance and TWD using GStreamer Thomas Vander Stichele

Overview ● ● ● TWD and USP TMS VKC of GST GPE prototyping LAP for SAD celebrating FTA Roadmap

software desktops ● Desktops have targets and philosophies – GNOME: less options, less bloat, HIG – KDE: full configurability – Mac. OS: eye candy, artistic personality – Windows: applications ● Desktops need a USP for better marketing ● After-hours business school applied to GNOME ● TMS done by Korean firm

Thorough Market Study

Thorough Market Study

Thorough Market Study

Next year at LWE

Karaoke ● What do we need ? ● Why use GStreamer ? ● What is GStreamer ? ● What needs to be done ?

What do we need ? ● Audio playback ● Visualization ● Video ● Subtitling

Why use GStreamer for it ? ● Lot of elements already present ● Framework is there for things you need ● – Output – Synchronization – Mixing multiple formats Prototyping is easy – gst-launch for quick tests – gst-editor for experimenting with pipelines – Gst-python for testing application structure

GStreamer ● What is it ? ● Why is it not ? ● Elements and pads ● Graph-based pipelines

What is it ? ● ● Core: streaming media framework – Arbitrary filter graphs – Extensible pluggable architecture – Data-agnostic, basic elements (GObjects) Plug-ins : provide functionality – Audio : filters, codecs, analysis, visuals – Video : filters, codecs, (de)muxers – Sources/Sinks : video/audio/media in/out

What is it ? ● GStreamer is first of all a library – ● Compare it to GTK We provide a few applications – A player as a test bed for features (~ gtk-demo) – An editor used in debugging (~ glade) ● We provide plug-ins ● We provide documentation

Why is it ? ● ● Linux has bad media handling many hacked single-purpose apps and libs, code copies and forks, . . . – ● other OS's have such a framework – ● avifile, Broadcast 2000, . . . Direct. Show, Quicktime framework, Be Media, Real. Media SDK LGPL core gives companies the chance to make binary plug-ins

What is it not ? ● It's NOT a sound server – one could be written with it (maybe) – not important for an app ● It's NOT Gnome-only ● It's NOT 100% ABI-stable (partial) ● It's NOT perfect

Elements and Pads ●element has pads ●chain- or loop-based ●buffer of data comes in on the sink pad ●element processes data ●Processed buffer goes out on the source pad ●This one decodes Ogg/Vorbis audio

Graph-based pipelines ●Pipeline managed by a bin (not visible) ●Links elements by their pads ●elements are chain-based or loop-based ●Data will flow left to right

What needs to be done ? ● Experiment – gst-launch – gst-editor ● Write elements ● Fix bugs – In elements – In applications like gst-editor

Let's get to it ! ● Audio playback – ● Visualization – ● Guadec-editor 1 Guadec-editor 2 Subtitling – Guadec-editor 3

Combining the three ● Normal: – ● Guadec-editor 4 Threaded: – Guadec-editor 5

Using video footage ● Threaded MPEG playback: – ● Guadec-editor 6 With subtitles – -guadec-editor 7

Ideas for improvement ● Use pitchshifter so Stallman can sing ● Use festival to replace Stallman ● – Play: Guadec-editor 8 – Save to vorbis: Guadec-editor 9 – Use with subtitles: guadec-editor 10 Personalise – Find video of friends: guadec-editor 11

Ideas for improvement ● Use live input – ● Use backing music – ● ● Mix in audience audio and video Mod files, midi files, . . . Use color font rendering Add a bouncing ball plugin

Gkaraoke ● Truth is stranger than fiction

Roadmap ● ● Actual stuff to work on What can you do ?

Actual stuff to work on ● Media-info library: – Save and load metadata and streaminfo – Cache it (-> thumbnails) – Playback parameters (mixing, volume) – Shared by all apps

Actual stuff to work on ● ● ● Nautilus: – Hoverplay for audio and video – Further audio view improvement Encoding profiles – Used by recorder, sound-juicer, . . . – Using human-readable names Editor: – Educational GStreamer tool

What can you do ? ● Users – Test our applications: ● ● ● – – – Gst-totem, gst-player, gst-editor Rhythmbox, quark, nautilus-media, sound-juicer Use media files on http: //gstreamer. net/media/ Suggest new applications Run the upcoming test suite on your computer Harass developers

What can you do ? ● Developers – – – Help us fix bugs and implement new features Document Write applications Write/fix elements Test and use the bindings Use the tools we provide ● ● – Build scripts Template tarballs Go to Ronald's talk