Post Desktop User Interfaces Multimedia Architectures Ines Frber

Post Desktop User Interfaces Multimedia Architectures Ines Färber and Alexander G. M. Hoffmann Advisor: Thorsten Karrer Thursday, 18 January 2007

Motivation n multimedia applications are of growing importance have to satisfy hard real-time and interaction requests good architectures are required Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 2

Introduction n “multimedia architecture” is not clearly defined multimedia is an integration of multiple digital media into one big media at least one combined media has to be timedependent Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 3

Introduction (2) n n typical applications: video games, video conferencing systems, speech recognition, animation, internet dealing with creation, manipulation, presentation, storage or communication high requirements concerning real-time, low latency and precise synchronisation all influences have to be well-regulated Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 4

Outline n SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems n Fran - Architecture for functional reactive animation generation n n Multimedia Information Service Enabling Conclusion Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 5

SAI Software Architecture for Immersipresence SAI is a new software architecture model for designing, analyzing and implementing applications performing distributed, asynchronous parallel processing of generic data streams. (Francois 2005) n general formalism n high bandwidth n low latency A. R. J. Francois. Software Architecture for Immersipresence, IMSC 2005 Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 6

SAI (2) n problems in actual complex systems ¨ integration ¨ resource intensive activity ¨ countless unforeseen problems n SAI ¨ distributed implementation ¨ easy integration Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 7

SAI (3) n dataflow architectures ¨ simple and intuitive design ¨ support parallel and distributed processing n problems: ¨ limitation of efficiency and modelling power ¨ shared data excess n blackboards ¨ but: not adapted to online and real-time processing Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 8

SAI – Fundamental principles n explicit account of time both in data and processing models ¨ Cells ¨ Sources n n distinction between persistent and volatile data asynchronous parallelism Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 9

SAI – graphical approach n n n data encapsulated in pulses cells=processing centres sources=persistent information repository persistent data (constant) volatile data (changing) Visual. SAI Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 10

Architecture n conceptual level ¨ set of sources and cells ¨ inter-connections ¨ descriptions of tasks n logical level ¨ cell: active and passive filters ¨ source: structure of passive pulse Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 11

Mu. SA. RT – These computations are defined by Music on the Spiral Array. Real-Time (ACM ‘ 03) four independent streams: n n MIDI input and event processing tonal analysis rendering of the Spiral Array structures control device (game pad) input and camera manipulation Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 12

Screenshot n The active triad is indicated by a coloured triangle red = major ¨ blue = minor ¨ n n grey dots = actual key(s) green line = main melody Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 13

Video http: //www-rcf. usc. edu/~mucoaco/ Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 14

Outline n SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems n Fran - Architecture for functional reactive animation generation n n Multimedia Information Service Enabling Conclusion Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 15

Problems of Creating Animations missing differentiation between presentation and modelling n animation is conceptually continuous and parallel, whereas a PC works discretely and sequentially visual authoring tool change of language Fran n Conal Elliott and Paul Hudak. Functional reactive animation IFCP’ 97 Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 16

Fran (Functional Reactive Animation) n implemented in Haskell Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 17

Pascal vs. Haskell n procedure quicksort(l, r : integer); var x, i, j, tmp : integer; begin if r>l then begin x: =a[l]; i: =l; j: =r+1; repeat i: =i+1 until a[i]>=x; repeat j: =j-1 until a[j]<=x; tmp: =a[j]; a[j]: =a[i]; a[i]: =tmp; until j<=i; a[i]: =a[j]; a[j]: =a[l]; a[l]: =tmp; quicksort(l, j-1); quicksort(j+1, r) end. n quicksort [] = [] quicksort [x: xs] = quicksort [n | n<-xs, n<x] ++ [x] ++ quicksort [n | n<-xs, n>=x] Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 18

Fran (Functional Reactive Animation) n n implemented in Haskell afford a declarative instead of an imperative kind of programming modelling-oriented high level vocabulary of data types, functions and primitive graphic routines Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 19

Fran: key concepts n n n behaviours: time-varying, reactive values events: sets of arbitrarily complex conditions, carrying possibly rich information modelling approach is established by four features: § temporal modelling § event modelling § declarative modelling § polymorphic media n n implicit treatment of time method for event detection based on interval analysis Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 20

Fran – examples behaviours left. Right. Charlotte = move. XY wiggle 0 charlotte = import. Bitmap ". . /Media/charlotte. bmp" up. Down. Pat = move. XY 0 waggle pat = import. Bitmap ". . /Media/pat. bmp" Conal Elliott and Paul Hudak. 1997 Modeling Interactive 3 D and Multimedia Animation with an Embedded Language Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 21

Fran – examples behaviours charlotte. Pat. Dance = left. Right. Charlotte `over` up. Down. Pat Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 22

Fran – example event red. Blue u = button. Monitor u `over` with. Color c circle where c = red `until. B` lbp u -=> blue Fran – example 3 D teapot = stretch 3 2 (import. X ". . /Media/tpot 2. x") red. Spinning. Pot = turn 3 z. Vector 3 time ( with. Color. G red teapot) Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 23

Outline n SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems n Fran - Architecture for functional reactive animation generation n n Multimedia Information Service Enabling Conclusion Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 24

Multimedia Information Services Enabling n n 3 -level Multimedia-Search Architecture extendable and scaleable multimedia information management systems extendable: easily adjusted to new data types scaleable: no performance lose while growing workload Erik Boertjes, W. Jonker, and J. Wijnands. Multimedia information services enabling: An architectural approach. ACM 2001 Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 25

Modules User Profiles and Personalization Module Metadata Module Communication and Data Transfer User Interfaces and Presentation Module Raw Data Servers Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 26

Three – layer model Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 27

Peggy n n Internet movie search retrieval service movie trailers are stored (rawdata) MPEG-7 format (metadata) Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 28

Spot – a – spot T n n n O - SP camera find free parking slot 0900 Feature - and Conceptual database PDOM search engine generates voice messages “Turn left, the empty spot Is between the black call-service car and the red car” Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 29

Outline n SAI - unifying approach to the distributed implementation of algorithms and their easy integration into complex systems n Fran - Architecture for functional reactive animation generation n n Multimedia Information Service Enabling Conclusion Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 30

Conclusion - SAI n n n universal framework distributed asynchronous algorithms integration into complex systems graphical interface Visual. SAI components, connectors and constraints Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 31

Conclusion Fran n n embedded language simplifies the animation programming process via declarative programming techniques presentation details are left to the underlying implementation possible improvements: more features for 2 D, 3 D and sound, improving performance the underlying ideas of Fran also formed the basis of Microsoft’s Direct. Animation Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 32

Conclusion MISE n n n 5 modules 3 layer model with new concept of raw- and metadatabases universal search extendable and scaleable modify and adjustable Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 33

Conclusion Multimedia Architectures n no universal architecture various approaches different approaches ¨ SAI is a graphical approach ¨ FRAN is animation oriented ¨ MISE handles Interactive Multimedia retrieval requests n incompatibility if we exchange media Multimedia Architecture Ines Färber and Alexander G. M. Hoffmann 18 January 2007 34

Thanks for your attention Multimedia Architectures Ines Färber and Alexander G. M. Hoffmann Advisor: Thorsten Karrer Thursday, 18 January 2007