Constraints for Multimedia Presentation Generation Joost Geurts Multimedia

Constraints for Multimedia Presentation Generation Joost Geurts, Multimedia and Human-Computer Interaction CWI Amsterdam email: Joost. Geurts@cwi. nl 1

Talk overview • Generating multimedia automatically • Cuypers multimedia generation engine • Multimedia and constraints – Quantitative constraints – Qualitative constraints • Cuypers demo • Conclusion, future directions 2

Multimedia Presentation • Multimedia Presentation – Image, Text, Video, Audio – Based on Temporal and Spatial Synchronization • Multimedia Document – SMIL, SVG, HTML – WYSIWYG – Static Content • Problem: Dynamic Content 3

Generating adaptive multimedia • Content – Large multimedia database • System profile – PC, PDA, WAP • Network profile – Modem, Gigabit • User profile – Language, Interests, Abilities, Preferences Too costly to author manually 4

Cuypers multimedia generation engine 5

Cuypers multimedia generation engine • Cuypers is based on – media independent presentation abstractions – transformation rules with built-in backtracking and constraint solving 6

Semantic structure Author does not specify complete presentation… …but only rhetoric relations 7

Communicative Devices …rhetoric relations are than transformed into presentation independent communicative devices… 8

Automatic multimedia generation • Designer does not specify complete presentation… …but only specifies requirements • System automatically finds a solution which meets requirements • How should the requirements be specified? – Declarative constraints 9

Constraint satisfaction • Constraints occur often in our daily lives – Agenda, Travelling, Shopping • Constraint paradigm for Problem Solving – Declarative Used for problems with: – Many variables – Large domains – Based on domain reduction paradigm 10

Intelligent reduction of possible values X {1, 2, 3, 4, 5}, Y {1, 2} ; X Y X {1, 2}, Y {1, 2} ; X Y 11

Traditional use of constraints Quantitative constraints – Integer domain – Reduction by arithmetic relations • Greater than (>) • Less than (<) • Equals (=) – Example (x < y ; x [0. . 10], y [5. . 10] ) (x + y = z 3 , x = u + 1 ; x , y , z , u ) 12

Solving a Constraint Satisfaction Problem • Problem SEND + MORE = MONEY • Modeling 1000 x S + 100 x E + 10 x N + D + 1000 x M + 100 x O + 10 x R + E = 10000 x M + 1000 x O + 100 x N + 10 x E + Y • Domain reduction / Search • Solution S=9, E=5, N=6, D=7, M=1, O=0, R=8, Y=2 13

Quantitative Constraints in Multimedia …Communicative devices generate constraint-graph which the system tries to satisfy… 14

Drawbacks of quantitative constraints • Too many (trivial) solutions that differ by: – 1 pixel position, or – 1 milliseconds in timing • Not sufficiently expressive • cannot specify “no overlap” constraint • Too low level • A. X 2 B. X 1 15

Allen’s 13 temporal relations Allen’s relations are used for both spatial and temporal lay-out 16

Solution: qualitative constraints • For non-typical domains – Boolean, – Three valued logics, – Allen’s relation • Advantages for Multimedia generation: – More intuitive – More expressive – Smaller domains 17

Domain Reduction Rules • Inverse A before B B after A A equal B B equal A • Transitive A before B , B before C A overlaps B, B during C A before C A overlap C or A during C or A starts C • Equals A overlap C, A [o, d, s] C A overlap C 18

Qualitative Constraints …Qualitative solutions translate automatically to lower level quantitative constraints… 19

New problem: What if constraints are insoluble? Solution: Constraint Logic Programming • Combine Prolog unification and backtracking with constraint solving • Use Prolog rules to generate constraints • Backtrack when constraints are insoluble 20

Cuypers generation engine • Multiple layers: – Communicative devices generate constraints – Qualitative constraints translate to quantitative constraints – Solution of both constraints provides sufficient information for final presentation 21

Cuypers demo: scenario • Client User is interested in Rembrandt and wants to know about the “chiaroscuro” technique • Server Query database Generate constraints according to: –System profile –User profile –Network profile • Server Solve constraints / revise constraints • Server • Client Generate SMIL presentation Play presentation 22

Conclusions • Quantitative constraints are insufficient for automatic multimedia presentation generation. Also need • Qualitative constraints to allow intuitive and effective high level specification, and • Backtracking for revising specific constraints which otherwise cause the entire set to fail 23

Discussion • Labeling – Choice of candidate variable – Choice of candidate value • Transitive Reasoning Rule – Infer implicit relations – Redundant • Allen’s Relations – Not very well suited for generating MM – Non interactive 24

Future directions • Best-first instead of depth-first – Choose “best” among possible solutions – Needs evaluation criteria • Improve knowledge management – Make design knowledge declarative and explicit – Preserve metadata in final presentation – Use standardized and reusable profiles 25

Thank you 26