Semantic Web Application Music Retrieval Ying Ding SLIS
Semantic Web Application: Music Retrieval Ying Ding SLIS, IU
What is the Semantic Web? l “An extension of the current Web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. ” l Sir Tim Berners-Lee et al. , Scientific American, 2001: tinyurl. com/i 59 p 2
Semantic Web -- Web 3. 0 l How to realize that: l l machine-understandable semantics of information, and millions of small specialized reasoning services that provide support in automated task achievement based on the accessible information
The current (syntactic / structural) Web
Was the Web meant to be more? Hyperlinks – typed hyperlinks Document - data
Ontology l The semantic Web is essentially based on ontologies l ontologies are formal and consensual specifications of conceptualizations… l providing a shared and common understanding of a domain that can be communicated across people and application systems
Metadata and Semantics
Semantic Web - Language tower
What is Semantic Web for? l l Integrating - trying to solve the problem of data and service integration Searching - Providing better communication between human and computers by adding machine-processable semantics to data. l Form keyword search data search query answer
What is current Semantic Web effort? l Lifting document web to data web l Weaving the data web through semantic links (types hyperlinks)
Bubbles in April 2008 >2 B RDF triples Around 3 M RDF links
http: //www. elec. qmul. ac. uk/easaier/ Enabling Access to Sound Archives through Integration, Enrichment and Retrieval
The EASAIER Project l l l EASAIER - Enabling Access to Sound Archives through Integration, Enrichment and Retrieval EU funded project, 30 month duration (started May 2006) Partners:
EASAIER - Goals l Overcome problems for many digital sound archives concerning online access l l l sound materials and related media often separate searching audio content limited EASAIER Framework l l Integration of Sound Archives Low level audio feature extraction (speech/music) Intelligent User Interface Enhanced Access Tools l l looping, marking of audio sound source separation time and pitch scale modification Semantic Search Evaluation
Semantics in EASAIER l Description of metadata using an ontology l High-level metadata l l l Low-level metadata l l l e. g. title, author of an audio asset sources are databases, files in e. g. DC, MARC e. g. speech event occurs at timestamp xyz feature extractor tools Semantic Search l l Search across variety of metadata Search across multiple archives Similarity Search Related content acquisition from the Web
The EASAIER System
Music Ontology l Overview l Merging existing related ontologies l Developed by QMUL l Cover the major requirements l Widely-adopted l Four core MO components l FRBR l FOAF l Event l Timeline http: //musicontology. com/
The Music Ontology: Timeline Ontology l Expressing temporal information, e. g. l l l This performance happened the 9 th of March, 1984 This beat is occurring around sample 32480 The second verse is just before the second chorus
The Music Ontology: Event Ontology l Event — An arbitrary classification of a space/time region l l l This performance involved Glenn Gould playing the piano This signal was recorded using a XXX microphone located at that particular place This beat is occurring around sample 32480
The Music Ontology: FRBR & FOAF l FRBR – Functional Requirements for Bibliographic Records l l l Work — e. g. Franz Schubert's Trout Quintet Manifestation — e. g. the "Nevermind" album Item — e. g. my "Nevermind" copy FOAF – Friend of a Friend Person Group Organization
The Music Ontology – Music Production Concepts l On top of FRBR: l Musical. Work, Musical. Manifestation (Record, Track, Playlist, etc. ), Musical. Item (Stream, Audio. File, Vinyl, etc. ) On top of FOAF: Music. Artist, Music. Group, Arranger, Engineer, Performer, Composer, etc. — all these are defined classes: every person involved in a performance is a a performer. . . On top of the Event Ontology: Composition, Arrangement, Performance, Recording Others : Signal, Score, Genre, Instrument, Release. Status, Lyrics, Libretto, etc.
The Music Ontology – Music Production Workflow
Metadata in RDF l Low-level metadata is output in RDF using Music Ontology l l Audio Feature extractor Speech recognition service Emotion detection service High-level metadata import l DB Schema Mapping l l Standardized Metadata import l l e. g. D 2 R, Virtuoso RDF Views DC, MARC, METS, . . . Linked Data ? l DBPedia, Geonames, . . .
Use Case: Archive Publication - HOTBED Publishing Hotbed Database Extending Music Ontology Querying the Semantic Archivist Hotbed RDF Features Extraction, Visualization, . . . Instruments Taxonomy Query Interface Sound Access tools
1) editing the ontology l using WSMT editor to extend the ontology Music Ontology Graphical Edit Music Ontology Text Edit
2) performing tests on the new extension l l What are the instruments in my taxonomy ? Did i forget any kind of [pipe] ?
3)mapping Scottish Instruments to a general Instruments taxonomy
4) relating and publishing Hotbed l l Relate tables from hotbed to concepts from the MO Publish on the semantic web via the D 2 R tool Hotbed Database Mapping RDF Publication via D 2 R tool The server offers a SPARQL end-point for external apps Music Ontology
Mapping Metadata to the Music Ontologies Title: File 2 Author: Oliver Iredale Searle Perfomers: Katie Punter Source Type: Audio Source: File 2 Instrument: Flute Instrument occurrence timings: 0"-16" Time Signature: 4/4 Beats per minute: 50 Tonality: Bb major Searle Testbed : music a mo: Signal ; dc: title "File 2" ; dc: author "Oliver Iredale Searle" ; : music-performance a mo: Performance ; mo: recorded_as : music ; mo: composer : Oliver. Iredale. Searle ; mo: instrument mo: flute ; mo: performer : Katie. Punter ; mo: bpm 50 ; mo: meter "4/4" ; mo: key #BFlat. Major. : Katie. Punter a foaf: Person. : ss 1 a af: Person. Playing; af: person : Katie. Punter; event: time [ tl: on. Time. Line : tl 1234; tl: begins. At "PT 0 S"; tl: duration "PT 16 S"; ].
Mapping Metadata to the Music Ontologies ALL web service output <xml version="1. 0" encoding="UTF-8"> <speech_retrieve. Result> <speech_descriptor word="power" audio_material="c: /hotbed/performance/1004. wav" position_sec="10" duration_sec="5" confidence="89" /> </speech_retrieve. Result> </xml> <http: //www. myarchive. org/signal/1234/event/power. PT 10 S> a af: Text; af: text "power"; af: confidence "89"; event: time [ a time: time. Interval; tl: on. Timeline <http: //www. myarchive. org/signal-timeline/1234>; tl: begins. At. Duration "PT 10 S"; tl: duration. XSD "PT 5 S"; ].
Mapping Metadata to the Music Ontologies Vamp Output <metadata type="audio"> <category name="vamp"> <feature name="beats" type="variablerate" description="Detected Beats" unit="N/A <data> <event idx="0" timestamp=" 0. 0928" duration="0" label="224. 69 bpm"/> </data> </feature> </category> </metadata> event: time [ a time: Instant ; tl: on. Time. Line : tl 898; tl: at "PT 0. 0928 S"; ]; mo: bpm "224. 69";
RDF Storage and Retrieval Component l Built on top of Open. RDF Sesame 2. 0 l Query interfaces l l Web Service (Servlet) l HTTP SPARQL Endpoint Web Service provides predefined SPARQL query templates l Themes l l Music, Speech, Timeline, Related media, Similarity l Dynamic FILTER constructs l Results in SPARQL Query Results XML Format Interface for RDF metadata import using the Archiver application
Enhanced Client
Web client
Related media Doubleclick
Related media on the web (1) Result search for author “Coltrane” Track selection Web related media search launched automatically according to the name of the author
Related media on the web (2)
Demo l l http: //www. elec. qmul. ac. uk/easaier/index 3. html http: //easaier. deri. at/demo/
Demo l l l Time and Pitch Scale Modification (demo) Sound source separation (demixing/remixing, Noice reduction, etc. ) (demo) Video time stretching (to slow down or speed up images while retaining optimal sound) (demo)
Scenario 1 – Artist Search l l Aggregation of music artist information from multiple web sources Ontology based search: l l Music. Brainz data mapped to the Music. Ontology Music. Brainz Web Service: l l Music. Brainz RDF Dump: l l l allows to retrieve artist URI by literal based search retrieve RDF use SPARQL to perform queries (e. g. resolve relationships) Web 2. 0 Mashups: l Retrieve data (videos, images) from external sources l l utilize RSS Feeds, APIs etc. from Youtube, Lyric. Wiki, Google more accurate results using references from Music. Brainz RDF data
Scenario 1 – Artist Search <URI> WS Interface “Beatles” <URI> process data. . . RDF Dump
Scenario 1 – Artist Search
Scenario 1 – Artist Search
Scenario 2 – Instrument Reasoning l l Reasoning over HOTBED instrument scheme Ontologize data from HOTBED (Scottish Music Archive) l l Usage of D 2 R to lift data from legacy DBs to RDF Ontologies: l l l Music. Ontology Instrument Ontology (domain related taxonomy) Subsumption reasoning: l l Retrieve instrument tree Search for persons that play an instrument l l Subclass relations: resolve persons playing more specific instruments Example: Wind-Instrument < Wood. Wind < Flute
Scenario 2 – Instrument Reasoning Example: Search for people playing instrument of type Woodwind
Demo 3 – Rules l Infer new knowledge with rules l l Domain Rule Sophisticated Query l l Albums based on certain Band/Artist/Instrument Use. Case: The Velvet Underground discography l Available information: l l l Membership durations Album release dates „Founders“ of the band ? exist _artist, <_band, has. Member, _artist>, <_artist, on. Duration, _duration> l forall ? x, <_band, has. Member, ? x>, < ? x, on. Duration, ? time> l <? time, not. Before, _duration> <_band, founder, _artist> l l Albums & corresponding members
Demo 3 – Rules Basic Information Band Founder Band Duration (Members & Albums) Album Tracks
Thanks l l l Contact Ying Ding LI 029 (812) 855 5388 dingying@indiana. edu
- Slides: 48