Optical Music Recognition Ichiro Fujinaga Mc Gill University

Optical Music Recognition Ichiro Fujinaga Mc. Gill University 2003

Content § Optical Music Recognition § Levy Project § Levy Sheet Music Collection § Digital Workflow Management § Gamera § Guido / Note. Ability

Optical Music Recognition (OMR) § Trainable open-source OMR system in development since 1984 § Staff recognition and removal • Run-length coding • Projections § § Lyric removal / classifier Stems and notehead removal Music symbol classifier Score reconstruction Demo

OMR: Classifier § Connected-component analysis § Feature extraction, e. g: § Width, height, aspect ratio § Number of holes § Central moments § k-nearest neighbor classifier § Genetic algorithm

Overall Architecture for OMR Image File Staff removal Segmentation Knowledge Base Feature Vectors Recognition Output K-NN Classifier Symbol Name Optimization Genetic Algorithm K-nn Classifier Off-line Best Weight Vector

Lester S. Levy Collection

Lester S. Levy Collection § North American sheet music (1780 – 1960) § Digitized 29, 000 pieces § including “The Star-Spangle Banner” and “Yankee Doodle” § Database of: § § text index records images of music (8 bit gray) lyrics (first lines of verse and chorus) color images of cover sheets (32 bit) http: //levysheetmusic. mse. jhu. edu

Digital Workflow Management § Reduce the manual intervention for large-scale digitization projects § Creation of data repository (text, image, sound) § Optical Music Recognition (OMR) § Gamera § XML-based metadata § composer, lyricist, arranger, performer, artist, engraver, lithographer, dedicatee, and publisher § cross-references for various forms of names, pseudonyms § authoritative versions of names and subject terms § Music and lyric search engines § Analysis toolkit

The problem § Suitable OCR for lyrics not found § Commercial OCR systems are often inadequate for non-standard documents § The market for specialized recognition of historical documents is very small § Researchers performing document recognition often “re-invent” the basic image processing wheel

The solution § Provide easy to use tools to allow domain experts (people with specialized knowledge of a collection) to create custom recognition applications § Generalize OMR for structured documents

Introducing Gamera § Framework for creation of structured document recognition system § Designed for domain experts § Image processing tools (filters, binarizations, etc. ) § Document segmentation and analysis § Symbol segmentation and classification • Feature extraction and selection • Classifier selection and combiners § Syntactical and semantic analysis Generalized Algorithms and Methods for Enhancement and Restoration of Archives

Features of Gamera § § § Portability (Unix, Windows, Mac) Extensibility (Python and C++ plugins) Easy-to-use (experts and programmers) Open source Graphic User Interface Interactive / Batchable (scripts)

Architecture of Gamera Graphic User Interface (wx. Windows) Scripting Environment (Python) Plugins (Python) Automatic Plugin Wrapper (Boost) Plugins (C++) GAMERA Core (C++)

Example of C++ Plugin // Number of pixels in matrix #include “gamera. hh” #ifdef __area_wrap__ #define NARGS 1 #define ARG 1_ONEBIT #endif using namespace Gamera; template <class T> feature_t area(T &m) { return feature_t(m. nrows() * m. ncols()); }

Example of Python Plugin // This filters a list of CC objects import gamera def filter_wide(ccs, max_width): tmp = [] for x in ccs: if x. ncols() > max_width: x. fill_matrix(0) else: tmp. append(x) return tmp

Gamera: Interface (screenshot in Linux)

Gamera: Interface (screenshot in Linux)

Histogram (screenshot in Linux)

Thresholding (screenshot in Linux)

Thresholding (screenshot in Linux)

Staff removal: Lute tablature

Classifier: Lute (screenshot in Linux)

Staff removal: Neums

Classifier: Neums (screenshot in Linux)

Greek example

GUIDO Music Notation Format H. Hoos, K. Renz, J. Kilian § “A formal language for score-level representation” § Plain text: readable, platform independent § Extensible and flexible § Adequate representation § Note. Server: Web/Windows § GUIDO/XML § Note. Ability (K. Hamel)

GUIDO: An example { [ beams. Off | clef<"treble"> key<"D"> f#*1/8. g*1/16 | a*1/4. d 2*1/8 d*1/4. c#*1/8 | e 1*1/2 _*1/4 f#*1/8. g*1/16 | c#2*1/4. b 1*1/8 a*1/4. g*1/8 | | e#*1/2 f#*1/4 f#*1/8. g*1/16 | a*1/4. d 2*1/8 d*1/4. c#*1/8 | e 1*1/2 _*1/4 f#*1/8 g | c#2*1/4. b 1*1/8 a*1/4. c#*1/8 ], …

Conclusions § Gamera allows rapid development of domain-specific document recognition applications § Domain experts can customize and control all aspects of the recognition process § Includes an easy-to-use interactive environment for experimentation § Beta version available on Linux § OS X version in preparation

Projections X-projections Y-projections back