Bi Onym A flexible workflow approach to taxon

Improving the current matchers • Propose several Taxonomic Authority Files as references to be matched with • Make flexible and customizable the control of the matching workflow (e. g. , selection of the sequence of the matching methods) • Give full control for advanced users [but still a set of default/standard workflow(s) for basic users] TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Bi. Onym approach • There is no one size that fits all!! • Some applications are ‘fault intolerant’ – E. g. compilation of authority lists – Have to minimise ‘false positives’, at the expense of less automation • Others are less sensitive to mistakes – E. g. synonymy expansion in a biogeographic query, find distribution records of a single species under different names or spelling variations • Will require different choices TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

A flexible workflow for taxon name matching: Bi. Onym Raw Input String. e. g. Gadis morua Lineus 1759 Parsing and Preprocessing Reference Source (ASFIS) Reference Source (Wo. RMS) Taxon Matcher 1 Matchers: • GSAy (new) • Lexical distances • Levenshtein • Soundex • Trigrams Workflows • Bi. Onym (new): User control • Emulation of Taxamatch • YASMEEN (new) Reference Source (Fish. Base) Taxon Matcher 2 Reference Source (any in Dw. C-A) … Taxon Matcher n Postprocessing Matching name qed Gadus morhua (Linnaeus, 1758) TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Developed in i. Marine infrastructure • i. Marine (D 4 Science): e-infrastructure • VREs: Virtual Research Environments exploiting data and tools in the infrastructure … VREs g. Cube Infrastructure TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

… and outside: i. Marine Data Bonanza OBIS Wo. R MS … Data. FAO Euro. S tat Validation Wo. R DS Private Cloud Sharing i. Marine Registries GBIF Enriching Commercial Cloud WOA My. Oc ean Co. L Processing ITIS NCBI IRMN G TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

i. Marine: Storage and Computing as Service • Secure • Fault-tolerant • Replication • Open source RDBMS • Up to 1 TB data Virtual Workspace Relational Databases 45 TB Currently Used 330 CPU Cores Currently Allocated Spatial Database Large and Active data storage • ISO 19115/10139 Metadata • Catalogue TDWG Annual Conference 2013, Firenze, Italy • Scalability and high availability • Across sites 31 st October 2013

Statistical Manager: Resources and Sharing TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Bi. Onym: Outline • Components – Taxonomic Authority Files – Matchers – Pre- and post-processing: parsers, synonym expansion , taxon resolution, performance statistics • Development frameworks – For Matchers – For Workflows (= sequence of Matchers) • Experiments – Results • Conclusions TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Available Taxonomic Authority Files • Co. L: • NCBI: Catalogue of Life National Center for Biotechnology Information • IRMNG: Interim Register of Marine and Non-marine Genera • ITIS: Integrated Taxonomic Information System • Wo. RMS: World Register of Marine Species • ASFIS: List of Species for Fishery Statistics Purposes; for commercial aquatic species • Fish. Base (+info from Cof. F: Catalog of Fishes): for finfishes TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Pre-Processing: name format standard • Split names in atomic components (genus, species, authority, author, year) if necessary (Dima Mozzherin’s parser) • Align variations in complementary words: var. /v. , aff. , conf. /cf. , comma in authority, etc. • Customize character/string substitutions TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Matchers: principle • Input: – Standard formatted file of names Input – Customized parameters (e. g. , thresholds for distances) • Character substitutions – E. g. dropping gender suffix – E. g. fuzzy matching of Tony Rees • A unique algorithm (e. g. , one lexical distance): – Using the customized parameters • Output: A set of names with matching rate – One subset being considered as matched – One subset considered as non-matching • The output of a matcher can be used as the input of another one TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Matchers: the built-in matchers • Lexical dist. : the minimum number of singlecharacter edits (insertion, deletion, substitution) required to change one word into the other • Soundex-Like dist. : an algorithm relying on an encoding of phonemes pronunciation in English. Our variant does not compress phonetic information • Trigrams / N-grams dist. : a similarity measure between sequences of letter triplets (a trigram representation) extracted from the input strings • One domain-knowledge based matcher (GSAy) … to be applied first in the context of Systematics TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

GSAy GSAY GSr. Ay GSr. AY GSA GSr. A Matchers: GSAy process (1) Step Score Complete match GSAy 100 Parentheses issue GSAY 97 Gender agreement issues GSr. Ay 94 Gender agreement and parentheses GSr. AY 91 Year issues GSA 88 Year and gender agreement issues GSr. A 85 TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

GSY Matchers: GSAy process (2) Step Rate Author issues, misspelling or wrong GSAy 82 GSr. Y GS Author and year issues, Homonyms GSr SAy Genus issues, other combinations Sr. AY TDWG Annual Conference 2013, Firenze, Italy GSAY 79 GSr. Ay 76 GSr. AY 73 GSA 70 GSr. A 67 31 st October 2013

Sr. Ay Matchers: GSAy process (3) Step Rate Genus issues, other combinations Sr. Ay 64 Sr. AY GAy GAY Species misspellings … but also … … species described in same genus by same author in same paper Sr. AY 61 GAy 35 GAY 32 61> >35 s e m a n d Matche other matchers Non-Matc hing nam TDWG Annual Conference 2013, Firenze, Italy es 31 st October 2013

Matcher: GSAy examples Genus Gadus Gadus Gadus Gadis Gadus Species Authority morhua Linnaeus, 1758 morhua (Linnaeus, 1758) morhuus Linnaeus, 1758 morhuus (Linnaeus, 1758) morhua Linnaeus, 1759 morhuus (Linnaeus, 1759) morhua Lineus, 1758 morhuus Lineus, 1758 morhua Lineus, 1759 morhuus Lineus, 1759 morhua Linnaeus, 1758 morhuus (Linnaeus, 1758) morthua Linnaeus, 1758 morthua (Linnaeus, TDWG Annual Conference 2013, 1758) Firenze, Italy Step Rate GSAy 100 GSAY 97 GSr. Ay 94 GSr. AY 91 GSA 88 GSr. A 85 GSY 82 GSr. Y 79 GS 76 GSr 73 SAy 70 Sr. Ay 67 SAY 64 Sr. AY 61 GAy 35 31 st. GAY October 201332

Workflow development framework • Builds flexible Workflows for Names Matching • A Java framework based on the g. Cube system (http: //www. gcube-system. org/) • Allows to exploit Cloud Computing Facilities • Presents Java interfaces to build Strings Pre. Processing, Parsing and Post-processing • Allows to define character substitutions • Allows to add new Matchers as plug-ins TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Workflow development framework • A series of operators acting as switches: – First apply ‘transformation’ (e. g. character substitution) – Then calculate distance between all possible pairs of names • Each switch decides, whether a pair of names should be considered as ‘matches’, and splits the input list in: – ‘matched’ names – ‘non-matching’ names. • Parameters in each switch are customizable TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Matcher framework: YASMEEN (FAO) • Yet Another Species Matching Execution ENvironment • Based on COMET – COncept Matching Engine and Tools • YASMEEN: a set of data models, formats and tools to perform species matching identification • Multiple matchlets, each dealing with a specific attribute of the species data model (genus, species, author etc. ) • New matchlets can be designed (just a few lines of code) and plugged in • Reference data in Dw. C-A format • Full support to distributed computation (split IN & REF data / join results) TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Workflow builder: YASMEEN (FAO) • Used as a matcher in Bi. Onym workflow • But can work as a standalone specific workflow • When used as standalone: • Lexical matchlets' scores can be computed with a combination of different strategies (Levenshtein distance, soundex similarity, N-grams similarity) • Overall matching score for an input / reference data pair is a weighted combination of the triggered matchlets' scores TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Workflow: Bi. Onym • Workflow consists of three parts – Preprocessing (including possibly parsing) – Chain of matchers; output from one is input in next – Postprocessing • E. g. present ‘ambiguous’ matches to end user • E. g. calculate performance statistics • Chain of matchers – Most restrictive first – Those based on domain knowledge first – Test names matched in one step are not passed on to next matcher TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

FAO use case Scientific names Austroglossus spp Austropotamobius pallipes Auxis rochei Auxis thazard, A. rochei Bagrus spp Bothidae Corallium sp. nov. Ex Mollusca Ex Pinctada spp Common Names Abalones nei Aesop shrimp African forktail snapper African lungfishes African moonfish African sicklefish Aka coral Akiami paste shrimp Alaska plaice Alaska pollock(=Walleye poll. ) TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Workflow: Emulation of Tony Rees’ Taxamatch • Normalization of species name into its root disregarding the gender issues in taxon name. • Modified Damerau-Levenshtein Distance Algorithm (MDLD) - the number of times of replace, delete or insert character to make the two strings the same • Phonetic algorithm (e. g. Soundex) • Authority Matching - which detects the similarity in substring TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Post-processing • Modalities governing how the results of the matching process are used/presented to the end-user • Will depend on the needs of the end user • Examples: – Synonymy expansion of queries in a biogeographical system – Reconciliation of check lists from different sources, for same area and taxon – Presenting end-user with ambiguous matches TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Experiments: a R implementation • Experimental system implemented in R and Postgre. SQL – R thin wrapper around Postgre. SQL statements – SQL used for the heavy lifting • Make use of Trigram indexes, for example • Tool for communication and prototyping • Developing tools to analyse performance – Generate confusion matrix… – For identical test sets, different workflows • Quantitatively compare sets of options and/or matchers TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Effectiveness s Tru hit eh se its Fal Example graph comparing performance of different settings (generated with R) Non hits TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Results of experiments (YASMEEN) Genus / species misspellings: IN: Lacheepenseer Perthseecous → REF: Acipenser persicus (Borodin, 1897) Scientific name matchlet, using Levenshtein similarity → 61. 5% No separation between genus / species, relevant misspelling: IN: acipnesreppeerseekoos → REF: Acipenser persicus (Borodin, 1897) Scientific name matchlet, using Levenshtein similarity → 47. 6% Inverted genus / species: IN: Platorhynchus Scaphirhincus → REF: Scaphirhincus platorhynchus (Rafinesque, 1820) Scientific name matchlet, using n-grams similarity → Score: 100. 0% Relevant misspellings (resolved with support from authorities data): IN: Casphinhi Platynchurs (Rafinesk, 1820) → REF: Scaphirhynchus platorynchus (Rafinesque, 1820) Genus matchlet (wgt: 75), Species matchlet (wgt : 100), Author name matchlet (wgt : 50), Year matchlet (wgt : 25), using Levenshtein similarity (wgt : 100) and Soundex similarity (50) → 58, 2% TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Experiments: a simple interface TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Efficiency • Application first version of Bi. Onym workflow (1000 species names) • First run: only one Worker node (~ 1 CPU) • Second run: Cloud Computing facilities assigned by i. Marine e-Infrastructure (computation distributed over 19 Worker nodes) • Result: Time reduction 76. 7% • This means that the workflow can be used also in interactive systems (no need for batch processing) TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Results of experiments: Matchers • Search Term: – Rhincodon typu Linneaus, 1758 • Output/s: Using GSAy – Rhincodon typus Smith, 1828 -> Score is 73% • Using taxamatch: – Rhineodon typus Smith, 1828 – Rhiniodon typus (Smith, 1828) – Rhinodon typicus Müller & Henle, 1839 – Rhinodon typicus Smith, 1845 TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Conclusions: Results • Workflows – Building of pre-set (default) – On the fly setting – Integrating taxonomic/nomenclature knowledge (GSAy) • Making the best from previous matchers (Taxamatch and subsequent various implementations) and other technologies (u. Bio/GNA/GBIF parser) • Effectiveness and Efficiency increased in i. Marine e-infrastructure TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Conclusions • Plans: interface (Nov. ), tests (Dec. ), open (Jan. ) • Other Taxonomic Authority File – FADA (Bio. Fresh) / PESI / … • Name reconciliation • Beyond scientific names – Common names / Vessels / … • New matchers integration = as matching methods are developed TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Future added value? Storage of knowledge • Make available the matches between raw and published names (and current valid names) • Self-learning system • Build a community of practice (Co. P), not alone … • GNA, Bio. Vel Collaborative development TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Special Thanks • Tony Rees (CSIRO) • Dmitry (Dima) Mozzherin (GNA project) TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013

Authors • Edward Vanden Berghe, Vrije Universiteit Brussel (VUB), Brussels, Belgium • Nicolas Bailly, World. Fish, and Fish. Base Information and Research Group (FIN), Los Baños, Philippines • Caselyn Aldemita, Fish. Base Information and Research Group (FIN), Los Baños, Philippines • Fabio Fiorellato & Anton Ellenbroek, Fisheries Statistics and Information (FIPS), FAO, Rome, Italy • Gianpaolo Coro & Pasquale Pagano, Istituto di Scienza e Tecnologie dell'Informazione A. Faedo (ISTI), CNR, Pisa, Italy TDWG Annual Conference 2013, Firenze, Italy 31 st October 2013