SANSKRIT ANALYZING SYSTEM Manji Bhadra Surjit Kumar Singh

SANSKRIT ANALYZING SYSTEM Manji Bhadra, Surjit Kumar Singh, Sachin Kumar, Subash, Diwakar Mishra Muktanand Agrawal, R. Chandrashekar, Sudhir K Mishra, Girish Nath Jha 3 rd ISCLS, Hyderabad 1

Introduction • It is an attempt towards analysis of laukika Sanskrit • Major goal is to build a machine translation system from Sanskrit to other Indian language. • The modules have been developed separately • We need to integrate these modules • We need to evaluate these modules 3 rd ISCLS, Hyderabad 2

Introduction • The system accepts full text inputs in Devanagari Unicode (UTF-8). • It supports two IMEs - Baraha and J-IME. • It has two major components– the shallow parser – the k raka analyzer 3 rd ISCLS, Hyderabad 3

Shallow parser The modules are as follows– sandhi analyzer – sam sa analyzer * – subanta analyzer – gender analyzer – k danta analyzer – taddhita analyzer* – ti anta analyzer – POS tagger 3 rd ISCLS, Hyderabad * Modules are under development 4

How does it work Show example 3 rd ISCLS, Hyderabad 5

Our platform • Java servlet based web application and services. • Java, JSP for frontend. • Unicode input/output with flatfiles, RDBMS (MS-SQL server 2005) • MS-JDBC driver for connectivity • Apache-Tomcat for web server • Javascript IME for unicode output with Itrans input 3 rd ISCLS, Hyderabad 6

Sandhi analyzer • Sandhi processing is critical for any further processing of Sanskrit. Without sandhivichheda it is not possible to get the word constituents for analysis. • At present, our sandhi analyzer does only vowel sandhi splitting. The consonant splitting is under development. • Our goal is to be able to parse a very complex string with potentially all kinds of sandhi 3 rd ISCLS, Hyderabad 7

Sandhi analyzer input Sanskrit text ↓ viccheda eligibility tests (pre-processing) ↓ subanta processing ↓ search of sandhi marker and sandhi patterns (sandhi rule base) ↓ generate possible solutions (result generator) ↓ search the lexicon ↓ subanta processing (to parse the vibhakti of first segment, if any) ↓ 3 rd ISCLS, Hyderabad output (segmented text) 8

Sandhi analyzer 1. tokenize by space (words) 2. preprocess (exclude puncts) -> puncts marked 3. check example base - if found stop 4. check subanta (it checks avyayas, verbs as well) -> pratipadikas -> avyayas marked -> verbs marked 3 rd ISCLS, Hyderabad 9

Sandhi Analyzer 5. check pratipadika list -> if found then dont process for Sandhi -> if not found then start sandhi processing 6. search of sandhi marker and sandhi patterns 7. generate possible solutions 8. search the lexicon 9. subanta processing (to parse the vibhakti of first segment, if any) 10. output (segmented text) 3 rd ISCLS, Hyderabad 10

Demo Live demo from JNU server Demo from localhost 3 rd ISCLS, Hyderabad 11

Subanta analyzer • Isolating the inflections and obtaining nominal bases and its case terminations is essential for morph analysis. • The system has Unicode Devanagari input/output mechanism and accepts complete text as well 3 rd ISCLS, Hyderabad 12

Subanta analyzer INPUT TEXT ↓ PRE-PROCESSOR ↓ VERB DATABASE LIGHT POS TAGGING AVYAYA DATABASE ↓ SUBANTA RECOGNIZER VIBHAKTI DATABASE ↓ SUBANTA RULES SUBANTA ANALYZER SANDHI RULES ↓ SUBANTA ANALYSIS 3 rd ISCLS, Hyderabad 13

Subanta Analyzer • Works on a subanta rulebase and examplebase • Subanta eligibility – check fixed lists (punctuations, avyayas, verbs) • If found tag • Else mark them SUBANTA • Check it in dictionary – If found store separately – Else start subanta processing 3 rd ISCLS, Hyderabad 14

Subanta Analyzer • check example-base – If found tag else continue • Template search – Evaluate string as per set templates – Split it in parts and match the viccheda patterns – If found obtain corresponding analysis – Else tag the input SUBANTA 3 rd ISCLS, Hyderabad 15

Demo Live demo from JNU server demo from localhost 3 rd ISCLS, Hyderabad 16

K danta Analyzer 3 rd ISCLS, Hyderabad 17

K danta Analysis • The process of k danta analysis mechanism is divided into two sections - recognition and analysis. • The k danta recognition starts by an exclusion process. The verb forms, avyayas and punctuations are excluded by running POS tagger • The nominal bases are obtained by the subanta analyzer These nominal bases are then checked in fixed lists. This may result in some of the subantas being marked for k danta. • The remaining subantas are sent to the k danta recognizer and analyzer system for recognition and analysis using following steps – 3 rd ISCLS, Hyderabad 18

K danta Analysis • check the k danta database, annotated corpus and k danta-tagged Monier Williams Sanskrit Digital Dictionary (MWSDD). • the subantas still untagged for k danta are sent to the rule base for k danta checking. • there may still remain an untagged k danta subanta. This will count as failure of the system. • ������� ["����� ", "��� +������ ", "noun_m"]_KR 3 rd ISCLS, Hyderabad 19

Ti anta Analysis • The methodology is a mix of using verb database and reverse Paninian processing • pre-processing • take token by token • confirm the verb (dict, check suffixes), ignore others • Check database • If not found start analysis – analyze suffixes – evaluate remaining string for base (dict check for bases) – result 3 rd ISCLS, Hyderabad 20

POS Tagger • Rule-based tagger is developed for Sanskrit Language. • There are three kinds of tags in this tagset- Word class main tags, feature sub-tags, punctuation tags. • The tag as a whole is a combination of word class main tag with feature sub-tags separated by an underscore • All the tags bear Sanskrit names with letter-digit acronymic in Roman script • Tagset (JNU server) • Tagset (localhost) 3 rd ISCLS, Hyderabad 21

POS Tagger Input text Pre processing Fixed list tagger Morph analyzer Disambiguator* Result normalizer Display tagged text 3 rd ISCLS, Hyderabad 22

Gender Analyzer • Along with the information of vibhakti and number, it is also necessary to have information of gender. In Sanskrit there is agreement within noun phrase in terms of vibhakti, number and gender. While translating a Sanskrit sentence into Hindi, it is necessary to know what would be collocational gender of the sentence, otherwise the whole translation may be wrong. 3 rd ISCLS, Hyderabad 23

Gender Analyzer Input Sanskrit text Un-anlayzed Text Lexical lookup Application of Subanta Analyzer Subanta Analyzed Text Lexical lookup Un-analyzed text Application of rulebase Check gender agreement within a noun phrase Suggest the gender of the noun phrase for Hindi translation 3 rd ISCLS, Hyderabad 24

Demo live demo from JNU server demo from localhost 3 rd ISCLS, Hyderabad 25

K raka Analyzer VERB ID VERB ANALYSIS NON—VERB ID SUBANTA ANALYSIS K K CHECK* K RAKA RULES* SPECIAL CONDITIONS K RAKA ASSIGNMENT 3 rd ISCLS, Hyderabad 26

Conclusion • The authors in this paper have presented an ongoing work for developing a complete SAS. Currently, the SAS has some modules partially developed and some under development. • Significant future additions will be the – Taddhita, samasa modules – ambiguity resolution modules – System integration module – Evaluation module 3 rd ISCLS, Hyderabad 27

Thank You! http: //sanskrit. jnu. ac. in 3 rd ISCLS, Hyderabad 28