Stemming Algorithms 9142608 9142609 From www mis nsysu

Stemming Algorithms 資訊擷取與推薦技術：期中報告 指導教授：黃三益 老師 學生： 9142608 黃哲修 9142609 張家豪 From www. mis. nsysu. edu. tw/~syhwang/Courses/IR/Stemming. Algorithms. ppt, modified by Sumanta

The Porter Algorithm n Word = Stem + Affix(es) n E. g. , generalizations = general + ization + s n Stemming is the determination of the stem of a given word n Porter’s stemmer is a rule-based algorithm n E. g. , ational → ate (apply: relational → relate) n Porter’s stemmer is heuristic, in that it is a practical method not guaranteed to be optimal

The Porter Stemmer: Definitions n CONSONANT: a letter other than A, E, I, O, U, and Y preceded by consonant (in TOY, consonants are T, Y; in SYZYGY they are S, Z, G) n VOWEL: any other letter n With this definition all words and parts of words are of form: [C](VC)m[V] C=string of one or more consonants (con+) and [C] indicates arbitrary presence of the contents, i. e. , possibly empty string as well. V=string of one or more vowels and [V] indicates arbitrary … n E. g. , n Troubles n C VC VC = C(VC)2 n m is the measure of the word n m = 0: TR, EE, TREE, Y, BY n m = 1: TROUBLE, OATS, TREES, IVY n m = 2: TROUBLES, PRIVATE, OATEN, ORRERY Spring 2002 NLE 3

Rule Format n Rules are of the form (condition) S 1 → S 2 where S 1 and S 2 are suffixes. Given a set of rules, only the one with the longest matching suffix S 1 is applies. n Conditions: 1. m --- measure of the stem m = k or m > k, where k is an integer 2. *X --- the stem ends with a given letter X 3. *v*--- the stem contains a vowel 4. *d --- the stem ends in double consonant 5. *o --- the stem ends with a consonant-vowel-consonant sequence, where the final consonant is not w, x or y, (e. g. , wil, hop) n Rules are divided into sets and in each successive step one set of rules is applied.

Porter Steps n Each step corresponds to a set of rules. The rules in a step are examined in sequence , and only one rule from a step can apply { } step 1 a(word); step 1 b(stem); if (the second or third rule of step 1 b was used) step 1 b 1(stem); step 1 c(stem); step 2(stem); step 3(stem); step 4(stem); step 5 a(stem); step 5 b(stem);

Should be: ti

if the second or third rule of step 1 b was used

Examples/Problems Step 1 a Step 4 n computers → computer → comput Step 1 b n singing → sing n generalizations → n information → n instructor → n Try words of your own …

Porter’s Mishaps n On-line Porter’s at http: //textanalysisonline. com/nltk- porter-stemmer gives n gas (noun) → ga n gases (plural) → gase n gasses (verb, present tense) → gass n gassing (verb, present continuous) → gass n gaseous (adjective) → gaseou This is not good – all these words should ideally reduce to the same stem. n Trade-off: More rules (accurate but slow) vs Less rules (efficient but sometimes wrong). n Google does give different results for gas and gases, so maybe they use these Porter rules: -)