Programming for Engineers in Python Lecture 4 Data

Programming for Engineers in Python Lecture 4: Data Analysis Autumn 2011 -12 1

Lecture 3: Highlights • Simulation: power lines and rare diseases • Plan before coding • Using Modules • Import • Constants 2

Tuples • Fixed size • Immutable (similarly to Strings) • What are they good for (compared to list)? • Simpler (“light weight”) • Staff multiple things into a single container • Immutable (e. g. , records in database) 3

Dictionaries (Hash Tables) • Key – Value mapping • Fast! • Usage: • Database • Dictionary • Phone book keys values 4

Dictionaries (Cont. ) 5

Dictionaries (Cont. ) 6

Dict – Initiate Dictionary from a List 7

Sorting Lists 8

Sorting Lists by Two Criteria First priority length, second lexicographical order 9

Today: Data Analysis of data is a process of inspecting, cleaning, transforming, and modeling data with the goal of highlighting useful information, suggesting conclusions, and supporting decision making. Data analysis has multiple facets and approaches, encompassing diverse techniques under a variety of names, in different business, science, and social science domains. • Descriptive • Predictive 10

Data Analysis Examples • Google • Stock market trends • Genome-disease association • Face recognition • Production yield • Business intelligence • Speech recognition • Text categorization 11

Text Categorization / Document Classification 12

How is it Done? • Manually • Automatically • Gather document statistics • Measure how similar it is to documents in each category • Today we will collect word-statistics from several well known books 13

Plan • Find data • Collect word statistics • Observe results 14

Find Data • This might be the hardest task for many applications! • Project Gutenberg (http: //www. gutenberg. org/) • Alice's Adventures in Wonderland (http: //www. gutenberg. org/cache/epub/11/pg 11. txt) • The Bible, King James version, Book 1: Genesis (http: //www. gutenberg. org/cache/epub/8001/pg 8001. txt) 15

Reading a Book 16

Flying 17

Print Most Popular Words (High Level) 18

Modular Programming • Top-down approach: first write what you plan to do and then implement the details • Clear for readers • Easy to debug and test • Easy to maintain 19

Print. Most. Popular Build Word-Occurrences Dictionary 20

Print. Most. Popular Sort Words by Occurrences ? http: //docs. python. org/library/operator. html 21

The Code 22

Results 23

And Now for Several Books 24

Results The word “to” as an example Bible L. Carroll 25

How is it Really Done? • Preprocessing (e. g. , words to lower case, remove punctuation signs) • Word count • Enhance statistics • • Discard stop words (e. g. , and, of, a) Stemming (e. g. , go & went) Synonyms ( )מילים נרדפות bigrams, trigrams • Similarity measures to existing documents / categories 26

How is it Really Done? Categories Topics: http: //www. cs. tau. ac. il/courses/py. Prog/1112 a/lectures/4/topics. rbb Categories Hierarchy: http: //www. cs. tau. ac. il/courses/py. Prog/1112 a/lectures/4/rcv 1. topics. hier. orig 27

How is it Really Done? Enhance Statistics Stop words: http: //www. cs. tau. ac. il/courses/py. Prog/1112 a/lectures/4/english. stop After processing: http: //www. cs. tau. ac. il/courses/py. Prog/1112 a/lectures/4/lyrl 2004 -non-v 2_tokens_test_pt 0. dat 28