Analysis of Social Media Trend Analysis YiChia Wang

Analysis of Social Media Trend Analysis Yi-Chia Wang LTI 2 nd year Master student

Introduction • Document streams • Arrive continuously over time • E-mail, news articles, search engine query logs, … • Identify topics in document streams • Topic detection and tracking • Text mining • Visualization • … • Is there a better organizing principle for the enormous archives of document streams? • Temporal information in document streams Oct-30 Analysis of Social Media 2007 Trausan-Matu et al. , 2007 2

“Burst of activity” • Topics appear, grow in intensity for a period of time, and then fade away. • Bursts correspond to points at which the intensity of message arrivals increases sharply • Problems with naive identification of bursts • Easily identifying large numbers of short bursts • Fragmenting long burst into many smaller ones • Goal: identifying bursts only when they have sufficient intensity Oct-30 Analysis of Social Media 2007 3

Bursty and Hierarchical Structure in Streams Jon Kleinberg Department of Computer Science Cornell University SIGKDD ‘ 02

• Idea: periods of lower message intensity interleave with periods of higher message intensity Two-state Automaton (A) Model q 1 q 0 q 0 time • A begins in state q 0 • A changes state with probability p • When in state q 0, messages are emitted at a slow rate; when in state q 1, messages are emitted at a faster rate Oct-30 Analysis of Social Media 2007 1 -p p q 0 q 1 p 5

Exponential Distribution • Modeling the message emission rate • Modeling the time gap between messages and • Modeling by exponential distribution with parameter being the rate of message arrivals Wikipedia Oct-30 Analysis of Social Media 2007 6

Two-state Automaton (A) Model • Formally, given: • • messages with specified arrival times : inter-arrival gaps • We want to determine the conditional probability of a state sequence Oct-30 Analysis of Social Media 2007 7

Two-state Automaton (A) Model • Finding a state sequence q maximizing the probability • Equivalently, minimizing the following cost function: Favoring state sequences that conform well to the sequence x of gap values Favoring sequences with a small number of state transitions Oct-30 Analysis of Social Media 2007 8

Infinite-state Automata Model • Cost Function Oct-30 Analysis of Social Media 2007 9

Computing a minimum-cost state sequence • THEOREM: If q* is an optimal state sequence in , then it is also an optimal state sequence in • Dynamic programming is used for searching an optimal state sequence Oct-30 Analysis of Social Media 2007 10

Bursts exhibit a natural nested structure A burst of intensity j is a maximal interval over which a part of state sequence is in a state of index j or Bursts can also be higher represented as a tree. Each burst is a node in the tree Oct-30 Analysis of Social Media 2007 11

Experiments • The model makes sense for many datasets (of an analogous flavor) • • Oct-30 Email Titles of conference papers U. S. Presidential State of the Union Addresses Web clickstreams Analysis of Social Media 2007 12

Email Dataset • Is the appearance of messages containing particular words exhibits a burst in the vicinity of significant times such as deadlines? • Author’s own collection of email • June 9, 1997 – August 23, 2001 • 34344 messages (41. 7 MB) • Focusing on the response set Oct-30 Analysis of Social Media 2007 13

Results for the Word - ITR • ITR is the name of a large NSF program • The author wrote 2 proposals for it in 1999 -2000; one is a small proposal while another is a large one • The intervals are annotated with the first and last dates of the messages • The first subtree splits further into 2 subtrees • For the 2 nd subtree, there is no burst since the author did not continue the submission Oct-30 Analysis of Social Media 2007 14

Results for the Word - prelim • Prelim is the term used at Cornell for non-final exams • The author taught courses in 4 of the 8 semesters covered by athe The 2 structures suggest how large folder of emailof might naturally beeach dividedof into a collection email, and hierarchical set of sub-folders around certain these courses had 2 prelims key events, based only on the rate of message • For thearrivals first of these courses, there was a special course email account • For remaining 3 courses, each corresponds to a long burst and 2 shorter, more intense bursts for the particular prelims Oct-30 Analysis of Social Media 2007 15

Titles of Conference Papers • Goal: extracting bursts in term usage from the titles of conference papers over the past several decades • Problem: conference papers arrive in discrete batches every half or one year no message inter-arrivals gaps • Modified automaton model: • Generating batched arrivals • For each state, there is an expected fraction of relevant documents • Bursty is identified if the fraction of relevant documents increases Oct-30 Analysis of Social Media 2007 16

Titles of Conference Papers • Cost function for each arrival batch: • The weight of the burst : the improvement in cost by using state q 1 rather than state q 0 Oct-30 Analysis of Social Media 2007 17

SIGMOD & VLDB, 1975 -2001 • Considering each word in paper titles • The 30 bursts of highest weight • The bursts with no ending date the interval extends to the most recent conference • These bursty words are different from a list of common words • The bursts are picking up trend in language use Oct-30 Analysis of Social Media 2007 18

STOC & FOCS, 1969 -2001 • The 30 bursts of highest weight • Particular titling conventions that were in fashion for certain periods • “How to construct random functions” • … Oct-30 Analysis of Social Media 2007 19

U. S. Presidential State of the Union Addresses Kleinbergh, SIGKDD ‘ 02 Oct-30 Analysis of Social Media 2007 20

Web usage data – clickstreams • Settings: • 80 undergraduate students • Two and a half months in Spring 2000 • For every URL w, all bursts in the stream of visits to w are determined • Focusing on high-weighted bursts as well as those that involve at least 10 distinct users • Results: • High-ranked bursts involve the URLs of the online class reading assignments, centered on intervals shortly before and during the weekly sessions at which they were discussed Oct-30 Analysis of Social Media 2007 21

Conclusions • Modeling streams using an infinite-state automaton • State transitions lead to bursts • First story detection: a single message on which the associated state transition occurred • The model offers a means of structuring the information from our patterns of interacting and communicating • Document streams have a strong temporal character • In many domains, we are accumulating detailed records of our own communication and behavior Oct-30 Analysis of Social Media 2007 22