The Trouble with House Elves Experiments in Computational

The Trouble with House Elves Experiments in Computational Folkloristics Timothy R. Tangherlini

2

A story… It was the old counselor from Skårupgård who came riding with four headless horses to Todbjærg church. He always drove out of the northern gate, and there by the gate was a stall, they could never keep that stall door closed. They had a farmhand who closed it once after it had sprung open. But one night, after he'd gone to bed, something came after the farmhand it lifted his bed straight up to the rafters and crushed him quite hard. Then the farmhand shouted and asked them to stop lifting him up there. "No, you've tormented us, but now you'll die. . . " I heard that's how two farmhands were crushed to death. He wanted to close the door and then they never tried to close it again. 3

• Some standard questions: – Role of ghosts in late 19 th century Denmark? – Origins of the story? – Structure of the story? – Who, what, where of this story? • Is there a need for a computational folkloristics? 4

Folklore • Early history of the discipline: – Philological – National Romanticism • Johann Gottfied Herder (1744 -1803) – Wilhelm (1786 -1859) and Jacob (1785 -1863) Grimm – Search for original forms 5

Romantic Nationalism in the Nordic lands • Asbjørnsen and Moe (Norway) – • • • Development of the Norwegian language Linnaeus and the rush to categorize (Sweden) The ballad, archaeology and Svend Grundtvig (Denmark) The Kalevala and folklore as a science (Finland) 6

Mapping Folklore • Historic-geographic method – Kaarle and Julius Krohn (1906 -1924) • Focused work on the Finnish epic, Kalevala • Led to the type index of folk literature (Antti Aarne) – Ripples on a pond theory of folklore diffusion 7

Maps in the study of culture • Geography is not an inert container, is not a box where cultural history "happens, " but an active force, that pervades the literary field and shapes its depth. Making the connection between geography and literature explicit. . . will allow us to see some significant relationships that have so far escaped us – (Moretti 1998, 3). 8

A New Historic-Geographic Method • Folklore as a process: – in time and space – emerges from the dialectic between individuals and tradition • Maps can help model relationships between: – People – Environment – Folk Repertoires 9

Study Corpus • Evald Tang Kristensen (1843 -1929) – Actively collected from 1865 -1923 – 219 collecting trips • 6500+ named informants • 24, 000 manuscript pages • 250, 000 published expressions 10

A multi-level folklore browser • People • Places • Stories 11

Experiments in mapping 1. Mapping collecting routes • 2. Challenge Question: Did Tang Kristensen’s published statements about his collecting accurately reflect his collecting work? Mapping individual repertoire distribution • • 3. CQ: Does individual mobility influence the range of places mentioned in stories? CQ: Do other informant features, such as gender, influence range of places mentioned? Mapping by story features against individual repertoire • CQ: Are there patterns, ala Moretti, that become apparent in the visualization of stories by repertoire, genre and/or story topic? 12

Experiment 1: Mapping Collecting Routes – ETK presents himself as a West Jutlander – Political motivations • Aftermath of Napoleonic wars and Danish bankruptcy (1814) • Loss of Schleswig to Bismarck (1865) • Urbanization – Search for “authentic” Danish culture – What do the collecting routes reveal? 13

Experiments 2 & 3: Mapping Repertoire • Theory: Individual biography influences repertoire and its features • Hypothesis: Classes of individuals have different degrees of physical mobility, and this is reflected in their storytelling • Hope: Maps reveal interesting patterns of placesmentioned – A Caveat: My main interest, and the vast majority of the collection, are based on legends, stories that refract the lived environments and social organization of the tradition participants 14

Experiment 2: Place Name Distribution and Mobility – Target: repertoires of 5 storytellers – Limit: only stories that mention places – Method • Plot place names mentioned by storyteller • Calculate Standard Deviation Ellipse distribution patterns for places mentioned in storyteller repertoires • Look for patterns in the underlying place name distribution 15

Experiment 3: Can unsupervised learning on text help in pattern discovery? 16

Experiment 3: Unsupervised learning and Repertoire clusters – Target: repertoires of 5 storytellers – Limit: only stories that mention places – Method • Convert stories to TFIDF vector representations • Force dimensionality reduction using SVD • Cluster: ECM by storyteller – eliminate small clusters • Project results into GIS • Calculate distribution ellipses for each cluster in each person’s repertoire 17

A Crisis… • Maps were informative since new patterns in the geographic distribution of stories were discovered… – why hadn’t I known about these patterns before? • What other types of patterns, some very small, some very large are lurking in the data? • How can I be sure that my selection of examples is representative or even accurate? 18

A Classic Folklore Problem • Classification in folklore – 1 text = 1 classifier • What happens when the classifier was designed for a different research problem? • Are we missing patterns that are not solely related to single topic classifiers? • Are we missing stories in our searches because of these single topic classifiers? • Does this limit our ability to work with a large archive? 19

Current folklore classifiers are very expensive 20

A lost story… It was the old counselor from Skårupgård who came riding with four headless horses to Todbjærg church. He always drove out of the northern gate, and there by the gate was a stall, they could never keep that stall door closed. They had a farmhand who closed it once after it had sprung open. But one night, after he'd gone to bed, something came after the farmhand it lifted his bed straight up to the rafters and crushed him quite hard. Then the farmhand shouted and asked them to stop lifting him up there. "No, you've tormented us, but now you'll die. . . " I heard that's how two farmhands were crushed to death. He wanted to close the door and then they never tried to close it again. 21

Networks to the rescue? • Folklore as traditional communication across social networks – Folklore networks • Social networks of tradition participants • Networks of scholars and collectors • Networks of stories – External networks • Communications networks • Transportation networks • Affiliation networks – Internal networks • Linguistic networks 22

Connecting the dots… P 1 P 7 I 3 P 8 S 2 S 1 I 1 P 2 I 3 I 2 P 4 P 5 P 3 S 4 P 6 I 3 23

Storyteller networks • Local networks • Connect all storytellers in a given parish • Connect all storytellers in a family • Fieldtrip networks • Connect all storytellers on a given fieldtrip • Collector-Storyteller networks • Connect all storytellers to all collectors with whom they worked • Inferred / Affiliation networks • Connect storytellers by work groups (eg millers, fiddlers, etc) • Connect storytellers by other affiliations (eg gender, age, education) 24

Story networks • Connect stories to: – People: • storytellers • people mentioned – Places • places collected • places mentioned – Each other • • By shared indexing By shared keyword (keyword extraction) By shared topic (topic modeling using LDA) By shallow ontology (tango index) 25

An initial graph of the ETK study corpus 26

Lost in a thicket of stories, keywords, etc 27

Folklore Spaghetti 28

Graph clustering • Use a tuned version of MCL clustering for graphs – iteratively generates stochastic matrices, also known as Markov matrices (van Dongen 2000) – 2973 nodes / 52663 edges 29

Structure emerges and the graph becomes useful 30

Remember our ghost story? • DS IV 650 • Classified as a story about manor lords, not ghosts! • Impossible to find in the archive • Can I use networks to find this story? • Will it help me find other stories of interest? 31

32

33

Almost all the surrounding stories are cataloged as ghost stories! 34

DS II B 147 is a story of interest—not a ghost story but strongly connected to DS IV 650… 35

DS II B 147 • A story about a house elf at a farm in Egå. . . • Ends as follows: – When they got home, the farmhand was happy because now he’d gotten something to use for feed, and afterward nis could go and feed just as much as he wanted to. Then they got another farmhand, and he didn’t want to let him go on like that. But he got lifted up in his bed and all the way up to the rafters, so he lay there dead when people got up the next morning. 36

The trouble with house elves… • You can’t always find them… • They act in unpredictable ways… • The things they do turn out to be pretty mean and nasty 37

New research question • What is the relationship between ghosts and house elves in 19 th century Denmark and why might there be such a relationship? 38

Some tentative conclusions 39

Directions for future work • Labeling – Can we automatically label nodes given a sparsely labeled graph? (LDA-G, Homophily algorithms) • Anomaly detection / Community detection – Can we automatically find “stories of interest” on our graph? • Multimodal networks – Integrate network information from several networks • Dynamic networks – Understanding how network changes over time • Geographic visualization of network models 40

A Very Special Thanks to • IPAM – Peter Jones – Mark Green – Russ Caflisch • Colleagues and friends from Search Engines 2007 41

Additional thanks to • • • Peter Broadwell, UCLA James Abello, DIMACS Tina Eliassi-Rad, LLNL/Rutgers Nischal Devanur, Rutgers UCLA’s Center for Digital Humanities 42

Funded by… – Nordic Council of Ministers – The American Council of Learned Societies – NSF Eager Grant IIS- 0970179 • With Lancaster (ECAI), Buckland (ECAI), Eliassi. Rad (Rutgers) and Faloutsos (CMU) – Google Books Humanities grant – Many ideas derived from • NEH Institute for Advanced Topics in Digital Humanities, “Networks and Network Analysis for the Humanities” (NEH HT 5001609) 43