DRAFT for internal project use only Education Modeling

DRAFT – for internal project use only Education Modeling Collaboration: LANL Initial Model Implementation Patrick Kelly Benjamin Sims Stephan Eidenbenz Joanne Wendelberger Steven Stringer Los Alamos National Laboratory UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 1

DRAFT – for internal project use only Initial model is complete n LANL’s initial model incorporates all the core elements needed for the final model. These include: • • • Model entities and their attributes (students, teachers, classrooms, etc. ) A structure to guide entity interactions (school system, schools, classes) A process for assigning students and teachers to school systems, schools, classrooms, and classes Functions for assigning scores/grades to students (based on student attributes, interactions with teachers and other students, and previous scores/grades in transcript) Methods for capturing and displaying model outputs UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 2

DRAFT – for internal project use only Model builds on input from the entire team n Ongoing interaction between modelers, statistical analysts, and education experts drives development • • • Previous discussions elicited a list of key parameters that influence student success, and related data sources Model core uses a few high-level parameters from these discussions The values of these high-level parameters will be tied to other parameters and data sources as model development continues More complex interaction structures and class assignment processes will be implemented based on actual school districts, curriculums, and policies Statistical analysis of SJUSD data will provide information for populating and calibrating the model UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 3

DRAFT – for internal project use only Benefits of multi-agent modeling approach n Multi-agent models represent and track individuals through a social system over time • n In this case, students and teachers in a school system Useful characteristics of multi-agent models • Bottom-up approach enables more detailed exploration of connections between individual behavior and systemic factors • Greater ability to manipulate and understand factors that shape individual student and teacher performance over time • Possible to model large populations so results can be statistically compared to real student and teacher data UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 4

DRAFT – for internal project use only Model contains this core set of objects n Student n Teacher n Classroom • n School • n Master “control mechanism” manages simulation and generates reports Student Transcript • n Mechanism for grouping teachers, students, classrooms School System • n Physical location where students and teachers interact in individual classes Record of achievement through the years (e. g. report card). Individual Class • An instance of a specific course, in a specific location, with a specific teacher, and specific list of students UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 5

DRAFT – for internal project use only Initial model uses a basic set of object attributes (Possible future attributes in gray) n Student • n Teacher • n Ability (motivation, gender, ethnicity, areas of expertise, …) Classroom • n Ability, grade level (motivation, socioeconomic status, age, gender, ethnicity, ESL status, …) (Quality) School • Curriculum UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 6

DRAFT – for internal project use only Current model runs assume a simplified school system n Students and teachers remain in the school to which they were originally assigned n All schools teach K-12 n A class lasts a full school year n All grade levels have the same set of classes (so they are more like general subject areas) n Student “score” (could represent test score or grade in the subject) is calculated once a year for each class UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 7

DRAFT – for internal project use only Model cycles through this loop each year Assign Classes Assign Scores Advance to Next Year (quarter, semester, etc. ) UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 8

DRAFT – for internal project use only Setting up a model run n From Input Files: School. System Schools (Curriculum) Classrooms Teachers (Ability) Students (Ability, Grade Level) n Initializations: Classrooms, Teachers, and Students are all assigned to the various schools by an internal algorithm. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 9

DRAFT – for internal project use only Example: school system setup School “A” C 1 C 2 T 1 S 2 School “B” C 3 C 4 T 2 … T 3 S 100 S 101 School “C” C 5 C 6 T 4 S 102 … UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA C 7 T 5 S 200 S 201 S 202 C 8 T 6 … S 300 Slide 10

DRAFT – for internal project use only Assigning classes n School has classrooms, teachers, students, and a curriculum n Curriculum is a list of each course to be taught: • • n Grade Level (e. g. K-12) Course Identifier (integer code for “Math”, “English”, “Social Studies” … ) Course Name (actual string for interpretation “Math”, “English” … ) Number Offered (e. g. “Three different classrooms offering Second-Grade Math”) Students are assigned to classrooms with teachers and students that vary from year to year • • Realistic to have some overlap and some differences in classmates and teachers from year to year Placing students with the same teachers and classmates each year leads to a completely static model UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 11

DRAFT – for internal project use only Initial class assignment method n Need a simple method that generates realistic mix of stability and change for the initial model • Set up one Individual Class for each entry in the defined curriculum • Number of Individual Classes for each grade/course in the schedule determined by the Number Offered value specified at initialization • Use a simple round-robin “card dealing” method to assign one teacher and one classroom to each Individual Class • For now, every student is enrolled into each course at their grade level. The specific Individual Class is determined by a heuristic shuffling algorithm. This changes the overall classmate mix as student progress through the system. — For courses at grade level G, we perform a “card dealing” strategy assigning G students at a time before moving to the next Individual Class UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 12

DRAFT – for internal project use only Example: Assigning students to classes Assume we have three courses offered for Fourth-Grade Math. Individual Class “A” Individual Class “B” Individual Class “C” Classroom: 205 Teacher: Mrs. Adams Classroom: 206 Classroom: 207 Teacher: Mr. Bradley Teacher: Mrs. Chavez S 1 S 2 S 3 S 4 S 13 S 14 S 15 S 16 S 25 S 26 S 27 S 28 S 5 S 6 S 7 S 8 S 17 S 18 S 19 S 20 S 29 S 30 S 31 S 32 S 9 S 10 S 11 S 12 S 21 S 22 S 23 S 24 S 33 S 34 S 35 S 36 (Truth be told, we are actually using G+1 in the code instead of G since Kindergarten is treated as “grade zero”. ) UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 13

DRAFT – for internal project use only Assigning scores n Score assignment is key driver of student outcomes n Initial model uses the following factors assign scores • • n These factors can be tied to a wide range of variables in the data set • n The student’s baseline “ability” to learn the material The teacher’s baseline “ability” to teach the material Average “ability” of students in classroom (captures interactive effects) How the student performed last year in this subject area In the initial model, values are assigned by the modeler We’ve experimented with two potential score-assignment functions • • Main approach: Averaging of factors Alternative approach: Incremental influence UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 14

DRAFT – for internal project use only Main approach to score assignment – averaging n All factors are weighted equally in initial model n Different weighting schemes can be tested or implemented based on data analysis and input from education experts UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 15

DRAFT – for internal project use only Alternative score assignment approach – incremental influence n Produces similar output to averaging approach – useful comparison Define scaled-sigmoid modifier function n Properties: n • • n Function output is always between zero and two. When x is zero, function output is one. When x is less than zero, function output is less than one. When x is greater than zero, function output is greater than one. Standard way of scaling model functions, does not represent behavior UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 16

DRAFT – for internal project use only Incremental influence (cont. ) represents the score a student would receive in the absence of knowledge concerning scores in previous years. We then modify to consider influence (bias? ) from the previous year’s score. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 17

DRAFT – for internal project use only Testing model response: Varying average student ability n Two Student Data Sets: • • Set “A” has narrower range of Ability values (higher average Ability) — Randomly selected between 0. 65 and 1. 00 (uniform). Set “B” has wider range of Ability values (lower average Ability) — Randomly selected between 0. 55 and 1. 00 (uniform). n Teacher Ability values distributed evenly between 0. 78 and 1. 00 n Three schools in the system. • • • n Each has five (5) classrooms assigned during initialization. Each has three (3) teachers assigned during initialization. Each has sixty (60) students assigned into their K-level classes. Follow students as they progress in one subject area through the end of 12 th grade (e. g. “math”) UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 18

DRAFT – for internal project use only Progression of students in group A (higher average) UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 19

DRAFT – for internal project use only Progressions of students in group B (lower average, wider distribution) UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 20

DRAFT – for internal project use only Isolated student score tracks from group B (for clarity) UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 21

DRAFT – for internal project use only These student progressions show that: n As expected for the initial model, there is some movement over time, but no drastic changes in ranking of individuals n The range of student scores at grade 12 is similar to the range at grade K • • Keep in mind that these are scores relative to grade level (if a student scores the same every year, they are progressing as expected) So the model shows that students are, on average, progressing as expected from year to year UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 22

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 K Slide 23

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 1 Slide 24

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 2 Slide 25

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 3 Slide 26

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 4 Slide 27

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 5 Slide 28

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 6 Slide 29

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 7 Slide 30

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 8 Slide 31

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 9 Slide 32

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 10 Slide 33

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 11 Slide 34

DRAFT – for internal project use only Group B score histogram over time UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA 5 0. 9 0 0 - 0. 9 0. 8 5– 0. 8 5 0 0– 0. 8 5 5– 0. 7 0– 0. 7 0. 6 5– 0. 7 5 12 Slide 35

DRAFT – for internal project use only These histograms show that: UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 36

DRAFT – for internal project use only Linking the model to observed data n Identify measurable quantities in our data sets that we can use to quantify model parameters n Tune internal mechanisms to generate data that is statistically similar to the observed data n This will involve: • • n Selection of a refined score assignment function. Tuning internal parameters for that function. Incorporate a more realistic mechanism for class assignments • More accurately reflects what happens in real world UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 37

DRAFT – for internal project use only Action plan n Continue model development to build in additional features. n Incorporate information from San Jose School District database into the modeling process. n Generate data from model that can be used for visualization of behavior over time. n Work with visualization experts to develop materials to communicate modeling results. n Work with education experts to obtain feedback and insights for future development of the model. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U. S. Department of Energy ’s NNSA Slide 38