Data 101 Numbers Graphs and More Numbers Emily

Data 101: Numbers, Graphs, and More Numbers Emily Putnam-Hornstein, MSW Center for Social Services Research University of California at Berkeley March 11, 2008 The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Agenda • Basic Terminology • Common Data Pitfalls • Graphics • Small Groups… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Data Basics… • Descriptive Data – Demographic characteristics of a population, place, office, etc. • Comparisons – Performance trends over time (one time period to another) – Differences/similarities between groups, counties, placement settings, interventions, etc. • Analyses – Exploring the relationship between two events (e. g. , reunifications and re-entries to care) – Looking at the contributions of various factors to some outcome • Y=a+b. X CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Computing a Percent Answers. com Dictionary: Rate • A measure of a part with respect to a whole; a proportion: the mortality rate; a foster care entry rate. What Percentage of Children who were reunified in 2005 reunified within 12 months of entering care? Raw Numbers (counts) # Reunified w/in 12 m = 290 # Reunified (total) = 440

Computing a Rate per 1, 000 Answers. com Dictionary: Rate • A measure of a part with respect to a whole; a proportion: the mortality rate; a foster care entry rate. What was the foster care entry rate in 2005? (i. e. , how many children entered care out of all possible children? ) Raw Numbers (counts) # Entered Care = 1, 333 # Child Population = 363, 376 Scales for a meaningful interpretation…

Measures of Central Tendency Mean: the average value for a range of data Median: the value of the middle item when the data are arranged from smallest to largest Mode: the value that occurs most frequently within the data 124 4 15 7 963 127 15 9 417 1763 7 =9 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley = 9. 7

Measures of Variability Minimum: the smallest value within the data Maximum: the largest value within the data Range: the overall span of the data 4 4 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 7 9 12 15 17 63

Disaggregation • One of the most powerful ways to work with data… • Disaggregation involves dismantling or separating out groups within a population to better understand the dynamics • Useful for identifying critical issues that were previously undetected Aggregate Permanency Outcomes CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Race/Ethnicity County Age Placement Type

2000 July-December First Entries California: Percent Exited to Permanency 72 Months From Entry 85% CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

2000 First Entries California: Percent Exited to Permanency 72 Months From Entry 88% CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 79%

2000 First Entries California: Percent Exited to Permanency 72 Months From Entry by Relative vs. Non-Relative Placement CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley =94% =84% =75%

3 Key Data Samples Entry Cohorts Data Point in Time CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Exit Cohorts

How long do children stay in foster care? January 1, 2005 July 1, 2005 January 1, 2006 Child 1 Child 2 Child 3 Child 4 Child 5 Child 6 Child 7 Child 8 Child 9 Child 10 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

California Example: Age of Children in Foster Care (2003 first entries, 2003 exits, July 1 2004 caseload) Entries % CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

California Example: Age of Children in Foster Care (2003 first entries, 2003 exits, July 1 2004 caseload) Entries Exits % CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

California Example: Age of Children in Foster Care (2003 first entries, 2003 exits, July 1 2004 caseload) Entries Exits Point in Time % CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Continuous vs. Discrete • The average foster child has 2. 6 placements while in foster care – This number makes little sense because the underlying dimension is discrete (i. e. , categorical, discontinuous) 1 2 2. 6 3 4 5 6 x There are 260 placements for every 100 foster children placements Continuous Data Age Days in Care Percentages / Rates CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Discrete Data Race/Ethnicity Placement Type Referral Reason

Correlation • Two “events” that covary with one another… Negative Positive Correlation== % Reentries Births to % Teen Moms Event 1 Event 2 or Event 1 Event 2 % Reunified within 6 months CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Percent Change Time Period 1 10 children Time Period 2 11 children CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Percent Change Time Period 1 10% Time Period 2 12% % % CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Exercise: Percent Change Calculation Comparison Referral Rate (time period 2): CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 48. 3 -4. 7% 12. 0 10. 8 -10% Percent Change: Minor Differences due to Rounding… Baseline Referral Rate (time period 1): 50. 7

CWS Outcomes System Summary CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

January 2004 -January 2008 California CWS Outcomes System: Federal Measures, Percent IMPROVEMENT CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Cross-Sectional vs. Longitudinal Cross-Sectional (repeated) * Figure 5. 23 retrieved from: http: //www. mrs. umn/edu/~ratliffj/psy 1051/cross. htm

There are three kinds of lies: Lies, Damned Lies and Statistics ^ Misused Statistics CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Six Ways to Misuse Data (without actually lying!): 1) Using Raw Numbers instead of Ratios 2) Rank Data 3) Compare Apples and Oranges 4) Use ‘snapshots’ of Small Samples 5) Rely on Unrepresentative Findings 6) Logically ‘flip’ Statistics 7) Falsely Assume an Association to be Causal CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

1) Numbers that conceal more than they reveal… Challenger: “Violent crime in Anytown, CA has increased over the last year. 100 more crimes were recorded. ” Incumbent: “Violent crime in Anytown, CA has decreased by 2% over the last year. ” Who is telling the truth? They both are. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“There approximately 82, 000 children in the child welfare system in California – 20% of foster children in the nation, and the largest foster care population of all 50 states. ” National Center for Youth Law, “Broken Promises”, 2006 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“There approximately 82, 000 children in the child welfare system in California – 20% of foster children in the nation, and the largest foster care population of all 50 states. ” NCYL, 2006 Factually true? • Yes Informative? • Not very. Ø What if California has one of the largest child populations of all states? Ø What if California has one of the smallest child populations of all states? Misleading? • Maybe… Ø What is the point being made? Ø Telling us that California has the largest foster care population does not shed any light on how the state is performing! CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

2) Rank Data Two streets in Anytown, CA…. $$ “Jane Doe is the poorest person living on Moneybags Avenue. ” e v A “Joe Shmoe is the wealthiest person living on Poverty Blvd. ” Pover ty Blv d CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley It’s all relative… And SOMEONE will always be ranked last (and first)

“San Francisco ranks 55 out of 58 counties when it comes to state and national performance measures…” SF Chronicle, “No refuge. For Foster youth, it’s a state of chance”, November 15, 2005 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“San Francisco ranks 55 out of 58 counties when it comes to state and national performance measures…” SF Chronicle San Francisco: AB 636 UCB State Measures (Used in NCYL Ranking) % IMPROVEMENT Jan ‘ 04 compared to June ‘ 06 (+) indicates a measure where a % increase equals improvement. (-) indicates a measure where a % decrease equals improvement. indicates a measure where performance declined. • Rankings mask improvement over time. • However, even improvement over time and relatively high rankings can be misleading.

3) Compare Apples and Oranges Two doctors in Anytown, CA… Doctor #1 Doctor #2 Doctor of the Year? 2/1000 20/1000 What if the populations served by each doctor were very different? CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento. ” SF Chronicle, “Accidents of Geography”, March 8, 2006 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento. ” 1. Different families and children served? 2. Different related outcomes? • First entry rates in Fresno are consistently lower • Re-entries in Fresno are also lower… 3. Other considerations… • Resources available, resource allocation choices • Performance trends over time CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

4) Data snapshots… Crime in Anytown, CA… Number of Crimes Period 1: 76 Period 2: 51 Average = 73. 5 No change. Crime jumped by 49%!! Period 3: 91 Crime dropped by 16% Period 4: 76 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area. . . ” SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…” Abuse in Care Rate Period 1: 1. 80% Period 2: 1. 64% 3: 0. 84% 0. 00% = 2/111 = 2/122 Period 100% improvement! = 1/119 Period 4: =0 0 Children Abused! Responsible use of the data prevents us from making any of these claims (positive or negative). The sample is too small; the time frame too limited. CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

5) Unrepresentative findings… Survey of people in Anytown, CA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium… But what if you were then told that respondents had been sampled from a list of season football ticket holders? CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers. ” “My Word”, Oakland Tribune, May 25, 2006 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers. ” Oakland Tribune Factually true? • Yes. Misleading? • Yes. – This was a survey of emancipated foster youth – Emancipated youth represent a distinct subset of the foster care population – This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

6) Logical “Flipping”… Headline in The Anytown Chronicle: 60% of violent crimes are committed by men who did not graduate from high school. “Flip” 60% of male high school drop-outs commit violent crimes? CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10, 000. ” Bath and Haapala, 1993 as cited in “Shattered bonds: The color of child welfare” by Dorothy Roberts CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10, 000. ” • In reading statistics such as the above, there is a tendency to want to directionally “Flip” the interpretation • But the original and flipped statements have very different meanings! 75% of neglect cases involved families with incomes under $10, 000 DOES NOT MEAN 75% of families with incomes under $10, 000 have open neglect cases Put more simply, just because most neglected children are poor does not mean that most poor children are neglected CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley Families with open neglect cases Families with incomes under $10, 000

7) False Causality… A study of Anytown residents makes the following claim: Adults with short hair are, on average, more than 3 inches taller than those with long hair. Hair Length X Height Gender Finding an association between two factors does not mean that one causes the other… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…” As reported in Sidebotham and Heron’s 2006 article CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

“A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…” • Should one conclude that prematurity is a causal factor in maltreatment? prematurity maltreatment a third factor (Drug use? ) CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Graphs / Charts • Keep it simple… • Use consistent color themes when possible • Think about the type of data being presented (discrete vs. continuous) • Label Clearly • Tell a story • Look at presentations on the UC site! CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Stacked Bar Chart Ethnicity and Path through the Child Welfare System: California 2006 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Pie Chart Ethnicity of Children in Foster Care: California 2006 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

3 D-Area Chart 2006 California: Referrals per 1, 000 by Age and Ethnicity *Series Total CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

(complex) Line Chart California: First Entries by Race/Ethnicity TOTAL Hispanic White Black Asian/PI Native American 1998 1999 2000 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 2001 2002 2003 2004 2005 2006 2007

(complex) Line Chart California: Foster Care Caseload by Race/Ethnicity TOTAL Black Hispanic White Asian/PI Native American 1998 1999 2000 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley 2001 2002 2003 2004 2005 2006 2007

Small Group Topics… Group 1: Explore County to County variation in Group Home use in 2007 Group 2: Miscellaneous Group 3: Describe any statewide trends in Group Home use (vs. other placements) over time Group 4: Explore the placement stability of the Group Home population in care for 24 months or mroe Group 5: Describe the Group Home Population in California in 2007 CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Group 1: Explore County to County variation in Group Home use in 2007 • How many children were in GH care in Sacramento County? Alameda County? • What percentage of the GH population is female in Humboldt County? – How does this compare with CA as a whole? – What conclusions can you draw about Humboldt? • Compare the ethnic distribution of the GH population in Los Angeles County with that of San Diego County. • Other observation(s)… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Group 2: Miscellaneous • In 1999, what percentage of children first entering foster care (“first entry”) had a first placement in a GH? What was the percentage in 2006? • In 1999, what percentage of children re-entering foster care (“other entry”) had a first placement in a GH? What was the percentage in 2006? – Any thoughts on why this may be the case? • In 2006, what percentage of children exiting from care with a last placement in a GH exited to emancipation? • The number of children exiting from GH to reunification has increased over time. What was the count in 1998? And in 2006? • Other observation(s)… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Group 3: Describe statewide trends in Group Home use (vs. other placements) over time • • • How has the size of the GH population changed over time? What percentage of the foster care population was in GH care on January 1, 1999? And in 2007? – How do you reconcile this with the fact that the count of children in GH care has gotten smaller over time? How has the size of the population in other placement settings changed over this same time period? – Kin? Foster? FFA? Shelter? – Overall out of home population? The overall out of home care population has decreased over time. What additional data do you need in order to assess whether this is a real change? Other observation(s)… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Group 4: Explore the placement stability of the Group Home population of children in care for 24 months or more • How has the total size of the population of children in care for 2+ years (and who are now in GH care) changed over time? – And what has been the trend over time for children in two or fewer vs. three or more placements been? • In 2006, what percentage of children in GH care for 2+ years had been in two or fewer placements? – What percentage in foster homes had been in two or fewer placements? – And kinship homes? • Is it reasonable to conclude that placement in Group Home Care causes placement instability? • Other observation(s)… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Group 5: Describe the Group Home Population in California in 2007 • What was the total PIT count of children in group home care in 2007? • Which age group had the greatest number of children in GH care? • Were there any infants in GH care? – Any thoughts on why this might be? • What percentage of children in GH care were ages 11 -15 years? • Are any gender differences observed? • Other observation(s)… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

A quick look at the website… CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley

Emily Putnam-Hornstein eputnamhornstein@berkeley. edu CSSR. BERKELEY. EDU/UCB_CHILDWELFARE Needell, B. , Webster, D. , Armijo, M. , Lee, S. , Dawson, W. , Magruder, J. , Exel, M. , Zimmerman, K. , Simon, V. , Putnam-Hornstein, E. , Frerer, K. , Ataie, Y. , Atkinson, L. , Blumberg, R. , Henry, C. , & Cuccaro. Alamin, S. (2007). Child Welfare Services Reports for California. Retrieved [month day, year], from University of California at Berkeley Center for Social Services Research website. URL: <http: //cssr. berkeley. edu/ucb_childwelfare> CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley