Challenges and Opportunities for Needs Assessment and Program

+ Challenges and Opportunities for Needs Assessment and Program Planning in the 21 st Century Panel Presentation by Ryan Watkins James W. Altschuld Hsin-Ling (Sonya) Hung Maurya West Meiers

+ 21 st Century Needs Assessment: Theory to Guide Practice Ryan Watkins George Washington University www. needsassessment. org

+ Kurt Lewin n "There is nothing so practical as a good theory. ” n "Experience alone does not create knowledge. ”

+ NEEDS ASSESSMENT n A systematic study of a problem or innovation, incorporating data and opinions from varied sources, in order to make effective decisions or recommendations about what should happen next (Rossett)

+ NEEDS assessment n What are we assessing? n What are NEEDS? n Individual, group, organizational, societal, all? n Expressed, felt, measured, comparative, etc. ? n Can we assess needs alone? n Should we assess needs alone?

+ NEEDS

+ ASSETS RESULTS NEEDS VALUES WANTS

+ CAPACITY SOCIAL MOTIVATION CAPABILITIES ASSETS RESULTS NEEDS JUDGEMENTS WANTS VALUES EXPECTATIONS INDIVIDUAL MOTIVATION

+ SYSTEMS EXPRESSED CAPACITY CAPABILITIES SOCIAL MOTIVATION ASSETS NORMATIVE DECISIONS RESULTS NEEDS JUDGEMENTS WANTS INDIVIDUAL MOTIVATION VALUES EQUITY FELT EXPECTATIONS ASCRIBED COMPARATIVE

+ EXPRESSED GILBERT VON BERTALANFFY SYSTEMS CAPACITY JAMES CAPABILITIES SOCIAL MOTIVATION BANDURA ASSETS NORMATIVE DECISIONS RESULTS MASLOW NEEDS KAUFMAN JUDGEMENTS VALUES FELT BRADSHAW SEN WANTS ALTSCHULD EXPECTATIONS ASCRIBED COMPARATIVE VROOM INDIVIDUAL MOTIVATION EQUITY MINTZBERG

+ So… Where to Start n How do we DEFINE needs? n Needs as things n Needs as gaps n Needs as goals n Needs as human condition

+ So… Where to Start n How do we DEFINE needs? n Needs as things n n Needs as gaps n n “Our need is to decrease by 80% the number of new HIV/AIDS cases in our community” Needs as goals n n “We need new computers in order to succeed” “People need to feel safe” Needs as human condition n “People need water to stay alive”

+ How We Define Needs… n Clarifies the goals of our assessment n Influences how we design our assessment n Determines what we measure, and therefore how we measure n Influences what we report, to whom, and in what format

+ Examples n n If you define need as a gap in results n Who knows (or has data)? n How can you get this information (survey, interviews, document review, etc. )? n What questions do you ask? n How do you analyze the data? If you define need as an object n Who knows (or has data)? n How can you get this information (survey, interviews, document review, etc. )? n What questions do you ask? n How do you analyze the data?

+ How you define NEED also… Determines if/how… n ASSETS n CAPACITY n WANTS … fit into our assessments

+ Theories to Guide Practice 1. Know your definition of “need” 2. Reflect on how your definition influences 3. n Your Design n Your Methods n Your Results Discuss this openly with your client

+ Needs ASSESSMENT n What does it mean to ASSESS? n How does it differ from ANALYSIS? n Can we assess without analyzing? n Should we assess without analyzing?

+ 21 st Century Needs Assessment: Data Collection Methods A Panel Presentation James W. Altschuld Professor Emeritus, Ohio State University October, 2013 The American Evaluation Association Washington, DC

+ Concepts to be Presented Abstract of Presentation Briefly Methodology authors Search for a Holy Grail One example of classification Quick overview of a few new/emergent methods Caveats and concerns

+ Abstract In their 1995 book, Witkin and Altschuld noted 20 + methods for assessing needs with the methodology mantra continuing now (Altschuld & Kumar, 2010; Watkins, West-Meiers, & Visser, 2012). In 2010 A. observed that 5 procedures predominate. In this presentation a few emergent methods are highlighted - rapid participatory appraisal, photo voice, unique uses of the Delphi, proposed variations of surveys, and a scaling option. This section concludes with the concern that there is a need to sharply improve NA surveys and the analysis of double and triple scaled data. Altschuld is the author/editor of 7 books on NA (new one in process) and with others has conducted/published research on methodology.

Some Authors and Background Book Authors (methods) What do you have against the Grail Witkin, 1984 OK but it’s unattainable Mc. Killip, 1987 Like most classifications there are exceptions W & Altschuld, 1995 A & W, 1999 -2000 NA Kit Authors, 2009 -10 Watkins, West-Meiers, & Visser, 2012 All breakdown somewhat, artificiality Examples are Causal analysis Altschuld, 2013 Surveys with barriers (into implementation) Numerous others Some qualitative usages Classifying methods Value added but futile quest for the Holy Grail of methods Others Guidance is good but do not get into absolutes

Methods Overview: Type of method not where it fits in NA process - W & A (1995) and A & K (2010) Data Type Archival (early needs activities but could be later into process as well) Techniques Records, logs, data bases, epidemiology, rates under treatment (9 types identified) Communicative, non- Surveys, observations interactive (5 types identified) (early on but later into process as well)

Methods Overview Continued Communicative, interactive (early and later as well) Analytic (later especially when translating needs into solutions) Focus groups, community forums, interviews, etc (8 techniques) Fault tree analysis, risk assessment, trend analysis (10 + techniques)

New Techniques, Photo Voice (see Altschuld, 2013 for an overview) What it is -Needs/assets via community -Community members collect the data -Minimum of guidance, instruction needed -Photos of assets/needs -Group review/discussion -Photographer generated interpretations -Input to deliberations -Wonderful technique Image from U. of Oregon CSWS Grant Winner Uses Photovoice in the Appalachian Coalfields Some comments -Limited expense -Some facilitation needed -Unimpeded views of community -Sampling issues -Potential for bias -The interpretive session -How to run broader community session -Other Image from http: //www. wvphotovoice. org/

New Techniques, Rapid Participatory Appraisal What is it -Collecting data from a community -Community members do it via interviews -Community members key to interpreting data -5 community members x 5 interviews – rapid collection -Many variations -Used for assets and needs -Appears more in community based settings -Needs facilitation and training but not excessive The Interview Dance takes Place

Comments about the Method Has worked well Been used extensively for ID assets/needs in community settings Fairly inexpensive (non professional/their training) Voice of the people comes through loud and clear Interviewers look like the community - rapport Very rapid collection of data Possible sampling and bias Easy to collect more data if there are issues in what has been learned so far Generally other data (harder) should be collected Might be better if done by teams Bit more informal

Other Methods – Just a Potpourri Unique use of Delphi What the technique is Many up and down sides Not employed in cross – cultural settings Hung et al 2007 - 2008 did that across Asia (7 – 8 countries) Issue of common language Even with common language – differential concepts Done mostly electronically Cross cultural research team See papers by Hung, Altschuld, & Lee Epidemiology What the technique is More common now in mixed methods NAs Tie in with qualitative aspects of NA More prominent with large, accessible data bases Seen in NA and asset/capacity building studies re health Strong complementariness dimension See Altschuld (2013)

Pot. Pourri continues Variations on surveys Some ideas for consideration What the technique is Meaning of DK, NA, etc Need for better wording (see examples) Meaning of what should be (see discussion) Careful inclusion of solutions (see Hunt et al, 2002) Mini sets of questions for multiple purposes MES approach ‘Fuzzy’ scales (see example) Needs What they are Assets Idea of item samples Studies in their use (see Lee et al, 2012 and 2013 Selected use of open ended q’s Issues in using them Barrier questions but be careful How well have they worked Scaling options, equal appearing intervals with Likert Call for more sophisticated research Make better use of literature re content, types of scales, etc Other traditional scaling issues – middle points, analysis, unbalanced scales, etc Need research on utilization

Example of Student Survey OSEA (2005)

Differences in Wording : Student & F/A about Services (Subtle within-method variation) F/A Importance Satisfaction Frequency of use Extent to which the service is important to the academic success of students. Your Frequency of satisfaction with students’ use of the service for this service. students. Student Extent to which the Extent to which How service is important you are satisfied frequently do to your academic with the service. you use this success. service?

Examples of Likert and Fuzzy Scales

What Should be’s Discussion initiated by J. White in 2010 When something is of high importance on a Likert scale what does it mean Many times items are rated high on importance , limited variability in scores One solution is normalizing the scores but… Result is that disparity comes from only what is not the what should be scores Raises question of upon what basis do people rate both what should be and what is Implications described by White and Altschuld (2012) and effects on understanding needs

Implications of what should responses Four Knowledge Conditions affecting discrepancies None Inaccurate Incomplete Accurate /complete Other issues affecting discrepancies Multiple groups Uniformly high what should be scores (skewed distribution) Value of sections within surveys Solutions continued -Provide more info but -Normalizing distribution -Ask about basis for responses -Valuing sections in survey -Visualizing differences -Do pilot studies to see basis for responses -Do follow-up studies -Myriad of possibilities More formal research should be done More observations from the field

Yet another technique Mapping procedures Paper currently in progress Google maps to study areas of a developing campus Links maps to crime rates Frequent crime rates Times of crimes Ties student perceptions of situations to maps Older technique applied to placement of MH centers by need Suspect newer and more innovative applications will come to the fore

Quick Recap & Major Concerns/Caveats Glimpse of some methods that hold promise Not random or systematic sample You have seen others, enhance and expand understanding Literature is not strong in re to research Need to add to it, encourage you to do it Pleased to see new methods/ideas but the downside Concerns -Lack of Publication, few NAs are published or may be publishable -Publications are ‘how to’ (including many of mine) -Publications, presentations do not look at pros and cons of techniques and approaches -Choices underlying what is done are not explained in any detail -Not consulting the literature for methods & issues within them, perpetuating poor or flawed methods -Good quality techniques vs just doing assessments – must improve

One Set of Caveats Examples with regard to surveys Often poorly designed/affect possibility of publication Limited evidence that literature is reviewed for design Not understanding wording, analysis, and even what choices might mean Sample size, multiple scales, and loss of n’s Value of DK, NA, NO, etc Subsequent interpretations of meaning Problems like these greatly affect value/validity of our data May or will reduce the importance ascribed to NA results Let’s do a better job

+ 21 st Century Needs Assessment: Decision Making Methods Hsin-Ling (Sonya) Hung University of North Dakota October, 2013 The American Evaluation Association Washington, DC

+ Needs Assessment and Decision Making n Identification and resolution of gaps is what needs assessment (NA) is n Many NA activities involve decision making n Decisions (good or bad) have an impact on the process and outcomes of NA as well as the program/organization

+ Areas Involving Decision Making n Determining the stages of NA to be conducted n Selecting Methods for collecting information n Defining “What is” and “What should be” n Identifying Discrepancies n Prioritizing Needs n Allocating Resources n Developing Action Plans n Many Others (see Stevahn & King, 2010, on decisions in post-assessment phrase, p. 10 )

+ Types of Decisions n Authority Decision Without Input n Expert Decision n Average Opinion Decision n Authority Decision After Input n Minority Decision n Majority Decision n Consensus Decision Source: Johnson & Johnson (2009) cited in Stevahn & King (2010). Needs Assessment Phase III: Taking Action for Change, pp 89 -90. Sage Publishing.

+ Decision Making Methods n Many methods in NA decision making from books by panelists (Altschuld; Watkins et al), and others. Here is a sampling • • • n Multicriteria Analysis 2 X 2 Matrix Decision Aids Fishbone Diagrams Root Cause Analysis Fault Tree Analysis Multi-Attribute Consensus Reaching (MACR) Delphi Method Sork Procedure Goal Attainment Scores (GAS) Some methods are better than others in a given situation depending on purposes (for example, consensus building, prioritization)

+ Challenges in NA Decision Making n Determining Types of Needs (Altschuld & Kumar, 2010) • Present Vs. Future? • Severe Vs. Slight? • Maintenance or Upgrade? • Levels 1, 2, or 3? • Single Organization or Collaborative? • Needs or Asset/Capacity Building? • Retrospective? n Identifying Data Collection Methods • Data from single method or mixed methods? • Which methods or combinations deal with needs better? • Role of causal analysis/priority setting in setting directions for the future?

+ Challenges in NA Decision Making Process n n Analyzing Information from Multiple Sources • What might happen when results don’t agree? • Are some sources better than others? Decision Makers • Who are the decision makers as the assessment progresses? • What might be the role of Needs Assessment Committee (NAC) in decision making? • How the voice of the people is utilized in the process? • How to involve others or stakeholder groups? • How to avoid subjective judgment influence?

+ Literature on Needs Assessment Decision Making Methods n Limited literature that solely addresses NA decision making methods n Limited systematic review of methods for NA decision making n Needs decisions based on results or evidence collected from various techniques/methods n According to the above, it seems “evidence-based” decision making has been done in NA but literature doesn’t adequately reflect it

+ Issues/Concerns on NA Decision Making n The use of tools to aid decision making sometimes ignore complexity of social well-being of the service recipients and the external force of the organization. n Decision making process and influential factors are far from straightforward. n Wide ranging factors may influence decisions, some of which are extraneous to the needs e. g. political interests of the organization/agency that called for NA. n Invalid data problems in evidence-based decision. n Many NA data collection techniques require participants making up their own minds before providing information or response. Participants’ decision process has not been addressed.

+ Issues/Concerns n Literature may describe a data collection process, but fail to describe how the decision was made or how decision maker(s) process data or information to make decisions n Needs based decision sometimes ignore risk analysis n Literature is not strong in this area.

+ So…. What should we go from here? n n n Decision making is a process, how the decision was made is as important as the decision itself. Providing explicit descriptions of NA decision-making process should be critical for us. The trend of connecting NA with Asset/Capacity Building as presented by other panelists indicates decision-making should not be as limited as in traditional practice. Understanding how decision making method deal with emergent dimensions in NA could be valuable. Let’s do a better job in addressing the concerns, issues, or challenges observed.

+ 21 st Century Needs Assessment: Planning and Dissemination Maurya West Meiers World Bank October 2013 The American Evaluation Association Washington, DC

+ Planning and Dissemination n To read more about overall planning, see n Watkins, West Meiers and Visser (A Guide to Assessing Needs) – Includes checklists for different types of needs assessments n Altschuld (Planning and Conducting Needs Assessments: A Practical Guide; Needs Assessment Kit) n Dissemination of your needs assessment – focus of today’s session

+ Dissemination – Getting Started n n Purposes n Information sharing for those affected and others who need to know n Obtain “buy-in” for the assessment – and the results n Aids data collection n Clarifications for needs assessors and stakeholders Basics n Use a planning checklist (and update it regularly) n Dissemination planning begins at the start of the needs assessment n Key issues n Stakeholders –Who has standing? Who else might value the information? n Participatory approach n Some methods of dissemination

+ Stakeholders & Informants n Primary stakeholders. Have some direct relationship with the assessment (e. g. , managers and employees, mayor’s office, neighborhood representative, village chief, community members, etc. ). n Secondary stakeholders. Have a lesser relationship to the assessment, but should not be overlooked (e. g. , people living in the next community, etc). n Informants. May have useful data to inform the assessment, but may not have standing (e. g. , experts, etc. ). n Researchers, others. Those who could benefit from the assessment. Publish your needs assessment methods and papers to build the literature base! Share your assessment information on www. needsassessment. org.

+ Participatory Approach n Emphasizing Our voices Participation n Right stakeholders n Early engagements n Two-way communications and creating opportunities for feedback n Transparency on assessment methods, non-confidential data, findings, etc. n Regular feedback and engagements (including data collection) n Targeted methods of dissemination for different audience group (Who needs what info, when, how often, which methods, etc? )

+ n n Newer Methods of Dissemination Websites for the NA with info/docs about such things as n background and purpose (e. g. , concept paper, baseline info, GIS maps, etc. ) n people involved (e. g. , assessors, funders, stakeholder voices) n methods (e. g. , data collection instruments, sampling techniques) n progress, results and decisions (e. g. , updates mid-stream, public comment opportunities, final reports) Blogging n n n good opportunity to provide more informal updates, interactions, guest bloggers, sharing of pictures, etc. Social media (Twitter, Facebook, Instagram, Linkedin, Flickr, Youtube, etc. ) n build followings – and connect to others who will be interested n good for getting the word out, sharing photos… Mobile updates, short ‘pulse’ surveys, etc.

+ Older - but still useful - Methods of Dissemination n Community or organization meetings (status reports, etc. ) n Existing networks (e. g. , NGOs, community leaders, village heads) n Posters, announcements or events in spaces visited by stakeholders (e. g. , municipal buildings, schools, libraries, rural telecenters, etc. ) n Street billboards n Letters to stakeholders/groups

+ Older - but still useful - Methods of Dissemination (Continued) n Newsletters n Newspaper articles, advertisements, media outreach n Radio broadcasts, call-in shows n Television, videos n Maps

+ Dissemination Tips n Different reports for different audiences (funders, project staff, beneficiaries, etc. ) n Written reports - Full, summary reports, fact sheets, etc. n Oral reports - Videotaping and sharing, podcasts, etc. n Feedback opportunities for stakeholders n Languages of reporting n Special needs of audiences (sight, hearing, meals, childcare, schedules, safety, etc. ) n Political and social considerations