资源描述
,Click to edit Master title style,Click to edit Master text styles,Second level,Third level,Fourth level,Fifth level,University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation,University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation,Unit 5: Collecting data,Collect data,Evaluation questions,Indicators:,Evidence that,answers your questions,Sources of,information:,program records,individuals, public,METHODS,Who,What,Source of evaluation information,Existing information,People,Pictorial records and observations,Quantitative:,numbers breadth generalizability,Qualitative:,words depth specific,Remember,Not everything that counts can be counted.,Paradigm dimensions,Qualitative,Naturalistic inquiry,Holistic, system-wide perspective,Uniqueness and diversity,Inductive reasoning,Qualitative data (words),Qualitative methods unstructured, open-ended,Purposeful sampling,Emergent, flexible design,Content analysis,Extrapolations,Quantitative,Scientific/experimental design,Independent, dependent variables,Standardized, uniform,Deductive reasoning,Quantitative data (numbers),Quantitative methods structured, standardized,Probabilistic, random sampling,Fixed, controlled design,Statistical analysis,Generalizations,Quantitative methods Qualitative methods,Quantitative,Qualitative,Surveys,Questionnaires,Focus groups,Tests,Unstructured interviews,Existing databases,Unstructured observations,Often, it is better to use more than one method.,Mixed methods for one program,Log of activities and participation,Self-administered questionnaires completed after each workshop,In-depth interviews with key informants,Observation of workshops,Survey of participants,Are the data reliable and valid?,Validity: Are you measuring what you think you are measuring?,Example:,Reliability: if something was measured again using the same instrument, would it produce the same (or nearly the same) results?,Example:,“Trustworthy” and “credible” data,What do these words mean relative to your evaluation information?,How can you help ensure that your evaluation data are trustworthy and credible?,Common data collection methods,Survey,Case study,Interview,Observation,Group assessment,Expert or peer reviews,Portfolio reviews,Testimonials,Tests,Photographs, videotapes, slides,Diaries, journals, logs,Document review and analysis,When choosing methods, consider,The purpose of your evaluation, Will the method allow you to gather information that can be analyzed and presented in a way that will be credible and useful to you and others?,The respondents, What is the most appropriate method, considering how the respondents can best be reached, how they might best respond, literacy, cultural considerations, etc.?,Consider,Resources available. Time, money, and staff to design, implement, and analyze the information. What can you afford?,Type of information you need. Numbers, percents, comparisons, stories, examples, etc.,Interruptions to program or participants. Which method is likely to be least intrusive?,Advantages and disadvantages of each method.,The need for credible and authentic evidence.,The value of using multiple methods.,The importance of ensuring cultural appropriateness.,Quality criteria for methods,UTILITY,Will the data sources and collection methods serve the information needs of your primary users?,Quality criteria,FEASIBILITY,Are your sources and methods practical and efficient?,Do you have the capacity, time, and resources?,Are your methods non-intrusive and non-disruptive?,Quality criteria,PROPRIETY,Are your methods respectful, legal, ethical, and appropriate?,Does your approach protect and respect the welfare of all those involved or affected?,Quality criteria,ACCURACY,Are your methods technically adequate to:,answer your questions?,measure what you intend to measure?,reveal credible and trustworthy information?,convey important information?,There is no one right method of collecting data.,Each has a purpose, advantages, and challenges.,The goal is to obtain trustworthy, authentic, and credible evidence.,Often, a mix of methods is preferable.,Culturally appropriate evaluation methods,How appropriate is the method given the culture of the respondent/the setting?,Culture differences: nationality, ethnicity, religion, region, gender, age, abilities, class, economic status, language, sexual orientation, physical characteristics, organizational affiliation,Is a written questionnaire culturally appropriate?,Things to consider:,Literacy level,Tradition of reading, writing,Setting,Not best choice for people with oral tradition,Translation (more than just literal translation),How cultural traits affect response response sets,How to sequence the questions,Pretest questionnaire may be viewed as intrusive,Are interviews culturally appropriate?,Things to consider:,Preferred by people with an oral culture,Language level proficiency; verbal skill proficiency,Politeness responding to authority (thinking its unacceptable to say “no”), nodding, smiling, agreeing,Need to have someone present,Relationship/position of interviewer,May be seen as interrogation,Direct questioning may be seen as impolite, threatening, or confrontational,Are focus groups culturally appropriate?,Things to consider:,Issues of gender, age, class, clan differences,Issues of pride, privacy, self-sufficiency, and traditions,Relationship to facilitator as prerequisite to rapport,Same considerations as for interview,Is observation culturally appropriate?,Things to consider:,Discomfort, threat of being observed,Issue of being an “outsider”,Observer effect,Possibilities for,misinterpretations,Cultural issues related to use of existing data/records,Need careful translation of documents in another language,May have been written/compiled using unknown standards or levels of aggregation,May be difficult to get authorization to use,Difficult to correct document errors if low literacy level,Culturally appropriate informed consent,How can we be culturally sensitive and respectful and ensure the protection of those involved in our evaluations?,Children,Marginalized, “less powerful” participants,Focus groups,Structured small group interviews,“Focused” in two ways:,Persons being interviewed are similar in some way (e.g. limited resource families, family services professionals, or elected officials).,Information on a particular topic is guided by a set of focused questions.,Focus groups,Focus groups are used.,To solicit perceptions, views, and a range of opinions (not consensus),When you wish to probe an issue or theme in depth,Survey,A structured way to collect information using questionnaires. Surveys are typically conducted through the mail (electronic or surface), phone, or internet.,Survey,Surveys are used,To collect standardized information from large numbers of individuals,When face-to-face meetings are inadvisable,When privacy is important or independent opinions and responses are needed,Steps in planning a survey,Decide who should be involved in the process.,Define survey content.,Identify your respondents.,Decide on the survey method.,Develop the questionnaire.,Pilot test the questionnaire and other materials.,Think about analysis.,Communicate about your survey and its results.,Develop a budget, timeline, and management process.,Response rate,The proportion of people who respond: divide the number of returned surveys by the total number of surveys distributed.,Example: If you distribute 50 questionnaires and you get 25 questionnaires back, your response rate is 50%.,Response rate,# that answered,# you contacted,= response rate,Response rate,High response rate,promotes confidence in results.,Lower response rate,increases the likelihood of biased results.,Response rate,There is no standard response rate. “The higher, the better.” Anything under 60% is a warning.,Why is high return important? Its the only way to know if results are representative.,Address low response. How are people who didnt respond different from those who did? Only describe your results in terms of who did respond.,How to increase response rate,Generate positive publicity for your survey.,Over sample.,Ensure that respondents see the value of participating.,Use a combination of methods.,Make (multiple) follow-up contacts.,Provide incentives.,Provide 1,st,class postage/return postage.,Set return deadlines.,Make the survey easy to complete.,If response rate is low,Use language that is suggestive rather than decisive.,Examples: “The data suggests” vs. “These data show”; “It appears” vs. “We can conclude”,Dont generalize findings to the entire group.,Clearly describe who the data represents.,Document review,Using information that already exists in records, receipts, meeting minutes, reports, budgetsrather than collecting new data,There is a wealth of information available on the web.,CHECK What information is already available?,Document review Sources,National Center for Education Statistics,Census Bureau,Bureau of Labor Statistics,National Center for Health Statistics,Childrens Defense Fund,Wisconsin Department of Public Instruction,Local school districts,ERIC searches,County government,Document review Advantages of using existing data,Available dont have to collect data,Low cost,Minimum staff required,Comparative or longitudinal data may be available,Document review Issues in using existing data,Missing or incomplete data,Confidentiality issues,Unknown, different, or changing definitions of data make comparison difficult,May not match what you need in terms of geographic location, same time period, or population may be too aggregated,Observation,Is watching people, programs, events, communities, etc.,Involves all 5 senses: sight, hearing, smell, touch, and taste,observation includes more than just “seeing”,Observation is used,To provide information about real-life situations and circumstances,To assess what is happening,Because you cannot rely on participants willingness and ability to furnish information,When is observation useful?,When you want direct information,When you are trying to understand an ongoing behavior, process, unfolding situation, or event,When there is physical evidence, products, or outcomes that can be readily seen,When written or other data collection methods seem inappropriate,Observations,Advantages,Most direct measure of behavior,Provides direct information,Easy to complete, saves time,Can be used in natural or experimental settings,Disadvantages,May require training,Observers presence may create artificial situation,Potential for bias,Potential to overlook meaningful aspects,Potential for misinterpretation,Difficult to analyze,Observation Purpose, benefits,Unobtrusive,Can see things in their natural context,Can see things that may escape conscious awareness, things that are not seen by others,Can discover things no else has ever really paid attention to, things that are taken for granted,Can learn about things people may be unwilling to talk about,Inconspicuous least potential for generating observer effects,Least intrusive of all methods,Can be totally creative has flexibility to yield insight into new realities or new ways of looking at old realities,Observation Limitations,Potential for bias,Effect of culture on what you observe and interpret,Reliability,Ease of categorization,Usually you do not rely on observation alone; combine your observations with another method to provide a more thorough account of your program.,Observation Ethical issues,Unobtrusiveness is its greatest strength; also potential for abuse in invasion of privacy,Can venture into places and gather data almost anywhere,Covert overt,Always consider ethics and human subjects protection.,Types of observation,Structured Unstructured,Observing what does not happen may be as important as observing what does happen.,Looking for,Looking at,Steps in planning for observation,Determine who/what will be observed.,Determine aspects that will be observed (characteristics, attributes, behaviors, etc.).,Determine where and when observations will be made.,Develop the observation record sheet.,Pilot test the observation record sheet.,Train the observers and have them practice.,Collect the information.,Analyze and interpret the collected information.,Write up and use your findings.,Who/what to observe,People (individuals, groups, communities),Characteristics,Interactions,Behaviors,Reactions,Physical settings,Environmental features,Products/physical artifacts,Observation Example,If you want information about,You would record,Who uses a particular service,Total number of users broken down by gender, age, ethnicity, etc.,Interactions between youth and adults,# and types of questions asked by each,Neighborhood safety,?,What to observe Example,Exhibit on tobacco use at a county fair,Information needed ,Number of youth who visit the exhibit: age, gender, cultural background,Can the information be observed accurately? e.g., gender may be more obvious than age or cultural background.,Will the observer affect the situation?,Example Plans for observing participation in an after school program,Who:,youth attending the program,What:,approximate age,gender, cultural background,length of time student stays in the program,When:,all hours the program is open for one week each month during 2007,Recording your observations,Observations need to be recorded to be credible. You might use:,Observation guide,Recording sheet,Checklist,Field note,Picture,Combination of the above,Observational rating scales,Written descriptions written explanations of each gradation to observe,Photographs series of photos that demonstrate each of the grades on the rating scale,Drawings, sketches, etc. other visual representations of conditions to be observed,Who are the observers?,You program staff,Participants,Stakeholders,Colleagues,Volunteers,College students,Training observers,Training is often necessary:,To learn what to look for,To learn how to record observations,To practice,When want standardized observations across sites: important that all observers use same methods, rate same observation in same way,Practice,For our workshop today, what observational data could we collect that would tell us ,whether learning is occurring,the characteristics of attendees,whether the setting is conducive to learning,whether the materials are easy to use,Practice,Imagine you are sitting in a room where ten youth are participating in a computer demonstration. If you were looking for indicators of student interest and learning from the demonstration, what would you look for?,(Remember to include verbal and nonverbal indicators.),Interviewing is,Talking and listening to people,Verbally asking program participants the program evaluation questions and hearing the participants point of view in his or her own words. Interviews can be either structured or unstructured, in person or over the telephone.,Done face-to-face or over the phone,Individual; group,Interviews are useful,When the subject is sensitive,When people are likely to be inhibited in speaking about the topic in front of others,When people have a low reading ability,When bringing a group of people together is difficult (e.g., in rural areas),Interviews,Verbally asking program participants the program evaluation questions and hearing the participants point of view in his or her own words.,Interviews can be either structured or unstructured, in person or over the telephone.,Interviews,Advantages,deep and free response,flexible, adaptable,glimpse into respondents tone, gestures,ability to probe, follow-up,Disadvantages,costly in time and personnel,requires skill,may be difficult to summarize responses,possible biases: interviewer, respondent, situation,Types of interviewing,Structured Conversational,Type: Structured interview,Uses script and questionnaire,No flexibility in wording or order of questions,Closed response option,Open response option,Type: Guided interview,Outline of topics or issues to cover,May vary wording or order of questions,Fairly conversational and informal,Type: Conversational interview,May not know that an interview is taking place,Spontaneous,Questions emerge from the situation and what is said,Topics or questions are not predetermined,Individualized and relevant to situation,Probing,Interview question:“What did you like best about this program?”,Response: “I liked everything.”,Probe 1: “What one thing stood out?”,R: “Being with my friends.”,Probe 2: “What about the program activities?”,R: “I liked it when we worked as a team.”,Probe 3: “How come?”,R: “It was neat to hear each others perspectives. I heard some things I hadnt considered before.”,Probe 4: “What is one thing that you learned?”,Interviewing tips,Keep language pitched to that of respondent,Avoid long questions,Create comfort,Establish time frame for interview,Avoid leading questions,Sequence topics,Be respectful,Listen carefully,Recording responses,Write down response,Tape record,Key in on computer,Work in pairs,Complete notes after interview,Questionnaires are,Data collection instruments used to collect standardized information that can be expressed numerically or through short answers,Basic instruments of surveys and structured interviews,Appropriate when,you want information from many people,you have some understanding of the situation and can ask meaningful questions,information is sensitive or private anonymous questionnaires may reduce bias,Questionnaires,Advantages,can reach large numbers,provide for anonymity,relatively inexpensive,easy to analyze,Disadvantages,might not get careful feedback,wording can bias clients response,response rate is often low,literacy demands,When should a questionnaire be used?,Respondents can provide useful information about the topic.,You know what it is you want to know and are reasonably sure that you can ask standardized questions to get the information.,Respondents can be relied upon to provide the information you need (perhaps with incentives). This means they can comprehend the questions and respond properly, they are truthful, and they are motivated enough to respond carefully.,Good questionnaires are NOT EASY!,Developing a good questionnaire, takes time, time, and more time.,Multiple (even a dozen!) drafts may be involved before the questionnaire is ready.,Its important to involve others in writing the questionnaire.,Questionnaire design Considerations,Kind of information: What do you want to know? Is the information already available?,Wording of questions and responses,Formatting the questionnaire,Pre-testing,Cover letters and introductions,When/where will the questionnaire be distributed?,How will returns be managed? How will the data be analyzed?,Who is responsible for each task?,Questionnaire design,Is the information already available?,Dont ask a question unless it has a use.,Eliminate the “nice to know.”,What will you do with each piece of information gathered?,Questionnaire design,Write questions through your respondents eyes.,Will the question be seen as reasonable?,Will it infringe on the respondents privacy?,Will the respondent be able and willing to answer the question?,Be,selective,and,realistic,when writing questions.,6 Steps in Developing Effective Questionnaires,Decide what information you need.,Determine sample respondents.,Develop accurate, user-friendly questionnaire.,Develop plan for distribution, return,
展开阅读全文