Evaluation of Health Programs A Postgraduate Overview Course

Evaluation of Health Programs A Postgraduate Overview Course

Evaluation of Health Programs A Postgraduate Overview Course MODULE 6: Evaluating Results Developed by the GEMNet-Health Task Group on Curriculum for Postgraduate Evaluation Courses Module 6 Learning objectives and sessions At the end of the module, students will be

able to: 1. Identify data sources to analyze program performance and track program outcomes 2. Illustrate the use of qualitative and quantitative data to assess program performance 3. Track program results and compare actual performance versus goals Sessions: Session 1: Developing Performance Monitoring Plans/M&E Plans (3 hours) Session 2a: Tracking Results/Changes (3 hours) Session 2b: Appraising Results (4 hours) Module 6, Session 2A

Tracking results/changes Objective and Outline Session 2A: Tracking results/changes Objective Understand the process of implementing an M&E plan Identify some challenges with implementing an M&E plan Outline Definition: What is meant by tracking program results Importance of tracking program results

Key steps in tracking program results Selecting and evaluating key performance indicators to monitor outcomes Developing a data collection plan Data analysis Dissemination plan Definition: Tracking program results M&E plan is a document that helps to track and assess the results of the interventions throughout the life of a

program/policy/intervention Tracking is a continual process (measuring, learning, improving) which involves data collection and measurement of program indicators to evaluate project/program target In monitoring, tracking results will mean: To track and assess performance through analysis and comparison of indicators over time In evaluation, tracking results will mean: To evaluate achievements or outcomes by comparing

indicators before and after an intervention Importance of Tracking Program results Evaluate program performance and success Identify specific areas for improvement or expansion Monitor trends against overall target Make well informed decisions about future policies, goals, and actions Key Steps for Tracking Program results 1. Identify key performance indicators

2. Setting the baseline year 3. Clearly define targets 4. Develop a plan to track results 5. Collect comprehensive data on the key indicators 6. Analyze data and evaluate short, medium, and long term impacts 7. Plan for dissemination Step 1: Identifying Key Performance indicators I An indicator is a variable that measures an aspect of a program or health outcome Characteristically, they can be measured as counts, percentages, ratios, rates, index,

thresholds, etc. Types of indicators Process indicators Outcome indicators Step 1: Identifying Key Performance indicators II Important steps Step 1A: Characteristics of good performance indicators include being: Measurable Accessible Relevant, in reference to the project

objectives Step 1B: Overview of the logic model (input, output and outcomes: Short term, medium, and long term) Note: The indicator should be able to provide enough relevant information to evaluate/ Characteristics of Good Indicators 1. Valid: accurate measure of a behavior, practice, or task 2. Reliable: consistently measurable in the same way by different observers 3. Precise: operationally defined in clear terms

4. Measurable: quantifiable using available tools and methods 5. Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities 6. Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact Validity: Class Activity 1 Class discussion: 1. Is parasitemia a valid measure of morbidity? 2. Is fever a valid measure for malaria? 3. Is parasite testing a valid measure for

parasite prevalence? 4. Is the number of people reached by behavior change communication (BCC) campaigns a valid measure of malaria knowledge? Measurable: Class Activity 2 Class discussion: How would you measure this (what method, source, tool, etc.)? 1. Number of ITNs distributed 2. Compliance to antimalarial treatment 3. Anemia 4. Parasitemia

Programmatically Important: Class activity 3 Class discussion: Are the following indicators programmatically important (i.e., linked to a public health impact or to achieving the objectives needed for impact)? Example 1: Insecticide-treated mosquito nets (ITN) distribution program Indicator: # of ITNs distributed in the past quarter Example 2: Program to increase access to ACTs through community-based health

Operationalizing Indicators Establish exactly how a given concept/ behavior will be measured Precise definition and metric How the value will be reliably calculated Anyone using the same data will arrive at exactly the same indicator value Challenges

Subjective judgment Local conditions Unclear yardsticks Skills of the users How Indicators Are Linked to Frameworks How Indicators Are Linked to Frameworks Results frameworks use indicators Step 2: Setting the Baseline Year

Choose a particular year that would be logical for the baseline year Remember: You will compare the performance of the project to the baseline year What do we need before we choose a particular year as the baseline line year? Key: Are data measuring key performance indicators available for that year? Step 3: Defining Target

Program target can be three-fold: Short-term Mid-term Long-term 1. Lets brainstorm about what goes into short-, medium-, and long-term targets 2. Lets give examples of outcome indicators and targets for each indicator Data Collection, Analysis, and interpretation Step 4: Developing a Plan Based

on the project goals develop a consistent, efficient, and for data collection reliable process for data collection and management Identify persons responsible for collecting data for each performance indicator, managing the dataset, creating and submitting reports Identify protocols, templates, sources, units to be used in collecting data Provide timelines for data collection, analysis, and reporting Brainstorm on how to ensure data integrity and security Allocate sufficient resources (money, time, staff, etc.) for each activity associated with tracking and reporting Minimize the burden of data requests by integrating with existing

processes and procedures Move to baseline data collection Start by collecting data for the baseline year Commence regular data collection based on the tracking plan Step 5: Collect Comprehensive Data on key indicators A. Sources of data for indicators B. Diagram of data collection, processing, analysis, and reporting system C. Data collection tools Patient records or registers Survey instruments Commodity management forms (e.g.,

condoms) Others? D. Management Roles and responsibilities of each group/member of the system Data Quality Constraints Describe known constraints to data quality and/or system performance and what will be done to address these Name of Data quality indicato issues r

Actions taken or planned to address this limitation (list by List possible risks How will the indicator) to the quality of identified possible data collected. risks to the quality Consider the five of data be criteria for data managed? quality: validity,

reliability, integrity, precision, and timeliness. Additional comments Step 6: Data Analysis and evaluation Evaluation is the process of analyzing changes in your indicators to determine which elements of your program are effective and which aspect of the project needs to be modified

to achieve project objectives Process evaluation: assess if the program is being implemented as planned or according to protocol Impact evaluation: assess whether or not the program is having the desired impact or effect and that the observed change can be attributed to the implementation of the Step 6: Data Analysis and evaluation Relevant questions: I. How much progress has been made between

the baseline data and the postimplementation data? II. Are the results on track for achieving interim and long-term targets? III. What other factors or programs could have influenced the change between the baseline and post-implementation year? IV. Do the data support our narrative? V. Document evaluation process and assumptions Methods for Data Collection and analysis Quantitative design: Structured questionnaire

to measure performance indicators, review of records (secondary data) Qualitative design: Focus group discussion, in-depth interview Analysis: Frequency, percent frequency, rate, counts, t-test for comparing difference in means, Z-test for comparing difference in proportion, and quasi-experimental design procedures Confounders A confounder is a variable that has an effect on a program, outcome variable of interest, or both Essential to control for confounders to

be able to quantify the effect of the program on the outcome of interest Ways of controlling for confounders: 1. Stratified analysisMantel Haenzel procedure 2. The use of regression models Selection Bias Selection bias arises when subjects are selected to receive the program based on characteristics that may also affect their outcomes. Nonparticipants are often a poor comparison

group for participants Participants are not a random sample of the target population Participants selected through program placement rules and self-selection Participants may differ systematically from nonparticipants Any observed difference in outcomes between program and comparison groups can be attributed to both the program impact and pre-existing differences between the groups.

Randomization Solves Selection Bias Selection bias occurs because of systematic differences between participants and non-participants: If program and comparison groups are drawn at random from the same underlying population, then the characteristics of the two groups will be identical on average The program and comparison groups are identical except that one got the intervention and the other did not The outcomes of the two groups would have been the same if neither received the interventionimplies

selection bias = 0 Any differences observed in outcomes between the two groups must be attributable to the intervention Power of a Statistical Test The statistical power is the ability of a statistical test to detect a difference of a specified magnitude given that this difference exists in the populations being compared Unlike , power is not the risk of a particular error; instead, it is the probability that a statistical

test will reach a particular correct conclusion Factors That Influence the Power of a Study Sample size: Increasing sample size increases power The variability of the observation: Power increases as variability of the observation decreases The effect size: Power increases as the effect size of interest increases Significance level: The power is greater if the significance level is larger. That is, type I error increases as the type II error

(decreases Interpretation and Discussion of Results Treatment on the treated estimate (TOT) Remember: The ITT and TOT estimates will be the same when there is full compliance; that is, when all units to whom a program has been offered actually decide to enroll in it Discuss results based on program objectives

Qualitative Designs for data collection and analysis Study Designs Phenomenology Purpose: to describe experiences as they are lived Grounded theory Purpose: theory development Ethnography Purpose: to describe a culture's characteristics and its impact on the

behaviour/outcome we are interested in Study Designs Narratives Purpose: describe and examine events of the past to understand the present and anticipate potential future effects Case study Purpose: describe, in-depth, the experience of one person, family, group, community, or institution Sampling Qualitative sampling is generally purposive

Respondents, settings, or events with particular characteristics that relate to the research question are selected to take part Theoretical sampling is a specific, theory-driven form of purposive sampling A small number of respondents selected and interviewed, a hypothesis is developed, and further respondents are selected who might challenge the Sampling Stratified purposive sampling Snowballing

Maximum variation Key informants/experts Consecutive Convenience Data Collection Approaches Interviews (in-depth/ semistructured) Focus groups Participant observation Observation Documentary analysis

Data Collection Tools Field notes Interviewers notes Interview guide Audio tape/digital voice recordings, transcripts Topic guide Document and artifacts Visual media Steps in the Data Management Process Make copies of all materials Put field notes in some order Management system for all interviews,

surveys, and questionnaires Create a catalog of all materials relevant to the research Check for missing data Data Collection and Analysis Data analysis informs further data collection Data collecte d Data analyse

d Data processe d Analysis Plan for Qualitative Data Identifying patterns, underlying themes, commonalities, and differences Thematic analysis Content analysis

Qualitative data analysis can be done manually or with software, such as: AtlasTi MAXQDA QSR NVivo EZ-TEXT 3.06C Interpreting Data Ethnography (Anthropology)

Immersion in a context; holistic; reflexive Grounded theory (Glaser and Strauss) Constant comparison Cyclical process of research Theory-driven Induction Framework method (Ritchie and

Spencer) Coding into a thematic framework Applied research Validity in Qualitative Research Internal validity: credibility External validity: transferability Credibility How believable are the research

findings? How to improve credibility: Good design appropriate to the question Triangulation Robust analysis (e.g., double coding) Checking with the participants Attention to negative cases Good description of methods in write-up Transferability Would your research findings apply in other settings? Transferability might not mean better research

Quantitative Designs for data collection and analysis Evaluation Designs We will identify various evaluation designs: 1. Experimental 2. Quasi-experimental 3. Non-experimental Types of Evaluation Designs

Design Experimental Strongest for demonstrating causality, most expensive Quasiexperimental Weaker for demonstrating causality, less expensive Nonexperimental Weakest for demonstrating causality,

least expensive Measuring the effect of an intervention Step 7: Plan for Dissemination and use A. Clearly defined users B. Databases for information storage C. Dissemination methods Reports (schedule and audience) Media Speaking events Others?

Reporting and Modifying project content Use the findings from your evaluation to inform stakeholders and improve program performance Know your audiences, their informational needs, and the best ways and times to reach them Identify program strengths, weaknesses, and opportunities Modify program activities and continue to track results to see the impact of the adjustments Consider how program results can be used to inform budget and strategic planning processes Continue the process of collecting and evaluating data, reporting progress, and making program

refinements for (at least) the duration of your program lass Activity 4: Data Collection Plan Using the case study, develop a data collection matrix to evaluation control and management of JE outbreak. Consider the following issues: Who will be responsible for data collection and its supervision? Who will be responsible for ensuring data quality at each stage? How will data quality be checked at every stage? How often will the data be collected, compiled, sent, and analyzed? What indicators will be derived from each data source?

How will the data be sent (raw; summary)? What tools/forms will be used, if any? What resources (staff, office supplies, computers, transportation) will be needed at each stage? Who will analyze the data? How often will analysis occur? How often will the results be compiled into reports? Implementi ng an M&E plan Implementing an M&E Plan A. Assessment of capacity to implement

plan B. A detailed work plan of each M&E activity Timing of each activity Party responsible for each activity Budget necessary for each activity C. Mechanism for reviewing and updating plan D. Costs Implementing an M&E Plan: Costs How much will it cost? Need to

budget: Costs of information systems (costs of data collection, processing, and analyzing) Costs of information dissemination and use Costs of the data quality control system Costs of coordination and capacity building Sample Costs for Selected Data Sources Information system National sentinel system Total annual costs (2001

$USD) $513,682 Frequency of data collection Continuous HMIS $2,119,941 Continuous Integrated disease surveillance

$4,270,943 Continuous Vital registration $719,427 Continuous DHS $854,164

Periodic (~4 years) National census $8,244,114 Periodic (~10 years) Source: Rommelmann, et. al. (2003). Costs and Results of Information Systems for Poverty Monitoring, Health Sector Reform, and Local Government Reform In Tanzania. Implementing the M&E Plan: The role of the M&E unit

Consensus buildingamong all stakeholders Coordination Data management, data analysis, and interpretation Reporting Information dissemination Training and capacity building Challenges with Tracking program results Lack of baseline data to be compared to mid- and long-term data collection for program evaluation The use of indicators that are not easily

measurable Poor data quality, especially on the over-reliance of secondary data Needs a lot of time and commitment Differences in indicators used at baseline and end-line Missing information on key performance Challenges in Implementing an M&E plan Need for plan not well perceived by stakeholders Costly Lack of expertise

Time consuming Objective and Outline Session 2A: Tracking results/changes Objective Understand the process of implementing an M&E plan Identify some challenges with implementing an M&E plan Outline

Definition: What is meant by tracking program results Importance of tracking program results Key steps in tracking program results Selecting and evaluating key performance indicators to monitor outcomes Developing a data collection plan Data analysis Dissemination plan

Cost of implementing M&E plan References 1. Developing Process Evaluation Questions. Evaluation briefs No. 4, CDC. February 2009, www.cdc.gov/healthyyouth/evaluation/pdf/brief4.pdf 2. http://www.fhi360.org/sites/default/files/media/documents/Monitoring%20HIV-AIDS%20 Programs%20%28Facilitator%29%20-%20Module%203.pdf 3. https://www.oecd.org/dac/peer-reviews/World%20bank%202004%2010_Steps_to_a_Re sults_Based_ME_System.pdf 4. Impact Evaluation in Practice, The World Bank. 2011, http://www.worldbank.org/pdt 5. Introduction to Impact Evaluation. Patricia Rogers. RMIT University. InterAction, https://www.interaction.org/impact-evaluation-notes 6. Kusek, J.Z., & Rist, R.C. (2004). Ten-steps to a results-based monitoring and evaluation system. A handbook for development practitioners. Washington, DC, USA: The World Bank.

7. World Health Organization. (2009). Monitoring and Evaluation of Health Systems Strengthening. An operational framework. Geneva, Switzerland: WHO. 8. Outline of Principles of Impact Evaluation. Part 1 Key Concepts. 9. Quantitative and Qualitative Evaluation methods. Center for Civic Partnerships. 2014, http://www.civicpartnerships.org/#!quantitative--qualitative-eval-methods/c1bel 10. Workshop materials from-GEMNet-Health Regional Workshop on Impact Evaluation of Population, Health and Nutrition Programs, https://www.measureevaluation.org/ resources/training/capacity-building-resources/workshop-on-impact-evaluation-of-popu lation-health-and-nutrition-programs/workshop-on-impact-evaluation-of-population-hea lth-and-nutrition-programs-landing-page This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the Carolina Population Center,

University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www.measureevaluation.org

Recently Viewed Presentations