In 1994 I went with Michael Kremer to visit the family he had lived with for a year in rural Kenya.¹ We met up with many of Michael’s old friends, including Paul Lipeyah, who told us of the work he was doing with International Child Support (ICS) Africa, a nongovernmental organization (NGO) helping government schools in Busia, a neighboring district in Kenya’s Western Province. Paul asked us what advice we might offer for improving the effectiveness of ICS programs. Could Michael help evaluate what they were doing? Michael suggested randomized evaluation: if ICS wanted to understand the impact of their...

Causal impact is the difference in outcomes that is caused by the program. In other words, to estimate the impact of a program, we need to examine how the people who participated in the program fared compared to how they would have fared if they had not participated in the program. This hypothetical condition is called the counterfactual. However, we never observe the counterfactual directly. We observe only what happens with the program, not what would have happened in the absence of the program, and so we have to make an inference about the counterfactual. This is the fundamental problem...

In the previous chapter we saw that performing a randomized evaluation confers a number of advantages for generating the information that we need for evidence-based policy. But undertaking a good randomized evaluation is hard and can be expensive. We should undertake one only when the benefits from the lessons we learn are likely to outweigh the costs. We should not conduct a randomized evaluation every time we implement a program. Nor do we need a randomized evaluation every time we have a question about whether and how a program is working. Many important questions can be answered by other types...

In order to evaluate the impact of a program or policy, our randomly selected treatment group must have more exposure to the program than the comparison group. We can control three aspects of the program or policy to create this differential exposure:

1. Access: We can choose which people will be offered access to the program.

2. Timing of access: We can choose when to provide access to the program.

3. Encouragement: We can choose which people will be given encouragement to participate in the program.

Because we control these three aspects of the program, these are also the three...

Specifying good outcomes and the indicators we will use to measure them requires a deep understanding of the program being developed, the objectives of those implementing the program, and potential pathways through which the program or policy can impact lives, both positively and negatively. Some of the terms we use are defined with examples in Table 5.1. A theory-of-change framework is a useful tool to use in systematically thinking through potential pathways to an impact and is best developed in close cooperation between those implementing and those evaluating a program. This theory of change is likely to lead us to...

We have completed our evaluation of a program designed to improve test scores in a poor community in India. We find that the average score on the test we administer is 44 percent in the treatment group and 38 percent in the comparison group. Can we be reasonably confident that this difference of 6 percentage points is due to the program, or could it be due to chance? We perform a statistical test designed to check whether the difference we observe could be due to chance. We find that the difference between the treatment and comparison groups is not statistically...

Some of the people in the treatment group may never be treated. For example, some students assigned to a training program may never attend the training. Some parents whose children are assigned to receive deworming drugs may not give their consent. Or impassable roads in the rainy season may keep a program from delivering fertilizer to some farmers in time for planting. We can measure the extent of this partial compliance from process data collected throughout the implementation as we check records of how many people attend the training program or how many parents do not give permission for deworming...

Before we start any analysis, we need to prepare (or “clean”) our data and make sure we have a good understanding of their properties. This usually involves taking the following steps:¹

Correcting obvious errors in the data

Checking for outliers

Calculating attrition rates

Calculating compliance rates

Plotting and describing the data

It is good practice to look at our data and make sure that we have corrected obvious errors in the data (sometimes called “cleaning the data”). Errors can occur when a respondent misunderstands the question, when an enumerator fills in the questionnaire incorrectly, or when data are entered incorrectly....

One benefit of the randomized evaluation methodology is that there are some basic criteria by which to judge whether a study is valid. This (nonexhaustive) checklist provides a summary of some of the most common mistakes made and refers the reader to the relevant sections of the book that discuss each issue in greater depth. We group the mistakes into those that are made at the design, implementation, and analysis stages.

Often providing a program to one member of a community will have implications for other members of the community. If we provide information to a random sample of farmers,...