The evaluation design for this activity changed over the course of the evaluation due to problems faced during implementation. In its original conception, the independent evaluator, NORC, and MCA-Honduras planned to use a randomized experimental design involving randomized assignment of communities (aldeas) to treatment. Following a series of implementation problems, the final approach used was an econometric model that relied on a model-based approach to impact evaluation. As raised in the evaluator report, one key assumption is that the causal models are correct. This assumes that all important unobserved variables affecting selection, such as proven ex-ante ability to grow horticultural crops, are time invariant (i.e., are constant between the two survey rounds). Assumptions of the econometric model are:
1. The stable unit treatment value assumption (SUTVA, no macro effects assumption, partial equilibrium assumption) is made. This means that the effect (potential outcomes) on one individual are not affected by potential changes in the treatment exposure of other individuals. This implies, for example, that the program is not so large that the outcomes are correlated (e.g., that farmers would produce such a large number of horticultural crops that the market would collapse).
2. The causal models are correct. The key assumption here is that all important unobserved variables affecting selection are time invariant (i.e., are constant between the two survey rounds).
3. The program intervention represents a “forced change” in (experimental control of) the agricultural system in Honduras.
4. The half of the country treated before this evaluation began is similar to the half yet to be treated, with respect to relationships among the important causal variables represented in the causal model underlying the statistical analysis.
Kind of Data
Sample survey data [ssd]
Unit of Analysis
The scope of the Farmer Training and Development study was:
• Household composition
• Individual characteristics
• Household expenditure
• Food expenses
• Non-food expenditure
• Agricultural production activities
• Access to marketing
• Loans and credits
• Training and technical assistance
• Animal husbandry
Migration & Remittances
Agriculture & Rural Development
Food (production, crisis)
Access to Finance
All villages in Honduras except Gracias a Dios Department, national parks and tourist areas (Islas de la Bahia Department), as well as any village where the study was introduced.
Producers and sponsors
National Opinion Research Center (NORC)
University of Chicago
Millennium Challenge Corporation
A two-stage survey design was used, in which a first-stage sample of 203 aldeas (villages) was selected, and a second-stage sample of households was selected from each sample aldea. The total number of aldeas in the sample frame (from the GIS, also from Census) was 3,675. After deleting aldeas in Islas de la Bahia and Gracias a Dios department, those having 100% of caserios in protected status, and those already processed by Fintrac, the sample frame was reduced to 1,822 aldeas. These are the primary sampling units for the survey. The sample sizes that were decided on were 113 treatment aldeas and 90 control aldeas, with an expected sample size of 9 program farmers and 20 other households in treatment aldeas, and 9 potential treatment farmers and 20 other farmers in control aldeas, for a total sample size of 203 aldeas and (expected) 203 x 29 = 5887 households in each survey round. This sample is constructed by selecting a sample of 113 matched pairs (226 units in all), randomly dividing them into treatment and comparison aldeas, and dropping 23 of the comparison aldeas (resulting in the desired sample size of 113 treatment and 90 comparison aldeas).
Deviations from the Sample Design
It was not possible to implement the original design, for a number of reasons (documented in the Final Report). The final sample (for the first survey round) consisted of the responding part of the original (experimental) design (3,981 households) and an additional sample of 545 clients (households) from the program implementer's client list (who entered the program at the same time as the program farmers of the original design). The final responding sample size for the first round survey was 4,526 households (farmers). The design was a panel design in which it was attempted to re-interview, in the second survey round, every household that had been interviewed in the first survey round. The number of households interviewed in the second round was 2,736, for a total of 7,262 household interviews in both survey rounds.
The sample sizes for the two survey rounds were 4,533 in the first round, with 4,533 responding (100 percent), and 3,063 in the second round, with 2,736 responding (89 percent).
The probability of selection for each sample aldea is included in the file RecodedExtract.xls. The "base" survey weights are equal to 1/prob.
Dates of Data Collection
Data Collection Mode
Once the “raw” survey data were available from INE, they were prepared for analysis by the ESA Consulters, the Honduras subcontractor. This cleaning and aggregation process are documented in detail in a series of Stata command (.do) files, Do*FTDAImpact.do (where “*” represents digits 1-11).
Estimates of Sampling Error
Standard errors are reported for all impact estimates presented in the final report. Standard errors were estimated using the "bootstrap" (resampling) procedure.
For each round of data collection INE trained a team of 15 to 20 data entry clerks and two supervisors. INE would conduct 5 day-training of data entry staff prior to the start of data entry. Staff were expected to complete the data entry of 20 surveys per day during an 8 hour workday for the first week and then increase to as many as 25 per day as they became more familiar with the instrument. They performed data entry using an in-house program, which was developed and tested by INE programmers and approved by MCA and NORC prior to the start of data collection. INE protocols require 100% double data entry. To ensure quality and detect any data entry errors, we required that each questionnaire be data entered twice, using different clerks for each of the two entries. Then, supervisors performed a reconciliation of all data entries to identify and correct any errors that were identified. The data entry program was designed to conduct consistency checks and perform a series of validation measures automatically. The next step in processing was to conduct a number of additional consistency and error checks. INE then generated frequencies and crosstabs in SPSS for validation. The data were delivered to the client within 6 - 8 weeks of the end of data collection in the field.
Caldwell, Joseph, Fidel Ordoñez, Michael Reynolds, Varuni Dayaratna, John Felkner, Impact Evaluation of the Millennium Challenge Corporation Farmer Training and Development Activity in Honduras: Merged Baseline and Endline Household Survey Data, NORC at the University of Chicago, November 15, 2013.
Disclaimer and copyrights
The user of the data acknowledges that the original collector of the data, the authorized distributor of the data, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses