Liz Bageant is a Research Support Specialist in Cornell’s Dyson School.
Economists tend to undervalue qualitative data, yet in many cases, our quantitative data fall short. In such cases, carefully collected qualitative data can be used at any or all points in the research process to strengthen validity and inference. Qualitative data can clarify research questions, provide crucial contextual information for survey instrument design, provide context-appropriate support for tough model specification decisions, shed light on unobserved heterogeneity and improve interpretation by explaining inconsistencies in or corroborating econometric results.
As part of this paper on gender and demand for index based livestock insurance (IBLI) in southern Ethiopia, I designed and conducted a qualitative study to complement the survey data and econometric methods that are standard in our field. The most important contributions of qualitative data in my work were to validate the survey data and probe the structure of measurement error, which improved my research dramatically and provided valuable information for future data users.
From this experience I learned that we often cloak the details of noisy household survey data in convenient assumptions that may, in fact, not hold. Below are two examples from my fieldwork of how qualitative data can help us explore those assumptions.
Example 1: Our respondents interpreted the term insuraansi horrii (livestock insurance) far more broadly than we expected. As the implementers of the IBLI livestock insurance scheme, the only one in the area, we were surprised to find that the term insuraansi horrii meant any broader activity associated with us, including implementation of the survey and visits to check on GPS collared animals. When we asked whether they had a positive experience with the insurance sales agent, or “the person selling insuraansi horrii,” a respondent might enthusiastically say, “Yes, I loved the insuraansi horrii person! He was perfect!” When we probed the respondent to explain this statement rather than taking it at face value, it turned out she was referring to the person who administered the household survey, or perhaps the person who came to her house to tell her the survey was coming again, or even the person who distributed the discount coupon, but not necessarily the insurance sales agent. The level of specificity we assume when we write survey questions may not be attained in practice.
Example 2: In quantitative interviews, we took time to explore in depth the process involved in deciding to purchase or not purchase IBLI was made in the household, rather than simply asking whether or not they purchased. Through this, we discovered that many households that reported in the survey that they purchased IBLI had not in fact purchased, or they purchased in the distant past and had misunderstood time period of reference in the survey question. And some households that had purchased did not report purchasing. Given the importance of this variable to any work done on impacts of index insurance, this was an important finding. Though we only interviewed a fraction of households in the sample, this discovery prompted us to validate all survey responses using administrative records from the insurance company and, in turn, provide two IBLI purchase variables to data users—one based on perceived (reported) IBLI coverage, and one on actual IBLI coverage as recorded by the insurance company. We were then able to choose the appropriate insurance purchase variable with respect to our research question. Deeper exploration through qualitative research can, despite small sample sizes, help us identify the patterns and potential biases associated with such misunderstandings and adjust our quantitative approaches accordingly.
Both of these examples represent issues that could theoretically be taken care of with extra care in the survey implementation. However, anyone who has administered a household survey in a difficult environment can attest that even the most rigorous enumerator training (in this case, training that lasted longer than actual data collection) and advanced data collection methods (such as real time data entry with built in consistency checks) cannot prevent misunderstandings in the field. Once data is collected, supplementing it with qualitative data as a form of validation and exploration of underlying noise can add significant value to the analysis and interpretation of survey data, errors and all.
