Reflection and Validity in Qualitative Research

Deborah Court, Bar-Ilan University, Israel

Deborah Court is a lecturer in the School of Education at Bar-Ilan University. She conducts ethnoghraphic research into educational cutlures in Israel and North America.

Abstract

Examples from two ethnographic studies illustrate how reflection is essential in order to perceive the truth of others. The first example describes how cultural misunderstandings between Moroccans and Canadians living in a small Israeli village engendered arguments, and how the author strived to gain greater objectivity and to examine the meanings she was imposing on events. The second example describes an interview study with Israeli Arab and Jewish educators, and the author’s attempts to externalize and examine her own feelings, values and opinions in order to achieve greater objectivity and stronger internal validity.

Introduction

As a Canadian living in Israel I have become more and more aware of how culture, values, insider and outside knowledge and personal experience influence the ways qualitative researchers analyze and present their data. The journey from Canada, where I shared the mainstream experiences and values (and thus seldom reflected on them) to Israel, where I am an outsider in so many ways, has immeasurably sharpened my sensitivity to the workings of culture. As Fetterman (1998) writes, the concept of culture “becomes immediately meaningful after cross-cultural experience… Attitudes or habits that natives espouse virtually without thinking are distinct and clear to the stranger” (p.17). This is certainly true, but in order to ascribe proper meanings to the attitudes or habits of those in a “strange” culture, the researcher must externalize and evaluate her own attitudes and habits. My values and experiences are those of a Canadian, an academic, a woman, a grandmother, a Jew. As I have sought to understand the experiences of Jews, Arabs and Christians in Israel I have discovered as never before how one’s native culture works as a set of tinted glasses through which the world is perceived. Through vignettes from two ethnographic studies (Court, 2002; Court, 2004) I will illustrate how reflection is essential in order to see beyond personal experience to perceive the truth of others.

The goal of many qualitative studies is to arrive at a ‘true’ description and interpretation of the lives of the people studied. This means identifying norms and values that underlie participants’ actions in a particular setting and reaching understanding of the meanings they ascribe to their actions, rather than imposing external meanings. A study’s internal validity is dependent on the extent to which the researcher succeeds at these tasks. Together with rigorous application of traditional methods of reaching internal validity, the qualitative researcher is called on to do continuous self-checking about her interpretations. In the end, the way the researcher interprets and presents her carefully collected data is dependent also on who she is, what she values and the extent to which she is willing to externalize and critique her intuitive understandings. Denzin (1997) writes that the most important criterion of internal validity is verisimilitude, a text’s “ability to reproduce and map the real” (p.10). A research report with high verisimilitude provides the reader, through vivid description and extensive narrative quotations, with the “opportunity for vicarious experience” (Stake, 1995, p.86) and allows the reader to make naturalistic (as opposed to propositional) generalizations. Naturalistic generalizations are “conclusions arrived at through personal engagement in life’s affairs or by vicarious experience so well constructed that the person feels as if it happened to themselves” (Stake, 1995, p.85).

Traditional methods of reaching internal validity include triangulation of research tools (seeking information through interviews, observations and document analysis) and informants (interviewing people with different roles, viewpoints and backgrounds), spending long periods in the field and inviting reliability checks from other researchers. These methods are essential, but they are not enough. In fact, since the researcher herself is the main data collection instrument (Merriam, 1991, p. 19) the single most important element in the quest for internal validity is continuous self-checking by the researcher of her interpretations. The way the researcher interprets her data is at least partly dependent on who she is, what she values, and the kinds of knowledge she holds as an insider, an outsider or some combination of these (Banks, 1998) in relation to the culture she is studying. She must be willing to externalize and critique her intuitive understandings and to write down and study these reflections throughout the course of a research project. “All we want are the facts, Ma’am” is not a dictum that applies to qualitative research. We do want the facts, but having them does not guarantee that the researcher will arrive at the right conclusions. Adding reflection to the internal validity toolbox helps her to make the necessary interpretative leap, beyond facts to the participants’ (rather than her own) meanings, and thus to reach new levels of insight into the lives and cultures of others.

First vignette: The meanings of the dog

My husband and I moved to a small village in Northern Israel because the land was cheap and the air clean. We were one of five newcomer Canadian families among the thirty or so veteran Israeli families, most of whom were of Moroccan origin. Their parents had come to Israel around the time of the beginning of the state. As observant Jews we all shared basic religious values, but it quickly became apparent, through heated clashes that sprang from the minutia of everyday life, that something was amiss.

I began an ethnographic study of our community after one of the Canadian women, my best friend, died suddenly, and the people of our little village sprang into action with funeral arrangements, food, prayer and support for the grieving family. As I attempted to understand the depth of this warmth and its seeming contradiction to the hot-blooded anger the Canadians so often roused in their neighbors, I began to see the extent of our cultural misunderstandings. Through cycles of conflict, feud, talking things over when tempers cooled, and finally resolution of arguments, we began to understand and accept one another more and to meld gradually into a community. These findings are described in detail elsewhere (Court, 2002). For the purposes of this paper I will relate how my written reflections around one powerful source of conflict began to free me from my own perspective so that I could see the same ‘facts’ from the perspective of ‘the other’. This powerful source of conflict was the family dog.

Most of the Moroccan families did not keep dogs, and many of their children were absolutely terrified of dogs, literally screaming in fear and running away at the sight, in the distance, of one of the Canadians walking a leashed dog. All but one of the five Canadian families had a dog (one family had two). These dogs were kept according to typical North American or European norms. They were fed dog food, with the occasional biscuit or treat thrown in, walked daily, taken to the vet at least once a year, bathed, let inside the house and talked to as if they were at least sentient beings and possibly children. Like any other member of the family, they were committed to for life. In our first few months in the community, it became clear that for the few Moroccan families that kept dogs, the norms were very different. These were watch dogs, tied up for their whole lives, fed table scraps, and rarely even vaccinated, although Israeli law does require it. They were not walked, petted or talked to. Alternately, in a family not afraid of dogs, a child might be given a puppy for a present. In every case this puppy would, after a few months, be dumped by the side of the road a few miles away as soon as it started to bark or chew up the laundry. Dogs were possessions to be discarded, not part of the family and not for spending money on.

Instead, chickens were often kept, running freely about, roosters crowing at 3 A.M. and infuriating the Canadians. Several serious conflicts arose because of this. We couldn’t seem to sleep through the crowing of roosters, and our neighbors couldn’t understand why. In addition, there were several incidents where a dog caught a chicken, and these exploded into angry accusations.

When I began to approach life in the community as research, recording incidents and observations, I began as well to try to externalize and write down the meanings I assigned to these events. Together with description of conflicts, I wrote my own, often angry, interpretations of dog-and-chicken events. As I read and reread these I began to see that my perception of the Moroccans was as ignorant, primitive and unfeeling. Surely only primitive people would choose chickens over dogs, and would discard puppies the minute they become an inconvenience. I began to understand that to our Moroccan neighbors, struggling to feed their large families and make ends meet, and raised in a place and time where money and food were for people, with nothing left over, the Canadian attitude to dogs represented the epitome of the spoiled ‘American’ culture. In this ‘American’ culture money grows on trees, people have few children, have never struggled to make ends met and have the shallow luxury to squander their money and affections on a creature which contributes nothing (except possibly as a watchdog). Chickens, on the other hand, produce eggs and do not drain a family’s resources.

As a participant in community conflicts I felt tremendous hostility to my neighbors. It was hard work to externalize and examine the meanings I placed on dog-and-chicken conflicts so that as a researcher I was finally able to understand the meanings these same events held for our Moroccan neighbors. This helped me to conceptualize our lurches forward to community building as cycles of conflict, feud, communication and resolution, with each cycle enhancing a bit more our understanding of one other.

Second vignette: Hearing others’ voices

After teaching a course on democracy and education to a class of Jewish, Christian, Druze and Moslem education students at a college in Northern Israel, I interviewed the students in the class who were already practicing teachers. All the students were completing an education degree. Most were not yet teaching, but six students, two Jews, one Druze and three Moslems, two of whom were Bedouin, were already teaching in local schools and had come back to college to finish their degrees. My goal was to capture the voices of experienced educators from these various groups and to understand their views on the Israeli democracy and how to teach democratic principles in schools. These results are reported in detail elsewhere (Court, 2004). Recommendations arising from the study included designing more opportunities for interaction between students from the different population sectors, investing more resources in schools in the population sectors that are lagging behind (especially the Bedouin sector), and better teacher training, so that teachers will be better able to engage students in thinking, weighing evidence and making choices. One of the most significant results of the study was the understanding I gained of how much the trust between Israelis from different groups has been damaged as a result of political strife, especially terrorism, which rips families apart and undermines everyone’s deepest need for security. Jewish Israelis during periods of heavy terrorist activity don’t know which Arabs they can trust; Arab Israelis feel that they are not trusted by Jews. Patterns of trust built up over decades and maintained by daily interactions in business, education and the marketplace, have been seriously disrupted.

My challenge in this research was first to externalize and examine my own feelings, values and opinions, and then to try and put them aside and listen in a compassionate and neutral manner to the voices of others. This was not easy, as everyone in this situation has strong feelings and I, as a participant in the society, an educator in the democracy and a passionately protective grandmother, have my own views on what should be done, as well as my own fears, hopes and dreams.

I listened carefully in the in-depth interviews and quoted extensively in the paper that I wrote, trying to intrude as little as possible on the stories that these six people told me. There were two significant steps in the final production of a research paper that contributed to its internal validity. The first was the disciplined reflection on my own position, described above but not detailed extensively in the first draft of the paper I submitted. The second came as a result of reviewers’ comments on this first draft. The reviewers were concerned that I might have inadvertently intimidated the teachers I interviewed, because I was their teacher, or that I might have unwittingly selected portions of their interviews for presentation in order to support a certain view. The reviewers requested that I fully reveal my own background and my relation to those I interviewed, in order that readers could place me in relation to the research. This stimulated a second round of reflection on the difficult issues explored in the paper and led me to introduce my own voice as one of those presented. There is no question that this strengthened the internal validity of the final product, giving readers a more fully developed portrait and more information on which to reach their own understandings.

Discussion

These two vignettes illustrate how one researcher’s personal values and background influenced her understanding of qualitative research data, and the importance of engaging in thoughtful written reflections during the processes of data collection and analysis. Such reflections must be written (not just thought about), because they are, in effect, part of the data, and require analysis just like data collected through interviews, observations and documents. Hostility, rejection, admiration and sympathy that a researcher may feel for her research subjects – strong feelings of any kind - spring from her personal perspectives. This can lead to a lack of balance, a tendency to regard some aspects of the data as extremely important and others as unimportant. Sharing written reflections with another researcher and sometimes with the participants themselves is also valuable. In the study described in the first vignette, I did discuss my reflections with both Canadian and Moroccan neighbors, getting feedback about their perceptions of shared events and about my developing understanding. In the second study, while I did not seek out external criticism, the journal reviewers’ comments on the first draft of the research report set in motion a further, and more self-critical round of reflection.

The researcher is the most sensitive data collection instrument possible, capable of incredible flexibility and depth. With eyes, ears, peripheral vision, sense of smell, and a wealth of knowledge and experience, the researcher not only records but interprets at every step of the research, progressing and changing direction according to new information and intuition. In this richness and sensitivity lie both the power and the danger of qualitative research, for intuition and previous experience can lead the researcher toward her own meanings rather than the meanings of those she is studying. As much as possible, qualitative researchers need to engage in the disciplined, ongoing writing of reflections throughout a research project. In addition, the final research paper should document the path taken by these reflections and reveal the researcher’s background and value stance, so that readers can evaluate interpretations made in the paper.

Not denying or suppressing, but externalizing, revealing and examining our reactions to people and events, is an essential part of qualitative researchers’ work, enriching our understanding and allowing us to approach internal validity. This paper has illustrated how one researcher approached that difficult task in settings where her personal experience, feelings and values clouded her vision and threatened the validity of her work.

References

Banks, J. (1998). The lives and values of researchers: Implications for educating citizens in a multicultural society. Educational Researcher 27(7) pp. 4-17.

Court, D. (2002). Unity and conflict in an Israeli village. Contemporary Jewry 22,

1-17.

Court, D. (2004). Education in a troubled democracy: Voices from Israel.

Curriculum Inquiry 34(1), 47-69.

Denzin, N. (1997.) Interpretive ethnography: Ethnographic practices for the 21^st century. Thousand Oaks, CA: Sage.

Fetterman, D. (1998). Ethnography (2^nd Ed) Thousand Oaks, CA: Sage.

Merriam, S. (1991). Case study research in education: A qualitative approach. San Francisco: Jossey-Bass.

Stake, R. (1995). The art of case study research. Thousand Oaks, CA: Sage.