Multiple site action research case studies: Practical and theoretical benefits and challenges

A curriculum initiative project was implemented in four schools in Singapore over a span of five to six weeks during 2004. The project employed a number of different schools: girls only, boys only and co-educational schools; different levels of performance in a graded situation; multiple teachers and classes within each site; and control and experimental conditions for the curriculum implementation. In conducting research in the diverse schools, there was also an opportunity to study the interactions between action research and multiple site case studies.

Though action research and case studies are frequently used in education to research curriculum initiatives, their interactions are seldom explored. Moreover, the practical benefits and challenges of multiple site case studies in action research are little discussed. In this paper, the interactions between action research and multiple site case studies as well as the practical and theoretical benefits and challenges are explored. By examining the particular benefits and challenges presented by this project, it is hoped that this paper will contribute to a better understanding of multiple site case study action research through describing the practical benefits and challenges, as well as the theoretical advantages and disadvantages, of multiple site action research case studies, and the means used to overcome the challenges that arose during the course of the case studies.

Background

The research was undertaken in naturalistic environments. The schools offered actual pre-existing classes and the teachers who normally taught these classes volunteered to teach under the program. The proposed curriculum was also implemented without any disruption to the normal working conditions or character of the participating schools and classes. The Literature Driven English Program (LDEP) was carried out as part of the normal operation in each of the four schools. In doing so, there was scope to study how well the curriculum worked in actual conditions, irrespective of the existence of the many differing variables inherent within and among the different schools. Since the project was action research through which it was hoped that there would be amelioration in the "rationality... of... the situations in which the practices are carried out" (Kemmis, 1988), it was important to ensure that the pre-existing conditions remained unaltered. Any study conducted in controlled conditions would not lead to an accurate understanding of the effectiveness of the curriculum or the factors that affect the implementation of any program.

Research questions: Major and sub-questions

The major research question is: Does an integrated English language and literature curriculum better develop the students' English Language skills? This question articulated into the following sub-questions which guided the development of the research methods.

1. Are the students able to write more effective narratives if they use a given literature text as a model of a good narrative?
2. Do students become better writers if they are taught holistically and contextually rather than in 'bits and pieces'?
3. Are students able to produce a more grammatically accurate piece of writing if they are taught in context through a literature text?
4. Are students able to comprehend better if they are taught comprehension skills in context through a literature text?
5. Are students better able to critically analyse a literature text if, in addition to content, they analyse the language of the text as well?

Participants

Six schools articulated initial interest and four schools eventually agreed to participate in the research. Among the four schools, two were co-educational schools, another was a boys school and the fourth, a girls school. The lower Secondary One (Year 7) students of these schools, who had just entered secondary education after completing the Primary School Leaving Examinations (PSLE), were selected to participate in the research. These students were selected because they were not sitting for any major examinations at the end of the year. This would ensure that participants' interests were safeguarded and any ill effects were minimised (NHMRC n.d., 4.3) The research also included the implementation of the specially designed LDEP for five to six weeks in each of the four schools from the period beginning July 2004 to end August 2004. The program was designed to run for such a short duration because schools could only allow a limited period of time for the research to be conducted in their schools. Thus, any inconvenience or detrimental effects which might occur due to the experimentation could be reduced (Dockrell 1990).

One of the aims of the research was to discover the effectiveness of the LDEP through experimentation by having experimental and control group of students. The experimental group of students were to be taught under the program whereas the control group of students were to follow the regular English and Literature Programs. However, one of the schools wanted all its Secondary One Express and Normal (Academic) classes to participate in the research. (Express students and Normal (Academic) students take four and five years respectively to complete their secondary education.) The three other schools, on the other hand, were willing to have some classes participate as control classes. Table 1 displays the four schools and information about participant classes and teachers in the schools. In all, there were seventeen experimental classes (including one which was excluded from the analysis) and eight control classes (Table 1). Thirteen teachers taught the experimental curriculum (Table 1).

School	Type	Experimental		Control
		Classes	Teachers	Classes	Teachers
1	Co-educational	8*	6	0	0
2	Boys	3	1	3	1
3	Girls	3	3	2	2
4	Co-educational	2	3	3	3
*Originally there were 9 classes, but one, a Normal (Academic) class, had to be excluded from analysis due to discrepancies in some of the data collected from the class.

One of the schools, the boys school, already had an integrated English and Literature Program. However, there were distinct differences between the school's program and the LDEP. The existing program in the school was a Literature Based English Program (LBEP) whereas the experimental program was a Literature Driven English Program (LDEP). In the LBEP, most of the teaching time is spent on literary analysis. Limited time is spent on teaching language skills. When these skills are taught, the literature text is not used as a medium of instruction. The LDEP aims to correct this imbalance by connecting the explicit teaching of language skills to the study of the literature text. The literature text becomes the vehicle through which students are taught to become better writers and readers. While in the LBEP it is hoped that students will gain language skills through interaction with the literature text, in the LDEP nothing is left to chance. The students are guided to see the literature text as a model of well crafted language. It is hoped they will, through the process of theory application, be able to transfer knowledge of the craft into their own use of the language.

The intent of the research was to discover if the program could be successful in a naturalistic environment with many mediating variables within schools and among schools. The mediating variables included the following.

• Type of schools, co-educational and single sex;
• Gender;
• Differing teaching styles of the teachers;
• Differing language fluency of the students, measured by the English grade they achieved in the Primary School Leaving Examinations;
• Differing class cultures;
• Differing streams, Express and Normal (Academic).

Multiple site action research case study

The curriculum was tested in four different types of schools with students of varying abilities (Figure 1). The curriculum was also taught to classes with varying cultures by teachers with different teaching styles (Figure 1). Therefore, the schools were viewed as cases with several sub-units based on the many variables found within and across schools. Thus, the use of multiple site case studies was considered the most suitable option for the research on testing the effectiveness on the LDEP in imparting language skills.

Figure 1: Overview of the research on the LDEP

Theoretical benefits

Instead of concentrating on a single site, it was decided to conduct the research in multiple sites because "[t]he same case study may involve more than one unit of analysis. This occurs when, within a single case, attention also is given to a subunit or subunits." (Yin 1994, p.41, emphasis is Yin's). Thus, each school and each class within each school becomes a case study with sub-units. The conclusions drawn from the findings from each school are studied in relation to the school as well as in comparison to the other schools. In this way, what Yin (1994) considers as a possible problem in embedded case study design can be avoided. As he puts it, a major problem with "[a]n embedded design... occurs when the case study focuses only on the subunit level and fails to return to the larger unit of analysis" (Yin 1994, p.44). While the effectiveness of the LDEP is analysed in relation to each mediating variable, in the end, the researcher returns to the main research question, namely is the LDEP effective in imparting language skills in spite of the mediating variables.

By studying the multiple sites as individual case studies as well as a larger single case study, sub-unit analysis as well as cross comparisons can be made. In doing so, it may be possible to arrive at either a literal replication or a theoretical replication (Yin 1994, p.46). A literal replication occurs when replication of measured outcomes is achieved. In a theoretical replication, there is an absence of comparable measurable outcomes. However, these contradictory outcomes generate explanations to account for the lack of comparability in relation to the variables in the design. These explanations could lead to the creation of hypotheses.

If a literal replication is made, in that the curriculum is effective across the various sub-units, then it may be possible to come up with a "theoretical framework ... [which] later becomes the vehicle for generalizing to new cases" (Yin 1994, p.46). On the other hand, if conflicting conclusions arise, then it may be possible to establish a theoretical replication. In the case of theoretical replication, the researcher may surmise that the curriculum may be effective in some schools or classes but not in others due to some pertinent mediating variables exerting an influence over the findings. The assumptions could then generate further research hypotheses and designs to evaluate the theories that arose out of the theoretical replication.

In addition, mixed methods were used which led to a richer analysis. A case study is "a unit of human activity embedded in the real world; which can only be studied or understood in context" (Gillham 2000, p.1). In order to have a full contextual understanding of multiple site case studies, in particular, quantitative data and qualitative data were collected. Indeed, "[t]he embedded case design allows for both qualitative and quantitative data and strategies of synthesis or knowledge integration" (Scholz & Tietje 2002, p.14). The quantitative data were derived from the pre-test and post-test results and the content analysis of the surveys that students completed. The qualitative data included data from interviews, observations, teachers' log book entries and field notes. The use of quantitative and qualitative data led to "the multiple sources of evidence [which] essentially provide[d] multiple measures of the same phenomenon." (Yin 1994, p.92).

Therefore, action research into a curriculum initiative in multiple site case studies provides a number of advantages which also argue for the validity of the research findings and conclusions drawn from them. These advantages include the following.

1. The curriculum can be tested in a naturalistic environment. Schools vary, students differ and so do classes. It would be difficult to reach any conclusion about a curriculum that has been tested on a single school. The school culture or organisational structure may have an influence on whether the curriculum succeeds or not. The school may have high achievers and thus, the motivated students could be a reason why the curriculum succeeds. There would be many plausible reasons for a curriculum succeeding in one school and just as many possible reasons why it may not in another. Therefore, dependence on a single site or school would result in less robust findings since these findings would be applicable only to that school and perhaps to schools with very similar characteristics. However, multiple site case studies afford the researcher with the opportunity to test the curriculum in situations that include more variables.

2. Multiple site case studies will lead to a greater coverage or sample of potential variables. In the case of the research on the LDEP, there were many variables with regard to ability, gender, school and class cultures, and teaching styles. These variables existed within as well as across the schools studied. Thus, the greater the number of variables, the greater is the chance that the conclusions about the LDEP drawn from the outcomes could be applied to more schools with similar variables and contexts.

3. It would be more likely that the findings will be more robust if there are more mediating variables. A curriculum that was tested in all four schools was the only constant amidst a myriad of variables. If it succeeded, it would be easier to draw a conclusion that the inherent merits in the curriculum were the most likely reasons for the improvement observed in the students' performances.

4. Any conclusion formed from an analysis of similar findings collected from multiple site case studies may lead to a "naturalistic generalization, derived at by recognizing the similarities of objects and issues in and out of context and by sensing the natural covariations of happenings. To generalize this way is to be both intuitive and empirical" (Stake 2000, p.22, emphasis is Stake's). Thus, the conclusions arrived at from such analyses could be extrapolated to other schools with similar contexts.

5. The naturalistic generalisation could result in a wider potential interest and audience. Other schools in Singapore, and perhaps, in other countries as well, would have a wider choice of schools with which to compare theirs. As such, the likelihood of finding schools with similar circumstances in multiple site case studies is high.

Theoretical challenges

1. Reconciling the many differences and conflicts in pertinent variables was one of the challenges. For instance, when the findings were analysed from the perspective of gender, the girls from the two co-educational schools and a girls school were grouped together. However, there are differences between the culture of a co-educational school and a single sex school. In addition, there were problems in the implementation of the program that were only encountered in the girls school. Therefore, in the analysis what was done was to analysis the performance of all the girls as one sub-unit of gender. Then, a second sub-unit was created to differentiate the performances of the girls from the co-educational schools from the performance of the girls in the girls school.

2. Deciding on the meaningfulness of the variables was a second challenge. A variable on its own is of no importance unless it has the potential to influence the findings. Initially, the outcomes of the performances in the writing assessment of the classes taught by the same teachers in the control and experimental groups were examined. However, such a comparison was found to be unnecessary and irrelevant. In the first place, the control group did not register any improvement. It would be more pertinent to compare the outcomes of the control and experimental classes taught by the same teachers in one of the schools. Then, the teaching style would be the control variable in both the experimental and control classes taught by the same teacher. The different curricula used in the experimental and control classes would become the mediating variable.

3. The third challenge lay in deciding how the discrepancy in outcomes between schools or classes could be explained. Since there were many variables, it would be difficult to decide on which variable or variables may have influenced the difference in the outcomes. For example, Schools 1, 2 and 4 registered similar improvements but the improvement achieved by the students in School 3 was significantly lower. Since the three other schools would also have many mediating variables, what was different about the variables found in School 3 that had an impact on the overall performance of the students? Would School 3 being an all girls school become an influencing factor since the other three schools were co-educational and boys schools? Or were other factors involved?

4. The final challenge rested on the premise that if there were discrepancies to which answers could not be easily sought, there could be no replication. Then, the results would be pertinent only to the individual cases and would have no significance for any other schools. Fortunately, in the research on the LDEP, there were more similarities than discrepancies and an attempt could be made to explain whenever discrepancies did crop up. Within classes with discrepancies there were similarities as well as differences between these classes and others.

Practical benefits

1. The diversity of variables led to a greater understanding of the degree of effectiveness of the curriculum. A curriculum that is effective in more schools with more classes would have more practical benefits than one that is found to be effective in only one school or a few classes. The possibility of coming up with a literal replication or a theoretical replication is higher. In all the four schools, there was improvement in the performances of the experimental classes, leading to a literal replication from which a theoretical framework could be created (Yin 1994). At the same time, there were some discrepancies in the outcomes of the writing assessment of one school and a couple of classes in particular. The degree of improvement was not as high as for the other schools or classes. A closer examination of the qualitative data led to a theoretical replication (Yin, 1994) whereby possible explanations were derived from the differences found in the way the program was implemented and in the similar attitudes of the students in the classes.

2. In multiple site case studies, there are also multiple conditions which reflect the naturalistic conditions found in the educational arena. By trying out the curriculum in these different conditions, without manipulating or controlling any of the variables, any finding in relation to the effectiveness of the curriculum would be more reliable. In the research on the LDEP, it was a 'take us as you find us' situation. The researcher did not attempt to impose any conditions but adapted and adjusted according to the needs and requirements of the schools. For example, in one school the periods were one hour long whereas in the other schools the periods were either thirty or thirty five minutes long. The lesson outline and lesson plans were adjusted to reflect these differences in timetabling.

3. Multiple site case studies also yield a large amount of data to work on. There are two advantages to this. The first advantage is that should there be any problem with any particular data, there are other sources on which the researcher can rely. In the LDEP research, data from the surveys and the pre-test and post-test results of one class were excluded when it was found that there was a high possibility that the data might be unreliable. However, there were data from sixteen other classes that could be analysed. The second benefit lies in the rich source of data which allowed for theoretical replication in the case of discrepancies in outcomes. The classes with discrepancies came from different schools but from observations, conversations with teachers and interviews with teachers and students, similarities between the classes emerged. In this instance it was possible to form a theory about the discrepancies.

Practical challenges

1. The time and resources available for the conduct of this research project were limited. There was only one researcher working alone in schools that could afford only a limited time of a few weeks to complete the program. It was a strain on the researcher and perhaps, if the attention was devoted to one case study instead of multiple case studies, some of the limitations of the research could have been eliminated. One limitation was that the time the researcher spent with the teachers from each school was considerably reduced. As a result it was left to the teachers to contact the researcher should they experience any difficult in the implementation of the LDEP. Unfortunately, many of them did not, and the problems became evident only during school visits and interviews at the end of the program.

2. There was also the problem of whether there would be replication since the research was being conducted in multiple sites with very different characteristics.

3. The large amount of quantitative and qualitative data meant a lot of time and energy were needed to analyse them.

Overcoming the practical challenges

Firstly, the teachers and schools were informed that the researcher could be contacted at any time should they need her assistance. The teachers made use of email and mobile phones to contact the researcher whenever they experienced difficulties or were in doubt. There were instances of emergencies, when assistance was needed immediately and assistance was provided without delay. On a particular occasion, the night before the teaching of a lesson one of the teachers had questions about the lesson and these questions were cleared immediately so that the teaching of the lesson could proceed smoothly.

Though the workload was great, the prospect of meeting the challenge of analysing the large amount of data proved to be exciting. The time and resources spent on discovering what the findings meant were worthwhile.

Conclusion

On the other hand, the theoretical and practical benefits derived from such research are enormous. The curriculum was tested in naturalistic environments with little manipulation of the existing mediating variables. The inclusion of a greater number of variables also meant that more variables were exposed to the experimentation making the findings more robust and giving rise to the possibility of arriving at a naturalistic generalisation to use Stake's (2000) term. There was also the possibility of more schools expressing an interest in the findings of the project since there was a greater likelihood of them finding participant schools comparable to their own. The practical benefits included the testing of the curriculum in different naturalistic environments which in turn led to a greater awareness of the extent of the effectiveness of the curriculum. Additionally, there were dual advantages in collecting the large amount of data. In the instance where a particular set of data proved unreliable, other sets could be relied upon and there were also more chances of developing theories should there be disparities in outcomes from the various sites. The theoretical advantages aid in validating the research and the practical benefits make the findings more reliable. For these reasons, a multiple site action research undertaken to explore a curriculum initiative was found to be fruitful and rewarding.

References

Gillham, B. (2000). Case study research methods. London: Continuum.

Kemmis, S. (1988). Action research. In J. P. Keeves (Ed.). Educational research, methodology, and measurement: An international handbook. Oxford: Pergamon Press.

NHMRC (n.d.). Research involving children and young people. [viewed 3 Oct 2003 at http://www.health.gov.au/nhmrc/hrecbook/01_commentary/04.htm, verified 24 May 2006 at http://www7.health.gov.au/nhmrc/publications/hrecbook/01_commentary/04.htm

Scholz, R. W. & Tietje, O. (2002). Embedded case study methods: Integrating quantitative and qualitative knowledge. Thousand Oaks, CA: Sage Publications.

Shofield, J. W. (2000). Increasing the generalizability of qualitative research. In R. Gomm, M. Hammersley & P. Foster (Eds), Case study method: Key issues, key texts. London: Sage Publications.

Stake, R. E. (2000). The case study method in social inquiry. In R. Gomm, M. Hammersley & P. Foster (Eds). Case study method: Key issues, key texts. London: Sage Publications.

Yin, R. K. (1994). Case study research: Design and methods, 2 Edn, Vol. 5. Thousand Oaks, CA: Sage Publications.

Authors: Mary Delfin Pereira is currently working towards a professional doctorate in Education from the University of Notre Dame Australia. Her research topic is related to discovering the effectiveness of an integrated English and Literature program in improving English Language skills. Prior to the commencement of her doctoral studies, she was working as an English Language and Literature teacher in a secondary school in Singapore. She had also worked in urban and sub-urban schools in the United States where she received her Master's degree in Instructional Systems Development. Email: delfinpereira@lycos.com Associate Professor Roger Vallance was Director of Research Training at Notre Dame in 2004-5. Dr Roger Vallance holds a PhD from Cambridge University. He has an earlier background of secondary science teaching and school administration and now explores research interests in educational and values based leadership, the education of boys and research methods particularly qualitative methods and research ethics. He has developed and taught in a Master of Leadership program, and is a Visiting Professor at the University of Illinois at Urbana-Champaign in the second half of 2005. Email: rvallance@dwu.ac.pg Please cite as: Pereira, M. D. and Vallance, R. (2006). Multiple site action research case studies: Practical and theoretical benefits and challenges. Issues In Educational Research, 16(1), 67-79. http://www.iier.org.au/iier16/pereira.html