|
| Title: | Rethinking Appraisal and Assessment |
| Editors: | Helen Simons, John Elliott (eds) Milton Keynes, UK: Open University Press, 1989 |
| Reviewer: | Graham Maxwell, Department of Education, The University of Queensland |
This book is a set of papers resulting from the fourth in a series of international invitational conferences on educational evaluation, held periodically at Cambridge, England, sponsored by the Nuffield Foundation, and attended by prominent exponents of 'naturalistic evaluation'. It is a worthy successor to the well-known and influential books which resulted from the first and second conferences.
The first conference was held in 1972 and resulted in Beyond the numbers game (Hamilton, Jenkins, King, MacDonald and Parlett, 1977). This book was influential in redirecting the practice of program and curriculum evaluation away from the dominant 'experimental' approach, which emphasised measurement of objectives, treatment-control comparisons and statistical analyses, towards a broader 'illuminative' approach, which attended to context, process and audience as well as outcomes. The second conference was held in 1975 and resulted in Towards a science of the singular (Simons, 1980) which was influential in promoting the adoption of case study approaches in educational evaluation and research. The third conference was held in 1979 and focussed on the consequences of the increasing adoption of managerial thinking and practices in education, but produced no communique.
The fourth conference, resulting in Rethinking Appraisal and Assessment, was held in 1985. A total of eight years have elapsed since the conference (four year to publication and four years since) but this has not affected the relevance and importance of the papers in this volume. The focus this time was teacher appraisal and student assessment. The participants came from England (14), the U.S.A. (5) and Austria (1) and included some of the foremost thinkers and practitioners of naturalistic approaches to educational evaluation.
The terms assessment, appraisal and evaluation are often used interchangeably. In the USA, 'evaluation' has usually been applied to any act of judging quality, whether it be focussed on students, teachers, programs, curricula or materials. British usage distinguishes between assessment of students, appraisal of teachers, and evaluation of programs, curricula and materials. In Australia, indiscriminant mixing of USA and British usage often causes confusion. British usage should be preferred for its conceptual clarity and its recognition that purposes, principles and practices can differ according to the focus of attention. It also assists emancipation of student assessment from the concepts of psychological measurement, as often found in books on 'educational and psychological measurement and evaluation'. Student assessment and teacher appraisal are not sub-components of either psychological measurement or program evaluation but separate areas of concern in their own right.
The papers in this book offer analyses of the theory and practice of teacher appraisal and student assessment. In her introduction, Helen Simons presents an interesting argument for the conjunction of these two issues. The argument is that they are linked through the increasing attention being given by politicians and bureaucrats to appraising teachers through assessment of their students' learning. This is a plausible justification. In fact, though, the link has not been maintained. Essentially, there are two sets of articles: the first(ten) articles on teacher appraisal; the second (five) articles on student assessment. They remain only loosely connected, although some attempt at integration is attempted by John Elliott in the final chapter.
The most memorable articles are the lead articles in each section, by Robert Stake and David Bridges respectively, both of which take more seriously than the others the central issue being considered, the relevance and practical implications of naturalistic evaluation.
The articles are considerably diverse in content and style. A brief summary of each is necessary to trace the trajectory of the book. These will be considered in order of presentation. First, there are the articles on teacher appraisal.
Robert Stake establishes the main thesis of the book by arguing for teacher appraisal to be undertaken in context and only then when it can be clearly demonstrated to be helpful to the individuals concerned (primarily) and to the educational system (secondarily). He places an emphasis on approaches which assist and encourage professional development, use multiple criteria which are appropriately adapted to the circumstances, use multiple sources of evidence which are contributed to and controlled by the teacher, and provide positive support for professional growth He sees the inquiry methods of naturalistic, phenomonological and ethnographic studies as providing appropriate support for these emphases. This paper is typical of Stake's insight and wisdom and repays careful reading.
Michael Eraut argues that annual appraisal is wasteful and ineffectual. Instead, he advocates three types of procedures: staff development interviews, monitoring and review. Staff development interviews should be annual interviews between teacher and supervisor to establish strengths and weaknesses, the teacher's priorities for professional development, and the school's priorities for professional development, as well as an action plan for the following year. Monitoring is an informal, supportive and ongoing process directed at identifying problems as they arise and undertaking remedial action and providing professional support where it is needed most. Review is more formal, more infrequent, perhaps on a three or five year cycle, broader in scope and linked to curriculum review. The article is short and the approaches are not elaborated but there are some important ideas here, especially relevant to the appraisal debate in Universities.
In his chapter Robert Burgess wrestles with the problem of criteria for teacher appraisal and with the confusion of managerial and methodological thinking in some of the literature. He argues that the only way of dealing with the contextual and ideological problems inherent in managerial-style appraisal is to institute research-based professional collaboration directed at teachers evaluating and improving their own professional practice. The case for more careful thinking and systematic research on criteria and procedures for appraisal is well sustained. However, collaborative evaluation is unlikely, by itself, to be publicly acceptable as a means of quality control. At a time of public disquiet about the quality of schooling, independent accounts of teaching activities and their effectiveness are needed. Alternatively, parents and students must be brought into the collaborative effort, and as true partners not as tokens. Burgess unwittingly reveals a 'professionalism' ideology which is as undemocratic as the 'managerial' ideology he rightly decries.
Clem Adelman argues similarly to Burgess for peer-involvement and self-evaluation. However, his analysis and propositions are quite different. He is concerned, on the basis of the process-product research into teaching that typical 'objective measures' of teacher effectiveness lack validity. He analyses two cases, in Michigan and in Oregon, where the failure to consult teachers themselves over criteria for successful teaching had dysfunctional effects. He suggests an action-research approach to professional evaluation within a framework of Teacher Consortia (each of about 30 teachers) whose standards would be moderated by a Professional Council on Teaching. This scheme is not elaborated and seems essentially unworkable. The three parts of this article make some interesting points but seem to be about quite different issues.
Richard Winter worries about the two-edged nature of teacher appraisal: the potential for professional improvement and the threat of 'disciplinary action'. He examines three models for their potential effects: the target output model; the performance criteria model; and the diagnostic model. All are seen to involve arbitrary and problematic threats; all focus on the products of teaching. Concern and conflict arises because of the tension between the product orientation of the bureaucracy and the process orientation of the profession. The proposed resolution has three separate components: a pre-promotion program, an accountability program, and a professional development program. The latter is seen as best served by a process-oriented action-research approach, echoing the previous two articles. The more difficult and more pressing problem of how best to implement an accountability program is not engaged.
Ernest House and Stephen Lapan want teacher appraisal to be used primarily for encouraging quality teaching, based on a professional/art view of teaching. They support a 'dimensions of merit' notion for evaluating teachers and place special emphasis on the teacher's 'inference structure', suggesting that this might be assessed through willingness to continue 'experimenting with new ideas', ability to 'explicate and justify the teaching moves' and through validation by peers(!). 'Eliminating incompetence' is seen to depend on evaluations of practice reflecting 'how teachers become good practitioners' such as 'grading practices, test construction, and classroom control'(!). A more satisfactory view of professional expertise can be found in the award winning article by Kennedy (1987).
At about this point in the book, the reader might reasonably expect some case studies implementing a naturalistic approach to teacher appraisal. Unfortunately, there are none. The litany of theoretical articles continues, offering variations on now familiar themes. Unfortunately, too, the variations are sometimes discordant with the original theme. It is easy to become confused. It would have helped the reader had the editors provided a commentary between the chapters. Re-reading Stake at this point in the book does at least re-establish the main theme.
Donald McIntyre seems aware of the lack of attention to practicalities because he apologises that 'this chapter does not report empirical research or development'. Instead, it offers an analysis of what is involved in justifying and clarifying criteria for judging the quality of teaching. The analysis illuminates the problems but it is by no means clear how McIntyre's conclusions can be put into practice. No attention is given to the problem of determining appropriate standards on the criteria. The impression given is that teacher appraisal is very difficult, maybe impossible, to do well.
With the article by Franz Kroath we come at last to an account of practical implementation. Kroath is a man of action not just rhetoric. He reports on a training program for supervisors of practice teaching in initial teacher education, where the criteria for supervisor appraisal were carefully specified and the appraisal procedures were designed to be consistent with principles of naturalistic evaluation. The appraisal was in terms of specified personality traits, teaching competencies and academic knowledge and had both selection and developmental functions. Implementation had most of the features noted in Stake's article as the hallmarks of naturalistic evaluation. This article should be of particular interest to those engaged in teacher education.
John Elliott returns to theoretical argument, focussing on whether teaching performance or personal qualities should be the subject of teacher appraisal. In a closely argued analysis, Elliott presents a case for the centrality of personal qualities in teacher appraisal, whether for dismissal, placement or professional development. Drawing on his previous work in police appraisal, he proposes a two-tiered model of teacher appraisal directed at reducing undesirable power-coercive effects of hierarchical management and enhancing teachers' professional self-development. The first tier involves collegially supported self-reflection and self-monitoring, the second involves the 'presentation of an account' to management of the teacher's own achievements, capabilities, aspirations and needs. Elliott also argues for a program of research to establish appropriate personal qualities in teaching through 'generic competencies' based on 'behavioural event interviews'. These ideas are relevant to the current debate on the assessment of teaching competencies in teacher education.
In terms of the scope and depth of its theoretical and practical understandings and suggestions, this is probably the most important article in the book. The reader should not be deterred by its difficulty. Careful study is worth the effort. There are plenty of ideas for further development and research. Its implications for teacher appraisal in all educational institutions, whether schools, colleges or universities, are profound. Elliott has provided what the previous authors in this volume mostly failed to provide: a clear program for implementing the principles enunciated in Stake's article.
Mary Louise Holly takes a grand philosophical view of human nature in which making sense' is seen as an imperative of life and development. She argues that teacher appraisal is only valuable if it engages the teacher personally in professional development which 'makes sense' of teaching through reflection, theorising, action-research and self-understanding. Although the perspective is different, these are themes which have already been well-rehearsed in previous chapters. The main practical suggestion is that teachers should be encouraged to keep and share a professional journal but does the author do this herself? Its impracticality is admitted by Holly herself: how to find the time?
This is the end of the section on teacher appraisal. Individually, they offer interesting ideas and viewpoints. Collectively, they are tediously repetitive and lacking in practical sensibility. Most of the articles direct the discussion away from appraisal for teacher accountability (summative) to appraisal for teacher development (formative) without resolving the issue of accountability. For readers in a hurry, I would recommend Stake and Elliott (and possibly Kroath) as the most important.
David Bridges offers the first of the five articles on student assessment. This is a very good article. It shows how some of the themes of naturalistic evaluation were finding expression in assessment reform, particularly in Britain, despite contradictory pressures towards centralised control and uniformity. The themes he identifies are similar to those which Stake identified as desirable for teacher appraisal. Here, student assessment is seen desirably as holistic, individualistic, emphasising positive capability, contextualised, multifaceted, negotiated and subject to 'educative dialogue', subject to pupil control over public access, integral to the curriculum, and supporting personal development and self-knowledge. Much of this is now threatened by the push to national curriculum and national assessment but needs to be rediscovered.
The article by Louis Smith and Carol Klaus is interesting but out of place in this book. They focus on some problems in determining appropriate dimensions for depicting the 'whole child' in naturalistic research studies. They speak to social science researchers rather than educational practitioners.
Harvey Goldstein makes a useful distinction between 'separated' and 'connected' assessment and analyses how the discourse of psychometric theory, which has historically underpinned separated assessment, has some deficiencies when applied to separated assessment and is entirely inappropriate when applied to connected assessment. This is an important and evocative article, particularly relevant to the Australian debate on school-based assessment and tertiary selection.
Mary James focuses on the tension between the formative, developmental and confidential aspects and the summative, reportative and public aspects of profile assessment and records of achievement. She draws a careful distinction between negotiation (of access and release) and dialogue (on facts, interpretations and judgements). Negotiation, deriving from naturalistic evaluation, is seen as appropriate for summative assessment because of its concern with democratic values and 'ownership' whereas dialogue is seen as appropriate for formative assessment because of its concern with developing the student's understanding of experience on the basis of evidence. The distinction is important and timely. It should assist in reducing some of the confusion concerning records-of-achievement and portfolios. However, as James indicates, balancing the tension between both processes is not easy. Her suggestion that they be implemented as separate but complementary aspects of recording achievement needs further elaboration and trial.
The article by Ian Stronach is somewhat out of character in this collection. It is argued that the intended effects of pupil profiling in British education, especially its promotion of individual identity and independence, have been subverted and inverted through the social and economic context. The effects of profiling are depicted as worse than useless. Stronach offers no hope and no solution. He is apparently more concerned with developing a 'sociology of assessment' than with improving educational practice. Perhaps he should come down out of the stands and join the field of play.
Finally, John Elliott offers a synthesis and commentary on the whole collection of articles. He attempts, somewhat unsuccessfully, a linking of teacher appraisal and student assessment under the banner of 'performance appraisal' and highlights some of the major themes and implications of 'naturalistic' principles and methods. Particular attention is given to the problem of apportioning responsibility for performance between the context and the person. Other authors in this volume have essentially avoided the issue by emphasising formative self-appraisal for professional development and inferring that summative evaluation for selection, appointment, advancement and promotion is somehow illegitimate. Elliott prefers to see a balance between the two and justifies this in terms of theories of symbolic interactionism but is unable to suggest how the balance is to be struck. The failure to address this issue is a major weakness of the book. Perhaps the group were all too much of the same mind to realise the insuffficiency of their arguments.
Despite its weaknesses, this book opens up many issues of great importance and deserves a wide readership. It is to be hoped that these issues will be thoroughly debated, challenged, elaborated and investigated and will mitigate the worst effects of current 'managerial' policies concerning teacher appraisal and student assessment. Elaboration is especially needed on the practical aspects of naturalistic appraisal. There needs to be much more research on the interaction between teaching context and teaching quality; claims of a one-way dependency (either way) seem unwarranted, but where is the evidence. Studies are also needed to identify dimensions of teaching ability which generalise across contexts, at least to some extent. Some recent research on the assessment of competencies in professional nursing (ANRAC, 1990) may provide a useful prototype for studying the generalisability of teaching abilities and for defining appropriate kinds of generic criteria for judging such abilities. It would also be helpful to have some case studies of the implementation of 'naturalistic appraisal' as promulgated in this book. We might hope that the authors are now hard at work doing just that.
Hamilton, D., Jenkins, D., King C., MacDonald, B. & Parlett, M. (eds) (1977) Beyond the numbers game: A reader in educational evaluation, London: Macmillan Educational.
Kennedy, M. (1987) 'Inexact sciences: Professional education and the development of expertise', in Review of Research in Education, ed Rothkopf, 14, 133-167.
Simons, H. (ed) ( 1980) Towards a science of the singular: Essays about case study in educational research and evaluation, Occasional Publication No. 10, Norwich: Centre for Applied Research in Education, University of East Anglia.
| Title: | Making Schools More Effective |
| Authors: | Barry McGaw, Kevin Piper, Diana Banks, Beryl Evans Australian Council for Educational Research, 1993 |
| Reviewer: | Reviewer: Errol McDonald, Assistant Executive Director (Studies) Metropolitan East Regional Office, Department of Education, Queensland |
The views of over 7,000 individuals and groups are contained in Making Schools More Effective - a concise report on the findings of a national survey about effective schools. Excluding Queensland, which did not formally participate in the project, more than a quarter of the schools in the country responded to the national survey.
Making Schools More Effective describes schooling for modern Australia as parents, students, teachers and school administrators would like it to be. The noticeable outcome from the monograph is that there is a broader and more qualitative Australian community vision for schools and schooling than some segments of the public media, and much of the current effective schools literature, would have us believe.
The strong message is that Australians want schools that stimulate intellectual development by setting high, but realistic, expectations for their students. They also want schools that develop students' personal and social skills. Above all, they want schools in which students learn to think well of themselves, to develop a sense of personal value and a confidence in themselves, to take with them to adult life. They want competition, but they want it to be with a student's own past performance, not with the performances of other students. They want the notion of 'personal best' translated into schooling as a goal for students in the way it is used with elite athletes.
The key to an effective school is seen to be its staff. They should be focused on learning, concerned with individual students' needs, and engaged in their own professional development. Local schools should have more say than at present in decisions about staffing - if not selecting the staff, then at least determining the staffing profile.
The main surprise in the findings was the lack of community and practitioner emphasis on such media favourites as public examinations, improved student performance in the 'basics', stricter school behaviour management, and improving the vocational skills of young people.
Making Schools More Effective can inform educational reform in Australia by way of its collation, for the first time, of the ideas of contemporary Australians about what they desire from the nation's schools. The findings of this study will cause school communities and policy makers to reflect on their current circumstances as well as the future direction of Australian schooling.
| Please cite as: QIER (1993). Book reviews. Queensland Researcher, 9(3), 44-53. http://education.curtin.edu.au/iier/qjer/qr9/bookreviews.html |