Standards framework: Developing scales of achievement in post-compulsory education: A case study

This paper presents the findings of a research project into the establishment of a standards based scale of achievement using the present Western Australian TEE Geography Year 12 Syllabus. The background to the research, including its relevance to the Curriculum Council's Post-compulsory Review, the use of formative assessment, the issue of standard-referenced assessment and the notion of courses of study, is discussed. A rationale for the choice of Geography as the subject context of the research is outlined and the paper continues by describing the procedure involved in undertaking the research project, including a brief discussion of Rasch analysis. The final section discusses the encouraging results of the investigation and suggests the usefulness of conducting further research on the development of scales of achievement.

Background

Implementing the Curriculum Framework in the post-compulsory years presents the opportunity to address a number of issues in the current system, which has operated since the last major review (McGaw, 1984). Essentially, the current review seeks to refocus post-compulsory education from a subject-centred, inputs-focused approach to one, which is student-centred and outcomes-focused.

From a broader perspective, global and national developments have significant implications for post-compulsory schooling: for example, structural and technological adjustments within industry over the past two decades, rapidly changing information and communication technologies, and patterns of international mobility and migration. Agreements and decisions by Commonwealth and State Ministers, at the national level, on the goals of schooling, benchmarking, standards and targets also have far-reaching consequences for the structure of schooling.

Standards and a new assessment paradigm

One of the key features of assessment practices, as outlined in the Curriculum Framework, is the emphasis on formative (developmental) assessment. This emphasis is somewhat different from present practices in post-compulsory Tertiary Entrance subjects, which tend to rely upon summative and normative assessment. The educational usefulness of formative assessment is well documented in the research literature. An example of such documentation is that of Black and Wiliam (1998, p. 3) who referred to a meta-study (of students aged from five years through to university graduates, across several school subjects and several counties), which showed that "...innovations which included strengthening the practice of formative assessment produce significant, and often substantial learning gains."

Essentially, Black and Wiliam (1998) argued that assessment regimes, which are built on clearly defined standards that provide a focal point for teacher-student interaction, are the most effective in improving learning. Teachers act as facilitators of learning. They work with students to (1) identify the current level of achievement, (2) target the desired level of achievement and (3) describe the changes that will be required to achieve the target level.

The corollary to the use of formative assessment is the emergence of scales of achievement (which describe clearly stipulated standards), against which students are able to monitor their progress. Teachers can also use scales of achievement in the planning of learning experiences designed to enable students to improve their level of achievement. An integral concept relevant to any scale of achievement is that of standard-referenced assessment. In describing this concept, Willis and Kissane (1997, p. 34) wrote "In essence, sets of standards are based on the notion of a continuum of increasing knowledge, quality or competence with the 'standards' intended to provide stable reference points or frameworks against which a particular student's quality of performance or level of attainment or achievement can be judged directly without reference to other students or to overall scores."

Sadler (1989) added to the evidence about the advantage of formative assessment when commenting on self-assessment by students as being a necessary part of formative assessment. He suggested that through such a process students acquire three types of information: the desired goal, evidence of present progress towards that goal and some understanding of how to narrow the gap between the two positions. Sadler's (1989) considered opinion appears to be supporting the use of a scale of achievement and, by implication, standard-referenced assessment.

Thus, the research project under discussion evolved from a combination of the previously described issues, namely, the Post-compulsory Review and the implementation of the Curriculum Framework, with the associated concepts of formative assessment and a standard-referenced scale of achievement.

The research project

Introduction

Methodology

Analysis of current year 12 geography syllabus

Step 1  Analysis of the current year 12 Geography syllabus to identify subject outcomes and their links with the Curriculum Framework outcomes. Step 2 Mapping of the subject outcomes to identify levels with existing progress maps: e.g. Education Department of Western Australia's Student Outcome Statements. Step 3 Using the levels analysis in step 3 to propose a generic scale of achievement for each subject outcome.

Analysis of student work samples

Step 4 Designing a marking key for the 1998 Geography TEE paper according to outcomes and trialing this key with a small number of 1998 TEE Geography scripts. Step 5 Marking a sample of scripts from the 1998 Geography TEE paper according to outcomes. Step 6 Analysing the marks using Rasch item response theory.

Identifying subject outcomes

This analysis established that the current Geography syllabus is relatively narrow in the range of Curriculum Framework outcomes it covers. Of the ten subject outcomes, eight were linked to the Place and Space outcomes. Furthermore, six of these eight outcomes were related to Features of Places. The only other Curriculum Framework outcome represented was Investigation, Communication and Participation (two subject outcomes). Both of these outcomes were linked to processing and interpreting information. There was no opportunity for students to develop in the areas of Planning Investigations, Conducting Investigations or Evaluating and Applying Findings.

Mapping against existing progress maps

Through this analysis it was concluded that the current Geography syllabus mapped to levels 6 to 8 for the Place and Space outcome, but at significantly lower levels for the Investigation outcome. In essence the current syllabus lacks breadth and depth.

This kind of analysis, as revealed in Table 2, would be most useful in identifying priorities for redeveloping the current syllabus.

Proposing a scale of achievement

Development of the marking key

• marks for an item indicate levels of achievement on the outcome assigned to it
• distinctly different levels of achievement on the item should be clearly identified from student work and should be described by "pointers" - i.e. what to look for in an answer
• Pointers should be described in such a way as to permit objectivity in marking, i.e. they should not require a wide range of subjectivity in interpretation
• different levels of achievement should be given successive integers (0. 1. 2. 3. etc.) as marks, beginning with "0" for "no achievement of the outcome". Disregard the original mark allocation in the TEE paper
• if an item does not address an outcome, leave it for discussion just before the marking key is finalised
• you do not have to include every item. Some may prove to be too difficult
• greater achievement of the outcome is what gets more marks. Simply writing more and more at the same level does not necessarily show higher achievement of the outcome.

This process also provided an insight into one useful form of professional development. A marker commented after the two-day exercise "It took us some time to shift our thinking, but now I've been through that I think I'm beginning to understand what an outcomes-focus is about. That is the most effective professional development I've had and I'll use a similar approach with my staff."

Marking a sample of scripts

Analysing the marks

The marks were transferred to a computer file, verified and analysed using RUMM (version 2.7q). This program fits the data to a unidimensional latent trait by calculating the locations, on a scale of achievement, of each item in the examination. As stated by Peck (1999a, p. 1) "The location is a measure of difficulty, or alternatively of progress along the dimension of achievement represented by the Outcome."

Using Rasch analysis it is possible to calculate the location of each threshold - a point on the scale between two consecutive levels of achievement. A student who is located above a threshold is more likely to score the higher of the two marks.

Results

Preliminary analysis revealed that four items had redundant response categories and were combined, and that another item misfitted badly and was removed. Figure 1 shows each item's thresholds. The reader should note that when interpreting this figure the criterion to consider is that if all of the individual thresholds are aligned on the scale, then generic performance levels have been demonstrated to be applicable in a number of contexts (item to item). In this particular case, there are few exceptions (for example, items 115 and 22a display reverse thresholds) to this criterion of judgement. However, Peck (1999a, p.4) suggests that these exceptions arise

because the sample of data is not large enough to smooth out the statistical fluctuations. These reversals are probably not serious enough to be of concern.

Analysis

In all probability, the alignment of levels of achievement would have been better if the task that students responded to had been written from an outcomes-focused syllabus, rather than in a TEE paper. The availability of responses to TEE questions and the lack of an outcomes-focused curriculum were the main reasons for the selection of the responses' sample. An Outcomes-focused system of pedagogy and assessment would have a set of generic achievement levels similar to those, which appear in Table 4 (Peck, 1999a, p. 3). Subsequently, assessment tasks and specific marking guides would be designed to give students the opportunity of demonstrating their level of achievement of the Outcomes.

In commenting further on the results, Peck (1999a, p. 5) stated, "An incidental product of the Rasch model analysis of this data was a measure of each student's ability. Although it was not the purpose of this analysis to compare these abilities with the results of marking the TEE with the traditional marking guide, it may be of interest to note that the correlation coefficient between the two sets of scores was 0.837. This relatively high correlation suggests that it would make little difference to students' ranking whether a TEE marking guide or an outcomes-focused marking guide was used. (Note: the benefits of an outcomes-focused education can not be demonstrated by a correlational comparison of this type.)"

Conclusion

The use of an achievement standards framework has the additional advantage that it can rank students for university entrance, while also reporting on what these students have actually learned. Further research into the development of scales of achievement should help contribute to demonstrating the credibility of this type of assessment process.

The results of this research provide a response to those critics of outcomes-focused curriculum who suggest that such an approach will result in the "dumbing-down" of education. Indeed, the higher reliability index of the outcomes-focused marking key, by comparison with the TEE marking guide, and the strong correlation in the ranking of the students when comparing the two marking keys suggests that, in this instance, such a suggestion is not in evidence. Furthermore, throughout the process of developing the marking key, the teachers involved were of the opinion that the required criteria to achieve at the higher levels were substantial.

The research process used in this investigation provides a model for future research in this field. With the development of outcomes-focused courses of study to occur in the near future, valuable information has already been gathered, particularly in respect to development of assessment tasks and associated marking keys. The way, in which the marking key was developed, with active teacher involvement, also provides guidelines for the development of effective professional development programs as curriculum change is implemented in the first decade of the 21st century in Western Australia.

The research demonstrated that it is feasible to set valid achievement standards using curriculum documents and student work samples. The research methodology used in the analysis of student work samples emphasised the role of social construction of valid standards; that is, as Wiliam (1996, p. 287) argued, the standards exist "by virtue of a shared construct in a community of practice". Further research is needed to test the robustness of these standards in 'high-stakes' assessment such as that used for university selection. A starting point for this research is the high degree of correlation noted by Peck (1999a) between the rankings achieved by the 319 students in the study sample on the outcomes-focussed marking scale and the actual rankings achieved in the 1998 TEE.

References

Barrett, R. (1999). Society and Environment Standards Project. Unpublished paper. Curriculum Council of Western Australia.

Black, P. and Wiliam, D. (1998). Inside the Black Box: Raising Standards Through Classroom Assessment. London: King's College.

Curriculum Council Act 1997. Perth: Government of Western Australia.

Curriculum Council of Western Australia (1998). Curriculum Framework. Osborne Park: Curriculum Council of Western Australia. [verified 2 Jan 2002] http://www.curriculum.wa.edu.au/pages/framework/framework00.htm

Curriculum Council of Western Australia (1999). Curriculum Update 12. Osborne Park: Curriculum Council of Western Australia.

McGaw Report (1984). Assessment in the upper secondary school in Western Australia: Report of the Ministerial Working Party on School Certification and Tertiary Admission Procedures. Perth: Government Printer.

Peck, R. (1999a). A Method for Defining Standards for Outcomes. Unpublished paper. Curriculum Council of Western Australia.

Peck, R. (1999b). Outcomes Standard Project. Unpublished paper. Curriculum Council of Western Australia.

Sadler, R. (1989). Formative Assessment and the Design of Instructional Systems. Instructional Science, 18, 119-144.

Wiliam, D. (1996). Meanings and Consequences in Standard Setting. Assessment in Education, (3)3, 287-308.

Willis, S. & Kissane, B. (1997). Achieving Outcome-Based Education. ACT: Australian Curriculum Studies Association.

Wright, B.D. & Stone, M.H. (1979). Best Test Design. Chicago: Mesa Press.

Authors: Originally trained as a History, Geography and English teacher, Mr Rees Barrett worked for eighteen years with the Education Department in schools, central office (Curriculum Branch), and a district office. Rees was involved in the development of the design brief and the profiles for Society and Environment learning area in the National Collaboration in Curriculum Project. More recently he has led the Curriculum Council's Key Competencies project and Common Assessment Framework project. Formerly a History Head of Department, Dr Graeme Lock also has extensive experience in lecturing graduate and postgraduate university students in areas including curriculum theory, practice and evaluation; the advanced study of teaching; educational policy studies and educational administration. Previous conference papers and presentations have covered aspects including curriculum evaluation, community participation in school decision-making, teacher occupational stress and small group learning. Please cite as: Lock, G. and Barrett, R. (2002). Standards framework: Developing scales of achievement in post-compulsory education: A case study. Issues In Educational Research, 12(1), 35-48. http://www.iier.org.au/iier12/lock.html