The previous articles in this journal issue support the conclusion that competency based assessment is a special case of standards based assessment and that there is no theoretical inconsistency if additional proficiency standards above the basic competency standard are adopted. Various options for assessing and reporting proficiency levels within a competency based assessment system can be identified. These options, taken singly or in combination, need serious consideration.

Both Peddie (1997) and Wilmut and Macintosh (1997) in this issue of the journal affirm that competence is not observed directly but is inferred from observations of performance, that is, that ascription of competence requires an assessor to make a 'subjective' judgement based on evidence from observable performances. Peddie (1997) points to the long history of recognition of the difference between 'competence' and 'performance' in research on learning, particularly child language acquisition.

This view of competence is supported by many other researchers, including Hager (1994), Hager, Gonczi and Athanasou (1994), Gonczi (1993), McGaw (1993) and Wood and Power (1987). It leads to what the Assessment Steering Group (1993) has characterised as the evidence-based judgement model of assessment in contrast to the measurement model of assessment. As Peddie (1997) says: '... there is no abstract, external, objective "competency standard" somewhere out there, just waiting to be incorporated into a unit. Similarly, there is no abstract, external and objective "competence" which will be acquired by the learner as one might acquire a hamburger from the corner takeaway.'

Wilmut and Macintosh (1997) draw on the work of Gonczi (1994) to affirm that 'setting the performance of particular tasks into a context of general attributes' is the only way to ensure that competency based assessment attends to the mastery of the knowledge and processes underpinning competence and does not become 'atomised', or in other words reductionist, non-generalisable and ad hoc. A similar view is taken by Hager (1994) who proposes an integrated approach to the assessment of competence which avoids the extremes of generic attributes and segmented tasks by assessing competence 'in terms of knowledge, abilities, skills and attitudes displayed in the context of a carefully chosen set of realistic professional tasks (intentional actions) which are of an appropriate level of generality'. That is, specific tasks are embedded within more general attributes. This is similar to the embedding of levels proposed for standards based assessment by Sadler (1986).

Standards based assessment (Sadler, 1987; Withers & Batten, 1990) is a general class of assessment methods to which competency based assessment belongs as a special case. The more general situation involves several standards along a continuum. As Hager (1994) affirms: 'Typically the standards relate to tasks which admit of many degrees of performance, as does the task of taking a traditional examination. In both cases, the existence of a minimum satisfactory level of performance is consistent with a full range of performances from excellent through to fail'. There may, however, be situations where the minimum satisfactory level of performance is so high that there is essentially no space for further differentiation above that level. Peddie (1997) characterises such situations as having no 'ceiling'. Alternatively, the 'competent' standard can be thought of as an adjustable 'floor'. If the floor is raised high enough, there is little height between the floor and the ceiling for differentiation. The image is an evocative one. The higher the ceiling above the 'competent' standard the more differentiation is possible.[1]

Competency based assessment is really a special case of standards based assessment where the proficiency continuum has been reduced to a simple dichotomy, 'competent' versus 'not competent', with only one 'standard', that of the competence threshold. Other standards could be specified above the competence threshold in those cases where there is sufficient height between the 'floor' and the 'ceiling', without undermining the fundamental focus on 'competence'.[2]

The impediment to differentiating levels of proficiency beyond basic competence is not theoretical. Various analyses, including those of Peddie (1997) and Wilmut and Macintosh (1997), support the view that it is possible to do so without compromising the overall intention of competency based assessment. However, the difficulty is in the execution, especially finding and implementing suitable assessment methods which are consistent with competency based assessment. Peddie (1997) suggests that 'there are no simple universal solutions'. Wilmut and Macintosh (1997) conclude that there is no 'wholly satisfactory' solution but that, as in all such policy decisions, 'any future action will have to be based on weighing the balance of advantages and disadvantages'.

OPERATING WITHIN THE PRESENT COMPETENCY BASED ASSESSMENT PROCEDURES

The most extreme view is that there ought not to be any change anyway. It is claimed by some that any change is likely to compromise the integrity of the system of competency based training on which vocational education and training has embarked. Competencies have been negotiated with employers and industry groups and are tied to particular jobs and wage levels. Any modifications might throw these arrangements into confusion and threaten the structure of training and employment agreements. It is claimed that these agreements should not be thrown in jeopardy by modifications which are peripheral to the main purpose of technical education. The main purpose of vocational education and training is ostensibly delivery of appropriate work skills not preparation of students for competitive entry to university courses.

Such an argument has some force. Competency based training offers a coherent philosophy which attaches particular levels of competence to particular qualifications. One advantage this offers is that qualifications can be interpreted as implying possession of the defined competencies. There is no mechanism within these agreements for recognition of some variation of these competencies nor for any discrimination among students in terms of differential standards of performance.

However, this 'maintain the faith' approach might have some unexpected consequences. In weighing their options, many students take into consideration the opportunities presented to improve their tertiary prospects. Some Vocational Employment Training (VET) students will find it desirable to take special tests such as the Special Tertiary Selection Test (STAT) to improve their selection chances; coaching schools may do well out of this. Other students may find it preferable to return to school or to undertake external Senior studies rather than to study in VET, reversing current trends. Others again may find the Open Learning courses attractive especially where opportunities for tutorial assistance are available through so-called 'Year 13' arrangements. It can be expected that better students will prefer to find other pathways through the tertiary system, leaving VET courses with only the weakest students. This is not to say that all students choose VET diplomas for their potential to provide a pathway into university studies. But many students wish to preserve their options. We can be certain that students will be quick to choose pathways which give themselves greatest potential advantage and to avoid those which do not.

Another factor should be noted. Not all employers are satisfied with competency based awards. In fact, there is considerable disquiet in some quarters, particularly in small business, where the bulk of employment is to be found. The essence of the problem is that results which provide no discrimination among students provide no information for employers to use in selecting among job applicants. If all applicants were identically qualified, then random choice would be sufficient. But employers do not accept that all students with the same qualifications are identically capable. Yet, to choose among applicants whose course results are identical requires recourse to other information, such as earlier academic performance or additional qualifications. Such information is less valid for assessing current proficiency than recent course-related information.

The suggestion has been made that the problem of differentiating among applicants with a VET diploma might be made by ballot. Although some nations (for example, Germany) make use of ballots in their selection procedures, Australians have never viewed such procedures favourably. The national interest in gambling does not extend to tertiary selection. Presumably, people can see the inequity. Peddie (1997) provides a good explanation of why a ballot would be inequitable and unacceptable. A ballot is only satisfactory where the applicants are identical, or at least indistinguishable, in their capabilities. Clearly, applicants with VET diplomas do not perceive themselves to be 'all the same'.

Irrespective of whether proficiency beyond competence is assessed, a standard of 'competence' will still apply. Part of the problem with the present tertiary selection schedules is that they equate the new 'competent' standard with the previous 'pass' standard. It is claimed by some in the technical education sector that the new 'competent' standard is higher than the previous 'pass' standard. On the face of it this appears to be the case. Typically, the previous 'pass' standard was pitched at 50 per cent of the achievable marks. The new 'competent' standard requires competence on all of the learning outcomes, not half of them. Further, competence on a learning outcome requires successful performance on all, or almost all, of a set of tasks indicative of that learning objective (or attribute) not half of them.

However, a proper comparison between the two systems requires comparison of the types of tasks set and their respective levels of complexity. It is facile to assume that 100 per cent on the present tasks represents a higher standard than 50 per cent on the previous tasks without demonstrating that the tasks are equivalent and that the assessment conditions are equivalent. It is likely, at least to some extent, that some of the more difficult tasks which previously enabled discrimination above the pass level are now not included. The proper comparison lies somewhere between the 50 per cent and 100 per cent levels of the previous system.

Rather than making comparisons of this kind, it would be better to re-analyse the relative merit of VET diplomas and other qualifications. This would require research on the demands placed on students obtaining a result of 'competent' on VET diplomas compared with the demands placed on students obtaining various results on other qualifications. For this to be consistent with the way in which other tertiary selection ranks are produced, this would need to be based on the generalised intellectual demands, perhaps through analysis of key competencies.

OPTIONS FOR FURTHER CONSIDERATION

Preparation of a portfolio

The portfolio is a rather raw, unprocessed form of information, involving little interpretation, speaking for itself. As such it is useful only when detailed attention is given to applicants and the selection process involves a personalised weighing of evidence. This happens in few cases in Queensland at present. At best, such information might be manageable for marginal decisions at the quota boundary but it is unlikely to be useable for the mass of decisions made in tertiary selection systems.

Various degrees of processing of the portfolio are possible. One possibility is the construction of a 'record of achievement' which provides 'a summary of the evidence in a portfolio, in the form of an organised set of descriptions or a profile of the student's achievements' (Wilmut & Macintosh, 1997). In addition, 'a summary report could also be generated, perhaps as an interpretation of what is in the portfolio, or to stand in its place. this would need to come from the teacher, perhaps in dialogue with the student' (Wilmut & Macintosh, 1997). This latter possibility approaches the notion of the 'principal's report', a detailed account of the characteristics of the student provided by the school principal acting on advice from the student's teachers. The principal's report has been explored as supplementary information in selection decisions especially at quota boundaries. Its validity, reliability and manageability are still unclear.

As Wilmut and Macintosh (1997) suggest, greater manageability is achievable through the use of standard formats and specified criteria. They also point out that this ensures that the criteria used in scrutinising the evidence are made public and accountable. The problem of comparability remains. Whether the simple uninterpreted portfolio, the record of achievement or the summary profile is presented, the question is how comparisons will be made across different applicants.

Standards based assessment of the whole course

Standards based assessment of a portfolio

Assessment of the portfolio is a logical extension of the previous discussion about portfolios. Wilmut and Macintosh (1997) see this assessment as involving generic criteria such as planning, information handling and evaluation, which are closely related to the student's capacity for autonomous learning, whereas Peddie (1997) suggests creativity, originality or flair. The criteria need to be clearly communicated to students in advance and embedded in the curriculum, that is, not tacked on without opportunity to acquire the necessary capabilities. Wilmut and Macintosh (1997) remind us that only a small number of 'levels' or 'standards' is feasible. Peddie (1997) warns that the number of students reaching each level is necessarily dependent only on their performance and not on any predetermined distribution, so that all students could potentially reach the top level or standard.

This last point is an important one. The standards cannot be adjusted after the fact. However, the standards ought to be set with some expectation of the likely distribution of capabilities. The risk of mistargeting can be minimised through careful analysis of the ways in which students typically differ in proficiency. Experience in setting and applying such standards in secondary schools suggests that this is not difficult to achieve and can be made consistent with course expectations.

Assessing the portfolio places an additional imposition on teachers beyond the usual expectations of the course but otherwise is minimally disruptive. Wilmut and Macintosh (1997) also point out that assessing the portfolio offers fewer comparability problems than other approaches because of the common criteria. However, there are always differences of interpretation unless supported by moderation procedures; and comparability across courses is still an issue if different criteria apply.

Standards based assessment of a special project

Another possibility is the use of a final assessable project. This is a task which requires the student to integrate and apply the various learning outcomes of the course. Therefore, it needs to be undertaken late in the course. Performance on this task does not affect overall competence in the course since competence is determined by performance on each of the learning outcomes. It offers a supplement to the overall competence of the student. Wilmut and Macintosh (1997) support the use of such an the integrating project or assignment and stress that it should 'reflect competencies gained through the course as a whole'. Peddie (1997) also suggests an additional assessment which involves integration and application, although he poses this in the form of a test.

Several issues concerning the assessable project need to be addressed, including:

• What proficiency scale is most appropriate? It has been suggested that honours, credit, pass might be sufficient. This provides very little differentiation among students but more levels might be difficult to manage with consistency across courses and motivations.

• How are the standards for the grade levels to be determined and expressed? The problem of moderation of these standards is a critical one because of the high stakes. Some form of standardisation and accountability would be necessary.

• Would this assessment be optional or compulsory? Students who do not intend to apply for other tertiary studies might not feel that they need it. On the other hand, they may change their mind later. It is safer to require it of all students. This does not prevent students from treating it lightly but this is less likely if it is clearly integrated into the course.

• Would this assessment also be available to part-time students? The difficulty is that some earlier aspects of a course may be less accessible after a lapse of time. This might also apply to full-time students but less severely. For example, in business courses, changes may have occurred in the type and availability of computer programs and spreadsheets. Skills may also need additional practice to restore to the peak level when they were initially assessed.

• Is it appropriate to attach so much importance to one single assessment? The categorisation into one of few grades has potentially serious consequences for the student. A 'pass' level presumably will have no effect on rank; performance at higher levels presumably will provide substantial advantage. Yet performance on a single task can be affected by contextual and temporal factors (such as the precise features of the task, recent facilitatory or inhibitory events, physiological well-being of the student).

• What kind of support would be provided for acquiring the skills necessary to undertake the project? It cannot be assumed that the necessary skills derive from the existing learning outcomes. These do not include the strategic skills necessary to plan, execute and report a comprehensive project which includes but extends beyond the course competencies.

• Is it appropriate to assess without prior practice of the necessary skills? This is an extension of the previous question. It is not good assessment policy to make the first attempt the final exam. Students need to be taught about and practiced on 'projects like this' before being required to demonstrate their acquired capability.

An alternative to the assessable project is the assessable module. This would provide an explicit framework of support for acquiring the necessary skills. The length of the module would depend on analysis of the strategic skills needed for a project of that type. A structured sequence of learning activities would develop and practice these skills. The final project needs to be similar to but different from those used for practice. Clear specifications should be given about the marking criteria.

This option addresses the final two questions of the previous option but leaves the remainder untouched. One possible advantage is that such a module could be separated from the course and made available to whichever students wished to avail themselves of the opportunity to upgrade their rank.

Standards based assessment of some or all units

Wilmut and Macintosh (1997) suggest that criteria for each proficiency level might be generic to the whole course or specific to each unit. Generic criteria '... could be spelled out in terms of some underlying attributes such as the quality of communication. On the other hand they could relate to some skills which are held in high regard, such as planning, organising information, analysis or evaluation'. The advantage of generic criteria would seem to be the commonality of standards across all units and the ease of interpretation this allows.

Unit specific criteria can be more closely aligned with the particular characteristics of the unit and more easily embedded in the context. On the other hand, less meaning can be attached to comparisons of proficiency levels on different units.

With both generic and specific criteria, rules of combination are needed for aggregation of the unit proficiency levels to produce course proficiency levels. This aggregation might be either numerical or judgemental.

Standards based assessment within some units

Just as for the course as a whole and the units as a whole, it is possible to assess some learning objectives at higher proficiency levels above the 'competent' threshold. For example, welds can be executed in a way which satisfies the minimum requirement for a satisfactory weld but variations in quality can still be observed among them, such as more efficient use of time or resources or more aesthetic appearance. Learning objectives lacking a 'ceiling' are not assessable in this way but many learning objectives do have a 'ceiling' and allow assessment of proficiency levels.

Learning objectives which can be used in this way need to be identified within the context of each course. Criteria can be generic or specific and need to be explicitly stated in advance. A profile of student proficiency is generated for each unit and the course as a whole. Methods for synthesising the elements of this profile to produce course proficiency levels need to be developed.

Assessment of desirable versus essential competencies

Another option which some VET teachers are enthusiastic about is to treat the currently defined competencies as essential competencies, that is, those necessary for satisfactory work performance at that level of qualification. Beyond that, it may be possible to define some desirable competencies, that is, competencies which would be beneficial but are not critical. This would require additional analysis of work situations and could not be accomplished quickly.

The attractiveness of this option is its consistency with the current approach to competency based assessment in VET. Assessment remains a matter of checking off the competencies acquired. Successful completion of the course would require all the essential competencies to be demonstrated. Grading beyond that would depend on the number of desirable competencies demonstrated. This has some similarities with the Type 6: Additional Outcomes suggestion by Peddie (1997).

Some issues need resolution:

• Who will define the desirable competencies? Such competencies are not part of the agenda of national competency standards. Considerable research and negotiation would be needed to bring them to fruition.

• Is it fair to grade on the number of competencies exhibited? Competency based assessment is supposed to allow students flexible time frames for completing each competency (although in practice this is severely constrained). Faster students may have more opportunity to cover more competencies in the time available.

• Might it be the case that all students could satisfactorily complete all competencies if given sufficient time? Unless the desirable competencies were sequenced according to increasing difficulty, there might be no differences among students. But increased difficulty can very easily lead to an overlap with the next competence level in the Australian Standards Framework (ASF).

CONCLUSION

REFERENCES

Gonczi, A. (1993). Integrated approaches to competency based assessment. Paper presented at the National Assessment Research Forum, Sydney, April 1993.

Gonczi, A. (1994). Competency based assessment in the professions in Australia. Assessment in Education, 1(1), 27-44.

Hager, P. (1994). Is there a cogent philosophical argument against competency standards? Australian Journal of Education, 38(1), 3-18.

Hager, P., Gonczi, A. & Athanasou, J. (1994). General issues about assessment of competence. Assessment and Evaluation in Higher Education, 19(1).

McGaw, B. (1993). Competency-based assessment: measurement issues. Paper presented at the National Assessment Research Forum, Sydney, April 1993.

Peddie, R. A. (1997). Some issues in using competency based assessment on selection decisions. Queensland Journal of Educational Research, 13(3), 17-46. http://education.curtin.edu.au/iier/qjer/qjer13/peddie.html

Sadler, D. R. (1986). General principles for organising criteria (Assessment Unit Discussion Paper 9). Brisbane: Board of (Senior) Secondary School Studies.

Sadler, D. R. (1987). Specifying and promulgating achievement standards. Oxford Review of Education, 13, 191-209.

Thomson, P. Mathers, R. & Quirk, R. (1996). The grade debate. Leabrook, South Australia: National Centre for Vocational Education Research.

Wilmut, J. & Macintosh, H. G. (1997). Possible options for differentiation in competency based assessment. Queensland Journal of Educational Research, 13(3), 47-71. http://education.curtin.edu.au/iier/qjer/qjer13/wilmut.html

Withers, G. & Batten, M. (1990). Defining types of assessment. In B. Low & G. Withers (Eds), Developments in school and public assessment. Melbourne: Australian Council for Educational Research.

Wood, R. & Power, C. (1987). Aspects of the competence-performance distinction: Educational, psychological and measurement issues. Journal of Curriculum Studies, 19(5), 409-424.

ACKNOWLEDGEMENT

ENDNOTES

1. Actually, the absence of a ceiling is one where the ceiling has been lowered to meet the floor, or perhaps where the floor has been raised to meet the ceiling, in a type of Edgar Allen Poe nightmare!

2. As Peddie (1997) suggests, in many cases there is probably no limit to the height of the ceiling.

Author details: Graham Maxwell is Senior Lecturer and Director of the Assessment and Evaluation Research Unit in the Graduate School of Education, The University of Queensland. Originally a secondary school teacher of mathematics and science, he has been involved for many years in teaching and research on assessment policies and practices as well as various assessment and evaluation projects and consultancies. Current research is concerned with selection for tertiary education and processes of teacher judgement of standards in performance assessments. Please cite as: Maxwell, G. S. (1997). Future directions for competency based assessment. Queensland Journal of Educational Research, 13(3), 71-84. http://education.curtin.edu.au/iier/qjer/qjer13/maxwell2.html