Editorial 20(3): ARC to revise the ranked journal lists

Atkinson, R.; McBeath, C.

Issues In Educational Research, Vol 20(3), 2010
[ Contents Vol 20 ] [ IIER Home ]

Editorial 20(3)

ARC to revise the ranked journal lists

Our last IIER Editorial commentary upon journal rankings was in Editorial 17(2) in 2007 (McBeath & Atkinson, 2007). At the time, the rankings exercise cited in the Editorial was to serve the Australian Government's Research Quality Framework ('RQF'). The Editorial recorded our feeling that "...IIER has good prospects within the 'Comprehensive' category". With the change of government in 2007, the RQF died (Atkinson, 2008), but from the ashes arose Excellence in Research for Australia ('ERA'), relocated to another Australian Government agency, the Australian Research Council (ARC), which gave new life to the proponents for journal rankings. The ERA process delivered a Ranked Journal List (ARC, 2010a; Lamp, 2010) in which IIER seemed to fare reasonably, being given a Tier B ranking in a normative scale (Tier A*, top 5%; Tier A, next 15%; Tier B, next 30%; Tier C, bottom 50%).

IIER's Tier B ranking survived unchanged during the ARC's revisions from its draft list in 2008 to its final list in 2010, although a number of Australian educational research journals suffered demotions, for example Higher Education Research and Development, Australian Educational Researcher, Australian Journal of Education, Australasian Journal of Educational Technology and Australian Educational Computing (Atkinson, 2010a; Atkinson, 2010b).

However, the Tier ranking for IIER (and all 20,712 journals in the current list) is now under a threat created by the ARC's recent announcement (ARC, 2010b) of a "Review of the ERA 2010 Ranked Outlet Lists":

The ranked journal and ranked conference lists form an integral part of the ERA evaluation process.
In preparation for the ERA 2012 round, the ARC will revise the ERA 2010 ranked journal and conference lists. This process will involve a public consultation period, followed by a review and finalisation phase supported by peak bodies and other academic groups. (ARC, 2010b)

Our perception of a threat arises from two aspects. Firstly, the first sentence of the ARC's announcement, phrase "...lists form an integral part..." seems to reiterate its determination to continue the Tiers ranking system, notwithstanding the criticism it has received. Secondly, the ARC's schema is rigidly normative (5% A*, 15% A, etc) and thus in order to promote one journal to a higher Tier, another journal has to be demoted to a lower Tier. Whilst Tier A* and Tier A journals are likely to be under a greater threat of a competitive demotion compared with Tier B journals, it would be prudent for IIER to be represented vigorously in the forthcoming public consultation period promised for "early 2011".

The public consultation period and its follow up in "a review and finalisation phase supported by peak bodies and other academic groups" is likely to become, as one well-known senior academic suggests, a stoush:

It's evident from recent coverage, that Australian academe is winding itself up for another stoush about journal rankings for the Excellence in Research for Australia exercise.
... It doesn't make sense to stick with the existing rankings as we move towards a second ERA in 2012. But can we afford to go through all that aggravation again? (Goodyear, 2010)

The word stoush is no doubt well-chosen, though as the Macquarie Dictionary indicates, it is quite a forceful word:

stoush Colloquial - noun 1. a fight 2. physical aggression; violence: the group were dealing out stoush to the terrified customers. 3. an artillery bombardment

Recent criticism about journal rankings has appeared in The Australian and other publications. Numerous illustrative snippets may be found readily, for example 'clique bodies' (Atkinson, 2010c); 'Ditching journal rankings and using a direct comparison of citations...' (Smith, 2010); '...the mysterious last-minute fiddling with rankings' (Goodyear, 2010); '...ranking of some Australian journals was improved if the editors and interested parties lobbied...' (Levey, 2010); '...the draft ranking was overturned by a process shrouded in mystery' (Bourke, 2010); 'Transparency should be a top priority when the ARC reviews the controversial journal ranking component of its research assessment process' (Rowbotham, 2010); 'Tier Review Process' (Atkinson, 2010d). We hasten to add that the purpose for 'snippets' or brief phrases is to tempt you to read widely on this topic, rather than fully inform you in a direct way! Academic journals and conference proceedings are now beginning to publish research probes into the ARC's use of journal rankings and its impact. Some examples are listed by Atkinson (2010c, 2010d), and this issue of IIER contains a fine contribution by Brian Hemmings and Russell Kay (Hemmings & Kay, 2010).

Criticism of the ARC's journal rankings is linked in many instances to perceptions of deficiencies in the assignment of journals to Fields of Research ('FoR') (ARC, 2010c), for example as indicated by Gough (2008), Graham (2008) and Sid Bourke (a former editor of IIER) (Bourke, 2010). In IIER's case, the FoR assignment is '13. Education', an FoR containing 130 ranked journals comprising 9 A* (6.9%), 16 A (12.3%), 42 B (32.3%) and 63 C (48.5%) (plus 1 unranked). From a selfish perspective, a journal in '13. Education' (for example) may seek to help retain or advance its Tier ranking by lobbying for re-assignments of higher ranked journals into other FoRs (e.g. 1301 Education Systems; 1302 Curriculum and Pedagogy; 1303 Specialist Studies in Education), and for re-assignments of lower ranked journals from other FoRs into '13. Education'.

Another angle for lobbyists will be to look for alternative rankings done by researchers other than the ARC's unknown rankers who did the '...mysterious last-minute fiddling with rankings' (Goodyear, 2010) or were drawn from 'clique bodies' (Atkinson, 2010c). For example, Australasian Journal of Educational Technology has published an alternative ranking for educational technology journals, using the Thomson Reuter's Impact Factor as a basis, and (naturally enough) it is favourable to AJET's case for promotion (Atkinson, 2010e). Genoni, Haddow and Dumbell (2009) provide a more extensive research study of selected social sciences and humanities journals.

We express the hope that the forthcoming "stoush" will be scholarly and evidence based, and that it will not degenerate into "... a 'divide and conquer' regime, as societies and other interested parties may seek to both promote 'their' journal, and belittle some of the higher ranked journals that 'their' journal could displace" (Atkinson, 2010d). We also wish to record with gratitude the support that Tier B ranked IIER continues to receive from authors and reviewers. We dare to hope that mostly or even in sizable majority they are not allowing The tyranny of Tiers to influence them unduly in their support for this journal.

Roger Atkinson and Clare McBeath
IIER Website Editor and IIER Editor

References

ARC (Australian Research Council) (2010a). Review of the ERA 2010 Ranked Outlet Lists. Announcement dated 1 November 2010. http://www.arc.gov.au/era/era_2012/review_of_era10_ranked_outlet_lists.htm (the same announcement appears also in http://www.arc.gov.au/pdf/ERAnews_Oct10.pdf)

ARC (2010b). Ranked Journal List. http://www.arc.gov.au/xls/ERA2010_journal_title_list.xls (Excel format, 5.27 MB, apparently dated 9 February 2010). For those who find the ARC's file too cumbersome and slow to handle on their personal computers, we recommend highly John Lamp's service (Lamp, 2010).

ARC (2009c). Australian and New Zealand Standard Research Classification (ANZSRC). (PDF, 4.5 MB). http://www.arc.gov.au/era/ANZSRC.htm (again, we recommend highly John Lamp's service).

Atkinson, R. (2008). AJET Editorial 24(2): IM 26: Obituaries for the RQF. http://www.ascilite.org.au/ajet/ajet24/editorial24-2.html

Atkinson, R. (2010a). AJET Editorial 26(1). Journal rankings: AJET demoted. http://www.ascilite.org.au/ajet/ajet26/editorial26-1.html

Atkinson, R. J. (2010b). Bibliometrics out, journalmetrics in! HERDSA News, 32(1). http://www.roger-atkinson.id.au/pubs/herdsa-news/32-1.html

Atkinson, R. (2010c). AJET Editorial 26(6). A Letter to the Editor: Defending the ERA initiative. http://www.ascilite.org.au/ajet/ajet26/editorial26-6.html

Atkinson, R. J. (2010d). ARC announces a Tier Review Process. HERDSA News, 32(3). http://www.roger-atkinson.id.au/pubs/herdsa-news/32-3.html

Atkinson, R. (2010e). AJET Editorial 26(5). Idle Moment 40: Impact Factor revisited. http://www.ascilite.org.au/ajet/ajet26/editorial26-5.html

Bourke, S. (2010). Rankings still rankle. In Letters to the Editor, Higher Education, The Australian, 17 November. http://www.theaustralian.com.au/higher-education/letters/rankings-still-rankle/story-e6frgcox-1225954652506

Genoni, P., Haddow, G. & Dumbell, P. (2009). Assessing the impact of Australian journals in the social sciences and humanities. In Proceedings ALIA Information Online 2009, Sydney, 20-22 January. http://conferences.alia.org.au/online2009/docs/PresentationC16.pdf

Goodyear, P. (2010). Rankings rancour. In Letters to the Editor, Higher Education, The Australian, 24 November. http://www.theaustralian.com.au/higher-education/letters/rankings-rancour/story-e6frgcox-1225959639713

Gough, N. (2008). Educational research and the ERA. AARE News, No 63, July. http://www.aare.edu.au/news/newsplus/news63.pdf

Graham, L. J. (2008). Rank and file: Assessing research quality in Australia. Educational Philosophy and Theory, 40(7). [verified 6 Dec 2010] http://bulletin.edfac.usyd.edu.au/wp-content/uploads/2008/11/rank-and-file_era-2008_graham-doc.pdf

Hemmings, B. & Kay, R. (2010). Journal ratings and the publications of Australian academics. Issues in Educational Research, 20(3), 234-243. http://www.iier.org.au/iier20/hemmings.html

Lamp, J. (2010). ERA Current Rankings Access. http://lamp.infosys.deakin.edu.au/era/

Levey, G. B. (2010). Rankings rancour. In Letters to the Editor, Higher Education, The Australian, 24 November. http://www.theaustralian.com.au/higher-education/letters/rankings-rancour/story-e6frgcox-1225959639713

The Macquarie Dictionary (1997). 3rd ed. Sydney: The Macquarie Library.

McBeath, C. & Atkinson, R. (2007). Editorial 17(2). Issues In Educational Research, 17(2), v-vii. http://www.iier.org.au/iier17/editorial17-2.html

Rowbotham, J. (2010). Journal rankings rankle academics. Higher Education Section, The Australian, 10 November. http://www.theaustralian.com.au/higher-education/journal-rankings-rankle-academics/story-e6frgcjx-1225950277254

Smith, A. (2010). Matching input and output assessments. In Opinion & Analysis, Higher Education, The Australian, 10 November. http://www.theaustralian.com.au/higher-education/opinion-analysis/matching-input-and-output-assessments/story-e6frgcko-1225950268027

Please cite as: Atkinson, R. & McBeath, C. (2010). Editorial 20(3): ARC to revise the ranked journal lists. Issues In Educational Research, 20(3), ii-iv. http://www.iier.org.au/iier20/editorial20-3.html

[ PDF version of this article ] [ Contents Vol 20 ] [ IIER Home ]
© 2010 Issues In Educational Research. This URL: http://www.iier.org.au/iier20/editorial20-3.html
Created 9 Dec 2010. Last correction: 17 Dec 2010.
HTML: Roger Atkinson [rjatkinson@bigpond.com]