Editorial 27(3)
Roger Atkinson
Dawn of a new ERA?
For ERA watchers, the first public sign of a major 'problem' with the ARC's handling of its ERA 2012 Ranked Outlets Consultation was the failure to meet one of its own deadlines. Until about the end of April 2011, the web page for the Review of the ERA 2010 Ranked Outlet Lists [1] stated that "A list of the participating peak bodies and academic groups will be made publicly available on the ARC website in mid-April." Sometime around the end of April or early May, "mid-April" was changed to "end of May". At the end of May there was no "list of the participating peak bodies", instead, dated 30 May 2011, something more dramatic appeared:
ERA 2012 Journal and Conference Lists
The ARC has refined the journal quality indicator for ERA 2012. Profiles of A*, A, B and C ranks will not be used...
... As a result of the above changes, the ARC will not be awarding contracts for ATM 66 - review and provision of recommendations for the ERA 2012 ranked outlets. [2]
Some detail about what is meant by "refined the journal quality indicator" appeared in the accompanying press releases from the Minister [3] and the ARC [4]. The Minister's media release contains about 1018 words in its running text, and one has to read to about word number 816 to get the key point: "I have made the decision to remove the rankings". So, "refined" means "removed"? However, the preceding text is important reading because it contains evidence of important advances in the ARC's understanding of matters, especially the recognition "that ERA could work perfectly well without the rankings" [3]. Earlier this year the ARC appeared to hold firmly the view that "The ranked journal and ranked conference lists form an integral part of the ERA evaluation process" [5]. So, within a few months, "integral part" has become "work perfectly well without". Now, how and why did that happen? An intriguing research question that may be forever unanswered.
The ARC's media release [4] was briefer, about 473 words, and less informative. It states that the ARC "will use a refined journal quality indicator for ERA 2012", without giving any tangible information about how its next attempt to refine will be better than its previous attempts. The media release reiterates the ARC's view that "journal quality is an important indicator of research quality", without referring to the core problem, the goodness (or poorness) of correlation between "journal quality" and the "research quality" of individual articles that a journal has published.
Media outlets very quickly characterised the Australian research community's general reaction to the Ministerial and ARC press releases:
Kim Carr bows to rank rebellion over journal rankings
Relief and a sense of vindication over its stand against journal rankings has characterised the research community's reaction to Innovation Minister Kim Carr's surprise dumping of the measure that made the government's research evaluation process so controversial. [6]
Journal rankings abolished - and unis couldn't be happier
After months of complaints, Kim Carr has axed the controversial ERA journal rankings.
Universities have welcomed an unexpected announcement by Research Minister Kim Carr to abolish journal rankings... [7]
Protests force research policy change
Widespread complaints about how Australian academic research is assessed has led to the scrapping of the most controversial aspect of the new assessment scheme: ranking academic journals. [8]
Troubled history of an ERA
In what must be an embarrassing backdown for the Australian Research Council and the Minister for Science and Research, Senator Kim Carr, one element of the controversial Excellence in Research for Australia assessment scheme has been scrapped: the scheme's method of ranking academic journals which had been roundly criticised from the moment anyone started to take notice of it. [9]
The Ministerial media release [3] contains several discordant passages that could be regarded by some as 'blame shifting'. Firstly, there is reference to the previous government and to the bibliometric advice that the ARC commissioned:
... the rankings themselves were inherited from the discontinued Research Quality Framework (RQF) process of the previous government, and were developed on the basis of expert bibliometric advice. [3]
Secondly, "institutional research managers" seem to be implicated for "ill-informed, undesirable behaviour in the management of research":
There is clear and consistent evidence that the rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings. One common example was the setting of targets for publication in A and A* journals by institutional research managers. [3]
So, the Tiers ranking was fine, but "ill-informed, undesirable behaviour in the management of research" has 'forced' the Minister to make "the decision to remove the rankings, based on the ARC's expert advice." Not a good 'rally the troops' call!
The Ministerial media release [3] promises "The introduction of a journal quality profile, showing the most frequently published journals for each unit of evaluation". At first guess it seemed that "most frequently published" should read "most frequently cited", leading to the speculation that "journal quality profile" will utilise the Thomson Reuters Impact Factor [10], or one of its competitors such as Elsevier's SNIP and SJR [11]. A partial clarification appeared on about 3 June 2011 in the Frequently Asked Questions page [12]:
What is the change to the journal indicator?
During the ERA 2012 evaluation, REC [Research Evaluation Committee] members will be provided with the new journal indicator, which will replace the ranked journal indicator used in the ERA 2010 evaluation. The new indicator will display, for each unit of evaluation, a table listing the journals in which the articles submitted to that UoE [Unit of Evaluation] are published.
...
The table will inform expert judgements regarding the relevance of the journals to the research being published e.g. 'Is this an appropriate journal for this research?'. 'Is it a highly regarded journal?' This will allow RECs to take into account any regional or applied focus of research in a UoE.
As before, REC members will be able to drill down to article level data from the table, so will not be making their judgments solely on the basis of journal titles or article counts. The table will not include information about the quality of journals... The new journal indicator will not use prescriptive A*/A/B/C ranks. [12]
It seems to be a partial clarification, as it contains a possibly ominous sentence, 'Is it a highly regarded journal?' We will have to wait and see the definitions, if any ever appear, for 'highly regarded journal', but we do note that the ARC is seeking tenders closing 16 June 2011 for "ATM 75 Provision of bibliometric information for ERA 2012" [13].
ATM 75 is dated 12 May 2011 and clearly was prepared and placed into distribution before the "the decision to remove the rankings". ATM is accompanied by 'Addendum 2 June 2011.pdf' [12] and its purpose was to update ATM 75 by removing various references to "discipline-specific tiered outlet rankings", and point out in several places that "Outlets will not be ranked in ERA 2012". Nevertheless, one reference to ranked outlets remained in clause 3.3.4:
Journal article and conference publication coverage will be assessed on the ERA 2010 ranked outlets lists available from http://www.arc.gov.au/era/key_docs10.htm [12]
However, the revised version of http://www.arc.gov.au/era/key_docs10.htm has an important new proviso, "Please note that material on this page is strictly related to the ERA 2010 process and is not relevant for the ERA 2012 process." The documents suggest that the critical decision meeting to "to remove the rankings" was on some date between about 13 May and 29 May 2011. Thus in a '5 Ws' [13] approach, we can estimate a when, though there's little prospect of progress with the other Ws. Most notably, on the why, a rather wide gap is obvious between the ARC-Ministerial perspective and the media commentary quoted above. To illustrate further the why from the ARC's perspective, the current Frequently Asked Questions page [12] states:
Why are ranks no longer being used?
The change enables journal quality to remain an indicator for ERA 2012, while discouraging the use of assessments of journal quality beyond their role as an ERA indicator. It will ensure that the indicator is kept in proper perspective, while maintaining ERA's rigour and focus on quality.
An essential feature of ERA is and has always been that it uses a range of indicators - not just journal rankings. Those most engaged with ERA understand this, however, the focus on journal rankings has led to some undesirable consequences. Experience from the ERA 2010 evaluation meeting also suggested that, although journal quality is an important indicator, the ranks themselves did not play a crucial part in the evaluation process. [12]
So "work perfectly well without" in the Ministerial media release [3] has become "did not play a crucial part" and "journal quality to remain an indicator". What will be the next morphing? Note again a somewhat supercilious implication that critics do not understand: "Those most engaged with ERA understand this ...".
However, the lack of information about what really happened, and lack of detail about the Tiers replacement, the new "journal quality profile", does have a brighter side from a journalism perspective. If there are few facts, creative writing may be unleashed, as in the witty, satirical speculation by Joseph Gora [14]:
Inside the ERA bunker
WIKIFREAKS has obtained a record of the following exchange that took place deep in the Australian Research Council bunker on the northern front of the Battle of the Journals' Ranking System. [14]
Where to now? No doubt many editors of journals will make cautious adjustments to their 'about this journal' pages. Cautious, because there will be uncertainty about this new unknown, "journal quality profile", and because the ARC has not 'un-published' the Tiers 2010 list [15], in contrast to its swift removal of the Tiers 2008 list. In AJET's case, we will work upon a sentence that states (in an academic way), that AJET was A in Tiers 2008, was demoted to B in Tiers 2010, and now is free from that artificial, arbitrary and counter-productive shackle.
In relation to Editorial advice to readers, authors and Society members, again a cautious line seems appropriate, because some previous predictions didn't happen. In Editorial 27(1) we speculated that:
The ARC may intend to ease the pressure a little by softening its stance on the rigidly normative nature of Tiers (5% A*, 15% A, 30% B, 50% C).... We note that references to 5% A*... seem to have disappeared from the ERA website, or have become deeply buried. Could we be on the verge of something less severely normative, e.g. 10% A*, 20% A, 35% B, 35% C? [16]
So, a wee bit off the mark there, normative is apparently abandoned, but we will repeat another RA prediction, made in Editorial 27(2):
The next big emerging topic for research into research management and leadership policies may be the topic of tensions between the "rewarding of excellence", and the "rewarding of research that aligns well with academic, community and Government priorities". [17]
However, as 'refinement' is in vogue, let's refine "rewarding of research that aligns well with academic, community and Government priorities" into rewarding of excellence that aligns well with academic, community and Government research priorities. One could expect that Government (some would add, especially a Labor Government) will be happier with that direction, compared with the demonstrably narrow direction fed to it by the current leadership of the ARC. The ARC seems to have fallen into the trap of becoming a petty bureacratic, controlling agency, losing any scope for being an influential and visionary contributor to academic research leadership in Australia and beyond.
Roger Atkinson and Catherine McLoughlin
AJET Production Editor and AJET Editor
Endnotes
- ARC (2011). Review of the ERA 2010 Ranked Outlet Lists. http://www.arc.gov.au/era/era_2012/review_of_era10_ranked_outlet_lists.htm
- The document ATM 66 detailed tender requirements for those seeking to become one of the "participating peak bodies". This document has been 'unpublished', but if searching for a copy, the original filename was 'ERA_2012_ranked_outlets.pdf'.
- Carr, Senator the Hon Kim (2011). Improvements to Excellence in Research for Australia. Media release, 30 May. http://minister.innovation.gov.au/Carr/MediaReleases/Pages/IMPROVEMENTSTOEXCELLENCEIN
RESEARCHFORAUSTRALIA.aspx
- ARC (2011). Excellence in Research for Australia 2012. Media release, 30 May. http://www.arc.gov.au/media/releases/media_30May11.htm
- This sentence was the first in reference [1] until about 30 May 2011. Now it has been 'unpublished', however records will continue to exist elsewhere, my own being in Atkinson, R. J. (2011). Technology and control of feedback: My encounters with ROCI. HERDSA News, 33(1). http://www.roger-atkinson.id.au/pubs/herdsa-news/33-1.html
- Rowbotham, J. (2011). Kim Carr bows to rank rebellion over journal rankings. The Australian, 1 June. http://www.theaustralian.com.au/higher-education/kim-carr-bows-to-rank-rebellion/story-e6frgcjx-1226066727078
- Woodward, S. (2011). Journal rankings abolished - and unis couldn't be happier. Campus Review, 6 June. http://www.campusreview.com.au/pages/section/article.php?s=News&idArticle=21128
- Maslen, G. (2011). Protests force research policy change. University World News, 174, 5 June 2011. http://www.universityworldnews.com/article.php?story=20110603184401784
- Dobson, I. (2011). Troubled history of an ERA. University World News, 174, 5 June 2011
- Thomson Reuters (2011). The Thomson Reuters Impact Factor. http://thomsonreuters.com/products_services/science/free/essays/impact_factor/
- Elsevier B. V. (2010). SNIP & SJR: A new perspective in journal metrics. http://info.scopus.com/journalmetrics/
- ARC (2011). Frequently asked questions: ERA 2012. [viewed 11 Jun 2011] http://www.arc.gov.au/era/faq.htm
- ARC (2011). Tenders. ATM75 Provision of bibliometric information for ERA 2012. http://www.arc.gov.au/about_arc/tenders.htm. Filenames 'Frequently Asked Questions ATM 75 3 June 2011.pdf', 'Addendum 2 June 2011.pdf' and 'Approach to Market (ATM) 75 - Citation provider 2012.pdf'
- Wikipedia. Five Ws. http://en.wikipedia.org/wiki/Five_Ws
- Gora, J. (2011). Inside the ERA bunker. The Australian, 8 June.
- ARC (2010). http://www.arc.gov.au/xls/ERA2010_journal_title_list.xls (Excel format, 5.27 MB, apparently dated 9 February 2010). For those who find the ARC's file too cumbersome and slow to handle on their personal computers, we recommend Lamp, J. (2010). ERA Current Rankings Access. http://lamp.infosys.deakin.edu.au/era/. See also John Lamp's bibliography, Publications on ERA Outlet Ranking, http://lamp.infosys.deakin.edu.au/era/?page=pubs
- AJET Editorial 27(1). Review of the ERA 2010 Ranked Outlet Lists http://www.ascilite.org.au/ajet/ajet27/editorial27-1.html
- AJET Editorial 27(2). Educational sector representation in educational technology journals http://www.ascilite.org.au/ajet/ajet27/editorial27-2.html
Conferences
advertised
in AJET 27(3) |
Hobart, 4-7 December 2011. http://www.leishman-associates.com.au/ascilite2011/
|
| HERDSA 2011
Higher education on the edge
Gold Coast, Queensland
4-7 July 2011
http://conference.herdsa.org.au/2011/ |
|
 | Moodlemoot AU 2011
Sydney Convention and
Exhibition Centre, 17-20 July
http://moodlemoot.org.au/
|
 | Association for Learning Technology: ALT-C 2011
Thriving in a colder and more challenging climate
University of Leeds, UK, 6-8 September 2011
http://www.alt.ac.uk/altc2011/ |
|
|
Nusa Dua, Bali, Indonesia, 2-5 October 2011.
http://www.ut.ac.id/icde2011/
|
 | ePortfolios Australia Conference 2011 (EAC2011)
Curtin University, Perth, 17-18 October 2011
http://eportfoliosaustralia.word
press.com/conference-eac2011/
|
 | Fifth Joint Conference of APACALL and PacCALL
De La Salle University
Manila, Philippines, 27-29 October 2011
http://glocall.org/ |
|
Adelaide, 24-25 November 2011. http://www.adelaide.edu.au/clpd/aall2011/
|
Singapore, 6-9 December 2011. http://www.cdtl.nus.edu.sg/tlhe/ |
The Australasian Journal of Educational Technology (AJET) is a refereed research journal published 7 times per year by the Australasian Society for Computers in Learning in Tertiary Education (ascilite). AJET retired its printed version (ISSN 1449-3098) at the end of Volume 23, 2007, and from Volume 24, 2008, the journal is open access, online only (ISSN 1449-5554), and does not have paid subscriptions.
© 2011 Authors retain copyright in their individual articles, whilst copyright in AJET as a compilation is retained by the publisher. Except for authors reproducing their own articles, no part of this journal may be reprinted or reproduced without permission. For further details, and for details on submission of manuscripts and open access to all issues of AJET published since the journal's foundation in 1985, please see http://www.ascilite.org.au/ajet/
For editorial inquiries, contact the Editor, Associate Professor Catherine McLoughlin, School of Education (ACT), Australian Catholic University, PO Box 256, Dickson ACT 2602, Australia. Email: Catherine.McLoughlin@acu.edu.au, Tel: +61 2 6209 1100 Fax +61 2 6209 1185.
For review process, production and business matters, contact the Production Editor and Business Manager, Dr Roger Atkinson, 5/202 Coode Street, Como WA 6152, Australia. Email: rjatkinson@bigpond.com, Tel: +61 8 9367 1133. Desktop publishing (PDF versions) and HTML by Roger Atkinson.
HTML Editor: Roger Atkinson [
rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet27/editorial27-3.html
Created 12 Jun 2011. Last correction: 16 Jun 2011.
