|AJET 27||AJET Home||AJET
ERA 2012 Journal and Conference ListsSome detail about what is meant by "refined the journal quality indicator" appeared in the accompanying press releases from the Minister  and the ARC . The Minister's media release contains about 1018 words in its running text, and one has to read to about word number 816 to get the key point: "I have made the decision to remove the rankings". So, "refined" means "removed"? However, the preceding text is important reading because it contains evidence of important advances in the ARC's understanding of matters, especially the recognition "that ERA could work perfectly well without the rankings" . Earlier this year the ARC appeared to hold firmly the view that "The ranked journal and ranked conference lists form an integral part of the ERA evaluation process" . So, within a few months, "integral part" has become "work perfectly well without". Now, how and why did that happen? An intriguing research question that may be forever unanswered.
The ARC has refined the journal quality indicator for ERA 2012. Profiles of A*, A, B and C ranks will not be used...
... As a result of the above changes, the ARC will not be awarding contracts for ATM 66 - review and provision of recommendations for the ERA 2012 ranked outlets. 
The ARC's media release  was briefer, about 473 words, and less informative. It states that the ARC "will use a refined journal quality indicator for ERA 2012", without giving any tangible information about how its next attempt to refine will be better than its previous attempts. The media release reiterates the ARC's view that "journal quality is an important indicator of research quality", without referring to the core problem, the goodness (or poorness) of correlation between "journal quality" and the "research quality" of individual articles that a journal has published.
Media outlets very quickly characterised the Australian research community's general reaction to the Ministerial and ARC press releases:
Kim Carr bows to rank rebellion over journal rankingsThe Ministerial media release  contains several discordant passages that could be regarded by some as 'blame shifting'. Firstly, there is reference to the previous government and to the bibliometric advice that the ARC commissioned:
Relief and a sense of vindication over its stand against journal rankings has characterised the research community's reaction to Innovation Minister Kim Carr's surprise dumping of the measure that made the government's research evaluation process so controversial. 
Journal rankings abolished - and unis couldn't be happier
After months of complaints, Kim Carr has axed the controversial ERA journal rankings. Universities have welcomed an unexpected announcement by Research Minister Kim Carr to abolish journal rankings... 
Protests force research policy change
Widespread complaints about how Australian academic research is assessed has led to the scrapping of the most controversial aspect of the new assessment scheme: ranking academic journals. 
Troubled history of an ERA
In what must be an embarrassing backdown for the Australian Research Council and the Minister for Science and Research, Senator Kim Carr, one element of the controversial Excellence in Research for Australia assessment scheme has been scrapped: the scheme's method of ranking academic journals which had been roundly criticised from the moment anyone started to take notice of it. 
... the rankings themselves were inherited from the discontinued Research Quality Framework (RQF) process of the previous government, and were developed on the basis of expert bibliometric advice. Secondly, "institutional research managers" seem to be implicated for "ill-informed, undesirable behaviour in the management of research":
There is clear and consistent evidence that the rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings. One common example was the setting of targets for publication in A and A* journals by institutional research managers. So, the Tiers ranking was fine, but "ill-informed, undesirable behaviour in the management of research" has 'forced' the Minister to make "the decision to remove the rankings, based on the ARC's expert advice." Not a good 'rally the troops' call!
The Ministerial media release  promises "The introduction of a journal quality profile, showing the most frequently published journals for each unit of evaluation". At first guess it seemed that "most frequently published" should read "most frequently cited", leading to the speculation that "journal quality profile" will utilise the Thomson Reuters Impact Factor , or one of its competitors such as Elsevier's SNIP and SJR . A partial clarification appeared on about 3 June 2011 in the Frequently Asked Questions page :
What is the change to the journal indicator?It seems to be a partial clarification, as it contains a possibly ominous sentence, 'Is it a highly regarded journal?' We will have to wait and see the definitions, if any ever appear, for 'highly regarded journal', but we do note that the ARC is seeking tenders closing 16 June 2011 for "ATM 75 Provision of bibliometric information for ERA 2012" .
During the ERA 2012 evaluation, REC [Research Evaluation Committee] members will be provided with the new journal indicator, which will replace the ranked journal indicator used in the ERA 2010 evaluation. The new indicator will display, for each unit of evaluation, a table listing the journals in which the articles submitted to that UoE [Unit of Evaluation] are published.
The table will inform expert judgements regarding the relevance of the journals to the research being published e.g. 'Is this an appropriate journal for this research?'. 'Is it a highly regarded journal?' This will allow RECs to take into account any regional or applied focus of research in a UoE.
As before, REC members will be able to drill down to article level data from the table, so will not be making their judgments solely on the basis of journal titles or article counts. The table will not include information about the quality of journals... The new journal indicator will not use prescriptive A*/A/B/C ranks. 
ATM 75 is dated 12 May 2011 and clearly was prepared and placed into distribution before the "the decision to remove the rankings". ATM is accompanied by 'Addendum 2 June 2011.pdf'  and its purpose was to update ATM 75 by removing various references to "discipline-specific tiered outlet rankings", and point out in several places that "Outlets will not be ranked in ERA 2012". Nevertheless, one reference to ranked outlets remained in clause 3.3.4:
Journal article and conference publication coverage will be assessed on the ERA 2010 ranked outlets lists available from http://www.arc.gov.au/era/key_docs10.htm However, the revised version of http://www.arc.gov.au/era/key_docs10.htm has an important new proviso, "Please note that material on this page is strictly related to the ERA 2010 process and is not relevant for the ERA 2012 process." The documents suggest that the critical decision meeting to "to remove the rankings" was on some date between about 13 May and 29 May 2011. Thus in a '5 Ws'  approach, we can estimate a when, though there's little prospect of progress with the other Ws. Most notably, on the why, a rather wide gap is obvious between the ARC-Ministerial perspective and the media commentary quoted above. To illustrate further the why from the ARC's perspective, the current Frequently Asked Questions page  states:
Why are ranks no longer being used?So "work perfectly well without" in the Ministerial media release  has become "did not play a crucial part" and "journal quality to remain an indicator". What will be the next morphing? Note again a somewhat supercilious implication that critics do not understand: "Those most engaged with ERA understand this ...".
The change enables journal quality to remain an indicator for ERA 2012, while discouraging the use of assessments of journal quality beyond their role as an ERA indicator. It will ensure that the indicator is kept in proper perspective, while maintaining ERA's rigour and focus on quality.
An essential feature of ERA is and has always been that it uses a range of indicators - not just journal rankings. Those most engaged with ERA understand this, however, the focus on journal rankings has led to some undesirable consequences. Experience from the ERA 2010 evaluation meeting also suggested that, although journal quality is an important indicator, the ranks themselves did not play a crucial part in the evaluation process. 
However, the lack of information about what really happened, and lack of detail about the Tiers replacement, the new "journal quality profile", does have a brighter side from a journalism perspective. If there are few facts, creative writing may be unleashed, as in the witty, satirical speculation by Joseph Gora :
Inside the ERA bunkerWhere to now? No doubt many editors of journals will make cautious adjustments to their 'about this journal' pages. Cautious, because there will be uncertainty about this new unknown, "journal quality profile", and because the ARC has not 'un-published' the Tiers 2010 list , in contrast to its swift removal of the Tiers 2008 list. In AJET's case, we will work upon a sentence that states (in an academic way), that AJET was A in Tiers 2008, was demoted to B in Tiers 2010, and now is free from that artificial, arbitrary and counter-productive shackle.
WIKIFREAKS has obtained a record of the following exchange that took place deep in the Australian Research Council bunker on the northern front of the Battle of the Journals' Ranking System. 
In relation to Editorial advice to readers, authors and Society members, again a cautious line seems appropriate, because some previous predictions didn't happen. In Editorial 27(1) we speculated that:
The ARC may intend to ease the pressure a little by softening its stance on the rigidly normative nature of Tiers (5% A*, 15% A, 30% B, 50% C).... We note that references to 5% A*... seem to have disappeared from the ERA website, or have become deeply buried. Could we be on the verge of something less severely normative, e.g. 10% A*, 20% A, 35% B, 35% C? So, a wee bit off the mark there, normative is apparently abandoned, but we will repeat another RA prediction, made in Editorial 27(2):
The next big emerging topic for research into research management and leadership policies may be the topic of tensions between the "rewarding of excellence", and the "rewarding of research that aligns well with academic, community and Government priorities". However, as 'refinement' is in vogue, let's refine "rewarding of research that aligns well with academic, community and Government priorities" into rewarding of excellence that aligns well with academic, community and Government research priorities. One could expect that Government (some would add, especially a Labor Government) will be happier with that direction, compared with the demonstrably narrow direction fed to it by the current leadership of the ARC. The ARC seems to have fallen into the trap of becoming a petty bureacratic, controlling agency, losing any scope for being an influential and visionary contributor to academic research leadership in Australia and beyond.
Roger Atkinson and Catherine McLoughlin
AJET Production Editor and AJET Editor
in AJET 27(3)
Hobart, 4-7 December 2011. http://www.leishman-associates.com.au/ascilite2011/
Higher education on the edge
Gold Coast, Queensland
4-7 July 2011
|Moodlemoot AU 2011|
Sydney Convention and
Exhibition Centre, 17-20 July
Nusa Dua, Bali, Indonesia, 2-5 October 2011. http://www.ut.ac.id/icde2011/
|ePortfolios Australia Conference 2011 (EAC2011)|
Curtin University, Perth, 17-18 October 2011
Adelaide, 24-25 November 2011. http://www.adelaide.edu.au/clpd/aall2011/
Singapore, 6-9 December 2011. http://www.cdtl.nus.edu.sg/tlhe/
The Australasian Journal of Educational Technology (AJET) is a refereed research journal published 7 times per year by the Australasian Society for Computers in Learning in Tertiary Education (ascilite). AJET retired its printed version (ISSN 1449-3098) at the end of Volume 23, 2007, and from Volume 24, 2008, the journal is open access, online only (ISSN 1449-5554), and does not have paid subscriptions.
© 2011 Authors retain copyright in their individual articles, whilst copyright in AJET as a compilation is retained by the publisher. Except for authors reproducing their own articles, no part of this journal may be reprinted or reproduced without permission. For further details, and for details on submission of manuscripts and open access to all issues of AJET published since the journal's foundation in 1985, please see http://www.ascilite.org.au/ajet/
For editorial inquiries, contact the Editor, Associate Professor Catherine McLoughlin, School of Education (ACT), Australian Catholic University, PO Box 256, Dickson ACT 2602, Australia. Email: Catherine.McLoughlin@acu.edu.au, Tel: +61 2 6209 1100 Fax +61 2 6209 1185.
For review process, production and business matters, contact the Production Editor and Business Manager, Dr Roger Atkinson, 5/202 Coode Street, Como WA 6152, Australia. Email: email@example.com, Tel: +61 8 9367 1133. Desktop publishing (PDF versions) and HTML by Roger Atkinson.
|AJET 27||AJET Home||AJET