|AJET 26||AJET Home||AJET
To begin with, there is a logical explanation for the adoption of a "literary heading" for this editorial, instead of an "educational technology research" heading. During 13-14 April 2010, ODLAA's email list for members  distributed some messages concerning recent advice from Thomson Reuters  that ODLAA's journal, Distance Education , had been accepted for indexing in Social Sciences Citation Index . The home page for Distance Education has a new line, "Now ISI listed!" , and in one email list posting the term "holy grail" appeared. I felt moved to point out to ODLAA members that:
The Thomson Reuters (was ISI) 'Impact Factor' used to be a holy grail, getting much influence, or far too much influence according to some of us, from its claim to select only the "best" journals for its citation analyses.A brief guide to the evidence has been published [6, 7], and my posting also cited one of my favourite criticisms of the Thomson Reuters 'Impact Factor', the article Impact factor wars: Episode V - The Empire Strikes Back . With the Holy Grail and The Empire Strikes Back in mind, the time seemed appropriate to explore some literary metaphors:
So the goalposts have shifted. Thus IMHO we need to change focus, move away from 'holy grails' and look instead for 'ERA dragon slayers'. The ERA's 2010 iteration seems to have downgraded Australian based journals relative to the American or UK/Europe based multinationals. We need to focus attention on research questions of the form, "What is the correlation between the real merit of a research work and the tier ranking of the journal in which the work was published?"; "What is the correlation between the real merit of a research work and its citation count according to Scopus?"; "What is the evidence base for the ERA's application of journalmetrics to research funding matters?", i.e. the potential "ERA dragon slayer" questions.Why explore literary metaphors? Well, we seem to have made little progress in applying educational research methods [6, 7] to gain an understanding of how the 2010 version of the ARC/ERA Tier rankings [9, 10] was derived. Perhaps we will have better luck with literary methods? Though, upon reflection, we recognised that "ERA dragon slayers" lacked an interpretative focus, hence the adoption here of a very recognisable literary phrase, "Beauty is in the eye of the beholder" . This has an interpretative focus, and as indicated by one source , can be linked to Shakespeare's Love's Labours Lost  (it's always safe to quote Shakespeare!). Taking some literary liberties and making a small adaptation:
Good Lord ARC, my beauty, though but mean,Will "Beauty is in the eye of the beholder" be a productive interpretative framework for the ARC/ERA Tier rankings? The answers will take some time to develop, because there are quite a large number of "beholders", even though the ranking of scholarly journals is a very specialised field with a relatively small number of researchers. Anyway, to "Begin at the beginning..." [13, 14], let us start with Elbeck and Mandernach (2009) . Other "beholders" will have to wait until future editorials!
Needs not the painted flourish of your praise:
Beauty is bought by judgement of the eye,
Not utter'd by base sale of bureaucratic tiers
[after W. Shakespeare, 12]
Elbeck and Mandernach (2009)  examined 46 scholarly journals pertaining to computer mediated learning, using metrics for popularity, importance, prestige, and overall rankings for each journal. First the good news, from AJET's perspective. Their Table 5: Overall Ranking of Scholarly CML Journals (N = 46) places AJET well ahead of all of the five Tier A educational technology journals listed in AJET Editorial 26(1), and also ahead of the Tier B journals therein. Proponents of the Tiers ranking could say that Elbeck & Mandernach's (2009) methodology is suspect, to which Tiers antagonists can retort that if you cannot publish your own methodology, you are not eligible to criticise a competing methodology.
Now for the "howevers". Elbeck & Mandernach's (2009) 'top five' (in Table 5) is rather perplexing, as one journal has ceased (Innovate ), one has changed recently from open access to closed (JALN ), one is closed access (AJDE ), and one (eLearning Papers ) was not on the AJET Editors' list of 'journals to watch'  (now added, though noting that eLearning Papers is not really international, being very 'Eurocentric'). In Elbeck and Mandernach's (2009)  'top five', the highest Tier ranking is B for IRRODL , with the other four being C ranked, except that eLearning Papers  is not Tiers listed.
To probe a bit deeper, the poor ranking in Elbeck and Mandernach (2009)  for the Tiers 'top five' namely BJET, C&E, ALT-J, JCAL and ETR&D (all Tier A, see  for references) is also perplexing. Elbeck and Mandernach's (2009) Table 5 ranks these journals in the range 31 to 45 in a field of 46, compared with AJET ranked 10. Considering the individual metrics, BJET, ALT-J and ETR&D fared poorly in popularity (Elbeck & Mandernach, 2009: Table 1), whilst C&E, ALT-J and JCAL fared poorly in prestige (Elbeck & Mandernach, 2009: Table 4). We need to note that the prestige metric may suffer from low reliability owing to the relatively small sample, N=23 (useable response rate 50%). However, the most important caveat concerns the use of Google PageRank data (Elbeck & Mandernach, 2009: Table 1). Their "zero" values of PageRank for BJET, C&E and ETR&D seem rather odd.
So we need to examine Google PageRank  more closely. These days website managers and researchers into these matters obtain PageRank data via a third party service, for example Elbeck and Mandernach (2009) used Top25Web . They did not report upon PageRank data obtained from other, similar services, for example Page Rank Checker , CheckPageRank.Net , Free Page Rank Checker , PageRankTool.net , and Check SEO . The problem is that the services cannot be relied upon to deliver consistent results free from spurious values, such as "zero" values of PageRank for BJET, C&E and ETR&D. Table 1 below illustrates this lack of consistency, and also indicates that investigators may need to check a number of URLs for a journal, and not just the current home page. Also for investigators, please resist the temptation to use only the instrument that gives results closest to the "desired result"!
|Top25Web  (b)||0||0||0||0||0|
|Page Rank Checker ||6||7||4(e)||6||7|
|CheckPageRank.Net  (c)||8||8||7||8||6|
|Free Page Rank Checker ||6||7||nr(d)||nr(d)||7|
|Check SEO  (b)||0||0||0||0||0|
Table 2 records a preliminary check on the popularity metric as defined by Elbeck and Mandernach (2009), using just one service, Check SEO . In this case, as indicated in Table 2, it is possible to compare the third party service with a Google search command. No discrepancies were found, but of course this does not indicate anything conclusive about the utility of Link popularity relative to PageRank. Table 2 is a work in progress and to date it indicates only that there is some degree of stability in rankings over a period of 15 months. There are potentially interesting suggestions of clues to be examined further, for example is JALN going into decline as a result of its change from open access top closed access; why is eLearning Papers sustaining a high level; and what is the reason for JCAL's marked movement?
|Journal (a)||Elbeck and Mandernach|
(28 Jan 2009) (b)
|AJET Editorial 26(2)|
(18 Apr 2010) (c)
|eLearning Papers ||14.83%||868|
With a little reluctance, we have to recognise that as "beholders", Elbeck and Mandernach's (2009) representation of "beauty" is perhaps a little flawed. There are problems with methodology and in the details of their methods. We are reluctant to be anything more than mildly critical, because AJET fared very well according to their metrics, and as suggested earlier, any presentation of methodologies for the ranking of scholarly journals has to be better than what amounts to apparently no presentation at all, as seems to be the case with the ARC's current Tiers.
In their concluding sentence, Elbeck and Mandernach (2009) seemed to look forward to:
... a definitive list and ranking of journals that will doubtlessly help faculty and administrators judge the relative value of publications for promotion and tenure purposes. Is that dangerous territory? Given that it's not hard to demonstrate that "Beauty is in the eye of the beholder", is the idea of "a definitive list and ranking of journals" some kind of medieval Holy Grail? What some of us are really trying to do is educate "faculty and administrators" about the counter-productive aspects (or the risks, or even the folly) of excessive reliance on "journalmetrics".
Let's conclude with a "beautiful" view from one more "beholder", David Jones  from Central Queensland University. We thank David warmly for his very kind words, and hope that his recommendation, written in 2009 when AJET was a Tier A journal, will not change as a result of AJET's demotion to Tier B in the 2010 list :
For my immediate purposes, it looks like AJET is a good fit. A journal that is open access. Also looking on the bright side, "Beauty is in the eye of the beholder" and the ranking of journals create a very sustainable source for AJET Editorials. At a consumption rate of two "beholders" per Editorial (or three if you count ARC/ERA's Tiers), the supply of Editorial material in this genre is assured for many years!
Roger Atkinson and Catherine McLoughlin
AJET Production Editor and AJET Editor
in AJET 26(2)
Sydney, 5-8 December 2010. Website http://www.ascilite.org.au/conferences/sydney10/
|Quality Connections - Boundless Possibilities:|
Through Open, Flexible and Distance Learning
25-28 April 2010, Wellington
Distance Education Association of New Zealand
|Global Learn Asia Pacific 2010|
17-20 May 2010, Penang, Malaysia
Association for the Advancement
of Computing in Education
|Reshaping Higher Education|
Melbourne, 6-9 July
MoodleMoot AU 2010, Melbourne, 11-14 July. http://moodlemoot.org.au/
20-23 Sept 2010
for Information Processing
Forum on Open Learning
24-28 November 2010, Kochi, India
Commonwealth of Learning and
Indira Ghandi National Open University
The Australasian Journal of Educational Technology (AJET) is a refereed research journal published 6 times per year by the Australasian Society for Computers in Learning in Tertiary Education (ascilite). AJET retired its printed version (ISSN 1449-3098) at the end of Volume 23, 2007, and from Volume 24, 2008, the journal is open access, online only (ISSN 1449-5554), and does not have paid subscriptions.
© 2010 Authors retain copyright in their individual articles, whilst copyright in AJET as a compilation is retained by the publisher. Except for authors reproducing their own articles, no part of this journal may be reprinted or reproduced without permission. For further details, and for details on submission of manuscripts and open access to all issues of AJET published since the journal's foundation in 1985, please see http://www.ascilite.org.au/ajet/
For editorial inquiries, contact the Editor, Associate Professor Catherine McLoughlin, School of Education (ACT), Australian Catholic University, PO Box 256, Dickson ACT 2602, Australia. Email: Catherine.McLoughlin@acu.edu.au, Tel: +61 2 6209 1100 Fax +61 2 6209 1185.
For review process, production and business matters, contact the Production Editor and Business Manager, Dr Roger Atkinson, 5/202 Coode Street, Como WA 6152, Australia. Email: email@example.com, Tel: +61 8 9367 1133. Desktop publishing (PDF versions) and HTML by Roger Atkinson.
|AJET 26||AJET Home||AJET