Editorial 26(2)

Beauty is in the eye of the beholder

To begin with, there is a logical explanation for the adoption of a "literary heading" for this editorial, instead of an "educational technology research" heading. During 13-14 April 2010, ODLAA's email list for members [1] distributed some messages concerning recent advice from Thomson Reuters [2] that ODLAA's journal, Distance Education [3], had been accepted for indexing in Social Sciences Citation Index [4]. The home page for Distance Education has a new line, "Now ISI listed!" [3], and in one email list posting the term "holy grail" appeared. I felt moved to point out to ODLAA members that:

The Thomson Reuters (was ISI) 'Impact Factor' used to be a holy grail, getting much influence, or far too much influence according to some of us, from its claim to select only the "best" journals for its citation analyses.[5]

So the goalposts have shifted. Thus IMHO we need to change focus, move away from 'holy grails' and look instead for 'ERA dragon slayers'. The ERA's 2010 iteration seems to have downgraded Australian based journals relative to the American or UK/Europe based multinationals. We need to focus attention on research questions of the form, "What is the correlation between the real merit of a research work and the tier ranking of the journal in which the work was published?"; "What is the correlation between the real merit of a research work and its citation count according to Scopus?"; "What is the evidence base for the ERA's application of journalmetrics to research funding matters?", i.e. the potential "ERA dragon slayer" questions.[5]

Good Lord ARC, my beauty, though but mean,
Needs not the painted flourish of your praise:
Beauty is bought by judgement of the eye,
Not utter'd by base sale of bureaucratic tiers
[after W. Shakespeare, 12]

Elbeck and Mandernach (2009) [15] examined 46 scholarly journals pertaining to computer mediated learning, using metrics for popularity, importance, prestige, and overall rankings for each journal. First the good news, from AJET's perspective. Their Table 5: Overall Ranking of Scholarly CML Journals (N = 46) places AJET well ahead of all of the five Tier A educational technology journals listed in AJET Editorial 26(1), and also ahead of the Tier B journals therein. Proponents of the Tiers ranking could say that Elbeck & Mandernach's (2009) methodology is suspect, to which Tiers antagonists can retort that if you cannot publish your own methodology, you are not eligible to criticise a competing methodology.

Now for the "howevers". Elbeck & Mandernach's (2009) 'top five' (in Table 5) is rather perplexing, as one journal has ceased (Innovate [16]), one has changed recently from open access to closed (JALN [17]), one is closed access (AJDE [18]), and one (eLearning Papers [19]) was not on the AJET Editors' list of 'journals to watch' [20] (now added, though noting that eLearning Papers is not really international, being very 'Eurocentric'). In Elbeck and Mandernach's (2009) [15] 'top five', the highest Tier ranking is B for IRRODL [21], with the other four being C ranked, except that eLearning Papers [19] is not Tiers listed.

To probe a bit deeper, the poor ranking in Elbeck and Mandernach (2009) [15] for the Tiers 'top five' namely BJET, C&E, ALT-J, JCAL and ETR&D (all Tier A, see [6] for references) is also perplexing. Elbeck and Mandernach's (2009) Table 5 ranks these journals in the range 31 to 45 in a field of 46, compared with AJET ranked 10. Considering the individual metrics, BJET, ALT-J and ETR&D fared poorly in popularity (Elbeck & Mandernach, 2009: Table 1), whilst C&E, ALT-J and JCAL fared poorly in prestige (Elbeck & Mandernach, 2009: Table 4). We need to note that the prestige metric may suffer from low reliability owing to the relatively small sample, N=23 (useable response rate 50%). However, the most important caveat concerns the use of Google PageRank data (Elbeck & Mandernach, 2009: Table 1). Their "zero" values of PageRank for BJET, C&E and ETR&D seem rather odd.

So we need to examine Google PageRank [22] more closely. These days website managers and researchers into these matters obtain PageRank data via a third party service, for example Elbeck and Mandernach (2009) used Top25Web [23]. They did not report upon PageRank data obtained from other, similar services, for example Page Rank Checker [24], CheckPageRank.Net [25], Free Page Rank Checker [26], PageRankTool.net [27], and Check SEO [28]. The problem is that the services cannot be relied upon to deliver consistent results free from spurious values, such as "zero" values of PageRank for BJET, C&E and ETR&D. Table 1 below illustrates this lack of consistency, and also indicates that investigators may need to check a number of URLs for a journal, and not just the current home page. Also for investigators, please resist the temptation to use only the instrument that gives results closest to the "desired result"!

Service (a)	Journal
	BJET	C&E	ETR&D	ALT-J	AJET
Top25Web [23] (b)	0	0	0	0	0
Page Rank Checker [24]	6	7	4(e)	6	7
CheckPageRank.Net [25] (c)	8	8	7	8	6
Free Page Rank Checker [26]	6	7	nr(d)	nr(d)	7
PageRankTool.net [27]	0	0	4(e)	6	7
Check SEO [28] (b)	0	0	0	0	0
Notes for Table 1: a. Results obtained 18 April 2010. See references [6] or [20] for journal names and URLs. b. Top25Web [23] and Check SEO [28] appear to be non-functional (returning only "zero" values). c. CheckPageRank.Net gives higher values because it returns a PageRank for the publisher's domain (i.e. it includes all of the publisher's journals rather than the nominated journal). d. Some returns were "no rank" (nr). e. For ETR&D these values are for the pre-2006 site (http://www.aect.org/Intranet/Publications/etrd/5302.asp) because the 2006 onwards site (http://www.springerlink.com/content/119965/) returned "zero".

Table 2 records a preliminary check on the popularity metric as defined by Elbeck and Mandernach (2009), using just one service, Check SEO [29]. In this case, as indicated in Table 2, it is possible to compare the third party service with a Google search command. No discrepancies were found, but of course this does not indicate anything conclusive about the utility of Link popularity relative to PageRank. Table 2 is a work in progress and to date it indicates only that there is some degree of stability in rankings over a period of 15 months. There are potentially interesting suggestions of clues to be examined further, for example is JALN going into decline as a result of its change from open access top closed access; why is eLearning Papers sustaining a high level; and what is the reason for JCAL's marked movement?

Journal (a)	Elbeck and Mandernach (28 Jan 2009) (b)	AJET Editorial 26(2) (18 Apr 2010) (c)
eLearning Papers [19]	14.83%	868
JALN	8.11%	147
AJET	3.46%	199
ET&S	2.49%	186
JCAL	1.95%	310
C&E	0.57%	48
BJET	0.46%	84
ETR&D	0.00%	0
ALT-J	0.00%	12
Notes for Table 2: a. Results obtained 18 April 2010. See [6] or [20] for journal names and URLs. b. Selected values (in %) from column "Google share", in Table 1, Elbeck and Mandernach (28 January 2009). c. Values obtained from Check SEO [29] (raw numbers: "how many incoming links there are to your site"). Returns were checked for agreement with Google search commands of the form link:URL (e.g. link:www.ascilite.org.au/ajet/) and no discrepancies were found.

With a little reluctance, we have to recognise that as "beholders", Elbeck and Mandernach's (2009) representation of "beauty" is perhaps a little flawed. There are problems with methodology and in the details of their methods. We are reluctant to be anything more than mildly critical, because AJET fared very well according to their metrics, and as suggested earlier, any presentation of methodologies for the ranking of scholarly journals has to be better than what amounts to apparently no presentation at all, as seems to be the case with the ARC's current Tiers.

In their concluding sentence, Elbeck and Mandernach (2009) seemed to look forward to:

... a definitive list and ranking of journals that will doubtlessly help faculty and administrators judge the relative value of publications for promotion and tenure purposes. [15]

Let's conclude with a "beautiful" view from one more "beholder", David Jones [30] from Central Queensland University. We thank David warmly for his very kind words, and hope that his recommendation, written in 2009 when AJET was a Tier A journal, will not change as a result of AJET's demotion to Tier B in the 2010 list [6]:

For my immediate purposes, it looks like AJET is a good fit. A journal that is open access. [30]

Roger Atkinson and Catherine McLoughlin
AJET Production Editor and AJET Editor

Endnotes

1. Open and Distance Learning Association of Australia.http://www.odlaa.org/
2. Thomson Reuters. http://thomsonreuters.com/
3. Distance Education. http://www.tandf.co.uk/journals/titles/01587919.asp [viewed 17 April 2010]
4. Social Sciences Citation Index. http://science.thomsonreuters.com/cgi-bin/jrnlst/jloptions.cgi?PC=SS
5. ODLAA Members email list posting by Roger Atkinson, 14 Apr 2010 09:43:51 +0800. See [6, 7] for references.
6. AJET Editorial 26(1). Journal rankings: AJET demoted. http://www.ascilite.org.au/ajet/ajet26/editorial26-1.html
7. Atkinson, R. J. (2010). Bibliometrics out, journalmetrics in! HERDSA News, 32(1). http://www.roger-atkinson.id.au/pubs/herdsa-news/32-1.html
8. Brumback, R. A. (2009). Impact factor wars: Episode V - The Empire Strikes Back. Journal of Child Neurology, 24, 260-262. http://jcn.sagepub.com/cgi/reprint/24/3/260.pdf
9. ARC (2010). http://www.arc.gov.au/xls/ERA2010_journal_title_list.xls (Excel format, 5.27 MB, apparently dated 9 February 2010).
10. Lamp, J. (2010). ERA Current Rankings Access.http://lamp.infosys.deakin.edu.au/era/ [viewed 18 Apr 2010; John Lamp's service is helpful for those who find the ARC's 5.27 MB file [9] too cumbersome and slow to handle on their personal computers]
11. Beauty is in the eye of the beholder. The Phrase Finder [viewed 17 Apr 2010]. http://www.phrases.org.uk/meanings/59100.html
12. Shakespeare, W. (1588). Love's Labours Lost, Act 2, Scene 1. http://shakespeare.mit.edu/lll/full.html
13. Carroll, L. (1865). Alice's adventures in wonderland. http://www.gutenberg.org/files/11/11-h/11-h.htm
14. Wikipedia (undated). Alice's Adventures in Wonderland. http://en.wikipedia.org/wiki/Alice's_Adventures_in_Wonderland
15. Elbeck, M. & Mandernach, B. J. (2009). Journals for computer-mediated learning: Publications of value for the online educator. International Review of Research in Open and Distance Learning, 10(3). http://www.irrodl.org/index.php/irrodl/article/download/676/1295
16. Innovate. http://innovateonline.info/?view=about
17. JALN (Journal of Asynchronous Learning Networks).http://www.aln.org/publications/jaln_main
18. AJDE (The American Journal of Distance Education). http://www.ajde.com/
19. eLearning Papers. http://www.elearningpapers.eu/
20. AJET: Journals relating to edtech. http://www.ascilite.org.au/ajet/about/ref/edtechpubs.html
21. IRRODL (International Review of Research in Open and Distance Learning). http://www.irrodl.org/
22. Wikipedia (2010). PageRank.http://en.wikipedia.org/wiki/PageRank
23. Top25Web. http://www.top25web.com/ [viewed 18 Apr 2010]
24. Page Rank Checker. http://www.prchecker.info/ [viewed 18 Apr 2010]
25. CheckPageRank.Net. http://www.checkpagerank.net/ [viewed 18 Apr 2010]
26. Free Page Rank Checker. http://www.free-pagerank-checker.com/ [viewed 18 Apr 2010]
27. PageRankTool.net. http://www.pageranktool.net/ [viewed 18 Apr 2010]
28. Check SEO. http://www.check-seo.com/PagerankChecker.php [viewed 18 Apr 2010]
29. Check SEO. http://www.check-seo.com/LinkPopularity.php [viewed 18 Apr 2010]
30. Jones, D. T. (2009). Choosing a research publication outlet. The Weblog of (a) David Jones. http://davidtjones.wordpress.com/2009/10/19/choosing-a-publication-outlet/

Conferences advertised in AJET 26(2)	Sydney, 5-8 December 2010. Website http://www.ascilite.org.au/conferences/sydney10/
		Quality Connections - Boundless Possibilities: Through Open, Flexible and Distance Learning 25-28 April 2010, Wellington http://www.deanz.org.nz/ Distance Education Association of New Zealand
		Global Learn Asia Pacific 2010 17-20 May 2010, Penang, Malaysia http://aace.org/conf/glearn/ Association for the Advancement of Computing in Education
		Reshaping Higher Education Melbourne, 6-9 July http://conference.herdsa.org.au/2010/
	MoodleMoot AU 2010, Melbourne, 11-14 July. http://moodlemoot.org.au/
		WCC2010 Brisbane 20-23 Sept 2010 http://www.wcc2010.com/ International Federation for Information Processing
		Sixth Pan-Commonwealth Forum on Open Learning 24-28 November 2010, Kochi, India http://www.pcf6.net/ Commonwealth of Learning and Indira Ghandi National Open University

The Australasian Journal of Educational Technology (AJET) is a refereed research journal published 6 times per year by the Australasian Society for Computers in Learning in Tertiary Education (ascilite). AJET retired its printed version (ISSN 1449-3098) at the end of Volume 23, 2007, and from Volume 24, 2008, the journal is open access, online only (ISSN 1449-5554), and does not have paid subscriptions.

© 2010 Authors retain copyright in their individual articles, whilst copyright in AJET as a compilation is retained by the publisher. Except for authors reproducing their own articles, no part of this journal may be reprinted or reproduced without permission. For further details, and for details on submission of manuscripts and open access to all issues of AJET published since the journal's foundation in 1985, please see http://www.ascilite.org.au/ajet/

For editorial inquiries, contact the Editor, Associate Professor Catherine McLoughlin, School of Education (ACT), Australian Catholic University, PO Box 256, Dickson ACT 2602, Australia. Email: Catherine.McLoughlin@acu.edu.au, Tel: +61 2 6209 1100 Fax +61 2 6209 1185.

For review process, production and business matters, contact the Production Editor and Business Manager, Dr Roger Atkinson, 5/202 Coode Street, Como WA 6152, Australia. Email: rjatkinson@bigpond.com, Tel: +61 8 9367 1133. Desktop publishing (PDF versions) and HTML by Roger Atkinson.