Australasian Journal of Educational Technology
2009, 25(1), 45-59. 		AJET 25

Sharing quality resources for teaching and learning: A peer review model for the ALTC Exchange in Australia

Geraldine Lefoe
University of Wollongong
Robyn Philip
Charles Darwin University
Meg O'Reilly
Southern Cross University
Dominique Parrish
University of Wollongong

An Outstanding Paper Award recipient, ascilite Melbourne 2008 Conference

The ALTC Exchange (formerly the Carrick Exchange), is a national repository and networking service for Australian higher education. The Exchange was designed to provide access to a repository of shared learning and teaching resources, work spaces for team members engaged in collaborative projects, and communication and networking services. The Australian Learning and Teaching Council (ALTC) established the Exchange for those who teach, manage and lead learning and teaching in higher education. As part of the research conducted to inform the development of the Exchange, models for peer review of educational resources were evaluated. For this, a design based research approach was adopted. Findings from the literature and feedback from key practitioners and leaders within the sector are discussed in this paper. Finally, key recommendations for implementation are identified.

Introduction

The ALTC Exchange (the Exchange) was designed as an online service that would "provide learning and teaching resources and functions to support communication and collaboration across the higher education sector" (ALTC, 2008). Originally called the Carrick Exchange, the repository and associated services for professional communication relating to learning and teaching was commissioned by the Australian Learning and Teaching Council (ALTC). During the development phase of the Exchange (2006-2007), funds were provided to support research by the Australasian Society for Computers in Learning in Tertiary Education (ascilite) into the needs of potential users of the system, the probable contexts of use for the system, and the policies and protocols required to facilitate engagement of and contribution by the higher education sector with the Exchange. Production of a model for peer review of educational resources contributed to the Exchange was integrated into the research brief. The aim was to investigate and derive formal and informal peer review protocols and mechanisms from the Australian higher education community, incorporating experience and research from international sources as relevant. This paper establishes the context of that research, describes the methodology used, and reports on the findings from the research and the model for peer review derived for the ALTC Exchange. The focus is mainly on the formal rather than informal processes generated.

Peer review for this context refers to the evaluation of teaching resources by those with qualifications and standing in the higher education sector who are capable of assessing the worth and value of teaching resources and similar artifacts. Reviewers might be educators, academics, consultants, designers, editors and other professionals. In general, peer review is a means of giving and receiving structured feedback so as to improve learning and teaching resources.

Peer review may be one of three types. It may be for quality assurance purposes, assessing e.g. the currency, educational design and construction of resources; copyright and digital rights; technical accuracy and reliability of resources. Secondly, it may be a formal process conducted by teams of experts, replicating scholarly peer review and publication methods. This tends to be a lengthy process, and in addition to assessing elements of quality assurance, may assess issues such as educational effectiveness, relevance, originality and currency. Formal peer review is generally conducted against specified standards, and resources considered worthy of formal peer review are likely to be perceived as high quality examples of best practice. The third type of peer review considered for this research is an informal process, known as "commentary", and includes peer evaluation submitted in an ad hoc fashion as discussion about, annotation of and comment on teaching resources and ideas. Any member of a community such as the ALTC Exchange may contribute commentary. Informal peer review is recognised as a support mechanism for peers around good practice in learning and teaching, and is essential to the development of communities of practice (Philip et al, 2007, p.846). It can also be a mechanism by which individuals show leadership and expertise in the field.

Examples of resources that might be peer reviewed in the context of the ALTC Exchange, either formally or informally, are: learning objects; learning designs; simulations; role plays; case studies; teaching and learning approaches; assessment schemes and programs; course and curriculum syllabi, outlines and materials; curriculum development documents; statements of teaching philosophy; and support materials for educational programs such as mentoring and leadership development.

Methodology

The methodology adopted for the project has been described in detail elsewhere (O'Reilly, Lefoe, Philip & Parrish, 2008). As consistent themes were identified from the literature, a series of research questions were identified to gather data from prospective users of the Exchange. The main research question sought to identify the Peer Review and Commentary protocols and mechanisms that could be derived from the higher education community. These were further expanded to include the following sub-questions:
  1. What new and current methods exist for peer review and commentary of resources that can be adapted for use for the reviewing of teaching and learning resources?
  2. What recommended policies and procedures are currently in use?
  3. What are the policy implications for the ALTC Exchange?
The ascilite research was conducted over an eighteen-month period (June 2006 to December 2007). A substantial literature review underpinned the research process. A design-based research approach was adopted for the research that used an iterative process for data collection and analysis, similar to action research in its inclusion of stakeholders in each stage. There are four elements in this design based research process:
  1. analysis of the problem by practitioners and researchers;
  2. development of solutions within a theoretical framework;
  3. evaluation and testing of solutions in practice; and
  4. reflection and documentation to produce design principles (Reeves, 2000).
The incorporation of the views of practitioners and researchers is crucial for this methodology. A process of four cycles of data collection was implemented, as identified in Figure 1. Each data collection cycle incorporated the four elements listed above. In Cycle 1 interviews were conducted with identified key practitioners and Cycle 2 incorporated focus groups with other key people. Cycle 3 was conducted with reference groups of ascilite members and finally Cycle 4 included international attendees at a symposium at the annual ascilite conference in 2007. Each successive cycle was informed by interpretation of the data from the previous cycle.

Figure 1

Figure 1: Project methodology

In order to analyse the problem, Cycle 1 data was derived from interviews with twenty-nine practitioners and leaders, key to the higher education community. Interviewees were recruited from a range of disciplines, institutions and fields of study. Interviews were recorded and transcribed, sent back to interviewees for validation, and systematically analysed with the aid of qualitative analysis software (NVivo). Development of solutions to identified problems was undertaken in Cycle 2, and data obtained through three focus groups. A total of twenty-two participants representing institutions from around Australia attended the sessions. The focus groups assisted with validation of the analysis of data from Cycle 1, prioritising issues and generating strategies to address the issues.

The draft strategies and policies created as a result were then circulated for review to members of the ascilite community in Cycle 3. A total of 24 ascilite members were chosen through a submission of expressions of interest for three reference groups conducted during Cycle 3. Reports from Cycles 1 and 2 were reviewed by reference group members. The resulting critical commentary was incorporated in Cycle 4 analysis and documentation. Cycle 4 data collation was completed at the 2007 annual conference of ascilite. International perspectives were provided by the MERLOT project (North America) and the CD-LOR project (United Kingdom) along with Australian ALTC Exchange findings, and presented at the conference symposium. Discussion and suggestions from conference delegates were incorporated into the final reports, which included recommendations and design principles generated for the project.

Although the views of participants on user engagement of repositories and communities, content contribution and peer review processes were sought during each cycle, the focus of this paper is specifically on the peer review processes.

Models of peer review

An initial literature review identified some of the key reasons for peer reviewing teaching and learning materials, many of which are relevant to this research. Four reasons for peer review of such technology resources were identified by Taylor and Richardson (2001, p.7) in a government funded report: In addition to these four reasons, the ascilite research found that peer review should:

Quality

One of the critical success factors for the design of a repository such as the Exchange is assurance of the quality and currency of resources contributed to it. Quality can be assessed through two related processes: 1) quality assurance methods and 2) peer review processes. The quality of contributed resources is of interest to both the managers and end users of repositories such as the Exchange. Feedback from participants in the research emphasised the importance of quality. Many saw the implementation of rigorous peer review processes as key to maintaining high standards.
Rigorous peer review is important for the outcomes and status of the review process. Having peer review processes in place supports the perception of quality resources and communication of best practice. (Focus group)

Recognition

The ascilite findings indicate that a national repository is a valid context in which to implement a formal peer review system. Whereas an institutional or discipline based repository may have insufficient resources for a fully implemented editorial board and ongoing review process, a national system could be expected to support such a structure. Small and local repositories, nonetheless, usually put in place some system of quality assurance. However, increasingly, with the proliferation of Web 2.0 technologies, more repositories are establishing facilities for informal peer review, i.e. incorporating commentary and annotation tools, and ratings systems.

Repositories of teaching resources and peer reviewed journals that provided important peer review and/or quality assurance models for this research included: from the USA, the repositories MERLOT (http://www.merlot.org/) and Educause (http://www.educause.edu/), the Health Education Assets Library (HEAL, http://www.healcentral.org/), and the journal Nature (http://www.nature.com/); the Cooperative Learning Object Exchange (CLOE) in Canada (was http://cloe.on.ca/, see http://cloe.desire2learn.com/); the national Jorum repository in the UK (http://www.jorum.ac.uk/), Intute (http://www.intute.ac.uk/), and the Journal of Interactive Multimedia in Education (JIME, http://jime.open.ac.uk/) at the Open University in the UK; and EducaNext (http://www.educanext.org/) in Europe. The researchers also examined a number of quality assurance systems in institutional and discipline based repositories in Australia and overseas, in addition to the Australian Universities Teaching Committee (AUTC) ICT based Learning Designs website (http://www.learningdesigns.uow.edu.au/), as a model of formally implemented peer review, and the LAMS Community (http://www.lamscommunity.org/) which implements informal peer review. Criteria and standards against which the teaching resources could be evaluated were most useful for the peer review models derived from the AUTC Learning Design Project, the CLOE and MERLOT websites, and in documents from the Health Education Assets Library (HEAL 2006), the University of Queensland et al (2004), and Souza and Persily (2005).

The MERLOT repository in the USA provides an exemplary model of formal peer review for educational resources (McMartin, 2004; Nesbit, Belfer & Vargo, 2002) on which, for example, CLOE has based its system. MERLOT has an "expertise orientated" approach (Worthen, Sanders & Fitzpatrick, 1997), derived from traditional academic peer review practices for scholarship and publication. In addition, using the affordances of repository database technologies, peer reviews are recorded as a means of formally recognising contributors; and as resources on the repository are used, patterns of usage are tracked and made available to authors on an annual basis for the purposes of documentation, recognition and promotion.

Nesbit, Belfour and Vargo (2002) created a "convergent participation model" to evaluate learning objects, based on the MERLOT model. In a two-stage process, two individual experts firstly assess resources; next, a combined group assessment is made and the reviews from both assessments are synthesised. In this model, students are included on the panel of reviewers. The Australian ACELL project, Advancing Chemistry by Enhancing Learning in the Laboratory (http://acell.chem.usyd.edu.au/), is an example of a discipline based repository that also incorporates learner feedback in the peer review process. Resources on the site are not published until they have been through evaluation by staff and students. Finally new technologies are able to both enhance and provide earlier access to publications as the American Chemical Society demonstrates with a new pilot project for their journals (http://pubs.acs.org/cen/news/87/i03/8703notw7.html).

Collaboration

Determining the standard of resources submitted for peer review places demands on reviewers, including the selection of criteria adopted to assess the items for review, the processes of moderation adopted, the time taken for the review process, and communication undertaken with authors. Referring to standard peer review of scholarly publications, Starbuck (2003) observes that "authors need to view reviewers' comments not as judgments about the value of their work, but as good data about potential readers of their articles." (p.344). During the peer review process there is often a hierarchy established between reviewer and author, a relationship that some suggest should be reassessed if educators are really interested in maintaining and raising standards of elearning resources. There is an argument for more collaboration between reviewers and authors throughout the review process, so that reviewers work less in judgement and more as peers in collaboration (Starbuck, 2003). Starbuck also notes that standard review processes have been shown empirically to incorporate elements of bias and randomness. A model of peer review for a repository of high quality learning and teaching resources might therefore be based on traditional methods of peer review, but modified to include more negotiation than usually expected in scholarly publication processes, in addition to informal peer review in the form of critical commentary or annotation on resources. Having more open peer review mechanisms, which could include a mix of formal and informal processes, may "increase accountability, fairness, and transparency" (Van Rooyen, Godlee, Evans, Black & Smith, 1999, p.44).

Grounded in scholarship

Feedback from participants in this research indicated that academics and educational designers wanted reward and acknowledgment for development and implementation of teaching related resources, and for it to be regarded on a par with scholarly research and publication. This same conclusion was drawn by Taylor and Richardson (2001) in their report, Validating Scholarship in University Teaching, research conducted into the establishment of a national system for peer review of ICT based teaching and learning resources. Taylor and Richardson considered the conceptual and procedural basis for such a scheme and stressed the importance of focusing on the scholarship around the development of ICT based resources. Key to the system was the "value adding" scholarship work of academics (p.7), that is, "those aspects of scholarship... that both inform the design of pedagogical practices and are evident in the resources constructed on the basis of that design" (p. x). Further, they maintained that "Activities of teaching and learning are now claimed as public territory... Our position is that the benefits of peer review still outweigh a situation where 'anything goes', whether in the field of research or teaching" (p.53). Peer review is a means of validating and recognising the scholarship that Taylor and Richardson claim is important for the process of bringing parity between two key areas of academic endeavour. The ascilite research corroborates this view, as participants consistently stressed the importance of recognition and reward for their scholarly work in both teaching and research.

Reviewers

In terms of who should do the reviewing, previous Australian research (Taylor & Richardson, 2001) had suggested that reviewers should be editors of journal and conference proceedings and it would be the responsibility of these editors to advertise the national peer review scheme to be developed. However, following the ascilite research, the following is recommended for the ALTC Exchange. Firstly, that reviewers be recruited from discipline groups that network around the Exchange, as in the MERLOT model. The collaborative Peer Review of Learning Materials Report (University of Queensland et al, 2004), suggests that staff for whom a significant part of their core business involves developing learning resources may be appropriate peer reviewers. Secondly, feedback from research participants suggested that: MERLOT operates on a strong discipline based system to promote contribution to the repository and to recruit peer reviewers. It utilises individuals engaged in other institutional initiatives as peer reviewers. To find peer reviewers, MERLOT adopts the following methods: Thirdly, for the ALTC Exchange, it was recommended that a group of cross-disciplinary experts should undertake the review process, namely discipline, educational and technical/media experts, plus an Editorial Chair. Not only teaching academics should be involved in the processes, but educational designers and IT personnel employed to assist with quality assurance and provide feedback on the design and potential effectiveness of any resource under review. A caution was offered that peer review should not be conducted by those in a line management relationship. Fourthly, those practitioners and learners who have used the teaching resource could provide effective feedback on the resource in question. Finally, the Exchange should not rely solely on volunteers to manage the process. Feedback from the sector and the experience of institutional repositories suggests that the ALTC Exchange may need to employ individuals to manage peer review processes. For formal peer review, the traditional system of "blind" peer review where reviewers are not identified, versus a system where reviewers are known was discussed with practitioners and stakeholders. Some participants favoured the latter system because of the accountability it demanded; others favoured the traditional and anonymous version because it appeared to assist with objective and democratic evaluation. Others suggested that where an author was able to engage in dialogue with the reviewers this was beneficial for developing a full understanding of the design dimensions, and intended and actual learning outcomes. In terms of authorship of informal peer review contributions, however, participants overwhelmingly indicated that it should be a transparent process where opinions were identified by the name of the author. The credibility of the person contributing the comments was an important factor for members considering the worth and value of others' commentary. Further, it was thought that "named contributions" would promote scholarly, thoughtful review of resources and lessen the likelihood of work being treated insensitively.

Open peer review models

Finding a balance between quality review (particularly in informal contexts) and democratic commentary open to all members of the Exchange will be challenging. One participant suggested:
There is a need to develop models and processes for peer review alongside issues of copyright, development of acceptable use policy and innovative use of technologies that model open source, participatory and interactive knowledge creation. (Reference group)
Reference group members encouraged the ALTC to investigate new methods of peer review:
Web 2.0 has sparked a growing number of "free" and "open" movements that challenge current publishing and peer review models - including the Free and Open Source Software movements, the Open Access Movement, Open Source Journalism and Creative Commons, and they ought to be considered in the new era of publishing as each contributes different perspectives and concerns on the emerging practices surrounding digital repositories and scholarly publishing. (Reference group)
While adopting traditional peer review methods for the assessment of ICT based resources has its merits, some scholars have noted that traditional types of blind peer review favour tradition and not innovation (Rogers, 2006). The traditional peer review system evolved when dissemination was difficult and expensive. Some journals have tried a more open approach, and feedback from the reference groups suggested that the developing open peer review systems trialed by a number of scholarly journals such as Nature be examined as alternative models. The consequences of publishing to the Internet as early as possible and not waiting until a resource has been through an exhaustive quality assurance and peer review process were seen as follows: In the future the Internet may be the first place of publication, as Anderson (2007) argues, and only the very best and most enduring works will be published in hard copy. Some journals are trialing more open peer review processes: the journal Nature in the USA and JIME in the UK are two examples. Nature began trialling a more open process in June 2006 (Rogers, 2006). Authors can choose a 'preprint' option of publication called Nature Precedings (Nature, 2007). Once a research article has passed an initial quality check, the author posts the paper to the journal's website, and anyone, provided they give their name and email address, may comment on the research; the traditional blind peer review process continues in the background. The traditional method of review is lengthy, and can take between four and twelve months. The open approach, where a preprint is out in the public domain immediately the research article is completed, allows for quicker access and dissemination. Where maintaining currency of resources is an issue, as in the context of the ALTC Exchange, this is an appealing model. Nature warns readers to treat the findings published in Nature Precedings with caution as possibly preliminary or speculative, but all documents are citable (have a DOI), and are archived under a Creative Commons licence where derivatives are allowed (Nature, 2007). In terms of what is accepted for peer review, certain judgements about standards are made before release in this pre-review form, i.e. about content and authorship: submissions must be deemed to be "genuine" scientific not pseudoscientific contributions, and submitted from "qualified" scientists with "recognised academic affiliation" (Nature, 2007, para. 3).

Another publication model worthy of noting here is JIME, the Journal of Interactive Media in Education. As a publication it does not have a chronological concept of "issue": articles are published for open peer review as soon as they are received. Final versions of articles are published as soon as the review process is completed. Authors have the right of reply; reviewers are named and accountable for their comments, and their contribution acknowledged; and the wider research community has the chance to shape a submission before publication. Submissions are potentially critiqued by many more reviewers than is possible in a conventional review process. It is also a public rather than private process. Preference is given to signed reviews, but anonymous contributions are permitted. Authors and reviewers must be willing to engage in negotiation to determine meaning, accuracy and quality.

Whilst the open review process speeds publication, the process of soliciting informal peer review comments was seen to be less than successful in the Nature trial. Reaction from authors and scientists was mixed. There was considerable traffic on the site where preprints were posted during the trial's four month period, and though the concept was reportedly well received, few readers commented on the papers. This reluctance to provide substantive comment in general on other repository, educational, professional development or social networking sites was noted by some participants in the ascilite research, and this lack of willingness to engage in commentary has been observed on the ALTC Exchange in the early stages of implementation in 2008. Comments in Scientific American (Waldrop, 2008) pick up on this debate and the value of collaboration versus competition in pre-publication of research to scientists' own websites, blogs and wikis, etc. As with participants in the ALTC Exchange research, there are real concerns with losing competitive advantage and opening oneself up to scorn from others who disagree with the preliminary findings. Further, reading and commenting on draft research, let alone keeping up with what is already formally published, is considered time consuming. Nonetheless, the argument for early publication and wide dissemination provided by the Nature example, may overshadow counter arguments that informal peer review is likely to be minimal and of little real value for development.

The ALTC Exchange peer review model

The following model (Figure 2) was submitted to ALTC as a result of ascilite's consultation with the higher education sector and consideration of models discussed above. To date this model has not yet been adopted as the Exchange progresses through its first stages of implementation (from May 2008). The model was derived from the models created for HEAL, MERLOT and CLOE repositories, and the work of Nesbit, Belfer and Vargo, (2002). The HEAL (2006) editorial policy provides useful detail on roles in the peer review process, as do Barton and Waters (2004). Features recommended for formal peer review processes for the ALTC Exchange included the following: that peer review be completed within an agreed timeframe, with specific goals and outcomes and clear identification of review focus; that the reviewers be identified; and that the criteria for assessment be clearly articulated. It may also be worthwhile for each resource contributor to have tools to conduct their own self evaluation prior to submitting their work for formal peer review (Bruce, 1997; Smith, 2005).

Figure 2

Figure 2: Adapted formal peer review process model for resource contributions

The major steps in the formal peer review process for the Exchange

  1. The author submits a resource to the Exchange and nominates it for peer review.
  2. Simultaneously, the resource may be made public, allowing for informal comment by members of the Exchange community. The resource is published and open for general review, at the same time as it is categorised as "under formal peer review".
  3. A technical and quality assurance check is undertaken, assessing for quality of content and copyright issues, educational effectiveness and ease of use/usability. The Exchange Administrator or Quality Assurance Editor oversees this step.
  4. After the resource passes the quality check, the Editorial Board Chair is notified that the resource is awaiting review.
  5. The Panel Chair convenes reviewers (probably to an online meeting). The Chair oversees the process and facilitates the discussion amongst the reviewers, and the final decision. This is communicated to the author as a numerical rating and written comments.
  6. The Panel consists of subject matter specialist, education and media/ information/ IT specialists. Reviewers conduct and record their reviews independently. They are then convened by the Panel Chair to discuss their assessments (provided numerically and as written comments): this helps mitigate against bias by the Chair and gives a more holistic evaluation, taking into account technical, educational and discipline specific issues. The Panel Chair writes the report.
  7. If any "informal" commentary on the resource has been received via the Exchange (see Step 2) the Chair and the Panel may take this into account in the final review.
  8. If modifications to the resource are required, the Panel Chair may elect to discuss the result with the Author, and negotiate amendments as required.
  9. Once accepted, the resource is published, marked as "Reviewed" and made available along with a summary of the reviewers' comments.
  10. Members of the community may continue to comment on the final resource and give it a "star" rating to indicate its quality to others.

Conclusion

The findings of the ascilite research for the ALTC Exchange indicated that stakeholders and practitioners from the Australian higher education sector saw the following four elements as important for a formal peer review scheme:
  1. a review panel comprising a mix of experts;
  2. adequate funding and support for the resource intensive process of peer review;
  3. sustainable strategies and protocols to support and manage the process in the longer term; and
  4. educational effectiveness to be regarded as a key review criteria.
Recommendations in four key areas were made as a result of the ALTC Exchange research.
  1. Protocols and mechanisms related to the goals of peer review
    Formal peer review policies and procedures of the Exchange should align with institutional policies and practices, and promote the scholarship of teaching and learning. The goals of formal peer review should be established and inform related policies and procedures. Peer review should enable judgements about quality and usefulness of a resource. Resources contributed to the Exchange should be classified according to the level of peer review received. Formal peer review should be appropriately resourced and adequately rewarded.

  2. Protocols and mechanisms related to the process and procedures for peer review
    The Exchange should establish processes, criteria and standards that enable consistent, equitable and fair peer review. The peer review process should meet the expectations of the sector and the needs of members. It is important that the process should be viewed as "institutionally neutral" to promote its value within the higher education community. A broad based team of reviewers would be required to provide support to a well recognised editorial chair in the disciplinary field.

  3. Protocols and mechanisms related to informal commentary
    Guidelines and protocols for informal commentary should be established to meet the needs and expectations of contributors and users. This would be an informal process whereby members of the community voluntarily respond to others' contributed resources, or resources stored elsewhere but linked to via the ALTC Exchange. Informal commentary might include discussion, feedback or comments made on resources and ideas contributed to the Exchange. Commentary is seen to be beneficial because it can provide informed users' ratings, perspectives and assessment.

  4. Protocols and mechanisms related to rewards and incentives for formal peer review and informal commentary
    The Exchange must implement rewards and incentives for participation in either formal peer review or informal commentary. They underpin the ultimate success of the entire review process. For contributors and users to engage in the review process there needs to be a perception of added value, either in terms of recognition through institutional processes such as promotion, or through ALTC recognition processes, such as awards.
Testament to the difficulties of implementing peer review systems of evaluation of teaching and learning resources, the work of Taylor and Richardson (2001) has not been implemented nationally, and as a consequence of reallocation of resources for the ALTC Exchange, a formal peer review system has not yet been initiated. It seems that an optimum "window of opportunity" (Taylor & Richardson, 2001, p.87) to establish a peer review scheme was missed in 2001, and may be again in 2008, unless steps are taken quickly to realise this goal.

Interviewees, focus group participants and reference group members consistently indicated that, to be successful, the formal peer review process should be appropriately resourced and adequately rewarded. The peer review process was seen as resource intensive, so only those resources which are of sufficient quality to warrant this assessment should be submitted for full peer review. The need for peer review processes to be integrated into other existing institutional processes was reiterated and echoes the findings of the Queensland research (University of Queensland et al, 2004).

As well as managing the peer review process, the Australian Learning and Teaching Council should make provision for a system of reward and recognition for authors and reviewers. Focus group participants strongly urged that membership of a peer reference group or formal review panel be viewed as prestigious and that peer review responsibilities valued as part of an academic's professional development. In agreement with findings of The British Academy (Shepherd, 2007), the researchers urge those responsible for the management of universities and research institutes to encourage and reward peer review activity, as this may encourage high calibre academics, already overburdened with work, from declining peer review duties, and may hearten other academics to more readily engage in the peer review process. Recognition and reward for teaching related activities is currently on the rise in Australian higher education. The ALTC Exchange needs to step up and take on the leadership role it is afforded through this enterprise to implement such a peer review process for teaching and learning resources.

Acknowledgments

This paper is based on the reports created for the Ascilite-Carrick Exchange project: http://www.ascilite.org.au/index.php?option=com_content&task=view&id=147&Itemid=119

References

ALTC (2008). The ALTC Exchange. Australian Learning and Teaching Council website. [viewed 22 July 2008] http://www.altc.edu.au/carrick/go/home/pid/534

Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education. Joint Information Systems Committee (JISC) Technology and Standards Watch, February. [viewed 25 July 2008] http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf

Barton, M. & Waters, M. (2004). Creating an institutional repository: LEADIRS Workbook. LEADIRS (Learning about Digital Institutional Repositories). Boston: MIT. [viewed 29 Sep 2007, verified 31 Jan 2009] http://www.dspace.org//implement/leadirs.pdf

Bruce, C. (1997). Peer review: A handbook. Queensland University of Technology. [viewed 29 July 2008, verified 31 Jan 2009] http://sky.scitech.qut.edu.au/~bruce/peerreviewhanbook/

Health Education Assets Library (HEAL) (2006). Editorial Policy. [viewed 22 Sep 2007, verified 31 Jan 2007] http://www.healcentral.org/services/policies/HEAL_Editorial_Policy_v2.0.pdf

McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and implementation. Journal of Digital Information, 5(3). [viewed 25 July 2008] http://jodi.tamu.edu/Articles/v05/i03/McMartin/

Nature (2006). Nature's peer review trial. December. DOI: 10.1038/nature05535. [viewed 25 July 2008] http://www.nature.com/nature/peerreview/debate/nature05535.html

Nature (2007). Nature Precedings is live. [viewed 25 July 2008] http://blogs.nature.com/peer-to-peer/2007/06/nature_precedings.html

Nesbit, J. Belfour, K. & Vargo, J. (2002). A convergent participant model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3). [verified 15 Feb 2009] http://www.cjlt.ca/index.php/cjlt/article/view/110/103

O'Reilly, M., Lefoe, G., Philip, R. & Parrish, D. (2008). The Carrick Exchange for Higher Education: Design factors for user engagement. In J. Luca, & E. Weippl (Eds.), Proceedings of Ed-Media 2008, World Conference on Educational Multimedia, Hypermedia & Telecommunications. pp. 211-230. Vienna, Austria: Association for the Advancement of Computing in Education.

Philip, R., Lefoe, G., O'Reilly, M. & Parrish, D. (2007). Community, exchange and diversity: The Carrick Exchange. In ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007, 844-845. http://www.ascilite.org.au/conferences/singapore07/procs/philip.pdf

Reeves, T. (2000). Enhancing the worth of instructional technology research through 'design experiments' and other development research strategies. Paper presented at the Annual Meeting of the American Educational Research Association, 27 April, New Orleans, LA. [verified 15 Feb 2009] http://it.coe.uga.edu/~treeves/AERA2000Reeves.pdf

Rogers, A. (2006). Get Wiki with it. Wired, 14 September, pp.30-32. [viewed 25 July 2008, verified 31 Jan 2009] http://www.wired.com/wired/archive/14.09/start.html?pg=3

Shepherd J. (2007). Help wanted. The Guardian, 4 September. [viewed 22 Sep 2007, verified 31 Jan 2009] http://www.guardian.co.uk/education/2007/sep/04/highereducation.research

Smith, C. (2005). Guidebook for individuals embarking on peer review of teaching. The Teaching and Educational Development Institute, University of Queensland. [viewed 29 July 2008] http://www.tedi.uq.edu.au/downloads/evaluations/Guidebook_PeerReview.pdf

Souza, K. H. & Persily, G. (2005). National peer review of digital learning materials in the health sciences. [viewed 22 Sep 2007] http://cit.ucsf.edu/events/event_mar05.php

Starbuck, W. (2003). Turning lemons into lemonade. Journal of Management Inquiry, 12(4), 344-351.

University of Queensland, Griffith University & Queensland University of Technology (2004). Peer review of learning materials report: A collaborative project involving the University of Queensland, Griffith University and Queensland University of Technology. 30 August 2004.

Taylor, P. & Richardson, A. (2001). Validating scholarship in university teaching. Canberra, ACT: DETYA, Commonwealth of Australia. [viewed 25 July 2008, verified 31 Jan 2009] http://www.dest.gov.au/archive/highered/eippubs/eip01_3/01_3.pdf

Van Rooyen, S., Godlee, F., Evans, S., Black, N. & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. British Medical Journal, 318, 23-27.

Waldrop, M. (2008). Science 2.0: Great new tool, or great risk? History of Science, Scientific American, 9 January, 1-5. [viewed 25 July 2008, verified 31 Jan 2009] http://www.sciam.com/article.cfm?id=science-2-point-0-great-new-tool-or-great-risk

Worthen, B., Sanders, J. & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd Ed.). New York: Longman.

This article received an Outstanding Paper Award at ascilite Melbourne 2008 Conference, gaining the additional recognition of republication of an expanded version in AJET. The reference for the Conference version is:
Philip, R., Lefoe, G., O'Reilly, M. & Parrish, D. (2008). A peer review model for the ALTC Exchange: The landscape of shared learning and teaching resources. In Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008 (pp. 766-775). Melbourne: Deakin University. http://www.ascilite.org.au/conferences/melbourne08/procs/philip.pdf
Authors: Dr Geraldine Lefoe, Centre for Educational Development and Interactive Resources, University of Wollongong NSW Australia 2522. Email: glefoe@uow.edu.au

Robyn Philip, Teaching and Learning Quality Group, Charles Darwin University. Email: rlphilip@bigpond.com

Dr Meg O'Reilly, Teaching and Learning Centre, Southern Cross University. Email: meg.oreilly@scu.edu.au

Dominique Parrish, Faculty of Education, University of Wollongong. Email: dom@parag.com.au

Please cite as: Lefoe, G., Philip, R., O'Reilly, M. & Parrish, D. (2009). Sharing quality resources for teaching and learning: A peer review model for the ALTC Exchange in Australia. Australasian Journal of Educational Technology, 25(1), 45-59. http://www.ascilite.org.au/ajet/ajet25/lefoe.html

[ PDF version of this article ] [ AJET 25 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet25/lefoe.html
Created 17 Feb 2009. Last revised 17 Feb 2009.
Page access count