It takes more than metadata and stories of success: Understanding barriers to reuse of computer facilitated learning resources

A national study in Australia in the late 1990s explored barriers to the adoption and reuse of computer facilitated learning (CFL) in Australian universities. These barriers will be summarised. One of these barriers is that it is hard to find information on courseware that is educationally sound; usually such courseware is expensive to produce and so reuse is especially desirable. However, even when information and access to electronic courseware exists, reuse may still not occur. Two cases will be described to illustrate the complexity of reuse. These cases are: 1) a collection of 169 plastic surgery websites; and 2) an international consortium of veterinary microbiology resources based on a well-evaluated case study design. Some strategies for improving reuse are suggested.

Just what is this paper about?

If it works with text books, why not with electronic media?

Pearson Education's reach extends across the globe through its seventy regional Web sites and twenty-five publishing centers, developing educational products for children, schools, universities, adults and corporations in thirteen languages. Long renowned as the world's market leader in English Language Training, Pearson Education maintains publishing operations in seven regions: the U.S., Europe, Middle East and Africa, Canada, Latin America, North Asia, South Asia, and Australia & New Zealand. Global imprints include Longman, Scott Foresman, Prentice Hall, Addison-Wesley, Prentice Hall-Financial Times, Markt & Technik, CampusPress, Direct English and Éditions du Renouveau Pédagogique. (http://www.pearsoned.com/globalPub.htm)

In order to get similar models operating with purely electronic courseware, what is needed? Is it just a question of organising electronic resources better? Or are there deeper issues about the use of electronic media in education? We are now in the position that there has been a great deal of development of electronic educational resources in universities world wide. This has occurred together with substantial investment of IT systems and infrastructure. However, there is little evidence of dissemination of these electronic resources and practices. Greater collaboration and sharing of resources is becoming an increasingly urgent issue. There are several existing databases of computer facilitated learning (CFL) materials, but these databases do not appear to have significantly increased the take up of CFL materials and strategies. In order to make the most of the valuable resources which exist, a range of educational, technological and management factors needs to be addressed. Let's look briefly at one study which examined adoption of CFL across Australian universities.

What major issues facilitate or mitigate against adoption of CFL?

This study found that the issues surrounding the adoption of CFL at university are complex, and no single factor will result in adoption. Instead, there is a range of factors, all of which must be addressed. Several universal factors in relation to widespread use of CFL were identified:

• coherence of policy across all levels of institutional operations and specific policies which impact on CFL within each institution;
• intellectual property, particularly the role of copyright in emerging online environments;
• leadership and institutional culture;
• staff issues and attitudes: namely, professional development and training, staff recognition and rewards, and motivation for individuals to use CFL; and
• specific resourcing issues related to funding for maintenance or updating of CFL materials and approaches, staff time release and support staff.

• the educational design of the CFL resources being produced,
• the incentives and support that exist for individuals to produce CFL resources,
• the technical design and access specifications for using these CFL resources,
• the experience of using the CFL resources in actual teaching contexts,
• evaluations carried out to determine how educationally effective these resources have been in practice,
• intellectual property and copyright issues which might affect the use by others, and
• how access can be obtained to these CFL resources from either colleagues in the same university or another institution.

How much will access to information about electronic material assist adoption and reuse?

1. How does one find, select and organise educationally appropriate websites?

Commercial search engines were used to identify plastic surgery websites in July 2000. One year later all sites were revisited. Those no longer available were excluded and the remaining sites were scored according to content, complexity, technical difficulty and readability using the Flesch-Kincaid Grade Level Score (e.g. Johnson, 1998). This readability index computes readability based on the average number of syllables per word and the average number of words per sentence. The score in this case indicates a grade school level. For example, a score of 8 means that an eighth grader would understand the document. Standard writing approximately equates to the seventh to eighth grade level. The maximum score is 10. Two hundred and five sites were identified in July 2000 and one year later 169 were still online. Less than 10% of sites scored 8 and above, 60% scored 4 and below. There is clearly a wide range of material available on the Internet but the question remains how to identify the sites with the greatest educational value, especially for students for whom English is a second language? Is reading ease a key factor in a visually rich environment? Are the objectively high scoring sites the best?

In order to begin to answer these questions, we asked 150 students to evaluate these 169 Internet sites according to their subjective perception of educational value, including language and content. Students related to the sites according to their own personal judgments of their educational interests and needs. They were not given specific criteria but ranked them on subjective impressions of their value to themselves at the time of ranking. The rankings from the objective (reading ease) and the subjective (personal interests and needs) scoring did not correlate. Clearly the reading ease of the text was not a key decisive factor in deciding students' preferences. While this is not a rigorous study, it has highlighted the wide variety of student preference about the style of web material they prefer.

We are left with a wealth of material but no clarity at this point about how to organise and frame this material so as to support individual student learning styles (Coates & Rowsell, 1998). How does one design the basic shell which points to these sites? How do we order and groups these sites? Certainly a list of 169 URLs is not likely be an educationally engaging site! Should detailed descriptions be added, pointing out a range of ways in which the material on each site might be used? Should we add questions which can be answered by searching a number of sites? Should we use these sites as a basis for case study questions? Some or all of the above? All we know at this stage is that an enormous amount of work will be needed before this potential wealth of the Internet can be realised for the benefit of medical students studying plastic surgery. And we are unclear about how to find the time and technical resources to do this work.

This case shows clearly that good educational design is one important element in integrating existing electronic resources into a coherent learning resource for students. Knowing about resources and having access to them is important, but knowing what to do with them in order to facilitate learning provides another set of complex challenges.

2. If it's too hard for one group to search, evaluate and design, can a consortium work better?

In January 1996 an international consortium of over 30 veterinary bacteriology and mycology teachers from Australia, Canada and the USA was formed to provide new online materials for veterinary courses. One of the main focuses in the formation of this 'consortium' was the emphasis on learning microbiology as a tool for preparing professional veterinarians, rather than learning microbiology for its own sake. PBL was to be a foundation principle for all projects. As a pilot, a range of materials was produced to allow members of the international consortium to comment on issues such as content and style. In order to gauge student response to these initial VetSource materials, five students were interviewed in September 1997. Student response to the initial VetSource materials was very encouraging (McNaught, Whithear, Browning, Hart & Prescott, 1998).

All this is very positive, but two years later the consortium had petered out. Only three universities actually provided materials and this is not enough to build the foundation for a true multi-institutional consortium. The funding that the consortium hoped to obtain through the US Department of Agriculture did not eventuate. The University of Melbourne was building a web based platform that would be highly suitable for the interactive nature of the VetSource modules (Goschnick, 1998). Despite a public launch of Creator (as the platform was called), funding for completion of the project was withdrawn. The lack of buy-in in terms of producing materials, the lack of a technical platform and the limited funding took its toll. The few VetSource modules were incorporated in the VetBac project at the University of Melbourne, which still continues, but these materials were not used by any of the other consortium members. The model of VetBac has influenced the development of other computer facilitated PBL veterinary work at the University of Melbourne (McNaught, Whithear & Browning, 1999) but the scale of operation has a much more limited scope than we hoped in the mid-1990s.

In this story there were three barriers-funding, a technical platform and insufficient production of the modules. A conclusion we might draw about why there was insufficient production of the modules is that the work environments in the universities in the consortium did not support the time and energy these academics needed to devote to making the consortium a sustainable entity. The models of how to do PBL exist; the willingness to engage can be kindled; but the work environment of higher education needs to be more supportive towards the time needed for innovation and collaboration.

So, what is needed to further the effective reuse of electronic media content?

1. effective educational design in combining a number of resources designed elsewhere, and

2. policies and processes which support the culture of collaborative work environments, and enable the recognition of this form of work.

1. What is meant by effective educational design in this context?

• improving the understanding of how electronic resources might work with other design elements of the student learning experience; and

• the development of generic designs.

Student learning need (after Open University, 1998)	Examples of appropriate use of online strategies
Building and maintaining motivation	engaging scenarios clear and current information*; early diagnostic feedback.
Negotiating choices	clear and current information*; topic choice based on graduate capabilities; assessment choice; discussion threads for negotiation; evaluation forms for two way feedback.
Information handling skills	web searching for online resources; use of databases; online sharing of resources; FAQ sites and/or glossary.
Independent learning skills	negotiated assignments; access to IT/info literacy and study skills resources; use of personal online journals.
Developing understanding	building websites; collaborative tasks (document sharing, specialist chat groups); problem solving exercises - from simple quizzes (for formative assessment) to the use of databases linked to case studies; concept mapping software; a collection of past/recent exams and sample tests (where appropriate).
Linking theory to practice	embedded media and simulations; online tutorials; students as mentors in online groups; samples of previous assignments/project work (with documented student permission).
Practising discussion, argument, articulation of ideas	online tutorials; online debates using a threaded discussion; role playing using a threaded discussion; sharing essays online.
Rehearsing skills and procedures	regular and frequent online quizzes with feedback; interactive activities using spreadsheets, multimedia objects, laboratory notes and guides.
Practising teamwork	group projects progressively sharing resources and annotating peer work using collaboratory software.
* Clear and current information includes items such as: direct access to the related approved subject or course guide; a current timetable/timeline related to outlining face to face tutorials, lectures, laboratory/ field work and online activities (with times, dates and location details); a map showing online learning activities clearly described/linked to learning outcomes; current contact details of lecturers, teachers and tutors; lecture outlines; and news announcements.

Improving the understanding of how electronic resources might work with other design elements of the student learning experience

Several people are working in this area. One model is given by McNaught, Kennedy & Majoor (2002). They describe a model of how different student learning needs can be supported by the various functions of and strategies for online learning. This model will hopefully further the understanding of how the various components of an online learning site, such as information areas, interactive tutorials, quizzes, and access to threaded discussions and chat can support learning for more students most effectively (Table 1). This may make it easier for teachers to see how a variety of electronic materials they have collected (such as the plastic surgery resources described earlier) might be used in designing a coherent student learning environment.

The development of generic designs

There is certainly a major interest in whether it is possible to develop generic designs. Generic designs can be described as a combination of clearly articulated learning designs combined with templates into which media elements can be inserted. The Australian University Teaching Committee (AUTC) has funded a project titled: 'Information and Communication Technologies (ICTs) and their role in flexible learning' The project is a two year endeavour and began in November 2000. The project's aims as stated at the end of 2001 were:

1. The identification of a range of learning designs that have been demonstrated to contribute to high quality learning experiences and which can be applied generically;

2. The design and subsequent development of a series of re-usable software, templates and/or exemplars for the learning designs previously identified; and

3. The development of a series of guidelines for good practice in the use of or implementation of the software, templates and/or exemplars in new contexts.

1. To facilitate the identification of learning designs that foster high quality learning experiences; and

2. To provide a mechanism to determine whether such learning activity designs have the potential for re-development in a more generic form. (Harper, Oliver & Agostinho, 2001, p. 255)

2. What policies and processes are needed to support the culture of collaborative work environments to foster adoption and reuse?

• provision of access to information about CFL resources; and
• policies and processes to enable collaborative work, both within and across institutions.

Access to information about CFL resources

While we have said this is not the determining factor in the use and reuse of CFL, it is important, and the current work being done on metadata and interoperability standards needs to be followed so that Internet architecture is developed with learning in mind. A good site to keep on eye on is the Australian IMS project site, hosted by EdNA Online at [http://ims.edna.edu.au/] It provides information on metadata standards and projects in Australia and elsewhere. Though not recently updated (most of the pages are August 2000), it does provide contact points for further searching.

AustLit, [http://www.austlit.edu.au/] (Australian Literature Gateway) and UniServe Science [http://science.uniserve.edu.au/] are examples of well-known information points and serve as continuing models of provision of access to relevant information and resources. In addition, UniServe Science organises events such as national symposiums in order to enact a role of fostering "a sense of community among tertiary teachers of science" (Johnston & Peat, 2002).

A recent Australian survey by the Department of Education, Science and Training (DEST) (Bell, Bush, Nicholson, O'Brien & Tran, 2002) investigated the extent of the use of online technologies by Australian universities. Fifty-four percent of university units now include an online component. The report describes the nature of online courses, what disciplines they are in, and how they are supported by online services. This is useful information and may hopefully be the beginning of more detailed sharing of information about online resources being used in Australian universities.

Policies and processes to enable collaborative work, both within and across institutions

Just how much activity is there in this area? Just how high up on the priority list for our governments is this type of policy item? If it is seen as a having a high priority at government level, it is much more likely to get attention at institutional level. Here we will focus on the Australian government. In order to get a rough idea of the number of studies and reports in the area of collaborative policy in higher education commissioned by the Australian government in recent years the Australian Department of Education and Science (DEST) website was searched using InfoCat. The InfoCat ('Information Catalogue' http://infocat.dest.gov.au/IE/) covers most research and statistical information published by DEST since 1995, with some information for work done prior to 1995. EdNA Online (http://www.edna.edu.au/) was also searched looking for Australian sites. The search words and results are shown in Table 2.

While this is hardly an exhaustive or rigorous search, it demonstrates two important points.

1. There is little development of policy which actively fosters collaboration in Australian universities at the present time. Without such policy a culture of sharing and reuse is unlikely to flourish.

2. Examples of collaboration can be found. It is important that where these educational collaborations exist, they are carefully evaluated to see whether and how such activities lead to increased sharing and reuse of any educational electronic resources.

'All words' search on 23 July 2002	Site and no. of results	Comment
Collaboration policy	DEST 5	Four of these are research related, rather than related to teaching and learning policy; the fifth is McNaught et al. (2000), which is largely an analysis of existing issues.
Collaboration process	DEST 6	One is research related, one is a 1992 curriculum document, one is about academic writing support; one is an equity report, and the fifth is on youth issues. The final is Taylor & Richardson (2001), which is a proposal for peer review of ICT resources. If implemented (and there is no certainty about this), this proposal could support reuse.
Collaboration policy	EdNA Australian sites. 20 out of 382,331 in the collection	Eight of these are focused more on research, industry or the general community. The other 12 do provide examples of organisations/ networks which illustrate collaborative policy.
Collaboration process	EdNA Australian sites. 19 out of 382,331 in the collection	Eleven of these relate to teaching and learning in some way (the others relate more to research or industry); a couple of examples of actual collaborative processes exist here.

One government initiative in Hong Kong is worth sharing because of its explicit recognition of the importance of inter-institution collaboration. In Hong Kong there are eight higher education institutions (see http://www.ugc.edu.hk/english/fund_inst.html [verified 5 Feb 2003]), each with a distinctive character. To date, there have been three rounds of Teaching Development Grants which are awarded on a competitive basis. These are substantial grants in the range $HK1-10M. The criteria on which grant proposals are judged are impact (60%), outcomes (20%), collaboration (10%) and alignment with institutional goals (10%). Within the collaboration category, the following scale is used,

5.	The project involves two or more institutions, all of which have contributed to the project design and will have a significant and well-defined role in the project's implementation and resulting benefit.
4.	The project involves two or more institutions, all of which have contributed to the project design, have a significant and well-defined role in its implementation, although the benefit for involved institutions is not equivalent.
3.	The project was designed by the submitting institution, but more than one institution will have a significant and well-defined role in its implementation, but benefit for all participants will not necessarily be equivalent.
2.	Individual staff members from outside the submitting institution appear to have contributed to the project design, but only one institution has a significant role in its implementation and will in all likelihood be the prime beneficiary of input upon completion.
1.	The project involves individual staff members from outside the submitting institution, but they appear to have only a pro forma involvement and thus benefit to their institution is questionable.
0.	The project does not involve more than one institution and benefit is restricted to the submitting institution.

While this might mean that some 'surface' collaborations are formed at grant writing time, this policy has resulted in several real collaborations where resources are produced and shared more widely.

Summary

References

Australian IMS project. http://ims.edna.edu.au/ [23 July 2002].

Australian Literature Gateway. http://www.austlit.edu.au/ [23 July 2002]

Bell, M., Bush, D., Nicholson, P., O'Brien D. & Tran, T. (2002). Universities Online: A survey of online education and services in Australia. Department of Education, Science and Training Higher Education Group, Occasional Paper Series 02-A. http://www.dest.gov.au/highered/occpaper/02a/02_a.pdf [23 July 2002, verified 5 Feb 2003]

Coates J., & Rowsell, A. (1998). Plastic surgical training and the Internet: How plastic surgeons learn. British Journal of Plastic Surgery, 51, 74-77.

EdNA Online (Education Network Australia). http://www.edna.edu.au/ [23 July 2002]

Goschnick, S. G. (1998). Design and development of Melbourne IT Creator TM - A System for authoring and management of online education. http://www.solidsoftware.com.au/Information/Paper/Tools98/Creator.html [23 July 2002, verified 5 Feb 2003]

Harper, B., Oliver, R. & Agostinho, S. (2001). Developing generic tools for use in flexible learning: A preliminary progress report. In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds), Meeting at the Crossroads. Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education. (pp. 253-262). Melbourne: Biomedical Multimedia Unit, The University of Melbourne. [23 July 2002, verified 5 Feb 2003] http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/harperb.pdf

InfoCat (DEST 'Information Catalogue'). http://infocat.dest.gov.au/IE/

Johnson, K. (1998). Readability. [23 July 2002, verifed 5 Feb 2003] http://www.timetabler.com/readable.pdf

Johnston, I. & Peat, M. (2002). Scholarly inquiry and flexibility. In Scholarly inquiry in flexible science teaching and learning, (pp. 1-2). Proceedings of Symposium, 5 April. UniServe Science: The University of Sydney. [verified 4 Feb 2003] http://science.uniserve.edu.au/pubs/procs/wshop7/schws001.pdf

McNaught, C., Kennedy, D. & Majoor, J. (2002). Designing online learning sites to cater for learners' needs. In Proceedings of the International Conference on Computers in Education (ICCE 2002). Auckland, New Zealand, 3-6 December.

McNaught, C., Phillips, P., Rossiter, D. & Winn, J. (2000). Developing a framework for a usable and useful inventory of computer-facilitated learning and support materials in Australian universities. Evaluations and Investigations Program report 99/11. Canberra: Higher Education Division Department of Employment, Education, Training and Youth Affairs. [verified 5 Feb 2003] http://www.detya.gov.au/highered/eippubs1999.htm#99_11

McNaught, C., Whithear, K. & Browning, G. (1994). The role of evaluation in curriculum design and innovation: A case study of a computer-based approach to teaching veterinary systematic bacteriology and mycology. In K. Beattie, C. McNaught & S. Wills (Eds), Interactive multimedia in university education: Designing for change in teaching and learning, (pp. 295-308). Amsterdam: Elsevier.

McNaught, C., Whithear, K. & Browning, G. (1999). Systems not projects: Focusing on evaluating the overall student experience, rather than isolated innovations. Higher Education Research & Development, 18(2), 247-259.

McNaught, C., Whithear, K., Browning, G., Hart, G. & Prescott, J. (1998). The best of both worlds: Redeveloping a multimedia project for the web. In T. Ottmann & I. Tomek (Eds), Proceedings of Ed-Media & Ed-Telecom 98. 10th World Conference on Educational Multimedia and Hypermedia and World Conference on Educational Telecommunications, volume 1, (pp. 946-951), Freiburg, Germany, 20-25 June.

Open University (1998). Technology Strategy for Academic Advantage. [23 July 2002, verified 5 Feb 2003] http://www2.open.ac.uk/ltto/tsaa/index.htm

Pearson Education Global Publishing. http://www.pearsoned.com/globalPub.htm [23 July 2002]

Prater, M. A. & Smith, D. J. (1989). Determining undergraduate curriculum content in plastic surgery. Plastic Reconstructive Surgery, 84, 529-533.

Shazly, E. L., Mohamed, M., & Maiwald, G. (2000). The Internet: A new friend to plastic surgeons. Plastic Reconstructive Surgery, 106, 235.

Taylor, P. G. & Richardson, A. S. (2001). Validating scholarship in university teaching. Constructing a national scheme for external peer review of ICT-based teaching and learning resources. Evaluations and Investigations Program report 01/3. Canberra: Higher Education Division Department of Employment, Education, Training and Youth Affairs. [23 July 2002, verified 5 Feb 2003] http://www.dest.gov.au/highered/eippubs/eip01_3/01_3.pdf

UniServe Science. http://science.uniserve.edu.au/ [23 July 2002]

Van Heijningen, R. I., Mannaerts, G. H. H., Blondeel, P. H. N. & Spauwen, P. H. M. (1998). PLink, Plastic surgery and the Internet. British Journal of Plastic Surgery, 51, 86-89.

Whithear, K. G., Browning, G. F., Brightling, P. & McNaught, C. (1994). Veterinary education in the era of information technology. The Australian Veterinary Journal, 71, 1-3.

This article was nominated for an Outstanding Paper Award at ASCILITE 2002, gaining the additional recognition of publication in AJET (with minor corrections). The reference for the Conference version is: McNaught, C., Burd, A., Whithear, K., Prescott, J. and Browning, G. (2002). It takes more than metadata and stories of success: Understanding barriers to reuse of computer facilitated learning resources. In A. Williamson, C. Gunn, A. Young and T. Clear (Eds), Winds of Change in the Sea of Learning: Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, pp451-460. Auckland, New Zealand: UNITEC Institute of Technology. Authors: Carmel McNaught, Centre for Learning Enhancement and Research, Chinese University of Hong Kong, Hong Kong. carmel.mcnaught@cuhk.edu.hk Andrew Burd, Department of Surgery, Chinese University of Hong Kong, Hong Kong. andrewburd@cuhk.edu.hk Kevin Whithear, Faculty of Veterinary Science, The University of Melbourne, Australia. k.whithear@vet.unimelb.edu.au John Prescott, Ontario Veterinary College, University of Guelph, Canada. prescott@ovcnet.uoguelph.ca Glenn Browning, Faculty of Veterinary Science, The University of Melbourne, Australia. g.browning@vet.unimelb.edu.au Please cite as: McNaught, C., Burd, A., Whithear, K., Prescott, J. and Browning, G. (2003). It takes more than metadata and stories of success: Understanding barriers to reuse of computer facilitated learning resources. Australian Journal of Educational Technology, 19(1), 72-86. http://www.ascilite.org.au/ajet/ajet19/mcnaught.html