The increased pressures on Universities to have demonstrable accountability measures in the form of quality assurance systems, coupled with the increasing use of flexible delivery technologies, produce a new environment in which to carry out the evaluation of teaching. This paper discusses one component of the evaluation of teaching - the evaluation of teaching by students, via online methods, at the University of South Australia. The advantages of this approach are described.
The results of the use of online evaluations in a large first year Art subject are presented and are compared with 'pencil and paper' instruments previously used. Examples of issues that needed to be addressed are discussed. The benefits and risks to students, staff and the institution as a whole are considered. Finally, future developments planned are outlined.
If you are about to apply for a place at university, you will realise that this is one of the most important choices that you will make. This site is designed to help prospective higher education students make informed study choices. It is one of a series of ten, one for each broad field of study. The site is a guide to Australian universities and the courses that they offer within each broad field of study. The site also gives information about the employment and study outcomes for past graduates and how they felt about their courses. This site is one important source of information. You should also seek out other information and advice so that your choice best suits you, taking into account your own study and career interests, your goals and ambitions. I wish you good luck with your future studies.
A cycle of evaluation and improvement based on student feedback is a fundamental component of the process of quality improvement in universities (Ramsden 1998). Whilst this fits well with a 'consumer' orientation of education, the question remains of how we define quality and how best to use information from students to improve it. Any evaluative mechanism can be shown to reflect particular constructions of 'good teaching' (Martens and Prosser, 1998). The Australian scene in this area is dominated by the Course Experience Questionnaire (CEQ), (Ramsden, 1991), and variants of it have been applied in many and various contexts (Santhanam et al, 2000). Perhaps its most powerful application is its use by the Good Universities Guide (Ashenden & Milligan, 2000) which, with its mix of student feedback on teaching quality and competitive rhetoric provides powerful reasons for universities to consider course quality within this framework. Add to this the advent of a Commonwealth Government quality agency, aiming to improve universities' 'international competitiveness', and universities are bound to respond in some way. In the words of the current Minister of Education (Kemp, 1999b):
Australia is part of a global community delivering higher education and the increased emphasis on quality assurance is a global phenomenon. We must have a national quality assurance framework that is internationally credible.It must also be recognised that the nature of learning environments, in which students find themselves, is rapidly changing. Key to this is the concept of flexible learning. Nicholl (1998) says the 'term "flexible learning" seems to be used increasingly in relation to the kinds of reformulations of course offerings' and goes on the critically review the way government policy focuses 'on the potential usages of distance, open, mixed-mode or flexible learning strategies'. (p291) While Nicholl challenges the role of policy on the theory of teaching, the argument for allowing students choices, being able to choose when, where, how they study is clearly what 'flexible delivery' is about. Flexible delivery is about teachers making use of a variety of teaching methods and resources to enable students to learn in a way that suits them. Online materials are one method of "flexible learning" that a university can choose to offer a greater range of resources for students. The online evaluation of courses seems congruent with online offerings, especially to an increasingly information technology literate cohort of students, even for subjects taught in traditional ways.
Not only is the general learning environment changing, but in addition the profile of the student body is undergoing a transformation. Smith and Webster (1997) predict the fate of the on-campus offering in the information age:
One should remember that the new technologies may now begin to be an increasing factor in decreasing demand for residential education. The frequently evoked scenario of the 'virtual university', in which students learn from and interact with leading academics irrespective of distance, is now, in principle at least, a practical possibility, even a reality (Laurillard, 1993). This technological solution to spiralling costs in higher education will have an inevitable appeal - perhaps excessively - to policy-makers. (Smith and Webster, 1997 pp.12-13)While Smith and Webster (1997) are critical of the pressure to get tertiary education 'online' in order to cut costs, others are more pragmatic. Martin (1999) points out that the pace of change in the information age has left teachers feeling undervalued and overworked. This has been accompanied with a 'decline in the traditional collegial ways of working' (p4), while universities need to become good at learning in order to cope with change. Martin claims that a proactive attitude to teaching is the only viable option and goes on to say:
'We can wait a long time for support and improved circumstance to come our way. Meanwhile, our lives may be dismal and depressing and we have only one life. There is much wisdom in the old adage that help comes to those who help themselves'. (p48).Spender (1996) is even more forceful in her championing of the online environment to the point where she says that 'the dividing line between teaching and learning ceases to become useful.' (p4)
There can be no doubt that universities are feeling these pressures. They are being exerted on learning organisations around the world. The processes of evaluation are not immune from their effects. As an example of how these pressures are being embodied in evaluative processes, we now move to describe an online evaluation instrument developed at the University of South Australia. This allows students enrolled in any subject across the university to evaluate the teaching in that subject, via a survey instrument prepared online by teaching staff. Staff are able to customise the instrument for particular learning contexts or to address particular quality improvement concerns. It can be seen as a response to the social and economic climate in which universities find themselves, utilising technologically mediated methods, and providing students and staff with flexibilities in operation that are fundamental to current teaching and learning approaches (Hicks, Reid and George, 1999). We now turn to a particular instance, of the teaching in a large first-year subject in the Art discipline.
The subject was taught in 1997 in face to face mode and evaluated with a paper and pencil survey. In 1998 some components of the subject were developed for online delivery and a simple web form was used to gain student responses. The evaluation instrument that is the subject of this paper was first trialled in 1999 when the subject was delivered entirely online, and in 2000 it will be used again when the delivery mode will be online, supplemented by face-to-face tutorial support. The online features were delivered initially by stand-alone web resources and progressively used the interactive features of the University's online platform, UniSAnet (Reid, 2000). A recent addition to the online tools in UniSAnet has been the online evaluation tool, TellUs. This is now described.
Based on these requirements, the TellUs evaluation tool was developed. It has the following components:
Figure 1: TellUs survey construction
Figure 2: TellUs participant survey
Figure 3: TellUs data analysis
Paper and pencil method is difficult for a small number of staff with limited time to aggregate the responses to the summative questions and make more than a superficial interpretation of the formative question responses.
During the 1998 delivery a web form/email SES was used to evaluate the subject, using an on-line form with the student response as an email generated by the server. This addressed the anonymity issue but not the verification of the student and still required manual aggregation.
The SES for 10447 during 1999 used the instrument developed by the FLC, now called TellUs, for the first time. The possibility of easily polling the students via the instrument prompted the subject team to try the SES twice, once about halfway through the semester and again at the end. The initial SES was very successful in numbers of responses, while the second was disappointing. The main reason for the poor response to the second SES seemed to be that the students expect to do the evaluation only once, as is usual in other subjects and 'curriculum overload' at the end of semester. In response to student comments in the first of the two SESs, part way through the semester modified the online study guide format. This provided a very useful way of acting on student feedback while teaching the subject, rather than waiting for the completion of teaching. UniSAnet allows for several study guides to be developed and each made available as needed.
Overall feedback from the students from the on-line SES showed the main issues were problems with access and lack of independent learning skills (related to transition to university learning). Jones (1996) refers to Chin and notes that these issues are typical of students' perceptions at the early stage of the web-based learning environment.
The replies to the formative questions show a tension between the freedom and flexibility studying on-line, and the structure that a traditional lecture/tutorial study mode offers. For example this response from the same student asked about the best and the worst of the subject,
(best) 'I found studying at my own pace, in my own time rewarding and productive.'Access to the Internet was another issue identified. General Purpose student computer pools at the University of South Australia provide access to the subject home page and any links to pages that reside on the University server, however not all students had external access to the Internet. Students in this predicament used the limited library facility or had access at home or visited an Internet café. This situation has since changed with the introduction of an allocation to all University of South Australia students for Internet access from public pools.
(worst) 'I found it a challenge because there are no tutors reminding you of what should be included in assignments and when to hand them in.'
In summary, the use of TellUs provided a feedback mechanism that was easy to administer, and also was flexible enough to use at numerous times throughout the semester. This allowed the teaching to be more responsive to students' needs, even though direct contact with students was not always possible.
Another interesting issue of the desirability of flexible evaluation methods for flexible teaching methods. The use of flexible delivery methods and acknowledgment that individual students learn at a different pace, reinforces the need for evaluation methods that can adapt to the needs of specific students. The TellUs instrument used within the online delivery of subjects enables teachers to get feedback about their teaching materials and students' progress more quickly by automating some of the data collection processes. This increases the flexibility with which the evaluation process can be undertaken.
Whilst there may be some uneasiness about a standardised instrument being used in this way, the broader environment within which we find ourselves, dictates moves in this direction. We certainly feel there is a need to use the flexibilities provided by an online instrument to provide opportunities to evaluate in more comprehensive ways - online or not. We have found that at least the instrument described here is easy to use, reduces work on the part of the teacher, is compatible with modern teaching methods, and can also be used for quality feedback upon which teachers can act for the benefit of their students. This after all, is the point of engaging in evaluative practices in the first place!
Jones, D. (1996). Solving some problems of university education: A case study. Proceedings of AusWeb96, The Second World Wide Conference Gold Coast, Australia [verified 19 Oct 2001] http://ausweb.scu.edu.au/aw96/educn/jones/paper.htm
Hicks, M., Reid I. & George, R. (1999). Enhancing online teaching: Designing responsive learning environments. Paper presented at the 1999 HERDSA conference Cornerstones: What do we value in higher education? Melbourne, 12-15 July. [verified 19 Oct 2001] http://herdsa.org.au/vic/cornerstones/pdf/Hicks.PDF
Kemp, D. (1999a). [verified 19 Oct 2001] http://www.detya.gov.au/tenfields/
Kemp, D. (1999b). Quality Assured: A new Australian quality assurance framework for university education. Speech given at a Seminar on the New Quality Assurance Framework. Canberra 10 December. [verified 19 Oct 2001] http://www.detya.gov.au/ministers/kemp/dec99/ks101299.htm
Martin, E. (1999). Changing Academic Work: Developing the Learning University. Buckingham: Society for Research into Higher Education: Open University Press.
Martens, E. & Prosser, M. (1998). What constitutes high quality teaching and learning and how to assure it. Quality Assurance in Education, 6(1), 28-36.
Nicoll, K. (1998). Fixing "the facts": Flexible learning as policy invention. Higher Education Research and Development, 17(3), 291-304.
Ramsden, P. (1991). A Performance Indicator of teaching in higher education: The Course Experience Questionnaire. Studies in Higher Education, 16, 129-150.
Ramsden, P. (1998). Learning to lead in higher education. London: Routledge.
Reid, I. C. (2001). A university goes online: Avoiding throwing the innovative baby out with the strategic bath water. In L. Richardson and J. Lidstone (Eds), Flexible Learning for a Flexible Society, 574-582. Proceedings of ASET-HERDSA 2000 Conference, Toowoomba, Qld, 2-5 July 2000. ASET and HERDSA. http://cleo.murdoch.edu.au/gen/aset/confs/aset-herdsa2000/procs/reid1.html
Santhanam, E., Ballantyne, C., Mulligan, D., de la Harpe, B. and Ellis, R. (2000). Student questionnaires and teaching evaluation: Cutting the cloth to fit the purpose. In A. Herrmann and M.M. Kulski (Eds), Flexible Futures in Tertiary Teaching. Proceedings of the 9th Annual Teaching Learning Forum, 2-4 February 2000. Perth: Curtin University of Technology. http://cleo.murdoch.edu.au/confs/tlf/tlf2000/santhanam2.html
Smith, A. & Webster, F. (1997). The Postmodern University? Contested Visions of Higher Education in Society. Buckingham: Society for Research into Higher Education: Open University Press.
Spender, D. (1996). Creativity and the computer education industry. [verified 19 Oct 2001] http://www.acs.org.au/ifip96/dales.html
Van Dusen, G. (1997). The Virtual Campus: Technology and Reform in Higher Education. ASHE-ERIC Higher Education Report, 25(5). Washington DC: The George Washington Graduate School of Education and Human Development.
|Contact details: Ian Reid, Coordinator Online Services, Flexible Learning Centre, University of South Australia. Telephone (08) 8302 7074 Fax (08) 8302 6363 Email firstname.lastname@example.org
Please cite as: Reid, I. C. and Welch, A. (2001). Evaluation goes online - What are the issues? In L. Richardson and J. Lidstone (Eds), Flexible Learning for a Flexible Society, 583-594. Proceedings of ASET-HERDSA 2000 Conference, Toowoomba, Qld, 2-5 July 2000. ASET and HERDSA. http://www.aset.org.au/confs/aset-herdsa2000/procs/reid2.html