Australian Journal of Educational Technology
2002, 18(3), 323-340.
AJET 18

Evaluating an online learning environment

Elizabeth Stacey and Mary Rice
Deakin University


Introduction

This paper reports on an evaluation undertaken under the auspices of the cross-institutional CUTSD funded project which was established to facilitate evaluations of computer facilitated learning with an action inquiry model of evaluation (Phillips, 2002). Our evaluation focused on students' learning processes and outcomes in an online learning environment established for postgraduate education students studying an Open and Distance Education Specialism in a Masters program at Deakin University. Online conferencing has been progressively used in this specialism, and was an integral part of the program at the time of this study, which followed students studying online in five units. The units had computer conferences established with FirstClass software for electronic discussion of tasks and activities, with the aim of engaging students in learning interactively. Our participation in the CUTSD project resulted from the need to evaluate the extent to which students engaged in the conferencing environment, as intended by the learning design. It was also designed to evaluate the effectiveness of established practice with a view to modifying it in response to student feedback.

Background

Learning online has enabled a potentially new type of learning community which provides a space for group discussion as well as access to other students for socialising and communication. The widespread use of computer conferencing in tertiary programs (Stephenson, 2001, Fisher, Phelps & Ellis, 2000, Oliver & Omari, 1999) shows the implementation of this potential. Stacey's research (1999) had shown the importance of group collaboration in online learning and her discussion of computer mediated communication, from a social constructivist perspective, has focused on interactive online group discussion as central to the learners' effective construction of new conceptual understandings. Her research found that in the social context of group interaction, the collaborative group develops a consensus of knowledge through communicating different perspectives, receiving feedback from other students and teachers, and discussing ideas, until a final negotiation of understanding is reached. Drawing on Vygotsky's (1978) theory that conceptual understandings are developed through verbal interaction, Stacey found that a socially constructed learning environment is essential for effective learning.

The refinement of the factors contributing to the establishment of an effective online learning environment showed the importance of communication that was affective as well as cognitive and the role of the teacher in establishing an online community where social presence is established through teacher modeling, discussed in the work of Salmon (2000) and Garrison, Anderson and Archer (1999), was the framework of research within which this evaluation occurred. Garrison, Anderson and Archer have defined the online group as a critical community of inquiry and have established a framework of analysis of the community into three elements: cognitive presence, social presence and teacher presence. These factors of analysis were similar to the content analysis framework defined by Stacey and were used to modify and explore the focus of evaluative analysis applied to this study.

Online learning in open and distance education units

The Open and Distance Education course had used computer conferencing since 1993 to teach and support distance students. Conferencing was established to provide opportunities for student interaction, and to break down the isolation of their distance learning experience. It was also a medium under research in distance education and as such was a way of providing students in this field with a means of analysing its use and effects on distance education. Since 1995, the electronic conferencing software, FirstClass, had enabled the establishment of a virtual campus in the Education faculty for most students studying postgraduate coursework degrees and higher degrees by research. (Stacey, 1997). Postgraduate students studying in the Open and Distance Education specialism used computer conferences as their main communication process for group interaction and their use had been integrated into our teaching in a range of ways.

This study will focus on a unit within the specialism which used computer conferencing as the central form of communication and learning. The unit required students to use the FirstClass environment to communicate and to access and share resources, both those provided and those they researched and evaluated through searching the World Wide Web. As the content of the unit was about online learning, students moderated discussions about issues of online learning, and worked in collaborative groups for an assessed task on researching the theory and process of collaborative learning online. In evaluating the effectiveness of computer facilitated learning in the medium of computer conferencing this study used a range of methods for data gathering and analysis .

Though we had previously used the online environment to discuss and reflect on students' evaluation comments, no overall attempt had been made to rigorously evaluate the online learning processes[1] and learning outcomes[2] in this specialism. The acknowledgement of the need for this prompted our participation in the CUTSD funded project. Time and expertise constraints are issues described by Taylor et al (2000) who describe a case study of "integrative evaluation" of an online course. Their method used the surrounding materials and activities as a means of evaluation via observation, interviews and questionnaires, web -administered questionnaires, paper questionnaires posted out, use of the computer conferences and by using the programming environment to gather data about student use. They found that the course team cannot always cope with the volume of evaluation data generated by a large course and can often be de-motivated by critical voices. Involving a mentor or evaluation team was a way of helping to identify what problems need solution and acting on these. In our case working as a team with a mentor helped us use our teaching time efficiently to ensure evaluation data were gathered and included in new development or to drive changes to ensure quality assurance.

Description of project

This evaluation focused on aspects of the implementation phase of the learning centred framework adapted from Alexander and Hedberg (1994) and Bain (1999). Curriculum analysis indicated that opportunities for off campus students to interact and engage in discursive activities were limited, therefore knowledge construction was often an individualised endeavour. FirstClass conferencing software was adopted to address this problem for reasons outlined earlier.

Previous monitoring of the environment focused on the scale of the technical and support structure needed (Goodwin, Rice, Stacey & Thompson, 1995). This work demonstrated that students found FirstClass easy and enjoyable to use. It reduced the isolation so often felt by off campus students and made them feel part of a community. Providing the technology infrastructure was stable, a majority of students found it easy to access resource materials and carry out the required discursive activities. Over 80% of students believed that CMC facilitated their learning through small group teamwork processes, though at this stage, no other data was gathered to affirm these beliefs.

The focus of researching the roles of the social, cognitive and teacher presence (Garrison, Anderson & Archer, 2001) described above defined the framework of the study and with that background, the present project was designed to evaluate the effects of the use of computer conferencing on students' learning. During the semester, data were gathered electronically, using the methods described below. The project used qualitative methods to gather students' perceptions and reflections on the effect the conferencing process had on their learning. Quantitative data was collected through analysis of the frequency and type of messages which occur on the conferences. Final student results for the units were also reviewed.

Methodology

Consistent with the approach suggested by Phillips et al (2000), an action inquiry process of plan, act, describe, and review was used as an underlying framework for this evaluation. This allowed practitioners to adjust methods and schedules in response to contingencies that arose.

Project participants

Students were invited to volunteer for the evaluation project through the unit electronic conferences on FirstClass. The final evaluation group were 17 part time students, 3 men and 14 women, most aged between 40 and 50 years, and working full time. They were widely dispersed geographically across Australia and overseas, and were therefore studying off campus with no opportunity for meeting face to face. All students were required to interact as a whole group as well as self selecting into four small groups during the semester. Online learning offered them group interaction more conveniently than studying on campus and enabled them to gain access to university study in a way that overcame their distance from the Deakin campus.

The team evaluating the project included the teacher and though this meant there could be response to student comments in a formative way, though the evaluation was ultimately a summative evaluation of the implementation of the computer facilitated learning (see Table 1). This meant that, as prescribed by ethical requirements of the university, most of the evaluation analysis occurred after the semester was complete and grades were recorded so that participants were able to respond during the semester in an open and unpressured way.

Evaluation objectives and questions

Evaluation objectives, questions and methods are summarised in Table 1.

Table 1: Evaluation model

ObjectivesEvaluation questionsMethods for all questions
To investigate the effectiveness of the computer conferencing approach by analysing patterns of online interaction and the interconnection between the cognitive, affective and system structures of the conference environment
  • How is computer conferencing used in teaching and learning for sharing ideas and constructing knowledge?
  • How do the students in the unit interact online?
  • Voluntary online focus group conference and individual emailed responses.
  • Online observation
  • Student results analysis
  • Conference analysis
  • Message frequency and distribution
To investigate the ways in which the learning processes and learning outcomes of the students studying the described units were affected by the use of computer conferencing.
  • How have students perceived the effect of online interaction on their learning?
  • Voluntary focus group and individual emailed interview responses
  • Summative online discussion

The project was designed to investigate the effectiveness of the computer conferencing approach in the unit by analysing patterns of online interaction and the interconnection between the cognitive, affective and system structures of the conference environment. The project also investigated the students' perceptions about the their learning processes and learning outcomes and how these were affected by the use of computer conferencing.

Data collection methods

Several methods were used because of the complexity of interactions within discursive environments in education and the chaotic nature of variables. Because of the need to summarise and analyse discursive data, the major emphasis of the evaluation has been on interpretive, qualitative approaches such as those described by Patton (1990). However, there was also a need to quantify particular categories of responses to determine how widespread they were. Therefore, usage statistics and response frequencies were used to test the credibility of students' comments and provide opportunities for triangulation of data.

As outlined in Table 1, the initial data collection methods included the following:

  1. A voluntary online focus group conference was established specifically for the project. Students were asked to respond both to questions about their experiences of learning online, and to reflections about online practice posted in this area during and at the end of the semester. These questions were also emailed to any student who had not participated in the focus group conference.

  2. Online observation was used to analyse communication and learning processes through ongoing response to student comments, and retrospective analysis of conference message archives.

  3. Analysis of conference message content was used to determine the students' learning processes online.

  4. Calculation of frequency and distribution of message use was used to establish communication patterns and as a form of triangulation of data.

  5. Comparison of students' results and interaction frequency to see if simple trends of interaction rate and outcome were interrelated.

  6. A summative online discussion was held three months after semester ended. Students were asked to comment on a summary of previously analysed findings which were posted on a short term computer conference set up for the purpose.
In keeping with an action inquiry approach, planned evaluation methods and schedules changed somewhat as the evaluation progressed. After the initial focus groups were set up, it became obvious that this was not a way all students wanted to report their reflections. Many students preferred to send emailed comments privately to the team so questions were sent individually to some participants with a much higher rate of return. The team also decided that using the medium under evaluation, a special computer conference, would be a suitable way to gather summative evaluation data in an interactive discussion. This meant that the final online evaluation discussion was a form of reflective practice as students consciously discussed the already analysed results of their focus group and emailed deliberations. The action evaluation model of the project enabled us to be open to different integrative evaluation processes during the study while encouraging us to consider new and different ways of evaluation.

Evaluation results

The results will be described in relation to the research questions.

1. How is computer conferencing used in teaching and learning for sharing ideas and constructing knowledge?

The focus group evaluation responses showed that the majority of students had been using the Internet for a few years and were often self taught or had had little formal training. Many used computers and online communication in their work and as a resource base for information. Computer conferencing was new to most students and some students noted that learning to use the FirstClass conference was a new process for many of them which initially made them feel insecure and uncomfortable. They thought that a structured process of learning steps was helpful at the beginning of a course. Most saw the potential of the online conference as an interactive interchange of ideas between participants. The students were appreciative of the required interaction as a way of making them engage more actively with the content of the course. It enabled them to construct their own understanding of the course content with feedback from other participants to assist that construction.
"...it provided me with the opportunity to construct my own knowledge/ understanding within my own context. Meant I had to revisit readings, interpret what others were saying in the group, respond, evaluate." Female student, <50, international
The different perspectives provided by the different students were particularly seen as an advantage to their learning as it took them out of their own more limited view of the subject (often with difficulty).
"Yes as it brought other people's perspectives to the issues raised which wouldn't occur by completing an assignment alone. our group had participants from England, Korea, Melbourne and Alice Springs which all had a wide range of diverse views and experiences in education " Male, 50, rural
Students found that the other participants challenged their ideas and provided new thinking.
"I was forced to think laterally and compromise my preferred style of learning. It forced me to take my blinkers off and open my mind to other interpretations. At first I was intolerant of other opinions that did not mesh with mine. Seeing words in black and white in front of you seems a lot louder than a voice." Female, >50, rural.

"..extended me in new areas, made me think about what I was saying and then had to rethink it when others took it differently. Gave me not only one experience but the experience of many to look at issues." Female, 40, international.

Often students compared online learning and face to face learning, judging CMC as more engaging and flexible environment for their learning.
"I had to really nut things out in order to feel confident in discussing things with the other members of my group. Putting your thoughts down in written form seems to require much more effort than face to face discussions." Female, 35, metropolitan.

"It challenged me to think about the questions and people's responses - CMC has certain advantages over the face to face tutorial in that here we often get EVERYONE'S response to a question - in a face to face class you would probably only hear from one or two people and then the tutor. CMC allows more reflection than in the face to face setting - we have time to read each other's comments and respond to them in our own time. We can come back to something later and respond to it - these opportunities rarely exist in face to face classes." Female, 35, rural.

"Had more time to think through the issues as there was no need to respond immediately. Often went back and reread materials to clarify issues and having this bank of materials readily available was helpful. Male, 50, rural."

The ability to "benchmark" their learning, to find out how others learned online was another advantage not always possible at a distance.
"Something interesting to me that did emerge was an understanding of how others think and work at a Masters level. This is the first time l have been able to observe peers at work, their commitment, their depth of involvement, their professionalism and actual research and language skills." Female, 40, international.
The role of the lecturer as conference facilitator, regularly interacting online was seen as essential to maintenance of activity and focus by students. How this role could be established and the effects of modeling online communication strategies were commented on by students and was a focus in the content analysis described below.
"The quality of the interaction is also affected by the frequency of the lecturer. Another site I attend has no lecturer involvement and the site feels "less relevant and important" Female >50, international

"The lecturer/ tutor has the ability to encourage and expand the learning base of all students when there is regular dialogue among the group." Female, 50, metropolitan

2. How do the students in the unit interact online?

Message frequency and student results

The teacher's online messaging total combined with the 17 students in the evaluation focus group totaled 1281 messages on all parts of the conference during the semester. These included whole group discussions, discussions of issues by topic as well as the small collaborative group discussions. As the unit required online interaction, frequent interaction could be measured by high message frequency, though this message tally gave no indication of the quality and length of the messages. Though comparison of message frequency and student results (see Table 2) could not be interpreted as providing conclusive evidence of the effect of online interaction on learning outcomes, the highest achieving students were also highly interactive, particularly in their small group interaction and lower achieving students were less active online. Only one very interactive student receiving a pass level (relating more to external factors) while all other frequently participating students gained a grade higher than a pass. Student 9, who was slightly less interactive due to family commitments restricting her less interactive time online, received a High Distinction due particularly to the quality of her interactions which though less frequent were prepared carefully and supported by extensive offline research. The failing students in the unit had also failed to interact, their absence online reflecting their lack of engagement with the course through group interaction which provided feedback from other students and from staff. Their consequent lack of submission of their assignments resulted in failure. Only one student had failed to complete the course after an interactive start and this was due to external work related relocation. Table 2 summarises the overall participation and results comparison.

Table 2: Comparison of whole group messaging frequency and semester results

StudentDemographics -
gender, age, location
Whole
group total
Small
group total
Total message
frequency
Results*
1Female, 41, overseas5093143High Distinction
2Female, 40, overseas7256128High Distinction
3Female, 51, local city512374Distinction
4Female, 51, overseas313970Distinction
5Male, 50, remote rural194261Distinction
6Female, 38, remote city276390Pass
7Female, 46, remote city243559Distinction
8Female, 58, remote rural233659Distinction
9Female, 35, local city223658High Distinction
10Female, 48, remote rural213051Distinction
11Female, 47, remote rural292150Distinction
12Female, 56, remote rural192746Distinction
13Female, 35, local city165369Distinction
14Male, 47, local city14519Pass
15Female, 34, local city12719Pass
16Male, remote regional11011No Assessment
17Female, 57, overseas101No Assessment
* HD = 80-100%, D = 70-79%, C = 60-69%, P = 50-59%

Message nature and content

Researchers have attempted to analyse communication, learning strategies and patterns of interaction in computer conferencing through content analysis and categorising of the text generated when messages are sent to computer conferences. Categories and methods are generated in many different ways depending on the focus of the research. Henri (1993) analysed messages into units of meaning and attempted to measure social dimensions, interactivity, cognitive skills, levels of processing and metacognitive knowledge and skills which were critiqued and developed by Gunawardena, Lowe and Anderson (1997) into a five phase constructivist interaction analysis model. Kanuka and Anderson (1998) applied this preliminary model successfully, suggesting modifications, and McLoughlin and Luca (1999) also adapted this model of analysis for a learner centred use of computer conferencing. Stacey in an earlier study (1998) categorised and calculated the online messages in her study into three types: course content, process of learning the technology and group learning and support before analysing the online discourse and other data into a model of attributes of online collaborative group learning. These attributes included clarification of ideas, feedback to ideas, diverse perspectives, group solutions and group resource sharing as well as factors of socio-affective collaborative support.

In this study the required online interaction in this unit was analysed focusing on the way the teacher established a model of social interaction through use of social presence factors. The level of cognitive engagement was analysed particularly through the continuing patterns of interaction and communication when students began to work mainly in small collaborative groups. This analysis used a categorisation that labeled units of meaning within each message for its primary purpose and content into:

The conference analysis focused on the development of social presence as a modeled and learned process. The first month of the main introductory conference was the space where students were asked to introduce themselves to the group and where the whole group established their online relationships before breaking into small groups. A period of the first four weeks of semester was analysed as well as two later periods (weeks 7 and 12) when small group conferences had been developed.

The teacher's role in the first week of semester, establishing a secure learning environment and modeling socially accepting processes of interaction, was shown to be a major factor in increasing the frequency of social presence factors in the whole group conference, as students in the second week of semester followed teacher direction and practised using aspects of the software while providing personal contextual information. There was a rise in the level of social presence factors in week 2 because, though the student participation rate did not vary from the other weeks analysed, their messages followed the modeled and explained process for establishing social presence exhibited in the analysed factors of social presence.

Small collaborative groups were established in this unit to facilitate continued group discussions and tasks. Social presence factors continued to be important in the communication of these groups, with high frequencies of interactive and cohesive units continuing to appear within messages and even rising towards the end of the semester. Even though cognitive content became a main focus in the group's interaction, the social interaction continued to be an important factor within a less formal space. As group participants negotiated over content, they interacted with one another's message text, asked questions and agreed with and complimented the others' ideas, an important social component of effective collaborative and cognitive learning.

3. How have students perceived the effect of online interaction on their learning?

Overall the majority of participants found the computer conference system used for online learning was enjoyable, easy to use, responsive and reliable and that conferencing was a positive experience. They thought that conferencing partially overcame the disadvantage of distance education in not being able to discuss things with others and that in their busy lives there was an advantage in the flexibility of access and times of study. They described how the group communication in the developing online community provided a motivation for learning and how they enjoyed the interaction which reduced their usual isolation in distance learning. They had a sense of community, particularly in the sharing of resources.
"I worked really hard, much harder than I am working on my present course because I felt a connection with my peers and felt I owed it to them to be on top of the conversations, contributing when I could and commenting on their thoughts when needed. " Female student <40, capital city
The shared resource base was seen as a great advantage of this type of learning as web resources have increased to such an extent that a group process of research and commentary on web sites provided students with a much better resource base than they could find themselves.
"The contributions from various others broadened the pool of resources to check and utilise. I was continuously grateful for the excellent resources offered by group members. I was able to access many valuable readings due to the industriousness of my colleagues. My own resources were okay, yet l found that sharing this task of finding materials gave a varied edge to the readings." Female, 40-50, international
One student summarised the advantages of interacting in an online environment as:
sense of learning community
discipline of regular feedback/responses to FirstClass
quick feedback from colleagues in the group
quick & easy access to Tutor
one place to go for group contact
less expensive than attending university campus
(male >50, metropolitan WA)
Most students recognised the value of group interaction though a few students elected to work independently on their assessment. They did identify the increased time spent on the subject as a disadvantage though this was an element that the students usually saw as a choice and as a self management issue. Some distance students complained that they chose to work at a distance as they preferred their independence and the ability to work at their own pace and did not learn well in groups, though they understood the advantages of the medium in their learning.

In the summative online discussion, students and the evaluation team participated in a month long conference which generated 94 messages, 59 of which were from students. The discussion occurred 3 months after the semester of study and the project participants had since joined other courses which had been taught in a variety of ways, both with and without computer conferencing,. This resulted in some comparative reflections which were a rich source of data. Students reported working harder than in other distance subjects as they were accountable to the group and engaged more with reading and reflection on a wider range of resources than they would have consulted alone. They were unable to just do the minimum amount of work for the assessment when they were asked to contribute to an ongoing discussion. Overall, the whole body of students were very positive about the results reported and saw many advantages in learning online and raised some useful issues for improvement. These included:

Some students thought it was hard to challenge ideas online, as they tended to be cautious of appearing confrontational. However, the majority of the students were convinced that conferencing was worth the effort, with the responsibility and accountability to the group helping their learning and engaging their interest while teaching them resource access skills.
"Having feedback is a motivator in itself, it's wonderful to have people acknowledge your ideas, you feel less isolated and bouncing ideas off one another is an important way to learn. Different experiences by people can either confirm or allow you to question the views you may have on a certain." Female, >40, international.

Conclusions

The intensity of evaluating in a team such as this cannot always be replicated in normal teaching. However the use of the online environment as a means of supplying constant formative feedback through selective content analysis at key points of the semester in the weeks indicated above provides the teacher with the means of awareness of the social and teacher presence in the online environment. Reflective summative discussion again using the conferencing environment is a manageable means of evaluation. The main findings of the evaluation have given the team an insight into how effectively computer conferencing has been incorporated into this postgraduate course and how well students are learning this way. Given similar online structures and contexts, such results could be expected in similar online postgraduate courses.

The effects on the learning processes and learning outcomes of the students have been analysed using a range of evaluation methods. In summary we found:

The involvement in the CUTSD evaluation project was an impetus and support for our team in evaluating and researching our computer conferencing in a rigorous and detailed way. The multiple methods of evaluation meant that we were able to triangulate student's individually expressed perspectives with content analysis and frequency statistics. A final summative discussion of results confirmed the results gathered which gave us a great deal of insight into how effectively computer conferencing had been incorporated into our course and the ways this was affecting student learning.

The role of the teacher in structuring and establishing cognitive and social presence of students studying online was defined more clearly through this study. The teacher must establish a secure interactive environment through modeling communicative behaviours while establishing online teacher presence. The online teacher's initial intense interaction can be followed by a teacher structured but student led online environment in which the teacher facilitates social and cognitive presence through carefully devised group tasks. The importance of providing time and activities for establishing social presence in an online learning environment should be considered in any new program being developed for online delivery. The cognitive and social strategies students use to learn online which were identified in this evaluative study have impacted on the program's development and have implications for similar courses taught in an online environment.

Endnotes

  1. Learning processes refers to all cognitive activities that contribute to learning, (e.g.problem solving, reflection) as well as the manner in which these acticities are carried out (e.g. individually, in groups, teacher or student-directed).
  2. Learning outcomes refer to the things students are able to do as a result of their engagement in a course of study. This includes both discipline-specific and generic skills.

References

Alexander, S., & Hedberg, J.G. (1994). Evaluating technology-based learning: which model. In K. Beattie, C. McNaught, & S. Wills (Eds), Multimedia in Higher Education: Designing for change in teaching and Learning. Amsterdam: Elsevier.

Bain, J. D. (1999). Introduction: Learning centred evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.

Evans, T. & Nation, D. (2000) (Eds). Changing university teaching: Reflections on creating educational technologies. London: Kogan Page

Fisher, K., Phelps, R. & Ellis, A. (2000). Group processes online: Teaching collaboration through collaborative processes. Educational Technology & Society, 3(3). [verified 22 Aug 2002] http://ifets.ieee.org/periodical/vol_3_2000/f06.html

Garrison, D.R., Anderson, T. & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105.

Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: SAGE Publications.

Gunawardena, C. N., Lowe, C. A. & Anderson, T. (1997) Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397-431.

Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed), Collaborative Learning Through Computer Conferencing. London: Springer-Verlag pp 117-136.

Jonassen, D., Prevish, T., Christy, D. & Stavrulaki, E. (1999). Learning to solve problems on the web: Aggregate planning in a business management course. Distance Education, 20(1), 49-65.

Kanuka, H. & Anderson, T. (1998). Online social interchange, discord and knowledge construction. Canadian Journal of Distance Education, 13(1), 57-74.

McLoughlin, C. and Luca, J. (1999). Lonely outpourings or reasoned dialogue? An analysis of text-based conferencing as a tool to support learning. Proceedings of 1999 ASCILITE Conference, Brisbane, December. [verified 22 Aug 2002] http://www.ascilite.org.au/conferences/brisbane99/papers/mcloughlinluca.pdf

Oliver, R. & Omari, A. (1999). Using online technologies to support problem based learning: Learners' responses and perceptions. Australian Journal of Educational Technology, 15(1), 58-79. http://www.ascilite.org.au/ajet/ajet15/oliver.html

Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park, CA: Sage.

Phillips, R. A. (2002). Learning-centred evaluation of computer-facilitated learning projects in higher education: Outcomes of a CUTSD staff development grant. Canberra: Committee for University Teaching and Staff Development, Commonwealth of Australia. [verified 22 Aug 2002] http://cleo.murdoch.edu.au/projects/cutsd99/

Phillips, R., Bain, J., McNaught, C., Rice, M. & Tripp. (2000). Learner-centred evaluation of computer-facilitated learning projects in higher education. Committee for University Teaching and Staff Development Project. [verified 22 Aug 2002] http://cleo.murdoch.edu.au/projects/cutsd99/handbook/handbook.htm

Reeves, T.C. (1997). Established and emerging evaluation paradigms for instructional design. In D.C. Dills & A.J. Romiszowski (Eds), Instructional development paradigms (pp. 163-178). Englewood Cliffs, New Jersey: Educational Technology Publications.

Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Canadian Journal of Distance Education, 14(2), 50-71.

Salmon, G. (2000). E-moderating: The Key to Teaching and Learning Online. Kogan Page, London.

Stacey, E (1997). A virtual campus: The experiences of postgraduate students studying through electronic communication and resource access. UltiBASE, Dec. [verified 22 Aug 2002] http://ultibase.rmit.edu.au/Articles/dec97/stace1.htm

Stacey, E. (1999). Collaborative learning in an online environment. Canadian Journal of Distance Education, 14(2), 14-33.

Stephenson, J. (Ed) (2001). Teaching and learning online: Pedagogies for new technologies. Kogan Page, London.

Taylor, J., Woodman, M., Sumner, T. & Blake, C. T. (2000). Peering through a glass darkly: Integrative evaluation of an online course. Educational Technology & Society, 3(4) [verified 22 Aug 2002] http://ifets.ieee.org/periodical/vol_4_2000/taylor.html

Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. (Translated by Cole, M.M. Lopez-Morillas, Luria, A. R & Wertsch, J.). Cambridge, MA: Harvard University Press.

Authors: Elizabeth Stacey, Senior Lecturer, Faculty of Education, Deakin University, Burwood Victoria 3125. Email: estacey@deakin.edu.au

Mary Rice, Lecturer, Education Design, Teaching and Learning Support Unit, Learning Services, Deakin University, Waterfront Campus, Geelong, Victoria 3217. Email: mrice@deakin.edu.au

Please cite as: Stacey, E. and Rice, M. (2002). Evaluating an online learning environment. Australian Journal of Educational Technology, 18(3), 323-340. http://www.ascilite.org.au/ajet/ajet18/stacey.html


[ AJET 18 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet18/stacey.html
Created 10 Oct 2002. Last revised 10 Oct 2002.

Page access count