|Australian Journal of Educational Technology
2002, 18(3), 323-340.
The refinement of the factors contributing to the establishment of an effective online learning environment showed the importance of communication that was affective as well as cognitive and the role of the teacher in establishing an online community where social presence is established through teacher modeling, discussed in the work of Salmon (2000) and Garrison, Anderson and Archer (1999), was the framework of research within which this evaluation occurred. Garrison, Anderson and Archer have defined the online group as a critical community of inquiry and have established a framework of analysis of the community into three elements: cognitive presence, social presence and teacher presence. These factors of analysis were similar to the content analysis framework defined by Stacey and were used to modify and explore the focus of evaluative analysis applied to this study.
This study will focus on a unit within the specialism which used computer conferencing as the central form of communication and learning. The unit required students to use the FirstClass environment to communicate and to access and share resources, both those provided and those they researched and evaluated through searching the World Wide Web. As the content of the unit was about online learning, students moderated discussions about issues of online learning, and worked in collaborative groups for an assessed task on researching the theory and process of collaborative learning online. In evaluating the effectiveness of computer facilitated learning in the medium of computer conferencing this study used a range of methods for data gathering and analysis .
Though we had previously used the online environment to discuss and reflect on students' evaluation comments, no overall attempt had been made to rigorously evaluate the online learning processes and learning outcomes in this specialism. The acknowledgement of the need for this prompted our participation in the CUTSD funded project. Time and expertise constraints are issues described by Taylor et al (2000) who describe a case study of "integrative evaluation" of an online course. Their method used the surrounding materials and activities as a means of evaluation via observation, interviews and questionnaires, web -administered questionnaires, paper questionnaires posted out, use of the computer conferences and by using the programming environment to gather data about student use. They found that the course team cannot always cope with the volume of evaluation data generated by a large course and can often be de-motivated by critical voices. Involving a mentor or evaluation team was a way of helping to identify what problems need solution and acting on these. In our case working as a team with a mentor helped us use our teaching time efficiently to ensure evaluation data were gathered and included in new development or to drive changes to ensure quality assurance.
Previous monitoring of the environment focused on the scale of the technical and support structure needed (Goodwin, Rice, Stacey & Thompson, 1995). This work demonstrated that students found FirstClass easy and enjoyable to use. It reduced the isolation so often felt by off campus students and made them feel part of a community. Providing the technology infrastructure was stable, a majority of students found it easy to access resource materials and carry out the required discursive activities. Over 80% of students believed that CMC facilitated their learning through small group teamwork processes, though at this stage, no other data was gathered to affirm these beliefs.
The focus of researching the roles of the social, cognitive and teacher presence (Garrison, Anderson & Archer, 2001) described above defined the framework of the study and with that background, the present project was designed to evaluate the effects of the use of computer conferencing on students' learning. During the semester, data were gathered electronically, using the methods described below. The project used qualitative methods to gather students' perceptions and reflections on the effect the conferencing process had on their learning. Quantitative data was collected through analysis of the frequency and type of messages which occur on the conferences. Final student results for the units were also reviewed.
The team evaluating the project included the teacher and though this meant there could be response to student comments in a formative way, though the evaluation was ultimately a summative evaluation of the implementation of the computer facilitated learning (see Table 1). This meant that, as prescribed by ethical requirements of the university, most of the evaluation analysis occurred after the semester was complete and grades were recorded so that participants were able to respond during the semester in an open and unpressured way.
|Objectives||Evaluation questions||Methods for all questions|
|To investigate the effectiveness of the computer conferencing approach by analysing patterns of online interaction and the interconnection between the cognitive, affective and system structures of the conference environment||
|To investigate the ways in which the learning processes and learning outcomes of the students studying the described units were affected by the use of computer conferencing.||
The project was designed to investigate the effectiveness of the computer conferencing approach in the unit by analysing patterns of online interaction and the interconnection between the cognitive, affective and system structures of the conference environment. The project also investigated the students' perceptions about the their learning processes and learning outcomes and how these were affected by the use of computer conferencing.
As outlined in Table 1, the initial data collection methods included the following:
"...it provided me with the opportunity to construct my own knowledge/ understanding within my own context. Meant I had to revisit readings, interpret what others were saying in the group, respond, evaluate." Female student, <50, internationalThe different perspectives provided by the different students were particularly seen as an advantage to their learning as it took them out of their own more limited view of the subject (often with difficulty).
"Yes as it brought other people's perspectives to the issues raised which wouldn't occur by completing an assignment alone. our group had participants from England, Korea, Melbourne and Alice Springs which all had a wide range of diverse views and experiences in education " Male, 50, ruralStudents found that the other participants challenged their ideas and provided new thinking.
"I was forced to think laterally and compromise my preferred style of learning. It forced me to take my blinkers off and open my mind to other interpretations. At first I was intolerant of other opinions that did not mesh with mine. Seeing words in black and white in front of you seems a lot louder than a voice." Female, >50, rural.Often students compared online learning and face to face learning, judging CMC as more engaging and flexible environment for their learning.
"..extended me in new areas, made me think about what I was saying and then had to rethink it when others took it differently. Gave me not only one experience but the experience of many to look at issues." Female, 40, international.
"I had to really nut things out in order to feel confident in discussing things with the other members of my group. Putting your thoughts down in written form seems to require much more effort than face to face discussions." Female, 35, metropolitan.The ability to "benchmark" their learning, to find out how others learned online was another advantage not always possible at a distance.
"It challenged me to think about the questions and people's responses - CMC has certain advantages over the face to face tutorial in that here we often get EVERYONE'S response to a question - in a face to face class you would probably only hear from one or two people and then the tutor. CMC allows more reflection than in the face to face setting - we have time to read each other's comments and respond to them in our own time. We can come back to something later and respond to it - these opportunities rarely exist in face to face classes." Female, 35, rural.
"Had more time to think through the issues as there was no need to respond immediately. Often went back and reread materials to clarify issues and having this bank of materials readily available was helpful. Male, 50, rural."
"Something interesting to me that did emerge was an understanding of how others think and work at a Masters level. This is the first time l have been able to observe peers at work, their commitment, their depth of involvement, their professionalism and actual research and language skills." Female, 40, international.The role of the lecturer as conference facilitator, regularly interacting online was seen as essential to maintenance of activity and focus by students. How this role could be established and the effects of modeling online communication strategies were commented on by students and was a focus in the content analysis described below.
"The quality of the interaction is also affected by the frequency of the lecturer. Another site I attend has no lecturer involvement and the site feels "less relevant and important" Female >50, international
"The lecturer/ tutor has the ability to encourage and expand the learning base of all students when there is regular dialogue among the group." Female, 50, metropolitan
The teacher's online messaging total combined with the 17 students in the evaluation focus group totaled 1281 messages on all parts of the conference during the semester. These included whole group discussions, discussions of issues by topic as well as the small collaborative group discussions. As the unit required online interaction, frequent interaction could be measured by high message frequency, though this message tally gave no indication of the quality and length of the messages. Though comparison of message frequency and student results (see Table 2) could not be interpreted as providing conclusive evidence of the effect of online interaction on learning outcomes, the highest achieving students were also highly interactive, particularly in their small group interaction and lower achieving students were less active online. Only one very interactive student receiving a pass level (relating more to external factors) while all other frequently participating students gained a grade higher than a pass. Student 9, who was slightly less interactive due to family commitments restricting her less interactive time online, received a High Distinction due particularly to the quality of her interactions which though less frequent were prepared carefully and supported by extensive offline research. The failing students in the unit had also failed to interact, their absence online reflecting their lack of engagement with the course through group interaction which provided feedback from other students and from staff. Their consequent lack of submission of their assignments resulted in failure. Only one student had failed to complete the course after an interactive start and this was due to external work related relocation. Table 2 summarises the overall participation and results comparison.
gender, age, location
|1||Female, 41, overseas||50||93||143||High Distinction|
|2||Female, 40, overseas||72||56||128||High Distinction|
|3||Female, 51, local city||51||23||74||Distinction|
|4||Female, 51, overseas||31||39||70||Distinction|
|5||Male, 50, remote rural||19||42||61||Distinction|
|6||Female, 38, remote city||27||63||90||Pass|
|7||Female, 46, remote city||24||35||59||Distinction|
|8||Female, 58, remote rural||23||36||59||Distinction|
|9||Female, 35, local city||22||36||58||High Distinction|
|10||Female, 48, remote rural||21||30||51||Distinction|
|11||Female, 47, remote rural||29||21||50||Distinction|
|12||Female, 56, remote rural||19||27||46||Distinction|
|13||Female, 35, local city||16||53||69||Distinction|
|14||Male, 47, local city||14||5||19||Pass|
|15||Female, 34, local city||12||7||19||Pass|
|16||Male, remote regional||11||0||11||No Assessment|
|17||Female, 57, overseas||1||0||1||No Assessment|
|* HD = 80-100%, D = 70-79%, C = 60-69%, P = 50-59%|
Message nature and content
Researchers have attempted to analyse communication, learning strategies and patterns of interaction in computer conferencing through content analysis and categorising of the text generated when messages are sent to computer conferences. Categories and methods are generated in many different ways depending on the focus of the research. Henri (1993) analysed messages into units of meaning and attempted to measure social dimensions, interactivity, cognitive skills, levels of processing and metacognitive knowledge and skills which were critiqued and developed by Gunawardena, Lowe and Anderson (1997) into a five phase constructivist interaction analysis model. Kanuka and Anderson (1998) applied this preliminary model successfully, suggesting modifications, and McLoughlin and Luca (1999) also adapted this model of analysis for a learner centred use of computer conferencing. Stacey in an earlier study (1998) categorised and calculated the online messages in her study into three types: course content, process of learning the technology and group learning and support before analysing the online discourse and other data into a model of attributes of online collaborative group learning. These attributes included clarification of ideas, feedback to ideas, diverse perspectives, group solutions and group resource sharing as well as factors of socio-affective collaborative support.
In this study the required online interaction in this unit was analysed focusing on the way the teacher established a model of social interaction through use of social presence factors. The level of cognitive engagement was analysed particularly through the continuing patterns of interaction and communication when students began to work mainly in small collaborative groups. This analysis used a categorisation that labeled units of meaning within each message for its primary purpose and content into:
The teacher's role in the first week of semester, establishing a secure learning environment and modeling socially accepting processes of interaction, was shown to be a major factor in increasing the frequency of social presence factors in the whole group conference, as students in the second week of semester followed teacher direction and practised using aspects of the software while providing personal contextual information. There was a rise in the level of social presence factors in week 2 because, though the student participation rate did not vary from the other weeks analysed, their messages followed the modeled and explained process for establishing social presence exhibited in the analysed factors of social presence.
Small collaborative groups were established in this unit to facilitate continued group discussions and tasks. Social presence factors continued to be important in the communication of these groups, with high frequencies of interactive and cohesive units continuing to appear within messages and even rising towards the end of the semester. Even though cognitive content became a main focus in the group's interaction, the social interaction continued to be an important factor within a less formal space. As group participants negotiated over content, they interacted with one another's message text, asked questions and agreed with and complimented the others' ideas, an important social component of effective collaborative and cognitive learning.
"I worked really hard, much harder than I am working on my present course because I felt a connection with my peers and felt I owed it to them to be on top of the conversations, contributing when I could and commenting on their thoughts when needed. " Female student <40, capital cityThe shared resource base was seen as a great advantage of this type of learning as web resources have increased to such an extent that a group process of research and commentary on web sites provided students with a much better resource base than they could find themselves.
"The contributions from various others broadened the pool of resources to check and utilise. I was continuously grateful for the excellent resources offered by group members. I was able to access many valuable readings due to the industriousness of my colleagues. My own resources were okay, yet l found that sharing this task of finding materials gave a varied edge to the readings." Female, 40-50, internationalOne student summarised the advantages of interacting in an online environment as:
sense of learning communityMost students recognised the value of group interaction though a few students elected to work independently on their assessment. They did identify the increased time spent on the subject as a disadvantage though this was an element that the students usually saw as a choice and as a self management issue. Some distance students complained that they chose to work at a distance as they preferred their independence and the ability to work at their own pace and did not learn well in groups, though they understood the advantages of the medium in their learning.
discipline of regular feedback/responses to FirstClass
quick feedback from colleagues in the group
quick & easy access to Tutor
one place to go for group contact
less expensive than attending university campus
(male >50, metropolitan WA)
In the summative online discussion, students and the evaluation team participated in a month long conference which generated 94 messages, 59 of which were from students. The discussion occurred 3 months after the semester of study and the project participants had since joined other courses which had been taught in a variety of ways, both with and without computer conferencing,. This resulted in some comparative reflections which were a rich source of data. Students reported working harder than in other distance subjects as they were accountable to the group and engaged more with reading and reflection on a wider range of resources than they would have consulted alone. They were unable to just do the minimum amount of work for the assessment when they were asked to contribute to an ongoing discussion. Overall, the whole body of students were very positive about the results reported and saw many advantages in learning online and raised some useful issues for improvement. These included:
"Having feedback is a motivator in itself, it's wonderful to have people acknowledge your ideas, you feel less isolated and bouncing ideas off one another is an important way to learn. Different experiences by people can either confirm or allow you to question the views you may have on a certain." Female, >40, international.
The effects on the learning processes and learning outcomes of the students have been analysed using a range of evaluation methods. In summary we found:
The role of the teacher in structuring and establishing cognitive and social presence of students studying online was defined more clearly through this study. The teacher must establish a secure interactive environment through modeling communicative behaviours while establishing online teacher presence. The online teacher's initial intense interaction can be followed by a teacher structured but student led online environment in which the teacher facilitates social and cognitive presence through carefully devised group tasks. The importance of providing time and activities for establishing social presence in an online learning environment should be considered in any new program being developed for online delivery. The cognitive and social strategies students use to learn online which were identified in this evaluative study have impacted on the program's development and have implications for similar courses taught in an online environment.
Bain, J. D. (1999). Introduction: Learning centred evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.
Evans, T. & Nation, D. (2000) (Eds). Changing university teaching: Reflections on creating educational technologies. London: Kogan Page
Fisher, K., Phelps, R. & Ellis, A. (2000). Group processes online: Teaching collaboration through collaborative processes. Educational Technology & Society, 3(3). [verified 22 Aug 2002] http://ifets.ieee.org/periodical/vol_3_2000/f06.html
Garrison, D.R., Anderson, T. & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105.
Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: SAGE Publications.
Gunawardena, C. N., Lowe, C. A. & Anderson, T. (1997) Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397-431.
Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed), Collaborative Learning Through Computer Conferencing. London: Springer-Verlag pp 117-136.
Jonassen, D., Prevish, T., Christy, D. & Stavrulaki, E. (1999). Learning to solve problems on the web: Aggregate planning in a business management course. Distance Education, 20(1), 49-65.
Kanuka, H. & Anderson, T. (1998). Online social interchange, discord and knowledge construction. Canadian Journal of Distance Education, 13(1), 57-74.
McLoughlin, C. and Luca, J. (1999). Lonely outpourings or reasoned dialogue? An analysis of text-based conferencing as a tool to support learning. Proceedings of 1999 ASCILITE Conference, Brisbane, December. [verified 22 Aug 2002] http://www.ascilite.org.au/conferences/brisbane99/papers/mcloughlinluca.pdf
Oliver, R. & Omari, A. (1999). Using online technologies to support problem based learning: Learners' responses and perceptions. Australian Journal of Educational Technology, 15(1), 58-79. http://www.ascilite.org.au/ajet/ajet15/oliver.html
Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park, CA: Sage.
Phillips, R. A. (2002). Learning-centred evaluation of computer-facilitated learning projects in higher education: Outcomes of a CUTSD staff development grant. Canberra: Committee for University Teaching and Staff Development, Commonwealth of Australia. [verified 22 Aug 2002] http://cleo.murdoch.edu.au/projects/cutsd99/
Phillips, R., Bain, J., McNaught, C., Rice, M. & Tripp. (2000). Learner-centred evaluation of computer-facilitated learning projects in higher education. Committee for University Teaching and Staff Development Project. [verified 22 Aug 2002] http://cleo.murdoch.edu.au/projects/cutsd99/handbook/handbook.htm
Reeves, T.C. (1997). Established and emerging evaluation paradigms for instructional design. In D.C. Dills & A.J. Romiszowski (Eds), Instructional development paradigms (pp. 163-178). Englewood Cliffs, New Jersey: Educational Technology Publications.
Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Canadian Journal of Distance Education, 14(2), 50-71.
Salmon, G. (2000). E-moderating: The Key to Teaching and Learning Online. Kogan Page, London.
Stacey, E (1997). A virtual campus: The experiences of postgraduate students studying through electronic communication and resource access. UltiBASE, Dec. [verified 22 Aug 2002] http://ultibase.rmit.edu.au/Articles/dec97/stace1.htm
Stacey, E. (1999). Collaborative learning in an online environment. Canadian Journal of Distance Education, 14(2), 14-33.
Stephenson, J. (Ed) (2001). Teaching and learning online: Pedagogies for new technologies. Kogan Page, London.
Taylor, J., Woodman, M., Sumner, T. & Blake, C. T. (2000). Peering through a glass darkly: Integrative evaluation of an online course. Educational Technology & Society, 3(4) [verified 22 Aug 2002] http://ifets.ieee.org/periodical/vol_4_2000/taylor.html
Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. (Translated by Cole, M.M. Lopez-Morillas, Luria, A. R & Wertsch, J.). Cambridge, MA: Harvard University Press.
|Authors: Elizabeth Stacey, Senior Lecturer, Faculty of Education, Deakin University, Burwood Victoria 3125. Email: email@example.com
Mary Rice, Lecturer, Education Design, Teaching and Learning Support Unit, Learning Services, Deakin University, Waterfront Campus, Geelong, Victoria 3217. Email: firstname.lastname@example.org
Please cite as: Stacey, E. and Rice, M. (2002). Evaluating an online learning environment. Australian Journal of Educational Technology, 18(3), 323-340. http://www.ascilite.org.au/ajet/ajet18/stacey.html