Evaluating an online learning environment

Introduction

Background

The refinement of the factors contributing to the establishment of an effective online learning environment showed the importance of communication that was affective as well as cognitive and the role of the teacher in establishing an online community where social presence is established through teacher modeling, discussed in the work of Salmon (2000) and Garrison, Anderson and Archer (1999), was the framework of research within which this evaluation occurred. Garrison, Anderson and Archer have defined the online group as a critical community of inquiry and have established a framework of analysis of the community into three elements: cognitive presence, social presence and teacher presence. These factors of analysis were similar to the content analysis framework defined by Stacey and were used to modify and explore the focus of evaluative analysis applied to this study.

Online learning in open and distance education units

This study will focus on a unit within the specialism which used computer conferencing as the central form of communication and learning. The unit required students to use the FirstClass environment to communicate and to access and share resources, both those provided and those they researched and evaluated through searching the World Wide Web. As the content of the unit was about online learning, students moderated discussions about issues of online learning, and worked in collaborative groups for an assessed task on researching the theory and process of collaborative learning online. In evaluating the effectiveness of computer facilitated learning in the medium of computer conferencing this study used a range of methods for data gathering and analysis .

Though we had previously used the online environment to discuss and reflect on students' evaluation comments, no overall attempt had been made to rigorously evaluate the online learning processes[1] and learning outcomes[2] in this specialism. The acknowledgement of the need for this prompted our participation in the CUTSD funded project. Time and expertise constraints are issues described by Taylor et al (2000) who describe a case study of "integrative evaluation" of an online course. Their method used the surrounding materials and activities as a means of evaluation via observation, interviews and questionnaires, web -administered questionnaires, paper questionnaires posted out, use of the computer conferences and by using the programming environment to gather data about student use. They found that the course team cannot always cope with the volume of evaluation data generated by a large course and can often be de-motivated by critical voices. Involving a mentor or evaluation team was a way of helping to identify what problems need solution and acting on these. In our case working as a team with a mentor helped us use our teaching time efficiently to ensure evaluation data were gathered and included in new development or to drive changes to ensure quality assurance.

Description of project

Previous monitoring of the environment focused on the scale of the technical and support structure needed (Goodwin, Rice, Stacey & Thompson, 1995). This work demonstrated that students found FirstClass easy and enjoyable to use. It reduced the isolation so often felt by off campus students and made them feel part of a community. Providing the technology infrastructure was stable, a majority of students found it easy to access resource materials and carry out the required discursive activities. Over 80% of students believed that CMC facilitated their learning through small group teamwork processes, though at this stage, no other data was gathered to affirm these beliefs.

The focus of researching the roles of the social, cognitive and teacher presence (Garrison, Anderson & Archer, 2001) described above defined the framework of the study and with that background, the present project was designed to evaluate the effects of the use of computer conferencing on students' learning. During the semester, data were gathered electronically, using the methods described below. The project used qualitative methods to gather students' perceptions and reflections on the effect the conferencing process had on their learning. Quantitative data was collected through analysis of the frequency and type of messages which occur on the conferences. Final student results for the units were also reviewed.

Methodology

Project participants

The team evaluating the project included the teacher and though this meant there could be response to student comments in a formative way, though the evaluation was ultimately a summative evaluation of the implementation of the computer facilitated learning (see Table 1). This meant that, as prescribed by ethical requirements of the university, most of the evaluation analysis occurred after the semester was complete and grades were recorded so that participants were able to respond during the semester in an open and unpressured way.

Evaluation objectives and questions

Objectives	Evaluation questions	Methods for all questions
To investigate the effectiveness of the computer conferencing approach by analysing patterns of online interaction and the interconnection between the cognitive, affective and system structures of the conference environment	How is computer conferencing used in teaching and learning for sharing ideas and constructing knowledge? How do the students in the unit interact online?	Voluntary online focus group conference and individual emailed responses. Online observation Student results analysis Conference analysis Message frequency and distribution
To investigate the ways in which the learning processes and learning outcomes of the students studying the described units were affected by the use of computer conferencing.	How have students perceived the effect of online interaction on their learning?	Voluntary focus group and individual emailed interview responses Summative online discussion

The project was designed to investigate the effectiveness of the computer conferencing approach in the unit by analysing patterns of online interaction and the interconnection between the cognitive, affective and system structures of the conference environment. The project also investigated the students' perceptions about the their learning processes and learning outcomes and how these were affected by the use of computer conferencing.

Data collection methods

As outlined in Table 1, the initial data collection methods included the following:

1. A voluntary online focus group conference was established specifically for the project. Students were asked to respond both to questions about their experiences of learning online, and to reflections about online practice posted in this area during and at the end of the semester. These questions were also emailed to any student who had not participated in the focus group conference.

2. Online observation was used to analyse communication and learning processes through ongoing response to student comments, and retrospective analysis of conference message archives.

3. Analysis of conference message content was used to determine the students' learning processes online.

4. Calculation of frequency and distribution of message use was used to establish communication patterns and as a form of triangulation of data.

5. Comparison of students' results and interaction frequency to see if simple trends of interaction rate and outcome were interrelated.

6. A summative online discussion was held three months after semester ended. Students were asked to comment on a summary of previously analysed findings which were posted on a short term computer conference set up for the purpose.

Evaluation results

1. How is computer conferencing used in teaching and learning for sharing ideas and constructing knowledge?

"...it provided me with the opportunity to construct my own knowledge/ understanding within my own context. Meant I had to revisit readings, interpret what others were saying in the group, respond, evaluate." Female student, <50, international

"Yes as it brought other people's perspectives to the issues raised which wouldn't occur by completing an assignment alone. our group had participants from England, Korea, Melbourne and Alice Springs which all had a wide range of diverse views and experiences in education " Male, 50, rural

"I was forced to think laterally and compromise my preferred style of learning. It forced me to take my blinkers off and open my mind to other interpretations. At first I was intolerant of other opinions that did not mesh with mine. Seeing words in black and white in front of you seems a lot louder than a voice." Female, >50, rural.

"..extended me in new areas, made me think about what I was saying and then had to rethink it when others took it differently. Gave me not only one experience but the experience of many to look at issues." Female, 40, international.

"I had to really nut things out in order to feel confident in discussing things with the other members of my group. Putting your thoughts down in written form seems to require much more effort than face to face discussions." Female, 35, metropolitan.

"It challenged me to think about the questions and people's responses - CMC has certain advantages over the face to face tutorial in that here we often get EVERYONE'S response to a question - in a face to face class you would probably only hear from one or two people and then the tutor. CMC allows more reflection than in the face to face setting - we have time to read each other's comments and respond to them in our own time. We can come back to something later and respond to it - these opportunities rarely exist in face to face classes." Female, 35, rural.

"Had more time to think through the issues as there was no need to respond immediately. Often went back and reread materials to clarify issues and having this bank of materials readily available was helpful. Male, 50, rural."

"Something interesting to me that did emerge was an understanding of how others think and work at a Masters level. This is the first time l have been able to observe peers at work, their commitment, their depth of involvement, their professionalism and actual research and language skills." Female, 40, international.

"The quality of the interaction is also affected by the frequency of the lecturer. Another site I attend has no lecturer involvement and the site feels "less relevant and important" Female >50, international

"The lecturer/ tutor has the ability to encourage and expand the learning base of all students when there is regular dialogue among the group." Female, 50, metropolitan

2. How do the students in the unit interact online?

The teacher's online messaging total combined with the 17 students in the evaluation focus group totaled 1281 messages on all parts of the conference during the semester. These included whole group discussions, discussions of issues by topic as well as the small collaborative group discussions. As the unit required online interaction, frequent interaction could be measured by high message frequency, though this message tally gave no indication of the quality and length of the messages. Though comparison of message frequency and student results (see Table 2) could not be interpreted as providing conclusive evidence of the effect of online interaction on learning outcomes, the highest achieving students were also highly interactive, particularly in their small group interaction and lower achieving students were less active online. Only one very interactive student receiving a pass level (relating more to external factors) while all other frequently participating students gained a grade higher than a pass. Student 9, who was slightly less interactive due to family commitments restricting her less interactive time online, received a High Distinction due particularly to the quality of her interactions which though less frequent were prepared carefully and supported by extensive offline research. The failing students in the unit had also failed to interact, their absence online reflecting their lack of engagement with the course through group interaction which provided feedback from other students and from staff. Their consequent lack of submission of their assignments resulted in failure. Only one student had failed to complete the course after an interactive start and this was due to external work related relocation. Table 2 summarises the overall participation and results comparison.

Student	Demographics - gender, age, location	Whole group total	Small group total	Total message frequency	Results*
1	Female, 41, overseas	50	93	143	High Distinction
2	Female, 40, overseas	72	56	128	High Distinction
3	Female, 51, local city	51	23	74	Distinction
4	Female, 51, overseas	31	39	70	Distinction
5	Male, 50, remote rural	19	42	61	Distinction
6	Female, 38, remote city	27	63	90	Pass
7	Female, 46, remote city	24	35	59	Distinction
8	Female, 58, remote rural	23	36	59	Distinction
9	Female, 35, local city	22	36	58	High Distinction
10	Female, 48, remote rural	21	30	51	Distinction
11	Female, 47, remote rural	29	21	50	Distinction
12	Female, 56, remote rural	19	27	46	Distinction
13	Female, 35, local city	16	53	69	Distinction
14	Male, 47, local city	14	5	19	Pass
15	Female, 34, local city	12	7	19	Pass
16	Male, remote regional	11	0	11	No Assessment
17	Female, 57, overseas	1	0	1	No Assessment
* HD = 80-100%, D = 70-79%, C = 60-69%, P = 50-59%

Message nature and content

Researchers have attempted to analyse communication, learning strategies and patterns of interaction in computer conferencing through content analysis and categorising of the text generated when messages are sent to computer conferences. Categories and methods are generated in many different ways depending on the focus of the research. Henri (1993) analysed messages into units of meaning and attempted to measure social dimensions, interactivity, cognitive skills, levels of processing and metacognitive knowledge and skills which were critiqued and developed by Gunawardena, Lowe and Anderson (1997) into a five phase constructivist interaction analysis model. Kanuka and Anderson (1998) applied this preliminary model successfully, suggesting modifications, and McLoughlin and Luca (1999) also adapted this model of analysis for a learner centred use of computer conferencing. Stacey in an earlier study (1998) categorised and calculated the online messages in her study into three types: course content, process of learning the technology and group learning and support before analysing the online discourse and other data into a model of attributes of online collaborative group learning. These attributes included clarification of ideas, feedback to ideas, diverse perspectives, group solutions and group resource sharing as well as factors of socio-affective collaborative support.

In this study the required online interaction in this unit was analysed focusing on the way the teacher established a model of social interaction through use of social presence factors. The level of cognitive engagement was analysed particularly through the continuing patterns of interaction and communication when students began to work mainly in small collaborative groups. This analysis used a categorisation that labeled units of meaning within each message for its primary purpose and content into:

• Cognitive/content, detailed discussion and commentary on the course content
• System messages (relating to learning the FirstClass software, or access issues) and administrative messages.
• Social content analysed into social presence factors using the multiscale Social Presence categories developed by Rourke, Anderson, Garrison and Archer (1999) for defining and measuring social presence in a computer conference. The three categories are:
• interactive responses, threaded responses with messages of socially appreciative nature
• affective responses expressing emotion, feeling and mood which are expressed by emoticons, humour and self disclosure
• cohesive responses which are group responses which build a cohesive group environment.

The teacher's role in the first week of semester, establishing a secure learning environment and modeling socially accepting processes of interaction, was shown to be a major factor in increasing the frequency of social presence factors in the whole group conference, as students in the second week of semester followed teacher direction and practised using aspects of the software while providing personal contextual information. There was a rise in the level of social presence factors in week 2 because, though the student participation rate did not vary from the other weeks analysed, their messages followed the modeled and explained process for establishing social presence exhibited in the analysed factors of social presence.

Small collaborative groups were established in this unit to facilitate continued group discussions and tasks. Social presence factors continued to be important in the communication of these groups, with high frequencies of interactive and cohesive units continuing to appear within messages and even rising towards the end of the semester. Even though cognitive content became a main focus in the group's interaction, the social interaction continued to be an important factor within a less formal space. As group participants negotiated over content, they interacted with one another's message text, asked questions and agreed with and complimented the others' ideas, an important social component of effective collaborative and cognitive learning.

3. How have students perceived the effect of online interaction on their learning?

"I worked really hard, much harder than I am working on my present course because I felt a connection with my peers and felt I owed it to them to be on top of the conversations, contributing when I could and commenting on their thoughts when needed. " Female student <40, capital city

"The contributions from various others broadened the pool of resources to check and utilise. I was continuously grateful for the excellent resources offered by group members. I was able to access many valuable readings due to the industriousness of my colleagues. My own resources were okay, yet l found that sharing this task of finding materials gave a varied edge to the readings." Female, 40-50, international

sense of learning community
discipline of regular feedback/responses to FirstClass
quick feedback from colleagues in the group
quick & easy access to Tutor
one place to go for group contact
less expensive than attending university campus
(male >50, metropolitan WA)

In the summative online discussion, students and the evaluation team participated in a month long conference which generated 94 messages, 59 of which were from students. The discussion occurred 3 months after the semester of study and the project participants had since joined other courses which had been taught in a variety of ways, both with and without computer conferencing,. This resulted in some comparative reflections which were a rich source of data. Students reported working harder than in other distance subjects as they were accountable to the group and engaged more with reading and reflection on a wider range of resources than they would have consulted alone. They were unable to just do the minimum amount of work for the assessment when they were asked to contribute to an ongoing discussion. Overall, the whole body of students were very positive about the results reported and saw many advantages in learning online and raised some useful issues for improvement. These included:

• discussion of access problems particularly in rural areas, how this was being responded to and the issues it raised for equality for remote Aboriginal communities.

• identification of a need for proficiency in English to fulfill teachers' expectations of reflective and deep thinking expressed in messages.

• discussion of the continued need for print material to backup online reading as it provided flexibility for learning away from the computer.

• differences in gender participation were discussed. One female students analysed her group's interactions and stated "I found that the men were more inclined to divulge references and links to good sites than "chat" informally. I felt that there was less of a banter and more of a statement made when the men participated." (Female, 35, metropolitan). The men agreed with this comment though they also confirmed appreciation the social support of the online conference.

• discussion of the need for other additional communication technologies layering the conferencing such as phone, meeting, or synchronous chat, was seen as essential for avoiding misinterpretation of messages.

• an ongoing need for teacher presence was supported unanimously with the lecturer's interaction seen as essential for a conference to be effective.

"Having feedback is a motivator in itself, it's wonderful to have people acknowledge your ideas, you feel less isolated and bouncing ideas off one another is an important way to learn. Different experiences by people can either confirm or allow you to question the views you may have on a certain." Female, >40, international.

Conclusions

The effects on the learning processes and learning outcomes of the students have been analysed using a range of evaluation methods. In summary we found:

• Student feedback supported the use of online conferencing in encouraging a learning community with teacher presence seen as central to this.

• Frequency analysis showed that required online involvement generated high frequency of messaging, a high teacher time requirement that needed more management with responsibility given to students. Patterns of communication showed that high teacher interaction encouraged high student response but in small groups this was devolved and required less teacher interactivity.

• Tasks designed for online discussion generated online interaction with a cognitive focus.

• Content analysis pointed to the role and importance of the conferences for social interaction and administrative sharing as well as for a cognitive focus.

• Summative discussion was a key evaluation innovation and confirmed previous findings establishing the reliability of formatively gathered results.

• Students perceived the value of considering other students' perspectives, ideas and resources as a major component of their successful learning online.

The role of the teacher in structuring and establishing cognitive and social presence of students studying online was defined more clearly through this study. The teacher must establish a secure interactive environment through modeling communicative behaviours while establishing online teacher presence. The online teacher's initial intense interaction can be followed by a teacher structured but student led online environment in which the teacher facilitates social and cognitive presence through carefully devised group tasks. The importance of providing time and activities for establishing social presence in an online learning environment should be considered in any new program being developed for online delivery. The cognitive and social strategies students use to learn online which were identified in this evaluative study have impacted on the program's development and have implications for similar courses taught in an online environment.

Endnotes

1. Learning processes refers to all cognitive activities that contribute to learning, (e.g.problem solving, reflection) as well as the manner in which these acticities are carried out (e.g. individually, in groups, teacher or student-directed).
2. Learning outcomes refer to the things students are able to do as a result of their engagement in a course of study. This includes both discipline-specific and generic skills.

References

Bain, J. D. (1999). Introduction: Learning centred evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.

Evans, T. & Nation, D. (2000) (Eds). Changing university teaching: Reflections on creating educational technologies. London: Kogan Page

Fisher, K., Phelps, R. & Ellis, A. (2000). Group processes online: Teaching collaboration through collaborative processes. Educational Technology & Society, 3(3). [verified 22 Aug 2002] http://ifets.ieee.org/periodical/vol_3_2000/f06.html

Garrison, D.R., Anderson, T. & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105.

Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: SAGE Publications.

Gunawardena, C. N., Lowe, C. A. & Anderson, T. (1997) Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397-431.

Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed), Collaborative Learning Through Computer Conferencing. London: Springer-Verlag pp 117-136.

Jonassen, D., Prevish, T., Christy, D. & Stavrulaki, E. (1999). Learning to solve problems on the web: Aggregate planning in a business management course. Distance Education, 20(1), 49-65.

Kanuka, H. & Anderson, T. (1998). Online social interchange, discord and knowledge construction. Canadian Journal of Distance Education, 13(1), 57-74.

McLoughlin, C. and Luca, J. (1999). Lonely outpourings or reasoned dialogue? An analysis of text-based conferencing as a tool to support learning. Proceedings of 1999 ASCILITE Conference, Brisbane, December. [verified 22 Aug 2002] http://www.ascilite.org.au/conferences/brisbane99/papers/mcloughlinluca.pdf

Oliver, R. & Omari, A. (1999). Using online technologies to support problem based learning: Learners' responses and perceptions. Australian Journal of Educational Technology, 15(1), 58-79. http://www.ascilite.org.au/ajet/ajet15/oliver.html

Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park, CA: Sage.

Phillips, R. A. (2002). Learning-centred evaluation of computer-facilitated learning projects in higher education: Outcomes of a CUTSD staff development grant. Canberra: Committee for University Teaching and Staff Development, Commonwealth of Australia. [verified 22 Aug 2002] http://cleo.murdoch.edu.au/projects/cutsd99/

Phillips, R., Bain, J., McNaught, C., Rice, M. & Tripp. (2000). Learner-centred evaluation of computer-facilitated learning projects in higher education. Committee for University Teaching and Staff Development Project. [verified 22 Aug 2002] http://cleo.murdoch.edu.au/projects/cutsd99/handbook/handbook.htm

Reeves, T.C. (1997). Established and emerging evaluation paradigms for instructional design. In D.C. Dills & A.J. Romiszowski (Eds), Instructional development paradigms (pp. 163-178). Englewood Cliffs, New Jersey: Educational Technology Publications.

Rourke, L., Anderson, T., Garrison, R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Canadian Journal of Distance Education, 14(2), 50-71.

Salmon, G. (2000). E-moderating: The Key to Teaching and Learning Online. Kogan Page, London.

Stacey, E (1997). A virtual campus: The experiences of postgraduate students studying through electronic communication and resource access. UltiBASE, Dec. [verified 22 Aug 2002] http://ultibase.rmit.edu.au/Articles/dec97/stace1.htm

Stacey, E. (1999). Collaborative learning in an online environment. Canadian Journal of Distance Education, 14(2), 14-33.

Stephenson, J. (Ed) (2001). Teaching and learning online: Pedagogies for new technologies. Kogan Page, London.

Taylor, J., Woodman, M., Sumner, T. & Blake, C. T. (2000). Peering through a glass darkly: Integrative evaluation of an online course. Educational Technology & Society, 3(4) [verified 22 Aug 2002] http://ifets.ieee.org/periodical/vol_4_2000/taylor.html

Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. (Translated by Cole, M.M. Lopez-Morillas, Luria, A. R & Wertsch, J.). Cambridge, MA: Harvard University Press.

Authors: Elizabeth Stacey, Senior Lecturer, Faculty of Education, Deakin University, Burwood Victoria 3125. Email: estacey@deakin.edu.au Mary Rice, Lecturer, Education Design, Teaching and Learning Support Unit, Learning Services, Deakin University, Waterfront Campus, Geelong, Victoria 3217. Email: mrice@deakin.edu.au Please cite as: Stacey, E. and Rice, M. (2002). Evaluating an online learning environment. Australian Journal of Educational Technology, 18(3), 323-340. http://www.ascilite.org.au/ajet/ajet18/stacey.html