Discursive evaluation in a distributed learning community

This paper describes a distance education project where a threaded discussion board was used for interaction amongst students and teachers. The experiences from the first year of the project shows that such a forum can be an important complement to other evaluative resources in order to monitor student's expectations and experiences. Furthermore, it is argued that discursive evaluations can establish a learning community with shared norms and forms of communication and collaboration. Vital properties of the discussion board are that it is continuous, online, public, asynchronous and auto-structuring.

1. Evaluation in distance education

In a distance educational setting, with students and teachers separated in time and space, these signals are drastically reduced. Compensation for this type of informal communication with computer mediated interaction is difficult, if even possible. Whittaker et al (1994) address the general issue of designing IT to support informal workplace communication, arriving at the conclusion that a shared workspace is an important aspect and that rich, synchronous video media and asynchronous, text based media should be combined and integrated when aiming for an acceptable design.

The focus of this paper is a threaded discussion board where students and teachers meet to discuss course related issues. This is an informal arena of conversation where students can express their attitudes, opinions, perceptions and experiences. Such a forum can serve as a valuable complement to traditional post-course questionnaires, even making it possible for teachers to challenge and manage the students' opinions. The arguments are based on experiences from a distance education project at a university in Scandinavia. The empirical data of the study are the entries made by students and teachers to a web based computer conference (discussion board). This conference is integrated into a web education system developed at the university. The entries have been analysed with respect to evaluative factors such as topic and nature.

2. The case

The study covers the courses during the first year of the project (Table 1). All courses had different teams of teachers and different methods of teaching and examination.

Mathematics and Statistics 10 weeks Computer Science 10 weeks Business Administration 10 weeks Finance 5 weeks C++ Programming 5 weeks

A common denominator in all courses was the use of learning technologies. Multiparty VC sessions were used for lectures, seminars and presentations, and a web education tool was used for distribution of course material and for text based interaction. The system provided a web site for each course, one of its components is the core object of this paper, namely the Discussion Board - a primitive web based conference, where discussion topics are divided into threads presented on the screen with indents that indicate the structure of the discussions. Each entry is presented with a hyperlinked title, a date and time stamp and the signature of the debater. The threads are sorted in descending time order with respect to the first entry of each thread. An entry is displayed in full text by clicking on the title, it is then possible to choose to post a follow up to the entry. In addition to text (with html tags if so desired) the author of an entry can choose to submit a picture, a URL and/or an email address. Each entry should be signed with a name or a pseudonym by the author. A new discussion thread is started in the same manner. Laurillard (1993) classifies the debate board to be a discursive medium defined as:

Both teacher's and student's conceptions are accessible to the other and both topic and task goals can be negotiable; students must be able to act on, generate and receive feedback on descriptions appropriate to the topic goal; the teacher must be able to reflect on student's actions and descriptions and adjust their own descriptions to be more meaningful to the student (p. 100).

2.1. Previous research

Typified communicative act having a socially defined and recognised communicative purpose with regard to its audience.

The Query genre was characterised by having the primary purpose of discussing exercises and aspects of the course literature. The purpose of the Smalltalk entries is to entertain and socialise even though the content is often related to some course activity. Social activities like this are often observed in distance learning projects (Fjuk 1998), and serve an important role in building and maintaining the learning community. These entries can to some extent be a substitute for subtle feedback signalling the mood of the students, as cited in the first section. However the focus in this paper is more directly linked to the Feedback genre which constitutes the foundation for the analysis reported in this paper.

3. Data collection and analysis

Topic	Description
Course context	Comments on administrative issues, student facilities or service
Course content	Assignments, course material, topics of study, literature, etc.
Learning technology	Comments on video conference or issues related to course web site (tech problems, features, etc).
Teacher performance	Comments on lectures and tutoring.
Debate climate	Comments on the form or style of other debaters.

The threads were grouped and classified with respect to the feedback topic or topics (Table 2). The process of grouping the threads into topic clusters with relevant labels was conducted in an iterative way, inspired by grounded theory techniques (Glaser & Strauss, 1980). If a thread did not stay focused on one topic, ie. if two or more entries belonged to another of the emergent topics, the thread was classified as having two topics.

Secondly, each debate board entry was analysed with respect to the nature of the feedback. Inspired by the classification of different types of feedback a teacher can give a student (presented by Draper 1997b) the following scheme was adopted.

Feedback types

Agree/Disagree with other debater
Success/Failure: Debater stating that something is bad (or good)
Behaviour recipe: Debater presenting a suggestion for change
Explanation: Elaboration on why something is wrong (or right)

Some simple quantitative measures such as number of entries, the number of follow up levels and the time span between first and last entry were recorded in order to give complementary information on the nature of the debate. In addition to analysing the debate entries, the log files containing data on all visits to the debate boards were analysed.

4. Results and analysis

4.1. Thread topics

Comparing the courses shows that the frequency of the debate is fairly equal in the first three courses and drops slightly in the last two. Also, the length of the threads drops from an average of 3-4 in the first 30 weeks, to no more than 2 in the last ten weeks. The fact that the 5-week courses tend to be more intense could be one possible explanation. Another explanation could be the fact that issues related to the web based system were frequently debated in the first courses, but gradually the system became more and more transparent to the students and was therefore not an interesting topic to debate. In the latter courses the technology threads related to how it was operated by teachers and integrated with course content.

Topic	Math/Stat	Computer Science	Business Admin	Finance	Program- ming
Context	4	1	7	3	6
Content	4	10	6	1	2
Technology	16	9	9	1	2
Teacher	5	1	1	3	0
Discussion	1	2	3	1	1
Disc/Content	0	1	0	0	0
Teach/Content	2	4	2	0	0
Teach/Tech	0	4	0	0	0
TOTAL	32	32	28	9	11

The short threads and solo entries could be found in all categories, but dominated in Learning technology and Debate climate. The longest discussions were concerned with the context or the content of the courses. The lifetime of most discussions were very short (1 or 2 days). Only three threads had a span of more than a week between the first and the last entry. One explanation to this could perhaps be found in the graphical interface of the debate board. Since the complete dialog history was presented on the screen, the threads disappeared from the start screen of the board after 15-20 entries have been made. Then the scroll bars must be used in order to read the entries or post a follow up. There were some examples where debaters chose to start a new thread continuing the debate on an existing topic, instead of adding to the original thread.

4.2. Nature of feedback

Cat	1	2	3	4	1&3	1&4	2&3	2&4	3&4	-
Descri- ption	Agree/ disagree (A/D)	Success/ failure (S/F)	Behav. recipe (BR)	Expla- nation (Exp)	A/D with BR	A/D with Exp	S/F with BR	S/F with Exp	BR with Exp	Other
Math	17	34	5	15	1	3	4	9	1	1
Comp.Sci	13	30	5	18	1	6	9	8	3	4
Bus.Admin	24	21	14	17	2	10	4	3	2	11
Finance	4	8	1	4	0	0	0	1	0	1
Program.	5	9	3	10	0	1	0	1	1	3
Total	63 (18%)	102 (29%)	28 (8%)	64 (18%)	4 (1%)	20 (6%)	17 (5%)	22 (6%)	7 (2%)	20 (6%)

4.3. Thread examples

Figure 1: Course context thread

The second example (Figure 2) is a short thread on Learning technology from the beginning of the first course and the third example (Figure 3) deals with the debate climate of the second course as well as the use of outdated software. Descriptions in italics are translations from debate board entries (in some cases, somewhat shortened).

Figure 2: Learning technology thread

Figure 3: Debate climate thread

5. Understanding the discursive evaluation

5.1. Community

Ågren (1999) argues that an important motivational factor, besides learning and interest in the topic, is the acquiring of social capital that comes from active participation in the discussions. In this case, it is merely possible to speculate regarding to what extent such a generic motivational factor contributed. Ackerman and Starr (1995) highlight how awareness of the activities of other community members can be valuable for members of a geographically dispersed community. In the context of the case reported in this paper, this could be interpreted as a collective interest in being aware of the conditions at the other (remote) sites. The discussion forum then serves as an arena for sharing experiences, expectations and frustration throughout the community. Hara and Kling (1999) explore the issue of student frustration, concluding that this is a major problem in distance education. In a case study of a North American DE project, they identify three sources of frustration: technology failure, lack of prompt feedback and ambiguous instructions, all of which are topics discussed at various times in this study. Viewing the discursive evaluation from the perspective of frustration, it seems as though engaging in discussions concerning the issues that are frustrating to the individual can help in reducing the frustration. This can be accomplished in several ways, for instance, by clearing misunderstandings, by finding out that others are experiencing the same, etc.

5.2. Evaluation

Regardless of whether the debate reflects a representative image of student attitudes or not, one can question the evaluative effect of such a study. Draper et al (1996) declare that any evaluation designed to monitor students' attitudes is weak with respect to the evaluation of educational effect, since what is measured is really students' experience compared to their expectations. They argue that the validity of such evaluations is particularly low when it comes to students' expectations of computer aided learning, since these educational settings are not familiar and well established for the students. Still, they acknowledge attitude measurements to be an important tool for teachers wanting to respond and manage these attitudes. Especially in a distance educational setting, when students and teachers seldom meet face to face, it can be difficult to keep track of students' attitudes (Hall 1997).

However, the debate board is very well suited for both discovering these perhaps diverse expectations, and even challenging and discussing them. In fact the debate could serve as means for smoothing out these differences. Hall (1997) reaches a similar conclusion. She says that using the content of a computer conference for evaluative purposes could be productive when wanting to find personal, subjective opinions among students. The illuminating qualities of the discourse could serve as a complement to a more summative post-course questionnaire. Such questionnaires are perhaps more valid in terms of representing the attitudes of the whole community, but often suffer from short comments without elaborate explanations on complaints and suggestions.

Another aspect that could be argued to add to the evaluative quality of the discussion board is the fact that students are probably less likely to perceive such a forum as an instrument of evaluation, thereby avoiding some of the bias that characterises other, more formal evaluative efforts. When a situation is identified as an evaluation, through for instance promises of anonymity, the students are likely to enter a role (Goffman, 1969), where the existing power relationships between them and the teacher affects what is and what is not said.

5.3. Application

The debate is public,

The fact that all entries can be read by everyone in the community serves at least three essential purposes. Firstly, the risk of minority opinions being overestimated is reduced since such entries are likely to be contradicted by others. Secondly, it makes it more interesting for students to express an opinion when they know that it will be read by more people than the teacher. This can be a strong implication for change. For example, when a student from one of the communities entered a complaint on bad service from a local facilitator, a student from another community responded with commending the perfect conditions on his site. The next entry (a day later) was from the first student reporting on a fast recovery. A third feature related to the public nature of the board is its social functions. The existence of the Smalltalk genre (Svensson 2002) points to the fact that debaters use the board for social purposes. It cannot be ruled out that an evaluative entry to the board is more of a social action than an actual wish to give feedback.

Auto-structuring,

The Debate Board was originally intended as a place were students could discuss issues related to the content of the course, but there were no attempts made to regulate or moderate the use in any way. The result of this study indicates a drift towards a more social and evaluative use. This kind of technology drifting is common when IT is used for collaborative purposes (Ciborra 1996). There is of course no guarantee that students will use the debate board for evaluation, but if they chose to do so, the entries are automatically sorted into threads, making topic oriented analysis easier for an evaluator or teacher. If the debate is regulated (ie. suggested topics for discussion) the threads can still provide good structure with respect to sub-topics. However, if the regulation is too strict, there is a risk of losing the discursive structure supported by the threads. At some point it will become arbitrary whether a new entry should start a new thread or add to an existing one.

Asynchronous,

Laurillard (1993) argues that the asynchronous nature of a discursive medium can contribute to students making more carefully thought through and 'well formulated' entries, reducing the risk for misunderstandings in analysing the content. Draper (1997a) argues that reflecting on one's own writing is a form of 'self feedback', that promotes learning. In the same way, taking time to reflect on other debater's viewpoints can induce a similar intellectual process.

...and Continuous

Draper et al (1996) says that an important constraint in evaluation is not to overload students. On the debate board they only express their opinion when they want to do so. This is not the case in traditional post-course evaluations, where many issues worth giving comments on are long since forgotten. Furthermore, it reduces the risk for students to think "Why should I bother to comment on this, when it's already too late for me to benefit from the impact of my viewpoints?" The possibility that suggested changes could be implemented and reported errors could be dealt with is likely to enhance students' willingness to give constructive feedback.

5. Conclusion

1. The discussion serves the purpose of supporting the creation of an online community of students and teachers where attitudes, expectations and opinions can be shared and discussed. The results reveal how such shared experiences, regarding for instance the students' frustration and satisfaction, are frequent topics for discussion. The existence of a meta-discussion on shared norms for the evaluative debate is also evidence of an online community.

2. The forum proved to be an interesting instrument of evaluation, with characteristics that complement traditional methods of evaluation. The results from the case study show discussions in many course related topics that are rich with constructive suggestions and argumentation. This feedback constitutes valuable input for teachers, but perhaps even more important, it makes it possible to challenge and influence students' attitudes and expectations. Especially in distance educational settings, where students and teachers seldom meet in person, it is important to identify ways of picking up sentiments and sources of irritation. A discursive medium like the discussion board presented in this paper should not be perceived as a formal instrument for objectively evaluating a learning context. On the contrary, it is a forum for conversations and discussions for the learning community of students and teachers, perhaps to some extent compensating for the loss of informal communication channels of the campus based education.

3. Finally the functionality and design characteristics of the application itself was argued to influence the nature of the discourse, thereby affecting both processes of community building and the educational evaluation. The continuous, public, asynchronous, and auto-structuring nature of the medium are all presented as key characteristics. The results also suggest that the graphical interface might impact aspects such as the lifetime of a discussion thread.

References

Bradner, E. & Kellogg, W. & Erickson, T. (1999). The adoption and use of 'Babble': A field study of chat in the workplace. Proceedings of the European Computer Supported Cooperative Work Conference (ECSCW'99).

Ciborra, C. U. (1996). Introduction: What does groupware mean for the organisation hosting it? In Groupware and Teamwork (p 1-22) John Wiley & Sons Ltd, Chichester, UK.

Draper, S. W., Brown, M. I., Henderson, F. P. & McAteer, E. (1996) Integrative evaluation: An emerging role for classroom studies of CAL. Computers & Education, 26(1-3), 17-32. [also at http://staff.psy.gla.ac.uk/~steve/IE.html verified 20 Aug 2002]

Draper, S. W. (1997a). Niche-based success in CAL. Proceedings of CAL'97.

Draper, S. W. (1997b). Feedback. http://staff.psy.gla.ac.uk/~steve/feedback.html [verified 20 Aug 2002]

Erickson, T. (1997). Social interaction on the net: Virtual communities as participatory genre. In J.F. Nunamaker, Jr. & R. H. Sprague, Jr. (Eds), Proceedings of the thirtieth Hawaii International Conference on Systems Science, Vol 6, pp. 23-300. IEEE Computer Society Press: Los Alamitos, CA, 1997.

Fjuk, A. (1998). Computer Support for Distributed Collaborative Learning: Exploring a complex problem area. Dr. Scient. Thesis 5 University of Oslo, Department of Informatics.

Glaser, Strauss (1980). The Discovery of Grounded Theory: Strategies for Qualitative Research. New York, Aldine Publications.

Goffman, E. (1969). The presentation of self in everyday life. London, Penguin.

Hall, J. (1997). Evaluating new technologies for teaching and learning in distance education: Current and future developments. In The New Learning Environments: A Global Perspectives. Proceedings from the 18th ICDEWorld Conference. The Pennsylvania State University.

Hara, N. & Kling, R. (1999). Students' frustration with a web based distance education course. First Monday, 4(12). [verified 20 Aug 2002] http://firstmonday.org/issues/issue4_12/hara/index.html

Laurillard, D. (1993). Rethinking University Teaching: A framework for the effective use of educational technology. Routledge, London.

Long, B. & Baeker, R. (1997). A taxonomy of Internet communication tools. Proceedings of WebNet 97, AACE, Charlottesville, USA.

Oliver, M. (1997). A framework for evaluating the use of educational technology. http://www.unl.ac.uk/tltc/elt/elt1.htm [verified 20 Aug 2002]

Orlikowski, W. & Yates J. (1994). Genre repertoire: The structuring of communicative practices in organizations. Administrative Science Quarterly, 39, 541-574.

Rheingold, H. (1993). The Virtual Communities. New York: Addison-Wesley.

Sproull & Kiesler (1991). Connections: New ways of working in the networked organization. The MIT Press, Cambridge Massachusetts.

Svensson, L. (2002). Interaction repertoire in a learning community. DVD-Proceedings of CSCL 2002. University of Boulder, Colorado, USA.

Svensson, L. & Sørensen, C. (2002). Designing community atmosphere barometers. Proceedings of 10th European conference on Information Science (ECIS 2002), University of Gdansk, Poland.

Whittaker, S., Frohlich, D. & Daly-Jones, O. (1994). Informal workplace communication: What is it like and how might we support it? In Human Factors in Computing Systems. CHI-94-4/94 Boston, Massachusetts, USA.

Wheeler, S. (1997). A framework for open and distance learning: Modes, methods and issues. Paper presented at CAL 97, Exeter University.

Yates, J. & Orlikowski, W. J. (1993). Knee-jerk anti-LOOPism and other email phenomena: Oral, written, and electronic patterns in computer-mediated communication. MIT Sloan School Working Paper No 3578-93.

Ågren, P-O. (1999). On the creation of social capital: A case study. Proceedings of IRIS 22 (Ed. Kekkölä), Keurru, Finland.

Author: Lars Svensson, Laboratorium for Interaction Technology University of Trollhättan Uddevalla, Box 795, 451 26 Uddevalla, Sweden Phone: +46 (0) 522 656020 Fax: +46 (0) 522 656099 Email: lars.svensson@htu.se Please cite as: Svensson, L. (2002). Discursive evaluation in a distributed learning community. Australian Journal of Educational Technology, 18(3), 308-322. http://www.ascilite.org.au/ajet/ajet18/svensson.html