|Australian Journal of Educational Technology
2003, 19(3), 371-387.
Recent research in content analysis has shown the difficulties of achieving acceptable levels of inter-rater reliability with CMC transcripts. This has lead to the development of semi-structured computer conferencing systems, in which participants choose the type of contribution that they are making from a limited set of alternatives. This article extends previous work with respect to semi-structured approaches to online discourse, suggesting coding categories relevant for problem based learning. The SQUAD approach to online discourse offers definitions for quality with respect to participation, interaction and cognition, when using the message as the unit of CMC transcript analysis, analysing for what the author has termed as 'online learning levels of engagement'. It is argued that the theoretical basis underpinning the SQUAD approach is beneficial for the development of teamwork and cognitive reasoning when learning in small groups, and that it is a relatively straightforward exercise to apply this approach in a different mode of study or subject area.
The SQUAD approach to CMC discourse adopts problem based learning (Barrows, 1996; Bridges, 1992; Oriogun et al, 2002) as an instructional method with the goal of solving real problems by:
In what ways can we measure the quality of online learning levels of engagement with respect to 'participation', 'interaction' and 'cognition'?
Other approaches (eg., Cook, 2001; Baker, 1994) are based on speech act theory, (Austin, 1962; Searle, 1969). In 'classical' speech act theory, only invented, isolated utterances are considered. More recently in speech act theory, dialogue is viewed as a sequence of speech acts, uttered by each party to achieve certain goals. For example, Cook (2001) presents an approach to using empirical data on human teacher-learner interactions to guide the development of a pedagogical agent for supporting musical composition learning. In addition, dialogue analysis is also used to ascertain whether or not a user interacts within a CMC system in a productive manner, in terms of the learning task outcome. This latter approach is the focus of the work described in this article. The SQUAD approach to CMC discourse invites students to post messages based on five given categories, namely, Suggestion, Question, Unclassified, Answer and Delivery.
By contrast, Howell-Richardson and Mellar (1996) used illocutionary act (from speech act theory) as a unit of analysis when they investigated participation, illocutionary properties and focus groups. Hara, Bonk and Angeli (2000) used paragraph as a unit of analysis for the same five variables as Henri (1992). Fahy et al. (2000) investigated interaction, participation and critical thinking, using sentence as a unit of analysis. Oriogun (2003) used message as a unit of analysis when he investigated participation and interaction.
A number of theories on knowledge building emphasise the socially distributed nature of cognition. Distributed cognition is therefore a process whereby individual cognition is extended to acquire something that an individual would be unable to achieve alone. Knowledge is constructed in associated networks of concepts and nodes. As learning occurs, new information is collected and coupled to existing knowledge networks. New information can then be easily retrieved to solve problems, and to apply in context. Students are expected to learn about the world based on their own research and study. Students determine their "knowns" and "unknowns". They seek knowledge to address their "unknowns". They engage in collaborative learning in their small groups to work on the problems (Wee, Kek & Sim, 2001, p159).
Bruer (1993) argues that learning is quicker when students possess self motivating skills generally referred to as metacognitive skills. Learning in PBL encourages metacognitive skills. In line with our usage of PBL in this article, we have adopted the adaptation of Henri's (1992) descriptors for "Reasoning Skills" as suggested by (Hara, Bonk & Angeli, 2000) in support of cognitive indicators for the SQUAD approach. See Table 1 for these descriptors.
|Elementary clarification||Observing or studying a problem, identifying its elements, and observing their linkages in order to come to a basic understanding.|
|In depth clarification||Analysing and understanding a problem to come to an understanding which sheds light on the values, beliefs, and assumptions which underlie the statement of the problem.|
|Inferencing||Induction and deduction, admitting or proposing an idea on the basis of its link with propositions already admitted as true.|
|Judgement||Making decisions, statements, appreciations, evaluations and criticisms. Sizing up.|
|Application of strategies||Proposing coordinated actions for the application of a solution, or following through on a choice or a decision.|
In line with the cognitive indicators underpinning the SQUAD approach (see Tables 1 and 3 respectively) the OLM was constructed. The OLM details the levels of online levels of engagements when using this new approach to categorising CMC discourse. Figure 1 is the consolidation of Table 2 in terms of «Low«, «Nominal« and «High« levels of online engagements. Consequently, messages posted with the title Question or Q, and Unclassified or U are deemed to be generally of "Low Level of Online Engagement"; messages posted with title Suggestion or S, and Delivery or D are deemed to be generally of "High Level of Online Engagement"; finally, messages posted with the title Answer or A remains as "Nominal". Therefore, depending on the level of granularity required, Figure 1 could also have been drawn showing all the five hierarchical ordering in Table 2.
|Message Category||Very Low||Low||Nominal||High||Very High|
|S - Suggestion||X|
|The process whereby the mere presentation of an idea to a receptive individual leads to the acceptance of the idea.||Students engage with other students within their coursework groups by offering advice, a viewpoint, or an alternative viewpoint to a current one.||-Elementary classification
-In depth classification
-Application of strategies
|Students engaging other students actively by taking a lead role in online discourse by posting meaningful and relevant messages to the group.||The message will be accessed and processed by other members of the group for the cycle of communication to complete.|
|A form of word address to a person in order to elicit information or evoke a response.||Students may seek clarification from the tutor or other students in order to make appropriate decisions relating to the group coursework.||-Elementary classification
|The message is posed in such a way that some or all the group members will engage in the ongoing discussion.||The message will be accessed and processed by other members of the group for the cycle of communication to complete.|
|Not in the list of categories of messages stipulated by the instigator of the task at hand.||This tends to happen at the start of the online postings. Students may be unsure of what the message is suppose to convey. In most cases, it falls within one of the four classified categories.||-Elementary classification||This type of message may or may not engage other students. In most cases, the message could be re-aligned to fall within the four classified categories by the coder of the final transcript at the end of semester.||This type of message may or may not engage other students. In most cases, the message could be re-aligned to fall within the four classified categories by the coder of the final transcript at the end of semester.|
|Reply, either spoken or written, as to a question, request, letter or article.||Students are expected to respond to this type of message with a range of possible solutions / alternatives.||-Elementary classification
-In depth classification
|Responding to a query or question will inevitably involve most, if not all the group members, especially if the response is not in line with other group members' opinions.||All group members are expected to deliver parts of the final product by working collaboratively.|
|The act of distribution of goods, mail etc.||Students are expected to produce a piece of software at the end of the semester. They all have to participate in delivering aspects of the artefacts making up the software.||-Elementary classification
-Application of strategies
|Each member of the group is expected to play active role in delivering parts of the artefact making up the final software product. This is also expressed in the marking scheme for the module.||All group members are expected to deliver parts of the final product by working together collaboratively.|
Each group was assigned a designated tutorial assistant. The author was the sole lecturer on the module. The tutorial assistants assisted the students during the laboratory sessions dealing with the practical aspects of the coursework for the module. There were two assessments for the course: group coursework (50%) and a final examination (50%). Group coursework had an individual element attracting 17% of the marks (this was the CMC discourse of the module). The SQUAD approach was adopted for the CMC discourse of the module throughout the semester. The author directed his students to use a previously developed, enhanced Problem Based Learning Grid (Oriogun et al., 2002) and its predecessor (Oriogun & Georgiadou, 2000) to help them to promote learning spaces as environments that provide the learner with the responsibility for their own development and allow the learner to participate and be incumbent in a new social dynamics (Oriogun et al., 2002 p497).
The five categories for the SQUAD approach were fully explained to the students at the beginning of the semester, namely that, when a student wishes to make a suggestion to the rest of the group regarding their ongoing assignment, the posting title must be Suggestion (S); in the case asking the group a specific or general question, the posting title must be Question (Q); if a student is answering a question the posting title must be Answer (A); where a student is delivering aspects of the artefacts for the group assignment the posting title must be Delivery (D); finally, the most demanding category to rate is the unclassified category. Postings leading to or with the potential of being categorised as Unclassified (U) are as follows:
The whole idea of the SQUAD approach is to reduce the inter-rater reliability measure of CMC discourse. It is therefore expected that once the categorisation has been explained to students fully as documented above, the only category that may be problematic is the Unclassified or U titled postings. The author re-categorised 13% of the messages posted under the category Unclassified (see more details under the results section of this article). Six of the actual messages posted by the students who participated in the study are shown in the Appendix. These were randomly selected to represent all of the categories in the SQUAD. Figure 1 below shows the contributions of the students (numbered S1 to S6) and the author during the semester.
Figure 1: Participant's online learning levels of engagement (see Table 2)
using the SQUAD approach to CMC discourse
The author sent 19.5% of all the messages posted throughout the semester. Out of the total of 45 messages sent by the author, 20% were of High level, 44% were of Nominal level, and 36% were of Low level. The author contributed most of the category 'S', 'U', and 'A' messages. The tutor asked very few questions, and he did not post any messages under 'D' for delivery of an artefact towards the students' coursework. Figure 1 above shows a graphical representation of the students' online learning levels of engagement.
Part of the project was to set up and use a yahoo groups account in order to record communication between members of the group, which was both assessed by and overseen by one of the course tutors. It was felt that the inclusion of one of the tutors created an artificial environment, partly because this meant that the developers were unable to express their feelings openly and also that some of the messages that were being posted to the site could be interpreted as seeking to impress the supervising member of staff rather than enhance the overall level of communication throughout the group.This coursework brought the students together as they had not worked together previously within a Problem Based Learning environment at the University. The group of students further reported on their experiences as part-time students undertaking a software engineering project as quoted below:
...we encountered a number of problems because we are part-time evening students. One of these problems was to work on files outside of the University computing facilities. We were effectively prevented from using FTP and Telnet, which meant any testing could only be done during opening hours of the computing building at the university. These hours were very limited. Eventually we were able to overcome the problems and connect to the department of computing server, but this resulted in the loss of a week's work at least... It is a testament to the dedication of the group that we were able to meet up as frequently as was achieved, and would probably go some way to explain the dependency on using other forms of communication, including the very large number of messages sent through the yahoo groups siteThe SQUAD approach is a semi-structured computer conferencing system, it is however, advisable for anyone interested in adopting the SQUAD for capturing what the author has termed as "online learning levels of engagement" to first experiment with either an inter-rater reliability measure (Cohen 1960; Henri 1992; Hara, Bonk & Angeli 2000; Fahy 2001) or the Transcript Reliability Cleaning Percentage TRCP as proposed by Oriogun (2003), in order to generate appropriate acronym from the initial categorisation of CMC messages within their subject discipline. The SQUAD approach is simple enough for academics with limited mathematical background and can provide insights into the nature of the learning process from the prospective of categorisation of online discourse.
It has been suggested by Mason & Romiszowski (1996) that
The most glaring omission in CMC research continues to be lack of analytical techniques applied to the content of the conference transcripts. Given that the educational value of computer conferencing is much touted by enthusiasts, it is remarkable that so few evaluators are willing to tackle this research area. (p.443).The SQUAD approach to CMC discourse is a way of addressing this suggestion by offering a theoretical framework that facilitates what the author is calling the "online learning levels of engagement".
Future extension to this research will be to develop the theoretical framework to include social and metacognitive elements within a CMC message, and to develop a dedicated tool for students to use when adopting the SQUAD approach to CMC discourse.
Baker, M. J. (1994). A model for negotiation in teaching-learning dialogues. Journal of Artificial Intelligence in Education, 5(2), 199-254.
Barrows, H. (1996). Problem-based learning in medicine and beyond: A brief overview. In L. Wilkerson and W. Gijselaers (Eds), Bringing Problem-Based Learning to Higher Education: Theory and Practice. New Directions for Teaching and Learning, 68, 3-11. San Francisco: Jossey-bass Publishers.
Barros, B. & Verdejo, F. (2000). Analyzing student interaction process in order to improve collaboration. The DEGREE approach. International Journal of Artificial Intelligence in Education, 11, , 221-241. [verified 9 Nov 2003] http://computing.unn.ac.uk/staff/cgpb4/ijaied/members00/archive/vol_11/barros/full.html
Boehm, B., Bose, P., Horowitz, E. & Lee M. J. (1995). Software requirements negotiation and renegotiation aids: A theory-W based spiral approach. IEEE Proceedings for the 17th ICSE Conference, 1995.
Bridges, E. M. (1992). Problem-based learning for administrators. ERIC Clearing House, University of Oregon.
Brine, J. & Shannon, S. (1994). Consolidating professional skills and developing the confidence of graduating students. In S. E. Chen, R. M. Cowdry, A. Kingsland & M. J. Ostwald (Eds), Reflections on Problem Based Learning. Sydney, Australia: Wild & Wooley Pty Ltd.
Bruer, H. S. (1993). Schools for Thought: A Science of Learning in the Classroom. Cambridge, MA: MIT Press.
Cook, J. (2001). Bridging the gap between empirical data on open-ended tutorial interactions and computational models. International Journal of Artificial Intelligence in Education, 12, 85-99. [verified 27 Oct 2003] http://computing.unn.ac.uk/staff/cgpb4/ijaied/members01/archive/vol_12/cook/full.html
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurements, 20, 37-46.
Duffy, T., Dueber, B. & Hawley, C. (1998). Critical thinking in a distributed environment: A pedagogical base for the design of conferencing systems. In C. Bonk & K. King (Eds). Electronic contributors: Learner-centered technologies for literacy, apprenticeship, and discourse (pp. 51-78). New Jersey: Erlbaum.
Fahy, P. J., Crawford, G., Ally, M., Cookson, P., Keller, V. & Prosser, F. (2000). The development and testing of a tool for analysis of computer mediated conferencing transcripts. Alberta Journal of Education Research, 46(1), 85-88.
Fahy, P. J. (2001). Addressing some common problems in transcript analysis, International Review of Research in Open and Distance Learning, 1(2). http://www.irrodl.org/content/v1.2/research.html#Fahy [viewed 24 Mar 2003, verified 18 Sep 2003]
Foreman, J. & T. Johnston. 1999. Key aspects of teaching and learning in business and management studies. In H. Fry, S. Ketteridge, and S. Marshall (Eds), A Handbook for Teaching and Learning in Higher Education, 372-390. London: Kogan Page.
Hara, N., Bonk, C. & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28(2), 115-152.
Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed), Collaborative learning through computer conferencing: The Najaden papers, pp 117-136. London: Springer-Verlag.
Howell-Richardson, C. & Mellar, H. (1996). A methodology for the analysis of patterns of participation within computer mediated courses. Instructional Science, 24, 47-69.
Hutton & Wiesenberg, (2000). Quality online participation: Learning in CMC classroom. RCVET Working Knowledge Conference Papers. Research Centre for Vocational Education and Training, University of Technology, Sydney, Australia, 10-13 Dec 2000. [viewed Mar 2003, verified 18 Sep 2003] http://www.rcvet.uts.edu.au/wkconference/working%20knowledge64.pdf
Mason, R. & Romiszowski, A. (1996). Analysing computer conferencing interactions. Computers in Adult Education and Training, 2(3), 161-173.
McDonald, J. (1998). Interpersonal group dynamics and development in computer conferencing: The rest of the story. In Proceedings of 14th Annual Conference on Distance Teaching and Learning, pp. 243-48. Madison, WI: University of Wisconsin-Madison [ERIC Document ED422864]
Oriogun, P. K. & Georgiadou, E. (2000). Towards making learning agents think of their roles in the learning process through a problem-based learning grid. Proceedings, 2nd Asia Pacific conference on PBL, Singapore, 4-7 Dec 2000. [verified 18 Sep 2003] http://www.tp.edu.sg/pblconference/full/OriogunGeorgiadouPaper.pdf
Oriogun, P. K., French, F. & Haynes, R. (2002). Using the enhanced Problem-Based Learning Grid: Three multimedia case studies. In A. Williamson, C. Gunn, A. Young & T. Clear (Eds), Winds of Change in the Sea of Learning: Proceedings of the ASCILITE Conference. Auckland, New Zealand: UNITEC Institute of Technology, 8-11 December 2002, pp495-504. http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/040.pdf
Oriogun, P. K. (2003). Content analysis of online inter-rater reliability using the transcript reliability cleaning percentage: A software engineering case study. Presented at the ICEIS 2003 Conference, Angers, France, 23-26 April 2003, pp.296-307, ISBN 972-98816-1-8.
Pilkington, R. (2001). Analysing educational dialogue interaction: Towards models that support learning (Introduction to the IJAIED Special Issue on Analysing Educational Dialogue). International Journal of Artificial Intelligence in Education, 12, 1-7. [verified 9 Nov 2003] http://computing.unn.ac.uk/staff/cgpb4/ijaied/members01/archive/vol_12/pilkington/full.html
Pilkington, R. M. (1999). Analysing Educational Discourse: The DISCOUNT Scheme (CBLU Technical Report 99/2). Leeds: Computer Based Learning Unit, The University of Leeds, Leeds, LS2 9JT.
Ryder, M. (1994). Augmentation of the intellect: Network instruments, environments and strategies for learning. [viewed 28 Jan 2003, verified 18 Sep 2003] http://carbon.cudenver.edu/~mryder/augment.html
Searle, J. R. (1969). Speech Acts: An essay in the philosophy of language. Cambridge: Cambridge University Press.
Strauss, A. and Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage: London.
Vygotsky, L. S. (1962). Thoughts and Language. Cambridge, Mass: MIT.
Wee, K. N., Kek, Y.C., & Sim H. C. (2001). Crafting effective problems for problem based learning. In P. Little & P. Kandlbinder (Eds), The Power of Problem Based Learning, 157-168. ISBN 0 7259 11387. Australia: Australian Problem Based Learning Network.
Weiss, R. & Morrison, G. (1998). Evaluation of a graduate seminar conducted by listserv. [ERIC Document Reproduction Service, ED 423868]
He is the customer so he would need to answer questions that you
would ask any customer. It is a fairly vague specification so it is a
very good idea to go back to him with a list of options.
From: "student 1" <student1 _email>
Date: Tue Nov 19, 2002 11:15 am
Subject: Q - Formatting messages
This is a petty question, but...
When we paste emails into the document - should we include the
original message, if a message is a reply?
I would say not, but we need to be consistent. Any thoughts?
From: "student 3" <student 3_email>
Date: Wed, 25 Sep 2002 22:10:11 -0000
Subject: U - [im54p21] Re: Help!
Was having similar problems earlier on, but everything seems to be OK now.
From: student 4 <student 4_email>
Date: Tue Nov 12, 2002 1:15 pm
Subject: A - web pages
Student 2 / student 5,
Both login and registration will have one page. The site will allow
access and give message post/reply functions to both message streams
(undergrad and postgrad) for any registered user.
Hope that answers it. Any chance of emailing draft html pages so
I can link some code to it? Thanks
From: student 5 <student 5_email>
Date: Tue Oct 15, 2002 7:52 am
Subject: D - an explanation as to life cycle plans
continuing my very busy morning, I have now put a file on with my
explanation for the life cycle plan I hope that it is readable, for this time
of the morning - I hope that this busy morning is taken into consideration
when I am completely incoherent at this evenings meeting
can't we do genetic engineering instead ?
|Author: Peter K. Oriogun|
Learning Technology Research Institute
Department of Computing, Communications Technology and Mathematics
London Metropolitan University
166-220 Holloway Road, London N7 8DB
Please cite as: Oriogun, P. K. (2003). Towards understanding online learning levels of engagement using the SQUAD approach to CMC discourse. Australian Journal of Educational Technology, 19(3), 371-387. http://www.ascilite.org.au/ajet/ajet19/oriogun.html