Figure 1: Distribution of my postings over four modules
| Australasian Journal of Educational Technology 2005, 21(2), 155-172. |
AJET 21 |
This paper presents an example of an approach that involves both the student and the instructor in the evaluation of an online discussion in a context of teaching and learning. According to this approach, the student conducts a self analysis of his or her contributions to the discussion, using criteria supplied in advance of the discussion. One student's four part self analysis of his contribution to an online discussion is presented to illustrate the approach. The self analysis focused on the number and length of postings, claims and grounds, and on knowledge construction, and was designed to engage students in higher levels of thinking. The approach to self analysis is discussed in terms of its modification for use in other contexts and implications for practice are presented.
Vygotsky's models of learning involving social interaction of peers, the scaffolding by more able peers and the use of language as a mediation tool of cognitive development all have opportunities of enactment in the use of the discussion forum within a web course environment. (Maximizing Learning, ¶ 2)Disseminating, evaluating and discussing information and options, solving problems, thinking critically, co-constructing knowledge, and scaffolding by more able peers: these represent opportunities that might be afforded by and taken advantage of in an OAD. These opportunities and the benefits of online discussions do not, however, occur automatically. Some studies that have focused on the content analysis of transcripts of online discussions have found that attempts to engage learners in, for example, knowledge construction, critical thinking or problem solving have not always been successful. Kanuka and Anderson's (1998) analysis of the transcript of an online discussion had similar findings of that of Gunawardena, Lowe, and Anderson (1997). In both cases, coding of discussants' postings placed them in the lower phases of a knowledge construction process:
Most of the conversation was of a sharing and comparing nature. Dissonance and inconsistency were not actively explored, little testing of evidence against experience or the literature was expressed, and rarely did participants state the relevance or application of new knowledge that was created (Kanuka & Anderson, 1998, Discussion section para 2)Murphy's (2004) analysis of an OAD transcript for evidence of collaboration found that discussants did not reach high levels of collaboration but instead engaged in more individual rather than group oriented processes. Likewise, analysis of the transcript of an OAD designed to engage participants in problem formulation and resolution revealed that, while participants engaged in the resolution processes, they did not engage much in problem formulation even though this was an expected outcome in the design of the OAD (see Murphy, 2004b).
These results suggest that, even in cases where an instructor or designer may have intentionally structured, organised or moderated the discussion in order to promote engagement in a particular cognitive process such as knowledge construction, there is no assurance that these intentions will affect learning outcomes. This lack of assurance makes evaluation of the discussion a priority and can reveal whether or not students engaged in a particular intended process and if the discussion was effective in promoting understanding and learning. However, although it serves an essential purpose, the instructor's evaluation of the quality of postings of each student in a course can be an onerous, cumbersome and daunting task. As an alternative, instructors might opt to reply solely on simple quantitative analyses such as those that emphasise the value of the number and length of postings yet these will not provide in depth insights into the quality of learning.
An alternative approach to the evaluation of students' contributions to online discussions might be to directly involve students in a process of making sense of, interpreting and analysing their own performance in such contexts. Support for this approach comes from Knowlton (2000) who argues that "For the benefits of online discussions to be realised, students must have formal opportunities for self evaluation" (Broadening the Powers of Evaluation: Self Evaluation, ¶ 1). He proposes a 'Self Analysis Model' that encourages students to metacognitively examine their own contributions to a discussion forum.
Student self assessment and self analysis are recognised in the literature as effective approaches to evaluation that can result in the development of higher order thinking and metacognition skills (Hansen, 1994) and improved student learning (Black and Wiliam, 1998). Such approaches conceptualise assessment not only as a means for grading and reporting but as a means for learning. Self assessment also has theoretical support in Schön's (1987) work on reflective practice which conceptualises learning as cyclical, developmental process that assesses existing knowledge and identifies gaps in order to improve learning in the future.
The purpose of this paper is to presents an illustrative case of how students can conduct a guided and systematic qualitative and quantitative analysis of their contribution to an online discussion. The case highlights the types of high level analyses students might engage in order to summatively assess their performance as well as identify ways of improving their future learning in online discussions. According to this approach, students conduct a self analysis of their contributions to the discussion using criteria supplied in advance of the discussion by the instructor. The student can subsequently present his self analysis of the discussion to the instructor at the end of the course or activity. It is this analysis that the instructor can then evaluate in order to provide a grade related to the student's participation in and contribution to the discussion. The following section of this paper presents one student's self analysis of his contribution to a discussion forum. The student's presentation is preceded by a section on methodology which explains the requirements of and context for the analysis.
Part III required a critical assessment designed to demonstrate how the student advanced the discussion and promoted knowledge construction. The findings for this part had to be presented not numerically but by including quotes from the student's actual messages or postings. Part IV involved drawing overall conclusions related to the analysis both in terms of participation in the discussion and also in terms of completing the self analysis. In this regard, the student was expected to articulate what might be some of the implications for his future participation in online discussions.
This part of the analysis was also expected to include a reflection of how he could have participated more effectively in order to promote greater sharing and construction of knowledge. The following sections present an illustrative example of one student's self analysis as it was submitted to the instructor. The self analysis is followed by a discussion of this approach to the evaluation of participation in an online discussion. The discussion is followed by some general conclusions and consideration of some implications for practice.
| Module no. | Module title | Weeks | Postings |
| 1 | Multiple contexts: Multiple issues | 2, 3, 4 | 7 |
| 2 | Digital repositories and learning objects | 5, 6 | 2 |
| 3 | Asynchronous communication | 7, 8 | 2 |
| 4 | Collaborative online learning | 9, 10, 11 | 37 |
Figure 1: Distribution of my postings over four modules
As revealed in Table 1 and Figure 1, the majority of my postings were made in the completion of Module 4 of the course. To account for the larger number of postings in Modules 1 and 4, as compared with Modules 2 and 3, one can observe from Table 1 that Modules 1 and 4 were both spread out over a three week period, whereas Modules 2 and 3 were only two weeks in length. Module 4 also involved the completion of a small group (3 members) online collaborative project and it is interesting to note that approximately twenty-four of my thirty-seven messages were directly related to this project.
The total number of messages I posted during weeks two through to eleven was forty-eight (48). The average length per posting was
| total number of words | = | 8221 | ~ | 171 words |
| total number of postings | 48 |
My longest message of 1031 words was posted during Week 05, and was in response to a series of questions asked regarding the sharing of resources, an aspect of Module 2. I posted the shortest message of 6 words during Week 10 in the Group 3 section of the discussion forum. The posting was in response to a question asked by another group member regarding the version of Windows installed on my computer.
A summary of the above results is provided in Table 2.
| average length of postings | ~ 171 words |
| longest message posted | 1031 words |
| shortest message posted | 6 words |
Figure 2 below provides a visual representation of the length of my postings and the number that falls into the following categories of message lengths: 0-100 words, 101-200 words, 201-300 words, 301-400 words, 401-500 words, 501-600 words, 601-700 words, >700 words.
Figure 2: Summary of my posting length
My analysis revealed that I provided more than one type of evidence or proof in support of a particular claim. For the sake of simplicity, the chosen unit of analysis is the message itself, and though several claims may have been present in a particular message, the message has been categorised on the basis of evidence of a certain type of claim or claims. For example, if a posting made three Judgment/Value claims and a Policy claim, I classified it under "J/V & P".
| Type of claim | No. of messages providing evidence of claim type |
| Fact (F) | 1 |
| Judgment/Value (J/V) | 5 |
| Policy (P) | 2 |
| Fact and Judgment/Value (F & J/V) | 5 |
| Fact and Policy (F & P) | 0 |
| Judgment/Value and Policy (J/V & P) | 10 |
| Fact, Judgment/Value and Policy (F, J/V & P) | 5 |
| No Claim (NC) | 20 |
Figure 3: Claims contained in messages
Of my forty-eight messages, twenty-eight contain claims of various types. This is not to imply that only twenty-eight claims were made in contributions to the discussion forum. Instead, this means that twenty-eight of my postings contain evidence of a claim or claims being made of the various types. The results from the analysis of my postings in relation to the types of claims for the entire time period under consideration are provided in Table 3 and Figure 3.
Table 4 and Figure 4 illustrate the distribution of types of claims over the four modules.
| Type of claim | Number of messages providing evidence of claim type | |||
| Mod 1 | Mod 2 | Mod 3 | Mod 4 | |
| Fact (F) | 0 | 0 | 0 | 1 |
| Judgment/Value (J/V) | 2 | 0 | 0 | 3 |
| Policy (P) | 0 | 0 | 0 | 2 |
| Fact and Judgment/Value (F & J/V) | 2 | 0 | 1 | 2 |
| Fact and Policy (F & P) | 0 | 0 | 0 | 0 |
| Judgment/Value and Policy (J/V & P) | 3 | 1 | 1 | 5 |
| Fact, Judgment/Value and Policy (F, J/V & P) | 0 | 1 | 0 | 4 |
| No Claim (NC) | 0 | 0 | 0 | 20 |
Figure 4: Distribution of my claims over modules
Supporting what one has to say through evidence and/or proof is important in establishing credibility and in encouraging others to listen and respect a particular perspective or view. In my twenty-eight postings in which I made claims, I found evidence of the following grounds: anecdotal evidence, quotations, reports, findings, physical evidence, and forms of reasoning. I did not provide statistical evidence to support any of my claims.
My analysis revealed that I provided more than one type of evidence or proof in support of a particular claim. Since some of my messages consisted of two or three claims, I provided different grounds to establish credibility for the points being made. The first set of results reflects the grounds evidenced in all twenty-eight messages containing claims, and further analysis will reveal the various types of evidence and proof provided in each of the modules.
| Grounds | Number of messages using grounds type |
| Anecdotal Evidence (AE) | 14 |
| Quotations (Q) | 3 |
| Reports (REP) | 2 |
| Findings (F) | 17 |
| Physical evidence (PE) | 6 |
| Forms of reasoning (REAS) | 23 |
Figure 5: Grounds used in messages containing claims
As depicted in Table 5 and Figure 5, I relied on various forms of reasoning, findings, and anecdotal evidence for the grounds related to the various claims. Table 6 and Figure 6 provide a breakdown of the grounds I used in the claims over the four modules.
| Grounds | Number of messages providing evidence of grounds | |||
| Mod 1 | Mod 2 | Mod 3 | Mod 4 | |
| Anecdotal Evidence (AE) | 3 | 1 | 1 | 9 |
| Quotations (Q) | 0 | 1 | 1 | 1 |
| Reports (REP) | 1 | 0 | 0 | 1 |
| Findings (F) | 3 | 2 | 2 | 10 |
| Physical Evidence (PE) | 1 | 1 | 0 | 4 |
| Forms of Reasoning (REAS) | 6 | 2 | 2 | 13 |
Figure 6: Grounds evidenced in the four modules
On closer examination of the course modules, I observed that I relied extensively on anecdotal evidence, findings, and forms of reasoning to provide evidence and proof for the claims I made. It is interesting to note that twenty-five of the twenty-eight messages that make claims are categorised as a judgment/value claim. Claims were only categorised as 'fact' if empirical evidence was provided in the message or if I made the claim based directly on research I had previously conducted. Modules 2 and 3 both contain only two postings, yet, all three types of claims are evidenced in each of the modules through the two messages. The length and detail of my postings, no doubt, contributed to this result.
The twenty messages which I characterised as having 'No Claim' all came from the completion of Module 4. Many of these messages were from the Group 3 section of the discussion forum and reflect the communication that occurred in organising and participating in a group project. Several of the postings are of a social nature, and though such postings may not necessarily contribute to the construction of new knowledge, they can be very important in establishing group dynamics and ensuring the eventual success of the collaborative project.
Message no. 468Normally, when I decide to contribute to a large group discussion, whether face to face or online, it is because I believe I have something significant to add. My online postings are usually a result of careful reflection and are well thought through. The seemingly average number of postings is typical of my contribution to such a discussion forum. The tables and figures presented throughout this analysis provide evidence that many of my postings contained numerous claims of varying types, and a variety of grounds was used in supporting these claims.
Posted by Jamie Loveless on Sunday, July 6, 2003 1:38pm
Subject Jamie's Application Activity
Online Discussions......by nature I am a relatively quiet person (those who know me personally can attest to this...I think ;-)) and even in a group setting among peers where conversation is occurring I am not an outspoken person...
One of the ways I try to contribute to a discussion is through not only responding to questions, but through posing more questions and disrupting mindsets as in the following examples:
(i).I tend to use a similar questioning technique in responding to the postings of others. The intent is to not only try and engage further discussion, but to have readers of the message search within themselves for possible answers to the question and, in turn, facilitate the construction of new knowledge, or restructure existing knowledge domains. An example of this questioning is evident in the following excerpt of a posting, given in response to a question asked by another student:
...do on-line courses tap into a certain independent learning style that many of our current students fail to possess?(ii).
...but what will happen to those students who, for whatever reason, are unable to make the necessary adjustment from the traditional classroom to the virtual classroom setting?
...Why, too, do we assume that our higher achieving students will succeed, regardless of the manner in which a course is offered? Is this always fair to our higher achieving students?
I sometimes wonder if it is really the scheduling of the online tutorials that is the issue, or is it the fact that teachers are not really comfortable with the medium through which they are being asked to learn?!...or is it a combination of both?In trying to advance the discussion and promote knowledge building, I aim to include a personal element in my posting to exemplify metacognitive activity taking place. If other learners witness the occurrence of personal reflection and introspection, they may be inclined to do the same. Challenging one's own thoughts and ideas and the creation of dissonance is a significant step in the learning process. The following two excerpts of postings illustrate this idea:
Are some of the demands and expectations being placed on teachers who may have limited experience with technology too great?
(i).In trying to avoid repetition among postings, which would detract from reader interest, I attempted to add novel thoughts and ideas to stimulate further discussion and broaden awareness of a particular topic or issue. This is evidenced in the following preamble to the main contents of the message:
It can be rather startling to "take a step back", on occasion, to observe and reflect on how quickly things are changing in our education system,...(ii).
... This, I know, is a personal reflection and may not apply to others in the class.
The responses posted thus far do a superb job explaining the importance of online collaboration and provide sound reasoning as to why online collaboration is used in the design and development of online courses.I included references in some of my postings to either support my claims or to provide an example of some aspect of the contents of my message. On some occasions, I provided online references to valuable math Web sites I felt would benefit others as a result of my sharing them:
A few additional thoughts...
(i).In relation to the communication and interaction that occurred in the design and development of the Group 3 collaborative project, I also made important contributions to the social dimension of the discussion. Although the contents of these twenty messages may not have directly contributed to the construction of knowledge, they played a role in establishing group dynamics and in facilitating the collaborative learning process. Even informing the group of my personal situation throughout the last week of the group project was critical in maintaining the level of trust and confidence that had been established within our group. Many of my messages contained positive feedback in relation to the group's efforts, thus contributing to the sense of morale that characterised our group. The following three examples provide evidence of this contribution:
...Consider, for example, the following Web site, useful for examining transformations of quadratic functions:
http://www.exploremath.com/activities/Activity_page.cfm?ActivityID=14(ii).
...free "Peanut Software" made available by Richard Parris at http://peanut.8m.net/, are examples of tools that can be used by mathematics educators and students.
(i).I argue that the absence of this social, 'human', element would have hindered the group's eventual success and detracted from the learning that occurred through completion of the collaborative project.
That's perfect. Also, if I happen to find any decent articles relating to topic 2, I'll let you know.
I would like to delve into topic 3...if that's ok!?!
Keep in touch!
Jamie :-)(ii).
WOW!!!!!!!!
The work you have done in the construction of the Web site is unbelievable...(iii).
As you are probably aware, this week is going to be very hectic for me...getting married Saturday!...
The above examples illustrate various ways I consciously attempted to advance the discussion and promote knowledge construction. It is difficult, however, to determine the true effectiveness of my contributions in relation to the learning experienced by other students. From a personal perspective, I know there were many postings from other students that stimulated my thinking and that resulted in the reconstruction and reshaping of my own knowledge, yet I did not respond to these postings in the discussion forum. This observation supports Kanuka and Anderson's (1998) argument that "It may also be possible that the construction of knowledge is not an observable activity" (Discussion, ¶ 5) and "individual participants might be processing information internally in a reflective manner but not sharing these thoughts with other participants" (Results, ¶ 16).
In conclusion, by conducting a self analysis of my contribution to an online discussion forum, I have developed an increased awareness of the potential benefits resulting from ongoing participation in this asynchronous mode of communication, including the sharing of experiences, challenging one's own thoughts and frames of cognition, the construction of new knowledge, and promoting collaborative efforts. As well, this activity helped me reflect on how I present my ideas and views in an online setting and how I tend to interact with others in a virtual environment. Through this self analysis, I have discovered ways I can potentially improve my participation in online discussion forums which will prove valuable in similar virtual settings in the future.
Part III was designed to bring the student to a deeper level of analysis which involved locating evidence of how the postings might have advanced the discussion and contributed to knowledge construction. This part of the analysis was also expected to draw on Parts I and II in order to evaluate the consequences of how, for example, the length of postings or their distribution might have affected others in the discussion. Part III engaged the student in cognitive processes related to evaluation, judging and critiquing. Finally, Part IV engaged the student in hypothesising, planning, drawing conclusions and considering implications. For example, it was at this stage that the student could consider the results of what might have happened in the discussion if he had participated differently.
The self analysis first focused on simple identification and descriptions of the number and length of postings, claims and grounds and on knowledge construction. It also required organising this information using graphs, charts and direct quotes from the discussion. The analysis was designed to engage the student, not only in cognitive processes related to describing or identifying but in higher levels of thinking such as interpretation, evaluation and concluding. Other contexts of self analysis of an online discussion might require that students assess additional aspects of their participation in and contribution to an online discussion. For example, students might be expected to determine and show evidence of how they engaged in critical thinking or in problem solving. Under these circumstances, the discussion would need to be structured or moderated to intentionally elicit this type of participation from students.
Students' results of their self analysis do not necessarily need to be presented using spreadsheet programs as was the case in the example presented in this paper. Students could use a table or simply not be asked to include a quantitative assessment of their postings. While the requirements of the self analysis can therefore vary depending on the context, there will always be a need to ensure that students engage in a variety of levels of cognitive processing. Additionally, this engagement should occur at the higher levels of analysis and interpretation and not simply at the levels of describing and recognising. Thus, instructors interested in involving their students in the assessment of their participation in and contribution to an online discussion will want to ensure that the self analysis, regardless of whether it involves critical thinking, problem solving or argumentation etc, requires engagement in higher level processes.
At the same time, requiring of students that they complete a self analysis will mean that instructors will have to clearly articulate the expectations for and requirements of participation in the discussion. It will not be enough to simply specify participation requirements in terms of numbers of postings. Nor will it be sufficient to require of students that they think critically, problem solve or contribute meaningfully to the discussion. Specific processes and indicators of those processes will need to be articulated and ideally some models of best practices provided.
As an example, an expectation that students engage in critical thinking will require of the instructor or designer of the online discussion that they present prompts and activities related to the construct of critical thinking. Instructors or designers can refer to the literature on critical thinking to determine the specific indicators and activities that might be suited to that construct. Ultimately, students' self analyses may help the instructor gain insight into and evaluate the degree to which he or she has effectively structured, organised and moderated the discussion. In this regard, the experience of students' self analysis of the discussion can become a learning tool for the instructor as well as the student.
Gunawardena, C., Lowe, C. A., & Anderson, T. (1997). Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing. Journal of Educational Computing Research, 17(4), 397-431.
Hansen, J. (1994). Literacy portfolios: Windows on potential. In S. W. Valencia, E. H. Hiebert & P. P. Afflerrbach (Eds), Authentic Reading Assessment: Practices and Possibilities (pp. 26-44). Newark, DE: International Reading Association.
Jonassen, D. H. (2000). Computers as mindtools for schools: Engaging critical thinking (2nd ed.). Upper Saddle River, NJ: Prentice-Hall Inc.
Kanuka, H. & Anderson, T. (1998). Online social interchange, discord, and knowledge construction. Journal of Distance Education, 13(1). [viewed 27 Jun 2003] http://cade.athabascau.ca/vol13.1/kanuka.html
Knowlton, D. S. (2000). Promoting durable knowledge construction through online discussion. Middle Tennessee State University [viewed 31 Jul 2003] http://www.mtsu.edu/~itconf/proceed01/11.pdf
Markel, S. L. (2001). Technology and education online discussion forums: It's in the response. Online Journal of Distance Learning Administration, 4(2). [1 Aug 2003] http://www.westga.edu/~distance/ojdla/summer42/markel42.html
Murphy, E. (2004). Recognizing and promoting collaboration in online asynchronous discussions. British Journal of Educational Technology, 35(4), 421-431.
Murphy, E. (2004b). Identifying and measuring problem formulation and resolution in online asynchronous discussions. Canadian Journal of Learning and Technology, 30(1), 5-20.
Murphy, E. (2003). Syllabus for Education 6620. [viewed 30 Jun 2003, not found 26 Apr 2005] http://webct.mun.ca:8900/SCRIPT/82157200203/scripts/serve_home
Schön D (1987). Educating the reflective practitioner. London: Jossey-Bass Publishers.
Toulmin, S. (1958). Uses of Argument. New York: Cambridge University Press.
| Authors: Elizabeth Murphy is an Associate Professor of Educational Technology and Second-Language Learning at Memorial University of Newfoundland, St. John's, Newfoundland, Canada. She has conducted several studies related to use of online asynchronous discussions in teaching and learning. These studies have focused on collaboration, critical thinking, and on problem formulation and resolution.
Jamie Loveless is a high school math teacher at Exploits Valley High (Maple Division), Grand Falls-Windsor, NL, Canada. He has recently completed the Master of Education Degree in Teaching and Learning offered through the Memorial University of Newfoundland. Dr Elizabeth Murphy, Associate Professor, Faculty of Education Memorial University of Newfoundland St. John's, Newfoundland, Canada, A1B 3X8 Tel: (709) 737 7634 Email: emurphy@mun.ca Website: http://www.ucs.mun.ca/~emurphy/ Jamie Loveless, Teacher, Exploits Valley High (Maple Division) Nova Central School District Grand Falls-Windsor, NL, Canada, A2A 1T6 Tel: (709) 489 4374 Email: jllovele@cdli.ca Website: http://www.angelfire.com/ma4/jloveless/ Please cite as: Murphy, E. and Loveless, J. (2005). Students' self analysis of contributions to online asynchronous discussions. Australasian Journal of Educational Technology, 21(2), 155-172. http://www.ascilite.org.au/ajet/ajet21/murphy.html |