Evaluating quality in online asynchronous interactions between students and discussion facilitators

Nandi, D.; Hamilton, M.; Chang, S..; Balbo, S.

Australasian Journal of Educational Technology
2012, 28(4), 684-702.

Evaluating quality in online asynchronous interactions between students and discussion facilitators

Dip Nandi, Margaret Hamilton
RMIT University
Shanton Chang
The University of Melbourne
Sandrine Balbo
Nuance, Australia

Online discussion forums have become an essential part of university courses, whether the course is conducted online, or face to face, or in mixed or blended mode. Discussion forums are considered to engage students better with the course content and encourage them to share and gain knowledge from each other. However, online engagement does not always happen automatically between students. Hence grading of discussion forum participation has been recommended to ensure quality student participation. Currently, a major focus has been put onto the better use of discussion forums, but the way in which the quality of participation can be evaluated has yet to be adequately investigated. Furthermore, evaluation of the instructor participation in a discussion forum and its impact on students and their contributions is lacking. In this paper, we report on our research into online discussion forum quality through analysis of discussion forum activities, along with student focus group meetings and instructor interviews. We have devised a set of criteria for evaluating discussion forum activities. Our results show that students depend highly on the instructor's feedback and the participation of the students can only be evaluated with reference to the moderation of the instructors.

Introduction

Since the introduction of the Internet and the adoption of learning management systems, discussion forums have been used and assumed to ensure the asynchronous interaction between learners and instructors. Online asynchronous discussion is argued to have many benefits for student learning, such as helping learners negotiate higher levels of understanding, share and develop alternative viewpoints (Rovai, 2000). In an earlier paper we developed a conceptual framework for assessing interaction quality in online discussion forums and noted that existing criteria do not focus on interaction or engagement or quality but instead they focus on content and objective measures such as participation rate (Nandi, Chang & Balbo, 2009; see Appendix A). According to Hawkes and Dennis (2003), establishing clear criteria for the assessment of online discussion is critical for successful use of this medium. A comprehensive framework with specific criteria assists in determining the quality of interaction. It can also assist students to manage and understand their own contributions to the online discussion. Jackson (2010) stated that a well defined framework not only evaluates the overall effectiveness of discussions, but also makes explicit and transparent to students the expectations of their engagement in discussions, and thereby shape that engagement. In this paper, we investigate the framework (Nandi et. al., 2009) by using it to evaluate the quality of learners' participations and their interaction with other learners and instructors, based on what is posted in the discussion forum. This leads to our research question:

How can assessors evaluate quality in online asynchronous interactions between students and between students and their students' facilitators?

Online learning and interaction

Online learning systems have been described as web based learning environments consisting of digitally formatted content resources via the use of the World Wide Web and communication devices to provide communication link between the instructor and students where they can actively interact (Piguet & Peraya, 2000).

"Interaction" has been recognised as the most significant attribute in any online system or course. The importance of interactivity is highlighted by several researchers who have conducted research in online learning systems (Maor & Volet, 2007; Al-Mahmood & McLoughlin, 2004; Sharples, 2000). Without interactivity, a discussion forum simply becomes a bulletin board for posting messages and information.

Discussion forum and assessment framework

Discussion forums have frequently been used successfully as communication tools in online learning environments to facilitate interaction between students to share knowledge (Rovai, 2002; Bradshaw & Hinton, 2004; Berner, 2003). There are different levels of participation in discussion forums. Firstly some are "lurkers" (Salmon, 2003) i.e. who just read the messages and don't participate. They may learn by reading the posts and incorporating the ideas into their assignments (Guzdial and Carroll, 2002). Secondly some people read the messages and treat it as a notice board posting their own position and having limited interactivity. Thirdly the participation is interactive and used to its full potential (Ho, 2002) for learning where collaboration and interaction facilitates the achievement of good learning outcomes.

On this point, a few authors, including Berner (2003) and Laurillard (2002), note that participation is more active if some sort of assessment is linked to it. Klisc, McGill & Hobbs (2009) uggested that incorporation of assessment of participation has a positive impact on learning outcomes. Indeed, whether courses are completely or partially online, Burkett, Leard and Spector (2004), Leh (2002) and Seo (2007) all indicated how grade points might be used as an incentive to enhance participation amongst learners.

However, assessing the participation in asynchronous discussions of the students is a major challenge and difficult for the instructors (Liu, 2007). The main concern is how to assess and what guideline to consider for assessment. This issue of assessment of student participation in the online discussion has been a heated topic among educators and researchers in online education (Bonk and Dennen, 2003).While there is some literature in this regard; there is a lack of empirical studies (Ho, 2002).

For assessment of discussion forum participation to work effectively, there needs to be a comprehensively defined framework that can assist the evaluators and students clearly. Having a comprehensive framework can also act as a guideline for participants and educators. Brannon and Essex (2001) stated the need for clear communication protocols and requirements for posting, and suggested that the continued development of an innovative evaluation framework is necessary to improve the quality of contributions to an online discussion. A rubric that explicitly describes levels of responses will stimulate learning by challenging students to reflect and think critically, rather than post basic statements of understanding and mere opinion. (Anderson & Krathwohl, 2001)

The role of instructors in online discussion forums

The term instructor, moderator and facilitator are often used inter-changeably in practice and in literature. Instructors have an important role to play so that collaborative and interactive participation by the students can be ensured (Meyer, 2002). Steel (2009) has investigated the level of adaption of learning management systems to ensure collaboration by instructors. Cheung and Hew (2010) explored how exhibited habits of mind of the facilitators influence student participation. Mazzolini and Maddison (2007) classified the instructor participation into four categories: 1. asking questions; 2. answering the questions posted by the students; 3. combination of answer and follow up questions; and 4. other administrative or housekeeping related posts. Mazzolini and Maddison (2007) further pointed out that students contradicted this classification and put more emphasis on periodic feedback which would answer their questions, suggest directions to achieve solutions for problems encountered, provide new angles of thinking, and discuss the solutions provided by the students. Garrison and Anderson (2003) pointed out that 'moderating' in online discussions involves such responsibilities that the instructor acts as being a tutor, a facilitator and a subject expert, and that this moderation is a significant factor in developing engagement and interaction. These include asking a question at the beginning to prompt a discussion, letting students answer questions and extending discussion by providing hints. The question remains whether this 'sage on the stage', to the 'guide on the side' or even 'the ghost in the wings' (Mazzolini & Maddison, 2003) role of the instructor can ensure quality discussion by the students or not. This paper does not seek to examine these roles but provide a framework which may be used flexibly across the roles. A structured framework would also assist in defining roles if multiple instructors or tutors are involved (Goold, Coldwell & Craig, 2010).

A conceptual framework for assessing quality in online discussion forums

A conceptual framework proposed by Nandi et. al. (2009) adapted from the works of Henri (1992), Newman, Webb and Cochrane (1996) and Garrison, Anderson and Archer (2001) defines several themes on which qualitative online interaction can be designed and assessed. In order to assess each criterion the authors have separated the above criteria into three broad categories:

Content: demonstrating the expertise of the learners in the discussion topic, by which the talent of the learners can be assessed.
Interaction quality: looking at the way learners interact with each other online in a constructive manner, which implies that the contribution should be collaborative and meaningful for the community of learners.
Objective measures: highlighting how consistently and frequently learners participate in discussion.

The content criteria cover clarification, justification, interpretation, application, prioritisation and breadth of knowledge, and a student's performance can be rated against these criteria. The interaction criteria cover critical discussion of contributions, new ideas emerging from the interactions, sharing and introducing outside knowledge, and using social cues to engage other participants in the forum. In order to fully support educators, this framework (Nandi et al, 2009) also provides a sub classification which clearly indicates what may be poor, satisfactory, good or excellent performance against each criterion. We have suggested that the relative importance and the relevance of the criteria that follow depend very much on the facilitators, the subject matter or discipline area, and the cohort and demography of the learners they are assessing.

Methodology

According to Yin (1989), a case study method is appropriate to examine "how" and "why" related questions. Hence we employed a university-level case study method to conduct this research.

We selected a postgraduate subject for this study because it provided a mostly online learning environment for the instructors and students, with only four face to face classes. The face to face classes were spread across the semester with one class in the first week, two in the middle of the semester and one at the last week of the semester. All other communications between the students and instructors were online through the learning management system provided by the university. Students were encouraged to participate in asynchronous discussion forums by the instructors and assessed on their participation, according to whether they understood the concepts themselves and contributed towards enhancing the understanding of the other students. These initiatives of engaging students in online asynchronous discussion forums were taken by the instructors to encourage the students to participate actively in productive interactions with others students and instructors by sharing views and ideas and knowledge.

Specific themes were uploaded by the instructors and students were expected to work through the readings on that theme and post their comments. Comments could be in the form of questions, opinions, analyses, etc. Students were expected and encouraged to work on each specific theme (with its related topics) for the duration of 3-4 weeks (for example, the first theme was studied for a period of four weeks).

Each week, the instructors provided a brief overview of the topic in the Content area. Then, to initiate discussion, the instructors posted questions in the discussion forum for the students to consider, which they were expected to respond to and discuss regularly by exploring the readings and concepts. An illustration of how the course and discussion forum activities were conducted is given in Table 1.

Students were expected to study the readings and answer those broad questions, post their views, agree or disagree with other's posts, whilst periodical formative feedback was provided by the instructors. New questions arose and were discussed through the week as the discussion progressed and students and the instructors engaged with the theme in more depth. Strategies for encouraging students to participate include prompting them with new and increasingly more complex questions, once previous ones had been discussed. These were complemented by the questions raised by the students themselves. In addition, students were encouraged by the instructors to answer each other's questions and to provide additional resources that might promote understanding. The instructor also prompted the few students who may not have contributed as much each week, to participate more. In addition, a 10% of total mark for the subject was allocated for discussion forum participation, which acted as an incentive for participation.

Table 1: Online discussion activities

Theme 1: Business and Information Systems Fundamentals (Weeks 1 - 4)
Good management requires a balanced approach to decision making that acknowledges that both the external environment (e.g. regulation, competition, customers) and the internal practices (e.g. what to produce, how to produce, how to market) are important to the success of the organisation. In the management of IT, similar rules apply. The first theme of the subject exposes you to three key concepts of social informatics, competitive forces and competitive advantage.
In Week 1, we will explore the Kling's review of the concept of social informatics, which highlights a fundamental approach to managing and understanding IT within organisations.
In Week 2, we will explore Porter's frameworks for understanding the external environment are greatly relied upon by IS professionals and academics.
In Week 3, we will explore the issues around competitive advantage and what it actually means for businesses. How can IS be used to create and sustain competitive advantage for businesses?
To kick the discussion off in Week 1, the instructors posted the following questions.
What do you think are some of the main principles behind social informatics?
Additionally, what did you take away from reading the PWC case?
From your own experiences, have you come across similar situations and could social informatics have assisted in understanding your own experience?

The students were assigned marks based on the consistency of their participation across the weeks (not just quantity), and also on the quality of their participation, by looking at the development of ideas over the weeks, ability to apply concepts learned to real world cases, helping other students understand complex concepts, and providing new insights into concepts broadly. There were 12 students enrolled in the course and all but one contributed consistently to the discussion forum.

In order to carry out the research, we conducted a focus group meeting with the students from the chosen postgraduate course and separately interviewed the two instructors of the course. The posts by the students and the instructors in the course discussion board were analysed. This data was processed using grounded theoretic approach (Strauss & Corbin, 1998) i.e. open, axial and selective coding (Neuman, 2006; Strauss & Corbin, 1990) so that information relevant to the research could be extracted

We systematically analysed the discussion board data to identify the themes-related participation. These themes provided a clear representation regarding the qualitative discussion between students and the instructors. Then we analysed the data from the focus group and the interviews with the students and instructors to find out on what aspects of discussion they put their most emphasis on and regarded as productive. By analysing and combining the data from the discussion board, focus group and the interviews we prepared the set of themes that the students and instructors exercised during their discussion and valued highly. As mentioned above this data analysis was performed using open, axial and selective coding method. A similar three stage data analysis technique was used by Vlachopoulos and Cowan (2010a, 2010b) while exploring the different styles and practice of e-moderation; and reports that this method is useful in gaining deep understanding of a phenomenon or a theme from raw data.

The purpose of open coding was to identify the themes or concepts within the discussion board, focus group and interview transcripts. Each separate concept in the data was labeled and similar ideas were grouped and labeled. Following open coding, the next step was axial coding, where the aim was to assemble coding categories into larger conceptual groupings. This process was repeated until no additional categories were identified and all the data had been analysed. The third and final coding step was selective coding. Again, the data were re-examined and the prior coding and grouping was revisited and verified or changed as required so that all the data are accounted for under a theme or sub-theme (Glaser & Strauss, 1967). This set of themes is presented in the "Findings" section.

We compared the themes arising from this data analysis with the set of criteria presented in the conceptual framework (Nandi et. al., 2009). The framework was extended according to the findings and details are discussed in the "Discussion" section.

The data analysis enabled the extraction of key and relevant information to the research and as a result, the research question was explored based on the results ascertained through these methods.

Data analysis and findings

The data analysis phase of the research uncovered a number of key themes that were mentioned on a number of occasions, by the different participants. Initially our main focus was on the participations of the students. However during data analysis the fact that students depend highly on the instructor's feedback in the discussion forum forced us to look at the participation of the instructors as well. Consequently two dimensions were looked at to explore themes. They are:

Quality of posts by the students
Type of moderations carried out by the instructors

1. Quality of posts by the students

In this dimension, we looked at the themes that are relevant to the quality of the student's postings in the discussion forum and a wide range of themes were discovered. These themes were clustered into three major categories focused on cognitive skills shown by the students, participation rates and type of language used, and is discussed below. The quotes provided here are reported verbatim from the discussion forum, focus groups and the interviews.

(a) Cognitive skills presented by the students in discussion forum participation
All the students presented a wide range of cognitive skills while participating in online interaction. Seven key themes under this category that came out are stated below along with quotations and discussion forum posts by the participants:

Ability to perform critical assessment: Almost all the students showed an excellent skill of assessing critically the readings and literatures suggested by the instructors. Analysis of posts showed that students did not just believe the readings and evaluated them critically. This criterion also came up in the focus group and the interviews with the instructors.
About wiki, I think it is a good resource. but i always remember one thing "start your research with wiki, but not end with wiki". [Student E, Forum]
Ability to draw personal expertise to the study: Many participants brought on personal experience to relate or solve a current problem in the readings. Personal experience in work or life was used to describe or debate with what is mentioned in the reading. Instructors also emphasised these criteria. The analysis of discussion forum posts showed that students were not reluctant to bring in personal experience but they also brought other real world examples into the discussion.
I've been working since before computers were commonplace in offices, (I remember 'typing pools'!) And it seems to me that the way people work, and what they do have changed so much that there's not much that comparable between the eras. [Student G, Forum]
The abilities to extend discussion: Carrying on discussion from another's post was a skill that was found in the participation of the forum. This ensured that discussions on specific topics covered a lot of areas around the topic of discussion. In doing so students raised new issues from other posts. This showed that students were prepared to take the discussion to a new level.
However, one question came up when you told that "it depends on an individual and on his "needs" as to how much he wants to get influenced by technology and that is where the "multiple effects" comes in from. [Student A, Forum]
The ability to respond to each other: It was found from the discussion forum analysis that students asked questions of fellow students about something they did not understand. This showed that students analysed the information, and if they did not understand, then prepared to query to clarify it directly. Students also responded to each other along with the questions asked by the instructors. In response, students suggested single or multiple solutions to problems raised by the instructors or other students. All participants agreed that this criterion of responding and suggesting single or multiple solutions is a valuable criterion to have in a discussion forum conversation.
What do you mean when you are talking about "decentralized production"? [Student C, Forum]
I like your idea about combining frameworks and models. When I read the article "Principles and modules for organizing the IT function" (2002), I found that every model has its specific strengths. [Student A, Forum]
[Students can get time to understand the concept, help each other and contribute enough. [Instructor, Interview]
The ability to support argument with references: It is important to justify opinions or information provided by the students by mentioning references so that it can be acceptable and accessible to all. Analysis of discussion forum participation and focus group showed that students exercised these criteria frequently and emphasised the importance of using these criteria in discussions. Students were using references as justification while agreeing or disagreeing with others, to validate their information and this criterion was emphasised in the focus group.
Here, I'd like to cite one concept from M [pseudonym] and F [pseudonym] to support my argument. [Student F, Forum]
Clarifying information in discussion: The ability to clarify one's position in a discussion about a specific topic is a skill that was shown by the students in the discussion forum. Instructors also emphasised this issue and stated that students should be able to clarify the information they post on the forum so that all can understand their points. Again analysis of discussion forum showed that students sometimes changed their previous views after going through readings or feedback from the instructors. In these cases, they should clearly mention their change of view in the discussion so that everyone can be updated with correct information.
When I was talking about centralization in every country, I meant that shops in every country will be interconnected (what I also mentioned in my other posts). [Student H, Forum]
Application of concept: Students demonstrated the ability to apply the concept learned in the subject to suggest a strategy for solving a problem. Instructors also explained this issue was important as applying the knowledge can show the students have understood the concept.
For instance, as you said, if it can establish a decentralized IS to all the stores and allow them to share the information (i.e., share the inventory information and do speeding coordination between stores in same regions), the company could even do response much quicker than it does ever before. [Student A, Forum]
Whether students understand concepts, applying their knowledge or not. [Instructor, Interview]

(b) Participation rates of the students
This category focuses upon quantitative measures of student participation. All participants were consistent in terms of their timing and frequency of posts in the discussion forum. Among all the posts, almost 85% were posted by the students, at a consistent rate. Both students and instructors mentioned that they wanted to see a steady participation so that the flow of discussion was always maintained.

It would be great to have a discussion which is consistent and not, like it doesn't stay standard for a time and you keep waiting for the post, it should all be flowing. [Student B, Focus Group]
All but one has participated consistently. [Instructor, Interview]

(c) Use of formal/informal language
This category looks on the type of language used during the online forum discussion. Apart from the cognitive skills presented by students during their consistent participation in the discussion forums, other issues such as the use of social cues (Henri, 1992) or informal language were also evident. Students did not participate only to obtain grades but also enjoyed their participation. Although some students in fact opposed the idea of using informal language in the discussion forums, they actually did use informal language on a few occasions. Instructors also had opposing ideas about the use of language. One thought that it was fine while the other preferred formal language, citing cultural differences amongst students.

Is this a matter of business strategy? I think I'm a little bit out of track on this topic... [Student E, Forum]
Should be formal, not everything informal, I would never say hi how you doing like that, I would never say it on the discussion forum, if I want to communicate with my friends I will invite them to Facebook. [Student A, Focus Group]
Academic English and use of formal language is preferred because of the mix of culture. As there is a cultural difference between students so the meaning of something may not be the same for everyone. [Instructor, Interview]
I am not worried about informal language, in discussion forum, it is to discuss. For discussion informal language can be used. [Instructor, Interview]

2. Type of moderation carried out by the instructors

This dimension focuses on the moderation activities carried out by the instructors in the discussion forum. Our initial research emphasis based on the development of the conceptual framework (Nandi et. al, 2009) was upon student participation. However, because of the dependency of students on their instructors, we also investigated the type of moderation performed by the instructors. According to Brannon and Essex (2001), lack of immediate feedback, daily dialogue and thoughtful discussion from instructors can create feelings of social disconnection among students, which is a major pitfall in asynchronous communication. Students revealed that they wanted the instructor to actively participate and provide them direction, while the instructors supported this idea by helping the students to keep on track with periodic feedback.

Four key themes that came out of the data analysis regarding the type of moderation activities by the instructors are stated below with appropriate quotations:

Initiating the start of discussion: The initiative to initiate the discussion was taken by the instructors and it provided the students with an initial direction to start their participation.
To kick it off, what do you think are some of the main principles behind social informatics? [Instructor, Forum]
Inspiring students: It is important to inspire the students while teaching and it was done by the instructors on a regular basis by suggesting potential benefits from the readings.
In doing the reading which is relatively long and detailed, you will find that this is how many organizations are having to assess IT Alignment at various different levels of their business and IT. [Instructor, Forum]
Providing regular feedback about information posted by the students: All the participants emphasised this concern about receiving regular feedback from the instructors. Students felt that feedback will verify the information that they post during discussion as different students post their views from different dimensions. The instructors valued feedback as important to keep the discussion on track, as there was a tendency to move away from the original topic. There were situations where students changed their views after feedback from the instructors.
As a guide, like answer our questions and tell us whether we are on the right track, how you going, how you can go to find books on the topic or this article. [Student A, Focus Group]
One drawback/lack is after a few round of discussion students tend to wander and move a bit away from the initial question. [Instructor, Interview]
Extension of the discussion: Occasionally there was a tendency for students to get stuck on a certain point of discussion and the instructors needed to take the initiative to extend the discussion and broaden the focus. This was regarded very highly by both students and instructors to keep the discussion going forward.
Good point Student E, now, you might want to look at alignment maturity which extends and expands on these ideas about 'fit between IT and business'. [Instructor, Forum]

Discussion

The key focus of this research was on the participation of the students in discussion forums. By analysing discussion forum participation, focus groups with students and interviews with instructors, we have uncovered several themes for online participation.

As mentioned earlier, we have compared the findings of the data collection with the set criteria from the conceptual framework (Nandi et al., 2009). Results of the comparison show that almost all the themes extracted from the data analysis are consistent with the criteria from the conceptual framework and can be used as an assessment framework. A few extra criteria for qualitative discussion also came out of the data analysis and are discussed below in detail.

1. Quality of posts by the students

On a general level, students are making the most out of their forum discussions to gain, share, deepen, apply and expand knowledge, as illustrated in Table 2. A number of criteria in the area of cognitive skills (Henri, 1992; Newman et al, 1996; Nandi et al, 2009), use of formal/informal language (Henri, 1992; Gerbic, 2006; Nandi et al, 2009), and frequency of participation (Henri, 1992; Nandi et al, 2009) were all evident in a way that students valued. A comparison is made between the themes shown by the students and the criteria from the conceptual framework. Table 2 shows good consistency between the findings from the data analysis and the criteria from the framework.

From Table 2 we realise that not all the criteria for quality derived from the literature review were exercised and valued by the students. The criteria of clarification, justification or judgment, application of strategies, breadth of understanding, relevance, participation rate and consistency of participation were very consistent with the initial criteria from the framework. Data analysis showed that students practised these criteria consistently throughout and instructors also emphasised these issues.

Table 2: Comparison of findings from data analysis with the conceptual framework (Nandi et al, 2009)

Criteria from the framework Is the criterion consistent with the result from data analysis?

Content Clarification Generally consistent with the criterion as students clarified their positions while discussing online.

Justification or judgment Consistent and the students tried to justify their opinions.

Inferencing or interpretation Semi-consistent with the criterion from the framework.

Application of knowledge (relevance) Very much consistent as almost all the posts done by the students were about the topic of discussion.

Prioritisation Initially most importance was given to answer the questions from the instructors. Often students did not try to find out important issues in the readings and discuss for themselves and waited the inputs and feedbacks from the instructors.

Breadth of knowledge Consistent as students showed their understanding while responding to the questions posted by the instructors.

Interaction
quality Critical discussion of contributions Consistent with criterion as evidence was imminent of this quality in the posts.

New ideas/solutions from interactions Very much consistent with the initial criterion.

Sharing outside knowledge or expertise Generally consistent with the criterion as personal outside experience used to support their opinion.

Use of social cues or emotions to engage with participants Not totally consistent with criterion as participants had disagreement with the use of informal language in the discussion forum.

Objective
measures Participation rate Consistent with criterion as found from the data.

Consistency of participation Reasonable consistency in participation.

Evidence of some new themes for quality arose along with a few extensions of the themes that we found from the literature review. These issues are discussed below, as they could be considered controversial and not agreed by everyone.

Inferencing or interpretation: This criterion was partially consistent with the initial proposition, as there was evidence of students' proposing new issues from the posts of other students, admitting what another says is true. This criterion was not exercised frequently by students, to their detriment.
Critical assessment: This criterion (Henri, 1992; Newman et al, 1996) refers to critically assessing the readings that the instructors recommended to them. Students went one step ahead and critically assessed other's participation which shows their interest in learning and sharing through the discussion forum. Consequently, it is important for ensuring quality that participants not only critically assess the readings, but also the postings of other students. While doing critical assessment of other student's post, students have agreed/disagreed by providing references which validate their points of agreement/disagreement and allow students to identify their areas of lack of knowledge.
Prioritisation: According to Newman et al (1996), importance refers to giving more emphasis to the important issues in the topic under discussion. But the data revealed that students put most emphasis on the questions asked by the instructors, and used all their efforts to answer those questions. In some cases, students did not go beyond answering the questions and participating in conversation with each other. This issue made us look into instructor participation, because in situations like this, the responsibility falls on the instructors to ensure that all the questions they ask to initiate the discussion, do actually cover all important issues. During discussions, some students moved away from the original topic, i.e. the important part of the topic, and instructors had to mediate to ensure that students stayed focused on the central issues under discussion.
New ideas/solutions: This criterion refers to proposing new ideas or solutions into the discussion, which shows the analytical capacities of students (Newman et al, 1996). Students suggested solutions to the problems, and also tried to come up with more than one solution. This shows that they thought about the topic and questions raised by the instructors from different dimensions, which is important for acquiring academic improvement (Weaver, 2005). Suggesting multiple solutions also ensures that the discussion can move on as other students have information from different dimensions to look at and assess.
Sharing outside knowledge or expertise: Students tried to bring outside knowledge or personal expertise to provide credibility to their opinions. Some used their personal experience, and some used real world examples, which added reliability to their solutions.
Use of social cues or emotions: The uses of social cues or emotions represent the fact that students are enjoying their participation in the discussion forum (Henry, 1992) which is very important to ensure active interaction (Weaver, 2005). But this proposition was not totally consistent with the initial proposition that came out the literature review. Both students and instructors were divided in this issue of using social cues or informal language during discussion. Some suggested that discussion forum is a place to share knowledge and as it is graded, it should be treated as official. Others suggested that formal academic English should be used as students come from different countries, backgrounds and cultures and so the meaning of something may not be same for all. On the contrary those in favor of using informal language mentioned that there should be no barrier in terms of using of informal/formal language.

Apart from the above criteria for quality of posts, a few additional themes came out of the data concerning to the quality of participation. They are:

Interaction types: There were both direct and indirect conversations in the discussion forum, which is considered important as it showed that participants were communicating with each other (Henry, 1992). This involved responding to each other as in face to face conversations and this issue was regarded as a valuable criterion for quality by the instructors. Analysis of data revealed that students were consistently asking direct questions to each other about their posted opinions so that the information becomes clearer to them. Again there was indirect participation while the students tried to answer the questions posted by the instructors only.
Continuing discussion based on other's posts: The students and instructors admitted that sometimes the discussion was based mainly on the questions asked by the instructors and could remain stuck there. So to add quality to discussion, it was necessary to take the step of continuing discussion based on what other students had posted.
Changing views after going through the reading: This criterion was seen in the participation of the students as they were brave enough to change their previous opinions. This represents the fact that students have analysed the study topics and found that their previous information is not totally credible.

From the above discussion we can find a pattern that students were mostly focused on the point of answering the questions and also feedback from the instructors. This led us to assess the instructor participation in discussion forum.

2. Emerging new criteria for instructor participation

Initially our research emphasis was on the issue of quality of student participation, i.e. deriving a set of criteria based on the student participation to be evaluated. Our analysis has revealed that students were highly dependent on feedback from the instructors and in some cases they did not consider information provided by others as correct until verified by the instructors. It also came up from the investigation that students were losing their way at times during discussion and moving away from the topic.

Consequently it can be said that quality of student participation cannot be evaluated fully unless the moderation of instructors is evaluated. The type of moderation carried out by the instructors has an impact on the student's participation and so on the quality of the discussion. The instructor's role is significant in providing encouragement, feedback and direct instructions in specific cases (Maor, 2008). Hence there should be a set of criteria for evaluating instructor participation. This can also provide valuable guidelines to the instructor on how the discussion forum can be moderated so that students can maintain the quality of discussion and in doing so can gain and share knowledge.

A number of criteria came up from the data which revealed how moderation by instructors can be evaluated.

Initiating the start of discussion: From the literature review we found that participation of the students in the online discussion is affected by the amount of interest they have in the subject (Weaver, 2005; Gerbic, 2006). This is supported by the data as instructors tried to post interesting engaging questions to kick off the discussion so that students are attracted to discuss. This is also considered as providing directions to the students to initiate the discussion so that the students know where to start from.
Regular feedback and inspiring students: Sheard, Ramakrishna and Miller (2003) reported that student participation is affected by their maturity and motivation. This theory was supported by the data from this research as instructors were consistently trying to inspire the students by suggesting the potential benefits they could get from their study. Again regular feedback by the instructors kept the students alert and created a feeling in the minds of the students that instructors were reading their posts. This also motivated the students to engage in qualitative participation for acquiring good grades. During feedback, the instructors verified the information provided by the students which ensured the students were kept on track.
Extension of discussion: Data from the research showed that the discussion was 'getting stuck' sometimes at a certain point, because of the student's intention to only answer the question asked by the instructors. This is the time when the instructors can moderate and try to extend the discussion and broaden the focus, e.g. introduce new perspectives in a related topic. Data analysis revealed that this periodic intervention enabled the discussion to move forward and gave students perspectives from different dimensions.

From the above discussion we can summarise that our framework can provide a clear guideline on how to assess quality of posts in discussion forums.

Conclusion and future research

We have compared the data analysis results with the framework and it does have the required categories to assess the quality of interaction between students and students' facilitators.

We include the following quotes as examples of what we understand to be good quality productive contributions to the online discussion forum under the two themes of application of knowledge and new ideas from interactions.

Application of knowledge

Whether all the companies should be aiming for level 5 depends on several factors, in my opinion. Firstly, how company ambitious is. Secondly, how high the information content in the industry where the company operates is. Different companies have different levels of information content. Thirdly, how big the organization is. For example, small stores in countryside. These organizations even do not have all the attributes in the six SAM criteria. Whether to pay more attention to some criteria or not, I believe that governance and partnership require special attention because:
1) In governance criteria business and IT strategies are identified. On these strategies mostly depends competitive abilities of the company. 2) In partnership criteria business perception of IT value and role of IT in business strategic planning are identified. If IT and business are not interconnected well, IT alignment will be unlikely. Your questions were very interesting. What do you think about them? [Student C, Forum]

New ideas from interactions

I like your idea about combining frameworks and models. When I read the article "Principles and modules for organized the IT function" (2002), I found that every model has its specific strengths. The first model enables IT to be the active partner in business innovation by having strong position of IT in top management team, whereas the second model allows business to provide services and infrastructure for business innovation, and to build enterprise-wide platforms and capabilities. Why can't we combine all the specific features and to create powerful IT function model? Maybe, it sounds unrealistic? Why not? [Student F, Forum]

Our framework has proved to be very useful for analysing discussion forums. Due to the exploratory nature of our research, the major focus was to identify key themes and sub themes which apply to online forums. A number of issues relating to effective online participation and engagement were raised by the participants, and discovered through the analysis of the information provided by them.

In order to have a better understanding of what it means by "quality" of posts, two major areas were considered, including type of participation by the students and moderation activities by the instructor. These were conducted under the umbrella of the concept "quality of posts", with the results of the research giving an indication about a set of criteria for quality interaction in discussion forums.

We have applied the framework in a blended learning environment and found it to be useful for evaluating quality in online interaction. Results showed that students were actively participating in the discussion forum activities by providing new information about the topics, justifying their position in a discussion by providing references from the literature, and examples from real world situations. They also show that the students are thinking critically and responding to each other's posts by agreeing or disagreeing with them. This productivity in participation can prove to be very helpful for the students in gaining valuable knowledge from each other's experiences.

Results also showed that instructors were setting up the discussions with initial engaging questions and providing feedback on a regular basis. This helped students to broaden their focus and inspiring them to continue their discussions. As mentioned earlier, the implementation of each and every criterion is influenced by the perceptions of the role of the instructors.

Identifying and evaluating the quality of interaction has established our framework to be unique and robust for the purpose of measuring the different dimensions of qualitative interaction between students and instructors.

There are several key areas that would benefit from future detailed research:

The usability of technology was an interesting area raised by the participants. A look into the future of the technology, and how technology affects participation within an online environment could prove to be an interesting topic for research.
The current research only looked at the student participation in a tertiary learning environment, specifically in a postgraduate course with limited number of students. Future research would benefit by adapting the framework in undergraduate courses with larger student cohort and multiple tutors. Professionals are also engaged in online communication now and research in this field could prove to be valuable in future. A comparison can also be carried out showing what a good quality discussion and a poor quality discussion is with illustrated examples to investigate and highlight the fundamental differences between them.

References

Al-Mahmood, R. & McLoughlin, C. (2004). Re-learning through e-learning: Changing conceptions of teaching through online experience. In Beyond the comfort zone: Proceedings ASCILITE Perth 2004 (pp. 37-47). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/al-mahmood.html

Anderson, L. & Krathwohl, D. (Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman Publishers.

Berner, R. T. (2003). The benefits of discussion board discussion in a literature of journalism course. The Technology Source, Sep/Oct. http://technologysource.org/article/benefits_ of_bulletin_board_discussion_in_a_literature_of_journalism_course/

Bonk, C. & Dennen, V. (2003). The 3 T's of online assessment: Technology, tools, and (saving) time. Workshop given at the 19th Annual Conference on Distance Teaching & Learning. University of Wisconsin, Madison. [verified 29 Apr 2012] http://www.trainingshare.com/download/wisc2/ts_part1.ppt

Bradshaw, J. & Hinton, L. (2004) Benefits of an online discussion list in a traditional distance education course. Turkish Online Journal of Distance Education, 5(3). http://tojde.anadolu.edu.tr/tojde15/articles/hinton.htm

Brannon, R. & Essex, C. (2001). Synchronous and asynchronous communication tools in distance education. TechTrends, 45(1), 36-42. http://dx.doi.org/10.1007/BF02763377

Burkett, R., Leard, C. & Spector, B. (2004). Using an electronic bulletin board in science teacher education: Issues and trade-offs. The Journal of Interactive Online Learning, 3(1), 1-9. http://www.ncolr.org/jiol/issues/pdf/3.1.1.pdf

Garrison, D. R., Anderson, T. & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. The American Journal of Distance Education, 15(1), 7-23. http://dx.doi.org/10.1080/08923640109527071

Garrison, D. R. & Anderson, T. (2003). E-learning in the 21st Century: A framework for research and practice. New York: Routledge Falmer.

Gerbic, P. (2006). To post or not to post: Undergraduate student perceptions about participating in online discussions. In Who's learning? Whose technology? Proceedings ascilite Sydney 2006. http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p124.pdf

Cheung, W. S. & Hew, K. F. (2010). Examining facilitators' habits of mind in an asynchronous online discussion environment: A two cases study. Australasian Journal of Educational Technology, 26(1), 123-132. http://www.ascilite.org.au/ajet/ajet26/cheung.html

Glaser, B. G. & Strauss, A. L. (1967). The discovery of grounded theory. Aldine, New York.

Goold, A., Coldwell, J. & Craig, A. (2010). An examination of the role of the e-tutor. Australasian Journal of Educational Technology, 26(5), 704-716. http://www.ascilite.org.au/ajet/ajet26/goold.html

Guzdial, M. & Carroll, K. (2002). Explaining the lack of dialogue in computer-supported collaborative learning. Paper presented at the Computer Supported Collaborative Learning Conference, CSCL 2002. http://www.umsl.edu/~wilmarthp/mrpc-web-resources/Explaining-the-Lack-of-Dialogue-in-Computer....pdf

Hawkes, M. & Dennis, T. (2003). Supporting and assessing online interaction. Educational Technology, 43(3), 52-56.

Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 115-136). New York: Springer.

Ho, S. (2002). Evaluating students' participation in on-line discussions. In Proceedings Australian World Wide Web Conference (AUSWEB), Sunshine Coast, Queensland, Australia. http://ausweb.scu.edu.au/aw02/papers/refereed/ho/paper.html

Jackson, K. (2010). What value assessment rubrics in shaping students' engagement in asynchronous online discussions? In Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010 (pp. 454-458). http://www.ascilite.org.au/conferences/sydney10/procs/Jackson-concise.pdf

Klisc, C., McGill, T. & Hobbs, V. (2009). The effect of assessment on the outcomes of asynchronous online discussion as perceived by instructors. Australasian Journal of Educational Technology, 25(5), 666-682. http://www.ascilite.org.au/ajet/ajet25/klisc.html

Laurillard, D. (2002). Rethinking university teaching: A framework for the effective use of learning technologies (2nd Ed.). London and New York: Routledge.

Leh, A. (2002). Action research on hybrid courses and their online communities. Educational Media International, 39(1), 31-38. http://dx.doi.org/10.1080/09523980210131204

Liu, S. (2007). Assessing online asynchronous discussion in online courses: An empirical study. TCC 2007 Proceedings. http://tcc.kcc.hawaii.edu/previous/TCC%202007/liu.pdf

Maor, D. (2008). Changing relationship: Who is the learner and who is the teacher in the online educational landscape? Australasian Journal of Educational Technology, 24(5), 627-638. http://www.ascilite.org.au/ajet/ajet24/maor.html

Maor, D. & Volet, S. (2007). Interactivity in professional learning: A review of research based studies. Australasian Journal of Educational Technology, 23(2), 227-247. http://www.ascilite.org.au/ajet/ajet23/maor.html

Mazzolini, M. & Maddison, S. (2007). When to jump in: The role of the instructor in online discussion forums. Computers & Education, 49(2), 193-213. http://dx.doi.org/10.1016/j.compedu.2005.06.01

Mazzolini, M. & Maddison, S. (2003). Sage, guide or ghost? The effect of instructor intervention on student participation in online discussion forums. Computers & Education, 40(3), 237-253. http://dx.doi.org/10.1016/S0360-1315(02)00129-X

Meyer, K. (2002). Quality in distance education: Focus on online learning. ASHE-ERIC Higher Education Report, 29(4), i-vii. San Francisco: Jossey-Bass.

Nandi, D., Chang, S. & Balbo, S. (2009). A conceptual framework for assessing interaction quality in online discussion forums. In Same places, different spaces: Proceedings ascilite Auckland 2009. http://www.ascilite.org.au/conferences/auckland09/procs/Nandi.pdf

Neuman, W. L. (2006). Social research methods: Qualitative and quantitative approaches, 6th Ed.

Newman, D. R., Webb, B. & Cochrane, C. (1996). A content analysis method to measure critical thinking in face-to-face and computer supported group learning. [verified 29 Apr 2012]. http://www.qub.ac.uk/mgt/papers/methods/contpap.html

Piguet, A. & Peraya, D. (2000). Creating web-integrated learning environments: An analysis of WebCT authoring tools in respect to usability. Australian Journal of Educational Technology, 16(3), 302-314. http://www.ascilite.org.au/ajet/ajet16/piguet.html

Rovai, P. A. (2000). Online and traditional assessments: What is the difference? The Internet and Higher Education, 3(3), 141-151. http://dx.doi.org/10.1016/S1096-7516(01)00028-8

Rovai, P. A. (2002). Building sense of community at a distance. The International Review of Research in Open and Distance Learning, 3(1). http://www.irrodl.org/index.php/irrodl/article/view/79/152

Salmon, G. (2003). E-tivities: The key to active online learning. London: Kogan Page

Seo, K. (2007). Utilizing peer moderating in online discussions: Addressing the controversy between teacher moderation and nonmoderation. The American Journal of Distance Education, 21(1), 21-36. http://dx.doi.org/10.1080/08923640701298688

Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & Education, 34(3-4), 177-193. http://dx.doi.org/10.1016/S0360-1315(99)00044-5

Sheard, J., Ramakrishnan, S. & Miller J. (2003). Modeling learner and educator interactions in an electronic learning community. Australian Journal of Educational Technology, 19(2), 211-226. http://www.ascilite.org.au/ajet/ajet19/sheard.html

Steel, C. (2009). Reconciling university teacher beliefs to create learning designs for LMS environments. Australasian Journal of Educational Technology, 25(3), 399-420. http://www.ascilite.org.au/ajet/ajet25/steel.html

Strauss, A. L. & Corbin, J. M. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory, 2nd ed. SAGE Publications, Thousand Oaks, CA, USA.

Strauss, A. L. & Corbin, J. M. (1990). Basics of qualitative research: Grounded theory procedures and techniques. SAGE Publications, Newbury Park, CA, USA.

Weaver, C. M (2005). What encourages student participation in online discussions? Unpublished PhD thesis, University of Southern Queensland. http://eprints.usq.edu.au/1523/

Vlachopoulos, P. & Cowan, J. (2010a). Reconceptualising moderation in asynchronous online discussions using grounded theory. Distance Education, 31(1), 23-36. http://dx.doi.org/10.1080/01587911003724611

Vlachopoulos, P. & Cowan, J. (2010b). Choices of approaches in e-moderation: Conclusions from a grounded theory study. Active Learning in Higher Education, 11(3) 213-224. http://dx.doi.org/10.1177/1469787410379684

Yin, R. K. (1989). Case study research: Design and methods. Thousand Oaks, CA: Sage.

Appendix: Conceptual framework for assessing quality in online discussion forums

Criteria		Poor	Satisfactory	Good	Excellent
Content	Clarification	Regurgitation of information	A clear explanation of available information	Explaining available information using relevant examples	Articulating available information to expand on ideas presented, including the use of examples
	Justification	No justification of points	Justification based on personal opinion	Justification using existing cases, concepts or theories	Justification using existing cases, concepts or theories and providing clear discussion of implications
	Interpretation	Misrepresentation of information	Basic paraphrasing of available information	Clear interpretation of available information	Critical discussion of available information
	Application of knowledge (relevance)	No application or discussion of relevance to questions asked	Application of knowledge to questions asked	Application of knowledge including discussion using relevant examples	Knowledge is critically applied and may include discussion of limitations
	Prioritisation	No prioritisation of information or knowledge	Some basic comparison of information	Ability to prioritise information and knowledge	Ability to prioritise information and knowledge based on criteria that learner has established
	Breadth of knowledge	Narrow and limited knowledge	Some indication of a wider view of the topics discussed	Presenting a wider view of the topics discussed by showing a good breadth of knowledge	Ability to point out other perspectives, including drawing from other fields of studies
Interaction quality	Critical discussion of contributions	No engagement with other learners' contributions	Some basic discussion about other learners' contributions	Consistent engagement with other learners' cont-ributions and acknowledge-ment of their comments	Contributing to a community of learners, with consistent engagement and advancement of each others ideas
	New ideas from interactions	No evidence of new ideas and thoughts from interaction	Some new ideas developed as a result of interaction	Some solutions and new ideas as a result of interactions	Collaborative approach to solution seeking and new ideas developed
	Sharing outside knowledge	No sharing of outside knowledge	Sharing generic information that is easily available from outside sources	Sharing real world examples that may not be immediately obvious to other learners	Sharing real life knowledge, personal experience and examples of similar problems/solutions
	Using social cues to engage other participants	No engagement with others in the discussion forum	Answering some basic question posed by facilitator or other learners	Engaging with the work and discussion of other learners	Engaging and encouraging participation with fellow discussants in the forum
Objective measures (this category is subject to facilitators' expectations)	Participation rates	None or less than 2 posts per week	Between 2 to 5 posts per week	Between 5 to 10 good quality posts per week	More then 10 good quality posts per week
	Consistency of participation	Rarely posts with occasional activity	Occasional activity	Consistent activity	Consistent and productive activity

Authors: Mr Dip Nandi, School of Computer Science and Information Technology, RMIT University, Melbourne, Victoria 3001, Australia.
Email: dip.nandi@rmit.edu.au

Dr Margaret Hamilton, School of Computer Science and Information Technology, RMIT University, Melbourne, Victoria 3001, Australia.
Email: margaret.hamilton@rmit.edu.au

Dr Shanton Chang, Department of Information Systems, The University of Melbourne, Victoria 3010, Australia.
Email: shanton.chang@unimelb.edu.au

Dr Sandrine Balbo, Nuance Australia. Email: sandrine@acm.org

Please cite as: Nandi, D., Hamilton, M., Chang, S. & Balbo, S. (2012). Evaluating quality in online asynchronous interactions between students and discussion facilitators. Australasian Journal of Educational Technology, 28(4), 684-702. http://www.ascilite.org.au/ajet/ajet28/nandi.html

[ PDF version of this article ] [ AJET 28 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet28/nandi.html
Created 17 May 2012. Last revised 17 May 2012.
Page access count

Criteria from the framework		Is the criterion consistent with the result from data analysis?
Content	Clarification	Generally consistent with the criterion as students clarified their positions while discussing online.
	Justification or judgment	Consistent and the students tried to justify their opinions.
	Inferencing or interpretation	Semi-consistent with the criterion from the framework.
	Application of knowledge (relevance)	Very much consistent as almost all the posts done by the students were about the topic of discussion.
	Prioritisation	Initially most importance was given to answer the questions from the instructors. Often students did not try to find out important issues in the readings and discuss for themselves and waited the inputs and feedbacks from the instructors.
	Breadth of knowledge	Consistent as students showed their understanding while responding to the questions posted by the instructors.
Interaction quality	Critical discussion of contributions	Consistent with criterion as evidence was imminent of this quality in the posts.
	New ideas/solutions from interactions	Very much consistent with the initial criterion.
	Sharing outside knowledge or expertise	Generally consistent with the criterion as personal outside experience used to support their opinion.
	Use of social cues or emotions to engage with participants	Not totally consistent with criterion as participants had disagreement with the use of informal language in the discussion forum.
Objective measures	Participation rate	Consistent with criterion as found from the data.
Objective measures	Consistency of participation	Reasonable consistency in participation.