AJET 21(3) Hammond and Wiriyapinit (2005) - learning through online discussion

Australasian Journal of Educational Technology
2005, 21(3), 283-302.

Learning through online discussion: A case of triangulation in research

Michael Hammond
University of Warwick, UK
Mongkolchai Wiriyapinit
Chulalongkorn University, Thailand

This paper reports on issues in carrying out research into online discussion. The context is a study of a distance learning module within an MBA program. The module required students to tackle problems based on real life scenarios within small online groups. Students were studying part time and shared similar professional backgrounds. The research looked at students' overall evaluation of the module, ways in which group work was conducted, and the contribution of the tutor. The approach taken was an interpretive case study using questionnaire survey, text analysis and interviews. The main findings from the study are reported, but the focus is on the strengths of, and difficulties in, using the research methods. Triangulation of methods provides the researcher with a greater degree of confidence in reporting findings, although subjective interpretation is still needed.

Background

There is an extensive literature on asynchronous online discussion (eg. Wallace, 2003) and widespread agreement that online discussion within distance learning programs enables interaction which would otherwise be difficult to achieve. A commitment to student-student and student-tutor interaction is often associated with a social constructivist approach to teaching and learning (eg. Yang & Tang, 2003; Angeli, Valanides & Bonk, 2003; MacDonald & Twining, 2002) which may be contrasted with non-interactive, traditional or transmission models (Snell, Hodgson & Mann, 1987). Some, but by no means all, research concerning online discussion has reported on relatively high rates of student participation with evidence of cooperative learning (eg. Aviv, Erlich, Ravid & Geva, 2003; Collings & Pearce, 2002; De Abreu Moreira & Quintino Da Silva, 2003; Hawkey, 2003; Hiltz, Coppola, Rotter, Turoff & Benbunan-Fich, 2000; Spiceland & Hawkins, 2003), while further literature argues for evidence of higher order thinking and knowledge building (eg. Anderson, Rourke, Garrison & Archer, 2001; McConnell, 2000; Åhlberg, Kaasinen, Kaivola & Houtsonen, 2001; Aviv, 2000; Curtis & Lawson, 2001; Thomas, 2002). Permanent storage of messages seems to provide support for reflection (eg. McConnell, 2000; Salmon, 2002) and expansion of available time for learning (eg. Macdonald & Twining 2002; Meyer, 2003). Some writers see asynchronous online discussion as adding value to the learning experience and to professional learning in general (eg. van Weert & Pilot 2003; de Corte, 2003).

Research into online discussion has drawn, not surprisingly, on similar sources of evidence: questionnaire survey (delivered electronically in many cases), interviews, and message analysis. Some writers have adopted more recognisably ethnographic perspectives (Taylor, 2001) while others have adopted experimental, or at least comparative, methods (eg. Hubscher-Younger & Narayanan, 2003; Koory, 2003; Parker & Gemino, 2001; Hiltz et al, 2000; Weller, 2000). Some have tried to engage students in formative evaluation of their online experiences (eg. Collings & Pearce 2002; Hawkey, 2003). Much of the research has relied on a single method, this may be a student survey (eg. Yang & Tang, 2003; Biesenbach-Lucas, 2003), but more frequently message analysis (eg. Anderson et al, 2001; Aviv et al, 2003; Cook & Ralston 2003; Kumari, 2001; Martinez et al, 2003; Swan, 2002; Watson & Prestridge 2003). Explicit commitment to triangulation of findings has been rare, though see McLoughlin (2002) and studies in which two methods of data collection have been used (eg. survey and interview in Galanouli & Collins, 2000; message analysis and interview in Light, Nesbitt, Light & White 2000; message analysis and survey in Seabrooks, Kenney & LaMontagne, 2000; Thomas, 2002; Tolmie & Boyle, 2000). The two most striking features of past research are, firstly, the use of message analysis, often cited as a unique method for those researching asynchronous online discussion, and, secondly, over reliance on a single, or dominant, method of data collection. This paper reports on asynchronous online discussion within a distance learning module and pays particular attention to the methods used. The context of the module is given along with the key findings, a description of methods, and a discussion of the value of triangulation. The paper aims to stimulate debate on how research into asynchronous online discussion is approached.

The context

This was a study of a distance learning module, for part time students, on the topic of e-business, within an MBA program at a local university. The module provided an opportunity for students to experience online learning within what had been a 'traditional' distance learning program. The e-business module was based on collaborative/cooperative groupwork and access to online material (eg. course notes, a library, and hyperlinks to further sources). Assessment covered both individual assignments and group products. This paper reports on the cohort of 43 students who took the module in 2002. There were 36 men and 7 women within the cohort, based in 17 countries, who shared broadly similar professional backgrounds.

The module was designed by the program management team and part time tutors were employed to support students. Induction and on going support for these tutors was provided. The module lasted for 18 weeks and covered three phases: Induction, the Study Units and Individual Assignment. The Induction lasted two weeks and covered introductions, addressing technical problems and an orientation to the module. The Study Units were discussed over 12 weeks. Students were placed in closed online groups of five or six members and joined by a tutor. Each group was asked to discuss readings and to tackle real life scenarios or 'cases' within e-business, resulting in the writing of agreed report. The groups were designed by the course team to contain a mix of nationalities and contained students for whom English was a first language and for whom English was a second language. At the close of these study units students had four weeks in which to complete an individual assignment, a 2,500 word report on a selected e-business topic. Students were expected to spend about 6 hours per week on study.

The research

The research set out to explore students' experience of the module and the nature and scope of group work. Key questions were: What was students' overall evaluation of the e-business module? How did students conduct their group work? How did students use the WebBoard (the conferencing software used in the module)? How was tutor support provided? How did students characterise the learning experience? The methodological approach was that of interpretive case study and the methods used were online questionnaire survey, message analysis and face to face interview.

Online questionnaire surveys

Students were asked about their general expectations at the start of the module, their experiences of collaborative learning mid-module, and to evaluate the course at the end of the module. Response rates were 76 per cent, 51 per cent and 42 per cent respectively. There were also questionnaires specifically for tutors.

The questions were mixed in format. Some required yes/no responses, for example 'have you used conferencing software before taking this module?' Some required selections from multiple choice responses. For example students were given a list of seven positive attributes of online discussion and asked to select those with which they identified (see Table 1).

Table 1: Positive attributes of online discussion: Percentage of students selecting each item

I like online discussion because: %

I can get feedback on my ideas 73

I can go back to read messages 73

I can mail when I like 64

I can mail where I like 50

I can get to know people from different cultures 36

I am not interrupted when I contribute 32

I can get to know other people in my group well 32

Further evaluation questions employed a Likert scale, for example did respondents 'strongly disagree', 'disagree', 'neither agree nor disagree', 'agree', or 'strongly agree' with statements such as 'I feel more comfortable about participating in group online discussion than in the face to face group' or 'I feel anxious when I post messages'.

In addition there were open-ended questions but these were not analysed in any depth as the response rate was low. However they offered insight into the individual difficulties that some students experienced.

Key findings from the surveys are reported briefly below:

All students were taking the module to learn more about e-business, some students also wanted to explore online learning, to get experience of using new technologies, to get to know other distance learning students, and to experience group work. A significant number of students (39%) chose the online module to avoid having to do a formal written examination. Students expected to find the learning environment more enjoyable and more collaborative.
Students experienced a high level of satisfaction with the module and the role of group discussion within it. They enjoyed the flexibility of participating in group asynchronous discussion and being able to go back to read messages, as contributions were permanently stored in WebBoard.
Students felt that their communication skills, consensus building skills, and cross cultural understanding had been developed. Most students did not feel anxious when posting messages and were comfortable with leaving their contribution permanently recorded. Students felt that the group activities were interesting and integrated into the module, and saw group work as an important part of the module. Nearly all had some previous experience of online working.
The strongest constraint on participation, although only stated by a minority, was finding the time to take part in online discussion. Tutoring was seen as appropriate but some students would have preferred more active tutor involvement.

Message analysis

Conferences were analysed both quantitatively and qualitatively. To get a sense of overall activity, messages were broken down by week, by topic, by group, by group member, by tutor, and by length. A key finding here was there was active participation in the conferences. For example, the number of messages sent within the working groups (in which there were five or six students plus one tutor) ranged from 250-618 messages over the 18 weeks. A mean of 64 messages were sent per participant, for women the mean was 44 messages, noticeably less than for men (70 messages). The mean number of messages sent by students with English as a second language (59 messages) was less than for those with English as a first language (72 messages). The most active groups tended to have the most active tutors, but tutors posted the least number of messages in each group.

Messages in three groups with respectively low, middle and high levels of activity were studied in greater depth. The mean lengths of messages were 64, 67 and 115 words. Many of these messages carried attachments of much longer, and more heavily edited, pieces of course work. The mean lengths of attachments for the three groups were, respectively, 967, 1260 and 918 words. There were considerable variations in participation rates between individuals, between groups and between weeks, but all students contributed to the discussions some of the time.

The main analysis of messages concerned input into the 'Unit 1 Plenary' discussion. This took place over a week and enabled each group to draw together, in an agreed report, their exploration of a company trying to establish a web presence. The plenary took place after students had become familiar with the technology, and with working together, and could be taken as exemplifying the group work process. As with other phases of the module, responsibilities were coordinated by a group leader, a role rotated around the group.

Messages (n=107) were again analysed within the three groups representing comparatively low, middle and high rates of participation. Each message, or more often part of a message, was tagged to show its function. After a long process of moderation, 30 functions were identified and a total of 306 tags made; the ten most frequently occurring functions accounted for 80 percent of the total number of function tags made. These ten functions were further analysed in terms of best fit within three overarching categories: 'independent', 'interactive', and 'strongly interactive'. Independent functions, such as 'stating', did not invite interaction; interactive functions, such as 'asking', invited feedback; strongly interactive functions, such as 'disagreeing', offered a different perspective on the group product and prompted deeper negotiation and group decision making. Table 2 suggests that there was a mix of independent and interactive styles of communication within the plenary, but that discussion was not strongly interactive.

Table 2: Overarching pattern of communication within 'Unit 1 plenary'

Overarching patterns
of communication Examples of functions Total
%

Independent Introducing, stating, reporting, telling, drafting, final-drafting. 37

Interactive Commenting, agreeing, expressing, requesting, clarifying, confirming, reasoning. 59

Strongly interactive Questioning, arguing, disagreeing. 4

Face to face interviews

Face to face interviews with a sample of course participants (n = 12) were carried out at a residential event. These took place during the MBA program but after students had completed the e-business module. This sample was broadly representative in terms of gender, student and tutor roles, those for whom English was a second language, and levels of participation. Interviewees were asked to provide feedback on their experience of the module. Themes covered learning, community, communication, participation, tutoring and curriculum design. Interviews lasted between 30 and 120 minutes, recordings were transcribed and coded. The coding was done manually, though in retrospect, a program such as Atlas or NVivo could have facilitated this process. Categories were constructed and properties associated with each category were identified. The example given in Table 3 shows the properties or associations with the category 'perception of other members of the group'.

Responses were examined and re-examined to draw out variations. For example, nearly all participants used words such as 'friendly', 'supportive' and 'non-confrontational' to describe other students in the group, but the consequences of this were seen differently. One interviewee felt that:

Within the group there was a recognition of a need to affirm each other, to value their contribution. I think that actually was done. Most people were very good in affirming each other's contributions. That generally was what was taking place and it worked very well.

However another interviewee saw this lack of disagreement more critically:

In the group work everything stays very polite. That might have been cultural or the fact that we are students. We were quite polite. Confrontational discussions were avoided... you could see the difference between contributions. People disagreed but the disagreement was not made explicit in messages and never came out.

Table 3: Tracking associations: How interviewees described the characteristics of other members of their group

Interviewee A B C D E F G H I J K L Tot

Friendly 1 1 1 1 1 1 1

1

8

Non-confrontational 1 1 1 1

1
1 1
7

Task focused 1 1 1
1

1
1

6

Collaborative 1 1

1 1

1
1 6

Impersonal
1

1 1
1

4

Formal

1
1

1 1

4

Informal 1 1 1

1

4

Supportive

1 1 1 1

4

Knowledgeable

1

1

1

3

Democratic 1 1

2

Easy going
1

1

2

Assertive

1

1

Honest

1

1

Punctual

1

1

Variation was further explored by comparing the three most active participants (who were male and English was their first language) with those who were less active. For example it was found that the most active participants were much more likely to talk about learning in a broader context and were more committed to learning in a group. Through group work they felt they would be learning in a way that was more relevant to them and more rooted in professional practice. Less active participants tended to see group work more as a support for understanding the course material, rather than to be valued in its own right.

Key findings from the interviews were:

The module was seen as enjoyable and learning objectives were met.
Communication was task focused and geared towards the production of reports on case problems.
Students were comfortable about taking part in online discussion and having their comments recorded.
Group work was structured around interpreting an activity, planning, drafting, and collating a group response.
There was scope for wider discussion but in practice there was not the time to do this. The focus was on fulfilling an individual commitment to the group by meeting schedules for discussion and the posting of individual viewpoints. The module was more time consuming than other 'traditional' modules.
Tutors were seen as supporting groupwork, but students in the less active groups wanted greater tutor involvement.
Collaboration was seen as successful in that tasks were completed and others within the group were supportive and 'democratic', but rarely challenging.
Commitment to group work was the most important factor in explaining higher levels of individual participation. Levels of participation were seen as stable and self reinforcing, for example active participants would access the discussion area more regularly and were drawn into more debate.
Active participants tended to be more spontaneous in their writing, for example they would normally compose messages directly within the conferencing software.
Both tutors and students tended to stress the shared professional background of the cohort rather than variation in gender, cultural or linguistic background.

Triangulation

Triangulation is a term used in different contexts (Denzin 1997) but has come to be associated most clearly with the use of more than one method for gathering data and an explicit concern for comparison of different sets of data. In this study the findings derived from each method were examined in respect to consistency (ie. there was a match between findings) and contrast (i.e. findings were contradictory). A third category complementarity referred to findings, derived from one method, which added a perspective unavailable, or simply not apparent, within the findings from a different method.

There was a high degree of consistency between findings. However there were also many instances of complementarity, so that relying on one set of data would give a partial, even a misleading, impression of the module. There were no instances of outright contrast but there was a marked difference of emphasis between survey and interview findings regarding the impact of time. The key findings from this exercise in triangulation, alongside a commentary on research methods, are:

Overall the module was very well received
Questionnaire survey and interview findings were consistent in showing a high degree of satisfaction with the organisation, content, assessment and group work associated with the module. This was consistent with message analysis which showed relatively high rates of student participation and the involvement of all students.
Time was a significant constraint on group work
Questionnaire survey and interview findings were consistent in showing pressure of time as the most cited obstacle to engaging with online discussion. However, interviews gave a much stronger and complementary perspective - the module required more time and commitment than 'traditional' distance learning. and students had to carefully manage their commitment to study. This was reinforced by message analysis in that strongly interactive messages were rare and writers tended to be on task and product focused.
There was a low threshold to taking part in online discussion
Questionnaire survey and interview findings were consistent in showing students felt at ease with online working and again message analysis showed relatively high rates of participation.
Online discussion was valued because there was permanent storage of messages
Questionnaire survey and interview findings were consistent in that both pointed to the permanent storage of discussions and any time access as the most valued attributes of online discussion. Both sets of findings showed that students put a high value on getting access to the views of others and rated this more highly than specific feedback on their own contributions.
Tutoring support was seen as appropriate but was less focused on pedagogical support
Questionnaire survey and interview findings were consistent in showing that tutor support was seen as appropriate and valuable. Interview findings gave a complementary perspective in that students in the least active groups were more critical of tutor activity and wanted more tutor involvement with more focus on pedagogic support. Message analysis showed that the most active groups tended to have the most active tutor.
Group work could best be described as cooperative rather than collaborative
Questionnaire survey and interview findings were consistent in showing that students were satisfied with the organisation of group work. However, interview findings provided a more complex picture: students were carrying out cooperative rather than collaborative group work (involving a leader, a role rotated within the group, and subdivision of tasks leading to a group plenary) with discussion very task focused.
There were variations in style and levels of participation
Variations in levels of student activity by person, by group and by phase were clear in the message analysis and confirmed within interviews. There was a high degree of consistency between how participants reported their level of online activity and quantitative analysis of their messages. Message analysis suggested gender and language background were important considerations in considering level of participation. Interviews gave a complementary perspective by showing that students' appreciation of the value of group work as a key factor.

Discussion

The study reported on an approach to online learning which was well received by students, offered benefits over traditional distance learning, and where there was a low threshold to participation. However it also showed limited levels of collaboration and constraints on participation, particularly lack of time. The approach taken to learning was largely strategic in that students effectively organised their studying methods, managed their involvement, and were alert to the assessment requirements (Entwistle & Ramsden, 1983), and cooperative in that there was a sub-division of tasks and responsibilities. This contrasted with more holistic approaches to collaboration (as discussed for example in Curtis & Lawson, 2001). This conceptualisation of strategic online cooperation emerged through triangulation of findings and an awareness of the value and the difficulties in using each research method. These are further discussed below.

Questionnaire survey: Strengths and weaknesses

Both closed and open-ended questionnaires were employed in this research and discussion of these is standard in research guides (see for example Cohen & Manion, 1989; Robson, 1993). Surveys have been used in much of the reported research into online learning (eg. Biesenbach-Lucas, 2003; Carswell, Thomas, Petre, Price & Richards, 2000; De Abreu Moreira & Quintino Da Silva 2003; Johnson, Suriya, Won Yoon, Berret & La Fleur, 2002; Mazzolini & Maddison, 2003; Morse, 2003; Oliver & Shaw, 2003; Shaw & Pieter, 2000; Spiceland & Hawkins, 2002; White & Le Cornu, 2002; Yang & Tang, 2003). Questionnaires were drafted and piloted to ensure clarity and to test reliability. Questions covered issues in learning, community, communication, participation, tutoring and curriculum design. Questionnaires were accessed electronically through an online form within the course web site - the advantage here being cost and time (particularly important, given the international spread of respondents). Many of the disadvantages of electronically delivered questionnaires (eg. see Woong Yun & Trumbo, 2000) were not apparent as all the survey population had access to the Internet and regularly accessed the course web site.

In this study, and in studies in general, the strength of the questionnaire survey lay in the breadth of the findings. It was not possible to interview everybody or analyse every message, but it was possible to ask everybody to complete an electronic questionnaire and so get an overview of student attitudes and behaviour. Survey returns required interpretation, for example, the Likert scale required the researchers to balance strength of response (eg. how important is it that some respondents 'agree', others 'strongly agree'?) with breadth of response (ie. what are the aggregated totals for those agreeing or strongly agreeing?).

However the major difficulty with interpretation concerned the degree to which respondents could be said to be representative of the cohort as a whole. Anonymity was preserved, so that while general reminders were sent out to the cohort, it was not possible to mail individuals encouraging them to reply or to track the activity of non-respondents within the message analysis. This created particular problems with the final evaluation questionnaire, delivered at the end of the module, to which less than half the cohort replied. The reason for this low response rate could have been the timing, as the module was drawing to a close. As it happened the degree of satisfaction with the module expressed within this final survey was broadly consistent with a previous one but there were continuing doubts that that those responding to the surveys were more likely to express positive feelings about the course, more comfortable with ICT, and less likely to be under extreme time pressure. Hence comparison with interview data was important. It was found that survey data did present a more positive view of the module than the interview data but this seemed more a consequence of the non-interactive format of the questionnaire, and the more probing nature of the interviews, rather than a problem of response rate per se.

Message analysis

Both quantitative and qualitative methods were used in the research. Descriptive quantitative findings were relatively unproblematic and a feature of nearly all papers looking at online discussion. Data on activity were generated automatically and although some manual calculation was needed, for example to compare tutor and student activity, this was not time consuming. As with questionnaire surveys, quantitative message analysis gave a broad overview of group activity, in this case showing relatively high rates of participation, patterns of participation linked to group work deadlines, and variation in activity between tutors and students. The limitation of quantitative analysis was of course that it gave no direct insight into the experience of the students, or their perspective on the nature of the discussions that had taken place. The quantitative overview was therefore complemented by a content analysis. Here a sample of messages needed to be identified and a protocol for classification adopted. The sample in this study was appropriate in the context of what was known about the conference as a whole. Categorisation was more difficult. It was decided to approach message analysis without recourse to existing frameworks, as this would force the researchers into a thorough examination of purpose and validity of the codings used. In the event codings were based on message functions, and in doing so the research shared a common assumption that it was more valuable to explore process rather than content. Categorisation of functions led to the notion of independent and interactive messages, albeit recognising that these were best fit descriptions.

Difficulties in content analysis have very often focused on reliability and unit of analysis (eg. Hew & Cheung 2003; Rourke, Anderson, Garrison & Archer, 2001). In this case the unit of analysis was thematic, functions within a message, rather than the entire message itself. Reliability was ensured by constant moderation of messages until very high rates of agreement had been achieved between researchers. However it needs emphasising that reliability involved agreement between researchers. No matter how painstakingly this was done, any statements about the student's intention in writing the message or indeed the experience of the student in reading the message were pure conjecture.

The modest scope of the content analysis in this study can be compared to other studies (eg. Heckman & Annabi, 2005). Two frequently cited categorisations were introduced by Henri (1992), and Gunawardena, Lowe & Anderson (1997). Henri focused on interaction and broke messages into the units of meaning and analysed them through five dimensions: participative, social, interactive, cognitive, and meta-cognitive. Each of these dimensions contains sub-categories. Gunawardena et al (1997) focused much more on the knowledge construction process, categories covered sharing/ comparing of information; the discovery and exploration of dissonance or inconsistency among ideas, concepts, or statements; negotiation of meaning/co-construction of knowledge; testing and modification of proposed synthesis or co-construction; and agreement statement(s)/ applications of newly constructed meaning. A further categorisation is offered by Garrison & Anderson (2003), who introduced a community of inquiry model which described online learning processes in terms of cognitive; social (Rourke, Anderson, Garrison & Archer, 1999); and teaching presences (Anderson et al, 2001).

The value of content analysis lies in describing the nature and scope of discussion forums. This enables comparative judgements to be made between one conference and another: for example is discussion A more interactive than B?; does group A show more evidence of higher order language functions than B?; in A does the tutor performs a wider range of functions? Mason (1991) argued that content analysis would enable judgements to be made about the educational value of asynchronous online discussion. However, content analysis has a valuable but more limited role in providing a set of data against which student perceptions may be compared and contrasted. Direct evidence of the student's perception of the educational value of the conference could only come from survey or interview data.

Interview data

Interviews allowed exploration of individual experiences through a schedule of open ended questions. The major value of interviewing, as opposed to survey questionnaire, is to delve into the student experience in greater depth through interactive conversation, albeit a conversation which the interviewer tries not to distort by inappropriate intervention, prompts and body language (Cohen & Manion, 1989; Robson, 1993). These interviews were carried out face to face with students. Interviews were transcribed, coded and later categorised with appropriate moderation between researchers. Interviews have been employed in a large body of research into asynchronous online discussion (eg. Brown, 2001; Hammond, 1999; Jones & Asensio, 2001; Lindblom-Ylänne & Pihlajamäki, 2003; Miller & Ewing, 2000; van Weert & Pilot 2003). However, the use of interview data is less common than content analysis, perhaps due to obvious difficulties of access to distance learners. In this study, face to face interviewing was made possible as students attended a residential event, but time constraints meant only a representative sample of students could be interviewed. These interviews took place some time after the module had finished, though participants had no apparent difficulty in recalling their involvement. The coding of data followed a grounded approach (cf Brown, 2001) in that categories emerged through immersion in the text. However, unlike 'pure' approaches to grounded theory (Glaser & Strauss, 1968), the researchers already had engaged in the literature before carrying out codings.

Strengths and weaknesses of methods

The strengths and difficulties in each research method are summarised in Table 4: the interviews provided depth, the survey breadth, and the message analysis a set of independent data against which perceptions could be compared. All three methods raised ethical issues concerning confidentiality and consent which were addressed.

Table 4: Summary of strengths and difficulties in using research methods

Method Value General issues to address Issues raised within this study

Interview Allows exploration of individual experience;
interactive;
in depth. Sample size and representative nature;
interview protocols;
transcribing and coding of messages;
aggregating findings. Access to interviewees;
date of interview.

Message
analysis Provides 'independent' data;
indicates nature and scope of forums;
indicates variation in participation. Categories for content analysis;
unit of analysis;
sampling of transcripts;
reliability in applying categories. Rejection of a priori approach to content analysis.

Survey
question-
naire Provides broad overview of student population. Nature of analysis;
representative nature of respondents;
non interactive format. Electronic delivery of questionnaire - response rate.

The value of triangulation

This study reinforced the case for triangulation and showed three major advantages:

There were some perspectives which could only be accessed via one method, eg. students' management of time, their engagement with reading and approaches to composing messages only emerged clearly during interviews.
Findings from one method could be put in a wider perspective through comparison with those from other methods, eg. students' accounts of their online activity could be compared to the objective data concerning frequency of message postings.
Consistency between findings gave greater authority in reporting, eg. the claim that students valued the module and adopted a task focused approach to group work is credible.

Indeed the only disadvantage with triangulation of data is its time consuming nature. However, there are qualifications before offering this particular triangulated approach as a model for research into asynchronous online discussion:

The methods used here were particularly appropriate to arrive at a holistic description of the module. Other research might have a more specialised focus and make use of a single method. For example, if research is examining the experience of the student the sole, or at least most obvious method, is interviewing (e.g. Jones & Asensio 2001; O'Regan 2003); if research is examining the nature of discourse, rather than the experience of engaging in discourse, then the method is necessarily content analysis (e.g. Anderson et al, 2001).
Other methods could be considered in addition to, or as alternatives, to the ones used here. For example researchers could engage more closely with students in the process of evaluation (as implied in Collings & Pearce, 2002; McLoughlin, 2002; Putz & Arnold, 2001) leading to student involvement in content analysis. Alternatively, assessment data could be used to explore learning outcomes (this need not imply an experimental approach) and researchers could engage more deeply with the subject matter rather than the process of knowledge construction within their content analysis.

Finally the limits on triangulation need setting out. Triangulation suggests a process akin to surveying in which accurate measurements are reached through calculation based on a set of readings. However, this is a misleading metaphor for educational research (Massey, 1999) as there is no agreement on the tools the researcher should use, each will be judged as to fitness for purpose. Triangulation assists in reaching judgements but these are still matters of best fit and personal interpretation; triangulation enhances the credibility and persuasiveness of an account but the researcher does not arrive at an objective truth. This raises a further dimension to triangulation - that of putting findings in the context of the wider literature. Here, the study confirmed the importance of the tutor role (put forward for example by Anderson et al, 2001; Hawkey, 2003; Miller & Ewing, 2000; Salmon, 2000). Variation in student activity was uncovered and seen primarily in terms of orientation to learning rather than confidence with subject matter (as in Brett, 2004); self confidence (eg. Wearmouth, 2004), or learning styles (eg. Carswell, Thomas, Petre, Price & Richards, 2000; Cunningham-Atkins, Powell, Moore, Hobbs & Sharpe, 2004; Meyer, 2003), albeit these three other factors are clearly associated with orientation to learning. Gender, cultural and language background offered further perspectives on levels of participation, but findings were not as clear cut as reported by, for example, Graddy (2004) and Morse (2003). Overall the study presented a cautiously optimistic case for using online discussion to support teaching and learning in which the constraints, widely reported elsewhere (eg. Angeli et al, 2003; Hammond, 1999; Lockhorst, Admiraal, Pilot & Veen, 2002; McCabe, 1999; Parker & Gemino, 2001; Seabrooks et al, 2000; Shaw & Pieter, 2000; Thomas, 2002), are recognised. This contrasted with the more straight forward optimism, even romanticism, of some of the literature (eg. Boder, 1992; Harasim, 1989; McConnell, 2000; Salmon, 2000).

References

Åhlberg, M., Kaasinen, A., Kaivola, T. & Houtsonen, L. (2001). Collaborative knowledge building to promote in-service teacher training in environmental education. Technology, Pedagogy and Education, 10(3), 227-238.

Anderson, T., Rourke, L., Garrison, R. & Archer, W. (2001). Assessing teacher presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2), 1-17. http://www.sloan-c.org/publications/jaln/v5n2/v5n2_anderson.asp

Angeli, C., Valanides, N. & Bonk, C. (2003). Communication in a web-based conferencing system: the quality of computer-mediated interactions. British Journal of Educational Technology, 34(1), 31-43.

Aviv, R. (2000). Educational performance of ALN via content analysis. Journal of Asynchronous Learning Networks, 4(2), 53-72. http://www.sloan-c.org/publications/jaln/v4n2/v4n2_aviv.asp

Aviv, R. Erlich, Z., Ravid, G. & Geva, A. (2003). Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks, 7(3), 1-23. http://www.sloan-c.org/publications/jaln/v7n3/v7n3_aviv.asp

Biesenbach-Lucas, S. (2003). Asynchronous discussion groups in teacher training classes: Perceptions of native and non-native students. Journal of Asynchronous Learning Networks, 7(3), 24-46. http://www.sloan-c.org/publications/jaln/v7n3/v7n3_biesenbach-lucas.asp

Boder, A. (1992). The process of knowledge reification. Journal of Computer Assisted Learning, 8(3), 177-185.

Brett, C. (2004). Off-line actors contributing to online engagement. Technology, Pedagogy and Education, 13(1), 83-95.

Brown, R. (2001). The process of community-building in distance learning classes. Journal of Asynchronous Learning Networks, 5(2), 18-35. http://www.sloan-c.org/publications/jaln/v5n2/v5n2_brown.asp

Carswell, L., Thomas, P., Petre, M., Price, B. & Richards, M. (2000). Distance education via the Internet: The student experience. British Journal of Educational Technology, 31(1), 29-46.

Cohen, L. & Manion, L. (1989). Research methods in education. London, Routledge.

Collings, P. & Pearce, J. (2002). Sharing designer and user perspectives of web site evaluation: A cross-campus collaborative learning experience. British Journal of Educational Technology, 33(3), 267-278.

Cook, D. and Ralston, J. (2003). Sharpening the focus: Methodological issues in analsysing on line conferences. Technology, Pedagogy and Education, 12(3), 361-376.

Cunningham-Atkins, H., Powell, N., Moore, D., Hobbs, D. & Sharpe, S. (2004). The role of cognitive style in educational computer conferencing. British Journal of Educational Technology, 35(1), 69-80.

Curtis, D. & Lawson, M. (2001). Exploring collaborative online learning. Journal of Asynchronous Learning Networks, 5(1), 21-34. http://www.sloan-c.org/publications/jaln/v5n1/v5n1_curtis.asp

De Abreu Moreira, D. & Quintino Da Silva, E. (2003). A method to increase student interaction using student groups and peer review over the Internet. Education and Information Technologies, 8(1), 47-54.

de Corte, E. (2003). Powerful learning environments. Pergamon, Oxford, UK.

Denzin, N.K. (1997). Triangulation in educational research. In P. Keeves (Ed), Educational research methodology and measurement: An international handbook. Pergamon Press, UK.

Entwistle, N. J. & Ramsden, R. (1983). Understanding student learning. London, Croom Helm.

Galanouli, D. & Collins, J. (2000). Using unmediated computer conferencing to promote reflective practice and confidence-building in initial teacher education. Technology, Pedagogy and Education, 9(2), 237-254.

Garrison, D. R. & Anderson, T. (2003). E-learning in the 21st century: A framework for research and practice. London, Routledge Falmer.

Glaser, B. G. & A. L. Strauss (1968). The discovery of grounded theory: Strategies for qualitative research. London, Weidenfeld and Nicolson.

Graddy, D. (2004). Gender and online discourse in the principles of economics. Journal of Asynchronous Learning Networks, 8(4), 1-12. http://www.sloan-c.org/publications/jaln/v8n4/v8n4_graddy.asp

Gunawardena, C., Lowe, C. & Anderson, T. (1997). Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing, Journal of Educational Computing Research, 17(4), 397-431.

Hammond, M. (1999). Issues associated with participation in on line forums: the case of the communicative learner. Education and Information Technologies, 4(4), 353-367.

Harasim, L. (1989). Online education: a new domain. In R. Mason & A. Kaye (Eds), Mindweave: Communication, computers and distance education. Oxford, Pergamon Press, 50-62.

Hawkey, K. (2003). Asynchronous text-based discussion: A case study with trainee teachers. Education and Information Technologies, 8(2), 165-177.

Heckman, R. & Annabi, H. (2005). A content analytic comparison of learning processes in online and face-to-face case study discussions. Journal of Computer-Mediated Communication, 10(2). http://jcmc.indiana.edu/vol10/issue2/heckman.html

Henri, F. (1991). Computer conferencing and context analysis. In A. Kaye (Ed), Collaborative learning through computer conferencing. Springer-Verlag, Berlin.

Hew, K. F. & Cheung, W. S. (2003). Models to evaluate online learning communities of asynchronous discussion forums. Australian Journal of Educational Technology, 19(2), 241-259. http://www.ascilite.org.au/ajet/ajet19/hew.html

Hiltz, R., Coppola, N., Rotter, N., Turoff, M. & Benbunan-Fich, R. (2000). Measuring the importance of collaborative learning for the effectiveness of ALN. Journal of Asynchronous Learning Networks, 4(2), 103-125. http://www.sloan-c.org/publications/jaln/v4n2/v4n2_hiltz.asp

Hubscher-Younger, T. & Narayanan, N. (2003). Authority and convergence in collaborative learning. Computers and Education, 41, 313-334.

Johnson, S., Suriya, C., Won Yoon, S., Berret J. & La Fleur, J. (2002). Team development and group processes of virtual teams. Computers and Education, 39, 379-393.

Jones, C. & Asensio, M. (2001). Experiences of assessment using phenomenography for evaluation. Journal of Computer Assisted Learning, 17(3), 306-313.

Koory, M. (2003). Differences in learning outcomes for the online and F2F versions of an introduction to Shakespeare. Journal of Asynchronous Learning Networks, 7(2), 18-34. http://www.sloan-c.org/publications/jaln/v7n2/v7n2_koory.asp

Kumari, D. (2001). Connecting graduate students to virtual guests through asynchronous discussions. Journal of Asynchronous Learning Networks, 5(2), 53-63. http://www.sloan-c.org/publications/jaln/v5n2/v5n2_kumari.asp

Light, P., Nesbitt, E., Light, V. & White, S. (2000). Variety is the spice of life, student use of CMC in the context of campus based study. Computers and Education, 34, 257-267.

Lindblom-Ylänne, S. & Pihlajamäki, H. (2003). Can a collaborative network environment enhance essay-writing processes. British Journal of Educational Technology, 34(1), 17-30.

Lockhorst, D., Admiraal, W., Pilot, A. & Veen, W. (2002). Design elements for a CSCL environment in a teacher training programme. Education and Information Technologies, 7(4), 377-384.

Macdonald, J. & Twining, P. (2002). Assessing activity-based learning for a networked course. British Journal of Educational Technology, 33(5), 603-618.

Martinez A, Dimitriades, Y, Rubia, B, Gomez, E. & de la Fuente, P. (2003). Combining qualitative evaluation and social network analysis for the study of classroom interactions. Computers and Education, 41, 355-368.

Mason, R. (1991). Methodologies for evaluating applications of computer conferencing. In A.R. Kaye (Ed), Collaborative learning through computer conferencing. Heidelberg: Springer-Verlag.

Mason, R. & Kaye, A. (Eds) (1989). Mindweave: Communication, computers and distance education. Oxford, Pergamon Press.

Massey, A. (1999). Methodological triangulation, or how to get lost without being found out. Studies in Educational Ethnography, 2, 183-197.

Mazzolini, M. & Maddison, S. (2003). Sage, guide or ghost? The effect of instructor intervention on student participation in on line discussion forums. Computers and Education, 40, 237-253.

McCabe, M. F. (1998). Lessons from the field: computer conferencing in higher education. Technology, Pedagogy and Education, 7(1), 71-87

McConnell, D. (2000). Implementing computer supported cooperative learning. London, Kogan Page.

McLoughlin, C. (2002). Computer supported teamwork: An integrative approach to evaluating cooperative learning in an online environment. Australasian Journal of Educational Technology, 18(2), 227-254. http://www.ascilite.org.au/ajet/ajet18/mcloughlin.html

Meyer, K. (2003). Face to face versus threaded discussions: The role of time and higher order thinking. Journal of Asynchronous Learning Networks, 7(3), 55-65. http://www.sloan-c.org/publications/jaln/v7n3/v7n3_meyer.asp

Miller, D. & Ewing, J. (2000). Beyond knowledge transmission? Computer-supported learning in teacher education: some benefits in terms of stress, control and self-belief. Technology, Pedagogy and Education, 9(3), 363-376.

Morse, K. (2003). Does one size fit all? Exploring asynchronous learning in a multicultural environment. Journal of Asynchronous Learning Networks, 7(1), 37-5. http://www.sloan-c.org/publications/jaln/v7n1/v7n1_morse.asp

Oliver, M. & Shaw, G. (2003). Asynchronous discussion in support of medical education. Journal of Asynchronous Learning Networks, 7(1), 56-67. http://www.sloan-c.org/publications/jaln/v7n1/v7n1_oliver.asp

O'Regan, K. (2003). Emotion and e-learning. Journal of Asynchronous Learning Networks, 7(3), 78-92. http://www.sloan-c.org/publications/jaln/v7n3/v7n3_oregan.asp

Parker, D. & Gemino, A. (2001). Inside on-line learning: Comparing conceptual and technique learning performance in place-based and ALN formats. Journal of Asynchronous Learning Networks, 5(2), 64-92. http://www.sloan-c.org/publications/jaln/v5n2/v5n2_parkergemino.asp

Putz, P. & Arnold, P. (2001). Communities of practice: Guidelines for the design of online seminars in higher education. Education, Communication and Information, 1(2), 181-195.

Robson, C. (1993). Real world research: A resource for social scientists and practitioner-researchers. Oxford, Blackwell.

Rourke, L., Anderson, T., Garrison, D. R. & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, 8-22.

Rourke, L., Anderson, T., Garrison, D. & Archer, W. (1999). Assessing social presence in asynchronous, text-based computer conferences. Journal of Distance Education, 14(3), 51-70.

Salmon, G. (2000). E-moderating: The key to teaching and learning online. London, Routledge.

Salmon, G. (2002). Mirror, mirror, on my screen: Exploring online reflections. British Journal of Educational Technology, 33(4), 379-391.

Seabrooks, J., Kenney, S. & LaMontagne, M. (2000). Collaboration and virtual mentoring: Building relationships between pre-service and in-service special education teachers. Technology, Pedagogy and Education, 9(2), 219-236.

Shaw, G. & Pieter, W. (2000). The use of asynchronous learning networks in nutrition education: Student attitude, experiences and performance. Journal of Asynchronous Learning Networks, 4(1), 40-51. http://www.sloan-c.org/publications/jaln/v4n1/v4n1_shawpieter.asp

Snell, R., Hodgson, V. & Mann, S. (Eds) (1987). Beyond distance learning towards open learning. Open University Press, London.

Spiceland, D. & Hawkins, C. (2002). The impact on learning of an asynchronous active learning course. Journal of Asynchronous Learning Networks, 6(1), 68-75. http://www.sloan-c.org/publications/jaln/v6n1/v6n1_spiceland.asp

Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication and Information, 2(1), 23-49.

Taylor, J. (2001). Virtual writing forum with Don Murray and the national writing project in an asynchronous environment. Journal of Asynchronous Learning Networks, 5(1), 99-134. http://www.sloan-c.org/publications/jaln/v5n1/v5n1_taylor.asp

Thomas, M. (2002). Implementing a CMC tutor group for an existing distance education course. Journal of Computer Assisted Learning, 18(3), 351-366.

Tolmie, A. & Boyle, J. (2000). Factors influencing the success of computer mediated communication (CMC) environments in university teaching: A review and a case study. Computers and Education, 34, 119-140.

van Weert, T. & Pilot, A. (2003). Task-based team learning with ICT, design and development of new learning. Education and Information Technologies, 8(2), 195-214.

Vandergrift, K. (2002). The anatomy of a distance education course: A case study analysis. Journal of Asynchronous Learning Networks, 6(1), 76-90. http://www.sloan-c.org/publications/jaln/v6n1/v6n1_vandergrift.asp

Wallace, R. (2003). Online learning in higher education: A review of research on interactions among teachers and students. Education, Communication and Information, 3(2), 241-280.

Watson, G. & Prestridge, S. (2003). A networked learning community approach to sustain teacher ICT professional development. Australian Journal of Educational Technology, 19(2), 227-240. http://www.ascilite.org.au/ajet/ajet19/watson.html

Wearmouth, J., Smith A. & Soler J. (2004). Computer conferencing with access to a 'Guest Expert' in the professional development of special educational needs coordinators. British Journal of Educational Technology, 35(1), 81-93.

Weller, M. (2000). Implementing a CMC tutor group for an existing distance education course. Journal of Computer Assisted Learning, 16(3), 178-83.

White, B. & Le Cornu, R. (2002). Email reducing stress for student teachers. Education and Information Technologies, 7(4), 351-357.

Woong Yun, G. & Trumbo, C. (2000). Comparative response to a survey executed by post, e-mail, and web form. Journal of Computer Mediated Communication, 6(1). http://jcmc.indiana.edu/vol6/issue1/yun.html

Yang, H. & Tang, J. (2003). Effects of social network on students' performance: a web based forum study in Taiwan. Journal of Asynchronous Learning Networks, 7(3), 93-107. http://www.sloan-c.org/publications/jaln/v7n3/v7n3_yang.asp

Authors: Michael Hammond, Centre for New Technologies Research in Education, Institute of Education, University of Warwick, CV4 7AL, United Kingdom. Email: hammond@warwick.ac.uk

Mongkolchai Wiriyapinit, Faculty of Commerce and Accountancy, Chulalongkorn University, Bangkok 10330, Thailand. Email: mongkolchai@acc.chula.ac.th

Please cite as: Hammond, M. and Wiriyapinit, M. (2005). Learning through online discussion: A case of triangulation in research. Australasian Journal of Educational Technology, 21(3), 283-302. http://www.ascilite.org.au/ajet/ajet21/hammond.html

[ AJET 21 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet21/hammond.html
Created 16 Sep 2005. Last revised 16 Sep 2005.
Page access count

I like online discussion because:	%
I can get feedback on my ideas	73
I can go back to read messages	73
I can mail when I like	64
I can mail where I like	50
I can get to know people from different cultures	36
I am not interrupted when I contribute	32
I can get to know other people in my group well	32

Overarching patterns of communication	Examples of functions	Total %
Independent	Introducing, stating, reporting, telling, drafting, final-drafting.	37
Interactive	Commenting, agreeing, expressing, requesting, clarifying, confirming, reasoning.	59
Strongly interactive	Questioning, arguing, disagreeing.	4

Interviewee	A	B	C	D	E	F	G	H	I	J	K	L	Tot
Friendly	1	1	1	1	1	1	1			1			8
Non-confrontational	1	1	1	1				1		1	1		7
Task focused	1	1	1		1			1		1			6
Collaborative	1	1			1	1				1		1	6
Impersonal		1			1	1		1					4
Formal				1		1			1	1			4
Informal	1	1	1				1						4
Supportive			1	1	1	1							4
Knowledgeable			1			1				1			3
Democratic	1	1											2
Easy going		1				1							2
Assertive				1									1
Honest				1									1
Punctual						1							1

Method	Value	General issues to address	Issues raised within this study
Interview	Allows exploration of individual experience; interactive; in depth.	Sample size and representative nature; interview protocols; transcribing and coding of messages; aggregating findings.	Access to interviewees; date of interview.
Message analysis	Provides 'independent' data; indicates nature and scope of forums; indicates variation in participation.	Categories for content analysis; unit of analysis; sampling of transcripts; reliability in applying categories.	Rejection of a priori approach to content analysis.
Survey question- naire	Provides broad overview of student population.	Nature of analysis; representative nature of respondents; non interactive format.	Electronic delivery of questionnaire - response rate.