|Australasian Journal of Educational Technology
2010, 26(3), 341-354.
Using wikis for collaborative learning: Assessing collaboration through contribution
Terry Judd, Gregor Kennedy and Simon Cropper
The University of Melbourne
Wikis are widely promoted as collaborative writing tools and are gaining in popularity in educational settings. However, while wikis include features that are designed to facilitate collaboration, it does not necessarily follow that their use will ensure or even encourage collaborative learning behaviour. The few empirical studies that have considered this issue report equivocal findings. We assessed students collaborative behaviour based on their contributions to a wiki-based shared writing task using a variety of text and time based metrics. We found little evidence of collaboration despite adopting a learning design that was intended to support it. While overall participation was high, a relatively small proportion of students did the bulk of the work and many students' contributions were superficial. Students made little use of the wiki's commenting feature - a critical tool for contextualising and coordinating their contributions for and with others, and the majority of contributions were made very late in the task, making the possibility of extensive collaboration unlikely. These findings are discussed in relation to factors that may lead to the more successful integration of innovative, technology based learning activities into broader undergraduate curricula.
Wikis have proven particularly popular (Bower, Woo, Roberts & Watters, 2006; Bruns & Humphreys, 2005, 2007). Wikis are websites that can be interactively edited by any number of people using simple online tools. An incremental version of the website is stored each time an edit is saved, making it possible to 'rollback' the site to any previous version if subsequent edits need to be undone. What sets wikis apart from other 'social' writing and publishing tools (e.g. blogs, photo-sharing, podcasts) is that more than one person typically contributes to the authoring and publishing of specific content, and most wiki tools include a range of features to facilitate these multi-author interactions. These include email or RSS (Really Simple Syndication) notifications of page edits and the ability for authors to add comments to a page prior to or after editing its content. This combination of functionality and features has led to wikis being promoted as powerful collaborative learning tools and they are increasingly being used to support group-based collaborative learning tasks.
Dillenbourg, Baker, Blaye and O'Malley (1996) make a useful distinction between collaborative and cooperative behaviour when students complete learning tasks. Cooperation is defined by the division of labour between individuals who are faced with a joint activity, while collaboration involves the "mutual engagement of participants in a coordinated effort to solve the problem" (p. 190). This distinction is supported by Scanlon (2000) who suggests,
Co-operation usually implies either splitting up the work or solving subtasks individually and combining the results into a Ūnal product. In contrast, collaboration can mean a coordinated attempt to solve and monitor a problem together, with perhaps some division of labour on aspects of the problem (p. 464-465).While, from a social learning theory perspective, peers collaborating is more desirable than them simply cooperating - "two heads are better than one" - there is limited evidence from the literature about the ways in which these distinct types of interactions are promoted by wiki-based learning tasks.
On the (largely) positive side, Forte and Bruckman (2006) investigated college students' use of a wiki to publish essays on public policy that other students could critique. Their findings, particularly with regards to collaboration, showed that students tended to post the largest edits (full drafts of essays) close to the assessment deadline while "smaller contributions like sharing resources and giving evaluations were more consistently spaced out over many days preceding due dates" (p. 184). They also found a high proportion of students (78.9%) addressed at least one student evaluation or critique of their draft essay in their final submission. Similarly, Minocha and Thomas (2007) used a wiki to encourage collaborative activities and discussions among groups of distance students undertaking a software engineering development project. These authors stated that "wiki activities facilitated collaborative learning and that a wiki is a good medium for collaborative work in a distance education course" (p. 198) and reported that three quarters of students involved in a collaborative writing task agreed that wikis facilitate collaborative learning.
Others have been more circumspect. Cole (2009) redesigned a course to include an activity that involved students publishing course-related material to a wiki, which was then the subject of review, seminar discussion, synthesis and reflection. However, at the halfway point of the semester not one of the 75 students enrolled in the course had contributed to the wiki. Students gave a number of reasons for their non-participation including time and work pressures and lack of interest in technology. Similarly, Ebner, Kikckmeier-Rust and Holzinger (2008) report on a wiki implementation in which none of the 287 students involved created new pages or edited existing pages over an entire semester. Even when participation is relatively high, much of the work may still fall to a relatively small proportion of contributors (Carr, Morrison, Cox & Deacon, 2007).
These and other findings suggest that wikis are not inherently collaborative and that additional components are required to promote participation and collaboration among students. Lack of incentives (such as the absence of formal assessment) and insufficient support for group work are often cited as barriers to the effective use of wikis (e.g. Grant, 2009; Bruns & Humphreys, 2007, Robertson, 2008; Neumann & Hood, 2009).
Although perception-based studies of educational wiki implementations continue to dominate the literature, a growing number of studies have drawn on the log data generated by wikis to support their investigations of student participation (Forte & Bruckman, 2006; Elgort, Smith & Toland, 2008; Cole, 2009; Trentin, 2009). Many educational technology environments, including most major learning management systems, include some capacity to automatically log activity and contributions by users. However, wikis are especially strong in this regard, as they routinely maintain detailed logs of user edits and comments, and often include robust notification systems (e.g. email alerts and/or RSS feeds) that can be used to monitor contributions in close to real time. Given that cooperative and collaborative behaviour by wiki users is predicated to a large degree on their patterns of contribution, routine analysis of the nature, scope and context of user contributions might lead to the development of relatively simple yet robust measures of such behaviour (e.g. Swan, Shen & Hiltz, 2006; Biuk-Aghai, Kelen & Venkatesan, 2008; Trentin, 2009).
This paper describes a wiki implementation of a collaborative writing task undertaken by undergraduate psychology students. Assessment of the task was based on minimum contribution requirements and these were successfully verified using a series of bespoke wiki log analysis routines. However, given the nature of the task, we were also interested in assessing the degree to which students behaved cooperatively and collaboratively in developing their contributions and whether this might be determined or inferred through a more detailed analysis of number, timing and scope of contributions. We explore this potential and in doing so identify prospective deficiencies in both our learning design and students' and educators' use of wikis. Potential approaches for mitigating or ameliorating such problems are then discussed.
The focus of this investigation was students' use of a wiki for a collaborative writing activity. In Week 6 of the semester, the lecturer presented two lectures on motion detection, a key concept in the cognitive psychology curriculum. Each lab class was then asked to collaboratively create a wiki-based submission on the subject of motion detection. Students were provided with guidance on the basic concepts that should be covered in their submission and were encouraged to create scholarly summaries of the key concepts using descriptions, reflections, quotes, images, web links and diagrams. Each group was provided a blank wiki page on which to develop their entry - the task was identical for all groups. Students were assessed individually rather than as a group, and satisfactory completion of the task - at least two non-trivial contributions to the wiki (textual changes of more than one sentence that were not primarily stylistic or related to formatting) - accounted for 10% of the relevant subject's assessment (4% of overall year mark).
Data harvesting and preparation
Each group's submission, comprising all versions of their wiki page or pages in HTML format, was retrieved at the completion of the task using the wiki's built-in archiving routines. A customised procedure was developed for extracting page comments, as these were not included in the archiving process.
In addition to page content, each version file contained information on the date and time of its creation and the author/editor's identity. To facilitate analysis, all HTML formatting (including references to embedded images, animation and video) was stripped from the retrieved pages, rendering them into plain text. Whitespace other than single spaces and paragraph returns were also removed. Approximately half of the 30 groups developed their submissions on a single wiki page, allowing simple comparisons and analyses of successive page versions. Each of the other groups used hyperlinks to develop multi-page submissions, and these were combined into single page entries prior to further analysis. Page comments were processed to create, for each final page version, a line-delimited list of comments comprising the author's identity, the date and time of the comment's submission and the text of the comment.
Analysis of page versions
A user identifier and timestamp were recorded for each page version. The scope of individual user edits was assessed by comparing successive page versions to derive the combined number of non-identical lines of text in pair-wise comparisons (e.g. version A vs. B plus B vs. A). These measures were then variously combined to provide measures of the number, timing and scope of edits (i) within the wiki generally (all users), (ii) within groups and (iii) by individual students. If a student created two or more successive versions over a short time (typically within an hour of each other) these were combined to form a single editing/authoring session.
Analysis of comments
A content analysis was used to code students' comments on the wiki into a series of non-exclusive categories (see categories and descriptions in Table 1). A comment was coded into a category if part or all of it matched the description, regardless of whether it was positive or negative. Each comment was independently coded by two researchers.
|Reply||A comment in response to an existing comment|
|Collaboration||A comment that showed the author was attempting to develop a shared understanding of some aspect of the page content|
|Content||A comment concerned with factual content on or relevant to the target page|
|Editing||A comment that concerned with some aspect of page editing on or relevant to the target page|
|Individual||A comment directed to an individual|
|Group||A comment directed to the group generally|
Details of the wiki tasks were provided to students in lectures (both verbally and through a series of printed notes) and at the beginning of the tasks. Time was allocated during scheduled practical classes for students to organise their roles relating to the tasks within their groups and to familiarise themselves with the use of the wiki. The main collaborative writing activity officially ran for three weeks but the wiki was open to students for several weeks beforehand and many students contributed to the discussion activity that was available during this period. However, very few students contributed to the collaborative writing activity before its formal start date.
Support was provided to tutors and demonstrators of the practical groups. These tutors were given notes on the collaborative writing activity in particular, and were given instructions for accessing the wiki that were similar to those given to the students. One author [GK] also attended one of the regular tutor briefing meetings to explain the project and the rationale behind the activity prior to it commencing. In this forum, there was a lengthy discussion, supported by brief notes provided by the project team, about how the activity could be introduced to students in the tutorials, and how to incorporate the activity into students' class time.
Students were given information about the wiki activity, support and advice before and during the task. Notes about the task, what was involved, and how to use a wiki effectively (including basic functions and etiquette) were provided to students in practical classes and were also included as a PDF download via the learning management system. The wiki incorporated detailed notes and help files for students, including notes about the task, what was involved, and how to use a wiki effectively. Students could access technical help by sending an email to the wiki administrator [TJ] and educational and administrative support was freely available through the course coordinator [SC].
Of the groups that developed multi-page submissions, a mean of 32% of participating students contributed to any given page, with 62% contributing to their 'home' page and 23% to each of the linked pages. The degree to which students contributed to the task varied considerably, with a relatively small number of students contributing most of the wiki content. Figure 1 illustrates the proportional contribution by the least and most productive 10, 25 and 50% of students. It reveals, for example, that while the most productive 10% of students contributed just over 40% of the total text, the least productive 25% of students contributed only 4.4%.
Figure 1: Proportional contribution of text (based on word counts) of the least and most productive students.
Figure 2: Temporal distribution of page edits over the time allocated to the task (task ended on day 26).
Figure 3: Percentage of students who contributed to the wiki on one or more days.
Examples of comments and their categorisations include:
is there a particular reason for the exclamation marks scattered throughout the page? [group, editing]There was a high level of agreement between the two independent coders (between 74 and 98%) with the collaboration category producing the highest level of inconsistencies. Given the coders were largely consistent in their reviews of students' comments, mean scores of their ratings are reported below (Fig. 4). All comments were scored in at least one of the contextual categories - group or individual (a small number were coded as both) and others as applicable, The overwhelming majority of students' comments were directed to their group generally rather than an individual and approximately 80% of comments were related to content or editing. Twenty-five percent of comments suggested collaboration between students and only 12% of comments were made in response to previous comments (Fig. 4).
hey! [is]anyone of us writing about the motion hierarchy? [group, content, collaboration]
I'm so sorry, I didn't realise that there was a link there. I will be really careful next time. [individual, editing]
hey [name removed], I put a picture in under your section since I thought it was pretty nifty, but feel free to get rid of it if you think it's confusing. Your definition is v. nice by the way! [individual, content, collaborate]
Figure 4: Percentage of all comments in each of the six contextual categories (mean of two coders).
Being mindful of such issues when designing our wiki activites, there was an expectation on our part that students would engage in cooperative and collaborative behaviour, supported variously by the design of the task, the involvement of teaching staff and the functionality of the wiki itself. It seemed, at least initially, as if our expectations were being met, as real time indicators of students' use of the wiki (monitoring of RSS and email alerts of contributions; regular in situ viewing of wiki pages and comments) appeared to indicate a relatively high level of coordinated and sustained activity within the wiki during both the preliminary and main writing tasks. However, a more detailed and objective analysis of students' contributions once the task was completed revealed a very different picture, evidenced by highly skewed patterns of edits with respect to both the timing and proportion of contributions.
Almost all students participated in the collaborative writing task and fewer than 10% failed to satisfactorily complete it (by contributing at least two non-trivial edits). However, the analysis of the equity of students' contributions provides evidence against a general tendency to cooperative and collaborative behaviour. The least productive 50% of students provided less than 15% of the total wiki content, which sits in stark contrast with the most productive 10% who contributed just over 40%. This suggests that many students were probably more concerned with simply meeting the task contribution requirements, which would require minimal cooperation or collaboration, than developing a comprehensive and coherent group submission, which would require establishing a cooperative or collaborative relationship with the other members of their group. Wagner and Prasarnphanich (2007) refer to such behaviour as 'social loafing' and suggest it often reflects primarily selfish or individualistic goals on the part of students - in this case meeting an individual hurdle requirement. Social loafing sits in direct contrast to the incidental or intentional altruism exhibited by those who provide more substantial and meaningful contributions - represented by the most productive 10% of students in our sample.
The timing and distribution of students' contributions also revealed a great deal about their capacity and willingness to cooperate and collaborate with other members of their group. The two major findings from our analysis in this area were that (i) the majority of contributions were made late in the activity, which is not surprising given many students' tendency toward last minute study (Bhuta et al., 2007), and (ii) most students contributed to the wiki on a single day. Both findings suggest that students would have had limited opportunities to interact, via the wiki, with other members or their groups. Clearly, the ability of two students to genuinely engage in bilateral online content creation and discussion is severely undermined if one or both of them wait until the last day of the activity to contribute. By making their contributions in the latter stages of the activity and restricting their contributions to a single day - or session even - many students seemed to preclude opportunities for either cooperative or collaborative behaviour.
The final piece of evidence that students were, on the whole, not engaging in a great deal of cooperative or collaborative behaviour is provided by the content analysis of students' page comments. These provide a simple yet powerful mechanism for students to both coordinate and contextualise their contributions with other members of their groups and even brief comments typically include statements that support one or other of these functions. Just as there was an inequity of contributions of content, so there was with page comments, with only one in ten students contributing at least one comment. While the overall number of comments was only small (less than one per 25 edits), the content analysis indicated that in a number of cases they included evidence of cooperative or collaborative behaviour. However, few comments either elicited responses or were made in response to other comments, suggesting that students' attempts to engage other students were often unsuccessful. The expected behaviour - that students would use page comments to develop a narrative around the development of their submissions - was not evident. There were too few comments to support this degree of collaborative activity, which was somewhat surprising given students vigorous participation in the voluntary familiarisation activity, which involved using page comments to respond to a series of 'opinion pieces' posted by the subject coordinator.
It can be seen, therefore, that the patterns of students' contribution to the collaborative writing task based on the amount and timing of their contributions, and the degree and content of their comments on the wiki, indicate that cooperation and collaboration among most students were low. This acts as a timely reminder that designing learning activities that are collaborative in nature, or simply using 'collaborative' technologies, does not guarantee students will work together in cohesive way. Whether the patterns of student behaviour observed in this study were incidental or intentional is unclear. Either way, there appear to be a number of potential explanations and remedies for the relatively low levels of cooperation and collaboration.
Minocha and Thomas (2007) discussed a range of issues - technical, administrative and educational - that can hamper collaboration in wikis particularly emphasising group management and coordination of edits, and the socialisation of participants (see also Choy & Ng, 2007; Elgort, Smith & Toland, 2008; Roberston, 2008). Students' preparation and support in collaborative writing activities is clearly critical to their success. A considerable amount of effort went into the design of the learning activity that was used in the study. As outlined above, substantial technical, administrative and educational support was provided to both staff and students. Moreover, a wiki familiarisation activity was integrated into the course well before the collaborative writing activity began, so that students were accustomed to the technology when they came to the collaborative writing task. Our findings suggest that this support and familiarisation did not go far enough. Improved preparation and familiarisation activities may help here but are unlikely to entirely resolve the difficulties associated with encouraging student collaboration.
A key area where the task could have been improved, thereby encouraging more meaningful contributions and collaborative activity on the part of students, is in the degree to which it was integrated with the curriculum and its assessment. It has often been said that assessment drives students engagement with learning and Biggs' (1996) notion of constructive alignment recognises the need to have close integration between the learning objectives of a course or activity, its instructional design and assessment. If the collaborative writing activity used in this study was allocated more class time for organisation and face to face group work, and the assessment of the task was weighted to reflect the value of group work, more meaningful collaboration within the wiki space may have been evident. As implemented, the task's relatively short time-frame coupled with its modest assessment weighting may not have motivated students sufficiently to effectively engage with the task.
The data and analysis presented here supports the idea, previously explored by authors such as Swan, Shen and Hiltz (2006), Bhuta et al. (2007) and Trentin (2009), that the records of user activity that are automatically recorded by a wiki can be used to analyse and interpret the nature, scope and context of user contributions. This represents an area of considerable potential, which could eventually lead to the routine use of automated wiki analysis to support both formative and summative programs of assessment. While systems based on automated content analysis are probably beyond our immediate reach, robust assessment schemes based on the timing, frequency or scope of students' contributions seem achievable.
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347-364.
Biuk-Aghai, R.P., Kelen, C. & Venkatesan, H. (2008). Visualization of interactions in collaborative writing. In Proceedings of the 2008 Second IEEE International Conference on Digital Ecosystems and Technologies (pp. 97-102), Phitsanulok, Thailand.
Bower, M., Woo, K., Roberts, M. & Watters, P. (2006). Wiki pedagogy - A tale of two wikis. In Proceedings of the International Conference on Information Technology Based Higher Education and Training (ITHET 06), (pp.191-202), Sydney, Australia.
Brown, J. S., Collins, A. & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42.
Bruns, A. & Humphreys, S. (2005). Wikis in teaching and assessment: The M/Cyclopedia project. In D. Riehle (Ed.), Proceedings of the 2005 International Symposium on Wikis (pp. 25-31). New York, ACM Press.
Bruns, A. & Humphreys, S. (2007). Building collaborative capacities in learners: The M/cyclopedia project revisited. In Proceedings of the International Symposium on Wikis, Montreal, Canada. [viewed 20 Apr 2010] http://www.wikisym.org/ws2007/_publish/Bruns_WikiSym2007_MCyclopedia.pdf
Carr, T., Morrison, A., Cox, G. & Deacon, A. (2007). Weathering wikis: Net-based learning meets political science in a South African university. Computers and Composition, 24(3), 266-284.
Cole, M. (2009). Using wiki technology to support student engagement: Lessons from the trenches. Computers & Education, 52(1), 141-146.
Dillenbourg, P., Baker, M., Blaye, A. & O'Malley, C. (1996). The evolution of research on collaborative writing. In E. Spada & P. Reiman (Eds), Learning in humans and machine: Towards an interdisciplinary learning science (pp. 189-211). Oxford, Elsevier.
Ebner, M., Kickmeier-Rust, M. & Holzinger, A. (2008). Utilizing wiki-systems in higher education classes: A chance for universal access? Universal Access in the Information Society, 7, 199-207.
Elgort, I., Smith, A. G. & Toland, J. (2008). Is wiki an effective platform for group course work? Australasian Journal of Educational Technology, 24(2), 195-210. http://www.ascilite.org.au/ajet/ajet24/elgort.html
Forte, A. & Bruckman, A. (2007). Constructing text: Wiki as a toolkit for (collaborative?) learning. In Proceedings of the 2007 International Symposium on Wikis (WikiSym) (pp. 31-42), Montreal, Canada. [verified 7 May 2010] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.105.5460&rep=rep1&type=pdf
Grant, L. (2009). 'I DON'T CARE DO UR OWN PAGE!' A case study of using wikis for collaborative work in a UK secondary school. Learning, Media and Technology, 34(2), 105-117.
Hughes, A. (2009). Higher education in a Web 2.0 World. Report of an independent Committee of Inquiry into the impact on higher education of students' widespread use of Web 2.0 technologies. The Committee of Inquiry into the Changing Learner Experience (CLEX). [viewed 20 Apr 2010] http://clex.org.uk/CLEX_Report_v1-final.pdf
Kennedy, G., Dalgarno, B., Bennett, S., Gray, K., Waycott, J., Judd, T., Bishop, A., Maton, K., Krause, K. & Chang, R. (2009). Educating the Net Generation: A handbook of findings for practice and policy. Australian Learning & Teaching Council. [verified 7 May 2010; 6.0 MB] http://www.altc.edu.au/system/files/resources/CG6-25_Melbourne_Kennedy_Handbook_July09.pdf
Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press, Cambridge, UK.
McPherson, K. (2006). Wikis and student writing. Teacher Librarian, 34(2), 70-73.
Minocha, S. & Thomas, P. (2007). Collaborative learning in a wiki environment: Experiences from a software engineering course. New Review of Hypermedia and Multimedia, 13(2), 187-209.
Neumann, D. L. & Hood, M. (2009). The effects of using a wiki on student engagement and learning of report writing skills in a university statistics course. Australasian Journal of Educational Technology, 25(3), 382-398. http://www.ascilite.org.au/ajet/ajet25/neumann.html
Robertson, I. (2008). Learners' attitudes to wiki technology in problem based, blended learning for vocational teacher education. Australasian Journal of Educational Technology, 24, 425-441. http://www.ascilite.org.au/ajet/ajet24/robertson.html
Scanlon, E. (2000). How gender influences learners working collaboratively with science simulations. Learning and Instruction, 10, 463-481.
Slavin, R. E. (1995). Research on cooperative learning and achievement: What we know, what we need to know. Contemporary Educational Psychology, 21(1), 43-69.
Swan, K., Shen, J. & Hiltz, S. R. (2006). Assessment and collaboration in online learning. Journal of Asynchronous Learning Networks, 10(1), 45-62.
Trentin, G. (2009). Using a wiki to evaluate individual contribution to a collaborative learning project. Journal of Computer Assisted Learning, 25, 43-55.
Vygotsky, L. S. (1978). Mind in society. Harvard University Press, Cambridge, MA.
Wagner, C. & Prasarnphanich, P. (2007). Innovating collaborative content creation: The rule of altruism and wiki technology. In Proceedings of the 40th Hawaii International Conference on System Sciences (pp. 18-27). IEEE Computer Society.
|Authors: Dr Terry Judd, Medical Education Unit, The University of Melbourne|
Email: firstname.lastname@example.org Web: http://www.meu.medicine.unimelb.edu.au/profiles/tjudd.html
Associate Professor Gregor Kennedy, Faculty of Medicine, Dentistry & Health Sciences.
Email: email@example.com Web: http://www.bmu.unimelb.edu.au/staff/gk/
Dr Simon Cropper, School of Behavioural Science, The University of Melbourne
Email: firstname.lastname@example.org Web: http://www.psych.unimelb.edu.au/people/staff/CropperS.html
Please cite as: Judd, T., Kennedy, G. & Cropper, S. (2010). Using wikis for collaborative learning: Assessing collaboration through contribution. Australasian Journal of Educational Technology, 26(3), 341-354. http://www.ascilite.org.au/ajet/ajet26/judd.html