AJET 24(2) Atkinson and Yeoh (2008) - Student and staff perceptions of the effectiveness of plagiarism detection software

Australasian Journal of Educational Technology
2008, 24(2), 222-240.

Student and staff perceptions of the effectiveness of plagiarism detection software

Doug Atkinson and Sue Yeoh
Curtin University of Technology

The aim of this research was to determine student and staff perceptions of the effectiveness of plagiarism detection software. A mixed methods approach was undertaken, using a research model adapted from the literature. Eight hours of interviews were conducted with six students and six teaching staff from Curtin Business School at Curtin University of Technology, which had trialled the plagiarism detection software, EVE2. A survey questionnaire was completed by 171 students involved in the trial. The summary indication was that students perceived that plagiarism is an important issue; detection software makes it easier for lecturers; it is fair to use detection software; students support its use; and it will have some effect in preventing plagiarism. However, students' concerns included being caught for unintentional plagiarism, teaching staff placing too much emphasis on detection results above student ability, and the accuracy of the software at detecting plagiarism. Concerns for teaching staff included the time taken for the detection process, limitation of the software to publicly based Internet sources and direct copying, and the extra workload involved with pursuing academic misconduct.

Introduction

Plagiarism is derived from the Latin word 'plagiarius', which means 'kidnapper' (The Concise Oxford Dictionary; Dawson 2004; Hawley, 1984). Clough (2000, p.1) suggests that plagiarism has occurred "when the work of someone else is reproduced without acknowledging the source". Hawley (1984, p.35) wrote that plagiarism is "perhaps best conceptualised as existing along a continuum of behaviours ranging from sloppy paraphrasing to the intentional copying of someone else's work verbatim without credit to the source". For the purpose of this research, plagiarism shall be defined as "the use of another's work without proper acknowledgement or permission".

A requirement of universities is to facilitate and assess individual development of knowledge and skills. This includes the acknowledgement of existing ideas in the literature and the development of original ideas. Plagiarism is a threat to individual scholarship as it becomes difficult for others to assess the knowledge and skills of the individual. In the worst cases an individual is cheating through "kidnapping" the ideas of others and deceiving the community into thinking he or she created that knowledge.

Plagiarism is probably as old as the first recorded documents but the rise of the Internet has made it particularly easy for individuals to search, retrieve, and copy and paste from electronic documents. There is thus an increasing number of plagiarism cases associated with Internet based documents. Coinciding with this incidence, there has been a growth in software designed to detect plagiarism (Lancaster & Culwin, 2004). Many of these software programs are from commercial vendors who recognised the business opportunities in a growth industry.

Plagiarism detection software (PDS) includes search engines, e.g. Google, service providers, e.g. Turnitin (Turnitin, n.d.), and client based software, e.g. EVE2 (Essay Verification Engine) (EVE2, n.d.). The basic premise is that such software saves teachers' time by automating the detection process and providing reports that indicate the plagiarised work. The software searches the Internet or repositories of student work, or both, to detect matching passages of text. The software is limited by its searching algorithms, access to document repositories, the document formats, the scope of the search, the nature of the reports and the speed of the processing. However, the software can greatly augment human detection of plagiarism (Bull, Collins, Coughlin & Sharpe, 2001).

The conception of plagiarism, detection of it, and the pursuit of academic misconduct cases is contentious because universities have been struggling to keep pace with changes in the environment. At the same time that Internet based sources have grown, more students are attending university than ever before. The student population is more culturally diverse with an increased proportion having English as a second language. Students are from countries that have different education cultures with different beliefs, values and attitudes regarding scholarship (Lahur, 2004). There are also increasing pressures for both students and teachers to do more in less time, and with less public funding. These pressures are conducive to a growth in plagiarism because it is an easy way for students and teachers to cope (Park, 2003).

Universities need to be careful about how they deal with plagiarism (Crisp, 2004). Detecting plagiarism on a large scale could be seen as success or failure, in much the same way as crime detection. Students who were eagerly encouraged into the university and then face academic misconduct and associated penalties may feel that they were not adequately educated or prepared. Teachers discovering academic misconduct may find themselves questioned by the institution and their peers about the way they teach and assess (Holloway, Joseph & Vuori, 2005). Teachers may also be subject to academic misconduct based on plagiarism (O'Keefe, 2007). The media could easily take a case out of context resulting in damage to a university's reputation. In extreme cases, badly handled plagiarism cases could become ongoing legal disputes with financial and reputation liabilities.

Hence whilst there is software available for detecting plagiarism, use of it is in the context of a social system where perceptions are shaped by past attitudes, beliefs and practices, and the norms, policies and procedures of the institution. Within this context, students undertake their learning and teachers are responsible for the assessment of that learning. It is thus of interest to investigate how the users (teachers) of plagiarism detection software perceive its effectiveness, and how the persons directly affected (students) perceive its use.

The research question is thus stated as:

What are student and staff perceptions of the effectiveness of plagiarism detection
software in detecting and preventing plagiarism in a university?

Having established the research question and the rationale for the research, the next section is a review of the literature. This includes an initial research model based around five themes. The detail of the mixed methods research design is then described. The findings from the interviews and questionnaire are discussed. The conclusion includes limitations and recommendations for practice and further research.

Literature review

Research model

To help guide the research, a research model was adapted from the literature (Figure 1). The original model presented by Vuori, Joseph and Gururajan (2004) consisted of three core concepts of education, detection, and prosecution. The model recognises that detection and prosecution is part of an interacting system that includes education. The education aspect is a natural fit within an education institution and helps address concerns that detection and prosecution without education is unfair to both students and staff.

In addition to the three core concepts, further concepts were added to represent key areas of concern. The definition of plagiarism was included as an understanding between students, teachers and the university which is central to managing plagiarism. Furthermore the education, detection, and prosecution processes take place in the wider context of the university and have impacts on the teaching staff and on the students. The focal concept was the effectiveness of the software (direct use by teachers on student assignments and also perceived by students), which was further elaborated by incorporating literature on information systems effectiveness.

Figure 1: Research model: Perceptions of plagiarism detection software
(adapted from Vuori, Joseph & Gururajan, 2004)

Types of plagiarism

Plagiarism was defined for this research as "the use of another's work without proper acknowledgement or permission". Within this definition there are a variety of types which have been described by Park (2004, p.293) and are listed Table 1. Universities generally regard poor referencing as an educational issue, whereas intentional cheating is dealt with as academic misconduct with penalties. Current software usually detects similar passages of text, the sources, and provides an indicator of the amount of plagiarism. It is left up to the teacher, the students and the university to establish the type of plagiarism, and the severity and the nature of follow up education and/or penalties. Thus the software and its detection effectiveness is just one part of the process.

Table 1: Types of plagiarism (Park, 2004, p. 293)

Collusion, where a piece of work prepared by a group is represented as if it were the student's own.

Commission or use of work by the student which is not his/her own and representing it as if it were. Examples include:
i. Purchase of a paper from a commercial service, including Internet sites, whether pre-written work or specially prepared for the student concerned.

ii. Submission of a paper written by another person.

Duplication of the same or almost identical work for more than one course.

Copying or paraphrasing a paper from a source text (in manuscript, printed or electronic form) without appropriate acknowledgement.

Submission of another student's work, with or without that student's knowledge or consent.

Plagiarism detection software

A key issue in detecting plagiarism is determining the source of the plagiarism. Software generally assumes an electronic original or copy and then searches for matches across a repository, e.g. the Internet. The advantage for the teacher is that the search and reporting is automated so time is saved in determining the plagiarism source and presenting the results. The commercially available software differ in the nature of the repository searched, the software use, and the purchase costs. Bull, Collins, Coughlin and Sharpe (2001) provide a review of five detection softwares including EVE2 (Essay Verification Engine) and Turnitin.

The software used in this study was EVE2. This is an example of a class of software that is tuned to detecting plagiarism of Internet sources, based on matching text. Curtin Business School at Curtin University of Technology was prepared to trial this software because it did not infringe copyright of student work, was relatively cheap, and had been used by some staff prior to the formal trial. After a license (single or site) has been purchased the software is installed on a teacher's personal computer. The student work is made available to the software on the teacher's computer and it then searches the Internet for matching text and reports the sources and the percentage of matching text.

Another software contender was Turnitin, which is based upon an application service provider. Here the student work is submitted to a website run by the service provider. The software does a similar job to EVE2 but, in addition, checks for matching text against a database of past student work, and can check for collusion within a student cohort. Whilst the sources checked appear more comprehensive than EVE2 there is also a greater cost involved (Bull et al., 2001). At the time of the trial deliberations, the University was concerned that the sending of student work to a third party might infringe the student's copyright. Hence in 2004 EVE2 was favoured by the Curtin Business School over Turnitin. It is noted that in 2007 the copyright issues appear to have been resolved. The University has trialled Turnitin, and is planning a wider implementation in 2008 (Yeo & Williams, 2008).

One limitation of the plagiarism detection software above is that the assumption of electronic sources means that copying of work from off line sources, for example books, journals, and assignments that are not electronic, goes undetected. Furthermore there are many databases on the Internet that require registration and/or subscription, and thus are not accessible by the detection software, but may be accessible to the plagiarist. A clever alternative detection process is based on writing style, noted as a trigger for manual detection by academics (Bull et al., 2001), and implemented in the software GPSD (Glatt Plagiarism Self-Detection). A quiz is presented to the student where every fifth word has been removed from a sample taken from the student paper. If the paper has been plagiarised then the student will struggle to use the same word to complete the prose. If the percentage of incorrect words reaches 50% then there is a likely case of plagiarism.

Effectiveness

Effectiveness of software in the context of an information system is a multidimensional concept that includes user perceptions of the usefulness and ease of use. Perceived usefulness is defined as "the degree to which a person believes that using a particular system would enhance his or her job performance" (Davis, 1989, p. 320). Perceived ease of use is defined as "the degree to which a person believes that using a particular system would be free of effort" (Davis, 1989, p. 320). Usability is a related concept associated with user interface and interactive product design and includes effectiveness, efficiency (e.g. time taken), safety (e.g. number of errors), utility (e.g. functionality), learnability (e.g. ease of learning) and memorability (e.g. ease of remembering) (Preece, Rogers & Sharp, 2002).

Whilst users of software for business purposes within an organisation are concerned primarily with achievement of purpose, there is also a broader sense of the user experience that includes concepts of satisfaction, enjoyment, reward, and motivation (Preece, Rogers & Sharp, 2002). Within the study reported here the primary users were teaching staff who used the EVE2 software during the trial. Semi-structured interviews conducted with the staff included a number of questions concerned with the perceived effectiveness, efficiency, and ease of use of the software. It was however important to gauge general opinions about plagiarism and detection software, hence the effectiveness questions were not as comprehensive or detailed as a formal usability study might demand.

Past research on trials of plagiarism detection software and its effectiveness

A difficult dilemma for educational institutions to deal with is that in the short term evidence of plagiarism produced through active detection can receive negative media coverage. On the other hand, allowing plagiarism to go unchecked in the long term affects academic quality, and produces long term negative impacts. It is thus a sensitive issue to divulge information publicly about plagiarism detection.

Nevertheless, universities have conducted trials and are reported to be actively using plagiarism detection software. One issue, identified by staff in a trial of Turnitin at Deakin University, is a potential increase in workload due to time required to confirm electronically detected cases for academic misconduct purposes. Furthermore, there was concern over whether the detection process was to be used primarily for punishment of students or for education. With regard to the software, the failure to distinguish between correctly referenced work and plagiarised work was seen as a weakness. Additionally, there was a lack of integration with existing e-learning software being used by the university (Sutherland-Smith & Carr, 2004).

A campus-wide implementation of Turnitin at Newcastle University in 2004 also revealed workload to be an issue. Additionally, it was noted that the detection software was not perfect although there was no indication of the failure rate. The implementation over a range of faculties also revealed that different disciplines have differing scholarship requirements and thus use of the software would have to be contextualised. A survey indicated strong support for the use of the software balanced with concern about the difficulties students had in it. (Eckersley, 2004).

A study at the University of Sydney using Turnitin collected both student and staff perceptions. Postgraduate students raised concerns about the compromising of their intellectual property. Staff were concerned about the workload involved in manually confirming electronically detected copying. Staff were however positive about the deterrent potential of using detection software (Savage, 2004). Mulcahy and Goodacre (2004) reported similar findings from the University of Tasmania. A focus group of students reported positive attitudes toward use of detection software. Staff also felt detection software could be useful as one tool amongst a number of strategies. There was however concern from staff with the workload of reviewing software reports to confirm plagiarism.

Consistent with the sensitivity of universities to negative publicity, it is difficult to get public statistics on the level of plagiarism. Alam (2004) reported that 57% of 99 postgraduate students surveyed admitted plagiarising at least once. A study of 1925 essays from six Victorian universities in 2002 using Turnitin software indicated that 14% of student essays had unacceptable levels of plagiarism (O'Connor, 2003). Definitions and measurement can also vary with context so it is difficult to combine results. The softwares usually report the percentage of total text that is the same as some source. EVE2 for example reports the percentage and issues a default warning if it is over 15%. It is then up to the human evaluators to determine if the text is not attributed, and if the scale is great enough to be classified as plagiarism.

Method

A mixed methods approach was adopted to deal with the richness of the context of plagiarism. Interviews were used to allow stakeholders to express their views about the social context and provide a deep response where the interviewer could interact with the interviewees and get deeper feedback on items. Questionnaires with closed Likert scale items and some open ended questions were used to get indicators from the wider group of students, whose assignments were subject to the detection software.

The interviews were held in October 2004 and were targeted at a small group of lecturers who had volunteered for a trial of the plagiarism detection software with the units (subjects) they coordinated for that semester. These persons were the primary users of the software and were also responsible for pursuing cases of academic misconduct that involved plagiarism. The questionnaires were targeted at the large body of students who were enrolled in the lecturers' units and were thus subject to having their assignments tested by the plagiarism detection software. To supplement the questionnaires and provide a deeper response, a small number of students were also interviewed.

The structure for the research instruments was based around the research model illustrated in Figure 1 and described in the previous section. The major themes were detailed into questions that made up the interview and questionnaire protocols. The instruments were pilot tested and refined based on feedback. The subjects for the interviews and the questionnaires were selected as follows. For the interviews the researcher approached the ten staff who had volunteered to trial the plagiarism detection software. Of these six agreed to be interviewed. Hour long interviews were held with these staff. The interviews were tape recorded and transcribed into Microsoft Word documents. The researcher then analysed the text looking for similarities and differences across the respondents.

For the questionnaires, the researcher gained permission to attend the lecturers' lecture classes and administered the questionnaires at the session, taking about 15 minutes to collect the data. This resulted in a good response rate amongst the students present, because the researcher was present during the data collection. Across the classes attended, a total of 171 completed questionnaires were collected. To further supplement the questionnaire data, six interviews were arranged with students who were identified from the questionnaires as being interested in further follow up. Again the interviews were tape recorded, transcribed and analysed for similarities and differences.

Results and discussion

The data from the interviews and questionnaires was summarised around the five themes of the research model presented in Figure 1, namely definition of plagiarism; University context; student impact; teaching staff impact; and effectiveness of the plagiarism detection software. Within this paper we decided to concentrate on the findings associated with effectiveness of the plagiarism detection software.

The quoting of comments from interviews used the following conventions, with pseudonyms used to protect student and staff confidentiality. Students were described by a first name only, e.g. Janey, whereas teachers were described by a title and a last name, e.g. Ms Hoover. Line numbers for the interview transcripts were included to facilitate data tracking.

Student perceptions of the effectiveness of plagiarism detection software

One reason for using plagiarism detection software on student assignments was to change student attitudes and behaviour so they would improve their scholarship, not plagiarise through ignorance, and not deliberately plagiarise. The students were not hands on users of the software, however they could experience consequences of being caught plagiarising, through academic misconduct procedures including penalties based on severity. It was thus important to gain feedback on student perceptions of the use of the software because their attitudes and behaviours would influence the success or otherwise of its implementation.

A subset of the questionnaire data collected from 171 students involved in the trial is reported. The statistical results including means, statistical significance, and percentages, are provided in Table 2, sorted in descending order of mean response. The summary indication is that students perceive that plagiarism is an important issue, PDS (plagiarism detection software) makes it easier for lecturers, it is fair to use PDS, they support its use, and it will have some effect in preventing plagiarism.

Furthermore there was no strong evidence that students felt unfairly targeted, or thought it would lead to distrust, or that it created unnecessary work for them. It is noted however that there were mixed views on these three items, with about one third agreeing, one third neutral, and one third disagreeing. It can be seen that "Importance" evoked the strongest response. Further description revealed that 77% of the 171 students surveyed agreed that plagiarism was an important issue for the University to deal with. This statistic alone provides support for some organised efforts to deal with it.

Table 2: Statistical summary of 171 student responses on a 5 point Likert, scale sorted by mean

Item Questionnaire statement Mean Std
dev Stat.
sig* Agree
(5,4) Neutral
(3) Disagree
(2,1)

Importance Plagiarism is an important issue for the University to deal with 3.9 0.9 Yes 77% 15% 5%

Ease for lecturers PDS makes it easier for lecturers to identify plagiarism 3.8 0.9 Yes 66% 26% 6%

Fairness It is fair for the University to use PDS on student assignments 3.4 1.0 Yes 53% 32% 8%

Prevents plagiarism Using PDS at University prevents student plagiarism 3.3 1.0 Yes 48% 34% 15%

Support I support the use of PDS at University 3.3 1.0 Yes 45% 40% 12%

Effectiveness PDS is effective in preventing plagiarism 3.2 0.9 Yes 44% 37% 13%

Unfairly targeted Students are unfairly targeted as a result of using PDS at University 3.1 1.2 No 34% 39% 25%

Distrust Using PDS will create distrust between lecturers and students 3.0 1.2 No 28% 36% 33%

Unnecessary work Students do unnecessary work as a result of the use of PDS 2.9 1.2 No 33% 33% 31%

* P-value < 0.05 Based on a t-test with degrees of freedom = 170 of:
H_o: population mean = 3 "not enough evidence to claim that population agrees with the statement"
H_a: population mean > 3 "population agrees with the statement"

Most students (53%) believe it is fair for the university to use PDS on student assignments, and 66% believe PDS will make it easier for lecturers to identify plagiarism. With regard to support by the students for the use of PDS software 45% agreed, 40% were neutral, and only 12% disagreed. So the majority agree, but a similar number remain to be persuaded that it is a good thing.

In addition to the statistical results, qualitative responses from questionnaires and interviews supported the above results and provided further detail of the variety of student views. It is not possible in this paper to provide such detail, however perhaps the most evocative quote was:

My degree would be worthless if I could cheat and get through! People might as well get a degree from a Weeties box. Student (questionnaire, Q.8)

Staff perceptions of the effectiveness of plagiarism detection software

The teaching staff in the trial were direct users of the software and thus the interviews were directed toward their perceptions of the software features and functions, and the limitations and benefits. The transcripts of the interviews were analysed for similarities and differences, and quotations were used to support the findings. The following discussion includes perceptions of the search capability, the detection process, identification of plagiarism, usefulness, user friendliness, accuracy, and limitations and benefits.

Search capability

Findings from the interviews with teaching staff confirmed that most staff members were aware of the general searching ability of EVE2 although there were a few uncertainties about its scope and the range of online information it could search.

Roughly it's searching the web, the Internet... only the information which is available online for example, if by any chance they take a database or a journal from the university, definitely that one it will not check it there because that one is located in one area and that is closed. While for example, if that journal is online definitely it will check it. Ms McConnell (line 207-220)

Several factors were brought to attention from the interviews as teaching staff described EVE2's searching scope. It was seen that EVE2 was only able to detect from text based online sources on the Internet and was restricted from searching private databases. Most teaching staff however were unsure of the searching potential of EVE2 indicating that staff members only had a basic understanding of the software.

It is difficult to really know the search capabilities and limitations of any plagiarism detection software without extensive testing and comparison. Vendors tend to verge on overselling the capabilities to ensure sales. Naïve users are probably more likely to overestimate the capabilities. This may not be a bad thing if it causes students and staff to take more care to avoid plagiarism.

Detection process

Staff members explained that student assignments first had to be in electronic format. This included other format requisites such as assignments having to be in text form, for example in a Microsoft Word document that did not contain images or graphics. Folders or directories were then created to store the assignments. For example, folder names were chosen to correspond with a number allocated by the staff member such as 'Lab 2'. The assignments were then loaded into the folder for the next stage of the detection process.

I just get students to submit assignments... put an electronic copy in the digital drop box [in Blackboard]... I just grab the copy from a Word document format from the digital drop box... Ms Hoover (line 181-182)
I asked them [students] to save the documents as [a] text file... just only text, no images, no nothing... I will be creating a folder and each folder will have the name of the lab... I'll copy the assignments... I'll put in a data folder, and after that I'll open the EVE software... Ms McConnell (line 232-249)

Staff members also mentioned additional procedures that they carried out. For instance, the naming conventions for student assignments were attended to as one staff member described:

...[I] didn't anticipate that students will all use generic names [for assignments] so what I had to do was to rename all those, the names, that takes quite a while... Surely if we [teaching staff] have fifty pieces of work then you'd be sitting there changing them or renaming them. Ms Krabappel (line 131-154)

Thus, the staff member considered the naming convention to be an important factor prior to the detection process. Another staff member who collected assignments on floppy disks commented that the disks required scanning for viruses:

...they will have it on a disk, first of all I would need to scan the disk [to see] if there's a virus... Ms McConnell (line 236-237)

Teaching staff needed to be aware of procedures to ensure that the plagiarism detection software ran efficiently. This included the formatting of assignments such as text based and the way in which they are to be submitted, storage options, naming conventions, and virus checks.

The next stage of the process was to initiate the detection procedure in EVE2. Teaching staff described pointing the software to selected folders where assignments were stored then starting EVE2's search function. Some staff members also stated that they left the software to run whilst out of the office. After this search process had completed, EVE2 compiled a report.

...I fire up EVE and I point EVE to that directory... I ran it on a batch of about seven or eight assignments and I actually ran it overnight... Mr Skinner (line 188-191)

Consequently, teaching staff explained that they had to examine reports provided by EVE2 for further investigation into whether plagiarism had occurred. Staff members added that the reports included a percentage level of suspected plagiarism and highlighted sections.

...I'd look at the reports that it provides... and in it are the main things it underlines or points out in red, I think, the bits that seem to be potentially plagiarised... it gives a percentage... you'd look to see if the percentage is close to a hundred percent (100%) then you're pretty concerned about it. Mr Skinner (line 197-205)
I get the report, I print out the report, keep it as a record... then I basically go through the report, have a look at it in detail... belligerent obvious uses of plagiarism so where we've got really high percentages. Mr Largo (line 118-133)

It was noted that the semi-automated detection was only the start of a potentially much longer and more onerous manual process which was the University's academic misconduct procedure. The logistics and outcomes of this procedure could be seen as a disincentive for individual staff to be overly diligent about detecting and prosecuting plagiarism in a formal manner. Furthermore, having known academic misconduct cases in a unit could be viewed either as a negative or positive for a staff member's reputation, depending on the comparison with other staff members and the prevailing politics in the institution.

Identification of plagiarism

Using screenshots of EVE2's results screens and an extract of a plagiarism report as prompts during interviews, staff expressed similar methods of identifying plagiarism.

Figure 2: EVE2 results screen

Figure 2 is EVE2's results screen where a percentage of plagiarism is displayed from a student's assignment. From interview findings, all teaching staff commented on using the percentage indicated by EVE2 as an estimate of the level of suspected plagiarism in student assessments. The software issues a guideline that anything above 15% warrants investigation. Staff noted this guideline but also varied in what they would investigate using 15%, 40%, and 80% as indicators. Given an ordered list, staff could work through the assignments from the highest percentage. With experience staff would find that the higher percentages were more likely to reveal serious plagiarism, whereas percentages closer to 15% may just be overuse of quotations.

Normally we go by... the percentage of the document... if it's below fifteen percent (15%), it would be pretty low so normally we tend to just ignore anything fifteen, twenty... I normally just check if it's from forty percent (40%) upwards... Ms Hoover (line 204-209)

Figure 3 is EVE2's detailed results screen which provides a list of web sites from where matching text was found. Teaching staff explained that they used this information as a second procedure in identifying plagiarism whereby they would check for plagiarism by selecting from the given list of web sites.

...I'll check the links [after looking at the percentage]... and see if I can identify anything further. Ms Hoover (line 220-222)
...I then go and have a look at the websites myself [after looking at the percentage] and I look at the matching [text], I use also the launch sites on my browser [to] have a look at them and see what they're about. Mr Largo (line 146-150)

Figure 3: EVE2 detailed results screen

Despite having these functions in identifying plagiarism, teaching staff were still concerned with the accuracy of the software and the generated percentages. Staff members remarked that the given percentage could not be taken as conclusive as they were unsure of how the software calculated the percentage and its trustworthiness.

...after that [looking at the percentage result] I'll start looking on the file [assignment] itself... not just only on these results, definitely not... because sometimes when I put my name on it [EVE2], my name it will be highlighted with red because it's copied from somewhere... So, I don't think ...that number [percentage result] is standing for something for me... Ms McConnell (line 273-289)

It was noted that there was no School or University guidelines for a benchmark percentage. The 15% was the guideline provided within the software. Furthermore the percentage was just seen as an indicator that the assignment warranted further investigation.

Usefulness

It was found that teaching staff held similar views of EVE2's usefulness. Most stated that it was fairly useful in detecting plagiarism, and in matching content from the Internet with student assignments. However, the software still required users to conduct further investigation to correctly identify that plagiarism had occurred. Consequently, whilst staff agreed that EVE2 was quite useful, they could not rely on it completely.

...at least it [EVE2] will tell you exactly which one is... direct copy and paste but... on the other hand, you need to check by yourself... I cannot depend on it [EVE2] one hundred percent (100%), definitely not. Ms McConnell (line 377-383)
...it was to some extent (useful)... because we're only checking on the internet... In terms of books obviously it can't detect that sort so I still have to do my own manual detection... so if it's just searching the Internet it serves its purpose. Ms Krabappel (line 245-253)
At the moment it's useful as a tool for helping with finding evidence of plagiarism but... you still have to investigate further. Ms Hoover (line 282-288)

User friendliness

Several teaching staff commented that EVE2 was fairly easy to use as they did not encounter any problems. The software was reasonably straightforward:

...it [EVE2] seems to be fairly logical so you... load up the assignments, it sets off doing its detection... you can set some levels of detection... you can say stop searching once you reach a certain level of plagiarism... there's not a thousand functions, it's just a few functions so... you don't get lost... Mr Skinner (line 495-507)

Other aspects of the software were mentioned by teaching staff to be user friendly. For example, its simple screen design and compatibility with Microsoft Windows which made the plagiarism detection reports easy to read.

The interface, button design, colour scheme, readability on the report, font size, font style. Mr Largo (line 270-271)
...it's [EVE2's] text report, is still screened to [Microsoft] Windows. It looks clear to me, not fancy but clear. Mr Bergstrom (line 259-261)

Lastly, one staff member stated that the functionality of EVE2 in evaluating student work to matched content from the Internet was also user-friendly. For instance, the software provided an automated feature to do so in a button which the user simply entered to initiate:

...this screen is good... if you need to confirm, [EVE2] can show it to you... if I click on that one [Show Comparison button], [it] will compare between that and the website and the student report and [it] will tell you exactly from where [it] grabbed this information. Ms McConnell (line 426-433)

Conversely, one staff member had experienced problems when viewing web sites of suspected sources within EVE2.

...I've maybe got into difficulty... I've set it [EVE2] off to do the searching and then it comes up with some websites and for whatever reasons maybe I get bumped out when I look at a site so I launch something in my web browser (instead)... Mr Skinner (line 515-520)

Additionally, despite staff members describing EVE2 as straightforward to use, it was mentioned that it took some time for users to go through the software's functions.

...at the beginning it takes a lot of time... in order to remember all these functions in it... Ms McConnell (line 416-420)

Another staff member also thought that the software's interface still required work to make it more attractive.

I think they could probably do a bit more with the user interface [of EVE], at the moment it's fairly basic... the screen is not all that attractive... make it a little bit more... appealing. Ms Krabappel (line 277-285)

Accuracy

Plagiarism detection software must be able to detect text within assessments that has been copied from other sources. The accuracy of this detection is an important factor in its effectiveness. Accuracy is one of the key criteria in the evaluation of plagiarism detection software as it measures the precision of the software in detecting plagiarism. Moreover, the software's ability to identify obvious cases of plagiarism is a main concern.

Using a screenshot of EVE2's essay / web page comparison result screens (Figures 4 and 5) as prompts during interviews, teaching staff evaluated the accuracy of the software. In particular, teaching staff discussed EVE2's inability to differentiate between different types of copying. These types include direct copy with no citation (strong plagiarism), direct copy with citation (maybe appropriate), and poor paraphrasing with citation (maybe appropriate) or without citation (plagiarism). EVE2 would only report matching text. However, as a first level of detection, compared to relying solely on manual means, this would seem to be a reasonable start.

Figure 4: EVE2 essay / web page comparison screen 1

Findings from the interviews revealed that staff members did not consider that EVE2 was entirely accurate in being able to differentiate between citations, as it did not distinguish between cited and non-referenced work.

...it [EVE2] doesn't discern that quality of where it's been actually cited correctly... Mr Skinner (line 565-567)
... it [EVE2] only counts the words that occur in the same two documents regardless whether those words occurred in the context that's being paraphrased or in the context of the direct quote without a reference. Ms Hoover (line 344-349)

Although it was conclusive amongst staff members that EVE2 was not accurate in making this distinction, they added that the software was accurate in doing its job of identifying suspected plagiarism. They regarded this as its primary purpose and the verification of correctly cited work consequently required their intervention.

...it's [EVE2] accurate in what it is doing, after all it's a computer software... In terms of checking, as users, as people, as humans, we still have to check to see the accuracy of that... Ms Krabappel (line 298-304)
...all you can do with EVE2 is getting a link of the sources in order to check it yourself because EVE2 can't actually differentiate... proper paraphrasing... and blatant plagiarism... Ms Hoover (line 335-339)

Figure 5: EVE2 essay / web page comparison screen 2

Benefits

The benefits reported by staff included the software's ability to help detect suspected cases of plagiarism and produce reports, and the service that it provided which aids teaching staff in their workload.

One of the benefits of EVE2 which teaching staff commented on was that it was useful in its goal of detecting suspected plagiarism within student work. As such, one staff member remarked how the software had already demonstrated its effectiveness in detecting a plagiarist.

...when I have a ninety eight percent (98%) [detected plagiarism result] I thought 'yes, EVE has done its work, served its purpose' and to me it's actually picked up what I want it to pick up... Ms Krabappel (line 344-347)

Additionally, staff members stated that the software provided them with evidence of plagiarism and detection results.

...it [EVE2] gives you all the sources [Internet websites] in one go. Ms Hoover (line 383-384)
...at least you have like evidence... Ms McConnell (line 485-486)
...the report's pretty good. It [EVE2] deals with the text, it highlights it, underlined in red... so that provides a pretty decent sort of audit trail and it gives a list of the websites... those two things together... would be enough to sustain proof if you like for an Academic Misconduct case... or to explain to a student... this is poor scholarship... Mr Skinner (line 624-631)

To add to the services provided by EVE2, a staff member stated that the software was able to search many sources at once which was not possible for teaching staff to do manually.

...if you selectively use EVE2 software to check certain suspicious cases, they can end up saving the lecturer's time I think because to manually check each and every source that takes a long time... you can't go through as many sources as EVE2... Mr Bergstrom (line 355-361)

Limitations

Limitations included the software's constraints in detecting material from sources which were not on the internet, its level of accuracy, the time taken for detection, and the extra workload in pursuing academic misconduct.

A majority of teaching staff pointed out EVE2's limitation in only detecting sources that were on the Internet. This restriction meant that teaching staff were unable to check against other materials such as textbooks, CDs, past student assignments, and restricted databases. For this reason, teaching staff regarded this as EVE2's main limitation as it did not provide a foolproof means of detecting plagiarism from all resources that are available to students. To add to this, teaching staff were concerned that the software could not detect plagiarism in the case of student collusion.

...even though EVE2 software is able to identify or check a large amount of online sources... there's still a lot of sources that's being left out... text book... CD-Rom, a lot of database things, if the students use materials from there, certainly EVE2 software couldn't detect for plagiarism. Mr Bergstrom (line 323-331)
...we [teaching staff] haven't been able to set it [EVE2] up to check for collusion... it's only looking at websites... (not) anything that's in the off line area i.e. books... Mr Skinner (line 592-599)

The accuracy of EVE2 was also mentioned as a limitation as EVE2 was unable to distinguish between correct and incorrect citations in student assessments. Furthermore, one staff member explained that the software was also restricted as it was not able to detect student work which had been rephrased;

...accuracy in terms of differentiating between citations and direct plagiarism... the degree of sophistication in terms of detecting identical ideas that has just been slightly paraphrased. Ms Hoover (line 355-359)

The time taken by EVE2 to run its detection process was identified as a limitation by staff members. The detection process for a single assignment ranged from a few minutes to thirty minutes which was considered by one staff member to be too long, specifically if application was to be scaled up to hundreds or thousands of assignments.

...it [EVE2] just takes too long to retrieve the whole report. Mr Bergstrom (line 243-244)
...it's [EVE2] only looking at the web... it's rather slow... if we're going to use it for the whole Department or for really big units, we might have a thousand assignments... so that's where you really want efficiency. Mr Skinner (line 398-434)

Whilst EVE2 was seen as useful in detecting plagiarism cases, some of which may have been overlooked previously, there were concerns about extra workload involved with pursuing academic misconduct. Once a case had been detected, the individual academic was responsible for documenting the evidence, filling out formal alleged plagiarism documentation and pursuing the case through the University academic misconduct procedures. In the case of serious plagiarism, this was potentially a lengthy process over several weeks involving numerous formal meetings with students and disciplinary panels, and considerable scrutiny of the evidence. Minor plagiarism cases would also require documentation and reporting. This extra workload and cost, ultimately borne by the institution, but primarily by the diligent academic, signals that prevention of plagiarism is preferred to expensive academic misconduct procedures.

So, EVE helps with the detection and the reporting but in doing that, for an individual lecturer it means then if they find ...a number of plagiarism cases then they may be up for several hours of ... the academic misconduct process in order to deal with it. Mr Skinner (line 694-699)

Conclusion

Perceptions of students and staff regarding plagiarism detection software were pursued via a mixed methods approach, using a research model, during a software trial. The summary indication based on quantitative survey results (Table 2) was that students perceive that: plagiarism is an important issue; detection software makes it easier for lecturers; it is fair to use detection software; students support its use; and it will have some effect in preventing plagiarism.

However, based on qualitative results from the survey and interviews, concerns for students included being caught for unintentional plagiarism, in teaching staff placing too much emphasis on detection results above student ability, and in the accuracy of the software at detecting plagiarism. The concerns for teaching staff, based on interviews, were in the time taken for the detection process, the restriction to publicly based Internet sources and direct copying, and the extra workload involved with pursuing academic misconduct.

The mixed methods approach had the benefits of combining richness, depth and generation of concepts from interviews with the broad statistical confirmation from the questionnaire survey. The approach is recommended for consideration by other researchers. The research model was useful for providing a framework of concepts for the study and to generate further sub-concepts.

The study was limited by the research question pursued (perceptions of students and staff), the research design (mixed methods), the respondents sampled (students and staff in the trial), the context of the University and the software trial (EVE2 in IS courses), and the researcher's subjectivity in the conduct (interviews and analysis) and reporting of the research.

Recommendations for universities proceeding with plagiarism detection software, based on this study and other authors, include:

Make it part of a larger, multi-faceted strategy (Alam, 2004; Carroll, 2002; Carroll & Appleton, 2001 p. 42; Macdonald & Carroll, 2006) that includes education of students and staff (software, support for good scholarship (McGowan, 2005), and communication skills), and policies for dealing with academic misconduct (Mulcahy & Goodacre, 2004);
Test and trial software to verify accuracy, reliability, strengths and limitations (Bull, Collins, Coughlin & Sharpe 2001);
Proceed incrementally with learning feedback to improve strategies and tactics;
Give students (and staff) access to the software for self testing and self education;
Consider how to scale from local lecturer-led testing to institutional level testing;
Do not underestimate the workload and the incentives for staff involved with processing assignments, confirming detection (Mulcahy & Goodacre, 2004), and pursuing academic misconduct.

Future research includes follow up studies within the institution, comparisons with implementations elsewhere, and further development of the research model.

References

Alam, L. S. (2004). Is plagiarism more prevalent in some form of assessment than others? In Beyond the comfort zone: Proceedings ASCILITE 2004 (pp. 48-57). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/alam.html

Bull, J., Collins, C., Coughlin, E. & Sharpe, D. (2001). Technical review of plagiarism detection software report. [viewed 5 Feb 2008] http://turnitin.com/static/pdf/luton.pdf

Carroll, J. (2002). A handbook for deterring plagiarism in higher education. Oxford: Oxford Centre for Staff and Learning Development.

Carroll, J. & Appleton, J. (2001). Plagiarism: A good practice guide. [viewed 5 Feb 2008] http://www.jisc.ac.uk/uploaded_documents/brookes.pdf

Clough, P. (2000). Plagiarism in natural and programming languages: An overview of current tools and technologies. [viewed 20 May 2004] http://www.dcs.shef.ac.uk/%7Ecloughie/papers/Plagiarism.pdf

Crisp, G. T. (2004). Plagiarism and the reputation of the university: How to distribute effort between educating students on attribution and rigorous detection of cheating? [viewed 7 Feb 2008] http://www.auqa.edu.au/auqf/2004/program/papers/Crisp.pdf

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340.

Dawson, J. (2004). Plagiarism: what's really going on? In Seeking Educational Excellence. Proceedings of the 13th Annual Teaching Learning Forum, 9-10 February. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2004/dawson.html

Eckersley, C. (2004). Turnitin - a university-wide implementation. [viewed 16 May 2005; not found 13 Feb 2008] http://www.caval.edu.au/prs/eprod/pd/University%20of%20Newcastle.pdf

EVE2 Essay Verification Engine (undated). Eve Plagiarism Detection System. [viewed 20 June 2007] http://www.canexus.com/eve/index.shtml

Glatt Plagiarism Services (undated). [viewed 20 May 2004] http://www.plagiarism.com/

Hawley, C. S. (1984). The thieves of academe: Plagiarism in the university system. Improving College & University Teaching, 32(1), 35-39.

Holloway D., Joseph, R. & Vuori, T. (2005). Australian university responses to student plagiarism: Shooting the messenger? Paper presented at The Reflective Practitioner: Teaching and Learning Forum 2005. Perth: Murdoch University. [viewed 7 Feb 2008] http://lsn.curtin.edu.au/tlf/tlf2005/abstracts.html#holloway [abstract only]

Lahur, A.M. (2004). Plagiarism among Asian students at an Australian university offshore campus: Is it a cultural issue? A pilot study. Proceedings HERDSA 2004 Conference, Curtin Sarawak, 4-7 July. [viewed 6 Feb 2008] http://www.herdsa.org.au/conference2004/Contributions/NRPapers/A033-jt.pdf

Lancaster, T. & Culwin, F. (2004). Using freely available tools to produce a partially automated plagiarism detection process. In Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 520-529). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/lancaster.html

Macdonald, R. & Carroll, J. (2006). Plagiarism - a complex issue requiring a holistic institutional approach. Assessment and Evaluation in Higher Education, 31(2), 233-245.

McGowan, U. (2005). Plagiarism detection and prevention: Are we putting the cart before the horse? Proceedings HERDSA Conference 2005. [verified 14 Feb 2008] http://www.itl.usyd.edu.au/herdsa2005/pdf/refereed/paper_412.pdf

Mulcahy, S. & Goodacre, C. (2004). Opening Pandora's box of academic integrity: Using plagiarism detection software. In Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 688-696). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/mulcahy.html

O'Connor, S. (2003). Cheating and electronic plagiarism - scope, consequences and detection. [viewed 16 May 2005, verified 14 Feb 2008] http://www.caval.edu.au/assets/files/PDFs/About_us/Staff%20Publications/Cheating_
and_electronic_plagiarism_scope_consequences_and_detection_EDUCASUE_May_2003.pdf

O'Keefe, B. (2007). Work so nice, they use it twice. The Australian, Higher Education, 6 June, p.29.

Park, C. (2003). In other (people's) words: Plagiarism by university students - literature and lessons. Assessment & Evaluation in Higher Education, 28(5), 471-488.

Park, C. (2004). Rebels without a clause: Towards an institutional framework for dealing with plagiarism by students. Journal of Further and Higher Education, 28(3), 291-306.

Preece, J., Rogers, Y. & Sharp, H. (2002). Interaction design: Beyond human-computer interaction. John Wiley and Sons, Hoboken, NJ.

Savage, S. (2004). Staff and student responses to a trial of Turnitin plagiarism detection software. Proceedings of the Australian Universities Quality Forum 2004. [viewed 16 May 2005, verified 14 Feb 2008] http://www.auqa.edu.au/auqf/2004/program/papers/Savage.pdf

Sutherland-Smith, W. & Carr R. (2004). Issues and concerns in the implementation of Turnitin at Deakin University. [viewed 16 May 2005; not found 14 Feb 2008] http://www.caval.edu.au/prs/eprod/pd/Deakin%20University.pdf

The Concise Oxford Dictionary (1995). 9th Edition. Clarendon Press, Oxford.

Turnitin (undated). Turnitin digital assessment suite. [viewed 20 June 2007] http://www.turnitin.com/

Vuori, T., Joseph, R. & Gururajan, R. (2004). Proposing a model to address issues of plagiarism in Australian tertiary education. In Seeking Educational Excellence: Proceedings of the 13th Annual Teaching and Learning Forum, 9-10 February. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2004/vuori.html

Yeo, S. & Williams, M. (2008). Turnitin: A tool for teachers. Paper presented at Preparing for the graduate of 2015: 17th Annual Teaching Learning Forum, 30-31 January. Perth: Curtin University of Technology. [abstract only] http://otl.curtin.edu.au/tlf/tlf2008/abstracts.html#yeo2

Authors: Doug Atkinson and Sue Yeoh, School of Information Systems,
Curtin University of Technology, GPO Box U1987, Perth WA 6845, Australia
Email: d.atkinson@curtin.edu.au

Please cite as: Atkinson, D. & Yeoh, S. (2008). Student and staff perceptions of the effectiveness of plagiarism detection software. Australasian Journal of Educational Technology, 24(2), 222-240. http://www.ascilite.org.au/ajet/ajet24/atkinson.html

[ PDF version of this article ] [ AJET 24 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet24/atkinson.html
Created 22 Feb 2008. Last revised 22 Feb 2008.
Page access count

Item	Questionnaire statement	Mean	Std dev	Stat. sig*	Agree (5,4)	Neutral (3)	Disagree (2,1)
Importance	Plagiarism is an important issue for the University to deal with	3.9	0.9	Yes	77%	15%	5%
Ease for lecturers	PDS makes it easier for lecturers to identify plagiarism	3.8	0.9	Yes	66%	26%	6%
Fairness	It is fair for the University to use PDS on student assignments	3.4	1.0	Yes	53%	32%	8%
Prevents plagiarism	Using PDS at University prevents student plagiarism	3.3	1.0	Yes	48%	34%	15%
Support	I support the use of PDS at University	3.3	1.0	Yes	45%	40%	12%
Effectiveness	PDS is effective in preventing plagiarism	3.2	0.9	Yes	44%	37%	13%
Unfairly targeted	Students are unfairly targeted as a result of using PDS at University	3.1	1.2	No	34%	39%	25%
Distrust	Using PDS will create distrust between lecturers and students	3.0	1.2	No	28%	36%	33%
Unnecessary work	Students do unnecessary work as a result of the use of PDS	2.9	1.2	No	33%	33%	31%
* P-value < 0.05 Based on a t-test with degrees of freedom = 170 of: H_o: population mean = 3 "not enough evidence to claim that population agrees with the statement" H_a: population mean > 3 "population agrees with the statement"