|Australasian Journal of Educational Technology
2009, 25(4), 509-523.
Case study of a computer based examination system
Andrew Fluck, Darren Pullen and Colleen Harper
University of Tasmania
Electronic supported assessment or e-Assessment is a field of growing importance, but it has yet to make a significant impact in the Australian higher education sector (Byrnes & Ellis, 2006). Current computer based assessment models focus on the assessment of knowledge rather than deeper understandings, using multiple choice type questions, and blocking access to more sophisticated software tools. This study explored a new system based on a customised version of an open source live CD, based on Ubuntu which was used with three groups of pre-service teachers (N=270). Students had divided opinions about using computers or paper for their examinations, but prior exposure to computer based assessment was a highly significant factor for preferring the computer medium. Reflecting upon their experience, students found the noise of computer keyboards a distraction during the eExamination and preferred fewer on-screen windows. The new system allowed eExaminations to be taken securely on student owned laptop computers and was supervised by invigilators without specialist information technology skills. The system has been made available for other researchers to use at http://www.eExaminations.org/
As flexible and online learning mediated by ICT becomes more pervasive, there is a growing need for educators to consider modes of assessment using similar tools. The cost of assessment in higher education is the most rapidly growing component of tuition fees (Ricketts, Filmore, Lowry & Wilks, 2003), whilst open content shrinks the cost of tuition and learning materials (Wales & Baraniuk, 2008). However, the increasing discrepancy between teaching through blended or online delivery with a learning content management system, and assessing using pen and paper, is another reason to consider ways in which candidates can verify their achievement whilst using computers. In our experience online, computer based assessment is fraught with difficulties which are discussed in this report. Therefore the study focuses on a new proctored and offline, computer based assessment system.
Researchers have often measured the 'wrong' things, looking for improvements in traditional processes and knowledge instead of new reasoning and new knowledge which might emerge from the ICT use (p. 34).Many similar studies have noted the overall impact of ICT integration as additional motivation to learn deriving from the Hawthorne effect of novelty; or a skill set to be mastered in addition to the content knowledge addressed. In the USA a large scale study found even good software had little learning benefit (Dynarski, Roberto, Heaviside, Novak, Carey, Campuzano, Means, Murphy, Penuel, Javitz, Emery & Sussex, 2007). Furthermore, Cuban (2001) described the ineffective use of computers in classrooms. Such studies demonstrate computer use in support of a pre-existing curriculum is of contested effectiveness, and possibly inefficient. To assess the latter, researchers might legitimately inquire as to whether learning outcomes are achieved more rapidly with, or without, ICT. In any case, Australian secondary schools are about to receive a major innovation as the Australian Government's Digital Education Revolution will provide a computer for every student in Years 9-12 (Gillard, 2008, p.57). Universities therefore need to prepare for an influx of laptop-savvy students in 2-4 years time.
Online assessment is now commonplace in many in Australian universities, but Byrnes and Ellis concluded this is largely superficial for both formative and summative purposes (2006). Online assessment is mostly used for quizzes, forums and digital assignment drop boxes. In many cases online assessment is conducted using an institutional learning management system (LMS) such as BlackBoard, WebCT, or an in house product (Pullen & Cusack, 2007; 2008). Online assessments however, offer several advantages for the institution and the learner. These include:
... developments in assessment are advocating alternative and diverse assessment methods, including peer assessment, portfolio, reflective journaling, self-assessment, and performance-based assessment, which are deemed to be constructive, authentic, contextualized assessment, and to promote deep learning and skills development. (BC consulting, 2006, p.3)In relation to assessment more generally, all assessment needs to be "valid, reliable, fair and flexible" (Booth, Clayton, Hartcher, Hungar, Hyde & Wilson, 2003, p.8). This can be just as difficult to achieve online as on paper, since some students have a greater aptitude for particular question types (e.g. multiple choice questions). A common response is to mix question types in any given test. Assessment equity and quality is commonly also achieved through the context of an individual written examination.
Only six per cent of partially online assessable units had a final exam that could be taken online. Of the partially online assessable units without an online final exam, 83 per cent had a traditional pen and paper final exam, indicating that the use of final exams is quite widespread. Overall, 84 per cent of partially online assessable units had a final exam. (Byrnes & Ellis, 2006)Outside Australia, there appears to be a greater uptake of eAssessment in schools, with 38% of awarding bodies surveyed in England using some form of eAssessment to deliver up to 60% of their assessment programme (Chapman, 2006). This sends a strong message to Australian quality assurance bodies and Boards of Studies which may find eAssessment worth investigation.
LMS based online testing environments offer useful tools for conducting assessments of knowledge. Automated marking is feasible for multiple choice questions, and for short answer questions where key words are sought in the response. Assessment feedback requiring an understanding of an essay, etc, is so far not widespread and responses which include diagrams might be difficult to mark automatically using vision recognition systems. The challenges faced by online systems are those posed by Booth et al. (2003). Assessment of student knowledge and skills within a web browser window or delivered by bespoke assessment software (specifically crafted for a particular set of questions) provides a restricted environment which prevents the demonstration of abilities associated with the use of specialist software or a combination of applications.
To be fair, online systems need to authenticate the individual undertaking the assessment - some systems have gone so far as to take photographs at random intervals to assure this (Rönnberg, 2001). However, the camera may not necessarily be pointing at the person undertaking the responses, so this is not a foolproof method for identifying candidates. Another aspect of fairness is generally eliminating collaboration: since the computer is online, this is hard to implement except by locking out critical functionality during the assessment. For example, E-Tests developed by Ko and Cheng (2008) required an Iomega zip disc peripheral; all the data files were encrypted, and the program would only run on computers with a pre-registered network card. Such a system handles large numbers of students very well, but is restricted to simple question types. Such systems primarily facilitate automated marking, whilst we sought to provide a comprehensive environment to move all assessment types into a digital modality.
Fairness can be improved by adopting a proctored or supervised testing environment, where all students are watched by an examiner as they undertake the assessment. But how can the examiner prevent collusion if all candidates are using computers? Bluetooth, wireless networking, infra-red and mobile phone connections are all feasible communication channels and these are not easily blocked. Thus, if assessment, particularly of high stakes summative examinations, is to move into an ICT environment, some other technological and pedagogical approach is required. An approach which can exploit student owned equipment would be particularly suitable, since few institutions can deploy many hundreds of computers for the small fraction of the year devoted to formal assessment. The institution may then be able to handle small numbers of students without computers or with faulty equipment, much as a biro might be loaned a student experiencing pen difficulties.
The value of online assessment appeared compromised by security concerns leading to the possibility of unfair advantage for some candidates. Our study therefore explored the practical implementation of offline eExaminations, student responses to eExaminations and factors related to acceptance or rejection.
Six to eight weeks prior to the eExamination, students were given a free, practice CD. They were shown how to use it during a practice tutorial to ensure they could operate the software. During the tutorial, students were taught the rudiments of the Ubuntu operating system, the use of Open Office Writer for word processing and shown how to create drawings using The Gimp software.
The exam procedure involved putting the CD into a computer, then switching it on. The computer then booted (began to operate) using the Ubuntu-Linux operating system from the CD. An exam folder appeared on the desktop. The examination questions within the folder had been prepared with Microsoft Word, so students double clicked on the test document to open it with OpenOffice Writer. The first instruction in the rubric told the student to immediately re-save the test to the desktop with a new filename comprising their surname and ID number.
Students then completed the answers to each question, saving at appropriate intervals (using good computer procedures such as before making a major change or after every 300 words or so of text input). When finished, they saved the document and closed all application(s), without switching off the computer or logging out. In the event of equipment failure, students were to report this quietly to the test supervisor. A paper version of the test would then be made available, or students could restart using another computer. Access to the Internet or any other digital resources was not allowed. Digitally facilitated collusion was prevented in three ways:
Figure 1: Modified Reconstructor options screen
The option for computer based networking allows examiners to conduct the digital equivalent of open book examinations. However, local communications such as Bluetooth (which can interface to mobile phones) or unwired networking can be disabled separately. To reduce distracting computer sounds in an examination hall, the loudspeaker drivers can be eliminated. If students are using their own laptops, or even loan pool laptops, local hard disk drive access can be eliminated.
The five step creation process allows the examiner to upload and include a folder containing examination materials onto the CD. The folder can contain all the files necessary for the examination, including a word processed document (as in our example), video, application software, etc. The examiner can also specify an image file which becomes a desktop background. This image is a useful security feature: any student whose screen does not show this unique image must have booted their computer from another source. For non-technical examination invigilators, this provides a quick visual check to confirm each student is using the correct materials. Once a master CD had been prepared, copies were produced by a standard duplicator with printed instructions on the top surface at about 50c each. This is similar to the cost of printing an examination paper.
Responses indicated only 38% of survey respondents had previously taken a computer based exam, so this was a unique experience for the majority. The most highly cited examples of previous experience included taking a multiple choice health questionnaire on the university learning management system; an ICT competency assessment used as a diagnostic in the first year of the course (Training and Development Agency for Schools, 2008); and pre-employment online training and assessment associated with paid work to support their studies. The teaching team felt a high degree of responsibility for making the eExamination experience one which students would like to repeat. This was a leading edge cohort, whose opinions were likely to shape sentiment about the innovation across the institution. The survey analysis considered responses from three distinct groups of students as described in Table 1.
|Group A||Group B||Group C|
|Exam date||October 2007||October 2008||December 2008|
|Students using own laptops||0||6||0|
|Exam type||Single document (edited in Open Office Writer)||PDF for questions; Open Office Writer document for responses. Ogg Vorbis video, PowerPoint student work sample, Word curriculum document.||PDF for questions; Open Office Writer document for responses. Ogg Vorbis video (Vorbis.com, undated).|
|Proportion reporting technical difficulties in the eExamination||23%||56%||0%|
When giving feedback on their personal preparation for the eExamination, 78% of respondents had used the practice CD before the eExamination, and 71% had found it very or moderately useful.
However, students who had previously taken a computer based exam preferred this medium (63% of respondents) compared with 37% of first-timers. The one way ANOVA confirmed a very significant difference due to prior computer exam experience with F(2,227) = 8.683, p<.001 with an effect size of 0.621. Thus a first experience of eExaminations appears to stimulate a preference for computer based testing, and is more significant than any subsequent technical difficulties.
In 2007 the examination consisted of a single document containing questions with spaces under each for student responses. We had two computer 'lockups', but both candidates had been able to resume after a restart, losing none of their work. In designing the 2008 eExamination, we realised students might inadvertently edit the questions, so prepared these as a separate PDF file. In addition we included a stimulus scenario video and PowerPoint student work sample. Inadequate testing on the variety of computers failed to reveal the video playback did not silence internal computer speakers on some models despite using headphones, which disturbed other candidates. In addition, the version of Open Office crashed when a PowerPoint file was closed - not a good experience for students new to eExaminations! These factors accounted for many of the technical difficulties reported by Group B. Nearly one third of the students who reported other technical difficulties included the comment 'the system was slow to respond'. This is an effect of using a 'live' CD operating system since commands must be loaded from the optical media, and led to the development of the variation used for Group C entirely based on a USB drive. Other specific reports of technical difficulties included:
Figure 2: Example comments from students concerning the eExamination
Sixteen students in Group A (12.8%) commented on the high noise level from computer keyboards during the eExamination, with notes such as:
Everyone in one room typing is LOUD! It was hard to concentrate.We warned Groups B and C about the problem of noise, and no students from these groups commented on this aspect of the eExamination. However, the latter groups did express frustration about having to manipulate multiple windows: "It was frustrating going between several documents. It would be better if answers & question [were] in one document". We are looking at the possibility of putting future eExaminations in a single file with questions in a distinguishing colour and protected from editing.
The noise of the clicking [of keyboards] was very distracting. The sound makes you feel rushed.
Betty: I prefer the computer based one because for me, I don't have very good handwriting, especially when I am under pressure. My handwriting gets pretty horrible, and I enjoyed... Like, we had a Maths (examination) a couple of days ago, and we had quite small spaces to put our answers in. But with this one we could use as much space as we wanted, which was especially useful in the last question where we could do an extended answer and make the text box as large as we wanted. Which is very good. It would have been stressful if there had been technical issues, but mine just worked.In further discussion the group debated giving students a free choice of writing tool, but acknowledged this choice does not always exist 'in the real world'. The different pace of the eExamination for Group B was significant. One question asked candidates to view a short video and respond in a personal way about their future actions. Peter liked the video:
Tom: I am a bit of a Luddite. So I was quite... I quite enjoyed it. It was sort of different. Just having to switch between the two screens, and the video, stretching a bit. I did find it very noisy. I have a bit of a headache at the moment. So that was one down for it. Also, I can't type. So instantly, I felt at a disadvantage. While everybody else is using all fingers, I am counting on two. So those were the main pitfalls. I felt that I wasn't able to keep up, so I had to be a bit more concise in what I wrote. Because I couldn't write a lot. With a paper based exam, everyone is more on a level playing field, in regards to what you could write within a set time .... My handwriting is atrocious. (widespread laughter). So it's like well OK, I can see the benefits of being able to read something keyboard - wise, but yes...
Peter: it breaks up the exam. I found it good. ... The footage was good, and it was engaging. It went to two minutes, which I thought was perfect. 120 seconds was all we needed. OK, I thought, I am awake again. I thought it was good. For example in the math exam, unless you step up and walk out to go to the toilet, there are no breaks in the intensity. So it was great that there was that footage. And it was easy to access.Although this respondent was commenting on the change of pace in the eExamination, the video example illustrates the benefit of moving into the new medium. The research team was exhilarated by the opportunity afforded to move beyond print in formal assessments, opening the door to authentic testing of skills.
This student cohort was ambivalent about the introduction of computer based examinations. To facilitate adoption of this innovation we make some recommendations:
BC Consulting (2006). Effective on-line assessment: Workshop support materials. University of New South Wales: Sydney. [viewed 11 Aug 2009]
Booth, R., Clayton, B., Hartcher, R., Hungar, S., Hyde, P. & Wilson, P. (2003). The development of quality online assessment in vocational education and training, Volume 1. National Centre for Vocational Education Research. [viewed 11 Aug 2009] http://www.ncver.edu.au/research/proj/nr1F02_1.pdf
Byrnes, R. & Ellis, A. (2006). The prevalence and characteristics of online assessment in Australian universities. Australasian Journal of Educational Technology, 22(1), 104-125. http://www.ascilite.org.au/ajet/ajet22/byrnes.html
Chapman, G. (2006). Acceptance and usage of e-assessment for UK awarding bodies - a research study. Proceedings 10th CAA Conference 4-5 July, Loughborough University, England. [viewed 18 Feb 2008, verified 15 Aug 2009] http://www.caaconference.com/pastConferences/2006/proceedings/Chapman_G_c2.pdf
Cox, M., Abbott, C., Webb, M., Blakeley, B., Beauchamp, T. & Rhodes, V. (2003). ICT and attainment: A review of the research literature. DfES, London. [verified 12 Sep 2009]
Cuban, L. (2001). Oversold and underused: Computers in the classroom. Harvard University Press, Cambridge, Massachusetts.
DEST (Department of Education, Science and Training) (2006). Family-School Partnerships Project: A qualitative and quantitative study. Saulwick Muller Social Research: Canberra. [verified 15 Aug 2009] http://www.familyschool.org.au/pdf/muller.pdf
Downes, T., Fluck, A., Gibbons, P., Leonard, R., Matthews, C., Oliver, R., Vickers, M. & Williams, M. (2001). Making better connections. ACSA, ACCE, TEFA, University of Western Sydney. http://www.dest.gov.au/archive/schools/publications/2002/MBC.pdf [viewed 11 Feb 2009, verified 15 Aug 2009].
Dynarski, M., Roberto A., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Means, B., Murphy, R., Penuel, W., Javitz, H., Emery, D. & Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort. U.S. Department of Education, Institute of Education Sciences, Washington, DC. http://ies.ed.gov/ncee/pdf/20074005.pdf
Finger, G., Russell, G., Jamieson-Proctor, R. J. & Russell, N. (2007). Transforming learning with ICT: Making IT happen. Pearson Education Australia.
Fluck, A. (2007). Can tertiary e-Assessment change secondary school cultures? In Proceedings 11th CAA Conference, 10-11 July, Loughborough University, England. [viewed 2 Feb 2008] http://www.caaconference.com/pastConferences/2007/proceedings/Fluck%20A%20f2_formatted.pdf
Gillard, J. (2008). The importance of being digitally engaged. Information Age, Dec08/Jan09, 56-59. [verified 15 Aug 2009] http://www.acs.org.au/iage/ia200812-01.pdf
Gvozdenko, E. & Chambers, D. (2007) Beyond test accuracy: Benefits of measuring response time in computerised testing. Australasian Journal of Educational Technology, 23(4), 542-558. http://www.ascilite.org.au/ajet/ajet23/gvozdenko.html
Institute for Interactive Media and Learning (2007). Online examinations. University of Technology Sydney. http://www.iml.uts.edu.au/assessment/online/guides/0222.html [viewed 3 Feb 2008, verified 15 Aug 2009].
Jonsson, T., Loghmani, P. & Nadjm-Tehrani, S. (2002). Evaluation of an authentic examination system (AES) for programming courses. http://www.ida.liu.se/~snt/teaching/HGUR-sept02.pdf [viewed 17 Feb 2008].
Ko, C. & Cheng C. (2008) Flexible and secure computer-based assessment using a single zip disk. Computers & Education, 50(3), 915-926.
Mathews, J. (2006). Let's teach to the test. Washington Post, 20 February, Page A21.
Moursund, D. (2005). Introduction to information and communication technology in education. University of Oregon, USA. http://uoregon.edu/%7emoursund/Books/ICT/ICTBook.pdf [viewed 11 Aug 2009]
Pullen, D. & Cusack, B. (2007). Schooling without borders. In J. Sigafoos & V. Green (Eds), Technology and teaching (pp. 101-114). Nova Science Publishers.
Pullen, D. & Cusack, B. (2008). Content management systems: The potential for open education. Fact Sheet FS01. Australian College of Educators, Canberra.
Ricketts, C., Filmore, P., Lowry, R. & Wilks, S. (2003). How should we measure the costs of computer aided assessment? Proceedings 7th Computer Assisted Assessment Conference, Loughborough University, UK. [viewed 11 Aug 2009] http://hdl.handle.net/2134/1924
Robertson, M., Grady, N. B., Fluck, A. & Webb, I. L. (2006). Conversations toward effective implementation of information communication technologies in Australian schools. Journal of Educational Administration, 44(1), 71-85.
Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Simon and Schuster.
Rönnberg, K. (2001). Security in learning management systems - User authentication in online assessment. Master's Thesis: Umea University, Sweden.
Training and Development Agency for Schools (2008). ICT practice materials. [viewed 30 Jan 2009, verified 15 Aug 2009] http://www.tda.gov.uk/skillstests/ict/practicematerials.aspx
Ubuntu: Community (2007). Our philosophy. [viewed 5 Feb 2008, verified 15 Aug 2009] http://www.ubuntu.com/community/ubuntustory/philosophy
Vorbis.com (undated). Get set up to listen to Ogg Vorbis! http://www.vorbis.com/
Wales, J. & Baraniuk, R. (2008). Technology opens the doors to global classrooms. The Australian, 2-3 February, p. 27.
|Authors: Andrew Fluck: Andrew.Fluck@utas.edu.au Web: http://www.educ.utas.edu.au/users/afluck
Darren Pullen: Darren.Pullen@utas.edu.au Web: http://fcms.its.utas.edu.au/educ/educ/pagedetails.asp?lpersonId=3583
Colleen Harper: Colleen.Harper@utas.edu.au
Address for all authors: Faculty of Education, University of Tasmania, Locked Bag 1307, Launceston, 7250, Tasmania.
Please cite as: Fluck, A., Pullen, D. & Harper, C. (2009). Case study of a computer based examination system. Australasian Journal of Educational Technology, 25(4), 509-523. http://www.ascilite.org.au/ajet/ajet25/fluck.html