AJET 25(4) Fluck, Pullen and Harper (2009) Case study of a computer based examination system

Australasian Journal of Educational Technology
2009, 25(4), 509-523.

Case study of a computer based examination system

Andrew Fluck, Darren Pullen and Colleen Harper
University of Tasmania

Electronic supported assessment or e-Assessment is a field of growing importance, but it has yet to make a significant impact in the Australian higher education sector (Byrnes & Ellis, 2006). Current computer based assessment models focus on the assessment of knowledge rather than deeper understandings, using multiple choice type questions, and blocking access to more sophisticated software tools. This study explored a new system based on a customised version of an open source live CD, based on Ubuntu which was used with three groups of pre-service teachers (N=270). Students had divided opinions about using computers or paper for their examinations, but prior exposure to computer based assessment was a highly significant factor for preferring the computer medium. Reflecting upon their experience, students found the noise of computer keyboards a distraction during the eExamination and preferred fewer on-screen windows. The new system allowed eExaminations to be taken securely on student owned laptop computers and was supervised by invigilators without specialist information technology skills. The system has been made available for other researchers to use at http://www.eExaminations.org/

Introduction

Many educators 'teach to the test', with varying degrees of institutional support (Mathews, 2006). Therefore for information and communication technology (ICT) to produce educational transformation (Downes, Fluck, Gibbons, Leonard, Matthews, Oliver, Vickers & Williams, 2001, p.10; Finger, Russell, Jamieson-Proctor & Russell, 2007, p.73), educators must consider which assessment techniques permit students to utilise the affordances of new technology. Without a suitable, computer based way of conducting examinations (as an example of rigorous or standardised assessment), curriculum transformation may be unlikely to occur because assessment is a major determinant in teaching (Ainley & Searle, 2007, p.7). This report is preliminary, describing the development of an eExamination method and its use in pre-service teacher education. Using open source software developments, a set of tools has been created and trialled for candidates to use their own personal computer under examination conditions. These tools are available for others to investigate and download.

As flexible and online learning mediated by ICT becomes more pervasive, there is a growing need for educators to consider modes of assessment using similar tools. The cost of assessment in higher education is the most rapidly growing component of tuition fees (Ricketts, Filmore, Lowry & Wilks, 2003), whilst open content shrinks the cost of tuition and learning materials (Wales & Baraniuk, 2008). However, the increasing discrepancy between teaching through blended or online delivery with a learning content management system, and assessing using pen and paper, is another reason to consider ways in which candidates can verify their achievement whilst using computers. In our experience online, computer based assessment is fraught with difficulties which are discussed in this report. Therefore the study focuses on a new proctored and offline, computer based assessment system.

Previous studies

The role of ICT in supporting the curriculum has been extensively evaluated by some large, governmen sponsored projects. The ICT and attainment review in England (Cox, Abbott, Webb, Blakeley, Beauchamp & Rhodes, 2003) found positive effects in almost all subjects, but particularly with specific uses such as word processing in English, modelling in mathematics, or using simulations in science. The authors noted:

Researchers have often measured the 'wrong' things, looking for improvements in traditional processes and knowledge instead of new reasoning and new knowledge which might emerge from the ICT use (p. 34).

Many similar studies have noted the overall impact of ICT integration as additional motivation to learn deriving from the Hawthorne effect of novelty; or a skill set to be mastered in addition to the content knowledge addressed. In the USA a large scale study found even good software had little learning benefit (Dynarski, Roberto, Heaviside, Novak, Carey, Campuzano, Means, Murphy, Penuel, Javitz, Emery & Sussex, 2007). Furthermore, Cuban (2001) described the ineffective use of computers in classrooms. Such studies demonstrate computer use in support of a pre-existing curriculum is of contested effectiveness, and possibly inefficient. To assess the latter, researchers might legitimately inquire as to whether learning outcomes are achieved more rapidly with, or without, ICT. In any case, Australian secondary schools are about to receive a major innovation as the Australian Government's Digital Education Revolution will provide a computer for every student in Years 9-12 (Gillard, 2008, p.57). Universities therefore need to prepare for an influx of laptop-savvy students in 2-4 years time.

Online assessment is now commonplace in many in Australian universities, but Byrnes and Ellis concluded this is largely superficial for both formative and summative purposes (2006). Online assessment is mostly used for quizzes, forums and digital assignment drop boxes. In many cases online assessment is conducted using an institutional learning management system (LMS) such as BlackBoard, WebCT, or an in house product (Pullen & Cusack, 2007; 2008). Online assessments however, offer several advantages for the institution and the learner. These include:

Time analysis of responses to the question level to better discriminate between candidates (Gvozdenko & Chambers, 2007).
Including video in questions, particularly for scenarios in authentic assessment.
Adaptive testing, where the next question to be posed is determined by prior response(s).
Question banks and randomisation of questions and response orders to reduce cheating.
Automated analysis of results from entire candidate cohorts.
Immediate feedback can be given.

These benefits and the difficulties associated with various forms of online assessment have been described by BC Consulting (2006). This suggests online assessment may not be effective for evaluating creativity, problem solving ability, critical thinking, reflection, or authentic learning; collectively the characteristics of deep and effective learning. The delivery technology itself creates problems of inter-candidate interaction and is prone to technical malfunctions which can affect many students simultaneously. Should the whole network fail, the examination needs to be rescheduled (Institute for Interactive Media and Learning, 2007). We have yet to see much use of ePortfolios, role plays and simulations, although these have been suggested as being powerful alternatives, and would be worthy of formal evaluation.

... developments in assessment are advocating alternative and diverse assessment methods, including peer assessment, portfolio, reflective journaling, self-assessment, and performance-based assessment, which are deemed to be constructive, authentic, contextualized assessment, and to promote deep learning and skills development. (BC consulting, 2006, p.3)

In relation to assessment more generally, all assessment needs to be "valid, reliable, fair and flexible" (Booth, Clayton, Hartcher, Hungar, Hyde & Wilson, 2003, p.8). This can be just as difficult to achieve online as on paper, since some students have a greater aptitude for particular question types (e.g. multiple choice questions). A common response is to mix question types in any given test. Assessment equity and quality is commonly also achieved through the context of an individual written examination.

Only six per cent of partially online assessable units had a final exam that could be taken online. Of the partially online assessable units without an online final exam, 83 per cent had a traditional pen and paper final exam, indicating that the use of final exams is quite widespread. Overall, 84 per cent of partially online assessable units had a final exam. (Byrnes & Ellis, 2006)

Outside Australia, there appears to be a greater uptake of eAssessment in schools, with 38% of awarding bodies surveyed in England using some form of eAssessment to deliver up to 60% of their assessment programme (Chapman, 2006). This sends a strong message to Australian quality assurance bodies and Boards of Studies which may find eAssessment worth investigation.

LMS based online testing environments offer useful tools for conducting assessments of knowledge. Automated marking is feasible for multiple choice questions, and for short answer questions where key words are sought in the response. Assessment feedback requiring an understanding of an essay, etc, is so far not widespread and responses which include diagrams might be difficult to mark automatically using vision recognition systems. The challenges faced by online systems are those posed by Booth et al. (2003). Assessment of student knowledge and skills within a web browser window or delivered by bespoke assessment software (specifically crafted for a particular set of questions) provides a restricted environment which prevents the demonstration of abilities associated with the use of specialist software or a combination of applications.

To be fair, online systems need to authenticate the individual undertaking the assessment - some systems have gone so far as to take photographs at random intervals to assure this (Rönnberg, 2001). However, the camera may not necessarily be pointing at the person undertaking the responses, so this is not a foolproof method for identifying candidates. Another aspect of fairness is generally eliminating collaboration: since the computer is online, this is hard to implement except by locking out critical functionality during the assessment. For example, E-Tests developed by Ko and Cheng (2008) required an Iomega zip disc peripheral; all the data files were encrypted, and the program would only run on computers with a pre-registered network card. Such a system handles large numbers of students very well, but is restricted to simple question types. Such systems primarily facilitate automated marking, whilst we sought to provide a comprehensive environment to move all assessment types into a digital modality.

Fairness can be improved by adopting a proctored or supervised testing environment, where all students are watched by an examiner as they undertake the assessment. But how can the examiner prevent collusion if all candidates are using computers? Bluetooth, wireless networking, infra-red and mobile phone connections are all feasible communication channels and these are not easily blocked. Thus, if assessment, particularly of high stakes summative examinations, is to move into an ICT environment, some other technological and pedagogical approach is required. An approach which can exploit student owned equipment would be particularly suitable, since few institutions can deploy many hundreds of computers for the small fraction of the year devoted to formal assessment. The institution may then be able to handle small numbers of students without computers or with faulty equipment, much as a biro might be loaned a student experiencing pen difficulties.

The value of online assessment appeared compromised by security concerns leading to the possibility of unfair advantage for some candidates. Our study therefore explored the practical implementation of offline eExaminations, student responses to eExaminations and factors related to acceptance or rejection.

The course and content

We developed and used the eExamination technique in a course unit focusing on the development of pedagogical ICT skills in the third year of a four-year Bachelor of Education program. Many units in this program use the WebCT-Vista LMS to support learning in a blended mode, so students are reasonably well versed in the use of ICT for their personal study. A first year unit ensures they have at least a minimal level of personal operational ICT skills - that is, they can operate a word processor, compose web pages, etc. Our aims in the unit were that students should learn to:

Demonstrate awareness of the extent and difference between home and school ICT access for pupils.
Facilitate and lead appropriate, equitable and responsible pupil use of information and communication technologies integrated into all areas of learning.
Demonstrate personal skills and knowledge of ICT in school education, and understand this requires continuous updating.
Inform parents and the community on how information technologies are, or would be, used in the school.

The course included lectures and tutorial workshops in computer laboratories for students to master a range of theoretical knowledge and practical skills. These included pedagogical approaches to ICT in schools; planning ICT integration; online learning objects and the use of other digital educational materials; robotics and game making; animations using free and open source software; software and hardware for students with special needs; methods for the integration of ICT into classroom practice and consideration of transformational changes in schooling through ICT.

The design of the eExamination

Student achievement of the learning outcomes was assessed through two equally weighted activities. The first was a take home assignment in which students explored new learning content by creating a new learning outcome for pupils, which can only realistically be achieved through their use of ICT. The second activity was a two-hour test under supervised conditions using a bootable CD operating system. Students were required to answer fourteen questions based on all the material in the unit. These questions comprised eleven short answers and three longer, open ended investigations, one of which required a critical review of pupil generated PowerPoint files. It needs to be stressed that the assessment was of understandings gained in the course, not of operational computer skills. It was an open book eExamination, but no communication with other persons was permitted during the test. This pilot study provided an initial proof of concept that formal assessment could be conducted with students using computers instead of pen on paper. The pedagogical transformations such a new process could enable were only modestly explored.

Six to eight weeks prior to the eExamination, students were given a free, practice CD. They were shown how to use it during a practice tutorial to ensure they could operate the software. During the tutorial, students were taught the rudiments of the Ubuntu operating system, the use of Open Office Writer for word processing and shown how to create drawings using The Gimp software.

The exam procedure involved putting the CD into a computer, then switching it on. The computer then booted (began to operate) using the Ubuntu-Linux operating system from the CD. An exam folder appeared on the desktop. The examination questions within the folder had been prepared with Microsoft Word, so students double clicked on the test document to open it with OpenOffice Writer. The first instruction in the rubric told the student to immediately re-save the test to the desktop with a new filename comprising their surname and ID number.

Students then completed the answers to each question, saving at appropriate intervals (using good computer procedures such as before making a major change or after every 300 words or so of text input). When finished, they saved the document and closed all application(s), without switching off the computer or logging out. In the event of equipment failure, students were to report this quietly to the test supervisor. A paper version of the test would then be made available, or students could restart using another computer. Access to the Internet or any other digital resources was not allowed. Digitally facilitated collusion was prevented in three ways:

Networking was disabled in the CD based operating system by omitting the interface modules and drivers from the operating system compilation.
Networking cables were withdrawn from the wall sockets in each computer laboratory. This was quick and easy to do, and visually monitored.
The gateway for the sub-net containing the laboratories was disabled by our IT technicians for the duration of the examination.

These three methods were used simultaneously. Each of them alone should have been sufficient to prevent network access. We were therefore confident candidates would be individually responsible for their submissions. We could have gone further to prevent collusion based upon observation of neighbour's work by installing privacy screen filters. These comprise very thin vertical bars which reduce the oblique viewing angle for a computer screen, but were not used because of the cost and our seating arrangement which minimised observation opportunities.

Preparing the eExamination

Ubuntu (http://www.ubuntu.com/) is one of several Linux distributions which is available either for installation onto a computer or which can be run from a 'live CD'. This latter option allows a computer user to try the operating system without over writing any existing installation on their machine. Ubuntu continues to be developed by a community initiative (sponsored by Canonical Ltd.) under the following philosophical principles (Ubuntu, 2007):

Every computer user should have the freedom to download, run, copy, distribute, study, share, change and improve their software for any purpose, without paying licensing fees.
Every computer user should be able to use their software in the language of their choice.
Every computer user should be given every opportunity to use software, even if they work under a disability.

The licence for Ubuntu and most of its associated software therefore allows copies and derivative products to be distributed freely. An application called Reconstructor is available to create new, live CD distributions of Ubuntu, allowing software packages to be added or deleted from the standard CD image. We have modified this application to simplify the creation of an examination CD by automating the process and reducing the number of steps required. One critical step for exam designers is to decide which customisation options to include. Figure 1 shows the options screen.

Figure 1: Modified Reconstructor options screen

The option for computer based networking allows examiners to conduct the digital equivalent of open book examinations. However, local communications such as Bluetooth (which can interface to mobile phones) or unwired networking can be disabled separately. To reduce distracting computer sounds in an examination hall, the loudspeaker drivers can be eliminated. If students are using their own laptops, or even loan pool laptops, local hard disk drive access can be eliminated.

The five step creation process allows the examiner to upload and include a folder containing examination materials onto the CD. The folder can contain all the files necessary for the examination, including a word processed document (as in our example), video, application software, etc. The examiner can also specify an image file which becomes a desktop background. This image is a useful security feature: any student whose screen does not show this unique image must have booted their computer from another source. For non-technical examination invigilators, this provides a quick visual check to confirm each student is using the correct materials. Once a master CD had been prepared, copies were produced by a standard duplicator with printed instructions on the top surface at about 50c each. This is similar to the cost of printing an examination paper.

Evaluation

At the end of the open book eExamination, students were offered a single page survey (see Appendix) with five questions to answer. This was voluntary, and had standard university ethics approval (reference H9714).

Responses indicated only 38% of survey respondents had previously taken a computer based exam, so this was a unique experience for the majority. The most highly cited examples of previous experience included taking a multiple choice health questionnaire on the university learning management system; an ICT competency assessment used as a diagnostic in the first year of the course (Training and Development Agency for Schools, 2008); and pre-employment online training and assessment associated with paid work to support their studies. The teaching team felt a high degree of responsibility for making the eExamination experience one which students would like to repeat. This was a leading edge cohort, whose opinions were likely to shape sentiment about the innovation across the institution. The survey analysis considered responses from three distinct groups of students as described in Table 1.

Table 1: Groups of students undertaking the eExamination

Group A Group B Group C

N 138 130 2

Respondents 125 106 2

Exam date October 2007 October 2008 December 2008

Exam media:
Delivery/collection CD/USB CD/USB USB/USB

Students using own laptops 0 6 0

Exam type Single document (edited in Open Office Writer) PDF for questions; Open Office Writer document for responses. Ogg Vorbis video, PowerPoint student work sample, Word curriculum document. PDF for questions; Open Office Writer document for responses. Ogg Vorbis video (Vorbis.com, undated).

Proportion reporting technical difficulties in the eExamination 23% 56% 0%

When giving feedback on their personal preparation for the eExamination, 78% of respondents had used the practice CD before the eExamination, and 71% had found it very or moderately useful.

Computer versus paper

The critical question on the survey form asked if respondents felt on balance, it is better to have a formal test conducted using computers or using more traditional handwriting on paper. Opinion was divided with 46% favouring each alternative and 7% selecting both options (230 valid responses). This compares with 94.5% preference for computer-based testing found in a case which included online feedback (Jonsson, Loghmani & Nadjm-Tehrani, 2002). However, opinions were significantly different between the groups of students F(2, 227) = 3.484, p = 0.032. Whereas 56% of Group A preferred exams on computers, only 35% of Group B preferred this medium. The team considered the greater reported incidence of technical problems in Group B could be responsible for this difference, but this was not confirmed by a one way ANOVA showing little difference in preferred medium with F(2,222) = 2.667, p=0.073.

However, students who had previously taken a computer based exam preferred this medium (63% of respondents) compared with 37% of first-timers. The one way ANOVA confirmed a very significant difference due to prior computer exam experience with F(2,227) = 8.683, p<.001 with an effect size of 0.621. Thus a first experience of eExaminations appears to stimulate a preference for computer based testing, and is more significant than any subsequent technical difficulties.

In 2007 the examination consisted of a single document containing questions with spaces under each for student responses. We had two computer 'lockups', but both candidates had been able to resume after a restart, losing none of their work. In designing the 2008 eExamination, we realised students might inadvertently edit the questions, so prepared these as a separate PDF file. In addition we included a stimulus scenario video and PowerPoint student work sample. Inadequate testing on the variety of computers failed to reveal the video playback did not silence internal computer speakers on some models despite using headphones, which disturbed other candidates. In addition, the version of Open Office crashed when a PowerPoint file was closed - not a good experience for students new to eExaminations! These factors accounted for many of the technical difficulties reported by Group B. Nearly one third of the students who reported other technical difficulties included the comment 'the system was slow to respond'. This is an effect of using a 'live' CD operating system since commands must be loaded from the optical media, and led to the development of the variation used for Group C entirely based on a USB drive. Other specific reports of technical difficulties included:

I could not remember how to create a diagram
Spelling issues - I kept touching the wrong keys
AutoCorrect changed my words

Other general comments included the following (Figure 2).

Figure 2: Example comments from students concerning the eExamination

Sixteen students in Group A (12.8%) commented on the high noise level from computer keyboards during the eExamination, with notes such as:

Everyone in one room typing is LOUD! It was hard to concentrate.
The noise of the clicking [of keyboards] was very distracting. The sound makes you feel rushed.

We warned Groups B and C about the problem of noise, and no students from these groups commented on this aspect of the eExamination. However, the latter groups did express frustration about having to manipulate multiple windows: "It was frustrating going between several documents. It would be better if answers & question [were] in one document". We are looking at the possibility of putting future eExaminations in a single file with questions in a distinguishing colour and protected from editing.

Focus group

Six students from Group B kindly agreed to participate in a focus group with two of the authors immediately after their eExamination. It quickly became clear that in this small group there were students with opposing views about eExaminations. The first speaker, Betty (pseudonyms have been used to preserve anonymity as required by our ethics approval), was supportive of using the computer, because she had more freedom of expression. The second speaker, Tom, felt disadvantaged by a slow typing speed, but acknowledged readability of the script compensated somewhat:

Betty: I prefer the computer based one because for me, I don't have very good handwriting, especially when I am under pressure. My handwriting gets pretty horrible, and I enjoyed... Like, we had a Maths (examination) a couple of days ago, and we had quite small spaces to put our answers in. But with this one we could use as much space as we wanted, which was especially useful in the last question where we could do an extended answer and make the text box as large as we wanted. Which is very good. It would have been stressful if there had been technical issues, but mine just worked.
Tom: I am a bit of a Luddite. So I was quite... I quite enjoyed it. It was sort of different. Just having to switch between the two screens, and the video, stretching a bit. I did find it very noisy. I have a bit of a headache at the moment. So that was one down for it. Also, I can't type. So instantly, I felt at a disadvantage. While everybody else is using all fingers, I am counting on two. So those were the main pitfalls. I felt that I wasn't able to keep up, so I had to be a bit more concise in what I wrote. Because I couldn't write a lot. With a paper based exam, everyone is more on a level playing field, in regards to what you could write within a set time .... My handwriting is atrocious. (widespread laughter). So it's like well OK, I can see the benefits of being able to read something keyboard - wise, but yes...

In further discussion the group debated giving students a free choice of writing tool, but acknowledged this choice does not always exist 'in the real world'. The different pace of the eExamination for Group B was significant. One question asked candidates to view a short video and respond in a personal way about their future actions. Peter liked the video:

Peter: it breaks up the exam. I found it good. ... The footage was good, and it was engaging. It went to two minutes, which I thought was perfect. 120 seconds was all we needed. OK, I thought, I am awake again. I thought it was good. For example in the math exam, unless you step up and walk out to go to the toilet, there are no breaks in the intensity. So it was great that there was that footage. And it was easy to access.

Although this respondent was commenting on the change of pace in the eExamination, the video example illustrates the benefit of moving into the new medium. The research team was exhilarated by the opportunity afforded to move beyond print in formal assessments, opening the door to authentic testing of skills.

Discussion and conclusion

This eExamination case study has revealed important information about ICT innovation which has implications for Australian schools and universities. Firstly, there was no initial clear cut preference for computers or paper based examinations amongst the Bachelor's degree students in the case study. The cohort was split fairly evenly between the two options. Further research could be undertaken in other parts of the country to validate this finding for similar cohorts, and to investigate preferences for older and younger (school aged) cohorts. Secondly, the preference for examination medium appeared to be strongly related to successful prior student experience of using computers in assessment, with a very significant difference and an effect size of 0.621. Such a finding is very valuable to innovation engineers - it tells us that creating the first positive experience of eExaminations is a vital first step to rapid adoption. This describes the chicken and egg aspect of innovation, where adoption is related to perceived utility. There are an increasing number of computer mediated assessment products on and entering the market. Apart from proprietorial interests of the suppliers, each of these generally restricts the demonstration of ability to a closed environment. Example systems we have found have the following restrictions:

Suited for a restricted discipline/knowledge domain, e.g. Pearson VUE [http://www.pearsonvue.co.uk/] for the UK Driving Standards Agency's Driving Theory Test [ http://www.pearsonvue.co.uk/OurClients/Pages/DSA.aspx]
Requires examiner expertise with a particular question setting or authoring software application, e.g. Articulate Quizmaker 2.0, http://www.articulate.com/products/quizmaker.php
Internet connectivity required throughout the examination, but candidate ability to run any other software is blocked. e.g. MaxExam [ http://www.maxexam.co.uk/info/whymaxexam_features.asp] or Securexam [ http://www.softwaresecure.com/student.htm]
Question types restricted to offered choices only, e.g. BrainsBuilder [ http://www.brainsbuilder.com/site/features.do]

One reason for supporting a change to eExaminations is the linkage between tertiary and secondary assessment methods. These are only loosely linked systems, yet it is likely methods used in one sector will be noticed and appropriately adopted in the other. Secondary sector assessments may be impeded from adopting computer mediated methods whilst tertiary systems rely heavily on high stakes written examinations (Fluck, 2007). Our informal scrutiny of pre-service teachers in schools substantiates this linkage, with few instances of computer based assessments observed in classrooms. We speculate that the low rate of eAssessment in schools could also be due to governance issues (Robertson, Grady, Fluck & Webb, 2006), lack of opportunity for teachers to perceive significant advantage (Rogers, 2003) or greater demands in schools which are 'busy places' (DEST, 2006, p. 121).

This student cohort was ambivalent about the introduction of computer based examinations. To facilitate adoption of this innovation we make some recommendations:

Attempt to find quieter keyboards
Advise the use of larger headphones covering the ear rather than ear buds worn internally.
Try to make time pressures in eExaminations less severe and therefore typing speeds less crucial.
Find some quicker way to collect competed scripts. We prefer a non-networking solution, such as write to CD, or boot from USB drive (as per Group C), to avoid conflicts about security and collusion and enable students to practise at home on personal laptop computers.
Examiners can consider allowing students to select writing tools of their personal choice for completing the examination.

In conclusion, the system used in this study had a number of useful attributes. The open source operating system on a live CD (and subsequently converted to bootable partitioned USB drives) provided a holistic examination environment which students could use to practise personal skills at home without copyright infringement. It also allowed eExaminations to be supervised by invigilators without specialist information technology skills by using a unique desktop background image for security. eExaminations can be easily constructed for a range of software environments, with candidates working on isolated workstations to 'open book' contexts with full internet connectivity. The use of the live CD allows candidates to use their personal laptop for the eExamination without gaining advantage because local disk access can be blocked. Future developments will focus on transfer to USB drives to improve operational speed, facilitate the collection of completed scripts and permit cross-platform compatibility (Intel based Macs and PCs). Although the cost of USB drives is higher than the 50 cents per CD, they can be re-used many times. Instructions for both these techniques and the open source 'Reconstructor for Exams' software are available at http://www.eExaminations.org/

Acknowledgments

The authors would like to acknowledge the contribution of Joshua Hesketh of Open Technology Solutions [ http://www.opentechnologysolutions.com.au/] for the practical implementation of some work described in this paper. Thanks also go to the students who have participated in trials and voluntarily responded to questionnaires over the last two years.

References

Ainley, J. & Searle, D. (2005). Students in a digital age: Some implications of ICT for teaching and learning. Melbourne. MCEETYA ICT in Schools Taskforce. [viewed 11 Aug 2009] http://pandora.nla.gov.au/pan/82404/
20080317-1603/www.icttaskforce.edna.edu.au/icttaskforce/webdav/site/icttaskforcesite/
users/root/public/Students_digital_age.pdf

BC Consulting (2006). Effective on-line assessment: Workshop support materials. University of New South Wales: Sydney. [viewed 11 Aug 2009] http://www.edtec.unsw.edu.au/inter/dload/flex_ed/resources/Online%20
Assessment/Online%20Assessment%20Workshop%20notes.doc

Booth, R., Clayton, B., Hartcher, R., Hungar, S., Hyde, P. & Wilson, P. (2003). The development of quality online assessment in vocational education and training, Volume 1. National Centre for Vocational Education Research. [viewed 11 Aug 2009] http://www.ncver.edu.au/research/proj/nr1F02_1.pdf

Byrnes, R. & Ellis, A. (2006). The prevalence and characteristics of online assessment in Australian universities. Australasian Journal of Educational Technology, 22(1), 104-125. http://www.ascilite.org.au/ajet/ajet22/byrnes.html

Chapman, G. (2006). Acceptance and usage of e-assessment for UK awarding bodies - a research study. Proceedings 10th CAA Conference 4-5 July, Loughborough University, England. [viewed 18 Feb 2008, verified 15 Aug 2009] http://www.caaconference.com/pastConferences/2006/proceedings/Chapman_G_c2.pdf

Cox, M., Abbott, C., Webb, M., Blakeley, B., Beauchamp, T. & Rhodes, V. (2003). ICT and attainment: A review of the research literature. DfES, London. [verified 12 Sep 2009] http://partners.becta.org.uk/upload-
dir/downloads/page_documents/research/ict_attainment_summary.pdf

Cuban, L. (2001). Oversold and underused: Computers in the classroom. Harvard University Press, Cambridge, Massachusetts.

DEST (Department of Education, Science and Training) (2006). Family-School Partnerships Project: A qualitative and quantitative study. Saulwick Muller Social Research: Canberra. [verified 15 Aug 2009] http://www.familyschool.org.au/pdf/muller.pdf

Downes, T., Fluck, A., Gibbons, P., Leonard, R., Matthews, C., Oliver, R., Vickers, M. & Williams, M. (2001). Making better connections. ACSA, ACCE, TEFA, University of Western Sydney. http://www.dest.gov.au/archive/schools/publications/2002/MBC.pdf [viewed 11 Feb 2009, verified 15 Aug 2009].

Dynarski, M., Roberto A., Heaviside, S., Novak, T., Carey, N., Campuzano, L., Means, B., Murphy, R., Penuel, W., Javitz, H., Emery, D. & Sussex, W. (2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort. U.S. Department of Education, Institute of Education Sciences, Washington, DC. http://ies.ed.gov/ncee/pdf/20074005.pdf

Finger, G., Russell, G., Jamieson-Proctor, R. J. & Russell, N. (2007). Transforming learning with ICT: Making IT happen. Pearson Education Australia.

Fluck, A. (2007). Can tertiary e-Assessment change secondary school cultures? In Proceedings 11th CAA Conference, 10-11 July, Loughborough University, England. [viewed 2 Feb 2008] http://www.caaconference.com/pastConferences/2007/proceedings/Fluck%20A%20f2_formatted.pdf

Gillard, J. (2008). The importance of being digitally engaged. Information Age, Dec08/Jan09, 56-59. [verified 15 Aug 2009] http://www.acs.org.au/iage/ia200812-01.pdf

Gvozdenko, E. & Chambers, D. (2007) Beyond test accuracy: Benefits of measuring response time in computerised testing. Australasian Journal of Educational Technology, 23(4), 542-558. http://www.ascilite.org.au/ajet/ajet23/gvozdenko.html

Institute for Interactive Media and Learning (2007). Online examinations. University of Technology Sydney. http://www.iml.uts.edu.au/assessment/online/guides/0222.html [viewed 3 Feb 2008, verified 15 Aug 2009].

Jonsson, T., Loghmani, P. & Nadjm-Tehrani, S. (2002). Evaluation of an authentic examination system (AES) for programming courses. http://www.ida.liu.se/~snt/teaching/HGUR-sept02.pdf [viewed 17 Feb 2008].

Ko, C. & Cheng C. (2008) Flexible and secure computer-based assessment using a single zip disk. Computers & Education, 50(3), 915-926.

Mathews, J. (2006). Let's teach to the test. Washington Post, 20 February, Page A21.

Moursund, D. (2005). Introduction to information and communication technology in education. University of Oregon, USA. http://uoregon.edu/%7emoursund/Books/ICT/ICTBook.pdf [viewed 11 Aug 2009]

Pullen, D. & Cusack, B. (2007). Schooling without borders. In J. Sigafoos & V. Green (Eds), Technology and teaching (pp. 101-114). Nova Science Publishers.

Pullen, D. & Cusack, B. (2008). Content management systems: The potential for open education. Fact Sheet FS01. Australian College of Educators, Canberra.

Ricketts, C., Filmore, P., Lowry, R. & Wilks, S. (2003). How should we measure the costs of computer aided assessment? Proceedings 7th Computer Assisted Assessment Conference, Loughborough University, UK. [viewed 11 Aug 2009] http://hdl.handle.net/2134/1924

Robertson, M., Grady, N. B., Fluck, A. & Webb, I. L. (2006). Conversations toward effective implementation of information communication technologies in Australian schools. Journal of Educational Administration, 44(1), 71-85.

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Simon and Schuster.

Rönnberg, K. (2001). Security in learning management systems - User authentication in online assessment. Master's Thesis: Umea University, Sweden.

Training and Development Agency for Schools (2008). ICT practice materials. [viewed 30 Jan 2009, verified 15 Aug 2009] http://www.tda.gov.uk/skillstests/ict/practicematerials.aspx

Ubuntu: Community (2007). Our philosophy. [viewed 5 Feb 2008, verified 15 Aug 2009] http://www.ubuntu.com/community/ubuntustory/philosophy

Vorbis.com (undated). Get set up to listen to Ogg Vorbis! http://www.vorbis.com/

Wales, J. & Baraniuk, R. (2008). Technology opens the doors to global classrooms. The Australian, 2-3 February, p. 27.

Appendix

Authors: Andrew Fluck: Andrew.Fluck@utas.edu.au Web: http://www.educ.utas.edu.au/users/afluck
Darren Pullen: Darren.Pullen@utas.edu.au Web: http://fcms.its.utas.edu.au/educ/educ/pagedetails.asp?lpersonId=3583
Colleen Harper: Colleen.Harper@utas.edu.au
Address for all authors: Faculty of Education, University of Tasmania, Locked Bag 1307, Launceston, 7250, Tasmania.

Please cite as: Fluck, A., Pullen, D. & Harper, C. (2009). Case study of a computer based examination system. Australasian Journal of Educational Technology, 25(4), 509-523. http://www.ascilite.org.au/ajet/ajet25/fluck.html

[ PDF version of this article ] [ AJET 25 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet25/fluck.html
Created 15 Sep 2009. Last revised 15 Sep 2009.
Page access count

	Group A	Group B	Group C
N	138	130	2
Respondents	125	106	2
Exam date	October 2007	October 2008	December 2008
Exam media: Delivery/collection	CD/USB	CD/USB	USB/USB
Students using own laptops	0	6	0
Exam type	Single document (edited in Open Office Writer)	PDF for questions; Open Office Writer document for responses. Ogg Vorbis video, PowerPoint student work sample, Word curriculum document.	PDF for questions; Open Office Writer document for responses. Ogg Vorbis video (Vorbis.com, undated).
Proportion reporting technical difficulties in the eExamination	23%	56%	0%