| Australian Journal of Educational Technology 2003, 19(3), 311-322. |
AJET 19 |
In this study we investigated the use of Internet technologies for administering and analysing a test of cognitive function in a web based psychology course. The aim was to determine whether the cognitive test actually measured the same functions when administered in three different settings: a supervised university based laboratory, an unsupervised university based laboratory, and a remote (or off campus) setting. Results showed that the different data collection processes generated similar data sets: the profile of responses over the four different tests of cognitive function were preserved across the different assessment settings, and improvements that occurred with practice were also consistent. Thus, the dataset can be used confidently to model key psychological principles, despite the added experimental noise created by having participants sit the tests in different settings, using different computers. The present findings highlight the potential use of web based technologies for delivering laboratory experiments in core psychology courses.
It is recognised increasingly that the addition of web based components to university psychology courses offers great advantages to students who study both on and off campus, and also to lecturers who can now develop new and innovative models for course delivery. The ability to disseminate high quality curricula to large numbers of students enables academic units to maximise teaching resources and provides a vehicle to update course content at regular intervals (Vodanovich & Piotrowski, 2001). Considerable development has already taken place in the use of web based applications to deliver lecture and tutorial material to university students generally and psychology students specifically. In contrast, development of laboratory exercises for web based learning modules has been slow. Indeed, this is characteristic not only of psychology but also of other disciplines in science including physics (Forinash & Wisman, 2002) and engineering (Lemckert & Florance, 2002).
One of the reasons for the under-development of web based psychology lab exercises is that many of these require that human subjects actually undergo some type of assessment, and in this assessment students can act as both assessor and subject. In the first instance, this allows data to be collected from a sufficient number of participants to ensure that the statistical power of experimental design clearly demonstrates the effect under investigation (Cohen, 1998). A second important aspect of the testing component of psychology lab exercises is that it allows students to develop the interpersonal skills and strategies necessary for the assessment of human subjects. While skills of this type are refined further in higher and professional degrees, for the many psychology students who do not continue with their studies this is the only training or experience they receive. Therefore, if web based psychology courses are to expand, there needs to be some experimental modules in which aspects of thinking or behaviour are assessed in actual human subjects, be they the students themselves or naïve participants. Furthermore, such modules should measure aspects of thinking or behaviour that are relevant to undergraduate training in psychology, be deliverable to off campus students as effectively as to on campus students, and provide a means whereby data collected from different students can be pooled and analysed quickly and then communicated back to students for the purpose of report writing (McGraw, Tew & Williams, 2001). In this manner, the training imperatives of the discipline can be preserved, highlighting most particularly to students the process of experimental research and its associated theoretical and practical issues.
In the current study we investigated the potential application of a web delivered computerised test of cognitive functions - specifically attention and memory. We sought to determine whether the cognitive test actually measured the same functions when used in a supervised university based laboratory, an unsupervised university based laboratory, or an off campus setting (that is, by students studying a course by distance education). Importantly, we needed to test whether the different data collection processes were each able to generate similar data sets and, hence, preserve the integrity of the underlying concepts that are the basis for the "class" exercise. To achieve this we determined whether the profiles of responses over the four different tests of cognitive function were preserved across the different assessment settings and, second, whether any improvement that occurred with practice on the tests operated the same way across all settings.
Within the CogState® test battery, stimuli for each task consist of playing cards drawn from a standard deck of 52, and all tasks are presented in the context of a game. Stimulus presentation - card type revealed face up - randomises both card selection for each task and correct response for tasks involving binary choices. Before presentation of each task, both written and interactive graphic instructions are presented on screen. These instructions disappear when the participant has made three successive correct responses to practice items, and the scored task then begins. Responses are indicated by pressing one of two keys on the computer keyboard ('d' or 'k') which represent 'yes' or 'no', respectively. All tasks present simple questions requiring a forced choice response - eg., "Is the card red?" A buzzer indicates abnormally slow or anticipatory responses, that is, failures to respond within 3500 ms or responses faster than 100 ms, respectively. Correct responses receive no auditory feedback. Response speed is recorded to the nearest 1 ms, while accuracy is given by the number of correct responses; the former was the focus of this paper. There are minimal or no practice effects and very high test-retest reliability (Collie et al., 2003; Darby et al., 2002). Users can familiarise themselves with the task requirements by practice tests until comfortable and confident with the battery.
The four tasks used were:
For each task students are asked to respond as quickly and accurately as possible. The performance measure used is the average reaction time (RT) to trials on which a correct response has been given. Before statistical analysis, this data is normalised using a logarithmic base 10 transformation.
| Setting | Responsible educator | Hardware to which test was down-loaded | Instruction | Results - data transfer | Results summary |
| Supervised university lab | Lab Tutor | Psychology lab computers | Complete the CogState test twice in 2 hours | Collected by lab tutor from computers | Provided by lecturer in lab |
| Unsupervised university | Tutor | Any university computer | Complete the CogState tests twice in 2 hours | Emailed to tutor | Provided by lecturer in class |
| Off campus | Lecturer | Home or office computer | Complete the CogState tests twice in 2 hours | Emailed to lecturer | Provided in emailed report |
To determine whether the effect of practice was equivalent across settings, a practice score was computed for each individual by subtracting the average RT for a specific task at Time 1 from the average RT on the same task at Time 2. Thus each individual student provided one practice score for each task. For each task the practice scores were compared between the three settings using a series of one way analysis of variance (ANOVA). Details of ANOVA can be found in Howell (2002).
Figure 1: Group mean (+SD) practice score (in s) for each task performed in a supervised university setting, an unsupervised university setting, and an off campus setting.
The second analysis determined whether the pattern of performance on the four CogState tasks was the same for all assessment settings. Therefore, each individual's mean RT (log10 transformed) averaged over time of assessment was submitted to a 3 (setting) x 4 (task) ANOVA with repeated measures on the task factor. The test of the hypothesis concerned the presence of any interaction between task and setting; a significant interaction would indicate that the effect of setting was not uniform across the different tasks.
Figure 2 shows the interrelationships between the different tests across the three settings, averaged over time. The ANOVA indicated a significant effect for task (F(3,13)=25.1, p<0.001) but no significant effect of setting (F(2,11)=1.1, p>0.05), nor setting by task interaction (F(3,13)<1). Investigation of the significant effect of task with dependent samples t-tests indicated that when averaged across settings, the speed of simple detection was significantly faster than the speed of identification decisions, which, in turn, was significantly faster than the speed of matching decisions. There was no significant difference between the speed of matching and the speed of remembering decisions.
Figure 2: Group mean log10RT (s) for each task across the three test settings
Interestingly, t-test revealed that the speed of matching decisions was not shown to differ from the speed of decisions made from working memory (ie., the remember task). Consideration of the task requirements indicates that in healthy young adults the speed with which objects can be matched in terms of their spatial properties is equivalent to the speed with which they can be matched in time (see also Collie et al., 2003). More generally, results are also consistent with work showing that existing web based technologies are sophisticated enough to permit the timing accuracy necessary to run chronometric experiments (McGraw, Tew & Williams, 2000). Model trends in the data that serve to illustrate key principles are replicable, despite the added experimental error (or noise) created by having participants sit the tests in different settings, using different computer hardware.
The present set of results serve to illustrate a number of core principles that fit well with current models of behaviour and their applications in different domains of psychology. These principles include response slowing with increasing task demands, faster decision times for identification decisions than matching decisions, and the equivalence of the thinking time necessary to match stimuli that occur at different points in time and those that occur (concurrently) in space.
More specifically, the pattern of results illustrates issues pertinent to experimental psychology (eg., the effect of information load on performance), to cognitive neuropsychology (eg., different processing times indicating the involvement of more complex neurocognitive networks), neuropsychology (e.g., brain disorders in which the interrelationships identified in the current study are disrupted), and to research design and analysis in psychology (e.g., the potential confounding effects of practice, and the statistics necessary to determine whether a given performance measure has changed over time or differs from another measure by virtue of cognitive load) (see also McGraw et al., 2000).
Results serve to highlight the efficacy of using interactive lab exercises in experimental psychology delivered via a web based medium. Several aspects of the software technology and computer user interface may explain the success of this approach. In the first instance, it is clear that all students in the unsupervised settings were able to download the CogState software successfully and initiate its application. Informally, no student reported difficulties in this regard, highlighting the use of clear download instructions on the source website. It has been shown that "download delay" can impair learning and performance under some circumstances (Davis & Hantula, 2001); for example, time spent on task learning decreases with large delays, particularly for inexperienced users. Download delay was not, however, a constraining factor in the present study.
It is important to note here that most students involved in this study were acquainted with online learning environments (e.g. BlackBoardTM) during their first year of undergraduate study. We believe that basic experience in downloading online resources (eg. plugins like QuickTime®) certainly enhanced the ability of students to follow download and software installation instructions in the present study. This type of experience may provide the type of orientation necessary to ensure success when students participate in online laboratories. A formal comparison between experienced and non-experienced online learners would prove instructive in this regard.
Secondly, the graphic design of the computer display is an effective medium to present (meaningful) stimuli and to communicate instructions clearly, within the context of a cognitive psychology laboratory. Results show that the integrity of the interface is not compromised in any significant way by remote delivery (see also Najjar, 1998) - the pattern of responding to visual stimuli of varying complexity and decision load is maintained across settings. In other words, the informational load and usability of the stimulus display is maintained despite variations in the immediate learning environment. As a result, participating students are assured of a consistent database that, in turn, enables them to interpret the pattern of results according to the same principles of human information processing. This level of consistency enhances communication between students in local and remote settings, particularly during the report writing phase where data is linked with past research and theoretical arguments are formulated.
We also believe that the concurrent use of written (or semantic) and visual prompts served to communicate clearly the types of responses that were required of participants. The transparency of this presentation is consistent with a dual coding model of information processing that finds broad support in the field of cognitive psychology (Paivio, 1991). From the perspective of this model, it is argued that there are two ways to represent concepts: (i) through a mental image, and (ii) through a verbal representation. Both forms of representation operate in concert, enhancing the ability to retrieve information. If the function of one system is reduced in some way (perhaps due to incomplete cues or the dominance of one processing modality over another), the other form of representation may compensate, allowing new information to be encoded in a usable and retrievable form. Importantly, presenting task instructions using a dual format is thought to be of particular benefit to naïve participants, the sample of choice in experimental studies (Blake, 1977).
Whether the same type of information can be communicated effectively to younger participants (children and young adolescents) using the same web based technologies is an empirical question; this is relevant to laboratories where undergraduate students are recruiting other people online in order to explore developmental issues. All indications are that older children and adolescents can decipher task cues of the type used in this study and make effective use of both semantic and pictorial information (Bjorklund, 2000).
It remains to be seen whether more complex task instructions can be communicated equally well as those presented here using web based technologies. We cannot discount the importance of the instructor's ability to clarify instructions on site and to help encourage participants to maintain their attention and to actively process key task cues (Burns, 1992). The communication of task instructions online may be challenged for decision making tasks involving multiple levels of analysis, from simple feature detection to more elaborate decision making in a real world (or virtual world) setting. This issue awaits further exploration.
In sum, the present findings highlight further the potential use of web based delivery in core psychology courses. In addition to its now common use in lecture delivery and presentation of simulations in tutorials, the medium is well suited to the remote delivery of core laboratory work. The technology can clearly communicate task instructions and provide a workable interface with the user, regardless of the type of setting, as well as providing a robust data set. Student participation in the research process and the integrity of the data set, in turn, reinforce principles of research design and theory that are integral to the discipline.
Bjorklund, D.F. (2000). Children's thinking: Developmental function and individual differences (3rd ed.). Belmont, CA: Wadsworth.
Burns, D.J. (1992). The consequences of generation. Journal of Memory and Language, 31, 615-633.
Cohen, J. (1988). Statistical Power Analysis for the Behavioural Sciences. Hillsdale, NJ: Lawrence Erlbaum and Associates.
Collie, A., Darby, D.G., & Maruff, P. (2001). Computerised cognitive assessment of athletes with sports related head injury. British Journal of Sports Medicine, 35(5), 297-302.
Collie, A., & Maruff, P. (2002). An analysis of systems of classifying mild cognitive impairment in older people. Australian and New Zealand Journal of Psychiatry, 36(1), 133-140.
Collie, A., Maruff, P., Darby, D.G., & McStephen, M. (2003). The effects of practice on the test performance of neurologically normal individuals assessed at brief test-retest intervals. Journal of the International Neuropsychological Society, 9(3), 419-428.
Darby, D.G., Maruff, P., Collie, A., & McStephen, M. (2002). Mild cognitive impairment can be detected by multiple assessments in a single day. Neurology, 59, 1042-1046.
Davis, E.S., & Hantula, D.A. (2001) The effects of download delay on performance and end-user satisfaction in an Internet tutorial. Computers in Human Behavior, 17, 249-268.
Forinash, K. & Wisman, R. (2002). Simple Internet data collection for physics laboratories. American Journal of Physics, 70, 458.
Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Boston: Duxbury.
Lemckert, C. & Florance, J. (2002). Real-time internet mediated laboratory experiments for distance education students. British Journal of Educational Technology, 33, 99-102.
McGraw, K., Tew, M.D., & Williams, J.E. (2000). The integrity of web-delivered experiments: Can you trust the data? Psychological Science, 11, 502.
McGraw, K., Tew, M.D., & Williams, J.E. (2001). Psychology Experiments. TechTrends, 45, 7-8.
Najjar, L.J. (1998). Principles of educational multimedia user interface design. Human Factors, 40(2), 311-323.
Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45, 255-287.
Vodanovich, S. J, & Piotrowski, C. (2001). Internet-Based Instruction: A National Survey of Psychology Faculty. Journal of Instructional Psychology, 28, 253-255.
Westerman, R., Darby, D., Maruff, P., & Collie, A. (2001). Computerised cognitive function testing of pilots. Australian Defense Forces Health Services Journal, 2, 29-36.
| Authors: Dr Peter H. Wilson Department of Psychology and Disability Studies RMIT University, City Campus PO Box 2476V, Melbourne Vic 3001, Australia Email: peter.h.wilson@rmit.edu.au Paul Maruff The Neurophysiology and Neurovisual Research Unit Mental Health Research Institute of Victoria Parkville, Melbourne, Australia Please cite as: Wilson, P. H. and Maruff, P. (2003). Validation of an Internet delivered and analysed test of cognitive function for use in web based psychology courses. Australian Journal of Educational Technology, 19(3), 311-322. http://www.ascilite.org.au/ajet/ajet19/wilson.html |