Australian Journal of Educational Technology
2002, 18(2), 208-226.
AJET 18

Iterative evaluation of Reflex Control of Blood Pressure

Debbi Weaver
University of Melbourne
Tony Gilding
James Cook University


Introduction

Academic teachers and designers of teaching materials are likely to need an iterative approach to evaluation where cycles of change, use, data collection and reflection will progressively improve the teaching resource to a point that satisfies its intended role within the curriculum. It is only through several iterative cycles that we appreciate how students relate to a new computer facilitated learning (CFL) teaching resource, and so can refine it to better suit our teaching and learning purpose.

This paper reports on two cycles of evaluation conducted on a CD-ROM tutorial, Reflex Control of Blood Pressure, during 2000. The evaluation was undertaken as part of a national ASCILITE and CUTSD project, Learning Centred Evaluation of Computer Facilitated Learning Projects in Higher Education.

The CD-ROM tutorial

Reflex Control of Blood Pressure is a standalone CD-ROM tutorial, originally developed to assist first year medical students, undertaking a traditional course, understand the baroreceptor reflex, which maintains blood supply to the brain during changes in posture, blood volume, etc. This reflex is one of the best and most simple examples of negative feedback, where an initial signal invokes responses to decrease the signal (eg. an initial drop in blood pressure triggers a set of responses that act to increase blood pressure, to return it to its set point).

Traditionally, the function of this reflex has been taught in lectures, using diagrams of a 'black box' to describe what was happening to nerve impulses in the brain. Not surprisingly, students memorised the diagram, but had little understanding of how negative feedback occurred. A common student misconception (reported by academics) was that negative feedback mechanisms would over correct blood pressure beyond the set point.

Literature and software searches for developments dealing with this area have to date been unsuccessful. Several simple animations of the baroreceptor reflex operating are available, but these are all of the "watch and learn" type, with little or no interactivity, or are merely question and answer exercises. Mathematical simulations have been attempted for some other biological mechanisms, but in the case of blood pressure control, which has a multitude of different factors all contributing to the outcome, it is easy to lose sight of the underlying concept of this reflex. The aim of this project was to develop a model building exercise, which would involve a very simple animation to demonstrate the outcome of student decisions, with no attempt to make this a "real world" or quantitative simulation. This would allow students to explore the consequences of different scenarios, and receive feedback on their decisions. Highly interactive model building exercises of this type are rare, and we have been unable to find anything similar in this field.

The software developed, Reflex Control of Blood Pressure, provides introductory information on the regions of the brain involved in this reflex pathway, definitions of feedback control, and an introduction to the symbols used to represent elements of a neural pathway. The major section of the program is a model building exercise, where students manipulate nervous system components to construct their own reflex pathway, and then can test this by increasing the blood pressure. Feedback in the form of animations and text panels are provided for every combination of elements. Once the model is completed, the effects of postural changes can be investigated. A case study is included, and a paper based Tasks sheet is supplied in the classroom (the questions from this Tasks sheet are also included in the program).

The software enables students to explore the subject by actively experimenting with outcomes of their decisions in a setting that assumes little background knowledge of cardiovascular physiology, yet capitalises on everyday knowledge of human function (eg. ability to stand up without fainting). The use of simulated environments like the one in this resource is consistent with the view of the learner as self directed and constructing knowledge afresh every time (Papert, 1996). Recalling Piaget's quote "To understand is to invent", the CD-ROM resource was designed to create conditions for invention, rather than provide ready made knowledge (quoted from Papert, 1996).

In addition, it was intended that students and teachers would use this resource within scheduled tutorial classes in ways that encouraged active discussion and debate about the decisions students made while building their model. Thus, the design of the program is consistent with Vygotsky's theory of the importance of the social context in the development of scientific concepts (Vygotsky, 1962).

Key learning objectives of the tutorial

The major aim of the CD-ROM is to understand how the central nervous system controls blood pressure with changes in posture or activity. Within this, the key learning objectives are to:

Educational context

This CD-ROM resource was originally designed for first year medical students at The University of Melbourne, within a traditional format curriculum. Students worked through the CD-ROM in a computer laboratory, as part of a scheduled class, and with several tutors present.

In 1999, the Medical course at the University of Melbourne changed to a form of problem based curriculum. The number of lectures and practical classes was drastically reduced, and replaced with a 'Problem of the Week', where students work in groups of 10-12 with a tutor for 4 hours each week. As a result of these course changes, the subject material of blood pressure control is now covered early in second year Medicine, rather than at the end of first year, so was not taught in 1999. The CD-ROM is a recommended self directed learning resource for two weeks of the new problem based learning (PBL) schedule. It is available on all computers in many computer laboratories and specialised PBL rooms across the Faculty. The effects of the change of course structure on how students use the program is not known, leading to a key question: How do students use a tutorial designed for a traditional curriculum in the newly re-designed problem based curriculum?

In the study undertaken for this paper, students were invited to work through the program in a scheduled session, with tutors present for assistance.

In addition, it was always the intention of the developers to design the program to be used by other courses taught in the Department of Physiology, particularly as part of the extensive computer facilitated learning (CFL) curriculum for second year students undertaking the 3 year Bachelor of Science degree. During the current study, the program was used as part of weekly scheduled CFL sessions by both second year Science and second year Biomedical Science students. Both courses are conducted under the more traditional format of scheduled lectures, practical classes and tutorials (in the form of CFL sessions with tutors present for assistance). Biomedical Science is a new 3-year degree course at The University of Melbourne, and this was the first cohort of second year students in this course.

Evaluation framework

History of the project

The CD-ROM was developed in 1997, and extensive formative evaluation undertaken with first year Medical students in 1998. Analysis of the data collected revealed several shortcomings in the program, which was rewritten as a result. This round of evaluation was presented at ASCILITE '99 (Weaver et al, 1999). The aims of the current round of evaluation were to determine the effectiveness of the recent modifications on student learning processes and student learning outcomes, and to determine whether this program was relevant to, and appropriate for, the different courses in which it has now been introduced.

Stage of evaluation

Using the learning centred framework adapted from Alexander, Hedberg and Bain (Phillips et al, 2000), this project was in both the development and implementation phases. Previous cycles of evaluation have concentrated on analysis and design stages of development, and on determining whether the project is functional, easily used by students, and an enjoyable learning experience for the students.

As several modifications were introduced as a result of these previous cycles, it was considered important to repeat evaluation into the monitoring of the learning environment to determine whether this CFL package was functional and enjoyable to use. In particular, previous evaluations had revealed students had experienced a great deal of frustration in getting started on the model building exercise. Several causes for this had been identified, one of which was the large cognitive load associated with learning how to use the tools provided, identifying the graphical representation of components of the nervous system, and at the same time trying to build a difficult model (cognitive load is defined as the level of "mental energy" required to process a given amount of information, after Cooper, 1990).

The current study also aimed to conduct evaluation into monitoring of the learning process, to determine whether this CFL was influencing the learning process as intended, and also to begin evaluation of the learning outcome. Some questions on the educational relevance of the project were also included in student surveys to begin evaluation on the innovation appropriateness. We recognise that the findings of this study are affected by the educational context of the tutorial, for the different cohorts of students using the program. The tutorial is now a recommended resource for medical students, rather the originally intended scheduled class, and this will have an affect on how the students view the importance of the program for their study.

Action inquiry evaluation of CFL

For the purposes of the CUTSD project (Phillips, Bain, McNaught, Rice and Tripp, 2000), action inquiry, which has its origins in action research, is defined as research, generally into social interaction, that has a major aim of producing a change in the environment or practice being researched, rather than producing written theories (Kember & Kelly, 1993). It requires active participation by the researcher in the process.

Characteristically, action inquiry is conducted in a cyclical, or spiral, manner, and involves one or more series of cycles, which can be summarised by the following stages:

Action inquiry is ideal for evaluation of developments of this kind, because of its cyclical nature, and because of the reflection of the CFL in use. It is essential that any modifications introduced as a result of early evaluation cycles be then evaluated to ensure they are appropriate and are producing the desired outcomes.

Evaluation cycles

Two cycles of evaluation were completed in 2000. In Semester 1, the CD-ROM was recommended to second year Medical students as a key self directed learning resource for their 'Problem of the Week'. The expectation was that students would complete the CFL program in the on campus computer laboratories during that week. However, for the purposes of this study, a voluntary scheduled session was advertised during lectures and on student online noticeboards, with a lecturer present to entice attendance.

In addition, the CD-ROM was also scheduled for use by second year Science and BioMedical Science students in Semester 2. Although both these courses are very different to the new PBL Medical course, evaluation with both groups of students have allowed us to test the impact of the modifications made since Semester 1.

170 second year Science students undertake weekly 2-hour scheduled CFL classes, with one tutor to about 30 students in each of 5 sessions. The course is of the traditional lecture / practical / tutorial structure, and we have been teaching this course for many years.

BioMedical Science is a new course, which commenced in 1999, and in 2000, we taught the first cohort of 80 second year students. The course is of the traditional format, with a weekly 2-hour timeslot of either CFL tutorial or laboratory practical classes. Students within the BioMedical Science course are derived from amongst the highest achieving secondary school students, and we anticipated these students would perform better when compared with students undertaking the more general Science course.

Evaluation questions

Key evaluation questions

Cycle One (identified from previous formative evaluation)
The major foci of this round were broadly classified as program usage and interface issues, and learning process issues. Under the first category, we wished to investigate the effect of design changes made as a result of the previous cycles of evaluation. At the same time, we wished to compare the way the program was used by students studying the new PBL course, with our experience from the previous traditional course.

Cycle Two (identified from Cycle One evaluation)
Again, the major foci of this cycle of evaluation included program usage and interface issues, and learning process issues, as well as beginning evaluation into learning outcomes. The first question was whether the modifications implemented as a result of Cycle One evaluation were effective in reducing student frustration and confusion. We were also interested in determining how student use of some of the features of the program may have changed as a result of these modifications. In addition, we also attempted evaluation into whether student use of this program has helped achieve the desired learning objectives.

Further elaboration of these questions is included in Table 1.

Table 1: Summary of the two evaluation cycles completed in 2000:
Evaluation questions, data analysis methods and key findings.

Action Cycle OneData Collection and AnalysisKey findings
1. Program usage/ interface issues
  • Has the simplification of the brain anatomy section resolved student frustration?
  • Has the introduction of the practice screen and other introductory screens helped reduce the cognitive load experienced by students?
  • Has the introduction of the Case Study helped reinforce the relevance of this material to the curriculum.
2. Learning Process Issues
  • Given that the context of learning has changed so significantly, are there any noticeable differences in the way students now use this resource?
  1. Anonymous written questionnaire collected from students on completing program
  2. Observations of student use of particular screens were noted
  3. General observations of student progress through the program were compared with previous student difficulties.
  4. Author attended tutor briefings before and after the relevant 'Problem of the Week' to monitor how the program was presented to students and tutors, and to gain tutor feedback about student comments.
  5. Online student noticeboards were monitored for any feedback directly from students.
All questionnaire responses were recorded and collated with observations, both by the author and tutors.
Students enjoyed the model building exercise, but most found it very challenging, and frustrating to get started.

Students were confused about the terms used for naming regions of the brain

Students did not use the practice screen for its intended purpose

Students were confused about the concepts of excitation / inhibition

Students forgot the colour codes and graphical represent-ation of the tools

Most students did not notice changes in rate of signalling

Students are confused what to do when finished building their model

The built in case study was not viewed by students

Action Cycle TwoData Collection and AnalysisKey findings
1. Program Usage / Interface Issues
  • Have the new names for the brain regions resolved the confusion about where different nervous systems arise?
  • Did student use of the practice screen increase?
  • Do students notice the changes in rates of signalling in their circuit?
  • Has student viewing of the case study increased?
2. Learning Process Issues
  • Does student use of the tutorial help explain the concepts of excitation / inhibition? Do students now read the hyperlinked definitions more? How do they use these?
  • Did student use of the practise screen better prepare students for focussing on the content of the tutorial later?
  • Does the new Summary screen help in highlighting the major learning issues of the tutorial?
  • Does viewing of the Case Study screens help put the aims of the CD-ROM into context?
3. Learning Outcome Issues
  • Do students now understand the different ways of achieving negative feedback?
  • Does student use of the CD-ROM improve their understanding of the key learning objectives?
  1. Questionnaires and observations as described in 1-3 above, but focussing on the impact of change made since Cycle 1.

  2. Students completed a short (3 MCQ) test prior to and after completing the CFL - (tests matched but anonymous) All questionnaire responses were recorded and collated.
Pre- and Post- CFL tests were marked and statistically analysed. (Wilcoxon Matched Pairs Ranks test).
Students still found the model building challenging, but achievable, and the level of frustration observed was much reduced.

Most areas of confusion identified in Semester 1 have been resolved by the modifications introduced.

Students are still experiencing difficulty with the concept of inhibition, (particularly the removal of inhibition) but the questions are now more advanced than previously.

Student performance on the tests showed significant improvement on Questions 2 & 3, but Q1 appeared to be too easy! (most got it right before and after the CFL).

(See Table 4 for test results)

Evaluation methods

Participants

This study reports on two evaluation cycles conducted during 2000.

Cycle One was conducted with second year Medical students in a PBL curriculum, in Semester 1, 2000. This CFL was recommended as a self directed learning resource, and the curriculum provides the opportunity for students to select this resource at a time convenient to them. Efforts were made to encourage students to attend a voluntary CFL session, but only 49 students (out of an enrolment of 180) attended during these times. It is possible that some students completed the tutorial at other times, but feedback from PBL tutors indicated that most students did not complete the resource at all.

Cycle Two was conducted in two stages, firstly with 82 second year BioMedical Science students in August, and then with 170 second year Science students in October. Attendances and rates of questionnaire returns are given in Table 2.

Table 2: Attendance and questionnaire response rates for Medical students
(Cycle one evaluation) and BioMedical Science and Science students (Cycle two evaluation).

CourseMedicineBioMedical
Science
Science
Enrolment18082170
Attendance at CFL session4975125
Number of questionnaires returned49
(100% of attendees)
75
(100%)
124
(99%)

Written questionnaire

All students (from all courses under investigation) were asked to complete an anonymous, written at the end of their CFL session. Questions included 5-point Likert scales of student perceptions of understanding of the topic, both before and after the CFL, and specific questions on student use of particular areas of interest, as identified in previous evaluation cycles. The questionnaire also included many open ended questions on areas of greatest and least difficulty, and relevance to the course.

Of the 14 questions on the survey, 8 questions focussed on interface or usage of the program, 2 on the learning process, and 3 could be classified as self reported learning outcome questions. The final question asked for suggestions for improvements.

The same basic questionnaire was used in both action inquiry cycles, with the only alterations being on relevance or appropriateness to the curriculum, allowing for differences in the two different types of courses being investigated. All student responses were transcribed, and collated into similar categories.

Observations

One of the co-authors was present at every CFL session using the CD-ROM tutorial, and also encouraged tutors to pay attention to areas of particular interest. Observations were collated with results from the questionnaire analysis for each student cohort, and a report circulated to the project group, other interested academics and all tutors involved in the study. For each cycle of the evaluation, the observations focussed on the key evaluation questions for that cycle.

A major development aim of the project, involving both the computer tutorial and the accompanying paper Tasks sheet, was to encourage group discussions (both student-student and student-tutor) of the problems presented. Accordingly, observations also focussed on the level and quality of discussion in the classroom.

The author was introduced to students as the Educational Programmer in Physiology and as the developer of this CFL, and was performing evaluation on this project as a participant-observer. It was made clear that she had no involvement in student assessment, but also has a strong background in Physiology, and was effectively acting as a senior tutor during all evaluation sessions.

Pre- and Post- CFL tests

A short test comprising 3 multiple choice questions was given to students at the start and again at the end of the CFL session (Semester 2 cycle only). Tests were anonymous but coded, to allow matching of pre- and post- results. Tutors were instructed not to assist students with this test, but full answers were provided in the week following this CFL session. Each question was designed to cover a particular learning objective, which had been previously identified as an area of difficulty. Question 1 was on the definition of inhibitory neurones, Question 2 was on ways of achieving negative feedback, and Question 3 was on regulation of blood pressure following haemorrhage.

The shortcomings inherent in pre- and post-tests is acknowledged by the authors, but this was never intended to be a standalone evaluation strategy, rather it provided additional information to be used together with data from observations and student feedback.

Evaluation team

The project group consisted of 5 members of staff of the Department of Physiology. All took active roles in development of the tutorial and are listed as authors of the CD-ROM program. The members can be described as: Other members of staff were consulted as required.

The evaluation was undertaken primarily by the educational programmer-developer of this tutorial. Inherent bias was minimised by extensive consultation with an external mentor (co-author of this paper), as part of a broader national network of mentors and mentees. An evaluation plan was submitted to an electronic forum, and evaluation strategies from other similar projects were consulted.

The primary stakeholders for the evaluation were the developer and project group, mentor and the ASCILITE and CUTSD network. In addition, reports were also circulated to teaching academics and tutors, and to curriculum coordinators.

Findings and discussion

First cycle

As discussed previously, this program was a recommended self directed learning resource in the new Medical course, available for students to use at their own convenience. Attempts to encourage students to attend a scheduled evaluation class using this program were largely unsuccessful, with only 49 out of 180 students attending. Those students who did attend the CFL session are likely to be amongst the most keen students, but their attendance could have resulted from a variety of reasons. Students who did attend rarely worked in groups, despite encouragement from tutors to do so.

All students attending the session completed a written questionnaire at the end of the session, giving a response rate of 100% for this group of students, and observations of student use of particular screens were noted. Attention of this cycle focussed predominantly on issues related to program use and interface design, particularly in relation to modifications introduced as a result of a previous cycle (Weaver et. al, 1999).

We were especially interested in any effects of the changes introduced in an attempt to reduce the reported high cognitive load associated with the model building exercise. Several modifications were made in an attempt to reduce this load, the key one being the introduction of a practice screen to encourage familiarity with the tools and animations, etc, involved. This cycle of evaluation concentrated on student use of these new features, and attempted to determine their effectiveness.

Student usage and interface issues
Analysis of the questionnaire, combined with observations and tutor feedback, revealed that students were still experiencing a great deal of difficulty and frustration with building their model, and amongst other findings, were not using the practice screen in the intended manner. Participation in group discussion was low, as most students worked through the program alone. The key findings are summarised in Table 3.

Students reported the best understood parts of the program were in the roles of different efferent neurones, but the number of responses for each different answer to this question was low. Least understood was the naming of regions of the brain, followed closely by problems understanding the concept of inhibition ("Do inhibitory interneurones increase or decrease signal, or have the same amount of signal but different neurotransmitter?"), both of which accorded well with our own observations. Most liked were the animations and visual representations, and the hands on approach of constructing their own model. Least popular was the perceived lack of explanations - many students asked for more directive hints ("Just tell me what to do") and more text they could copy into their workbooks.

Learning process issues
Our second focus of this cycle of evaluation was to determine whether there were any noticeable differences in how students use the program in the new PBL course. Due to the low level of student attendance, it was difficult to make any conclusions about how student use of this resource may have changed. The questionnaire did not ask questions on reasons for attendance at the CFL session or completion of the tutorial, as we had expected the majority of students to use this resource, but this does signal an issue of concern for self directed learning curricula. This resource does not appear to be seen by students in this curriculum as important, and this has been conveyed to the relevant coordinators, but may also indicate a problem with the construction of the PBL course.

Table 3: Summary of design issues identified in Cycle One evaluation
and relevant action taken prior to Cycle Two evaluation.

Design issues
(from Semester 1)
Modification introduced
(for Semester 2)
Brain regions:
Students experienced difficulty with the functional names given to regions of the brain- due to our use of terms which are also applied elsewhere.
The brain regions were re-named, keeping functional terms, but avoiding terms which are also used to describe the end effects of this system.
Inhibition:
Students were confused about the definition of excitatory vs inhibitory - what exactly makes a neurone inhibitory? We have not encountered this problem before, but this confusion is highlighted by the problem of naming of brain regions.
Existing definitions of excitation and inhibition were clarified, and we attempted to increase the viewing of these definitions by making the hypertext definitions appear automatically at the end of a popular animation, as well as by the existing link.
Practice screen:
This screen was introduced to reduce the large cognitive overload experienced by students starting on the model building exercise. The current evaluation found that students were not using these practice features built into the CFL - mainly because it was not clear that they could do so.
The title and instructions of the practice screen were changed, to highlight that this is for practising using the tools.
Colour and graphical representation:
Some students reported that while they were building their model, they forgot the colour coding / graphical representation used to depict different elements of the nervous system. (It was hoped that use of the practice screen would familiarise them with this, but these students may not have used this screen.)
A box showing examples of the nervous system elements was included on the practice screen, and same box was also included on the main model building screen, accessible by clicking on an 'information' button.
Rate of signalling:
Most students did not notice the changes in rate/ number of signals (action potentials) moving around their model, adding to the confusion about inhibition, since they did not notice when the signal increased or decreased. Also, they were unaware that they tested their circuit by increasing the original signal (increasing blood pressure at the receptor site) - the button they clicked to do this was named "Test Circuit", so gave no indication of what this test was.
The first time the model is tested and an animation is run, a dialogue box appears to prompt students to consider the rate/ magnitude of the signal moving around the circuit.
The "Test Circuit" was renamed "Raise BP", and clearer explanations provided about how the circuit is tested.
Finishing:
Students are confused what to do next when they reach the end of the model building exercise.
A new screen ("Summary") was included, after the model building exercise, to summarise the major points of the tutorial.
Case study:
A case study (3 screens of a large cartoon image and small amount of text) had been included since the last version, and was intended to give some applied context to the main learning issue. The current evaluation revealed that very few students even saw this case study.
The case study was moved to be the first item in the Contents menu, and screens were re-positioned, so that students who do not use the Contents menu to navigate (ie. those who just click on the right arrow to proceed through the program linearly) will still view the case study.

26 out of 49 respondents thought the CFL was relevant to their Problem of the Week, and a further 5 students believed it was important material for their course, although not directly relevant to the Problem of the Week. Medical students rated their own understanding of this topic prior to the CFL as 2.8 (1 = understood topic not at all, 5 = understood very well), and this improved to 3.8 after the class.

After analysis of the Semester 1 data, the project group met and decided on a range of modifications to the program. Most of the issues raised by students were considered to be related to various aspects of the program or interface design, and could be addressed by relatively simple re-design of the program. These responses are also summarised in Table 3.

Second cycle

Cycle Two was conducted during Semester 2, with BioMedical Science students in August, and then with Science students in October. Two different courses were surveyed, as this resource is now being used by a variety of several different courses, and this also extends the range of student responses collected. BioMedical Science is a new course at The University of Melbourne, and this is the first time we had taught this course to these students. Entry scores into this course are high, and we expected these students to be higher achievers compared to the Science students. Both courses are traditional lecture based curricula.

The same evaluation methods were used by both groups, and included a similar questionnaire as previously, observations and tutor feedback as well as pre- and post-CFL tests, in an attempt at evaluation into learning outcomes. All students were encouraged to complete the questionnaire, and no students refused to do this (one Science student left the classroom prior to completing the tutorial) (see Table 2). Again, observations focussed on the effects of modifications introduced since the previous cycle, particularly on whether student use of introductory material and the practice screen was improving their ability to get started on the model building exercise, but also attempted to gather evidence on learning process and learning outcome issues.

Student usage and interface issues
Results from the second cycle of evaluation were very positive. Problems identified earlier with different aspects of the interface or use of the program appear to be largely rectified. Nearly all students used the practice screen in its intended manner, and fully completed the short model building exercise contained there. This appeared to have the desired effect of reducing the cognitive load previously found with the major task of the program, and was reflected in generally much more positive statements about the program from the students. The most popular aspect of the program was again the animated model building exercise, and the high level of interactivity ("Building the circuit myself, rather than just reading and answering questions, required more thought and really improved my learning"). Responses to least popular were much less than previously seen, with the highest number of responses being "Nothing", followed by "Not enough explanation".

Learning process issues
Overwhelmingly, both groups of students believed the CFL was relevant and appropriate for their course ("helped reinforce what was covered in lectures and extended some of the topics covered"; "I wouldn't have understood the lecture material without the tutorial"; "Pitched at a level that can be understood in second year"). Most students also thought it helped to integrate material from different lectures or different subjects ("Autonomic nervous system and cardiovascular physiology were done separately in lectures so this was good to integrate them"; "Covers a number of topics in both our core subject and also aids in consolidation of material from Pharmacology").

Classroom discussion had moved from the technical difficulties of using the program or interpreting the feedback statements, to more physiological investigation of the reflex circuit, and tutors all reported that they were challenged more than they had ever been to resolve wider ranging areas of discussion, as students attempted to integrate the topics covered in this CFL with knowledge from other sources. Other areas of difficulty previously identified (eg. confusion over naming of brain regions) were not reported and so were no longer apparent.

Learning outcome issues
Students reported the best understood areas of the program related to the overall physiological responses and control of blood pressure ("I now understand the effects of changes in blood pressure on sympathetic and parasympathetic activity"), which is a major aim of the innovation. Least understood was the role of the inhibitory interneurone, particularly the concept of decreased inhibition producing an increased response. Observations by both the author and tutors revealed that no students claimed negative feedback would over correct blood pressure beyond its set point, a common student misconception reported by academic staff teaching in previous years. BioMedical Science student rating of their own understanding of this topic increased from a pre-CFL rating of 2.2 (1 = understood topic not at all, 5 = understood very well) to a post-CFL rating of 3.7. Science students' perception of their understanding increased from 2.4 (pre-CFL) to 3.5 (post-CFL).

Pre- and post-CFL tests

Answers to the three multiple choice questions were analysed in an attempt to determine whether the CFL session had achieved an improvement in student understanding of the key concepts. Each question had been designed to highlight a particular learning objective. Results of the analysis of these tests are shown in Table 4. It was recognised that any improvement in a test score may not arise purely from the CFL session itself, but that the pre-CFL test may focus student attention on the particular areas, which would not happen if no test was conducted (note that the CFL tutorial was designed to increase group discussion, both amongst peers and with tutors, so it is impossible to separate any improvement in students' understanding arising from the tutorial from improvements arising from discussions, etc).

A difficulty in analysis arose because there were 2 correct answers to question 2. This was deliberate, as one of the key learning objectives of this program is to identify and understand the two different methods of negative feedback, and we were interested to see whether students understood that there were two mechanisms involved. However, where students selected only one correct option, it is impossible to tell whether they believed there was only one mechanism, or whether they did not read the (bold) instruction at the top of the test that there may be more than one correct answer. The following results are based on students receiving 0.5 marks for one correct answer, and only receiving the full mark for both correct options.

Question 1 was not difficult to work out, or at least take an educated guess at, and this is reflected in the group results - a high number of students answered this correctly prior to the CFL, so it was difficult to see a significant improvement in performance. Performance on Question 2 improved similarly in both courses, with the BioMedical science students generally performing better both prior to and after the CFL, but showing about the same level of improvement.

Table 4: Results of pre- and post-CFL tests (using Wilcoxon Matched Pairs Ranks Test).
Each question was worth 1 mark. Results given are the average mark for
each question, and the average score (out of 3) for the whole test.

Biomedical Sci
- Aug 2000
Pre-CFLPost-CFLSignificance
Question 10.85 ± 0.36
(n = 75)
0.89 ± 0.31
(n = 75)
N.S.
(p = .496, n = 75)
Question 20.58 ± 0.26
(n = 75)
0.77 ± 0.25
(n = 74)
p < 0.001
(n = 74)
Question 30.36 ± 0.48
(n = 75)
0.49 ± 0.50
(n = 74)
N.S.
(p = .185, n = 74)
Whole test1.79 ± 0.66
(n = 75)
2.13 ± 0.59
(n = 75)
p < 0.005
(n = 75)
Sci - Oct 2000Pre-CFLPost-CFLSignificance
Question 10.84 ± 0.37
(n = 124)
0.90 ± 0.31
(n = 115)
N.S. (p = .18)
(n = 115)
Question 20.48 ± 0.27
(n = 123)
0.67 ± 0.34
(n = 117)
p < 0.001
(n = 116)
Question 30.24 ± 0.43
(n =1 23)
0.50 ± 0.50
(n = 115)
p < 0.001
(n = 114)
Whole test1.56 ± 0.58
(n = 124)
2.03 ± 0.59
(n = 117)
p < 0.001
(n = 117)

Performance on Question 3 was lower than expected, given that it is very similar to a question on the Tasks sheet which students had been working on. It was observed that a few groups of students did not know the definition of haemorrhage - some said it was a bruise, others used the word clot. The question on the Tasks sheet refers to blood loss, so the different term used may have led students to believe this was a completely different question. But since most students completed the test without discussion, as instructed, it is impossible to conclude where they had difficulty with this.

From observations only, we tend to believe that the CFL session itself was responsible for the improved performance on questions 2 and 3, since we did not observe students referring back to the questions during the class, or discussing these topics more than in similar sessions without tests, even though they had been informed they would be undertaking the same test at the end of the session. Even when specific reference was made in discussions with tutors to points which had been included as options on the multiple choice questions, no students commented on this or seemed to recognise these from the test. However, further evidence is needed before any conclusions can be reached on the effect of the tutorial in meeting the learning objectives.

Implications for evaluation

Both stages of Cycle Two of the evaluation study revealed that student frustration with building the model has been reduced and the desired student learning outcomes appear to have been achieved, although the reasons for improvement have not been clearly ascertained. Students generally enjoyed using the tutorial and believed that their understanding of the subject matter had increased. In a separate Quality of Teaching survey conducted by the University over the whole of Semester 2, students rated this CD-ROM the best of all the CFL sessions they have undertaken in this course. The features most popular were the interactivity, or hands on approach, and the animated responses to represent physiological mechanisms operating. Students reported that concentrating on the concepts involved, rather than the quantitative detail, helped them achieve a deeper understanding ("Helped to visualise the concepts and apply them to problem solving"), and improved their recall of the concepts later ("Easier to understand and remember concepts if you put them into practise"). Student discussion of these learning issues was greatly increased when group work was encouraged (eg. in smaller computer laboratories, where computers must be shared). One Medical student who first completed the CFL working alone, and then repeated the tutorial in a group setting, wrote that "Having demonstrators and other students to discuss the questions was great - I understood the model much better after discussing it than when I was alone".

The original intention was to look at the CFL tutorial and its immediate context, and not at the whole curriculum, however, this evaluation project did identify some issues about the broader context in which these types of application can be used.

The process of action inquiry allowed us to evaluate different versions of the CD-ROM, not just for interface or useability issues, but on the whole learning experience it offered to students, and to systematically identify areas of concern, address these issues in an appropriate manner, and to focus further evaluation rounds on the impact of these changes.

We are seldom able to anticipate all the different ways in which students relate to CFL, consequently an evaluation framework must be integral to the use of CFL in teaching today. Such a strategy needs to be ongoing throughout any project, adopting a cyclic method geared towards continuous improvement in a systematic way.

Clearly, there was considerable value in the iterative evaluation of this CFL resource as we were able to better understand how students related to this resource and made use of it in their learning. We were unable to fully anticipate the way students understand the software interface or the context in which it is used. Iterative approaches to evaluation, like the one we have employed in this project, can greatly improve the quality of the resource and its use.

Future directions

Several Physiology Departments at other institutions, both national and international, have been trialling this CD-ROM with their students, and feedback has been positive. It is also being trialled with students from different disciplines (eg BioMedical Engineering). The CD-ROM in its current form was released for commercial sale in December 2000 (Weaver et. al, 2000). It would be useful to employ the iterative evaluation strategy applied in this project with feedback from further use in other institutions in an attempt to achieve a greatly improved product and understanding of how to effectively use it medical education.

References

Cooper, G. (1990). Cognitive load theory as an aid for instructional design. Australian Journal of Educational Technology, 6(2), 108-113. http://www.ascilite.org.au/ajet/ajet6/cooper.html

Kember, D. & M. Kelly (1993). Improving teaching through action research. HERSDA Green Guide No. 14.

Papert, S. (1996). The connected family - bridging the digital generation gap. Longstreet Press, Atlanta, Georgia, USA.

Phillips R., Bain J., McNaught C., Rice, M. & D. Tripp (2000). Handbook for Learning-Centred Evaluation of Computer-Facilitated Learning Projects in Higher Education. ASCILITE / CUTSD publication. [verified 24 Jul 2002] http://cleo.murdoch.edu.au/projects/cutsd99/handbook/handbook.htm

Vygotsky, L. (1962). Thought and Language. MIT Press, Cambridge, Mass., USA.

Weaver D., Kemm R., Petrovic T., Harris P. & Delbridge L. (1999). Learning about control systems by model building - A biological case study. In Responding to Diversity. Proceedings of ASCILITE 99 Brisbane, Qld, pp. 381-389. http://www.ascilite.org.au/conferences/brisbane99/papers/weaverkemm.pdf

Weaver, D.A, Delbridge, L.M.D., Harris, P.J., Petrovic, T. and Kemm, R.E (2000). Blood Pressure: Reflex control. Pub. AD Instruments, Sydney. ISBN 0 7340 2085 6

Authors: Debbi Weaver, Department of Physiology, The University of Melbourne
Tony Gilding, Teaching and Learning Development Support, James Cook University

Please cite as: Weaver, D. and Gilding, A. (2002). Iterative evaluation of Reflex Control of Blood Pressure. Australian Journal of Educational Technology, 18(2), 208-226. http://www.ascilite.org.au/ajet/ajet18/weaver.html


[ AJET 18 ] [ AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet18/weaver.html
Created 16 Aug 2002. Last revised 27 Aug 2002.

Page access count