Figure 1: A screen from the package, showing the rollover audio, the video
of the patient and feedback and questions used to prompt students.
| Australian Journal of Educational Technology 2000, 16(2), 104-125. |
AJET 16 |
This paper reports on the development rationale and evaluation of a computer facilitated learning (CFL) package which aimed to assist medical students with their clinical communication skills and develop an integrated biopsychosocial approach to identifying a patient's problems. Using a formative evaluation framework developed at the University of Melbourne the CFL package, Communicating with the Tired Patient, underwent a three stage review. Initial evaluation consisted of both formal and informal conceptual and technical review by content experts, instructional designers and evaluators. The final stage of the evaluation involved the survey and observation of 110 medical students interacting with the package. Students were very positive about the instructional and interface design of the package and indicated that the package assisted with their understanding of issues associated with clinical communication. Nevertheless, a number of areas were highlighted where either the package or the learning environment could be modified or improved.
A second aspect of a clinical interview, or any interpersonal interaction, is the communication microskills used by the participants. Microskills refer to the interviewing strategies that are used to help facilitate a successful interview and to establish rapport between the interviewer and the interviewee. Microskills may be verbal or non-verbal. Verbal strategies include using different types of questions (open, closed or leading), reflecting, paraphrasing, minimal encouragers and using different tones of voice. Non-verbal strategies include posture and facial expressions, and encouragers like nods and eye contact. A skilled clinician listens to verbal responses and observes the non-verbal behaviour of the patient and is able to monitor how comfortable and forthcoming a patient is and alter his or her interview accordingly.
Thus, the computer facilitated learning (CFL) package Communicating with the Tired Patient had two overarching aims. First, it aimed to help students develop an understanding of the need to consider biological, psychological and social factors when conducting a clinical interview and ultimately in their identification of a patient's problems. Second, the package challenged students to reflect on an array of microskills used and a range of verbal and non-verbal behaviours exhibited, by both the doctor and the patient, in a clinical interview.
In the package students are asked to play the role of the doctor in a clinical interview. As the doctor students are given a number of options regarding the types of questions they are able to ask the patient which reflect different microskills common in clinical interviews. Students are asked to listen to the different questions and to select one they would like to ask the patient. Students are able to see the ramifications and implications of asking different questions by listening to and observing the patient's video response. Consistent with the concept of a virtual interview, the patient responds directly to the student, as in Figure 1.
Figure 1: A screen from the package, showing the rollover audio, the video
of the patient and feedback and questions used to prompt students.
To assist them with their reflection, questions which are specific to the last audio-video segment pop-up on screen which students can answer in a workbook which supplements the package. Students are encouraged to comment on what they noticed about the patient's verbal and non-verbal responses and are challenged to think about the direction in which their interview is heading. In addition to considering the microskills which are reflected in students' choices, students are asked to reflect on the possible causes of the patient's presenting complaint. This process aims to promote students' understanding of the biopsychosocial aspects of a patient's presenting problems.
The development team saw a number of advantages in developing the package in this way. First, students not only actively participate in the interview with the patient but they actively construct it as well, controlling both its tenor and direction. Each student is able to select the questions, options and pathways in a clinical interview which make sense to him or her. By interacting with the package in this way, it is hoped that students will build up new understandings in the content area based on their current-and often implicit-knowledge of appropriate interviewing methods. Kennedy and McNaught (1997) suggest that one of the critical challenges facing educators is "enabling passive learners to use more active approaches to learning and new knowledge construction..." (p. 391). Communicating with the Tired Patient aims to confront this challenge head on by actively involving and engaging students in the learning experience rather than simply presenting information for them to absorb. The theoretical underpinnings of the package are, therefore, consistent with a constructivist philosophy of courseware design and development.
A second important aspect of the learning design of the module was the promotion of critical reflection by students. Boud, Keogh and Walker (1985) define reflection as "those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations" (p. 19). Schon (1983) describes reflection-in-action as analogous to "thinking on your feet" or "learning to adjust once you are out there" (pp. 54-55). It is reflecting in the midst of an operation rather than at a post-mortem. The structure of the package meant that students were involved in a process that required them to critically reflect on their decisions and tacit understandings during and after their virtual interview. The package helped students, both implicitly and explicitly, to reflect on their choices. After students have selected a particular doctor's question they are able to see the implications of their selection through the audio and video response of the patient (implicit feedback). In addition, students are supported with specific feedback and are challenged with further questions about their selections (explicit feedback). Through reflection students are able to critique how they have gone about their learning experience and identify areas where these processes could be improved (Koschmann, Kelson, Feltovich, & Barrows 1996; Schon 1983). By returning to their learning experiences, attending to their feelings and re-evaluating these experiences, students are able to associate and integrate information. This will hopefully foster greater understanding (Boud et al, 1985).
Another advantage of the development team's approach was that through the use of audio and video and by placing the student in the role of the doctor, the developers attempted to create a realistic scenario. Using the model of situated cognition, the developers recognised that the media of audio and video could be used effectively to place learners in a real life context (Brown, Collins & Duguid, 1989; Herrington & Oliver, 1995, 1997). Brown, Collins and Duguid (1989) argue that "useful learning" takes place when students are set authentic tasks or are placed in authentic contexts. They argue that "much school work is inauthentic and thus not fully productive of useful learning" (p. 34). The challenge for multimedia developers, therefore, is to create learning activities and contexts which are authentic and maximise the potential for student learning and facilitate the application of what is learned. The developers attempted to meet this challenge with Communicating with the Tired Patient.
Details of this program are reported elsewhere (Kennedy, Petrovic & Keppell, 1998; Kennedy 1999a, 1999b), and will not be covered in detail here. However, the program has a number of key elements:
The major concerns during this phase of the evaluation were reducing the size of the project so that it was manageable both conceptually and technically. A number of scripts were produced by the content experts for the proposed clinical interview case studies, showing the details of interactions between doctors and patients. There was a need to keep scripts as simple as possible without undermining the learning objectives. Scripts of relatively simple interactions were produced because of technical and storage limitations as well as the target audience of first year students. Once working versions of these scripts had been established, estimates were made of the file sizes associated with each of the clinical interviews. Given the extensive use of audio and video in the project it was decided that cross-platform CD-ROM was the most appropriate delivery mode. Due to the proposed number of clinical interview case studies, the length of these interviews and the large file sizes associated with capturing these interviews on audio and video, it was clear to the development team that compromises were needed in terms of production. These compromises were manifest in two ways. First, the number of clinical interview case studies that were initially proposed was reduced. Second, it was determined that video and audio would be produced for both the doctor and the patient for the reduced number of clinical interviews but the development team would initially focus on producing two of these as interactive learning modules.
A number of major modifications were made to the package as a result of this evaluation. These modifications involved providing more support for students, both in terms of functionality and educational goals. In terms of functionality, the evaluator and content expert agreed that students may not be clear about how to negotiate their way around the package. The evaluation recommended more specific and clearer instructions be provided in the introduction. It was also recommended that students be provided with more specific on-screen instructions when selecting a doctor and a patient for the interview and when students were required to operate the audio functions.
In terms of educational support, it was thought that the package was in danger of being too open and lacking an educational direction. There was a clear tension between adopting a constructivist development approach on the one hand and providing enough instructional support for students on the other. It was thought that if the package remained in its current form it would not be clear to students what the purpose of the package was. At this stage it was decided that students would be prompted to reflect on the specific audio-video interview segments they had chosen. The content expert provided an array of supporting questions to fulfil this function. For example, after a particular audio-video sequence a question would pop-up on the screen: "You asked Mrs Nacarella a closed question in a business like fashion. Was the response what you expected?" It was decided at this stage that a workbook containing all the questions available in the package would be used to supplement the package.
In addition to these modifications a number of minor amendments were made to the package at this stage. A conclusion was included in the package to provide students with a summary of the biopsychosocial implications of the patient's presenting complaint. A number of technical glitches, such as audio not being activated by rollovers and poor screen transitions, were noted and subsequently rectified. In one section (the glossary) the readability of the text was improved. Finally, areas of the package which were included in the initial interface design but currently lacked functionality (the tutorial and a number of clinical cases) were either removed from the interface or were clearly labelled as "under construction".
| Response | Frequency |
| The video was fine/ good/ improved the package | 54 |
| The audio was fine/ good/ improved the package | 49 |
| The audio was too soft/ unclear | 29 |
| The video was too small | 9 |
| The questions were too similar/ repeated too often | 8 |
| You should be able to go back and repeat video showings | 8 |
| The interview was not sequential/ in the wrong order | 4 |
| The video was blurry | 3 |
Students were also asked to rate the quality of the audio and video on nine-point bipolar scales where '1' indicated high quality and '9' indicated low quality. Mean scores for the audio and the video indicated students thought that generally both were of high quality (audio mean = 4.08, SD = 2.34; video mean = 3.25, SD = 1.80). However, the high standard deviations, especially for the audio, indicated there was a high degree of variability in these responses. Paired T-tests revealed that students were more positive about the video than the audio (t (108) = 3.94; p < .001).
Technical difficulties were experienced with volume control when the package was installed on computers in the computer lab. As such the audio problems experienced were not a result of the package per se, but resulted from its integration into the learning context. Students may have had difficulties hearing the audio for one of three reasons: their headphones were plugged into the wrong audio socket, the sound control software was set too low, or the mute button was on. Students were often not experienced enough with the computers to rectify this situation on their own.
While content experts were not asked specifically about the audio and video, two mentioned its high quality. Content experts also noted the use of audio and video was excellent for creating a "real-life" scenario and for training students to listen to and observe their patients. Student observation supported this conclusion. Students spent a lot of time reviewing the audio options and some students closed their eyes when listening to the audio in order to focus on the difference between the audio options. One student was heard to say "No, no, no. I don't want you to say that!" after making an audio choice that she was unhappy with. Students were also observed concentrating fiercely on the video responses of the patients in order to pick up both verbal and non-verbal cues. These observations suggest that the audio and video were useful in engaging students in the content of the package. This is reflected in the two most common comments by students regarding the best aspects of the package which were the video (46) and the audio (36) (see Table 5 below). It seems, therefore, that other than the technical difficulties surrounding sound quality, that the audio and video were well received.
| Response | Frequency |
| The workbook and the program worked well together/ they were easy to use together | 42 |
| Negative comments about the workbook questions | 25 |
| The workbook assisted learning/ helped direct attention/ was stimulating | 18 |
| Positive comments about the questions | 9 |
| The workbook interrupted the flow of the program | 7 |
| There was no need to write down so much in the workbook | 6 |
| It was difficult to know which question you were dealing with | 5 |
| It was difficult to switch between the workbook and the program | 4 |
| It was too confusing | 3 |
| The workbook left too much space for answers | 3 |
Relatively few students thought the workbook affected the continuity of the program (7) or had trouble moving between the workbook and the program (4). When asked explicitly about moving between the workbook and the program the vast majority (72.7%) indicated they had found this easy with only around thirteen percent of students (12.7%) indicating this presented them with difficulties. Observations suggested that students were, on the whole, not having too many difficulties negotiating transitions between the package and the workbook. On occasions, students would be distracted from the screen when they looked for the next question in the workbook or reviewed their previous answers. The concern that the attention of some students may sometimes be divided between the workbook and the interview may be rectified by allowing students to "rewind" their interview. This is something the students themselves requested (see below).
Students were, however, critical about the nature of the questions in the workbook. The most common criticism was that questions in the workbook were too repetitive or monotonous (12). This is consistent with students' comments about the package structure generally (see Table 4 below) which is not surprising given students use of workbook is contingent on their path through the package. Other comments were that the questions were too difficult (7), were not in sequence (3) or were ambiguous (2).
| Response | Frequency |
| The problematic nature of the Doctor's questions | 40 |
| The options or paths were too similar or limited | 24 |
| The sound/ volume was poor | 21 |
| There was no going back to repeat questions | 16 |
| The interview lacked continuity | 12 |
| It was too boring, long or slow | 12 |
| Technical problems or bugs | 7 |
| The screen was too small | 6 |
| It was too confusing/ there was no familiarisation | 6 |
| You couldn't see the Doctor | 5 |
The second most common comment about the doctors questions was that the questions were too similar (8). This should be seen as related to the second most common complaint about the package generally which was that the options or paths that were available were either too similar or too limited. This raises two related issues. First, students found it difficult to discriminate between the types of questions that the Doctor was asking in the virtual interview; and second they therefore saw themselves as having few options with the kinds of questions they could ask. The issue of question similarity was mentioned by the content experts. One content expert thought that the "nuances in the way the questions were read out may not be of big enough difference for overseas students to pick up" while another said "some questions were too similarÉstudents can't make a choice". The latter content expert suggested labelling questions to "aid learning and selection". This may, however, undermine a fundamental goal of the package which is to assist students recognise and discriminate between different types of verbal questions and responses. By labelling questions students would effectively be told what type of verbal response is being given. This issue will be explored further in the discussion.
Other common comments about the worst aspects of the package were technical issues related to sound and volume (16), the lack of a "back" or "rewind" option (16) and the difficulty with continuity (12). This latter comment is reflected in the statistic that one-fifth of students (20.9%) indicated they found the flow of information from screen to screen confusing. While this may be because of workbook distractions or technical difficulties it seems that the open structure of the package resulted in a lack of coherency in the "virtual interview". The major problems seem to be that questions were revisited, the sequence of information in the interview was not logical for students, and the information revealed in the interview could be quite disjointed depending on the paths chosen. A number of students also found the package long, boring or slow (12). This may be related to the fact that many students found the package repetitious or that their questions or options were too similar.
| The package generally ('1' = not at all; '9' = extremely) |
M | SD |
| Engaging | 5.88 | 1.51 |
| Appealing | 6.03 | 1.58 |
| Confusing | 3.44 | 1.87 |
| Interesting | 6.31 | 1.60 |
| User-friendly | 6.93 | 1.54 |
| Valuable | 6.32 | 1.67 |
| Boring | 4.02 | 1.89 |
| Enjoyable | 5.95 | 1.56 |
| Logical | 6.16 | 1.80 |
| The interface generally ('1' = poor; '9' = excellent) |
M | SD |
| Useability | 6.67 | 1.38 |
| Clarity | 6.57 | 1.62 |
| Structure | 6.05 | 1.83 |
| Appeal | 6.20 | 1.75 |
Another indication of students' positive attitude towards the package (and the possible reasons for it) was from responses to the statement "Please note what you thought were the three best aspects of the package". A summary of the ten most frequent responses is presented in Table 5. Aside from the audio and video, which has been mentioned above, students thought the interactivity, the options and paths available and the ease of use were the most positive features of the package. Other comments centred on the content and the ability to observe the patient's body language. It is worth noting that some features of the package which were regarded as the best features by some students, were regarded as the worst features by others (eg. different options, the questions).
| Response | Frequency |
| The video | 46 |
| The audio | 36 |
| The interactivity | 23 |
| The options available and the different paths | 20 |
| The user-friendliness or ease of use | 20 |
| The content: learning about communication and interview skills | 15 |
| The good design and presentation | 13 |
| Being able to see the patient's responses and body language | 11 |
| The good questions | 9 |
| The chance to apply skills learnt in tutorials | 8 |
| Response | Frequency |
| My understanding of the use of questions, language and communication in general was improved | 53 |
| I didn't learn much new; the program reinforced material from the tutorials | 12 |
| I had an improved idea of the general process of interviewing | 7 |
| I learnt more about observing body language and non-verbal cues | 6 |
| I had an improved understanding about dealing with Non English Speaking Background patients | 4 |
| The lack of feedback limited learning | 3 |
| It was much better learning than reading from a book | 3 |
| It improved my understanding of how to approach patients | 2 |
| It showed me the importance of listening | 2 |
| It showed me the importance of empathy | 2 |
One recommendation which emerged from the alpha version evaluation related to the continuity, flow and sequence of information contained within the module. The finding that some students found the package repetitious and disjointed was of concern to the development team. These perceptions seemed to be fundamentally linked to the use of doctor's questions within the package. It was recommended that the development team review their approach to the use of audio and video segments that underpin the doctors questions and the patients responses. It was expected that if modifications were made in how these segments were used, then the way in which students created their virtual interview could be enhanced, directly addressing the issues of continuity and repetition.
Rather than dealing with relatively short segments of audio and video it was decided that longer audio-video segments, representing particular aspects of the doctor/patient interaction, be created. Each segment should make sense on its own (for example, a set of segments on sleep problems, diet or depression). By using larger segments it will be easier to organise the program so that repetition is avoided. Students will also be provided with access to a site map which will record their pathway through the interview. At any point in their interview students will be able to call up either a text or an audio-visual transcript of their virtual interview. Students will be able to review audio-video segments directly or they can access them via hyperlinks from a transcript of their interview. Thus, students will be able to use the site map to review their last few interactions or to review their entire interview. It is hoped that by providing this facility, students will reflect more generally on their interview, rather than on specific sequences. It is hoped that students will see continuity in their interview by using this function.
Considering the almost infinite permutations in variations in nuance, style, words and non-verbal behaviour of both patient and doctor and the fact that we could not incorporate past experience into the model, it is an achievement that only a minority of students felt that they had limited options or pathways within the package. A difficulty that both content experts and students mentioned was the trouble some students had in detecting often rather subtle differences between the doctor's questions. One content expert suggested labelling the questions may be a way to circumvent this problem. However, as mentioned above, specifying the nature of the questions would undermine a major thrust of the package, as it would effectively tell students what type of microskill is being employed by the doctor. Nevertheless, a number of other recommendations emerged from the evaluation. First, it was recommended that the development team take more care in the selection of the doctor's questions so that there is a clearer demarcation between question types. Second, the inclusion of a tutorial module in the CFL package was suggested which would give students audio and video examples of different types of microskills used in the interviews contained within the package. Through interactive tasks in the tutorial students could see more explicitly the advantages and disadvantages associated with using different types of questions and microskills in a variety of interview situations.
Other than these recommendations, which were accepted by the development team, it also seemed worthwhile to find a solution to the problem of question discrimination which was embedded within the virtual interview itself. A possibility that the development team is currently considering is specifying the nature of the doctor's question on the screen after students have made their initial, unguided question selection. By asking for an expert comment, students would be able to recognise the type of question they have selected on the previous screen and this information would help students appreciate the patient's response. By incorporating "rewind" and "go back" functions, students would be more easily able to investigate the implications of asking different types of questions.
While there were not too many negative comments about the interface, one content expert noticed that there was inconsistency in some of the functions of the buttons and links. The development team was aware that in the alpha version of the package there was inconsistency in whether buttons were rollovers, whether they were clickable or whether they were purely cosmetic. Some content experts expressed concern that the instructions which were given to students were not adequate, while many students requested that rewind or go back functions be incorporated into the design. The difficulties that were encountered with the audio quality indicated the need for a more user-friendly means of volume control. These results regarding the interface and graphic design led to a recommendation that the interface be redesigned. The development team, who were already planning this action, decided that the revised interface would incorporate a volume control, a rewind function and a go back function. The interface will endeavour to be more intuitive and will make a distinction between general program navigation and audio/video controls. One final, but crucial, criterion for the revised interface will be to employ consistent functionality of buttons.
Brown, J. S., Collins, A. & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32-42.
Herrington, T., Herrington, J. & Oliver, R. (1999). Providing reflective online support for preservice teachers on professional practice in schools. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (Ed-Media 99) (pp. 166-171). Charlottesville, AACA.
Herrington, J. & Oliver, R (1995). Critical characteristics of situated learning: Implications for the instructional design of multimedia. In J. Pierce (Ed.), Proceedings ASCILITE'95. University of Melbourne, Australia. [verified 12 Jul 2000] http://www.ascilite.org.au/conferences/melbourne95/smtu/papers/herrington.pdf
Herrington, J. & Oliver, R. (1997). Multimedia, magic and the way students respond to a situated learning environment. Australian Journal of Educational Technology, 13(2), 127-143. http://www.ascilite.org.au/ajet/ajet13/herrington.html
Kennedy, D. M. & McNaught, C. (1997). Use of concept mapping in the design of learning tools for interactive multimedia. Journal of Interactive Learning Research, 8(3/4), 389-406.
Kennedy, G. E. (1999a). Defining the dimensions of a formative evaluation program: A multi-method, multi-perspective approach to the evaluation of multimedia. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (Ed-Media 99) (pp. 604-609). Charlottesville, AACA.
Kennedy, G. E. (1999b). Critical reflection on a program of development evaluation. In J. Winn (Ed.), Proceedings ASCILITE'99, 165-171. Queensland University of Technology, Australia. [verifed 12 Jul 2000] http://www.ascilite.org.au/conferences/brisbane99/papers/kennedy.pdf
Kennedy, G. E., Petrovic, T. & Keppell, M. (1998). The development of multimedia evaluation criteria and a program of evaluation for computer aided learning. In R. M. Corderoy (Ed.), Proceedings ASCILITE'98, 407-415. University of Wollongong, Australia. [verified 12 Jul 2000] http://www.ascilite.org.au/conferences/wollongong98/asc-98pdf/kennedypetrovickeppel.pdf
Keppell, M. & Bennett, S. (1997). Digital Learning. A guide to multimedia and on-line development in the Interactive Multimedia Unit. Central Queensland University.
Keppell, M., Elliott, K. & Harris, P. (1998). Problem based learning and multimedia: Innovation for improved learning of medical concepts. In R. M. Corderoy (Ed.), Proceedings ASCILITE'98, 417-424. University of Wollongong, NSW. [verified 12 Jul 2000] http://www.ascilite.org.au/conferences/wollongong98/asc98-pdf/keppellelliotharris.pdf
Koschmann, T., Kelson, A. C., Feltovich, P. J. & Barrows, H. S. (1996). Computer-supported problem-based learning: A principled approach to the use of computers in collaborative learning. In Koschmann, T. (Ed), CSCL Theory and Practice in an emerging paradigm. (pp 83-124). Erlbaum.
Medical AudioVisual Communications Inc (1998). A practical guide to communication skills in clinical practice. CD-ROM, University of Texas & CPSO Canada.
Schon, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books Inc.
| Authors: Teng Liaw, Department of General Practice and Public Health Gregor Kennedy, Biomedical Multimedia Unit Mike Keppell, Biomedical Multimedia Unit John Marty, Department of General Practice and Public Health Ruth McNair, Department of General Practice and Public Health Faculty of Medicine and Dentistry and Health Sciences The University of Melbourne Please cite as: Liaw, T., Kennedy, G., Keppell, M., Marty, J. and McNair, J. (2000). Using multimedia to assist students with communication skills and biopsychosocial integration: An evaluation. Australian Journal of Educational Technology, 16(2), 104-125. http://www.ascilite.org.au/ajet/ajet16/liaw.html |