Figure 1: Simulation outputs depicting a compliant patient
| Australasian Journal of Educational Technology 2012, 28(4), 671-683. |
AJET 28 |
Challenges in integrating a complex systems computer simulation in class: An educational design research
Swee-Kin Loke, Hesham S. Al-Sallami, Daniel F. B. Wright, Jenny McDonald, Sheetal Jadhav and Stephen B. Duffull
University of Otago
Complex systems are typically difficult for students to understand and computer simulations offer a promising way forward. However, integrating such simulations into conventional classes presents numerous challenges. Framed within an educational design research, we studied the use of an in-house built simulation of the coagulation network in four pharmacy undergraduate classes. Drawing upon audio recordings of small group discussions, focus group interviews, and class observations, we identified implementation challenges related to: adaptation of simulation to align with student needs; compromises to learning design; and classroom infrastructure. These findings can serve to guide teachers and staff developers on the common challenges that are likely to arise from integrating computer simulations meaningfully into realistic contexts.
We built a computer simulation based on a model of the coagulation network (Wajima, Isbister & Duffull, 2009). The simulation was built in MATLAB (R2008a) (MathWorks, undated), a programing environment featuring algorithm development, data analysis, and visualisation. Using the simulation, pharmacy students were able to manipulate variables including warfarin dose, patient compliance, and genotype (patient's genetic sensitivity to warfarin). They could then run simulations for various scenarios and generate predictions of INR response over time (see Figures 1 and 2).
Figure 1: Simulation outputs depicting a compliant patient
Figure 1 depicts a compliant patient and Figure 2 a non-compliant patient who missed a dose on Day 5. All plots are over time with days on the x-axis. The upper panels represent the plasma concentration-time profile of warfarin, the middle panels the four main clotting factors that warfarin affects, and the lower panels the INR response.
We attempted to integrate this simulation into conventional classes to exemplify the ideals of the research-teaching nexus (Griffiths, 2004), helping students to understand the latest research findings in pharmacology and to engage in inquiry-based activities.
Figure 2: Simulation outputs depicting a non-compliant patient
Similarly in medical/pharmacy education, studies were often contextualised within controlled settings (e.g., Hariri, Rawn, Srivastava, Youngblood & Ladd, 2004). In a naturalistic setting, Wu-Pong and Cheng (1999) evaluated a teacher's (not students') use of a complex systems simulation (projected on screen during lectures).
All authors jointly designed the new workshop, with three authors taking on more specific roles: author 2 was the tutor facilitating the workshop and authors 1 and 4 were staff developers/researchers observing the workshop.
The 120 students came from four classes and each class attended one of the four sessions of the new workshop. Each class of 30 students was further divided into groups of five or six students during each session (see Figure 3). Each group worked with a laptop running MATLAB and remotely connected to the School of Pharmacy's server. The four sessions were held over two days in three different classrooms within the same teaching complex.
Figure 3: Division of groups in each class
The students undertook the workshop in three parts. In part 1, the tutor introduced the case to the students (see Appendix) and invited them to execute the first run (default parameters) to find out: (a) the time the patient took to reach the desired INR; and (b) the patient's INR on Day 10. In part 2, the students selected one of five genetic markers (which indicate how sensitive a patient will be to warfarin), predicted the outcomes for (a) and (b), and ran the simulation again. The tutor made sure all five options were covered and noted the students' predictions in a Word document displayed to everyone. In part 3, the students picked one of five incidences of non-compliance (e.g., missing two doses, halving a loading dose), predicted the outcomes for (a) and (b), and ran the simulation.
The audio data (approximately 3 hours of classroom discourse and 45 minutes of interview data) were fully transcribed. To identify the three core challenges below, we highlighted those that impacted all four sessions in the most significant ways and that other educators would most likely encounter.
To identify students' ways of thinking about the complex phenomenon of warfarin-dosing (to support assertion #2 below), authors 1 and 4 read the transcripts independently and coded student utterances illustrating particular ways of thinking based on Jacobson's (2001) clockwork-complexity categories. These categories emerged from Jacobson's (2001) study in which complex systems experts and novices were asked how they would design a large city efficiently. Complex systems experts tended to think in terms of the following complexity categories: non-linear relationships (e.g., accepting that a small perturbation in a remote area can cause a large effect in the central business district); de-centralised interactions (e.g., construing drivers, driving patterns, and models of freeways as interacting without a centralised controlling agent); multiple causes (e.g., assuming that any perturbation is likely to be caused by multiple factors); and stochasticity (e.g., construing alternative city configurations as not being completely predictable). Complex systems novices tended to think in terms of clockwork categories (e.g., linear relationships, centralised control) which were incongruent in order to understand complex systems.
Authors 1 and 4 then met to compare and negotiate our coding. Like all forms of human intelligence, these ways of thinking are dispositional (Perkins, Tishman, Ritchhart, Donis & Andrade, 2000): the workshop sessions and focus group interviews were mere opportunities that students could seize (or dismiss) to exhibit particular ways of thinking. In other words, the appropriate instantiations of these ways of thinking were more important than their frequency (more instantiations does not imply better thinking).
The "trustworthiness" (Guba & Lincoln, 1989, p. 233) of our findings was maximised in the following ways: validity was enhanced by triangulating multiple sources of evidence (i.e., each assertion reported below is supported by evidence from both workshop sessions and focus group interviews); reliability was increased by carrying out four identical workshop sessions with four different groups; and objectivity was reinforced by maintaining both insider (tutor) and outsider (researcher) viewpoints throughout the study. Any emerging assertion was tested in the entire data corpus and negative examples were actively sought. The researchers also conducted a peer debrief between the workshop sessions to share developing understandings of the study and to re-focus follow-up observations and interviews.
The majority of students in FG1 and FG2 did identify the relationship between dose, time of dose, measurement of clotting function, and warfarin effect as a key learning point of the workshop. However, some students expressed that the learning activity was a little beyond their understanding: "We've got no clue [what the exact duration to reach the desired INR is], we're just guessing" [WS4]; "We were only taught about the pharmacokinetics (i.e., relationship of dose to concentration) of warfarin? So... we just kind of guessed by comparing that to other drugs [e.g., antibiotics] that we know about" [FG1]. The difficulty in matching learning outcomes and simulation is well-documented (e.g., Davies, 2002; Moizer, Lean, Towler & Abbey, 2009) and is accentuated when teachers attempt to simplify the latest research findings for their students. Additional scaffolds to facilitate understanding will have to be considered for subsequent cycles of our educational design research.
At the point of conception of this project, we had planned to adopt Jonassen's (1999) design principles for constructivist learning environment, positioning our computer simulation as an example of "problem manipulation spaces" (p. 223) where students - in order to solve an authentic problem - formed and tested their own hypotheses and received feedback from the simulation via changes in the graphs. However, the experimentation with 'what-if' scenarios would have required an unpredictable number of runs which the time constraint did not allow. The issue of fitting more open-ended learning activities within well-defined durations is also reported in Tüzün (2007). Among the eight groups we sat with, only one managed to test out a fourth scenario on their own (to find out the threshold of missed doses before their patient's INR fell below therapeutic level) because they had started their third run earlier than the rest [WS4]. Upon reflection, many students expressed the desire to try out their own scenarios at their own time: "[Testing] what makes INR change faster" [FG1]; "Test it... until you really understood it" [FG2].
To accommodate software limitations, we redesigned the learning activity, persisting to keep "meaning open or 'performable'" (Bruner, 1986, p. 26). We asked students to predict the outcomes of one of five pre-determined scenarios and then to articulate the assumptions behind their predictions. This design decision was informed by Jacobson and Wilensky's (2006) contention that students' interactions with simulations can potentially encourage them to articulate and modify their assumptions, and by Jonassen's (1997) that students' articulation of their solutions was a good indicator of what they know.
The majority of students did appreciate that predicting made them "actively think" [FG2] and tested their understanding [FG2], without which the simulation outputs would have been "pointless" [FG1]. They also stated that they would recommend this learning activity to their peers [FG1, FG2]. In addition, two groups [WS2, WS3] articulated and modified their mistaken assumptions while interacting with the simulation (more details in the following paragraph). However, many students noticed that choosing among the five pre-determined options reduced the relevance of the activity: "Really doesn't matter which way we go" [WS3]; "We just had to choose 1 to 5. Anything [will do]" [WS1]. The mixed reactions suggest that designing a constructivist learning environment need not be an all-or-nothing undertaking: compromises to the learning design can still result in some meaningful outcomes. Working within less-than-ideal situations, we recommend the preservation of key characteristics of constructivism (student meaning-making, in our case) to maximise the potentials of our learning activity. However, we speculate that the reduction of student agency in trying out their own scenarios limited the scope in their thinking (see Table 1).
The most apparent shift in thinking happened in WS2 and WS3 where two groups we followed modified their mistaken assumptions regarding the dose-INR relationship from linear to non-linear. Their reassigning of the dose-INR relationship from a clockwork to complexity category constitutes a "conceptual change" (Chi, Slotta & de Leeuw, 1994, p. 27) through which they have begun to understand the concept "dose-INR relationship" in an ontologically different way. Such ontological shifts are necessary in order to understand complex systems (Jacobson, Kapur, So & Lee, 2011). Some students expressed their developing and imperfect understanding in tentative terms such as "no concentration-INR ratio" [WS2, FG1]: while they were right in that INR responses are not directly proportional to the concentration of warfarin, they had used an expression that has no meaning among pharmacists. This is characteristic of emergent words/expressions arising from local learning activities that may be "stabilised" or "discarded" (Roth, 2005, p 123) through further interactions. Many students in FG1 and FG2 also identified this change of perspective (linear-nonlinear) as a key learning point from the workshops. Shifts in other ways of thinking about complex systems were less apparent. We speculate that if the students had been asked to test their own hypotheses, a wider range in their thinking would have surfaced.
No evidence was found of students conceptualising coagulation as stochastic. We speculate that the fidelity of the simulation had a corresponding impact on the students' thinking about complex systems. Using Sheard and Mostashari's (2009) definition of "complex systems" (p. 296), we accept that the coagulation network we had developed is not entirely complex: our model has all the elements of complex systems except stochasticity. We question the mutual exclusivity of "determinism" and "stochasticity" and affirm that, while mechanism-based models of complex biological systems are usually deterministic, the effect is often unpredictable and can appear stochastic.
| Ways of thinking | Clockwork | Complexity |
| Non-linear relationships | WS2 (part 3): predicting output of non-compliance S7: Is it because you halved it [loading dose] so you just have to halve whatever you have? S8: Slower [time to reach desired INR], yah. S6: Yeah, it will be very slow. WS3 (part 2): predicting output of genetic variation (Extended dialogue from one group where they interpreted the shift of genotype from 1*1* to 2*2* as doubling metabolism and predicted that the duration to reach therapeutic level would approximately double from Day 5 to Day 9. After viewing the simulation's output, one student joked that they had mistaken the relationship as "mathematical"). |
FG1 (post-workshop) S6: When you take antibiotics, your loading dose is important, cos it just gets your plasma concentration high quickly? So halving that, logically, you'd think... your INR would take longer to increase... but it really didn't make a difference. So it shows you that there is no concentration-INR ratio. WS3 (part 3): viewing output of non-compliance S13: It's like there's a huge dip in concentration doesn't mean it'll affect its [INR]. WS4 (part 3): viewing output of non-compliance S16: Why is there a delay in the decrease in concentration and INR? I mean how you see the concentration went down but INR didn't go down as much? |
| De-centralised interactions | (No utterances on centralised control: e.g., "INR is controlled by the patient's genes.") | FG1 (post-workshop) R1: If the patient's INR keeps increasing with 7mg maintenance dose, what would you do? S4: Decrease the dose. (...) S5: Maybe the patient is changing his diet? (xxx) S2: Check with the patient that they're taking the right amount of pills and stuff. (...) S6: Maybe herbal stuff like vitamins. |
| Multiple causes | (No utterances on single causality: e.g., "If the patient's INR keeps increasing, it's because he is overdosed.") | WS2 (part 3): viewing output of non-compliance R1: Did you notice this drop in concentration, it's a bit different from what you had. S6: Yeah. (...) S8: But - why though? S6: Because I guess there's more accumulation? Still going up? S7: You metabolise slowly. (...) S9: It really depends on the clearance of warfarin as well. FG2 (post-workshop) S20: If you double or miss a dose, then you kind of expect more effect because you know it's closely monitored? So there must a reason why it is. But it's more because of the food and everything which affects the enzymes [rather than the dose]. |
| Stochastic agents | WS4 (part 3): viewing output of non-compliance S16: If you removed 1 more dose, it'd be lower than 2 definitely. |
(No utterances on stochasticity: e.g., "INR is not completely predictable. Sometimes it varies within the same patient and we don't really know why.") |
Even though we had scaled down our simulation, it remained resource-intensive. The teaching complex we used featured a wireless network, but having tested it, we preferred instead to use the classrooms' wired network for better reliability and performance. Although each classroom had 16 LAN points, only the one behind the teacher's desktop was activated. To activate the other LAN points, we would normally have had to wait up to two weeks and pay a fee. Because of the research support allocated to this project, we were able to go directly to our IT department to get the ports activated for our workshop quickly and at no additional cost. However, it is noteworthy that this option would usually not be available for standard teaching sessions and that classroom infrastructure has been identified elsewhere as a barrier to teaching with simulations (Moizer et al., 2009; Tüzün, 2007).
Despite the challenges in the three domains reported above, we noted several positive outcomes: many students came to understand that the dose-INR relationship was non-linear [WS2, WS3, FG1, FG2]; two groups articulated and modified their mistaken assumptions while interacting with the simulation [WS2, WS3]; the simulation enabled students to "visualise" warfarin's activity in humans which in turn helped them understand the reasons behind dosing regimens [FG1]; and the majority of students agreed that they would recommend this learning activity to their peers [FG1, FG2].
Given our experience, we plan to make the simulation available to students (outside of the workshop sessions) to test their own hypotheses in their own time in subsequent cycles of our educational design research. The workshop will be redesigned to exploit the students' experimentations. We also envisage the inclusion of another tutor (author 3) to facilitate the workshop with a view to sustaining the new workshop in the course.
Barab, S. & Squire, S. (2004). Design-based research: putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1-14. http://dx.doi.org/10.1207/s15327809jls1301_1
Bruner, J. (1986). Actual minds, possible worlds. Cambridge: Harvard University Press.
Cazden, C. B. & Beck, S. W. (2003). Classroom discourse. In A. C. Graesser, M. A. Gernbacher & S. R. Goldman (Eds.), Handbook of discourse processes (pp. 165-198). USA: Harvard Graduate School of Education.
Chang, K. E., Chen, Y. L., Lin, H. Y. & Sung, Y. T. (2008). Effects of learning support in simulation-based physics learning. Computers & Education, 51(4), 1486-1498. http://dx.doi.org/10.1016/j.compedu.2008.01.007
Chi, M. T. H., Slotta, J. D. & de Leeuw, N. (1994). From things to processes: A theory of conceptual change for learning science concepts. Learning and Instruction, 4(1), 27-43. http://dx.doi.org/10.1016/0959-4752(94)90017-5
Davies, C. H. J. (2002). Student engagement in simulations: A case study. Computers & Education, 39(3), 271-282. http://dx.doi.org/10.1016/S0360-1315(02)00046-5
Goldstone, R. L. & Wilensky, U. (2008). Promoting transfer through complex systems principles. The Journal of the Learning Sciences, 17(4), 465-516. http://dx.doi.org/10.1080/10508400802394898
Griffiths, R. (2004). Knowledge production and the research-teaching nexus: The case of the built environment disciplines. Studies in Higher Education, 29(6), 709-726. http://dx.doi.org/10.1080/0307507042000287212
Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Los Angeles: SAGE.
Hariri, S., Rawn, C., Srivastava, S., Youngblood, P. & Ladd, A. (2004). Evaluation of a surgical simulator for learning clinical anatomy. Medical Education, 38(8), 896-902. http://dx.doi.org/10.1111/j.1365-2929.2004.01897.x
Hung, W. (2008). Enhancing systems-thinking skills with modelling. British Journal of Educational Technology, 39(6), 1099-1120. http://dx.doi.org/10.1111/j.1467-8535.2007.00791.x
Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41-49. http://dx.doi.org/10.1002/cplx.1027
Jacobson, M. J., Kapur, M., So, H. J. & Lee, J. (2011). The ontologies of complexity and learning about complex systems. Instructional Science, 39(5), 763-783. http://dx.doi.org/10.1007/s11251-010-9147-0
Jacobson, M. J. & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. The Journal of the Learning Sciences, 15(1), 11-34. http://dx.doi.org/10.1207/s15327809jls1501_4
Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory (pp. 215-239). Mahwah, NJ: Lawrence Erlbaum Associates.
Jonassen, D. H. (1997). Instructional design model for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65-94. http://dx.doi.org/10.1007/BF02299613
Laurillard, D. (1992). Learning through collaborative computer simulations. British Journal of Educational Technology, 23(3), 164-171. http://dx.doi.org/10.1111/j.1467-8535.1992.tb00327.x
MathWorks (undated). MATLAB - the language of technical computing. http://www.mathworks.com.au/products/matlab/
Moizer, J., Lean, J., Towler, M. & Abbey, C. (2009). Simulations and games: Overcoming the barriers to their use in higher education. Active Learning in Higher Education, 10(3), 207-224. http://dx.doi.org/10.1177/1469787409343188
Park, S. I., Lee, G. & Kim, M. (2009). Do students benefit equally from interactive computer simulations regardless of prior knowledge levels? Computers & Education, 52(3), 649-655. http://dx.doi.org/10.1016/j.compedu.2008.11.014
Perkins, D., Tishman, S., Ritchhart, R., Donis, K. & Andrade, A. (2000). Intelligence in the wild: A dispositional view of intellectual traits. Educational Psychology Review, 12(3), 269-293. http://dx.doi.org/10.1023/A:1009031605464
Pirmohamed, M. (2006). Warfarin: Almost 60 years old and still causing problems. British Journal of Clinical Pharmacology, 62(5), 509-511. http://dx.doi.org/10.1111/j.1365-2125.2006.02806.x
Reeves, T. C., McKenney, S. & Herrington, J. (2011). Publishing and perishing: The critical importance of educational design research. Australasian Journal of Educational Technology, 27(1), 55-65. http://ascilite.org.au/ajet/ajet27/reeves.html
Riley, D. (2002). Simulation modelling: Educational development roles for learning technologists. ALT-J: Research in Learning Technology Journal, 10(3), 54-69. http://dx.doi.org/10.1080/0968776020100305
Roth, W. M. (2005). Talking science: Language and learning in science classrooms. Lanham: Rowman & Littlefield Publishers, Inc.
Sabelli, N. H. (2006). Complexity, technology, science, and education. The Journal of the Learning Sciences, 15(1), 5-9. http://dx.doi.org/10.1207/s15327809jls1501_3
Sheard, S. & Mostashari, A. (2009). Principles of complex systems for systems engineering. Systems Engineering, 12(4), 295-311. http://dx.doi.org/10.1002/sys.20124
Tüzün, H. (2007). Blending video games with learning: Issues and challenges with classroom implementations in the Turkish context. British Journal of Educational Technology, 38(3), 465-477. http://dx.doi.org/10.1111/j.1467-8535.2007.00710.x
Wajima, T., Isbister, G. K. & Duffull, S. B. (2009). A comprehensive model for the humoral coagulation network in humans. Clinical Pharmacology and Therapeutics, 86(3), 290-298. http://dx.doi.org/10.1038/clpt.2009.87
Wu-Pong, S. & Cheng, C. K. (1999). Pharmacokinetic simulations using cellular automata in a pharmacokinetics course. American Journal of Pharmaceutical Education, 63(1), 52-55. http://www.highbeam.com/doc/1P3-40642328.html
On examination, Violet had no signs of dyspnoea, cyanosis, or fever. Her blood pressure was 120/80 mm Hg. Physical examination was normal except for swelling of the left lower leg below the knee and Homans' sign (pain on passive dorsiflexion of the foot). The operation wound was healed with no signs of inflammation or bleeding.
Violet was commenced on enoxaparin 70 mg SC BD (she weighed 70 kg) and warfarin (given once daily). She was later discharged to the care of her GP after learning how to self-administer the enoxaparin injection. Target INR is 2-3.
Use the warfarin simulation software to explore the relationship between warfarin dosage and genetic covariates on its pharmacokinetic and pharmacodynamic profiles.
Run #1 - default setting:
Run #2 - genetic variation:
Run #3 - compliance:
| Authors: Swee-Kin Loke is a professional practice fellow (educational design) in the Higher Education Development Centre, University of Otago, 65 Union Place West, Dunedin 9013, New Zealand. Email: swee.kin.loke@otago.ac.nz
Hesham Al-Sallami is a lecturer in the School of Pharmacy, University of Otago, PO Box 56, Dunedin 9054, New Zealand. Email: hesham.al-sallami@otago.ac.nz Daniel Wright is a professional practice fellow in the School of Pharmacy, University of Otago, PO Box 56, Dunedin 9054, New Zealand. Email: dan.wright@otago.ac.nz Jenny McDonald is a lecturer in the Higher Education Development Centre, University of Otago, 65 Union Place West, Dunedin 9013, New Zealand. Email: jenny.mcdonald@otago.ac.nz Sheetal Jadhav is a research assistant in the School of Pharmacy, University of Otago, PO Box 56, Dunedin 9054, New Zealand. Email: sheetal.jadhav@otago.ac.nz Stephen Duffull holds a chair in clinical pharmacy and is Dean of the School of Pharmacy, University of Otago, PO Box 56, Dunedin 9054, New Zealand. Email: stephen.duffull@otago.ac.nz Please cite as: Loke, S.-K., Al-Sallami, H. S., Wright, D. F. B., McDonald, J., Jadhav, S. & Duffull, S. B. (2012). Challenges in integrating a complex systems computer simulation in class: An educational design research. Australasian Journal of Educational Technology, 28(4), 671-683. http://www.ascilite.org.au/ajet/ajet28/loke-2.html |