Creating simulations for an "Introduction to Research Methods" course

Bob Adamson
Hong Kong Institute of Education

This paper describes the production of a software program for Master of Education students studying the "Introduction to Research Methods" course at a tertiary institution in Hong Kong. The course was originally delivered in a lecture mode, which proved unsatisfactory in providing sufficient learning support for the students. The paper describes how the program was designed for a blended learning environment, drawing on constructivist learning principles and the use of simulations. The software program that emerged was a compromise to resolve a number of tensions, such as the need to present a rationalised, pedagogically-oriented version of a complex and messy reality, and the desire to enhance student autonomy while working within the limitations of technical boundaries. The paper concludes by reflecting on the lessons learnt in the design process.

Introduction

At the outset, we had two broad principles in mind-a commitment to constructivist views of learning that was derived from our own pedagogical beliefs, and a wish to explore the use of simulations to bridge the theoretical and practical aspects of the course, which comprised an introduction to several common kinds of research and their application in educational contexts. Bringing together these two broad principles proved difficult in some respects, despite the fact that they are complementary in many ways. The tensions centred around the messiness of reality in research design on the one hand and the provision of a clearly structured learning experience on the other. Another challenge was finding ways to allow the students as much autonomy as possible within the program without requiring too much interaction with real life tutors.

The course required attention for several reasons. Pedagogically, its existing design was proving ineffective. The course consisted of a series of ten classes, each lasting for three hours on a weekday evening. The majority of the students are studying part time, and are attending class after a full day's work teaching in local schools. Teachers in Hong Kong tend to have very heavy teaching and administrative workloads (Morris & Adamson, 2010) and so the students tend to be tired, hungry and listless in class, which was unsatisfactory for all participants, particularly given that this course was viewed by the institution as an essential component of the master's degree. Also, being a core course, it was attended by all students on the program (totalling several hundred students each year), resulting in crowded lecture rooms and a predominantly expository lecturing style by the tutor, with the students expected to get to grips with the subject through self-directed reading and the essay assignment. It was thus difficult to ensure that each student received adequate support to engage deeply with the concepts.

Another concern was financial: tertiary institutions in Hong Kong are becoming increasingly reliant on self-funded programs such as the master's. This program was intended to be economically viable; lecturers' salaries represented the largest challenge to profitability, so e-learning represented an opportunity to reduce these costs. At the same time, staff were under increasing pressure to enhance their research output, as this was an increasingly important criterion for tertiary funding and prestige in Hong Kong, so any exploration of alternative modes of delivery that would reduce face to face teaching time without sacrificing (and, ideally, improving) the quality of the students' learning experience was welcomed by the teaching staff and senior management alike. These dynamics, plus the fact that this course was the first to be converted to a software supported form of delivery, facilitated the successful bid for a Teaching Development Grant within the institution.

We were therefore attracted to the possibilities of blended learning (that is, the convergence of face to face teaching with technology-enabled learning) for both pedagogical and pragmatic reasons. Graham (2004) identifies in blended learning possibilities for enabling students to have greater access and flexibility in organising their study, for enhancing pedagogical practices, and for transforming the way in which the students could learn. While the face to face contact would enable the lecturers to introduce the students to the concepts of research design, technology-enabled learning would allow students the opportunity to learn in a more comfortable environment (rather than a lecture hall), at their own pace and with the opportunity and support for greater engagement with the concepts.

In a review of technology-enabled learning, Gao, Choy, Wong and Wu (2009) note a shift from technology being used as instruction tools to support conventional teaching, to its use as cognitive tools that enhance active, constructive, authentic and cooperative learning. These learning characteristics matched those that we wished to develop in our students, although we did not arrive at a solution that allowed us to incorporate them all in the software program that we designed. By using the software program on their own and at a time and place of their own choice, the students were provided with the opportunity to develop into autonomous learners, in charge of their own learning to some extent, which is an indicator of successful learners (Mok, Lung, Cheng, Cheung & Ng, 2006). This was an important aim, as the software program was intended to replace several face to face lectures. The convenience and comfort afforded by the opportunity for the students to study the information provided in the program and to undertake the different tasks according to their own schedule was designed to alleviate some of the negative effects of the lecture timetable, which, coming at the end of a heavy day's work for many students, was not an optimal arrangement for facilitating learning.

Design principles

Pedagogical principles

Hmelo-Silver, Duncan and Chinn (2007) argue that well-scaffolded, problem-based learning is an effective pedagogical approach. We felt that this approach was appropriate for the circumstances, as carefully structured learning support could be supplied through the software program. The scaffolding that we employed comprised theoretical input and experience sharing of the practical application of research design from lecturers, and a built-in interactive mechanism whereby students would receive feedback on their work as they progressed, and be provided with an opportunity to revise their work before it is submitted as the final product - an aid to learning that has been called "feedforward" (Carless, Joughin, Liu & Associates, 2006).

Simulations

Simulations in software programs have been promoted by instructional designers such as Aldrich (2005), Quinn (2005) and Schank (2002) for their value in motivating learners and enhancing the relevance of their studies to the realities of their work. Black, Thalheimer, Wilder, de Soto and Picard (1994) developed four key principles for the use of the use of simulations in technology-enabled learning:

1. the knowledge and skills and the design of tasks should be contextualised in realistic situations;
2. multiple contexts and diverse representations of content should be provided to prepare students to transfer the new knowledge and skills to a variety of contexts;
3. multiple perspectives on the knowledge and skills should be offered in order to avoid cognitive straitjacketing; and
4. opportunities should be given for the students to collaborate and generate their own knowledge.

We built the simulations around a series of educational research projects that were all likely to occur in educational settings in Hong Kong - an evaluation of a music program to see if it represents value for money; an investigation of pedagogical concepts in curriculum documents, textbooks and lesson plans relating to science; a critical analysis of social equity issues in language education; a gauging of satisfaction levels with innovations in assessment in the subject of Personal, Social and Humanities Education; and an action research project intended to improve the quality of student learning in mathematics. The equipment for developing a research design plan in real life would include word processing, emails and access to a library. The first two were built into the program at appropriate stages (with emails being simulated) and the students would already have online access to the Institute library. The psychological fidelity was addressed by including Virtual Tutors acting as mentors in the program, as well as having real life lecturers presenting both their own research experience and their suggestions on the simulated projects.

In order to provide multiple contexts and perspectives on the knowledge and information that we wished the students to acquire, we decided to design several tasks for them to complete. The tasks were similar, in that they asked the students to follow a set sequence of design, but they were different in terms of the setting (for instance, some projects were set in primary schools, others in secondary schools), the content (each project focused on a different school subject), and the purpose of carrying out the research (such as evaluation, investigation or explanation). Opportunities for the students to interact and construct knowledge together proved difficult to incorporate in the program. Students would have some opportunities in the broader context of the blended learning arrangement rather than it being a feature of the software. The face to face class time would allow students to meet up and discuss their approaches to the tasks in the software and the institute's e-learning platform, Blackboard, allowed for online collaboration and discussion with fellow students and lecturers. However, the integration of the program with these channels for peer interaction has yet to be successfully achieved.

Features of the program

The program that we designed takes a narrow view of the research process, being limited to the development of a research proposal. We did not incorporate a simulation of data collection, analysis and presentation of findings. There were two reasons for this. The first reason was to limit the cognitive load on the students - for many of them, the introduction to different types of research was sufficiently demanding in a 39-hour course. The second reason was more technical: we realised that trying to incorporate a simulation of collecting and analysing data and of producing a research report in a stand alone program would be very difficult and resource consuming to achieve. For these reasons, we decided that the program would be integrated into the taught course at an early stage, so the real life lecturer could explore issues of data collection, analysis and presentation of findings in the latter part of the course.

Having located the program in the context of the taught course, we decided that the students would need a considerable amount of scaffolding. The design of the program therefore had two main components - the simulations that took the students step by step through the design of five research projects, and the learning support, presented by lecturers who are experts in research methods. Our first move proved a false step. We recorded a series of five talking-head videos by each lecturer, giving a general introduction to the types of research to be covered by the course (scientific research, investigative research, evaluative research, critical research and action research). At that stage we had not designed the simulations, and had not identified the main teaching points. In the event, these videos provided very general introductions and did not have close or direct relevance to the projects that eventually formed the basis of the simulations.

Therefore we returned to the lecturers and asked them to present two more videos: a parallel research project that the lecturer had actually undertaken, and specific advice for the project in the simulation - both in the form of semi-structured interviews to ensure that the lecturers' contributions were more relevant to the simulations and also focused on specific teaching points (i.e. the decisions made by a researcher in the design of a project). The videos were between three and 10 minutes in length. The challenge was to achieve succinctness without over-simplification so that they would have pedagogical value within technological constraints such as bandwidth and download times. We were conscious that there is a fine line to walk between over-scaffolding - providing too much support and making the task too simple for the students - and under-scaffolding, which might make the students frustrated. We chose to err on the side of over-scaffolding, given the introductory nature of the program.

The program opens with a split screen, which is a consistent feature of the program. On the left hand side are the supporting instructional resources, while the simulations unfold on the right hand side (Figure 1). An introductory tour is presented by an avatar, Paul.

Figure 1: Home page with avatar on left

Note: This screenshot and those in subsequent figures are taken from a test of the software conducted by the design team. One of the team, Denis Williamson, has logged on in the role of a student.

In the simulations section, students are presented with a choice of the five educational problems drawn from five subject areas in the school curriculum: music, science, languages, assessment and mathematics. As noted above, the problems are linked to five different kinds of research (scientific, interpretive, evaluative, critical and action research). The students can explore the projects in more detail before committing to a particular one: by clicking on the title of a project, they are taken to a page where they see a video of an actor playing the role of Project Sponsor (who is commissioning the project in the simulation) and a brief written description of the project. They may select any of the five - the projects are not sequenced in any way (for instance, the first project in the list is not necessarily the least complex). The reason for not doing so was our desire to give the students freedom of choice in the projects that they wished to undertake: we believed that this choice would be more motivating for the students. This was a conscious decision to enhance student autonomy, and we also took the view that students might devise their own route from less complex to more complex as they have the flexibility to leave one project at any time in order to pick up or start another one.

Having chosen a project, the students follow a number of steps to create a research proposal. Although we wished to achieve realism in the program, we also acknowledged that the holistic complexity and "messiness" of reality would be difficult to capture, and, indeed, might overwhelm the students and adversely affect their learning. We decided to rationalise the research design process as a linear sequence with several decision points. For each decision, students would be provided with instruction and advice. The decision points that we initially chose were: selecting a research topic; identifying an appropriate research question; conducting a literature review; choosing the research methods; collecting data; analysing data; and presenting findings. As noted above, these required subsequent modification and we reduced the steps to choosing the topic, carrying out a literature review, selecting a research question, selecting appropriate research methods, and finally submitting the research proposal for consideration by the funding body.

Figure 2: Inserting the references from the literature review

For the literature review, the students have to identify three relevant publications from the library or from websites, and input the references using a word processing function before they can continue to the next stage. (Figure 2 shows an example; the examples are all taken from the mathematics project). Given the problem of building feedback into the software for these unpredictable responses, we decided that a real life tutor would be needed in order to check and give feedback to the students on the references that they had chosen. Therefore, once they have inserted the three references, the students are presented with a print button on the screen. They may then, if the real life tutor wishes, hand in the set of references as an assignment.

Student autonomy in the following stages is more controlled, as the total number of options and ramifications available to them is limited by the software. Messages from Paul shepherd them through the stages, while simulated emails from Virtual Tutors (one for each project) offer support and feedback (Figure 3). The emails tell the students what is happening and offers them some options for proceeding: this could be an email from the Virtual Tutor offering to review the students' research proposal prior to submission, or directing them to particular videos, or it might be a message from the project sponsor announcing the result of the students' submitted research proposal. The use of emails offered another realistic element and also permitted simulated personal interaction. The Virtual Tutors are portrayed by lecturers whom, in real life, the students would be likely to encounter in the course of their studies. Mindful of this, we took care in selecting the lecturers in order to achieve some kind of balance between the genders as well as ethnicity, given that the Institute employs staff from different countries. To enhance personalization, the emails in which the Virtual Tutors introduce themselves to the students and offer advice and support were actually written by these real life lecturers in their own style.

Figure 3: Tutor's introductory email (right)

In the first email message, the Virtual Tutor refers the students to the three videos that are available for the project, presented by the real life lecturer who appears as the Virtual Tutor. Initially we made available on the screen all the videos we had recorded with the five lecturers and merely asked the students to identify the videos that they would find the most useful for the particular task they were undertaking. This meant that they were faced with a choice of 15 videos, that is, three for each lecturer, and some of these videos were of considerable duration - up to 15 minutes long. On reflection, we decided that this did not provide sufficient scaffolding, and resolved to simplify the students' decision by only displaying the three videos that were pertinent to the task in hand. This required a simple technical adjustment. The students are free to select from these - the general introduction to the specific type of research by the lecturer, an example of research carried out by lecturer, and the lecturer's advice to the student on undertaking the particular project in the simulation. As a means of providing further support, the videos are divided into titled chapters; key ideas were presented as subtitles; and a brief glossary of key terms was also provided.

Moving to the next screen, the students are asked to select a research question from four options (Figure 4). The questions took several drafts before we were satisfied that there were no blatant linguistic clues that would enable the students to select the most appropriate option without using their knowledge of research. For example, the presence of words in the research project re-occurring in the research question was an obvious lead to the answer, so paraphrasing was used as far as possible. This juggling act reflected the tension between providing too much and too little support.

Figure 4: Choosing the research question

In the next step, students are presented with a set of research methods from which they choose as many as they wish so long as they deem these research methods to be applicable and feasible ways of addressing the research question (Figure 5).

Figure 5: Choosing the research methods

Once this choice has been made, the students are shown a summary of their proposal, incorporating a description of the project, the research questions and methods, and the literature review. They have several options. They could, if they wish, revise the proposal. Alternatively the students may wish to respond to an offer from the Virtual Tutor to appraise and advise on their research proposal, which would allow the students to have an opportunity to revise their draft prior to submission. If they accept the offer, the students will be advised on the appropriateness of their choices and, if they do not accord with the judgements built into the software, the students are advised on how they might reconceptualise their proposal (Figure 6).

Figure 6: Virtual Tutor's feedback on the proposal

The students also have the option of submitting their draft research proposal directly to the Project Sponsor without consulting the Virtual Tutor. Such a course of action is obviously riskier as only two consequences can result, either acceptance of the project proposal or rejection. The students are made to wait for the Project Sponsor's decision, partly to build up the tension and partly to enable us to insert a multiple choice quiz focusing some of the key features of the particular type of research, a move that is intended to consolidate the students' learning.

The students' progress in completing each project successfully or unsuccessfully is tracked on the home page (Figure 7). Once the students have completed all five projects, they are deemed to have finished the program and they then return to face to face classes to study research design in more depth.

Figure 7: Flowchart for tracking progress through the projects

Evaluation

A further quality assurance mechanism was to invite a team of international consultants from the USA to review the product. The consultants in this case were two highly experienced professionals in the production of multimedia learning programs in the field of education. One was a former university professor with several years' experience in mentoring research students; both had previously worked for the BBC for many years and were currently based in Silicon Valley in California, working for a company specialising in educational software. The consultants carried out a rigorous review of the product and submitted a detailed report. Unlike the student reviewers, who were mainly concerned with bugs and individual problems with the software, the international consultants took a macro view of the product and addressed issues such as the overall design, interface, and the pedagogical underpinnings of the program. Their final report made a number of suggestions, one of which pointed out the paradox that achieving simplicity of interface is actually a highly complicated process. A major technical revision they recommended was a less complicated use of the real estate, replacing the split screen format with a single screen and making the contents less "busy". In terms of pedagogy, they also suggested creating more opportunities for students to revise the projects and reducing the prospect of rejection by the Project Sponsors.

While we were able to respond to many of the suggestions put forward by reviewers, we did not undertake a major overhaul of the interface design to further simplify the navigation because of pressure of time. With hindsight, it would have been better to invite the input of the consultants at a much earlier stage of the design process so that their suggestions could have been formative in guiding the development of the first version program, rather than asking them to provide a summative evaluation when there was little scope for revisions. However, the ideas can be incorporated in the next version of the program.

Another weakness identified by the consultants that we will attend to in a subsequent revision of the program is the presentation of choices for the research question and methods as a multiple choice. This presentation is uncontextualised and unrealistic in format. However, it did allow us to develop a structure that was controlled, as we could determine the number of ramifications involved. For example, by limiting the options to four choices rather than five or six at the first decision point, we reduced the number of ramifications at the second stage, and by similarly reducing the number of options at the second and third stage the total number of possible outcomes was mathematically reduced and the amount of scripting that was required to provide feedback to the students on the choices that they had made following the different pathways that were possible was manageable. However, we also recognise that constructing the software around a multiple choice question format imposed limitations on the freedom of choice and therefore the autonomy of decision making given to the students and also means that there was a strong degree of predictability, a strong curriculum frame, to use Bernstein's (1971) term. In future versions, we intend to replace this list by suggestions from various sources, such as videos of colleagues, emails and simulated Internet searches. This change would also allow us to increase the sense of personal interaction that is a feature of the simulation, one which was praised by the consultants.

Another technical challenge that impaired our ability to offer the students more options and more scaffolding concerned the size of the final product, which, at 500 MB, is currently quite substantial for the computers that are used in the institute and by the students in their school or home environments. We planned to make the program available to all students to download from the institute's electronic learning platform, Blackboard, but this proved impractical, as the program in its integrity exceeded the available space. Therefore we had to separate out the embedded videos. This might slow down the access to the videos, and also requires that the students be connected to the Internet when working on the program - we had originally planned that the whole program could be simply copied onto the students' hard disk, as not all students have Internet access at home and would only be able to work on the program at work or on campus. A third option would be to burn the program on to CDs or USB memory sticks, which would require the students to transfer the program onto their hard disks in order to preserve the save function that records the progress that the students have made to date.

Overall, we believe that the program adds value to the previous lecture delivery format. Through the use of simulation, students meet some of the basic approaches to successful research in an environment that is not clouded by real world constraints such as shortage of time, lack of delivery of materials, or inadequate responses from contributors. They learn how to scope a research project and how to focus on the relevant and available means of collecting the necessary data. Moreover, students gain access to an array of experts in educational research talking first hand about their own experiences and giving experienced guidance on the different types of research in the local and regional context. The step by step approach to research design provided by the program will prove valuable once students start on their own research projects. It gives students the opportunity to think through the steps they need to take in their future research projects and trains them in a methodical way, which is in keeping with current pedagogical trends in Hong Kong. It also challenges them to think in depth about the topic they are researching. Feedback given during the software application empowers the students in handling some of the obstacles they will meet once they start on their own research work.

Discussion

There is also a degree of realism that is sacrificed through the use of Virtual Tutors with a limited number of possible responses rather than a real life tutor. Furthermore, the fact that most of the students would work through the program on their own, as there is no component in the program that facilitates, for instance, team work (which would be likely to occur in an actual research project in a school in Hong Kong), meant that interactivity is ultimately constrained.

The second tension is pedagogical and relates to the incorporation of the features of constructivist learning approaches and the realism of the simulation. The program presents the student with readily accessible expertise in a neatly packaged form, something that would not be easily available in a real life setting. It also sets out clear and rational steps and specific decision making points to guide the students' through a process that, in reality, seldom lends itself to rationality.

The sacrifices to the accurate portrayal of reality were, in our view, justified by the fact that our primary aim was to promote learning rather than to produce a highly realistic simulation that captured the messiness of the process. We tried to avoid over-emphasising the support of student learning by helping them too much - instead, the Vygotskyan notion of the ZPD was a useful guide in determining the degree and nature of the scaffolding to be provided.

On balance, we believe that we have created a program that, despite its compromises, does simulate and support learning about the decision-making processes involved in the research design process in the field of education studies, with particular reference to the Hong Kong context. With some further refinements, we hope that this application can act as a template to prepare a range of students working on research projects in different fields of study.

Acknowledgments

References

Bernstein, B. (1971). Class, codes, and control. London: Routledge & Kegan Paul.

Black, J., Thalheimer, W., Wilder, H., de Soto, D. & Picard, P. (1994). Constructivist design of graphic computer simulations. Paper presented at The National Convention of the Association for Educational Communications and Technology. [verified 11 Dec 2010] http://www.ilt.columbia.edu/publications/cdgcs.html

Bruner, J. (1967). A study of thinking. New York: Science Editions Inc.

Bruner, J. (1977). The process of education. Cambridge, Massachusetts: Harvard University Press.

Carless, D., Joughin, G., Liu, N-F. & Associates (2006). How assessment supports learning: Learning-oriented assessment in action. Hong Kong: Hong Kong University Press.

Clark, J. L., Scarino, A. & Brownell, J. A. (1994). Improving the quality of learning: A framework for Target-Oriented Curriculum renewal in Hong Kong. Hong Kong: Hongkong Bank Language Development Fund.

Fritz, P. Z., Gray, T. & Flanagan, B. (2007). Review of mannequin-based high-fidelity simulation in emergency medicine. Emergency Medicine Australasia, 20(10), 1-9.

Gao, P., Choy, D., Wong, A. F. L. & Wu, J. (2009). Developing a better understanding of technology based pedagogy. Australasian Journal of Educational Technology, 25(5), 714-730. http://www.ascilite.org.au/ajet/ajet25/gao.html

Graham, C. (2006). Blended learning systems: Definition, current trends, and future directions. In C. J. Bonk. & C. Graham (Eds), Handbook of blended learning: Global perspectives, local designs. San Francisco, CA: Jossey-Bass.

Hmelo-Silver, C.E.,Duncan, R.G. & Chinn, C.A. (2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller and Clark (2006). Educational Psychologist, 42(2), 99-107.

Mok, M. M. C., Lung, C. L., Cheng, D. P. W., Cheung, R. H. P. & Ng, M. L. (2006). Self-assessment in higher education: Experience in using a metacognitive approach in five case studies. Assessment and Evaluation in Higher Education, 31(4), 415-433.

Morris, P. & Adamson, B. (2010). Curriculum, schooling and society in Hong Kong. Hong Kong: Hong Kong University Press.

Piaget, J.-P. (1963). The origins of intelligence in children. New York: W.W. Norton & Company, Inc.

Quinn, C. (2005). Engaging learning. San Francisco: John Wiley & Sons.

Schank, R. (2002). Designing world class e-learning. London: McGraw-Hill.

Vygotsky, L. F. (1978). Mind in society: The development of higher psychological principles. Cambridge, MA: Harvard University Press.

Authors: Professor Bob Adamson Department of International Education & Lifelong Learning Hong Kong Institute of Education 10, Lo Ping Road, Tai Po, New Territories, Hong Kong SAR Email: badamson@ied.edu.hk Web: https://oraas0.ied.edu.hk/rich/web/people_details.jsp?pid=11548 Bob Adamson is Professor of Curriculum Studies and Head of the Department of International Education & Lifelong Learning at Hong Kong Institute of Education. Please cite as: Adamson, B. (2010). Creating simulations for an "Introduction to Research Methods" course. Australasian Journal of Educational Technology, 26(7), 917-931. http://www.ascilite.org.au/ajet/ajet26/adamson.html