|Australasian Journal of Educational Technology
2012, 28(5), 896-911.
Design-based research principles for student orientation to online study: Capturing the lessons learnt
University of Technology, Sydney
Mary Jane Mahony
The University of Sydney
Few institutions have reported research on students' use of orientation programs designed for mature students returning to study in contemporary learning environments now regularly amalgamating distance and online strategies. We report within a design-based research framework the student experience of GetLearning, the third stage of an innovative online orientation program. Integral to the design is active student engagement with activities to extend their skills whilst still supported in a risk free environment. Analysis included observations of the patterns of student activity, students' responses in evaluation instruments, our reflections, and identification of the limitations to deployment of such orientation programs. We also provide a set of design principles to guide further development work and research.
Literature describing the student experience of beginning online study (Levy, 2006; Motteram & Forrester, 2005; Price, Richardson & Jelfs, 2007), substantial local anecdotes and our own research (Wozniak, Mahony, Pizzica & Koulias, 2007) all indicate that pre-semester orientation as well as support during the semester is required. As Salmon (1998) indicated in her early work, there is a need for learners to be provided with opportunities to experiment with the technology and to make mistakes in a supported environment.
Parallel investigations have also been directed at determining the factors that contribute to student satisfaction with online learning. Elements that have been identified as influencing student satisfaction include ease of use of the online system, learner anxiety with the technology, perceived usefulness of the materials, instructor attitudes, course design and flexibility (Chang & Tung, 2008; Sun, Tsai, Finger, Chen & Yeh, 2008).
Reports that describe orientation initiatives for online learners (Bozarth, Chapman & LaMonica, 2004; Levy, 2006; Scagnoli, 2001; Tomei, Hagel, Rineer, Mastandrea & Scolon, 2009), note the achievement of increased student enjoyment and confidence. However with the exception of Levy (2006), they retain a sense of atomised, superficial and rather piecemeal strategies without a coherent underpinning conceptual framework for delivering such improvements. Instead, what is often recommended are resources to improve computer skills or a review of static information loaded webpages without consideration as to how this is best achieved. However, the 2010 release of the Student Induction to E-Learning (SIEL) report (IMS Global Learning Consortium Inc, 2010; Krause & McEwen, 2009) recognised the need for best practice frameworks to guide student induction to e-learning. This report provides a checklist and maps a best practice framework for promoting post-secondary student retention associated with induction to e-learning. It suggests the need for a pro-active approach to orientation. Although this guidance undoubtedly can contribute to improved practice, there remains a need to examine more closely the student experience of this crucial stage which lays the foundation for successful study in the following years (Crosling & Heagney, 2009), echoing the suggestion made earlier by Motteram and Forrester (2005).
... we consider that more student-centred studies exploring online distance learning are needed to assess how students manage the transition to online learning, how they perceive their initial online tasks, what strategies (if any) they adopt, and how the appropriate use of technology and pedagogy could enhance distance programmes, benefit students' learning, and enrich their educational experience (p295)Our research describes the implementation of one such approach known as the GettingOnTrack suite of resources founded on a simple, yet effective conceptual framework (Wozniak, Mahony, Lever & Pizzica, 2009).
Implicitly and explicitly directed at creating products and processes for the improvement of student learning, teacher skills, and institutional and systemic reform (p.32).This paper is structured into five sections as recommended by Collins, Joseph and Bielaczyc (2004) for reporting design-based research:
GettingOnTrack is designed to begin prior to enrolment (and even admission) via the GetReal open access, web-based interface (Lever, Mahony & Wozniak, 2007). Prospective students can assess their readiness for study by comparing course design attributes with their perceived needs, gauge their technological capabilities, and engage in a time management task. After admission and enrolment in their initial subjects, students are provided with a simple guide (GetStarted) to assist them to navigate and log into the learning management system (LMS) site. They are then able to access GetLearning, housed in the university's enterprise LMS, where they encounter a modularised introduction to key elements of learning in online environments. The focus in this paper is on the third component, GetLearning.
This suite of resources was designed in keeping with the emerging conceptual framework (Wozniak et al., 2009). Design and content drew on prior work carried out with undergraduate students in blended learning environments evaluating the effectiveness of activity based online communication tasks (Wozniak, 2007; Wozniak & Silveira, 2008), and on other activities piloted in a range of faculties across the university (Freeman, Clarkeburn & Treleaven, 2007). Iteration 1 encompassed a comprehensive expert teacher evaluation. Reviewers included local champions of online learning and over twenty teaching and support staff from relevant postgraduate degree programs (at least some of whom were targeted stakeholders for the new orientation resource). A proforma reporting sheet was used and more than twenty pages of comments (both general and specific) were received, enabling further refinement. The broad design concept was confirmed, with no major design faults.
Consequently, there is a broadly sequential design starting with simple navigation exercises, moving through communicating with others online and working in collaborative online groups, continuing to essentials such as submitting assignments online, understanding relevant university policies and accessing information sources, and avoiding plagiarism. The activities are learner-centred, self-motivating, low maintenance and applicable to different course contexts. Further discussion of the underlying design principles are explained in Wozniak et al (2009). Also included is an overview of the five modules contained in the GetLearning resource including the types of interactive activities and the supports available to assist new online learners as they progress through each module.
Approval was gained from the University's Human Research Ethics Committee. Data sources included a wide variety of both quantitative and qualitative types collected across two academic semesters from February 2007 to November 2007. This included:
The site was made available to postgraduate students prior to the commencement of each of two semesters in 2007 (one week prior for semester 1 and two weeks prior for semester 2). Students were sent an advisory email by a teacher responsible for the unit or course in which they were enrolled. They were provided with the GetStarted guide in PDF or hardcopy, and encouraged to work through the five modules, commencing immediately and recommended to be finished by the end of the semester's first week. Their participation in the GetLearning site was not mandated and did not contribute to assessment scores. On entry into the site they received an announcement about the research and were asked to indicate if they would accept an invitation to participate in a focus group session. Student response to this was insufficient and focus groups were not conducted (see the section 'Lessons' for further discussion). Students were asked to complete an anonymous '3 minute feedback' evaluation survey at the completion of each module.
This approach allowed the moderators and designers of the site to compare their expectations of how students would use the site with how students did use the site. In combination, these evaluation methods allowed the moderators to act upon the students' patterns, providing further targeted support to students.
For Iteration 3, the moderation activities were reviewed and a 'minimalist model' developed whereby students were not prompted to explore additional modules in the site. This was a pragmatic response to limited resources as well as an opportunity to observe student activity under limited moderation conditions.
Data were collected by downloading tracking statistics and survey responses. SPSS was used to provide descriptive statistics and all open ended answers were read by all researchers and grouped according to thematic emphasis. Patterns of access were coded by analysing the tracking sequences taken by students. As noted above during Iteration 2 in Semester 1, students were given further advice and support on how to better utilise the module resources, and tracking data was again captured to determine the impact of the intervention suggestion provided by the moderator. The effect of these interventions on participation during iteration 2 has been reported previously (Wozniak et al., 2007).
Accessing the site and completing the activities in each module was recommended to students by their course coordinators, but was not mandated or assessed. In Semester 1, 79% of students (n=142) accessed the site within the first three weeks of semester. In Semester 2, 69% (n=202) accessed the site in the same timeframe.
pattern from first access
|1. Accessed modules||Selected one of the 5 modules from the homepage||45%|
|2. Accessed homepage only||Accessed the homepage but did not explore further during this initial access||23%|
|3. Accessed site but not via the homepage||Accessed site from icons or course tools that lead them directly to announcements, or other area such as a discussion posting rather than through the homepage||20%|
|4. No access||Students never accessed the site||12%|
The second pattern, demonstrated by group 2, accessing the homepage without any further access to deeper pages on the first visit, was also a common pattern. Of greater significance is determining if this group returned to the site at a later date. In semester 1, 28 of the 41 students (68%) returned to the site from a few hours to 10 weeks later, and likewise 24 of the 43 (56%) in semester 2. This may illustrate the need for a clear stimulus to assist the learner to go beyond the initial browse of the homepage.
Another pattern (Group 3), not initially expected, occurred when students navigated on their first access directly to the embedded activities and tools such as the discussions, mail, assignments, and calendar and missed the homepage and content of the modules. This occurred when they clicked on the icons below the GetLearning title link rather than the title link which would take them to the homepage. A far greater number of students explored the site in this manner in semester 2 possibly because they had become accustomed to this approach from their earlier studies online in semester 1. The main limitation to this type of access on the first visit to the site is that students miss the context of the site. For example, going directly to the discussion board without reviewing the activity information may confuse the student by not locating the activity in the context of development of online communication skills (module 2). Of the students who accessed the site in this manner a proportion did return to the site to access the modules at a later date (21 of 35 (60%) in semester 1 and 57 of 141 (40%) in semester 2).
Table 2 outlines the overall type of participation in the GetLearning resource for all students who accessed the site during iterations 2 and 3. There is a marked difference in participation between each semester with greater access to the module content and greater participation in all the activities during semester 1. During semester 2 there was a greater student 'lurking' role in the discussion board where students were noted as reading the discussion board postings rather than engaging in active participation. It was noted that if students did access the module they then navigated to deep pages within that modular content area, but this did not always lead to active participation in the related activity (discussion, quiz or assignment submission).
Findings also indicated that students returned to GetLearning throughout the semester, multiple times. In semester 1, 25% of students accessed the site more than 10 times during the semester. This was considerably less in semester 2 (8%). However in both semesters students continued to access the site after the conclusion of semester teaching periods (25 students in semester 1 and 23 in semester 2), presumably to review different elements as the need arose. Exploration of this unexpected outcome is outside the scope of this paper, with further investigation of student use of orientation resources over time being warranted.
|Modules||Accessed modules regardless of their entry route into the site||119||108|
|Accessed by any means but didn't review any modules||38||118|
|Read discussion postings but did not post any messages||32||51|
|Read and posted a discussion item||62||30|
|Started and completed a quiz activity||74||27|
|Started and completed an assignment submission||30||10|
|Module number: Content focus||Semester 1|
n=119 students who accessed
one or more modules
|Module 1: Navigation||40||27|
|Module 2: Communication||41||21|
|Module 3: Groups||28||5|
|Module 4: Assignments||29||8|
|Module 5: Academic integrity||12||5|
|Total responses for all surveys||150||66|
Analysis of the survey results indicated that participants agreed or strongly agreed that the modules were helpful (Semester 1, 96.7%, Semester 2, 80.3%) and relevant (Semester 1, 88.9%; semester 2, 81.8%). Respondents were also asked to self-assess if they felt the module objectives had been accomplished. Combining these responses across all modules indicated that in Semester 1, 78% (n=117) and in Semester 2, 62.1% (n=41) felt they had met all the objectives in the modules. Further, in Semester 1, 11.3% (n=17) and in Semester 2, 21.2% (n=14) felt they had met one or more of the objectives.
In open-ended questions participants were invited to comment on what worked well, why it worked, what needed improvement and what further assistance they felt they needed. Comments were grouped into the following themes.
Many participants commented on the "hands on" and "practical" approach of the activities in the site. The instructions and guidelines for activities in each module were described as "well structured", "easy to read" and "easy to follow". Importantly, students felt they were "not overwhelmed with too many instructions" and actively participated in the activities embedded in each module. Participants reported active behaviours such as "clicking around", "practising", "experimenting", "making mistakes and correcting them", "flipping through the windows", "enacting the tasks" and referring to the instructions "repeatedly".
Participants expressed the perceived relevance of the site in a variety of ways. They saw the activities in the site as both "real" and "enabling". The use of real discussion groups drew many positive comments as encouraging participation "beyond the simple posting of singular messages" and providing support and encouragement such as seeing "what other people had to say regarding the problems they were having also". In activities such as submission of assignments, participants noted how closely the practice task mirrored the requirements of assessment tasks within the courses in which they were enrolled. In activities such as managing email accounts, searching electronic databases and locating and downloading electronic journal articles through the library, participants expressed enthusiasm and relief at being able to see the immediate results of their efforts and being guided to successfully complete tasks which had previously been a "source of intimidation and concern". Only one participant indicated that they would have found the site better if it had been made available only to students within the same course.
A safe environment
Aligned to participants' valuing the relevance and the opportunity to practise, many also enthusiastically expressed the value of doing so in an environment they considered to be "non-threatening". The opportunity for practice activities was seen as "confidence building" and allaying "anxiety".
The timing of access to the resource was mentioned by both Semester 1 and Semester 2 participants. Continuing students offered the resource in Semester 2 indicated that, although the resource was still useful, access to it in the first semester of their enrolment would have been preferable.
I found that the example questions worked well as they enabled a better insight into the module. It is a shame this approach to course requirements was not incorporated at the beginning of the first semester.Commencing students also indicated that earlier access to the resource would have been helpful (timely availability is addressed in the Lessons section below)
It came a bit late should have been introduced on the first semester on my part. But it is good tool.
I felt I should have received this much earlier, especially when I was accepted on the course, now I feel am rushing this, when there is a lot of other things that have to be taken into consideration. I think this is a brilliant idea & is good for those who haven't studied in ages
Most things worked well but I could not log into any of these activities until the 6th of March which made completing them before the start impossible. I was looking daily...
Extra time at the start would have saved me hours later on. It works well because it is like a practice run to the real thing.Examining the initial student access patterns to GetLearning has helped to identify the most pertinent types of interventions that can assist the student's entry to the real online learning environment. Individualised supports and feedback can assist students to make the transition to online learning and feel comfortable about participating in online learning communities. GettingOnTrack models these interventions and provides advice to other academics considering how they might incorporate this into their own context.
At the same time this research made visible some conundrums for student support and for research into its impact, with factors that promote student engagement potentially at odds with institutional constraints and the culture of academic autonomy. In addition there are continuing challenges for conducting useful research with busy students.
The Student Induction to E-Learning report (IMS Global Learning Consortium Inc, 2010) highlights the need for a sustained and multifaceted approach, beginning prior to enrolment, as well as during the early stages of study, recommending guidance and support for learner centred activities as well as the development of remediation action plans for students at risk of withdrawing from study. The concept of 'just in time' support and incremental development of skills has been recommended as a key issue requiring further investigation (Forrester & Motteram, 2005). There is an emerging trend towards universities providing opportunities for new students to engage more actively with the university culture. One example is the University of Western Australia's Facebook page which is directing its support to students in the pre-enrolment period (Cluett, 2010) - such an approach circumvents institutional administrative limitations, but also potentially takes students into less secure territory where issues of digital identity become more apparent (Wozniak, Uys & Mahony, 2008).
Moderation in our view is a broader role, described as both risk management for individual students and establishing institutional and limited teacher presence (Garrison & Anderson, 2003). Our findings indicated the value of using LMS tracking data to identify students who have not entered the site (inactive), who appear to have lost their way in its technicalities, and/or who have chosen a pathway to suit their own perception of their needs with attendant risks to being ill-prepared in the areas skipped. This was the most concerning for Semester 2 where moderation did not occur, and about 50% of students did not access any of the module content. Perhaps this was because in Semester 2 students were not new to online study but perhaps they needed greater scaffolding and support. In the case of GetLearning, our particular concern was for those students who did not engage in the discussion board activities included in modules 2 and 3. Moderation of discussion board in a minimalist but visible presence is necessary, not only for the activities recommended by Salmon (2000), but to contribute to the sense of institutional teacher presence (Shin, 2003) for students new to the institution.
Coupled with this dilemma is the consideration as to whether to mandate participation in such resources. Recently Carruth et al (2010) described an online course for graduate nursing students which over time, has become a compulsory component for entry of all new masters level students. Students receive continuing education credits for completion, which contributes to their nursing registration (a reward for learning activity which may be perceived by students as additional to their formal coursework). Requirements include participation in discussion forums with feedback provided via an assessment rubric, completion of a quiz as well as engaging in other collaborative activities such as using a wiki. These authors also note the advantage of students experiencing and participating in a non-threatening online learning environment with survey evaluations reinforcing the value of practising "being an online learner" as a way to reduce the anxiety and apprehension felt by commencing students.
Carruth, A. K., Broussard, .P. C., Waldmeier, V. P., Gauthier, D. M. & Mixon, G. (2010). Graduate nursing online orientation course: Transitioning for success. Journal of Nursing Education, 49(12), 687-690. http://dx.doi.org/10.3928/01484834-20100831-06
Chang, S. C. & Tung, F. C. (2008). An empirical investigation of students' behavioural intentions to use the online learning course websites. British Journal of Educational Technology, 39(1), 71-83. http://dx.doi.org/10.1111/j.1467-8535.2007.00742.x
Cluett, L. (2010). Online social networking for outreach, engagement and community: The UWA Students' Facebook page. In Educating for sustainability. Proceedings of the 19th Annual Teaching Learning Forum, 28-29 January 2010. Perth: Edith Cowan University. http://otl.curtin.edu.au/tlf/tlf2010/refereed/cluett.html
Collins, A., Joseph, D. & Bielaczyc, K. (2004). Design research: Theoretical and methodological Issues. Journal of the Learning Sciences, 13(1), 15-42. http://dx.doi.org/10.1207/s15327809jls1301_2
Crosling, G. & Heagney, M. (2009). Improving student retention in higher education: Improving teaching and learning. Australian Universities Review, 51(2), 9-18. http://www.nteu.org.au/library/view/id/502
Freeman, M., Clarkeburn, H., & Treleaven, L. (2007). A collaborative approach to improving academic honesty. In A. Brew & J. Sachs (Eds.), Transforming a university: The scholarship of teaching and learning in practice (pp. 153-162). Sydney: Sydney University Press. http://hdl.handle.net/2123/2127
Garrison, D. R. & Anderson, T. (2003). E-learning in the 21st century: A framework for research and practice. London: RoutledgeFalmer.
IMS Global Learning Consortium Inc. (2010). IMS GLC Student Induction to E-Learning: A best practices framework for promoting post secondary student retention associated with induction to e-learning. http://www.imsglobal.org/siel/index.html
Kennedy, G. E., Judd, T. S., Churchward, A., Gray, K. & Krause, K. L. (2008). First year students' experiences with technology: Are they really digital natives? Australasian Journal of Educational Technology, 24(1), 108-122. http://www.ascilite.org.au/ajet/ajet24/kennedy.html
Kehrwald, B., Rawlins, P. & Simpson, M. (2011). Learner experiences of online learning in a blended learning situation: Different cohorts, different needs. In Changing demands, changing directions. Proceedings ascilite Hobart 2011. (pp.671-676). http://www.ascilite.org.au/conferences/hobart11/downloads/papers/Kehrwald-concise.pdf
Ko, S. & Rossen, S. (2010). Teaching online: A practical guide (3rd ed.). New York: Routledge.
Krause, K.-L. & McEwen, C. (2009). Student induction to e-learning: A progress report. University of Southern Queensland report by Link Affiliates Team within the Australian Digital Futures Institute. http://www.linkaffiliates.net.au/Publications/SIeL_Mar09_Report.html
Lever, T., Mahony, M. J. & Wozniak, H. (2007). GetReal: Building and managing essential academic learning from the academic periphery. In ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007. http://www.ascilite.org.au/conferences/singapore07/procs/lever.pdf
Levy, P. (2006). 'Living' theory: A pedagogical framework for the process support in networked learning. Research in Learning Technology, 14(3), 225-240. http://dx.doi.org/10.1080/09687760600837025
Motteram, G. & Forrester, G. (2005). Becoming an online distance learner: What can be learned from students' experiences of induction to distance programmes? Distance Education, 26(3), 281-298. http://dx.doi.org/10.1080/01587910500291330
O'Donnell, C. M., Sloan, D. J. & Mulholland, C. W. (2006), Evaluation of an online student induction and support package for online learners. European Journal of Open, Distance and E-learning, 01.04.2006, 1-12. http://www.eurodl.org/?p=archives&year=2006&halfyear=1&article=220
Price, L., Richardson, J. T. E. & Jelfs, A. (2007). Face-to-face versus online tutoring support in distance education. Studies in Higher Education, 32(1), 1-20. http://dx.doi.org/10.1080/03075070601004366
Reeves, T., Herrington, J. & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 96-115. http://dx.doi.org/10.1007/BF02961476
Reeves, T., McKenney, S. & Herrington, J. (2011). Publishing and perishing: The critical importance of educational design research. Australasian Journal of Educational Technology, 27(1), 55-65. http://www.ascilite.org.au/ajet/ajet27/reeves.html
Scagnoli, N. I. (2001). Student orientations for online programs. Journal of Research on Technology in Education, 34(1), 19-27.
Salmon, G. (1998). Student induction and study preparation online. Paper presented at the Telematics in Education Seminar. http://atimod.com/research/presentations/Finland2.doc
Salmon, G. (2000). E-moderating: The key to teaching and learning online. London: Kogan Page.
Shin, N. (2003). Transactional presence as a critical predictor of success in distance learning. Distance Education, 24(1), 69-86. http://dx.doi.org/10.1080/01587910303048
Sun, P. C., Tsai, R. J., Finger, G., Chen, Y. Y. & Yeh, D. (2008). What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1183-1202. http://dx.doi.org/10.1016/j.compedu.2006.11.007
Tomei, L., Hagel, H., Rineer, A., Mastandrea, L. A. & Scolon, J. (2009). Do orientation materials help students successfully complete online courses? International Journal of Information and Communication Technology Education, 5(2), 73-89. http://www.igi-global.com/article/orientation-materials-help-students-successfully/2375
Wozniak, H. (2007). Empowering learners to interact effectively in asynchronous discussion activities. In M. Bullen & D. Janes (Eds.), Making the transition to e-learning: Strategies and issues (pp. 208-228). Hershey, PA: Idea Group Publishing.
Wozniak, H., Mahony, M. J., Lever, T. & Pizzica, J. (2009). Stepping through the orientation looking glass: A staged approach for postgraduate students. Australasian Journal of Educational Technology, 25(2), 221-234. http://www.ascilite.org.au/ajet/ajet25/wozniak.html
Wozniak, H., Mahony, M. J., Pizzica, J. & Koulias, M. (2007). How do students 'get learning'? Unexpectedly diverse pathways in an activity based, online orientation site. In ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007. http://www.ascilite.org.au/conferences/singapore07/procs/wozniak.pdf
Wozniak, H. & Silveira, S. (2008). Transforming learning: Using structured online discussions to engage learners. In A. Brew & J. Sachs (Eds.), Transforming a university: The scholarship of teaching and learning in practice (pp. 163-174). Sydney: Sydney University Press. http://hdl.handle.net/2123/2128
Wozniak, H., Uys, P. & Mahony, M. J. (2012). Digital communication in a networked world. In J. Higgs, R. Ajjawi, L. McAllister, F. Trede & S. Loftus (Eds; 3rd ed), Communicating in the health sciences. Melbourne: Oxford University Press.
Zaritsky, R., Kelly, A. E., Flowers, W., Rogers, E. & O'Neill, P. (2003). Clinical design sciences: A view from sister design efforts. Educational Researcher, 32(1), 32-34. http://dx.doi.org/10.3102/0013189X032001032
|Authors: Helen Wozniak, Senior Lecturer, eLearning Coordinator, Northern Territory Medical Program|
FlindersNT, Flinders University, PO Box 362, Charles Darwin University NT 0815
Email: Helen.Wozniak@flinders.edu.au Web: http://www.flinders.edu.au/people/helen.wozniak
Jenny Pizzica, Lecturer, Institute for Interactive Media and Learning
University of Technology, Sydney PO Box 123 Broadway NSW 2007
Email: Jenny.Pizzica@uts.edu.au Web: http://datasearch.uts.edu.au/iml/staff/details.cfm?StaffId=3848
Dr Mary Jane Mahony, Honorary Senior Lecturer
CoCo, Faculty of Education and Social Work, University of Sydney
Email: firstname.lastname@example.org Web: http://sydney.edu.au/education_social_work/coco/
Please cite as: Wozniak, H., Pizzica, J. & Mahony, M. J. (2012). Design-based research principles for student orientation to online study: Capturing the lessons learnt. Australasian Journal of Educational Technology, 28(5), 896-911. http://www.ascilite.org.au/ajet/ajet28/wozniak.html