|Australasian Journal of Educational Technology
2006, 22(4), 568-580.
This research presents the results of a study of alternative response methods for in class formative questioning. Students' anonymity from their peers and instructor was studied through a research design that maintained a constant interactive teaching strategy in a large lecture hall, in all respects except for the method used by students to respond to the in class questions. A handheld electronic response keypad was the only approach affording complete anonymity. Student perceptions of the benefits of anonymity were obtained from a survey conducted at the end of the course. The results suggest that anonymity is a critical factor affecting student willingness to participate with in class exercises. Furthermore, the results indicate that students' propensity to engage with in class questions increases with the degree of anonymity provided to the student in revealing their response. The benefit of anonymity, combined with the increased availability and affordability of electronic response systems, will be of interest to academics keen to design engaging learning environments.
It is well accepted that quality learning outcomes are more likely to occur when students adopt a deep approach to learning (Ramsden, 2003). While there are various factors that determine a student's approach to learning, such as prior learning experiences, the student's perception of the learning context is a strong determinant over which academics have influence. Assessment is the most obvious way academics can influence perceptions and behaviour. Ramsden (2003, p.182) takes this further by concluding that "from our students' point of view, assessment always defines the actual curriculum". Academics can also impact the student's perception of the learning context by the in class experience. In a large research project with physics classes, Hake (1998) shows that interactive classes were correlated with higher student achievement. While Biggs (2003) reiterates that it is what the student does that directly affects learning outcomes, achieving interaction and engagement in large classes is more difficult.
Promoting interaction in larger lecture halls can be achieved with students attempting questions individually and/or with peers (Boud, Cohen & Sampson, 1999). Student responses can be revealed by asking for or selecting one or more students to share their thoughts. In large classes either of these can be intimidating (e.g. unsure students may feel uncomfortable being selected, and shy students may feel uncomfortable being the focus of attention even if they know the correct response). Such alternatives are possibly ineffective and time consuming as well (e.g. hearing responses from students at the back of the lecture hall). Asking for a show of hands (i.e. 'hands up if you think the answer is A?') is somewhat less intimidating because some anonymity is possible amidst a crowd of hands (unless a further public verbal response might be required). While students can gauge their relative understanding from viewing the distribution of responses and the confirmation of the correct response from the instructor, it is limited by the class participation. While relative understanding can only be approximated by the hands up method, it should be sufficient for the academic to decide if further discussion is needed. An alternative to a sequential show of hands is to provide students with different coloured paper for each choice (e.g. Apricot for A, Blue for B etc) and thus seek their responses simultaneously (Harden, Wayne & Donald, 1968). However, in addition to it being administratively cumbersome (e.g. distribution of the coloured paper), this method still fails to provide complete anonymity to the students.
An alternative, more anonymous approach is to use a paper based formative quiz that is marked on completion in class. Dihoff, Brosvic and Epstein (2004) report a novel approach that can provide feedback even sooner. Feedback is available immediately following each quiz question when students use a pre-formatted response sheet (called an Immediate Feedback Assessment Test or IF-AT). Students scratch off their preferred response as if they were scratching a lottery ticket. If the first response scratched is incorrect, further scratches are made which can earn partial credit. Dihoff, Brosvic & Epstein (2004) report improved student achievement in subsequent tests or exams from students receiving such immediate feedback. This is no surprise because students are ripe for learning at such 'teachable moments' (Dempsey, Driscoll & Swindell, 1993). The main drawback with this method is the loss of spontaneity for the instructor to ask new questions since all questions must be pre-ordered to fit the pattern of answers beneath the scratchable response sheet's pre-formatted surface.
The final approach for eliciting and gathering students' responses is facilitated by electronic means. Quizzes and tests can be marked by portable electronic scanner in the classroom, interactive computer assisted quizzes in an online or lab context, mobile phone, or a handheld audience electronic response keypad. Michaelsen, Knight and Fink (2004) report positive benefits from the use of portable scanners, but with a loss of flexibility for questioning for formative purposes. While computer labs are a scarce resource and wireless enabled laptops and mobile phones are still emerging as solutions for electronic questioning (Prensky, 2005) handheld electronic response systems appear to be the most viable current solution. Mazur (1997) provides an overview of how ERS can be used to facilitate student learning in large lectures and particularly with peer discussion.
The level of anonymity provided by these different in class response methods is represented in Figure 1. At one end of the spectrum handheld ERS and other electronic means for obtaining student responses provide a high level of anonymity. At the other extreme is a student being randomly selected in class to provide a response to an instructor's question - many students, especially the shy, less confident or unprepared ones, may feel threatened in such circumstances. Large lecture halls exacerbate this problem.
Figure 1: Anonymity provided by different interactive teaching strategies
Even with academics purposefully designing for student-instructor interaction in large lecture halls using ERS, there will still be barriers for some students to participate actively. This reluctance can be alleviated partially by giving space for student-student interaction to achieve peer learning, sometimes combined with self reflection, prior to student-instructor interaction. However, Draper and Brown (2004) observed that even in small classes when students were uncertain of the most appropriate response, they were more reluctant to join a discussion. This occurred even where students knew each other well and previously had not hesitated to join an oral group discussion.
Draper, Cargill and Cutts (2002, p.15) note the advantage over the hands up alternative, in that ERS "crucially offers more privacy (it's a secret ballot, and important for just the same reasons)". Crouch and Mazur (2001) make a similar observation. Poulis et al. (1998) suggest ERS use is associated with a reduction in fear and potential peer ridicule. Banks (2003) suggests that cultural background may be significant with a preference for ERS from those for whom criticism is problematic. Draper and Brown (2004) propose that increased understanding arises from anonymity because it induces students to pick a definite answer even when they are quite uncertain, and it is the latter that encourages student effort to then engage to produce an answer. While this may be similar to the pressure on a juror to make a decision when they feel ill equipped to do so, it is different in that a student's decision does not have major irreversible consequences. Academics can choose to provide instant feedback on wrong answers, and if in class discussion is insufficient, students are more likely to be motivated to clarify their understanding after the class. While new technology may have some novelty effect, Draper and Brown (2004; p.89) conclude in relation to ERS that "it seems to last only somewhere between 5 and 50 minutes".
However, notwithstanding these advantages of ERS, there is considerable research to show that technology in and of itself does make any significant difference to student learning achievements (Cuban, 1986; Russell, 1999; Alexander & McKenzie, 1998; Kirkwood & Price 2005). What matters is the pedagogical design within which technology is used to facilitate learning. Following their review of three decades of ERS research, Judson and Sawada (2002; p.167) agree that this principle also applies specifically to ERS educational technology, concluding that the
...literature points to the pedagogical practices of the instructor, not the incorporation of the technology as being key to student comprehension. Electronic response systems are viewed as a tool that holds a promise of facilitating earnest discussion.This research explores whether the anonymity afforded by ERS is a key contribution to the learning environment, valued by students over alternative response methods. It follows up on a study (Freeman & Blayney, 2005) that compared the use of an ERS with a traditional response method (i.e. a show of hands), when students were exposed to an equivalent learning context in all respects except response method, which alternated weekly between hands up and ERS. Students were encouraged to engage in the learning activity by interacting with their peers before providing a response and by strong instructor encouragement to take a stand. While the instructor successfully elicited close to 100% participation with the ERS a much lower response rate was achieved with the show of hands instances. An unexpected finding was that students' perceptions of their level of interaction in lectures, their understanding and their ability to gauge their own understanding, was significantly higher when the ERS was used. Anonymity was one factor highlighted in the qualitative research responses with statements like "everyone gets to participate without coming out" and "I don't know why I am afraid to ask the lecturer questions". The current study has attempted to evaluate anonymity as provided by the ERS as the key factor behind these improved learning experiences afforded by ERS educational technology.
|Stage of in class learning environment|
Table 1 summarises the two teaching strategies:
Figure 2: A typical ERS enhanced learning activity
The survey sought student evaluations on seven rating statements using a five-point Likert scale (strongly disagree to strongly agree). Four of these statements sought ratings on various aspects of learning design and the remaining three statements directly addressed the role of anonymity.
To assist with the further analysis various demographic data was collected including gender, age, tuition basis (i.e. government subsidised or fully self funded), hours per week working for payment, and first language. Finally, the survey sought students' evaluations using two free response items to obtain qualitative feedback on the best aspects of the course and those aspects needing improvement.
Statistical analysis of the survey results was undertaken in several different forms. Students' responses to the Likert scale items were analysed using both the "percentage agreement" metric (percentage endorsing either "agree" or "strongly agree") and the "percentage disagreement" metric (percentage endorsing either "disagree" or "strongly disagree"). Students' ranking of response preferences were analysed using the Friedman test (Conover, 1980) with a Bonferroni correction for follow up multiple comparison tests between specific preferences (Hsu, 1996). Mann-Whitney and t-tests were conducted to test for differences due to demographic variables.
|Greater class preparation||Greater follow up of learning gaps||Greater learning by peer discussion|
The issue of anonymity was directly addressed in Table 3. Students generally prefer to not reveal their identity when they respond to in-class questions. Students value anonymity to their instructor (68%) more than anonymity to their peers (62%). Anonymity is important when students are uncertain of the answer (63%). Of course, there is always likely to be a proportion of students who are less shy and more self confident and thus for whom anonymity is not an issue. We surmise that it is those students who might be more likely to participate in a show of hands or even offer to respond if the instructor makes a general request to the class for an answer. Again there were no significant differences in these ratings for any of the demo graphic variables.
|Preference for anonymity|
|Preference for anonymity|
from other students
|Preference for anonymity|
Table 4 indicates students' preferred response method as identified by the mean ranking per preference. The method affording the greatest anonymity, namely the ERS, had the highest ranking (mean ranking 1.4), followed by show of hands (mean ranking 2.3), volunteering an answer (mean ranking 2.65) and being selected by the instructor as least preferred (mean ranking 3.66). A complete set of rankings was given by 124 students. The overall test was significant, Q(3) = 194.86, p < 0.001, indicating that statistically reliable differences existed between students' rankings of preferences. The results of the Bonferroni-corrected pairwise comparisons between rankings, using a critical value for the difference of 53.42 are in the final three columns of Table 4.
|Matrix of pairwise|
|1.||Using a keypad to submit my answers with my identity anonymous to academic and peers||1.40||277.50 *||154.00 *||110.50 *|
|2.||Raising my hand when asked how many students thought the answer was "a", "b", "c", etc.||2.30||167.00 *||43.50 ns|
|3.||Raising my hand when I knew the answer and then being asked to respond||2.65||123.50 *|
|4.||Being selected by the academic to provide my answer||3.66|
|* difference between preference rankings statistically significant|
ns difference between preference rankings not statistically significant
The results in Table 4 concur with our expectations, namely that the level of anonymity afforded by a response method is a good predictor of students' preferences. This preference is significant in all cases except between alternate variations of raising hands.
Further analysis of the above ranking data using Mann-Whitney tests found no significant difference in preference ratings between any of the demographic variables. Interestingly, there was no significant difference in preference ratings between students whose first language was English or those for whom it was not. Using first language as a rough proxy for culture, and in particular openness to criticism, these results might be seen to contradict Banks (2003) who suggested that cultural background could impact preference for ERS usage. Future research should explore this with a better proxy.
While open discussion is possible (and even the norm) in small post-graduate classes, this is not the case in a typical university lecture hall with a large class of undergraduate students. As class sizes increase due to funding constraints, the findings in this study are increasingly important for academics wishing to engage students in such challenging contexts. However, despite the benefits for student learning, the costs to instructors of adopting ERS are not necessarily small. While the monetary costs of such systems have declined with technological improvements in recent years, and even with textbook publishers strongly supporting such adoptions (with limited or no cost to the instructor), penetration into higher education contexts appears to be slow and quite limited. Physics appears to be the exception (Sharma et al., 2005; Crouch & Mazur, 2001). This may be due in part to strong cultural factors within disciplinary circles and the earlier influential work of Mazur (1997), specifically in physics. Indirect costs on instructors, such as the time and hassle of the distribution and collection of keypads, also need to be further explored. The instructor experience with ERS is the subject of other systematic research currently underway.
The limitations of this study include sample size, the single site context, and focus on perceptions. These suggest profitable future opportunities for research relating to ERS and anonymity benefits to students that can be achieved by larger samples and other contexts. Experimental research methods that focus on increased understanding, and learning contexts designed to provide anonymity using ERS or other technologies are possible research avenues. Better proxies for cultural factors, such as openness to criticism to unpack the influence on anonymity, may also be useful to investigate. Finally, further research focussing on the instructor experience with ERS would be particularly useful.
Notwithstanding these limitations, the finding of the benefit of anonymity in promoting in class engagement and interaction, combined with the increased availability of and anonymity afforded by electronic response systems, will be of interest to instructors keen to design engaging learning environments. These findings will also be of interest to those investigating educational technologies more generally that afford anonymity since the latter appears to be an impo rtant aspect increasing engagement and interaction.
Banks, D. (2003). Using keypad-based group process support systems to facilitate student reflection. In G. Crisp, D. Thiele, I. Scholten, S. Barker & J. Baron (Eds), Interact, Integrate, Impact: Proceedings of the 20th ASCILITE Conference, Adelaide. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/37.pdf
Biggs, J. (2003). Teaching for quality learning at university. 2nd ed. Buckingham: Open University Press.
Boud, D., Cohen, R. & Sampson, J. (1999). Peer learning and assessment. Assessment and Evaluation in Higher Education, 24(4), 413-426. [verified 31 Oct 2006] http://www.education.uts.edu.au/ostaff/staff/publications/db_23_bcs_aehe99.pdf
Conover, W. J. (1980). Practical nonparametric statistics. New York: John Wiley.
Crouch, C. H. & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American Journal of Physics, 69(9), 970-977.
Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. New York: Teachers College Press.
Dempsey, J. V., Driscoll, M. P. & Swindell, L. K., (1993). Text-based feedback. In J.V. Dempsey & G. C. Sales (Eds), Interactive Instruction and Feedback (pp.21-54). Englewood Cliffs, NJ: Educational Technology Publications.
Dihoff, R. E., Brosvic, G. M. & Epstein, M. L. (2004). Provision of feedback during preparation for academic testing: Learning is enhanced by immediate but not delayed feedback. The Psychological Record, 54(2), 207-231. [verified 31 Oct 2006] http://www.med.wright.edu/aa/facdev/_Files/PDFfiles/ImmediateFeedback.pdf
Draper, S. W. & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94.
Draper, S. W., Cargill, J. & Cutts, Q. (2002). Electronically enhanced classroom interaction. Australian Journal of Educational Technology, 18(1), 13-23. http://www.ascilite.org.au/ajet/ajet18/draper.html
Freeman, M. & Blayney, P. (2005). Promoting interactive in-class learning environments: A comparison of an electronic response system with a traditional alternative. In S. L. Cheung (Ed), Innovation for Student Engagement in Economics: Proceedings of the Eleventh Australasian Teaching Economics Conference, pp.23-34. http://hdl.handle.net/2123/199 [viewed 13 Oct 2006, verified 31 Oct 2006]
Hake, R. (1998). Interactive engagement versus traditional methods: a six thousand - student survey of mechanics test data for introductory physics course. American Journal of Physics, 66(1), 64-74.
Harden, R. M., Wayne, E. & Donald G. (1968). An audio-visual technique for medical teaching. Journal of Medical and Biological Illustration, 18(1), 29-32.
Hsu, J. C. (1996). Multiple comparisons: Theory and methods. London: Chapman & Hall.
Judson, E. & Sawada, D. (2002). Learning from past and present: Electronic response systems in college lecture halls. Journal of Computers in Mathematics and Science Teaching, 21(2), 167-181. [verified 31 Oct 2006] http://www.aace.org/dl/files/JCMST/JCMST212167.pdf
Kirkwood, A. & Price, L. (2005). Learners and learning in the twenty first century: What do we know about students' attitudes towards and experiences of information and communication technologies that will help us design courses? Studies in Higher Education, 30(3), 257-274.
Mazur, E. (1997). Peer instruction: Manual. New Jersey: Prentice Hall.
Michaelsen, L., Knight A. & Fink, L. (2004). Team-based learning: A transformative use of small groups. Stylus Publishing LLC.
Poulis, J., Massen, C., Robens, E. & Gilbert, M. (1998). Physics lecturing with audience paced feedback. American Journal of Physics, 99(5), 439-441.
Prensky, M. (2005). What can you learn from a cell phone? Almost anything!. Innovate, 1(5). http://www.innovateonline.info/index.php?view=article&id=83 [viewed 15 June 2005, verified 31 Oct 2006]
Ramsden, P. (2003). Learning to Teach in Higher Education. 2nd ed. London: Routledge.
Russell, T. (1999). The 'No Significant Difference' Phenomenon. http://www.nosignificantdifference.org/
Sharma, M. D., Khachan, J., Chan, B. & O'Byrne, J. (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. [viewed 21 Aug 2006] http://www.ascilite.org.au/ajet/ajet21/sharma.html
|Authors: Mark Freeman|
Office of Learning and Teaching in Economics and Business
Faculty of Economics and Business, The University of Sydney
Paul Blayney, Discipline of Accounting
Faculty of Economics and Business, The University of Sydney
Paul Ginns, Institute for Teaching and Learning, The University of Sydney.
Please cite as: Freeman, M., Blayney, P. and Ginns, P. (2006). Anonymity and in class learning: The case for electronic response systems. Australasian Journal of Educational Technology, 22(4), 568-580. http://www.ascilite.org.au/ajet/ajet22/freeman.html