An investigation of the effectiveness of electronic classroom communication systems in large lecture classes

Since 2002 we have been investigating the use of an electronic classroom communication system in large first year lecture classes. Handheld keypads were distributed to teams of students during a lecture class. Students used the keypads to answer two step multiple choice problems after a discussion within their group. The questions were generated using students' answers from previous exams. We have evaluated our use of the classroom communication system using a survey about how comfortable students are with this type of interaction. In addition, we have tried to determine if the use of the classroom communication system can be linked to student performance on exams. Our results show that students are comfortable with this technology and feel that, on the whole, interactive lectures are useful. At a first glance, there is an improvement in students' exam performance, but there are too many competing factors to clearly say that this improvement is solely due to the use of the classroom communication system. Even though this paper is based in physics and a physics example is used to illustrate points, the technique can be applied to other discipline areas.

Introduction

Classroom communications systems have been used in various ways for enhancing teaching and learning. Burnstein & Lederman (2001) and Shapiro (1997) have used a CCS for summative assessment in large lecture classes, while Dufresne et al. (1996) have used a CCS in highly structured lecture courses based on a specially designed learning and question cycle. In general they can be used for revision, reinforcement, calculations, quizzes and assessment, each of which can be carried out interactively (for examples see Irving et al., 2000; Jones et al., 2001; Elliot, 2003; McCabe & Lucas, 2003; Williams, 2003). A design rationale for introducing CCS is presented by Draper et al (2002) and our proposal for using the CCS is in Sharma et al. (2002a). We are changing the teaching method as well as the technology used in delivery. No attempt has been made to add to the Clark-Kozma debate by separating the effects of the teaching method and technology (Clark, 1983; Kozma, 1991).

At the University of Sydney we are using the CCS in large first and second year physics lecture classes, both as an integral part of the lectures and as revision. The objectives are to

• encourage deeper thinking by the students,
• reinforce and link concepts,
• promote student-student and lecturer-students interactions,
• provide instantaneous feedback to lecturer and students on the students' progress.

The classroom communication system

In our case a hand held unit is shared between two or three students who are expected to collectively choose one of several options and a histogram of the number of students choosing each option is displayed. The data can be readily exported for further analysis.

The system has the following additional features that allow flexibility of use.

• Each handheld unit has an individual number that is displayed on the video projector so that students can tell that their response has been registered.
• The lecturer can determine the maximum number of times a unit can respond before the answer is locked, thus allowing students to change their minds.
• The lecturer sets a clock that counts down but the clock can be paused or reset while it is counting.
• Students can indicate how confident they are about their answer.
• Data can be displayed in other forms that depend on the type of interaction that the lecturer aims to achieve.

Teaching and learning strategies used

A schematic of the process used in class is shown in Figure 1.

Figure 1: A schematic of the learning cycle used with the CCS.

A multiple choice problem is displayed on an overhead projector in parallel with the video projector displaying the CCS screen. The lecturer reads the question then starts the clock, giving the class 2 to 3 minutes to discuss the question and submit an answer (box 1 in Figure 1). When the time has run out a histogram displays the collective responses to the whole class. Almost always the histogram is greeted with a few seconds of silence prior to team discussions as students compare their answers with those displayed (i.e. observe, box 2). The result provides the lecturer with instant feedback from the class and he/she provides instant feedback to the class by interactively discussing each option. Students are more inclined to interact because they become aware that others in the class think similarly to them. In addition, they have discussed and committed to an answer as a team and have an argument to defend. The challenge for the lecturer is to find flaws in the argument without deflating student enthusiasm and confidence. The lecturer-students dialogue aims to resolve students' conflicting ideas (box 3). The follow-up second multiple choice question is then displayed and students submit answers (box 4). The ensuing discussion summarises the ideas and reinforces the main points (box 5).

This cycle could be used with show of hands, or short written quizzes where the lecturer collects responses and student comments at the end of the class. However, the ability to get a response from everyone and provide instant feedback with the CCS makes the cycle more effective.

The answer, observe, resolve, answer, reinforce cycle in Figure 1 is a learning cycle similar to but not adopted from the predict, observe, explain cycle devised by Champagne et al. (1980). Such cycles are particularly useful when demonstrations or experiments can be used during the observe part of the cycle (for example Sokoloff & Thornton, 1997; Kearney, 2002). We have adapted this cycle to discuss abstract ideas that cannot be easily 'demonstrated' in large lectures to generate discussion and resolve conflicting ideas that students may have. The use of a teaching and learning cycle to enhance learning is important in view of the work done by Van Dijk et al. (2001). They showed that using a CCS with no learning cycle can generate poorer post-test results than not using a CCS, whereas the class with both a learning cycle and CCS did the best on the post-test. Mazur (1997) and Defresne et al. (1996) use a similar combination of a cycle with a CCS.

An example of using the CCS in class

We selected questions from 1999 and 2000 end of semester examination papers that required descriptive answers. A set of 50 student answers for the selected questions were sorted by the authors into phenomenographic categories that captured the variations in the answers (Marton & Saljo, 1976; Sharma et al., 2004). Phenomenography aims at describing the conceptions of learners, in contrast to the teacher's official view of a phenomenon. The method consists essentially of asking open ended questions (in this case, the exam problem requiring explanations), reading the open ended written responses repeatedly, and coming up with a post-hoc categorisation that assigns every answer to one of a few categories, corresponding to different types of conceptions. The perhaps surprising observation of researchers in this area is that this can nearly always be done: that there are just a few common conceptions of a phenomenon.

The categories were used to produce multiple choice questions and a range of answers. We found that for most exam problems the categories could be readily used to generate a pair of related multiple choice questions (to form a two step multiple choice problem). Figure 2 shows a typical two step multiple choice problem. In fact the pair arises naturally because the original open ended exam problem required an answer and an explanation. The first multiple choice problem addresses the answer and the second arises out of the explanation of the correct choice from the first multiple choice problem. The main advantage for using this methodology when writing choices is that it reflects what students are thinking, and these choices are often not present in existing databases of questions. For example, choice 2 from the first multiple choice problem in Figure 2 is rarely seen in database questions, but students do choose this option. The set of two step multiple choice problems developed so far are available at the website: http://www.physics.usyd.edu.au/super/CCS/Questions.html

The website also includes other questions that have been generated in the same way and tested in class. Of course this way of formulating a multiple choice question can be applied to any discipline.

A taut string is attached to a distant wall. A lecturer holding the unattached end moves his hand up and down to create a pulse travelling towards the wall. The lecturer now wants to produce a pulse that takes a longer time to reach the wall. Different ways of achieving this are considered below, select the best answer.

Figure 2: A typical two step multiple choice problem used in class with the CCS. Correct answers are 5 and 1.

A variation of this approach called two tiered multiple choice question asks students to make a choice then validate their choice as a written explanation. This technique explicitly probes students' understanding of a concept (Treagust, 1988; Tyson & Bucat, 1995; Christianson & Fisher, 1999). Since we already have students' explanations (as exam responses) we have used those to generate a second multiple choice question.

The multiple choice problems are 'wordy' and based on more than one concept. Solving them requires comprehension, interpretation of scientific jargon, and skills in analysing the situation and linking different concepts. Students also need to apply their understanding and knowledge of physics content. Other questions we have used ask students to choose between graphical answers which test understanding and interpretative skills: http://www.physics.usyd.edu.au/super/CCS/Questions.html

The problems are of the analysis and synthesis type described by Bloom (1956) and lend themselves to student-student and lecturer-students interactions. In comparison, questions of the recall type do not necessarily encourage discussion but serve other purposes such as revision.

Figure 3 shows the team responses during lectures to the two step multiple choice problem shown in Figure 2 (earlier presented in Sharma et al., 2002b). The question was used in two lectures denoted by L1 and L2 with 34 and 17 hand held units responding in the respective lectures. In most cases, each hand held unit was used by three students. We estimate that in total about 140 out of the 220 students enrolled in the course were present when the CCS was used with the question being discussed. That is, about 64% of the students were exposed to the problem used with CCS.

We note three important features from Figure 3. First, there are more incorrect answers to the first part in comparison to the second part of the problem. The improved responses to the second part are understandable and satisfying since displaying the first histogram and discussing the responses is designed to influence student responses to the second part. Second, the distributions for the two lectures are quite different. There are various possible explanations such as the type of lecturer-student interactions and differences in the student cohort. Third, 35% of teams from L1 and 25% from L2 have selected choice 2 for the first part. As noted earlier, choice 2 is not a response usually offered as an option in questions. It is interesting to note that we as lecturers do not expect this response.

Evaluation

Figure 3: Student group responses logged using the CCS with the two step multiple choice problem shown in Figure 2 (after Sharma et al. 2002b). The same two step multiple choice problem was used in both classes.

Evaluation using a survey

• how comfortable students were with particular teaching and learning strategies used in large lecture classes
• lecturing styles preferred by students
• the use of keypads.

Table 1 shows how comfortable students were with various interactive teaching and learning strategies. Almost all the students were comfortable or very comfortable with using the keypads and having small group discussions. In contrast, about 60% of the respondents were not comfortable with asking and answering questions in lectures.

Table 2 shows students' perceptions of how useful particular lecturing styles are. It is interesting but not surprising to note that out of the lecturing styles listed, the totally non-interactive lecture has the highest percentage of students saying that it is not useful and the least percentage saying that it is very useful. The use of demonstrations are consistently popular with students, but tend to be "show and tell" exhibitions rather than interactive. Students, in principle agree that "lectures where the lecturer makes you think" are useful. Even though students prefer "lectures where the lecturer asks questions and uses discussions" they are not very comfortable with asking and answering questions in lectures (see Table 1). The Fundamentals students, on the whole, prefer interactive lecturing with the electronic keypads rated as being very useful by 45% of the respondents and useful by 46% of the respondents.

Table 1: How comfortable were students with particular teaching and learning strategies used in large lecture classes. The values indicate the percentage of students who responded. A total of 138 students responded.

Table 2: Students' preferred lecturing styles. The values indicate the percentage of students who responded. A total of 138 students responded.

The survey had 4 open ended questions. For each question we simply grouped comments that were similar. If a response had different comments then they appeared in two or more groups. The comments to questions 1 and 2 were firstly grouped independently to allow groups for each question to form naturally. The groups formed for each question were then compared. We found a one to one match between the groups for these two questions.

The first question was

Q1. What was the best thing about the keypads?

In total, 120 students responded to this question and 176 comments were noted. There were 7 groups of comments. The groups, with comments that indicate the range of comments in the group, are listed below.

Q1.1. Thinking/understanding/learning (18 comments)

makes you think about problems.....
...an opportunity to check whether what you have learnt is correct
...helps to understand the lecture and concepts
.....got to reinforce what you learnt

participating in the lecture
It's an innovative way of involving every single student in the class
got to discuss the questions with peers.....get their points of view...
group work is also beneficial, friends can sometimes explain things better than the lecturer

....interesting and a valuable learning experience
it was more fun than just sitting and listening
makes you feel more at ease-you get to have an input, but don't have to worry about being embarrassed
answering a question and knowing other answers while remaining anonymous

its something different to the ordinary lectures
just to split up the lecture a bit
brings up interesting graphs/ratios
being given other student ideas concerning physics
get to see questions from previous exams
...gives appropriate exam techniques

convenient, quicker
the fun thing about the keypads is you press a button and your number comes up on the screen
....they are easy to use.....
...and they are really novel
they gave quick graphical results
playing with buttons
....maybe if there were more flashing lights-red and yellow-blue ...
its more futuristic
you don't need to write down the answer

nothing-they were a waste of time
wastes learning time
nothing they don't serve to illustrate anything, just opinions on questions of the class

feeling like I was on "who wants to be a millionaire"
the lecture material stops

Q2. What was the worst thing about using the keypads?

In total, 81 students responded to this question and 87 comments were noted. The groups, with comments that indicate the range of comments in the group, are listed below.

Q2.1. Thinking/style of question or answer (9 comments)

the answer is often surprising and confusing because solution choices are similar & sometimes ambiguous
...did not have to think about answers because it was multiple choice
sometimes questions are vague and too confusing-maybe do simpler questions then harder questions
having conflicting ideas about the correct answer
when I get a wrong answer

you have to wait for the others to answer
difference in opinions between members
everyone in group had to decide on answer
discussing with people you don't know

the pressure to make an answer
they're disruptive
its really boring

time consuming during lessons
using them takes up precious, precious lecture time-maybe lectures should go for longer to accommodate these activities
they use up half the lecture and have very little benefit
the point being made in the question using the controllers, could be made without the controller, saving 5 minutes for another point/example

not being sure if your answer was recorded (hard to find number on screen)
pressing on the number while everyone is pressing
concentrated on pushing the button rather than getting the answer
maybe each student should have one

its not used more
nothing bad
well don't know yet

I always get grey (probably referring to the colour display in response to pressing the keypads)

Q3. Should the keypads be used more often?

Seventy-three students indicated that the keypads should be used more often, while 23 said they should not be. In addition 10 students said that the usage was fine, and there were 5 miscellaneous responses. In total 111 students responded to this question.

The final question asked for any other comments. 38 students responded and 13 comments were made about the use of the keypads in class. Four of these were negative, 7 positive and 2 neutral with the following being the most creative response.

We should work on a system where if you get the question wrong you get an electric shock.

Evaluation: A comparison of student answers on exams

1. Amplitude of pulse
2. Duration of pulse- relating to changes in period or frequency
3. Tension/mass- relating to changing physical properties of the string.

Fifty students' answers from 2002 (when the CCS was used) were compared to 50 from 1999 (when the CCS was not used). Students' answers were compared in two ways. First, we counted the number of answers that contained each technique that would affect the speed of the pulse. Hence an answer that contained two techniques was counted twice. A miscellaneous category was added for responses we could not understand. Second, we counted the number of answers that contained only one technique, and those that contained multiple techniques were placed in miscellaneous. Table 3 shows the numbers of answers from 1999 and 2002 that contained each technique and Table 4 those containing only one technique.

There are two points worthy of note when comparing the 1999 and 2002 data. First, more students in 2002 indicated that the categories are indeed mutually exclusive. Second, more students in 2002 obtained correct answers. A chi-squared test for the differences between the distributions of answers that indicate 'only one technique' gives a p-value that is significant for the two samples' being drawn from the same population (chi squared (3, N = 100) = 19.7, p <0.05). This means that there is a statistically significant difference between the 1999 and 2002 distributions for the case when 'only one technique' was used.

Table 3: A comparison of 1999 and 2002 student answers where we have counted the number of times each technique is mentioned. The categories are not mutually exclusive and the totals are greater than the 50 exam scripts used in the study. Undecipherable responses are placed in "uncategorisable". Tension is the correct answer.

Table 4: A comparison of 1999 and 2002 student answers where we have counted the number of times only one technique is mentioned. The categories are mutually exclusive and the totals equal 50 - the number of exam scripts used in the study. Undecipherable responses are placed in "uncategorisable", together with those that contain multiple techniques. Tension is the correct answer.

A variety of factors influence this result:

• There were subtle differences in the way the questions were phrased between the two years, making it difficult to directly compare the two years' responses.

• We do not take attendance during lectures, hence, for the 2002 cohort we cannot identify students who did actually use CCS. However, as noted earlier, approximately 64% of 2002 'Fundamentals' students have been exposed to the CCS.

• Students revise past years' exam papers, so some of the 2002 students are likely to have revised the 1999 exam question. The concepts covered by this question have also appeared in various guises in older exam papers. A fraction of the 2002 cohort met the question for the first time through CCS and for the second time in the exam while the majority of the 1999 cohort met the question for the first time in the exam.

• We have assumed that the samples for 1999 and 2002 are comparable in terms of ability, background, etc, and that the sample is representative. In 2002, no changes were made to the existing Fundamentals course apart from using the CCS in 6 out of the 39 lectures.

Evaluation: Anecdotal evidence

The CCS was enthusiastically received by a substantial number of students. They were highlighted as an interesting learning exercise in the Staff-Student Liaison Meetings held twice in the year.

Discussion

The two step multiple choice problems are generated from students' answers from previous exams, allowing the variations in the manner in which students answer questions to be selected as options rather than what we think the students may be thinking. Of course one can use interviews and other methods to determine what students are thinking.

The results of this study show that students are comfortable using the CCS, more so than with asking and answering questions in lectures. We have tried to determine if the use of the CCS has an impact on student performance on exams. There is an improvement in students' exam performance as determined by how students have answered an exam based on the same concepts with and without the CCS. However, this was not a controlled experiment in that the two groups of students were from different years. A controlled experiment allowing some students from the same class to use the CCS and others not, would not be practical in our educational setting. The use of the CCS has contributed to improvement in exam performance. On the whole, staff and students have valued the interactivity introduced by the use of CCS in lectures.

Acknowledgments

References

Bloom's Taxonomy. [verified 15 Mar 2005] http://www.officeport.com/edu/blooms.htm

Burnstein, R. A. & Lederman, L. M. (2001). Using wireless keypads in lecture classes. The Physics Teacher, 39, 8-11.

Champagne, A. B., Klopfer, L. E. & Anderson, J. H. (1980). Factors influencing the learning of classical mechanics. American Journal of Physics, 48, 1074-1079.

Christianson, R. G. & Fisher, K. M. (1999). Comparison of student learning about diffusion and osmosis in constructivist and traditional classrooms. International Journal of Science Education, 21, 687-698.

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53, 445-459.

Draper, S. W., Cargill, J. & Cutts, Q. (2002). Electronically enhanced classroom interaction. Australian Journal of Educational Technology, 18, 13-23. http://www.ascilite.org.au/ajet/ajet18/draper.html

Dufresne, R., Gerace, W., Leonard, W., Mestre J. & Wenk, L. (1996). A classroom communication system for active learning. Journal of Computing in Higher Education, 7, 3-47.

Elliot, C. (2003). Using a personal response system in teaching economics. International Review of Economics Education, 1. [verified 15 Mar 2005] http://www.economics.ltsn.ac.uk/iree/i1/elliott.htm

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand student survey of mechanics data for introductory physics courses. American Journal of Physics, 66, 64-74.

Irving, A. Read, M., Hunt, A. & Knight, S. (2000). Use of information technology in exam revision. Fourth International Computer Assisted Assessment Conference. [verified 15 Mar 2005] http://s-d.lboro.ac.uk/caaconference/conf2000/pdfs/readm.pdf

Jones, C., Connolly, M., Gear, A. & Read, M. (2001). Group interactive learning with group process support technology. British Journal of Educational Technology, 32(5), 571-586.

Kearney, M. (2002). Predict-observe-explain strategy supported by the use of multimedia. Learning Designs. [viewed 8 Apr 2004, verified 15 Mar 2005] http://www.learningdesigns.uow.edu.au/exemplars/info/LD44/

Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61, 179-211.

Marton, F. & Säljö, R. (1976). On qualitative differences in learning: I - outcome and process. British Journal of Educational Psychology, 46,4-11.

Mazur, E. (1997). Peer Instruction: A User's Manual. Upper Saddle River, NJ: Prentice-Hall.

McCabe, M. & Lucas, I. (2003). Teaching with CAA in an interactive classroom: Death by PowerPoint - life by discourse. Seventh International Computer Assisted Assessment Conference. [verified 15 Mar 2005] http://s-d.lboro.ac.uk/caaconference/conf2003/pdfs/mccabe2.pdf

Poulis, J., Massen, C., Robens, E. & Gilbert, M. (1998). Physics lecturing with audience paced feedback. American Journal of Physics, 66, 439-441.

Shapiro, J. A. (1997). Student Response System. Rutgers University Department of Physics and Astronomy [verified 15 Mar 2005] http://www.physics.rutgers.edu/~shapiro/SRS/index.html

Sharma, M. D., Khachan, J., Chan, B., Stewart, C., Hogg, K. & O'Byrne, J. (2002a). Interactive lecturing using a classroom communication system. Proceedings of the UniServe Science Workshop, April 2002, University of Sydney, 87-89. [menu; verified 15 Mar 2005] http://science.uniserve.edu.au/pubs/procs/wshop7/

Sharma, M. D., Khachan, J., Chan, B., Stewart, C., Hogg, K. & O'Byrne, J. (2002b). A classroom communication system for large first year physics classes. Proceedings of the 15th Australian Institute of Physics Congress, Sydney, July 2002, 247-249.

Sharma, M. D., Millar, R., Smith, A. & Sefton, I. (2004). Students' understandings of gravity in an orbiting space-ship. Research in Science Education, 34, 267-289.

Sokoloff, D. R. and Thornton, R. K. (1997). Using interactive lecture demonstrations to create an active learning environment. The Physics Teacher, 35, 340-347.

Thornton, R. K. and Sokoloff, D. R. (1998). Assessing student learning of Newton's laws: The Force and Motion Conceptual Evaluation and the evaluation of active learning laboratory and lecture curricula. American Journal of Physics, 66, 338-352.

Tobias, S. (1994). They're Not Dumb, They're Different: Stalking the Second Tier. Tucson, USA: Research Corporation and Foundation for the Advancement of Science.

Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students' misconceptions in science. International Journal of Science Education, 10, 159-169.

Tyson, L. M. & Bucat, R. B. (1995). Chemical equilibrium: Using a two-tier test to examine students' understanding. Western Australia Science Education Association Annual Conference.

Van Dijk, L. A., Van Den Berg, G. C. & Van Keulen, H. (2001). Interactive lectures in engineering education. European Journal of Engineering Education, 26, 15-28.

Williams, J. B. (2003). Learning by remote control: Exploring the use of an audience response system as a vehicle for content delivery. In Interact: Integrate: Impact: Proceedings of the 20th ASCILITE Conference, Adelaide. http://www.adelaide.edu.au/ascilite2003/docs/pdf/739.pdf

Authors: Manjula D. Sharma, Joe Khachan, Ben Chan and John O'Byrne Sydney University Physics Education Research Group School of Physics, University of Sydney, NSW 2006, Australia Phone: 02 9351 2051 Fax: 02 9351 7726 Corresponding author email: m.sharma@physics.usyd.edu.au Please cite as: Sharma, M. D., Khachan, J., Chan, B. and O'Byrne, J. (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. http://www.ascilite.org.au/ajet/ajet21/sharma.html