Figure 1: Off line contact-Online response event sequence
| Australian Journal of Educational Technology 2002, 18(3), 275-292. |
AJET 18 |
The paper describes one specific approach to implementing web based survey research and reviews a range of online survey techniques. The advantages and disadvantages of various web based survey approaches are discussed. The paper describes one research project that used an 'off line contact-online response' in an educational setting. The researchers conclude that it is not possible to describe one process that would work well in all situations, given the variations in the way technology is currently utilised in schools and homes. The paper discusses the problems encountered and makes suggestions for future web based survey research.
Survey research has traditionally played an important role in many areas of the social sciences and business, such as policy development, marketing and consumer research, health issues, educational practice, media polling, and political polling (Weisberg, Krosnick & Bowen, 1996). More importantly, as technology evolves in the context of the digital economy, the implementation of web based surveys will become increasingly prevalent. The appeal of using the Internet to collect large amounts of data is obvious. Computer automation can reduce the cost associated with data entry while simultaneously reducing error to zero (Nicholls, Baker & Martin, 1997). On the other hand, the cost of developing web based surveys can be extremely high when questions are complex (eg., when they entail branching logic), and when graphics, video, and audio are included. In addition, if respondents are required to log back into the survey numerous times, the cost of instrument development will also increase (Witmer, Colman & Katzman, 1999; Valentine & Lucas, 2001).
As Dillman (2000) astutely points out, the number of households in the US with members who can access email is in excess of 35% or about the same level of households with telephones in 1936. During that year, Literary Digest conducted an infamous telephone survey of over a million people and erroneously predicted that Roosevelt would lose the presidential election. According to Dillman, the impact of this inaccurate prediction was partly responsible for setting back telephone survey research until the 1970s. It is therefore important that care be taken in researching the best approaches to web survey design in order to avoid this type of setback.
The purpose of this paper is to provide an overview of the design, development, and implementation of Internet based educational survey research. The paper draws on the authors' experience of conducting a large scale web based survey of elementary school teachers.
The authors of the article are involved in a larger research project on the use of Canadian children's literature in the elementary school setting. The Canadian Children's Literature Survey project is designed to collect information from schools throughout the Province of Alberta regarding the use of Canadian children's literature in Alberta provincial elementary schools.
It was originally intended that the method of contact and data collection be conducted by postal mail. However, during the early (1997) conceptual phase of this project the Alberta Provincial government announced its intention to have all schools throughout the province linked to the Internet before the year 2000. We viewed the government commitment as an opportunity to design, develop and deliver our survey over the Internet because our target population would have equal access to the Internet. The following is an overview of two general approaches to web based survey research, and a description of the approach we employed and the issues we encountered.
In one early study of email users, Parker (1992) administered an electronic questionnaire to a sample of AT&T employees. His results indicate a 68% return rate in contrast to 38% rate for the same survey by postal mail. Parker claimed the higher response rate was the result of a novelty effect. In an email survey at the Chinese University in Hong Kong, Tse et al. (1995) reported the opposite results - 27% return rate by postal mail as compared to 6% return rate for email. Tse et al. attributed the poor result to a belief that email responses could more easily be traced and fear of the new email technology. In another study Bachmann and Elfrink (1996) indicated very little difference in the return rate (both at 56%) between the email and postal mail when they surveyed employees at the Lotus Corporation. Couper, Blair & Triplett (1997), in a survey of employees at government statistical agencies, reported a 71% response rate by postal mail as compared to 43% response rate by email. People also appear to respond differently to email than regular post in that email can be immediately processed (deleted or answered) - whereas paper requests take more handling time (Kittleson, 1997). Clearly, the reported response rates from these researchers indicate a great deal of variability. One can hardly conclude that an email contact-email response approach will produce higher response rates than a survey conducted by postal mail.
From a technical perspective, email questionnaires raise a number of implementation problems. The first problem that may arise is multiple response copies of the same email questionnaire. It is easy for the respondent to press the send key more than once. Attempting to keep the responses anonymous makes it almost impossible to control for multiple submissions. In contrast, numerous strategies can be employed with postal mail questionnaires to ensure anonymity (Weisberg, Krosnick, & Bowen, 1996).
A second problem with questionnaires returned via email is that they must be converted into individual records with data fields. Typical email software filtering techniques can usually classify email based on the text contained in the subject section of the email. A more difficult problem arises when the body of the message must be parsed to create a respondent's record and data fields (one email response should translate into one respondent record). There are various technical problems associated with this parsing approach and a number of methods have been developed to handle such problems (White, Carey & Dailey, 2000). In almost all cases, automating this process requires solutions that incorporate various levels of programming expertise. The nature of the programming problems often depends on the hardware and software configuration of the respondent (client side) and surveyor (server side).
In the simplest case the respondent simply answers the questionnaire items and emails the response back to the surveyor. At this point the researcher must employ a "simple but labor intensive approach" (White, Carey & Dailey, 2000; p. 3) to organising the data into a respondent's record. For example, each respondent's email message must be moved to an editor so it can be modified into a suitable record format and then appended to a cumulative data file. This cumulative data file will continue to grow, one record at a time, until responses are no longer received.
Given that the researcher can overcome ethical and technical problems, there do appear to be some advantages to using the email contact-email response approach. The two most obvious ones are speed of access and reduced cost. Existing research indicates much faster return rates for email surveys (Schaefer & Dillman, 1998, Yun & Trumbo, 2000). The email contact-email response approach is usually less costly than postal mail and this cost saving will grow larger as the sample size increases (Watt, 1999). Another advantage occurs in cases where the email is returned to the sender as "non-delivered." The researcher knows almost immediately the portion of the sample that did not receive the questionnaire and can then employ alternative methods of contact.
A major concern with using web form questionnaires is the type of hardware platform and web browser being used by the sample subject. The two most popular browsers are Netscape Navigator (Netscape Corporation) and Internet Explorer (Microsoft Corporation). In an educational setting, such as a school district, it is often difficult to predict which version of Netscape or Explorer a potential respondent will be using. Browser software packages are revised regularly with newer versions coming out yearly and, depending on software bugs, modifications to these versions can appear at almost anytime. Compounding this problem is the 'between group variability' in the hardware platforms used by sample subjects (Macintosh or Intel PC) and the 'within group variability' of these hardware platforms (various versions of the hardware, eg., Pentium III vs Celeron, G-series Macintosh vs iMac). Even differences in monitors can affect screen presentations of the web form. If one considers the possible combinations: a) browsers and hardware platforms (between group variability), and b) browsers and hardware versions (within group variability), the number of confounding variables that can interfere with web based data collection is high. This does not include the use of client side software technologies such as JavaScript, VBScript, and Cookies, which can further act as intervening variables. Thus the survey researcher must be extremely careful to ensure some type of standardisation.
Technical problems aside, there are clearly a number of advantages to using web form questionnaires in survey research:
Figure 1: Off line contact-Online response event sequence
In a recent study Yun and Trumbo (2000) compared the responses to a survey executed by post mail, email, and web form. Their target audience was the 900 National Association of Science Writers (NASW) who listed both their email and postal addresses in a 1998 membership directory. They randomly selected 360 of these subjects and contacted them all via both postal mail (survey letter with paper survey) and email. All sample subjects had the ability to respond either by postal mail, email (in the body of the email), or by using a web form (a URL was provided in the email and postal contact letters). The response rate was 72%, but only about one third of these responses were completed electronically. This result indicates that, given the choice, the respondents in this study still preferred the paper form of the questionnaire. Yun and Trumbo do, however, indicate that respondents using the email were much faster to respond - many responding (either by email or web form) within the same day of receiving the original email.
In summary, there are a number of advantages and disadvantages to employing the Internet based survey strategies described thus far. A researcher's approach to conducting an Internet survey depends entirely on the technical resources available at the research site and the technical capabilities of the target subjects. Educational survey researchers must pay particular attention to such issues given the complex task of dealing with the various technical systems employed by school jurisdictions.
A number of factors guided our approach to the web based survey. First, access to the survey instrument should be as simple as possible for all survey participants. Second, the respondents should be able to fill out the survey instrument with the same relative ease as if they had received the survey on paper. Third, a relatively simple security system (that we call "respondent authentication") should be in place to ensure the integrity of the data. Fourth, all data obtained through the survey must be safe from any external tampering. Finally, the data should be stored in a format that can easily be read by the data analysis software package, in our case SPSS. What follows is a brief discussion of these factors.
In order to ensure that the survey was easy to complete, a pilot study was undertaken in a computer laboratory with 15 participants. During the pilot testing phase, the average time to complete the online survey was twenty minutes. We designed the online forms to ensure respondents did not have to download files (eg., cookies) and thus deal with issues concerning file incompatibility. No user expressed frustration at having to complete all items before submitting the form. On reflection, however, it would have been extremely useful to conduct a "think aloud" process with the pilot respondents (Luchs, Zulli, Gullickson, & Barbour, 2000). A think aloud procedure would have enabled us to follow the respondents' thinking as they navigated the survey. The "think aloud" could have been audiotape recorded and the respondents could have spoken face to face with a research assistant at the time of completing the survey. A protocol could have been developed to identify any problems the respondents encountered, and track and monitor the processes they engaged in as they completed the survey. We could also have provided information to the respondents, on their request, that later could have been incorporated into the general instructions for completing the survey.
School boards (through the school superintendent) and school principals provided initial consent to the project. The principals invited one teacher in each school to participate. This guaranteed anonymity for the teachers. The teachers were not to be named by the school principal. On filling in the survey, individual teachers first encountered a web page that explained the project, notified them of their rights and informed them that by responding to the questionnaire they were consenting to participate. They were not to provide their name or any other identification.
Further compounding the problem was that home users could choose from a variety of Internet Service Providers (ISPs), whereas the school districts were all standardised on one ISP. This proved to be problematic because different ISPs use different software, some of which caused problems when the data were being transferred to our server. This was also evident when either schools or private ISPs used firewall technology to filter communications. For example, firewall technology often got in the way of our respondent authentication strategy by preventing potential respondents from accessing our secure web site.
Our expectations that the schools would be able to converse easily via email, should they encounter problems, were undermined by errors in the way computer systems were configured and supported at individual schools and throughout school districts. We discovered that when we sent emails to teachers and principals at their schools, they frequently did not receive them. If we faxed the schools and asked teachers or principals to email the researchers then there was a successful communication. Thus, email use may be a matter of time constraint or of ease of access (depending on the number of computers in the school).
Some potential participants were lost because the survey was conducted online. For example, two teachers accessed the questionnaire online, printed it, filled it in by hand and submitted it via postal mail. Three other schools asked if they could receive the survey in printed form. At the same time we probably gained some respondents because of the "novelty effect" of completing a survey instrument on the web, or from what Yun and Trumbo (2000) refer to as the "techie effect" - in this case those teachers drawn to any topic that uses some aspect of technology or the web.
The support system put in place by the researchers to help teachers in accessing the survey could have been stronger. We relied on faxes, email and phone calls from teachers, and it took a few weeks before we realised that teachers needed more specific information than we had provided to them in the initial letter. For example, there were differences in the screens appearing on Macintosh and PC computers that had to be explained to the teachers in the letter. Such specific items had not been anticipated at the outset even thought we had tested our software on both PC and Macintosh computers.
Changing technology provides exciting new opportunities for conducting surveys, but the results will clearly be biased toward the views of a technological elite. Indeed, it is not always possible to tell who is answering questions on such systems - whether adults are giving serious answers or whether children are just punching buttons. (p. 42).Clearly, one must be careful not to see Internet and web based technologies as a panacea for carrying out survey based research.
The project described in this paper had a number of goals, one of which was to design and develop a process for web based survey research. We now realise that it may not be possible to describe one process that would work well in all situations given the large variations in the way technology is currently utilised in schools and at home. In other words, the approach we describe in this paper is one of many that could be used to carry out a web based survey.
Essentially, a self administered web based questionnaire appears to have a number of advantages. Two important ones are that it provides fast access to the survey instrument, and the data obtained can be automatically analysed or saved in file format which can be directly uploaded into a program such as SPSS. In all cases there is a financial saving because reliance on the traditional postal service is greatly reduced and data input and coding errors are a non-issue (assuming the program works correctly).
Some disadvantages include: the need to have access to the Internet, subjects must be willing to use a computer to complete the survey, and both the respondents' (client) system or network and surveyors' (host) system or network can encounter problems. For example, we are unsure of how many schools simply turned down our request for participation because they feared using technology or how many potential respondents in the schools agreeing to participate failed to complete the survey due to technology phobia. As Fink and Kosecoff (1998) note, "for some time to come, certain respondents (such as some who prefer to take cyberspace slowly) will continue to mistrust computers and computer based surveys." (p. 7).
Furthermore there is still a great deal of variability in hardware and software used in both school and home environments. This means it often becomes difficult to assume that everyone who has a computer hooked to the Internet actually has equal access. For example, we know of some individuals who tried to access our web survey and continuously encountered problems. We are not completely sure how to solve such problems (especially those of a technical nature) given that some individuals are great distances away and without local technical support. We are also unsure of how many "potential respondents" tried to complete our survey but because of technical problems just gave up!
Overall, as well as successfully collecting useful content data, important lessons were learned in conducting web based survey research. Items we shall keep in mind for future projects include the use of a "think aloud" procedure with the pilot group. This would have provided us with invaluable information regarding the thought processes and the difficulties, if any, that pilot respondents encountered in completing the survey. We also need to conduct a more thorough testing of the compatibility of various browsers and hardware platforms. In addition, we need to attend to the recent advent of the firewall technology used in schools, as this technology can interfere with both the transmission and collection of data. We need a better understanding of the range of technology (hardware/ software) used within school jurisdictions and in the teachers' homes. Finally, we need to explore the possibility of including graphics, audio and video in a web based questionnaire.
Buchanan, T. & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. The British Journal of Psychology, 90(Feb), 125-144.
Carbonaro, M. & Bainbridge, J. (2000). Design and development of a process for web-based survey research. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
Couper, M. P., Blair, J. & Triplett, T. (1997). A comparison of mail and e-mail for a survey of employees in federal statistical agencies. Paper presented at the American Association for Public Opinion Research, Norfolk, VA.
Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method. New York: John Wiley & Sons.
Fetterman, D. M. (1998). Webs of meaning: Computer and Internet resources for educational research and instruction. Educational Researcher, 27(3), 22-30.
Fink, A., & Kosecoff, J. (1998). How to conduct surveys: A step-by-step guide (2nd ed). Thousand Oaks, CA: Sage.
Fraenkel, J. R., & Wallen, N. E. (1997). How to design and evaluate research in education (3rd ed). Toronto, ON: McGraw-Hill.
Kittleson, M. J. (1997). Determining effective follow-up of e-mail surveys. American Journal of Health Behavior, 21(3), 193-196.
Luchs, S., Zulli, R., Gullickson, A. & Barbour, S. (2000). The challenge of validity: A case study exploring pilot testing strategies for web-based survey development. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.
Newsweek (1995, Feb. 15), p. 32.
Nicholls, W. L., Baker, R. P. & Martin, J. (1997). The effect of new data collection technologies on survey data quality. In Lyberg, Biemer, Collins, deLeeuw, Dippo, Schwartz and Trewin (Eds), Survey measurement and process quality. New York: John Wiley & Sons.
Owsten, R. D. (1997). The World Wide Web: A technology to enhance teaching and learning? Educational Researcher, 26(2), 27-33.
Parker, L. (1992). Collecting data the e-mail way. Training & Development, 46(7), 52-54.
Roschelle, J. & Pea, R. (1999). Trajectories from today's WWW to a powerful educational infrastructure. Educational Researcher, 28(5), 22-25.
Schaefer, D. R. & Dillman, D. A. (1998). Development of standard e-mail methodology: Results of an experiment. Public Opinion Quarterly, 62(3), 378-397.
Schmidt, W. C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments and Computers, 29, 274-279.
Smith, C. B. (1997). Casting the Net: Surveying an Internet Population. Journal of Computer-Mediated Communication, 3(1). [verified 6 oct 2002] http://www.ascusc.org/jcmc/vol3/issue1/smith.html
Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, and Computers, 29, 496-505.
Tse, A. (1998). Comparing the response rate, response speed and response quality of two methods of sending questionnaires: E-mail vs. mail. Journal of the Market Research Society, 40(4), 353-361.
Tse, A. C. B., Tse, K. C., Yin, C. H., Ting, C. B., Yi, K. W., Yee, K. P. & Hong, W. C. (1995). Comparing two methods of sending out questionnaires: E-mail versus mail. Journal of the Market Research Society, 37(4), 441-446.
Valentine, J. W. & Lucas, S. E. (2001). The design and implementation of a web-based national study of middle level principals and schools. Paper presented at the Annual Meeting of the American Educational Research Association, Seattle, WA.
Watt, J.H. (1999). Internet systems for evaluation research. In G. Gay and T.L. Bennington (Eds), Information technologies in evaluation: Social, moral, epistemological, and practical implications (pp. 23-44). San Francisco, Jossey-Bass.
Weisberg, H. F., Krosnick, J. A. & Bowen, B. D. (1996). An introduction to survey research, polling, and data analysis (3rd ed). Thousand Oaks, CA: Sage.
White, J., Carey, L.M. & Dailey, K. (2000). Improving web-based survey research data collection. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA. [verified 6 Oct 2002] http://www.coedu.usf.edu/jwhite/survey1/
Witmer, D. F., Colman, R. W. & Katzman, S. L. (1999). From paper-and-pencil to screen-and-keyboard: Toward a methodology for survey research on the Internet. In S. Jones (Ed), Doing Internet research: Critical issues and methods for examining the net (pp. 145-161). Thousand Oaks, CA: Sage Publications, Inc.
Yun, G. W., & Trumbo, C. W. (2000). Comparative response to a survey executed by post, e-mail and web form. Journal of Computer-Mediated Communication, 6(1). [verified 6 Oct 2002] http://www.ascusc.org/jcmc/vol6/issue1/yun.html
| Authors: Mike Carbonaro is an assistant professor in the Department of Educational Psychology at the University of Alberta, Edmonton, Canada. His interests focus on the application of computer technology in the area of teaching and learning as well as the relationship between computational cognitive modeling and human learning. Dr Mike Carbonaro, Dept of Educational Psychology, 6th floor, Education North, University of Alberta, Edmonton AB, Canada T6G 2G5. Phone: 1 780 492 2646 Fax: 1 780 492 1318. Email: mike.carbonaro@ualberta.ca
Joyce Bainbridge is a professor in the Department of Elementary Education, University of Alberta. She teaches courses in children's literature and the language arts. Her current research explores the role of Canadian children's literature in Canadian elementary classrooms.
Brenda Wolodko is an associate professor in Early Childhood Education at the University of Toledo, Ohio, USA. She is a doctoral candidate in the Department of Elementary Education at the University of Alberta, pursuing an inquiry in mathematics education. Please cite as: Carbonaro, M., Bainbridge, J. and Wolodko, B. (2002). Using Internet surveys to gather research data from teachers: Trials and tribulations. Australian Journal of Educational Technology, 18(3), 275-292. http://www.ascilite.org.au/ajet/ajet18/carbonaro.html |