|Australasian Journal of Educational Technology
2011, 27(1), 55-65.
Publishing and perishing: The critical importance of educational design research
Thomas C. Reeves
The University of Georgia
University of Twente
The outcomes of educational systems continue to lag far behind expectations at all levels, primary, secondary, and tertiary. Meanwhile, the sheer amount of educational research published in refereed journals has expanded enormously. There is an obvious disconnect between the educational research papers published in professional journals or presented at academic conferences and any form of beneficial impact on the students, teachers, and other stakeholders in educational systems. This problem can be traced back to those professors and research supervisors engaged in the preparation of educational researchers who fail to convey to novice researchers important distinctions between the goals and methods of educational research. Educational design research provides a potentially viable alternative to the kind of educational research that is commonly conducted in the field of educational technology. Educational design research has the twin objectives of developing creative approaches to solving human teaching, learning, and performance problems while at the same time constructing a body of design principles that can guide future development efforts. This paper argues that a reconsideration of educational research approaches is crucial and that the time for greater uptake of educational design research is now.
The problem of declining achievement is also evident in tertiary education (Bok, 2006). Although the elite universities and colleges in the USA are still highly ranked, the institutions of higher education that the vast majority of American students attend have arguably become high-cost providers of mediocre outcomes (Hacker & Dreifus, 2010; Taylor, 2010; Whitehurst, 2010). Similar criticisms have been levelled at higher education in Australia (Coady, 2000), the United Kingdom (Hussey & Smith, 2009), and other countries.
While highly developed nations such as Australia, the United Kingdom, and the USA seek to improve their educational outcomes at all levels, education in developing countries still lags far behind with respect to both opportunities and achievements at even primary levels. In 2001, the United Nations stated eight Millennium Development Goals (MDGs) that are to be achieved by the year 2015 (UN Millennium Project, 2005). The second of these goals is: "Achieve universal primary education." Sadly, there is little evidence of significant gains in reaching this goal in most under-developed nations (Tarabini, 2010).
Many paths are being taken in the seemingly never-ending search for solutions to the problems inherent in weak educational outcomes in both developed and under-developed nations. Testing, national standards, eliminating tenure, increased accountability, charter schools, educational technology, vouchers, and better teacher training are just a few of the numerous "reforms" touted as remedies for educational problems around the world.
Oddly, at the same time when educational attainments have been declining or remained stagnant in most nations, the number of educational research publications has expanded enormously. In 2006, Ulrich's Periodicals Directory, the definitive reference for bibliographic information about scholarly publications, listed 1226 active, refereed scholarly journals related to education (Togia & Tsigilis, 2006), and there are no doubt more today with the explosion in web-based journals. Each year, these increasing numbers of educational research journals are filled with "refereed" articles, although the referring process itself has become rather dubious. No less an authority than Richard Horton, editor-in-chief of The Lancet, the leading medical journal in the United Kingdom, at the time wrote:
The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability - not the validity - of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong. (Horton, 2000, p. 148)If the journal refereeing process in medical research is flawed, than it is likely to be much more so in education where the stakes are much less clear-cut and the chances of replication by other researchers are much lower. Many of the articles in educational journals report on the predictive testing of a specific, often esoteric, tenet of an educational theory or some type of innovation using quasi-experimental methods. Many other refereed papers report on the use of one or more theories or principles to inform the "theoretical foundation" for an interpretivist study of some phenomenon related to education using qualitative methods.
Regardless of the type of methodology used, there appears to be fundamental disconnect between the conduct and reporting of educational research and serious improvement in educational outcomes. How can the quantity of educational research journals and articles have increased so much over the last 50 years while the world's educational achievements remained unacceptably low or actually regressed in many nations? It may be possible that good research is being done, but that it is simply not being presented in formats or venues acceptable to practising teachers. However, this seems highly unlikely. Hattie's (2009) synthesis of 800 meta-analyses of the relationship between 138 educational treatments and educational achievement indicates that most classroom innovations have been trialled at all levels of education with weak results.
Although the realities of climate change and the limited chances we appear to have of radically reducing carbon omissions are arguably more serious than the crisis in the quality, value, and impact of educational research, the hypocrisy is no less. How can we as educational researchers go on padding our vitae with refereed journal publications and book chapters that have no demonstrable impact on the state of education today?
By contrast, consider the field of oncology that deals with the study and treatment of cancer. Until the early 1970s, there were no oncologists as distinct medical practitioners and people diagnosed with cancer usually were sent to surgeons who tried to cut the cancer out of them. But over the last 40 years, cancer researchers and oncologists as well as other cancer specialists have collaborated in ambitious research and development efforts that have yielded a dramatic improvement in cancer survivorship around the world (Pardee & Stein, 2009).
There is no parallel impact with respect to educational research (Ball & Forzani, 2007). It certainly cannot be found in enhanced educational outcomes (Hattie, 2009). Nor are there greater opportunities for access to high quality learning environments because of educational research. Equal opportunities for education with respect to gender and socio-economic status have been improved in some countries, but these enhancements cannot be easily traced back to educational researchers. Sadly, it is extremely difficult to trace the impact of educational research to anything that really matters.
The hypocrisy of the educational research charade is augmented by the fact that more and more countries and universities are trying to measure the "impact" of their journal publications using some form of citation analysis (Atkinson, 2010; Goodyear, Brewer, Symms-Gallagher, Tracey, Claiborn, Lichtenberg, et al., 2009; van Aalst, 2010). But as Togia and Tsigilis (2006) clearly demonstrate, the utility of these citation analyses rests simply in the ease with which they can be calculated and counted in academic decision processes concerning promotion, tenure, and grant awards. Their value and meaningfulness in the real world, on the other hand, are seriously undermined by many technical and human factors, including:
How is this accomplished? First, educational design research requires practitioners and researchers to collaborate in the identification of significant teaching and learning problems. There are many problems that can be addressed. For example, what can be done to encourage more girls and members of minority groups to take the study of mathematics and science more seriously? How can boys be motivated to read as a fundamental method of self-improvement and enjoyment? What can be done to ensure that university graduates have the knowledge, skills, attitudes, and intentions to become productive members of a society more concerned with issues like climate change, the reduction of poverty, and public health than simply with accumulating material wealth? What can be done to ensure that children in the world's poorest countries have equitable access to learning opportunities at the primary and secondary levels, and eventually in the post-secondary sector?
Second, educational design research engages practitioners and researchers in the creative activity of developing prototype solutions to these and other serious problems, based on existing design principles. Theory almost seems to be an afterthought in many other approaches to educational research, but in educational design research, it plays a primary role in the shaping of prototype innovations that address serious problems.
Third, educational design research involves the close collaboration of practitioners and researchers in the testing and refinement of both the prototype solutions and the design principles upon which they are based, until satisfactory outcomes have been reached by all concerned. Educational design research is not "done" until desirable results are attained, results that represent progress in solving the problems with which the research projects began.
Of course, educational design research is not an activity that individual researchers conduct in isolation from practice. It requires close collaboration among all the major stakeholders in the problem area. By its very nature, educational design research ensures that progress will be made with respect to, at the very least, clarification of the problems facing educators and the public at large, and ideally, the creation and adoption of effective innovations in tandem with the clarification of robust design models and principles.
We use the term educational design research for three primary reasons. First, the simpler term of design research is also used by researchers in the field of human-computer interface design and industrial engineering (Laurel, 2004). The modifier educational is our attempt to distinguish this research approach from those used in other fields. Second, although the term design-based research has many adherents, it seems to us to over-emphasise the design aspects of the approach. Third, the two primary books published to date concerning educational design research (Kelly, Lesh & Baek, 2008; van den Akker et al., 2006) have not adopted the term design-based research to any great extent.
Researchers with theoretical goals are focused on explaining phenomena through the logical analysis and synthesis of theories, principles, and the results of other forms of research such as empirical studies. This type of research is relatively rare because it requires levels of synthesis, generalisation, and theory construction beyond the abilities of most researchers. In addition, this type of research follows a long-term agenda that is sustained for many years. A classic example of research with theoretical goals within the field of educational technology is the seminal work of Gagné (1997) to describe the basic conditions of learning and a theory of instruction.
Researchers with predictive goals are focused on determining how education works by testing conclusions related to theories of teaching, learning, performance, assessment, social interaction, instructional design, and so forth. Educational technology researchers with this type of goal usually employ experimental (or quasi-experimental) methods to determine the effects of some form or aspect of a technological innovation under controlled conditions. This type of research has dominated educational technology for decades, but reviews reveal that it is often done poorly (Clark, 1998). Its popularity over many decades stemmed from the fact that until interpretivist research became legitimatised within the academy, it was practically the only goal graduate students and young researchers were encouraged to pursue. In addition, empirical studies using quasi-experimental methods take less time and logistical support than other approaches, and many research journals remain more receptive to reports of empirical studies than other forms of research. Although such studies are often flawed, there are examples of competent research such as the investigations of the effects of multimedia variables on learning conducted by conducted over several decades by Mayer (2001).
Researchers with interpretivist goals are focused on portraying how education works by describing and interpreting phenomena related to teaching, learning, performance, assessment, social interaction, innovation, and so forth. Educational technologists with interpretivist goals draw upon naturalistic research traditions borrowed from other sciences such as anthropology and sociology. The popularity of conducting research from an interpretivist perspective has increased dramatically among educational researchers over the past 20 years, even among educational technologists. A pioneering example of interpretivist research within the field of educational technology is Neuman's (1991) naturalistic observations of learning disabled children using commercial courseware.
Researchers with postmodern goals are focused on examining the assumptions underlying contemporary educational programs and practices with the ultimate goal of revealing hidden agendas and empowering disenfranchised minorities. Although increasingly evident among academic researchers with multicultural, gender, or political interests, research in the postmodern tradition is rare within the field of educational technology. There are several reasons for this, not the least of which is the fact that there are relatively few educational technologists capable of mentoring graduate students or young researchers in this approach. Another is the difficulty postmodern researchers have in finding scholarly outlets for their papers. DeVaney's (1998) analysis of the field of educational technology in relation to race, gender, and power is an important example of research with this goal.
Researchers with design/development goals are focused on the dual objectives of the developing creative approaches to solving human teaching, learning, and performance problems while at the same time constructing a body of design principles that can guide future development efforts. Educational design research has recently received endorsements from notable experts (van den Akker et al., 2006; Kelly et al., 2008), but it has not been widely adopted. One well-known example of educational design research is the work of Sasha Barab and his colleagues at Indiana University who are constructing sophisticated virtual learning environments (e.g., Quest Atlantis) while at the same time developing a Learning Engagement Theory (Barab, Thomas, Dodge, Carteaux & Tuzun, 2005).
Researchers with action/evaluation goals are focused on a particular program, product, or method, usually in an applied setting, for the purpose of describing it, improving it, or estimating its effectiveness and worth. Sometimes called action research or evaluation research, research with action goals is similar to design research except that there is little or no effort to construct theory, models, or principles to guide future design initiatives. The major goal is solving a particular problem in a specific place within a relatively short timeframe. Some theorists maintain that this type of inquiry is not research at all, but merely a form of evaluation. However, despite its primary focus on considerations of use for local practitioners, it can be regarded as a legitimate form of research, provided reports of it are shared with wider audiences who may themselves choose to draw inferences from these reports, in a manner similar to interpretivist papers. One example of this research is an evaluation of a project-based undergraduate engineering course conducted by Reeves and Laffey (1999).
Only after clearly understanding one's research goal or intent, should anyone select a specific research method (Krathwohl, 2009). There are many methods that can be chosen, for example, literature review, historical analysis, experimental studies, interviews, observations, artefact analysis, and deconstruction just to name a few. Educational design research almost always involves mixed methods designs (Creswell & Plano-Clark, 2006).
Of course, it is obvious to anyone who has had cancer or has lost a loved one to the disease that cancer research still has a long way to go. Gladwell (2010) describes how there are two primary schools of researchers seeking cures for cancer. "The first school, that of 'rational design,' believes in starting with the disease and working backward-designing a customized solution based on the characteristics of the problem" (Gladwell, 2010, p. 71). This approach is akin to educational design research.
According to Gladwell (2010), "The other approach is to start with a drug candidate and then hunt for diseases that it might attack. This strategy, known as "mass screening," doesn't involve a theory. Instead, it involves a random search for matches between treatments and diseases" (p. 72). This seems much more in keeping with much of existing educational research, especially in the field of educational technology in which the authors of this paper situate much of their work. As Cuban (2001, 1986) has described, virtually every new media or technology to come along for over 100 years has been followed by a shotgun-blast like surge in studies aimed at finding the educational efficacy of the target of the research. This has not worked in the past and it won't work in the future.
Educational design research is a viable approach to solving real problems in practice and to advancing theoretical understanding as well. Many challenges remain. As a relatively new genre of inquiry, researchers are still learning about how to share the outcomes of studies beyond the immediate contexts in which they are conducted. Important questions about publishing educational design research must be addressed. What pieces of the story of an educational design research project are useful/interesting to a broader audience? Long, complex stories are difficult, perhaps impossible, to tell in a traditional journal format. Books can be quite rich, but risk being useful to a limited audience (or not recognised by the author's institution). It must be admitted that engaging in educational design research can be risky in that the amount of work that goes into any given project can be far more demanding than that required in many other forms of research, and the publications stemming from those efforts may be fewer in quantity, although we would argue educational design research publications have the potential to be much more valuable with respect to quality and impact.
Educational design researchers in academe must seek to advance their careers just as other academic staff members do. Therefore, educational researchers who share our concern for increasing our impact should seek significant changes in how promotion and tenure decisions are made by seeking to establish better alignment between career-path/recognition and relevant scientific and social outputs. For example, research-based products themselves should 'count' as meaningful outputs and this credit should be shared with the designers and teachers involved in creating and refining these products.
Educational design researchers try to focus on the bigger picture of lessons learned and not just the immediately developed results. Similarly, promotion and tenure committees should focus on the overall impact of a line of research rather than simply the number of publications. Publication should not be a goal in itself. Even if this was acceptable, it is severely flawed in education just as it is in medical research (Ioannidas, 2005). In his article about the challenges of cancer research, Gladwell (2010) described the work of Dr Safi Bahcall, who holds a PhD in theoretical physics, but has dedicated his career to the search for effective cancer drugs. According to Gladwell, Bahcall maintains that "In physics, failure was disappointing. In drug development, failure was heartbreaking" (p. 69) because he and his colleagues are seeking to save the lives of the patients involved in their clinical trials. It is time that the continued failure of educational researchers to have meaningful impact should also be regarded as "heartbreaking." To end this deplorable situation, we must adopt new approaches. In our view, educational design research is the approach with the greatest promise.
Ball, D. L. & Forzani, F. M. (2007). What makes education research "educational"? Educational Researcher, 36(9), 529-540.
Barab, S. A., Thomas, M., Dodge, T., Carteaux, R. & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research & Development, 53(1), 86-108.
Bok, D. (2006). Our underachieving colleges: A candid look at how much students learn and why they should be learning more. Princeton, NJ: Princeton University Press.
Bransford, J. D., Stipek, D. J., Vye, N. J., Gomez, L. M. & Lam, D. (Eds.) (2009). The role of research in educational improvement. Cambridge, MA: Harvard Education Press.
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2, 141-178.
Clark, R. E. (1998). Bloodletting, media and learning. In T. L. Russell (Ed.). The no significant difference phenomenon (pp. i-iv). Raleigh, NC: North Carolina State University Press.
Coady, T. (Ed.) (2000). Why universities matter: A conversation about values, means and directions. Sydney: Allen & Unwin.
Collins, A. (1992). Towards a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). Berlin, Germany: Springer.
Creswell, J. W. & Plano-Clark, V. L. (2006). Designing and conducting mixed methods of research. Thousand Oaks, CA: Sage Publications.
Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press.
Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. New York: Teachers College Press.
Darling-Hammond, L. (2010). The flat world and education: How America's commitment to equity will determine our future. New York: Teachers College Press.
De Vaney, A. (1998). Can and need educational technology become a postmodern enterprise? Theory into Practice, 37(1), 72-80.
Gagné, R. M. (1997). The conditions of learning and theory of instruction. New York: Holt, Rinehart and Winston.
Gladwell, M. (2010). The treatment: Why is it so difficult to develop drugs for cancer? The New Yorker, 86(17 May), 68-77.
Goodyear, R. K., Brewer, D. J., Symms-Gallagher, K., Tracey, T. J. G., Claiborn, C. D., Lichtenberg, J. W. & Bruce E. Wampold, B. E. (2009). The intellectual foundations of education: Core journals and their impact on scholarship and practice. Educational Researcher, 38, 700-706.
Hacker, A. & Dreifus, C. (2010). Higher education: How colleges are wasting our money and failing our kids - and what we can do about it. New York: Times Books.
Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses related to achievement. New York: Routledge.
Herrington, J., McKenney, S., Reeves, T. C. & Oliver, R. (2007). Design-based research and doctoral students: Guidelines for preparing a dissertation proposal. In C. Montgomerie & J. Seale (Eds.), Proceedings of EdMedia 2007: World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 4089-4097). Chesapeake, VA: AACE.
Horton, R. (2000). Genetically modified food: Consternation, confusion, and crack-up. Medical Journal of Australia, 172, 148-149.
Hussey, T. & Smith, P. (2009). The trouble with higher education: A critical examination of our universities. London: Routledge.
Ioannidis J. P. A. (2005). Why most published research findings are false. Public Library of Science, Medicine, 2(8), e124. http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124
Kelly, A. E. (2003). Research as design. Educational Researcher, 32(1), 3-4.
Kelly, A., Lesh R. & Baek, J. (Eds.) (2008). Handbook of design research methods in education. New York: Routledge.
Krathwohl, D. R. (2009). Methods of educational and social science research: The logic of methods (3rd ed.). Long Grove, IL: Waveland Press.
Laurel, B. (2004). Design research: Methods and perspectives. Cambridge, MA: MIT Press.
Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
Neuman, D. (1991). Learning disabled students' interactions with commercial courseware: A naturalistic study. Educational Technology Research & Development, 39(1), 31-49.
Newman, D. (1990). Opportunities for research on the organizational impact of school computers. Educational Researcher, 19(3), 8-13.
Pardee, A. B. & Stein, G. S. (2009). The biology and treatment of cancer: Understanding cancer. Hoboken, NJ: John Wiley & Sons.
Ravitch, D. (2010). The death and life of the great American school system: How testing and choice are undermining education. New York: Basic Books.
Reeves, T. C. (2000). Socially responsible educational technology research. Educational Technology, 40(6), 19-28.
Reeves, T. C. & Laffey, J. M. (1999). Design, assessment, and evaluation of a problem-based learning environment in undergraduate engineering. Higher Education Research and Development, 18(2), 219-232.
Smil, V. (2010). Energy myths and realities: Bringing science to the energy policy debate. Washington, DC: American Enterprise Institute.
Tarabini, A. (2010). Education and poverty in the global development agenda: Emergence, evolution and consolidation. International Journal of Educational Development, 30(2), 204-212.
Taylor, M. (2010). Crisis on campus: A bold plan for reforming our colleges and universities. New York: Alfred A. Knopf.
Togia, A. & Tsigilis, N. (2006). Impact factor and education journals: A critical examination and analysis. International Journal of Educational Research, 45(6), 362-379.
UN Millennium Project (2005). Investing in development: A practical plan to achieve the Millennium Development Goals. New York: United National Development Program.
van Aalst, J. (2010). Using Google Scholar to estimate the impact of journal articles in education. Educational Researcher, 39, 387-400.
van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, N. Nieveen, R. M. Branch, K. L. Gustafson & T. Plomp (Eds.), Design methodology and developmental research in education and training (pp. 1-14). The Netherlands: Kluwer Academic Publishers.
van den Akker, J., Gravemeijer, K., McKenney, S. & Nieveen, N. (2006). Educational design research. London: Routledge.
Whitehurst, G. J. (2010). Spurring innovation through education: Four ideas. Brookings Institute Policy Brief No. 174. [verified 18 Jan 2010] http://www.brookings.edu/~/media/Files/rc/papers/2010/0603_innovation_whitehurst/pb_174.pdf
|This article received an Outstanding Paper Award at ascilite Sydney 2010, gaining the additional recognition of republication in AJET (with minor changes). The reference for the Conference version is:
Reeves, T. C., McKenney, S. & Herrington, J. (2010). Publishing and perishing: The critical importance of educational design research. In Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010. http://ascilite.org.au/conferences/sydney10/Ascilite%20conference%20proceedings%202010/Reeves-full.pdfAuthors: Thomas C. Reeves, The University of Georgia, USA
Susan McKenney, University of Twente, The Netherlands
Jan Herrington, Murdoch University, Australia
Please cite as: Reeves, T. C., McKenney, S. & Herrington, J. (2011). Publishing and perishing: The critical importance of educational design research. Australasian Journal of Educational Technology, 27(1), 55-65. http://www.ascilite.org.au/ajet/ajet27/reeves.html