Design-based research

Definition

According to Collins et al (2004: 15) “the term "design experiments" was introduced in 1992, in articles by Ann Brown (1992) and Allan Collins (1992). Design experiments were developed as a way to carry out formative research to test and refine educational designs based on principles derived from prior research.”

According to Sandoval (talk, 2007). DBR is about intervention: when it works, how it works and for who it works.

See also:

• design science, design language, conjecture map and design methodology
• educational modeling language
• Research and practice models in education

What is DBR ?

A short history

Design-based reasearch (DBR) in education is probably very old, but recent interest can be traced back to the early nineties, e.g. Brown (1992) and Collins (1992).

According to Reeves (2000:8), Ann Brown (1992) and Alan Collins (1992) defined critical characteristics of design experiments as:

• addressing complex problems in real contexts in collaboration with practitioners,
• integrating known and hypothetical design-principles with technological affordances to render plausible solutions to these complex problems, and
• conducting rigorous and reflective inquiry to test and refine innovative learning environments as well as to define new design-principles.

According to the Design-Based Research Collective (2003): “First, the central goals of designing learning environments and developing theories or “prototheories” of learning are intertwined. Second, development and research take place through continuous cycles of design, enactment, analysis, and redesign. Third, research on designs must lead to sharable theories that help communicate relevant implications to practitioners and other educational designers. Fourth, research must account for how designs function in authentic settings. It must not only document success or failure but also focus on interactions that refine our understanding of the learning issues involved. Fifth, the development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interest.”

More recently, special issues of Educational Researcher (e.g. Kelly 2003), the Journal of Learning Sciences (e.g. Barab 2004) and the Educational Psychologist (e.g. Sandoval & Bell 2004) reopened the debate. In addition some researchers joined in the Design Based Research Collective.

According to Collins et al (2004: 16), design research was developed to address several issues central to the study of learning, including the following:

• The need to address theoretical questions about the nature of learning in context.
• The need for approaches to the study of learning phenomena in the real world rather than the laboratory.
• The need to go beyond narrow measures of learning.
• The need to derive research findings from formative evaluation.

For Anderson and Shattuck (2012:24), “DBR seems have been used to make a difference—but mostly at the level of small-scale interventions and in the lives of individual teachers and schools.” The authors “concure with Dede, Ketelhut, Whitehouse, Breit, and McCloskey’s (2009) claim that “DBR offers a ‘best practice’ stance that has proved useful in complex learning environments, where formative evaluation plays a significant role, and this methodology incorporates both evaluation and empirical analyses and provides multiple entry points for various scholarly endeavors””

What is not DBR ? For example:

• Single case one-shot field studies of various sorts
• User centered technical implementation studies that do not document output of various cycles
• Studies that do not produce design rules as output.
• Studies that are vague about educational workflows, etc. I.e. the ones that do not use any kind of semi-formal modeling.

In other words: Studies that neither precisely describe a design, nor its mechanics, nor its "making", nor contextual variables that make it work.

DBR v.s. traditional empirical research

Reeves (2000:9, 2006) draws a clear line between research conducted with traditional empirical goals and that inspired by development goals leading to "Design-principles".

Predictive and design research approaches in educational technology research. In Reeves, T.C. (2006), Design research from the technology perspective

Action orientation

There is clearly an action-research oriented perspective, i.e. researchers must try to change things.

The overall goal of research within the empirical tradition is to develop long-lasting theories and unambiguous principles that can be handed off to practitioners for implementation. Development research, on the other hand, requires a pragmatic epistemology that regards learning theory as being collaboratively shaped by researchers and practitioners. The overall goal of development research is to solve real problems while at the same time constructing Design-principles that can inform future decisions. In Kuhn's terms, these are different worlds." (Reeves, 2000: 12).

For Juuti and Lavonen (2006) three aspects constitute design-based research: “(a) a design process is essentially iterative starting from the recognition of the change of the environment of praxis, (b) it generates a widely usable artefact, (c) and it provides educational knowledge for more intelligible praxis.”

Design-based research can be seen as combination of action research and ordinary instructional methods such as ADDIE. Richter et al. point out that design-based research also could be more critical and that reflexive design assumes that neither the problem nor the possible solutions are given but are actually created in the process of design.

Situatedness and complexity

Context, i.e. situation-specific knowledge is an other important feature:

``A core part of design-based research as applied work involves situating the work in "naturalistic contexts".´´ (Barab & Squire, 2004: 11)

Prototypically, design experiments entail both engineering particular forms of learning and systematically studying those forms of learning within the context defined by the means of supporting them. This designed context is subject to test and revision, and the successive iterations that result play a role similar to that of systematic variation in experiment. (Cobb, diSessa, Lehrer, & Schauble (2003:9)

Related to complexity and situatedness is the idea of iteration.

Theory as output

DBR often produces theory as output, in particular an instructional design model with a design rule at its heart.

Such theory is often very contextual and not necessarily applicable to a wider context, i.e. it needs futher corroboration with more traditional research approaches.

According to Sandoval (talk at EPFL, 2007), types of knowledge that DBR typically can produce:

• Design knowledge (Edelson, 2002)
• Ontological innovation (DiSessa & Cobb, 2004)
• Local instructional theories (Cobb, Gravemeijer, 2004). There was some debate in his talk wether one could organize some comparative analysis from similar projects (a bit like in political science' similar systems design).

Example approaches

DBR is put into practise in different ways and from different perspectives ...

Reeves's recommendations

Verbatim quote from Reeves (2000:12):

• Focus on chronically difficult problems related to human learning and performance.
• Engage teachers, students, and colleagues in long-term collaborative research agendas.
• Carefully align any prototype technological solutions with instructional objectives, pedagogy, and assessment.
• Clarify the theoretical and practical design-principles that underlie prototype technological solutions, and conduct rigorous studies of these principles, their inherent assumptions, their implementation, and their outcomes in realistic settings.
• Share the results of your design experiments in multiple ways, including refereed and commercial publications, web-pages, conferences, and workshops.
• Expect to work very hard. Be patient and persevere. And enjoy the challenge and reward of a career worth having for its contributions to the greater good.

BGuILE

• See Conjecture map and BGuILE.

Design-Based Implementation Research (DBIR)

Design-Based Implementation Research (DBIR) (Penuel, Fishman, Cheng & Sabelli, 2011) also includes the study and design of implementation factors, e.g. change management

Methodology

Design-based research uses the full range of social science methodologies, but if faces specific challenges (most of which can be be found in other design-oriented research designs.

More or less according to Sandoval (EPFL talk, 2007):

• Complexity. How to avoid too much data (e.g. videotape everything) and conversely how to focus on the right data.
• Validity. Same as most qualitative research
• Generalization. Even to make general theoretical claims one need lots of design experiments.
• Replicability. Generally speaking it is not. (But features may be of course)
• Trajectory. How to organize the whole process in a systematic way, map conjectures to design changes etc.

Regarding the "trajectory", i.e. the organization of the whole process, Sandoval developed the use of conjecture maps, i.e. a kind of concept map that allows to visually organize major theoretical, design and outcome elements.

Writing up a DBR

Design-based research in longer articles or thesis can be presented in various ways. For example, Christopher Hoadley (via a e-mail message) made the following suggestions of which we present the summary.

I. Intro
II. Lit review/research questions
III. Methods: DBR
	A. description of context
	B. description of DBR generally
	C. initial research plan
IV. Design narrative: description of iterations
V. Results
VI. Discussion

I. Intro
II. Lit review/research questions
III. Methods
	A. description of context
	B. description of DBR generally
IV. Iteration 1
	A. questions and design propositions
	B. design narrative
	C. data + analysis (findings)
	D. discussion
V. Iteration 2
VI. Iteration 3
VII. Results and discussion across iterations

I. Intro
II. Lit review/research questions
III. Methods
	A. description of context
	B. description of DBR generally
	C. Quick design narrative across iterations
IV. Theme 1
	A. questions and data sources across iterations
	B. analysis (findings)
	C. discussion
V. Theme 2
VI. Theme 3
... and so on
VII. Discussion across themes

Links

• Design-based Research Webliography
• Resources on DBR
• http://www.designbasedresearch.org/index.html
• Researchblog of Ilya Zitter. See also her list of interesting articles
• Design Based Research EPSS (includes peer tutorial, bibliography with texts online, expert interview, etc.)
• Graduate Student Resource Hub in Design Research in Education. Good link for Master and PhD students who plan to engage in this kind of research. “You may feel unsure about what it means to do design research. This website has collected materials and ideas that may offer you some insight into this type of research. You will find, for example, videos about the theory and the practice, a bibliography, reviews, a study guide and a glossary. You are encouraged to join the design research community and contribute directly to this evolving website.” (retrieved, nov 2007).
• Following a Design-based Research Cycle by Mike M Rook, November 30, 2010. (summary of Nelson et al. 2005).
• Educational Design Research. An International Journal for Design-Based Research in Education.

References

• Anderson, T., Shattuck, J. (2012). Design-Based Research: A decade of progress in education research, Educational Researcher', vol. 41, n° 1, p. 16-25. http://edr.sagepub.com/content/41/1/16

• Bannan, B. (2012). Design research and twice exceptional children: Toward an integration of motivation, emotion and cognition factors for a technology-based intervention. In Dai, D.Y. (Ed.), Design Research on Learning and Thinking in Education Settings: Enhancing Intellectual Growth and Functioning. Mahwah, N.J.; Lawrence Erlbaum Associates.

• Barab, S. A., & Kirshner, D. (Eds.) (2001) Special issue: Rethinking methodology in the learning sciences. Journal of the Learning Sciences, 10(1&2), 1-222.

• Barab, S. A., & Kirshner, D. (Eds.). (2001). Rethinking methodology in the learning sciences. [Special Issue] Journal of the Learning Sciences, 10(1&2).

• Barab, S. A., & Squire, K. (Eds.). (2004). Design-based research. [Special Issue] Journal of the Learning Sciences, 13(1).

• Barab, S. A., Dodge, T., Thomas, M. K., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263-305.

• Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1) 1-14.

• Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243-253.

• Bell, Philip, Christopher M Hoadley, and Marcia C Linn. 2004. Design-based research in education. In Marcia C Linn, Elizabeth A Davis, and Philip Bell (eds.), Internet environments for science education, 73-85. Lawrence Erlbaum.

• Boling, E. (2010). The need for design cases: Disseminating design knowledge. International Journal of Designs for Learning, 1(1). https://scholarworks.iu.edu/journals/index.php/ijdl/article/view/919/978 (not design-based research, but a complementary approach)

• Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178.

• Burkhardt, Hugh and Alan H. Schoenfeld, Improving Educational Research: Toward a More Useful, More Influential, and Better-Funded Enterprise, Educational Researcher , Vol. 32, No. 9 (Dec., 2003), pp. 3-14 JStor

• Class, Barbara et Schneider, Daniel. (2013). La Recherche Design en Education : vers une nouvelle approche ? : Educational design research: a use inspired basic research approach?. frantice.net, Numéro 7 - septembre 2013. Récupéré du site de la revue : http://www.frantice.net/document.php?id=764. ISSN 2110-5324

• Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

• Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. Journal of the Learning Sciences, 10(1&2), 113-163.

• Cobb, P., diSessa, A., Lehrer, R., Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. [1]

• Collins, A. (1992). Towards a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). Berlin: Springer.

• Collins, Alan., Diana Joseph & Katerine Bielaczyc (2004). Design Research: Theoretical and Methodological Issues, The Journal Of The Learning Sciences, 13(1), 15-42.

• A Collins (2010). Design Experiments, http://www.elsevierdirect.com/brochures/educ/PDF/Design_Experiments.pdf

• Design-Based Research Collective (2003) Design-Based Research: An Emerging Paradigm for Educational Inquiry. Educational Researcher, Vol. 32, No. 1, pp. 5

• diSessa, A. A. (1991). Local sciences: Viewing the design of human-computer systems as cognitive science. In J. M. Carroll (Ed.), Designing Interaction: Psychology at the Human-Computer Interface. NY: Cambridge University Press, 162-202.

• diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103.

• Edelson, D. C. (2002). Design research: what we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121.

• Enyedy, N. (2005). Inventing mapping: creating cultural forms to solve collective problems. Cognition and Instruction, 23(4), 427-466. (this is an example study).

• Ford, C., McNally, D., & Ford, K. (2017). Using Design-Based Research in Higher Education Innovation. Online Learning, 21(3). doi:http://dx.doi.org/10.24059/olj.v21i3.1232

• Gutierrez, K. D., & Penuel, W. R. (2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43(1), 19-23.

• Gravemeijer, K. (2004). Local instruction theories as means of support for teachers in reform mathematics education. Mathematical thinking and learning, 6(2), 105-128.

• Hake, R. R. (2007). Design-based research in physics education: a review. (M. C. Linn, E. A. Davis, & P. Bell, Eds.)Education, (March), 1-24. Lawrence Erlbaum. Citeseerx

• Henrick, Erin and Paul CobbEducational (2012?). Design Research to Support System- Wide Instructional Improvement, [2]

• Herrington, J. A. (2006). Authentic e-learning in higher education: design principles for authentic learning environments and tasks. In T. C. Reeves & S. Yamashita (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2006 (pp. 3164-3173). Chesapeake, VA: AACE.

• Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer Support for Collaborative Learning 2002 (pp. 453-462). Mahwah, NJ: Lawrence Erlbaum Associates.

• Hoadley, C. (2004). Fostering collaboration offline and online: Learning from each other. In M. C. Linn, E. A. Davis & P. L. Bell (Eds.), Internet Environments for Science Education (pp. 145-174). Mahwah, NJ: Lawrence Erlbaum Associates. (good example)

• Kelly, A. E. (Ed.). (2003). Theme issue: the role of design in educational research. [Special Issue] Educational Researcher, 32(1).

• Kelly, Anthony, E. (2003), Research as Design, Educational Researcher, 32 (1), 3-4.

• Kelly, Anthony, E. Design Research in Education: Yes, but Is It Methodological?, The Journal of the Learning Sciences , Vol. 13, No. 1, Design-Based Research: Clarifying the Terms. Introduction to the Learning Sciences Methodology Strand (2004), pp. 115-128 JStore.

• Kelly, Anthony, Richard Lesh & John Baek (eds.) (2008). Handbook of Design Research Methods in Education, Routledge, ISBN 978-0-8058-6059-7. This volume is designed as a guide for doctoral students, early career researchers and cross-over researchers from fields outside of education interested in supporting innovation in educational settings through conducting design research.

• Kelly, A. E. (in press). Developing Validity and Reliability Criteria for Assessments in Innovation and Design Research Studies” in David Yun Dai (Ed.), Design Research on Learning and Thinking in Educational Settings: Enhancing Intellectual Growth and Functioning. New York: Taylor & Francis.

• Kelly, A. E. (2009). When is design research appropriate? In T. Plomb & N. Nieveen (Eds.), Introduction to Educational Design Research (pp. 73-88), PDF

• Juuti, K., & Lavonen, J. (2006). Design-Based Research in Science Education: One Step Towards Methodology. NorDiNa, 4, 54-68.

• Lehrer, R., & Romberg, T. (1996). Exploring children's data modeling. Cognition & Instruction, 14(1), 69-108. (example study)

• Lesh, R. A., & Kelly, A. E. (2000). Multitiered teaching experiments. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 197-230). Mahwah, NJ: Lawrence Erlbaum Associates.

• Levin, J. R., & O'Donnell, A. M. (1999). What to do about educational research's credibility gaps? Issues in Education, 5(2), 177-229.

• Linn, M. C. (1987). Establishing a research base for science education: Challenges, trends, and recommendations. Journal of Research in Science Teaching, 24(3), 191-216.

• Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science Education, 22(8), 781-79 (example study)

• McKenney, Susan & Thomas Reeves (2014). Educational Design Research, in J. Micheal Spector, M. David Merrill, Jan Elen, and M.J. Bishop (eds.) Handbook of Research For Educational Communications and Technology, 4th edition, Springer.
  • If you want free access to this excellent text, become a member of AECT, the Association for Educational Communications and Technology.

• McKenney, Susan & Thomas Reeves (2019). Conducting Educational Design Research. 2nd edition. London and New York: Routledge.
• Nelson, B., Ketelhut, D. J., Clarke, J., Bowman, C., and Dede, C. (2005). Design-based research strategies for developing a scientific inquiry curriculum in a multi-user virtual environment. Educational Technology, 45(1), 21-27.

• Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing Research and Development at the Intersection of Learning, Implementation, and Design. Educational Researcher, 40(7), 331‑337. https://doi.org/10.3102/0013189X11421826

• Reeves, Thomas C. (2000). Enhancing the Worth of Instructional Technology Research through Design Experiments and Other Development Research Strategies, Paper presented on April 27, 2000 at Session 41.29, International Perspectives on Instructional Technology Research for the 21st Century, a Symposium sponsored by SIG/Instructional Technology at the Annual Meeting of the American Educational Research Association, New Orleans, LA, USA. PDF.

• Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86-109). London: Routledge. (A prior version of this text is above, i.e. Reeves, 2000)

• Reeves, T.C. (2011) Can Educational Research Be Both Rigorous and Relevant? Educational Designer, 1(4). Retrieved from: http://www.educationaldesigner.org/ed/volume1/issue4/article13/

• Reeves, Thomas C. & Susan McKenney (2012). 7 Things You Should Know About Educational Design Research, EduCause

• Reeves, Thomas C., Susan McKenney & Jan Herrington (2011), Publishing and perishing: The critical importance of educational design research, Australasian Journal of Educational Technology, 27(1), 55-65. HTML PDF

• Reiser, B. J., Tabak, I., Sandoval, W. A., Smith, B. K., Steinmuller, F., & Leone, A. J. (2001). BGuILE: Strategic and conceptual scaffolds for scientific inquiry in biology classrooms. In S. M. Carver & D. Klahr (Eds.), Cognition and instruction: Twenty-five years of progress (pp. 263-305). Mahwah, NJ: Lawrence Erlbaum. (example study).

• Reymen M. M. J., D. K. Hammer, P. A. Kroes, J. E. van Aken, C. H. Dorst, M. F. T. Bax and T. Basten (2006), A domain-independent descriptive design model and its application to structured reflection on design processes, Research in Engineering Design, 16 (4), 147-173. Abstract PDF/HTML (Access restricted) (This is also a good overview article)
• Richter, C. & Allert, H. (2017). Design as critical engagement in and for education. EDeR - Educational Design Research, 1(1), 1-20. http://dx.doi.org/10.15460/eder.1.1.1023

• Sandoval, William A. & Philip Bell (2004), Design-Based Research Methods for Studying Learning in Context: Introduction, Educational Psychologist, Vol. 39, No. 4: pages 199-201. doi:10.1207/s15326985ep3904_3

• Sandoval, W. A., & Bell, P. (Eds.). (2004). Design-based research methods for studying learning in context. [Special Issue] Educational Psychologist, 39(4).

• Sandoval, William A. (2004a). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Preprint. [To appear in Educational Psychologist, Vol. 39, No. 4, Pages 213-223, see below]

• Sandoval, William A. (2004). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Educational Psychologist, Vol. 39, No. 4, Pages 213-223.

• Shavelson, R. J., Phillips, D. C., Towne, L., & Feuer, M. J. (2003). On the science of educational design studies. Educational Researcher, 32(1), 25-28.

• Simonson, Michael (1985).Design-based Research, Applications for distance education, Distance Education (1985) Volume: 7, Issue: 2, Pages: pp. vii-viii

• Smith, K. M. (2010). Producing the rigorous design case. ​ International Journal of Designs for Learning, 1(1). https://scholarworks.iu.edu/journals/index.php/ijdl/article/view/917/980

• Steffe, L. P., & Thompson, P., W. (2000). Teaching experiment methodology: underlying principles and essential elements. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 267-306). Mahwah, NJ: Lawrence Erlbaum Assoc.

• Tabak, I. (2004). Reconstructing context: negotiating the tension between exogenous and endogenous educational design. Educational Psychologist, 39(4), 225-233.

• University of California at Berkeley, Field Guide to Design Experiments in Education, [3]

• Van den Akker Jan, Koeno Gravemeijer, Susan McKenney, Nienke Nieveen (2006). Educational Design Research, Routledge, ISBN ISBN 0415396352 - ISBN 9780415396349
• Van den Berg, N & Teurlings, C. with collaboration of Janssen, B (2019). From practical issue to research question. Knowledge Roundabout, The Netherlands. Flyer. Full book.

• Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23.

• White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition & Instruction, 16(1), 3-118. (example)

• Zitter, Ilya (2006), Design of competency-based, ICT-supported learning environments in higher education: The role of artefacts, ICO Toogdag research meeting PDF

Influences from other fields

• Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116-127.

• Erickson, F. (1982). Classroom discourse as improvisation: relationships between academic task structure and social participation structures in lessons. In L. C. Wilkinson (Ed.), Communicating in the classroom (pp. 153-181). New York: Academic Press.

• Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26.

• Koschmann, T., Kelson, A. C., Feltovich, P. J., & Barrows, H. S. (1996). Computer-supported problem-based learning: a principled approach to the use of computers in collaborative learning. In T. Koschmann (Ed.), CSCL: theory and practice of an emerging paradigm (pp. 83-124). Hillsdale, NJ: Erlbaum.

• Winn, W. (2003). Research methods and types of evidence for research in educational technology. Educational Psychology Review, 15(4), 367-373.

• Zhang, Jianwei; Marlene Scardamalia, Mary Lamon, Richard Messina and Richard Reeve (2007). Socio-cognitive dynamics of knowledge building in the work of 9- and 10-year-olds, ETR&D, 55 (2), 117.145 http://dx.doi.org/10.1007/s11423-006-9019-0.