Design-based research

Definition

According to Collins et al (2004: 15) “"the term "design experiments" was introduced in 1992, in articles by Ann Brown (1992) and Allan Collins (1992). Design experiments were developed as a way to carry out formative research to test and refine educational designs based on principles derived from prior research.”

According to Sandoval (talk, 2007). DBR is about intervention: when it works, how it works and for who it works.

See also:

• design science, design language and design methodology
• educational modeling language

What is DBR ?

A short history

Design-based reasearch (DBR) in education is probably very old, but recent interest can be traced back to the early nineties, e.g. Brown (1992) and Collins (1992).

According to Reeves (2000:8), Ann Brown (1992) and Alan Collins (1992) defined critical characteristics of design experiments as:

• addressing complex problems in real contexts in collaboration with practitioners,
• integrating known and hypothetical design-principles with technological affordances to render plausible solutions to these complex problems, and
• conducting rigorous and reflective inquiry to test and refine innovative learning environments as well as to define new design-principles.

According to the Design-Based Research Collective (2003): “First, the central goals of designing learning environments and developing theories or “prototheories” of learning are intertwined. Second, development and research take place through continuous cycles of design, enactment, analysis, and redesign. Third, research on designs must lead to sharable theories that help communicate relevant implications to practitioners and other educational designers. Fourth, research must account for how designs function in authentic settings. It must not only document success or failure but also focus on interactions that refine our understanding of the learning issues involved. Fifth, the development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interest.”

More recently, special issues of Educational Researcher (e.g. Kelly 2003), the Journal of Learning Sciences (e.g. Barab 2004) and the Educational Psychologist (e.g. Sandoval & Bell 2004) reopened the debate. In addition some researchers joined in the Design Based Research Collective.

According to Collins et al (2004: 16), design research was developed to address several issues central to the study of learning, including the following:

• The need to address theoretical questions about the nature of learning in context.
• The need for approaches to the study of learning phenomena in the real world rather than the laboratory.
• The need to go beyond narrow measures of learning.
• The need to derive research findings from formative evaluation.

DBR v.s. traditional empirical research

Reeves (2000:9, 2006) draws a clear line between research conducted with traditional empirical goals and that inspired by development goals leading to "Design-principles".

Predictive and design research approaches in educational technology research. In Reeves, T.C. (2006), Design research from the technology perspective

Action orientation

There is clearly an action-research oriented perspective, i.e. researchers must try to change things.

The overall goal of research within the empirical tradition is to develop long-lasting theories and unambiguous principles that can be handed off to practitioners for implementation. Development research, on the other hand, requires a pragmatic epistemology that regards learning theory as being collaboratively shaped by researchers and practitioners. The overall goal of development research is to solve real problems while at the same time constructing Design-principles that can inform future decisions. In Kuhn's terms, these are different worlds." (Reeves, 2000: 12).

Situatedness and complexity

Context, i.e. situation-specific knowledge is an other important feature:

``A core part of design-based research as applied work involves situating the work in "naturalistic contexts".´´ (Barab & Squire, 2004: 11)

Prototypically, design experiments entail both engineering particular forms of learning and systematically studying those forms of learning within the context defined by the means of supporting them. This designed context is subject to test and revision, and the successive iterations that result play a role similar to that of systematic variation in experiment. (Cobb, diSessa, Lehrer, & Schauble (2003:9)

Related to complexity and situatedness is the idea of iteration.

Theory as output

DBR often produces theory as output, in particular an instructional design model with a design rule at its heart.

Such theory is often very contextual and not necessarily applicable to a wider context, i.e. it needs futher corroboration with more traditional research approaches.

According to Sandoval (talk at EPFL, 2007), types of knowledge that DBR typically can produce:

• Design knowledge (Edelson, 2002)
• Ontological innovation (DiSessa & Cobb, 2004)
• Local instructional theories (Cobb). There was some debate in his talk wether one could organize some comparative analysis from similar projects (a bit like in political science' similar systems design).

Example approaches

DBR is put into practise in different ways and from different perspectives ...

Reeves's recommendations

Verbatim quote from Reeves (2000:12):

• Focus on chronically difficult problems related to human learning and performance.
• Engage teachers, students, and colleagues in long-term collaborative research agendas.
• Carefully align any prototype technological solutions with instructional objectives, pedagogy, and assessment.
• Clarify the theoretical and practical design-principles that underlie prototype technological solutions, and conduct rigorous studies of these principles, their inherent assumptions, their implementation, and their outcomes in realistic settings.
• Share the results of your design experiments in multiple ways, including refereed and commercial publications, web-pages, conferences, and workshops.
• Expect to work very hard. Be patient and persevere. And enjoy the challenge and reward of a career worth having for its contributions to the greater good.

BGuILE

• See Conjecture maps below and BGuILE.

Methodology

Design-based research uses the full range of social science methodologies, but if faces specific challenges (most of which can be be found in other [[Methodology tutorial - design-oriented research designs|design-oriented researche designs].

More or less according to Sandoval (EPFL talk, 2007):

• Complexity. How to avoid too much data (e.g. videotape everything) and conversely how to focus on the right data.
• Validity. Same as most qualitative research
• Generalization. Even to make general theoretical claims one need lots of design experiments.
• Replicability. Generally speaking it is not. (But features may be of course)
• Trajectory. How to organize the whole process in a systematic way, map conjectures to design changes etc.

Regarding the "trajectory", i.e. the organization of the whole process, Sandoval developed the use of conjecture maps, i.e. a kind of concept map that allows to visually organizse major theorectical, design and outcome elements.

Writing up a DBR

Design-based research in longer articles or thesis can be presented in various ways. For example, Christopher Hoadley (via a e-mail message) made the following suggestions of which we present the summary.

I. Intro
II. Lit review/research questions
III. Methods: DBR
	A. description of context
	B. description of DBR generally
	C. initial research plan
IV. Design narrative: description of iterations
V. Results
VI. Discussion

I. Intro
II. Lit review/research questions
III. Methods
	A. description of context
	B. description of DBR generally
IV. Iteration 1
	A. questions and design propositions
	B. design narrative
	C. data + analysis (findings)
	D. discussion
V. Iteration 2
VI. Iteration 3
VII. Results and discussion across iterations

I. Intro
II. Lit review/research questions
III. Methods
	A. description of context
	B. description of DBR generally
	C. Quick design narrative across iterations
IV. Theme 1
	A. questions and data sources across iterations
	B. analysis (findings)
	C. discussion
V. Theme 2
VI. Theme 3
... and so on
VII. Discussion across themes

Links

• Design-based Research Webliography
• Resources on DBR
• http://www.designbasedresearch.org/index.html
• Researchblog of Ilya Zitter. See also her list of interesting articles
• Design Based Research EPSS (includes peer tutorial, bibliography with texts online, expert interview, etc.)
• Graduate Student Resource Hub in Design Research in Education. Good link for Master and PhD students who plan to engage in this kind of research. “You may feel unsure about what it means to do design research. This website has collected materials and ideas that may offer you some insight into this type of research. You will find, for example, videos about the theory and the practice, a bibliography, reviews, a study guide and a glossary. You are encouraged to join the design research community and contribute directly to this evolving website.” (retrieved, nov 2007).

References

• Barab, S. A., & Kirshner, D. (Eds.) (2001) Special issue: Rethinking methodology in the learning sciences. Journal of the Learning Sciences, 10(1&2), 1-222.

• Barab, S. A., & Kirshner, D. (Eds.). (2001). Rethinking methodology in the learning sciences. [Special Issue] Journal of the Learning Sciences, 10(1&2).

• Barab, S. A., & Squire, K. (Eds.). (2004). Design-based research. [Special Issue] Journal of the Learning Sciences, 13(1).

• Barab, S. A., Dodge, T., Thomas, M. K., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263-305.

• Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1) 1-14.

• Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243-253.

• Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178.

• Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

• Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. Journal of the Learning Sciences, 10(1&2), 113-163.

• Cobb, P., diSessa, A., Lehrer, R., Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. [1]

• Collins, A. (1992). Towards a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). Berlin: Springer.

• Collins, Alan., Diana Joseph & Katerine Bielaczyc (2004). Design Research: Theoretical and Methodological Issues, The Journal Of The Learning Sciences, 13(1), 15-42.

• Design-Based Research Collective (2003) Design-Based Research: An Emerging Paradigm for Educational Inquiry. Educational Researcher, Vol. 32, No. 1, pp. 5

• diSessa, A. A. (1991). Local sciences: Viewing the design of human-computer systems as cognitive science. In J. M. Carroll (Ed.), Designing Interaction: Psychology at the Human-Computer Interface. NY: Cambridge University Press, 162-202.

• diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103.

• Edelson, D. C. (2002). Design research: what we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121.

• Enyedy, N. (2005). Inventing mapping: creating cultural forms to solve collective problems. Cognition and Instruction, 23(4), 427-466. (this is an example study).

• Herrington, J. A. (2006). Authentic e-learning in higher education: design principles for authentic learning environments and tasks. In T. C. Reeves & S. Yamashita (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2006 (pp. 3164-3173). Chesapeake, VA: AACE.

• Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer Support for Collaborative Learning 2002 (pp. 453-462). Mahwah, NJ: Lawrence Erlbaum Associates.

• Hoadley, C. (2004). Fostering collaboration offline and online: Learning from each other. In M. C. Linn, E. A. Davis & P. L. Bell (Eds.), Internet Environments for Science Education (pp. 145-174). Mahwah, NJ: Lawrence Erlbaum Associates. (good example)

• Kelly, A. E. (Ed.). (2003). Theme issue: the role of design in educational research. [Special Issue] Educational Researcher, 32(1).

• Kelly, Anthony, E. (2003), Research as Design, Educational Researcher, 32 (1), 3-4.

• Kelly, Anthony, Richard Lesh & John Baek (eds.) (2008). Handbook of Design Research Methods in Education, Routledge, ISBN 978-0-8058-6059-7. This volume is designed as a guide for doctoral students, early career researchers and cross-over researchers from fields outside of education interested in supporting innovation in educational settings through conducting design research.

• Lehrer, R., & Romberg, T. (1996). Exploring children's data modeling. Cognition & Instruction, 14(1), 69-108. (example study)

• Lesh, R. A., & Kelly, A. E. (2000). Multitiered teaching experiments. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 197-230). Mahwah, NJ: Lawrence Erlbaum Associates.

• Levin, J. R., & O'Donnell, A. M. (1999). What to do about educational research's credibility gaps? Issues in Education, 5(2), 177-229.

• Linn, M. C. (1987). Establishing a research base for science education: Challenges, trends, and recommendations. Journal of Research in Science Teaching, 24(3), 191-216.

• Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science Education, 22(8), 781-79 (example study)

• Reeves, Thomas C. (2000). Enhancing the Worth of Instructional Technology Research through Design Experiments and Other Development Research Strategies, Paper presented on April 27, 2000 at Session 41.29, International Perspectives on Instructional Technology Research for the 21st Century, a Symposium sponsored by SIG/Instructional Technology at the Annual Meeting of the American Educational Research Association, New Orleans, LA, USA. PDF.

• Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86-109). London: Routledge. (A prior version of this text is above, i.e. Reeves, 2000)

• Reiser, B. J., Tabak, I., Sandoval, W. A., Smith, B. K., Steinmuller, F., & Leone, A. J. (2001). BGuILE: Strategic and conceptual scaffolds for scientific inquiry in biology classrooms. In S. M. Carver & D. Klahr (Eds.), Cognition and instruction: Twenty-five years of progress (pp. 263-305). Mahwah, NJ: Lawrence Erlbaum. (example study).

• Reymen M. M. J., D. K. Hammer, P. A. Kroes, J. E. van Aken, C. H. Dorst, M. F. T. Bax and T. Basten (2006), A domain-independent descriptive design model and its application to structured reflection on design processes, Research in Engineering Design, 16 (4), 147-173. Abstract PDF/HTML (Access restricted) (This is also a good overview article)

• Sandoval, William A. & Philip Bell (2004), Design-Based Research Methods for Studying Learning in Context: Introduction, Educational Psychologist, Vol. 39, No. 4: pages 199-201. doi:10.1207/s15326985ep3904_3

• Sandoval, W. A., & Bell, P. (Eds.). (2004). Design-based research methods for studying learning in context. [Special Issue] Educational Psychologist, 39(4).

• Sandoval, William A. (2004a). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Preprint. [To appear in Educational Psychologist, Vol. 39, No. 4, Pages 213-223, see below]

• Sandoval, William A. (2004). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Educational Psychologist, Vol. 39, No. 4, Pages 213-223.

• Shavelson, R. J., Phillips, D. C., Towne, L., & Feuer, M. J. (2003). On the science of educational design studies. Educational Researcher, 32(1), 25-28.

• Steffe, L. P., & Thompson, P., W. (2000). Teaching experiment methodology: underlying principles and essential elements. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 267-306). Mahwah, NJ: Lawrence Erlbaum Assoc.

• Tabak, I. (2004). Reconstructing context: negotiating the tension between exogenous and endogenous educational design. Educational Psychologist, 39(4), 225-233.

• University of California at Berkeley, Field Guide to Design Experiments in Education, [2]

• Van den Akker Jan, Koeno Gravemeijer, Susan McKenney, Nienke Nieveen (2006). Educational Design Research, Routledge, ISBN ISBN 0415396352 - ISBN 9780415396349

• Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23.

• White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition & Instruction, 16(1), 3-118. (example)

• Zitter, Ilya (2006), Design of competency-based, ICT-supported learning environments in higher education: The role of artefacts, ICO Toogdag research meeting PDF

Influences from other fields

• Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116-127.

• Erickson, F. (1982). Classroom discourse as improvisation: relationships between academic task structure and social participation structures in lessons. In L. C. Wilkinson (Ed.), Communicating in the classroom (pp. 153-181). New York: Academic Press.

• Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26.

• Koschmann, T., Kelson, A. C., Feltovich, P. J., & Barrows, H. S. (1996). Computer-supported problem-based learning: a principled approach to the use of computers in collaborative learning. In T. Koschmann (Ed.), CSCL: theory and practice of an emerging paradigm (pp. 83-124). Hillsdale, NJ: Erlbaum.

• Winn, W. (2003). Research methods and types of evidence for research in educational technology. Educational Psychology Review, 15(4), 367-373.

• Zhang, Jianwei; Marlene Scardamalia, Mary Lamon, Richard Messina and Richard Reeve (2007). Socio-cognitive dynamics of knowledge building in the work of 9- and 10-year-olds, ETR&D, 55 (2), 117.145 http://dx.doi.org/10.1007/s11423-006-9019-0.