Design-based research

Definition

According to Collins et al (2004: 15) “"the term "design experiments" was introduced in 1992, in articles by Ann Brown (1992) and Allan Collins (1992). Design experiments were developed as a way to carry out formative research to test and refine educational designs based on principles derived from prior research.”

According to Sandoval (talk, 2007). DBR is about intervention: when it works, how it works and for who it works.

See also:

• design science, design language and design methodology
• educational modeling language

What is DBR ?

A short history

Design-based reasearch (DBR) in education is probably very old, but recent interest can be traced back to the early nineties, e.g. Brown (1992) and Collins (1992).

According to Reeves (2000:8), Ann Brown (1992) and Alan Collins (1992) defined critical characteristics of design experiments as:

• addressing complex problems in real contexts in collaboration with practitioners,
• integrating known and hypothetical design principles with technological affordances to render plausible solutions to these complex problems, and
• conducting rigorous and reflective inquiry to test and refine innovative learning environments as well as to define new design principles.

According to the Design-Based Research Collective (2003): “First, the central goals of designing learning environments and developing theories or “prototheories” of learning are intertwined. Second, development and research take place through continuous cycles of design, enactment, analysis, and redesign. Third, research on designs must lead to sharable theories that help communicate relevant implications to practitioners and other educational designers. Fourth, research must account for how designs function in authentic settings. It must not only document success or failure but also focus on interactions that refine our understanding of the learning issues involved. Fifth, the development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interest.”

More recently, special issues of Educational Researcher (e.g. Kelly 2003), the Journal of Learning Sciences (e.g. Barab 2004) and the Educational Psychologist (e.g. Sandoval & Bell 2004) reopened the debate. In addition some researchers joined in the Design Based Research Collective.

According to Collins et al (2004: 16), design research was developed to address several issues central to the study of learning, including the following:

• The need to address theoretical questions about the nature of learning in context.
• The need for approaches to the study of learning phenomena in the real world rather than the laboratory.
• The need to go beyond narrow measures of learning.
• The need to derive research findings from formative evaluation.

DBR v.s. traditional empirical research

Reeves (2000:9) draws a clear line between research conducted with traditional empirical goals and that inspired by development goals leading to "Design Principles":

Reprinted without permission for the moment (Reeves, 2000:9)

Action orientation

There is clearly an action-research oriented perspective, i.e. researchers must try to change things.

The overall goal of research within the empirical tradition is to develop long-lasting theories and unambiguous principles that can be handed off to practitioners for implementation. Development research, on the other hand, requires a pragmatic epistemology that regards learning theory as being collaboratively shaped by researchers and practitioners. The overall goal of development research is to solve real problems while at the same time constructing design principles that can inform future decisions. In Kuhn's terms, these are different worlds." (Reeves, 2000: 12).

Situatedness and complexity

Context, i.e. situation-specific knowledge is an other important feature:

``A core part of design-based research as applied work involves situating the work in "naturalistic contexts".´´ (Barab & Squire, 2004: 11)

Prototypically, design experiments entail both engineering particular forms of learning and systematically studying those forms of learning within the context defined by the means of supporting them. This designed context is subject to test and revision, and the successive iterations that result play a role similar to that of systematic variation in experiment. (Cobb, diSessa, Lehrer, & Schauble (2003:9)

Related to complexity and situatedness is the idea of iteration.

Theory as output

DBR often produces theory as output, in particular an instructional design model with a design rule at its heart.

Such theory is often very contextual and not necessarily applicable to a wider context, i.e. it needs futher corroboration with more traditional research approaches.

According to Sandoval (talk at EPFL, 2007), types of knowledge that DBR typically can produce:

• Design knowledge (Edelson, 2002)
• Ontological innovation (DiSessa & Cobb, 2004)
• Local instructional theories (Cobb). There was some debate in his talk wether one could organize some comparative analysis from similar projects (a bit like in political science' similar systems design).

Example approaches

DBR is put into practise in different ways and from different perspective

Reeves's recommendations

Verbatim quote from Reeves (2000:12):

• Focus on chronically difficult problems related to human learning and performance.
• Engage teachers, students, and colleagues in long-term collaborative research agendas.
• Carefully align any prototype technological solutions with instructional objectives, pedagogy, and assessment.
• Clarify the theoretical and practical design principles that underlie prototype technological solutions, and conduct rigorous studies of these principles, their inherent assumptions, their implementation, and their outcomes in realistic settings.
• Share the results of your design experiments in multiple ways, including refereed and commercial publications, web-pages, conferences, and workshops.
• Expect to work very hard. Be patient and persevere. And enjoy the challenge and reward of a career worth having for its contributions to the greater good.

BGuILE

• See BGuILE

Methodology

Some challenges

More or less according to Sandoval (EPFL talk, 2007):

• Complexity. How to avoid too much data (e.g. videotape everything) and conversely how to focus on the right data.
• Validity. Same as most qualitative research
• Generalization. Even to make general theoretical claims one need lots of design experiments.
• Replicability. Generally speaking it is not. (But features may be of course)
• Trajectory. How to organize the whole process in a systematic way, map conjectures to design changes etc.

Conjecture Maps

The idea is to map design elements to "intermediary variables" to desired outcomes. Each link is a kind of hypothesis for which various kinds of data could provide some support. (Sandoval/BGuiLE, ... more to come).

Example conjecture map by William A. Sandoval

Here is a skeleton to get started:

Theoretical conjectures	Design elements	What's happening (intermediate outcomes)	Objective outcomes
	Rather features than just "names"

Links

• http://www.designbasedresearch.org/index.html
• Researchblog of Ilya Zitter. See also her list of interesting articles

References

• Barab, S. A., & Kirshner, D. (Eds.) (2001) Special issue: Rethinking methodology in the learning sciences. Journal of the Learning Sciences, 10(1&2), 1-222.

• Barab, S. A., & Kirshner, D. (Eds.). (2001). Rethinking methodology in the learning sciences. [Special Issue] Journal of the Learning Sciences, 10(1&2).

• Barab, S. A., & Squire, K. (Eds.). (2004). Design-based research. [Special Issue] Journal of the Learning Sciences, 13(1).

• Barab, S. A., Dodge, T., Thomas, M. K., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263-305.

• Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1)

• Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243-253.

• Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178.

• Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

• Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. Journal of the Learning Sciences, 10(1&2), 113-163.

• Cobb, P., diSessa, A., Lehrer, R., Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. [1]

• Collins, A. (1992). Towards a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). Berlin: Springer.

• Collins, Alan., Diana Joseph & Katerine Bielaczyc (2004). Design Research: Theoretical and Methodological Issues, The Journal Of The Learning Sciences, 13(1), 15-42.

• Design-Based Research Collective (2003) Design-Based Research: An Emerging Paradigm for Educational Inquiry. Educational Researcher, Vol. 32, No. 1, pp. 5

• diSessa, A. A. (1991). Local sciences: Viewing the design of human-computer systems as cognitive science. In J. M. Carroll (Ed.), Designing Interaction: Psychology at the Human-Computer Interface. NY: Cambridge University Press, 162-202.

• diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103.

• Edelson, D. C. (2002). Design research: what we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121.

• Enyedy, N. (2005). Inventing mapping: creating cultural forms to solve collective problems. Cognition and Instruction, 23(4), 427-466. (this is an example study).

• Kelly, A. E. (Ed.). (2003). Theme issue: the role of design in educational research. [Special Issue] Educational Researcher, 32(1).

• Kelly, Anthony, E. (2003), Research as Design, Educational Researcher, 32 (1), 3-4.

• Lehrer, R., & Romberg, T. (1996). Exploring children's data modeling. Cognition & Instruction, 14(1), 69-108. (example study)

• Lesh, R. A., & Kelly, A. E. (2000). Multitiered teaching experiments. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 197-230). Mahwah, NJ: Lawrence Erlbaum Associates.

• Levin, J. R., & O'Donnell, A. M. (1999). What to do about educational research's credibility gaps? Issues in Education, 5(2), 177-229.

• Linn, M. C. (1987). Establishing a research base for science education: Challenges, trends, and recommendations. Journal of Research in Science Teaching, 24(3), 191-216.

• Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science Education, 22(8), 781-79 (example study)

• Reeves, Thomas C. (2000). Enhancing the Worth of Instructional Technology Research through Design Experiments and Other Development Research Strategies, Paper presented on April 27, 2000 at Session 41.29, International Perspectives on Instructional Technology Research for the 21st Century, a Symposium sponsored by SIG/Instructional Technology at the Annual Meeting of the American Educational Research Association, New Orleans, LA, USA. PDF

• Reiser, B. J., Tabak, I., Sandoval, W. A., Smith, B. K., Steinmuller, F., & Leone, A. J. (2001). BGuILE: Strategic and conceptual scaffolds for scientific inquiry in biology classrooms. In S. M. Carver & D. Klahr (Eds.), Cognition and instruction: Twenty-five years of progress (pp. 263-305). Mahwah, NJ: Lawrence Erlbaum. (example study).

• Reymen M. M. J., D. K. Hammer, P. A. Kroes, J. E. van Aken, C. H. Dorst, M. F. T. Bax and T. Basten (2006), A domain-independent descriptive design model and its application to structured reflection on design processes, Research in Engineering Design, 16 (4), 147-173. Abstract PDF/HTML (Access restricted) (This is also a good overview article)

• Sandoval, William A. & Philip Bell (2004), Design-Based Research Methods for Studying Learning in Context: Introduction, Educational Psychologist, Vol. 39, No. 4: pages 199-201. doi:10.1207/s15326985ep3904_3

• Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39(4), 213-223.

• Sandoval, W. A., & Bell, P. (Eds.). (2004). Design-based research methods for studying learning in context. [Special Issue] Educational Psychologist, 39(4).

• Sandoval, William A. (2004). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Educational Psychologist, Vol. 39, No. 4, Pages 213-223.

• Shavelson, R. J., Phillips, D. C., Towne, L., & Feuer, M. J. (2003). On the science of educational design studies. Educational Researcher, 32(1), 25-28.

• Steffe, L. P., & Thompson, P., W. (2000). Teaching experiment methodology: underlying principles and essential elements. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 267-306). Mahwah, NJ: Lawrence Erlbaum Assoc.

• Tabak, I. (2004). Reconstructing context: negotiating the tension between exogenous and endogenous educational design. Educational Psychologist, 39(4), 225-233.

• University of California at Berkeley, Field Guide to Design Experiments in Education, [2]

• Van den Akker Jan, Koeno Gravemeijer, Susan McKenney, Nienke Nieveen (2006). Educational Design Research, Routledge, ISBN 9780415396349

• White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition & Instruction, 16(1), 3-118. (example)

• Zitter, Ilya (2006), Design of competency-based, ICT-supported learning environments in higher education: The role of artefacts, ICO Toogdag research meeting PDF

Influences from other fields

• Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116-127.

• Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26.

• Koschmann, T., Kelson, A. C., Feltovich, P. J., & Barrows, H. S. (1996). Computer-supported problem-based learning: a principled approach to the use of computers in collaborative learning. In T. Koschmann (Ed.), CSCL: theory and practice of an emerging paradigm (pp. 83-124). Hillsdale, NJ: Erlbaum.

• Winn, W. (2003). Research methods and types of evidence for research in educational technology. Educational Psychology Review, 15(4), 367-373.