Design-based research

Definition

According to Collins et al (2004: 15) “"the term "design experiments" was introduced in 1992, in articles by Ann Brown (1992) and Allan Collins (1992). Design experiments were developed as a way to carry out formative research to test and refine educational designs based on principles derived from prior research.”

According to Sandoval (talk, 2007). DBR is about intervention: when it works, how it works and for who it works.

See also:

• design science, design language and design methodology
• educational modeling language

What is DBR ?

A short history

Design-based reasearch (DBR) in education is probably very old, but recent interest can be traced back to the early nineties, e.g. Brown (1992) and Collins (1992).

According to Reeves (2000:8), Ann Brown (1992) and Alan Collins (1992) defined critical characteristics of design experiments as:

• addressing complex problems in real contexts in collaboration with practitioners,
• integrating known and hypothetical design-principles with technological affordances to render plausible solutions to these complex problems, and
• conducting rigorous and reflective inquiry to test and refine innovative learning environments as well as to define new design-principles.

According to the Design-Based Research Collective (2003): “First, the central goals of designing learning environments and developing theories or “prototheories” of learning are intertwined. Second, development and research take place through continuous cycles of design, enactment, analysis, and redesign. Third, research on designs must lead to sharable theories that help communicate relevant implications to practitioners and other educational designers. Fourth, research must account for how designs function in authentic settings. It must not only document success or failure but also focus on interactions that refine our understanding of the learning issues involved. Fifth, the development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interest.”

More recently, special issues of Educational Researcher (e.g. Kelly 2003), the Journal of Learning Sciences (e.g. Barab 2004) and the Educational Psychologist (e.g. Sandoval & Bell 2004) reopened the debate. In addition some researchers joined in the Design Based Research Collective.

According to Collins et al (2004: 16), design research was developed to address several issues central to the study of learning, including the following:

• The need to address theoretical questions about the nature of learning in context.
• The need for approaches to the study of learning phenomena in the real world rather than the laboratory.
• The need to go beyond narrow measures of learning.
• The need to derive research findings from formative evaluation.

DBR v.s. traditional empirical research

Reeves (2000:9, 2006) draws a clear line between research conducted with traditional empirical goals and that inspired by development goals leading to "Design-principles".

Predictive and design research approaches in educational technology research. In Reeves, T.C. (2006), Design research from the technology perspective

Action orientation

There is clearly an action-research oriented perspective, i.e. researchers must try to change things.

The overall goal of research within the empirical tradition is to develop long-lasting theories and unambiguous principles that can be handed off to practitioners for implementation. Development research, on the other hand, requires a pragmatic epistemology that regards learning theory as being collaboratively shaped by researchers and practitioners. The overall goal of development research is to solve real problems while at the same time constructing Design-principles that can inform future decisions. In Kuhn's terms, these are different worlds." (Reeves, 2000: 12).

Situatedness and complexity

Context, i.e. situation-specific knowledge is an other important feature:

``A core part of design-based research as applied work involves situating the work in "naturalistic contexts".´´ (Barab & Squire, 2004: 11)

Prototypically, design experiments entail both engineering particular forms of learning and systematically studying those forms of learning within the context defined by the means of supporting them. This designed context is subject to test and revision, and the successive iterations that result play a role similar to that of systematic variation in experiment. (Cobb, diSessa, Lehrer, & Schauble (2003:9)

Related to complexity and situatedness is the idea of iteration.

Theory as output

DBR often produces theory as output, in particular an instructional design model with a design rule at its heart.

Such theory is often very contextual and not necessarily applicable to a wider context, i.e. it needs futher corroboration with more traditional research approaches.

According to Sandoval (talk at EPFL, 2007), types of knowledge that DBR typically can produce:

• Design knowledge (Edelson, 2002)
• Ontological innovation (DiSessa & Cobb, 2004)
• Local instructional theories (Cobb). There was some debate in his talk wether one could organize some comparative analysis from similar projects (a bit like in political science' similar systems design).

Example approaches

DBR is put into practise in different ways and from different perspective

Reeves's recommendations

Verbatim quote from Reeves (2000:12):

• Focus on chronically difficult problems related to human learning and performance.
• Engage teachers, students, and colleagues in long-term collaborative research agendas.
• Carefully align any prototype technological solutions with instructional objectives, pedagogy, and assessment.
• Clarify the theoretical and practical design principles that underlie prototype technological solutions, and conduct rigorous studies of these principles, their inherent assumptions, their implementation, and their outcomes in realistic settings.
• Share the results of your design experiments in multiple ways, including refereed and commercial publications, web-pages, conferences, and workshops.
• Expect to work very hard. Be patient and persevere. And enjoy the challenge and reward of a career worth having for its contributions to the greater good.

BGuILE

• See Conjecture maps below and BGuILE.

Methodology

Some challenges

More or less according to Sandoval (EPFL talk, 2007):

• Complexity. How to avoid too much data (e.g. videotape everything) and conversely how to focus on the right data.
• Validity. Same as most qualitative research
• Generalization. Even to make general theoretical claims one need lots of design experiments.
• Replicability. Generally speaking it is not. (But features may be of course)
• Trajectory. How to organize the whole process in a systematic way, map conjectures to design changes etc.

Conjecture Maps

According to Sandoval (2004a:2), “designed learning environments embody conjectures about learning and instruction, and the empirical study of learning environments allows such conjectures to be refined over time. The construct of embodied conjecture is introduced as a way to demonstrate the theoretical nature of learning environment design, and to frame methodological issues in studying such conjectures.”

"Design" in this context refers to to interventions, including designed technologies, curricular materials, and participation structures. More precisely, “designed learning environments embody design conjectures about how to support learning in a specific context, that are themselves based on theoretical conjectures of how learning occurs in particular domains. Learning theory is often underspecified with respect to learning environment design and the educational research should understand the formulation of appropriate design conjectures as theoretical activity.”

An embodied conjecture is a conjecture about how theoretical propositions might be reified within designed environments to support learning. Designed environments include tools (like software), materials, and activity structures (defined as the combination of task structure, how a task is organized, and social participation structures, Erickson, 1982).

Embodied conjectures should predict outcomes at two levels:

• intermediate outcomes are observable patterns of behavior predicted by a model of how an embodied conjecture functions should support learning.
• intervention outcomes, refer to the sort of outcome that psychologists look for, e.g. whether students learn what they are intended to learn.

Embodied conjectures also should predict interactions with their contexts of use.

The idea of a conjecture map is to link design elements to "intermediary variables" to desired outcomes. Each link is a kind of hypothesis for which various kinds of data could provide some support.

BGUILE example

“ExplanationConstructor and related aspects of the BGuILE learning environment were designed by conjecturing that epistemological conceptions influence scientific inquiry to the extent that they frame particular goals for the kinds of knowledge inquiry can generate, while such ideas are practically helpful only to the extent that conceptual knowledge of particular domains can generate possible explanations and investigative strategies. This conjecture led to the design conjecture to integrate conceptual and epistemic scaffolds for inquiry, which was predicted to help students both learn a particular science topic and about scientific epistemology.” (Sandoval 2004a).

Example conjecture map by William A. Sandoval

It also may be useful to link design elements (embodied conjectures) to general learning and instructional theory. Here is a skeleton to get started:

Theoretical conjectures	Embodied conjectures	Intermediate outcomes	Objective outcomes
General theory not embodied, not operational	Design elements: Rather features than just "names"	What's happening ? What students and teachers do	What students will learn

Daniel K. Schneider wonders if sometimes one also might a fifth column to separate activity design from tools design. Both together form a cognitive instrument, but to get/configure affordable tools can be research problem by itself.

Theoretical conjectures	Embodied conjectures - activity types	Embodied conjectures - tools	Intermediate outcomes	Objective outcomes
General theory not embodied, not operational	Design elements: Teacher and learner activity elements	Design elements: Tools with features (e.g. software modules)	What's happening ? What students and teachers do	What students will learn

Links

• http://www.designbasedresearch.org/index.html
• Researchblog of Ilya Zitter. See also her list of interesting articles
• Design Based Research EPSS (includes peer tutorial, bibliography with texts online, expert interview, etc.)
• Graduate Student Resource Hub in Design Research in Education. Good link for Master and PhD students who plan to engage in this kind of research. “You may feel unsure about what it means to do design research. This website has collected materials and ideas that may offer you some insight into this type of research. You will find, for example, videos about the theory and the practice, a bibliography, reviews, a study guide and a glossary. You are encouraged to join the design research community and contribute directly to this evolving website.” (retrieved, nov 2007).

References

• Barab, S. A., & Kirshner, D. (Eds.) (2001) Special issue: Rethinking methodology in the learning sciences. Journal of the Learning Sciences, 10(1&2), 1-222.

• Barab, S. A., & Kirshner, D. (Eds.). (2001). Rethinking methodology in the learning sciences. [Special Issue] Journal of the Learning Sciences, 10(1&2).

• Barab, S. A., & Squire, K. (Eds.). (2004). Design-based research. [Special Issue] Journal of the Learning Sciences, 13(1).

• Barab, S. A., Dodge, T., Thomas, M. K., Jackson, C., & Tuzun, H. (2007). Our designs and the social agendas they carry. Journal of the Learning Sciences, 16(2), 263-305.

• Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1)

• Bell, P. (2004). On the theoretical breadth of design-based research in education. Educational Psychologist, 39(4), 243-253.

• Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The Journal of the Learning Sciences, 2(2), 141-178.

• Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13.

• Cobb, P., Stephan, M., McClain, K., & Gravemeijer, K. (2001). Participating in classroom mathematical practices. Journal of the Learning Sciences, 10(1&2), 113-163.

• Cobb, P., diSessa, A., Lehrer, R., Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. [1]

• Collins, A. (1992). Towards a design science of education. In E. Scanlon & T. O'Shea (Eds.), New directions in educational technology (pp. 15-22). Berlin: Springer.

• Collins, Alan., Diana Joseph & Katerine Bielaczyc (2004). Design Research: Theoretical and Methodological Issues, The Journal Of The Learning Sciences, 13(1), 15-42.

• Design-Based Research Collective (2003) Design-Based Research: An Emerging Paradigm for Educational Inquiry. Educational Researcher, Vol. 32, No. 1, pp. 5

• diSessa, A. A. (1991). Local sciences: Viewing the design of human-computer systems as cognitive science. In J. M. Carroll (Ed.), Designing Interaction: Psychology at the Human-Computer Interface. NY: Cambridge University Press, 162-202.

• diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103.

• Edelson, D. C. (2002). Design research: what we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121.

• Enyedy, N. (2005). Inventing mapping: creating cultural forms to solve collective problems. Cognition and Instruction, 23(4), 427-466. (this is an example study).

• Kelly, A. E. (Ed.). (2003). Theme issue: the role of design in educational research. [Special Issue] Educational Researcher, 32(1).

• Kelly, Anthony, E. (2003), Research as Design, Educational Researcher, 32 (1), 3-4.

• Lehrer, R., & Romberg, T. (1996). Exploring children's data modeling. Cognition & Instruction, 14(1), 69-108. (example study)

• Lesh, R. A., & Kelly, A. E. (2000). Multitiered teaching experiments. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 197-230). Mahwah, NJ: Lawrence Erlbaum Associates.

• Levin, J. R., & O'Donnell, A. M. (1999). What to do about educational research's credibility gaps? Issues in Education, 5(2), 177-229.

• Linn, M. C. (1987). Establishing a research base for science education: Challenges, trends, and recommendations. Journal of Research in Science Teaching, 24(3), 191-216.

• Linn, M. C. (2000). Designing the knowledge integration environment. International Journal of Science Education, 22(8), 781-79 (example study)

• Reeves, Thomas C. (2000). Enhancing the Worth of Instructional Technology Research through Design Experiments and Other Development Research Strategies, Paper presented on April 27, 2000 at Session 41.29, International Perspectives on Instructional Technology Research for the 21st Century, a Symposium sponsored by SIG/Instructional Technology at the Annual Meeting of the American Educational Research Association, New Orleans, LA, USA. PDF.

• Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86-109). London: Routledge. (A prior version of this text is above, i.e. Reeves, 2000)

• Reiser, B. J., Tabak, I., Sandoval, W. A., Smith, B. K., Steinmuller, F., & Leone, A. J. (2001). BGuILE: Strategic and conceptual scaffolds for scientific inquiry in biology classrooms. In S. M. Carver & D. Klahr (Eds.), Cognition and instruction: Twenty-five years of progress (pp. 263-305). Mahwah, NJ: Lawrence Erlbaum. (example study).

• Reymen M. M. J., D. K. Hammer, P. A. Kroes, J. E. van Aken, C. H. Dorst, M. F. T. Bax and T. Basten (2006), A domain-independent descriptive design model and its application to structured reflection on design processes, Research in Engineering Design, 16 (4), 147-173. Abstract PDF/HTML (Access restricted) (This is also a good overview article)

• Sandoval, William A. & Philip Bell (2004), Design-Based Research Methods for Studying Learning in Context: Introduction, Educational Psychologist, Vol. 39, No. 4: pages 199-201. doi:10.1207/s15326985ep3904_3

• Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39(4), 213-223.

• Sandoval, W. A., & Bell, P. (Eds.). (2004). Design-based research methods for studying learning in context. [Special Issue] Educational Psychologist, 39(4).

• Sandoval, William A. (2004a). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Preprint. To appear in Educational Psychologist, Vol. 39, No. 4, Pages 213-223.

• Sandoval, William A. (2004). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs, Educational Psychologist, Vol. 39, No. 4, Pages 213-223.

• Shavelson, R. J., Phillips, D. C., Towne, L., & Feuer, M. J. (2003). On the science of educational design studies. Educational Researcher, 32(1), 25-28.

• Steffe, L. P., & Thompson, P., W. (2000). Teaching experiment methodology: underlying principles and essential elements. In A. E. Kelly & R. A. Lesh (Eds.), Handbook of research design in mathematics and science education (pp. 267-306). Mahwah, NJ: Lawrence Erlbaum Assoc.

• Tabak, I. (2004). Reconstructing context: negotiating the tension between exogenous and endogenous educational design. Educational Psychologist, 39(4), 225-233.

• University of California at Berkeley, Field Guide to Design Experiments in Education, [2]

• Van den Akker Jan, Koeno Gravemeijer, Susan McKenney, Nienke Nieveen (2006). Educational Design Research, Routledge, ISBN 9780415396349

• White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition & Instruction, 16(1), 3-118. (example)

• Zitter, Ilya (2006), Design of competency-based, ICT-supported learning environments in higher education: The role of artefacts, ICO Toogdag research meeting PDF

Influences from other fields

• Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30(2), 116-127.

• Erickson, F. (1982). Classroom discourse as improvisation: relationships between academic task structure and social participation structures in lessons. In L. C. Wilkinson (Ed.), Communicating in the classroom (pp. 153-181). New York: Academic Press.

• Greeno, J. G. (1998). The situativity of knowing, learning, and research. American Psychologist, 53(1), 5-26.

• Koschmann, T., Kelson, A. C., Feltovich, P. J., & Barrows, H. S. (1996). Computer-supported problem-based learning: a principled approach to the use of computers in collaborative learning. In T. Koschmann (Ed.), CSCL: theory and practice of an emerging paradigm (pp. 83-124). Hillsdale, NJ: Erlbaum.

• Winn, W. (2003). Research methods and types of evidence for research in educational technology. Educational Psychology Review, 15(4), 367-373.