Docimo

Introduction

"DOCIMO" is an online platform to create online and offline assessments, i.e. an assessment management system (AMS). It was developed by Assess Group, a spin off from a Liège university centre specialized in evaluation engineering. The system has been developed and tested with several partners from industry and educational institutions.

Draft

The system can be used in English settings.

See also:

• Learner assessment
• Assessment management system

Creating tests

Gilles distinguishes four evaluation levels according to Kirkpatrick (1959): (1) Satisfaction, (2) Learning, (3) Behavioral change and (4) Institutional change. Docimo targets level 2 and 3.

Design method

Gilles (2002,2010) create a model for creating standardized assessments, called Construction Cycle and Quality Controls for Standardized Testing (CCQCST). According to Gilles (2017), “The CCQCST model includes 8 steps: (1) Analysis of the objectives to be evaluated; (2) Design / test plan; (3) Questions / items elaboration; (4) Training / Information of the trainees to be evaluated; (5) Testing / assessment; (6) correction and analysis of results; (7) feedbacks and (8) cycle regulation for continuous improvement of subsequent evaluations. The CCQCST cycle, as shown above, provides quality control procedures to ensure quality assessments.”

The methods is based on Deming's quality wheel: Plan (prepare an action), do (implement), check (if objectives are met), act (sustainable, corrective)

Majors steps using the software:

• Identify Points to evaluate (PE), i.e. a list of learning objectives to test.
• Define categories of performance (CP) for each PE
• Create a table of specifications (TDS), i.e. a matrix PE x CP with a cognitive level for each CP (e.g. recall, understand, apply, analyse, evaluate, create)
• Define Modality of Questioning (MQ) for each CP, i.e. a matrix PE x CP x MQ

Validity and reliability concerns

• Construct validity: A testing score allows measuring learning gains for a given cognitive ability, using an accepted theoretical construct, e.g. Krathwohl's taxonomy
• Content validity: Allows measuring contents taught in the class.
• Reliability: Coherence of response items with overall testing score. See also: inter-judge and intra-judge agreement.
• Sensibility: Measures should be precise but take into account certitudes (confidence level) of the learner
• Communicability: Learners should understand the principle behind the evaluation, e.g. understand what cognitive level(s) will be measured (e.g. recall, understand, analyse, evaluate)

To calibrate questions and response items, Docimo allows computing various coefficients, e.g. biserial correlation coefficient (R bis) and CMoy coefficients between correct answers for a question and correct answers total.

Confidence level

In order to prevent random answers, users must enter confidence degrees. According to the selected degree, positive and negative points change.

Docimo test item where the users selects a confidence level for a selected answer

Docimo - Testing results example

Authoring

A few screenshots illustrate some of the functionality.

The authoring interface requires that tests are built in a more or less hierarchical manner

Docimo - define points to evaluate

Docimo

Docimo

Docimo - Table of specifications

Links

• http://www.assess-group.be (old domain)
• https://itinera-group.com/docimo/ (main site)

• The GIRIEF is integrated by a group of researchers, teachers, technologists in areas related to education, psychology, computer science with experts in statistical test analysis, and teaching and learning models.

Additional resources (In french):

• Degrés de certitude
• Le réalisme avec degrés de certitude

Bibliography

Dunberry, A., & Péchard, C. (2007). L’évaluation de la formation dans l’entreprise: état de la question et perspectives. UQAM/CIRDEP. https://www.cirdef.uqam.ca/wp-content/uploads/2015/03/CIRDEF-PSRA-Dunberry_EvaluationEntrepriseLitt2007.pdf

Gilles, J.-L., & Tinnirello, S. (2017). DOCIMO: an Online Platform Dedicated to the Construction and Quality Management of Learning and Impact Assessments in the Digital Age. Poster presented at the The #dariahTeach Open Resources Conference Lausanne, Suisse. Retrieved from http://hdl.handle.net/20.500.12162/107

Gilles, J.-L., Henzi, P., et Tinnirello, S. (2018). Instrumentation du cycle de construction et de gestion qualité des évaluations standardisées : apports de la plateforme en ligne DOCIMO. Dans M. Milmeister et C. Weis (Eds.), Colloque 2018 de l'ADMEE-Europe : L’évaluation en éducation et en formation face aux transformations des sociétés contemporaines - Livret des résumés (pp. 29-30). Esch-sur-Alzette, Luxembourg: Université du Luxembourg. http://hdl.handle.net/20.500.12162/106

Gilles, J.-L. (2010). Qualité spectrale des tests standardisés universitaires. Sarrebruck, Allemagne: Editions universitaires européennes. http://hdl.handle.net/20.500.12162/238

Jean‐Luc Gilles, Pascal Detroz & Jean‐Guy Blais (2011) An international online survey of the practices and perceptions of higher education professors with respect to the assessment of learning in the classroom, Assessment & Evaluation in Higher Education, 36:6, 719-733, DOI: 10.1080/02602938.2010.484880 https://srhe.tandfonline.com/doi/abs/10.1080/02602938.2010.484880#.XN06CaTgpnQ

Gilles, J.-L., et Lovinfosse, V. (2004). Utilisation du cycle SMART de gestion qualité des évaluations standardisées dans le contexte d’une Haute Ecole : regard critique en termes de validité, fidélité, sensibilité des mesures, diagnosticité, praticabilité, équité, communicabilité et authenticité. Dans S. Arzola Medina (Ed.), Proceedings of World Education for Educational Research (WAER). Santiago, Chile: Pontifica Universidad Catolica de Chile. http://hdl.handle.net/20.500.12162/890

Kirkpatrick, D.L. (1994). Evaluating training programs: the four levels, Berrett-Koehler, San Francisco (1994)