Docimo

Introduction

"DOCIMO" is an online platform to create online evaluations developed by the spin off Assess Group, a spin off from a Liège university centre specialized in evaluation engineering. The system has been developed and tested with several partners from industry and educational institutions.

The system can be used in English settings.

Creating tests

Gilles distinguishes four evaluation levels: (1) Satisfaction, (2) Learning, (3) Behavioral change and (4) Institutional change. Docimo targets level 2 and 3.

Design method

Gilles (2002,2010) create a model for creating standardized assessments, called Construction Cycle and Quality Controls for Standardized Testing (CCQCST). According to Gilles (2017), “The CCQCST model includes 8 steps: (1) Analysis of the objectives to be evaluated; (2) Design / test plan; (3) Questions / items elaboration; (4) Training / Information of the trainees to be evaluated; (5) Testing / assessment; (6) correction and analysis of results; (7) feedbacks and (8) cycle regulation for continuous improvement of subsequent evaluations. The CCQCST cycle, as shown above, provides quality control procedures to ensure quality assessments.”

Validity and reliability concerns

• Construct validity: A testing score allows measuring learning gains for a given cognitive ability, using an accepted theoretical construct, e.g. Krathwohl's taxonomy
• Content validity: Allows measuring contents taught in the class.
• Reliability: Coherence of response items with overall testing score. See also: inter-judge and intra-judge agreement.
• Sensibility: Measures should be precise but take into account certitudes (confidence level) of the learner
• Communicability: Learners should understand the principle behind the evaluation, e.g. understand what cognitive level(s) will be measured (e.g. recall, understand, analyse, evaluate)

To calibrate questions and response items, Docimo allows computing Rbis and CMoy coefficients for answers.

Confidence level

In order to prevent random answers, users must enter confidence degrees. According to the selected degree, positive and negative points change.

Docimo test item where the users selects a confidence level for a selected answer

Links

• http://www.assess-group.be (old domain)
• https://itinera-group.com/docimo/ (main site)

• The GIRIEF is integrated by a group of researchers, teachers, technologists in areas related to education, psychology, computer science with experts in statistical test analysis, and teaching and learning models.

Additional resources (In french):

• Degrés de certitude
• Le réalisme avec degrés de certitude

Bibliography

Gilles, J.-L., & Tinnirello, S. (2017). DOCIMO: an Online Platform Dedicated to the Construction and Quality Management of Learning and Impact Assessments in the Digital Age. Poster presented at the The #dariahTeach Open Resources Conference Lausanne, Suisse. Retrieved from http://hdl.handle.net/20.500.12162/107

Gilles, J.-L., Henzi, P., et Tinnirello, S. (2018). Instrumentation du cycle de construction et de gestion qualité des évaluations standardisées : apports de la plateforme en ligne DOCIMO. Dans M. Milmeister et C. Weis (Eds.), Colloque 2018 de l'ADMEE-Europe : L’évaluation en éducation et en formation face aux transformations des sociétés contemporaines - Livret des résumés (pp. 29-30). Esch-sur-Alzette, Luxembourg: Université du Luxembourg. http://hdl.handle.net/20.500.12162/106

Gilles, J.-L. (2010). Qualité spectrale des tests standardisés universitaires. Sarrebruck, Allemagne: Editions universitaires européennes. http://hdl.handle.net/20.500.12162/238

Jean‐Luc Gilles, Pascal Detroz & Jean‐Guy Blais (2011) An international online survey of the practices and perceptions of higher education professors with respect to the assessment of learning in the classroom, Assessment & Evaluation in Higher Education, 36:6, 719-733, DOI: 10.1080/02602938.2010.484880