View source for Docimo

== Introduction ==

"DOCIMO" is an online platform to create online and offline assessments, i.e. an [[assessment management system]] (AMS). It was developed by [http://www.assess-group.be Assess Group], a spin off from a Liège university centre specialized in evaluation engineering. The system has been developed and tested with several partners from industry and educational institutions.
{{stub}}
The system can be used in English settings.

See also:
* [[Learner assessment]]
* [[Assessment management system]]

== Creating tests ==

Gilles distinguishes four evaluation levels according to Kirkpatrick (1959): (1) Satisfaction, (2) Learning, (3) Behavioral change and (4) Institutional change. Docimo targets level 2 and 3. 

=== Design method ===

Gilles (2002,2010) create a model for creating standardized assessments, called Construction Cycle and Quality Controls for  Standardized  Testing (CCQCST).  According to Gilles (2017), {{quotation|The CCQCST model includes 8 steps:  (1)  Analysis  of  the  objectives  to  be  evaluated;  (2) Design / test plan; (3) Questions / items  elaboration;  (4)  Training  /  Information  of  the  trainees to be evaluated; (5) Testing /  assessment; (6) correction and analysis of  results;  (7)  feedbacks  and  (8)  cycle  regulation  for continuous improvement of subsequent  evaluations. The CCQCST cycle, as shown  above,  provides  quality  control  procedures  to  ensure  quality  assessments. }}

The methods is based on Deming's quality wheel: Plan (prepare an action), do (implement), check (if objectives are met), act (sustainable, corrective)

Majors steps using the software:
* Identify Points to evaluate (PE), i.e. a list of learning objectives to test.
* Define categories of performance (CP) for each PE
* Create a table of specifications (TDS), i.e. a matrix PE x CP with a cognitive level for each CP (e.g. recall, understand, apply, analyse, evaluate, create)
* Define Modality of Questioning (MQ) for each CP, i.e. a matrix PE x CP x MQ

=== Validity and reliability concerns ===

* Construct validity: A testing score allows measuring learning gains for a given cognitive ability, using an accepted theoretical construct, e.g. [[Learning level|Krathwohl's taxonomy]]
* Content validity: Allows measuring contents taught in the class.
* Reliability: Coherence of response items with overall testing score. See also: inter-judge and intra-judge agreement. 
* Sensibility: Measures should be precise but take into account certitudes (confidence level) of the learner
* Communicability: Learners should understand the principle behind the evaluation, e.g. understand what cognitive level(s) will be measured (e.g. recall, understand, analyse, evaluate)

To calibrate questions and response items, Docimo allows computing various coefficients, e.g. biserial correlation coefficient (R bis) and CMoy coefficients between correct answers for a question and correct answers total.

=== Confidence level ===

In order to prevent random answers, users must enter confidence degrees. According to the selected degree, positive and negative points change.

[[image:docimo-1.png|none|600px|thumb|Docimo test item where the users selects a confidence level for a selected answer]]

[[image:docimo-3.png|none|600px|thumb|Docimo - Testing results example]]

== Authoring ==

A few screenshots illustrate some of the functionality.

The authoring interface requires that tests are built in a more or less hierarchical manner

[[image:docimo-spec-1.png|none|600px|thumb|Docimo - define points to evaluate]]

[[image:docimo-spec-3.png|none|600px|thumb|Docimo]]

[[image:docimo-spec-4.png|none|600px|thumb|Docimo]]

[[image:docimo-spec-2.png|none|600px|thumb|Docimo - Table of specifications]]

== Links ==

* http://www.assess-group.be (old domain)
* https://itinera-group.com/docimo/ (main site)

* [http://servidor1.ieiaserver.com/w02/ The GIRIEF] is integrated by a group of researchers, teachers, technologists in areas related to education, psychology, computer science with experts in statistical test analysis, and teaching and learning models.

Additional resources (In french):

* [https://sites.google.com/site/dchepl/accueil Degrés de certitude]
* [https://www.erudit.org/en/journals/mee/2015-v38-n1-mee02527/1036553ar/ Le réalisme avec degrés de certitude]

== Bibliography ==

Dunberry, A., & Péchard, C. (2007). L’évaluation de la formation dans l’entreprise: état de la question et perspectives. UQAM/CIRDEP. https://www.cirdef.uqam.ca/wp-content/uploads/2015/03/CIRDEF-PSRA-Dunberry_EvaluationEntrepriseLitt2007.pdf

Gilles, J.-L., & Tinnirello, S. (2017). DOCIMO: an Online Platform Dedicated to the Construction and Quality Management of Learning and Impact Assessments in the Digital Age. Poster presented at the The #dariahTeach Open Resources Conference Lausanne, Suisse. Retrieved from http://hdl.handle.net/20.500.12162/107

Gilles, J.-L., Henzi, P., et Tinnirello, S. (2018). Instrumentation du cycle de construction et de gestion qualité des évaluations standardisées : apports de la plateforme en ligne DOCIMO. Dans M. Milmeister et C. Weis (Eds.), Colloque 2018 de l'ADMEE-Europe : L’évaluation en éducation et en formation face aux transformations des sociétés contemporaines - Livret des résumés (pp. 29-30). Esch-sur-Alzette, Luxembourg: Université du Luxembourg. http://hdl.handle.net/20.500.12162/106

Gilles, J.-L. (2010). Qualité spectrale des tests standardisés universitaires. Sarrebruck, Allemagne: Editions universitaires européennes. http://hdl.handle.net/20.500.12162/238

Jean‐Luc Gilles, Pascal Detroz & Jean‐Guy Blais (2011) An international online survey of the practices and perceptions of higher education professors with respect to the assessment of learning in the classroom, Assessment & Evaluation in Higher Education, 36:6, 719-733, DOI: 10.1080/02602938.2010.484880 https://srhe.tandfonline.com/doi/abs/10.1080/02602938.2010.484880#.XN06CaTgpnQ

Gilles, J.-L., et Lovinfosse, V. (2004). Utilisation du cycle SMART de gestion qualité des évaluations standardisées dans le contexte d’une Haute Ecole : regard critique en termes de validité, fidélité, sensibilité des mesures, diagnosticité, praticabilité, équité, communicabilité et authenticité. Dans S. Arzola Medina (Ed.), Proceedings of World Education for Educational Research (WAER). Santiago, Chile: Pontifica Universidad Catolica de Chile. http://hdl.handle.net/20.500.12162/890

Kirkpatrick, D.L. (1994). Evaluating training programs: the four levels, Berrett-Koehler, San Francisco (1994)

[[Category:Learner assessment and evaluation]]