Wiki metrics, rubrics and collaboration tools

The educational technology and digital learning wiki
Jump to navigation Jump to search

Draft

<pageby nominor="false" comments="false"/>

Introduction

The main purpose of this article is to survey various strategies, methods and tools that allow to measure what participants (and students in particular) do in a wiki. Surveyed literature also will include topics that are only indirectly related to this area, but are of technical interest, e.g. the literature on trust metrics or wiki health.

In addition we will look at strategies, tools and tactics that will enhance wiki participation and collaboration. I.e. modern wikis such as Mediawikis allow to add simple collaboration visualizations or social networking that could enhance spontaneous collaboration. Of course, in education, it's the pedagogical scenario (the instructional design model) that will have the most impact and that will not discussed here.

Finally, we will try to outline a few paths for further development, i.e. we will try to formulate some desiderata for educational wiki software with respect to organization, orchestration, monitoring and assessment.

It will take some time before this piece will achieve good draft status - 10:51, 4 November 2011 (CET).

Contribution and collaboration metrics

Research and development about wiki collaboration and contributions has interested researchers from various fields and for various reasons. A lot of research was triggered directly or indirectly by the success of the WikiMedia projects that includes Wikpedia, Wikibooks and Wikiversity. Directly, because it's interesting how such large sites grow and indirectly because it's a huge database that researchers from many fields can play with. Examples of such research fields are visualization, data-mining, language processing and social networking Finally, the Mediawiki foundation itself sponsors internal research projects. Below we shall look at a few projects that are of interest to education and related fields.

Arazy et al. (2009:172) developed “a wiki attribution algorithm that: (a) calculates the authors’ contributions to each wiki page, (b) cannot be easily manipulated, (c) estimates the extent of a contribution using a sentence as the basic unit of meaning, and (d) distinguishes between contributions that persist on the page from those that are deleted”. More particularly, the algorithm is based mainly based on sentence ownership. Sentences between the current and previous release are compaired and considered to similar (i.e. the same) if the sentence in the current release is very similar to one in the previous release. From that principle, the authors can compute (a) the total amount of work and (b) the total contributions that persist, i.e. measure the relevancy of a sentence and therefore the quality of a user's contribution. In addition the number of internal and external links and word-level changes are measured.

Meishar-Tal and Tal-Elhasid describes the methodology developed in the Open University of Israel (OUI) to measure collaboration among students in wikis. They used three indicators: relative diversity (how many students in a group participate), interactivity and intensity.

  • Relative diversity, was calculated by the formula:
Relative Diversity = Diversity / No. of potential editors
E.g. “If a group of five students is able to work on a page but only four students are active and contributing, the level of collaboration would be 0.8”
  • Interactivity, was (simply) measured by:
Number of interactions among the participants, i.e. the number of edits
  • Intensity, is expressed by the following ratio.
Intensity= Interactivity / Diversity

Pedro Puente (2007) asked the question on how to measure student work within an experiential learning scenario, that is composed of the following stages: (1) "Concrete Experience", (2) "Reflective Observation", (3) "Abstract Conceptualization", and (4) "Feedback or Active Experimentation". Students were requested to choose explicitly from a list of contribution types:

caption List of contribution types (de Pedre Puente, 2007)
importance
(***=important)		 contribution type Descriptiong
Others (report) Other contribution type not listed in the menu at present (report which one to teachers)
* Organizational aspects Proposals and other questions related to the organization of the work team, scheduling,....
* Improvements in markup Improvements in markup, spelling, etc. (bear in mind that final document quality for printing (nicer tables, paginated table of contents, page markup...) will be performed at the end
* Support requests Simple questions, help requests, etc. without too much making of previous information
** Help partners Help group or course mates who asked questions, requested support, formulated doubts, etc. (group or course forum)
** New information New information has been added to text or discussion
*** New hypotheses New hypothesis has been prepared from preexisting information, and possibly, some new information (if so, mark option 'New information ' also)
*** Elaborated questions and new routes to advance Elaborated questions and new ways to move forward in the work which they were not taken into account previously (not just simple questions or elementary requests of support)
*** Synthesis / making of information Synthesize or refine speech with preexisting information

This was implemented as pulldown menu in a Tikiwiki.

Knowing both type and size of contributions provides an initial method of valuing each student's contribution, which the author labelled process (P) In addition, a final submitted text in word processing format, labelled final product (FP) was also graded. Both types of indicators then are then used in the following grading rubric.

caption Evaluation criteria, estimation methods and grading percentage.
Criteria Estimation method % of grade
Teamwork (P): Sum of sizes of all contribution types 15
Synthesis & clarity of information (P): Statement (corrected) of students, and sum of sizes of specific contribution type (*** Synthesis / making of information) 30
Quantity & quality of contributed information (a) (P): Statement (corrected) of students and sum of sizes of related contributions types (** New Information, *** New hypothesis and *** Synthesis / making of information")
(b) (FP): Revision by teachers of individual attribution statements of each section at final printed document		 40
Formal quality of work (a) (P): Statement of the students and specific contribution size and type (* Improvements of presentation")
(b) (FP): Arbitrary scoring by teachers to last work by editors-inchief to final document prior to printing		 15


Wiki health

Roth et al (2008), identified a simple list of indicators that allow to measure the viability of a wiki as whole.

  • rank - Daily rank of the wiki in terms of content growth
  • id - Internal identifier of the wiki
  • name - Full name of the wiki
  • total - Total number of pages
  • good - Number of real pages after discarding system pages
  • edits - Number of edits
  • views - Total number of views
  • admins - Number of users with administrator privileges
  • users - Total number of users
  • images - Number of images uploaded to the wiki
  • ratio - Ratio of good pages over total
  • type - Wiki engine
  • url - wiki root url

A study of over 360 wikis that had an initial population between 400 and 20000 users that both structural and governance factors have an impact on the growth rate. (to be completed ....)

Roth-Taraborelli-Gilbert Wiki Viability (2008)

Taraborelli et al (2009) then proposed a WikiTracer system that would allow to follow to measure the performance and growth of wiki-based communities with a standard plug-in system. It introduces over twenty indicators grouped into several categories: Wiki identification, population indicators, content indicators, governance indicators, access control and system information.

Social Text: Wiki Analytics asks the question whether there are “algorithmic ways of determining the health of a Wiki”. The following categories are suggested:

  • Page names - The better the names, the more likely people will link to those pages, both intentionally and accidentally. Goodness can be measured by "smaller is better", i.e. number of characters, words/tokens and number of non-alphabetic characters.
  • Link (Graph Analysis)- If no pages are linked to anything else, then every page is an island of one, and you are probably not using the Wiki in a useful way. This is measured by number of "single connections" (blocks), patterns of interconnectedness, longest diameter (path to reach another page) and number of incoming/outgoing links.
  • Page Content Analysis. Simple metrics would include page size, number of sections and word clouds.
  • Time Analysis - how content evolves -- how it is refactored (or not), how conflict is resolved, and in general, what the patterns of interaction look like
  • Tags (categories), i.e. another kind of incoming/outgoing link
  • Usage patters. Measured with edits/accesses, how many times orphan pages are assessed and edited, most wanted pages and numbers of editors.

See also Wiki metrics that provides more operational definitions of some indicators.

EduTechWiki does fairly well with respect to all of these, execpt the usage pattern ....

Wiki design

Built-in wiki collaboration and quality tools

“Based on a classification made Carley (2004), Wiki-specific networks can be arranged in four categories (Figure 1): social perspective (who knows who), knowledge perspective (who knows what), information perspective (what refers to what), and temporal perspective (what was done before). Relationships in one network usually imply relationships in another” (Müller and Meuthrath, 2007).

Some wikis include a reviewing process that allows authors to write drafts that only are published once it has been reviewed. A typical example is found in Wikimedia's Wikibooks. This is implemented with the Flagged Revisions extension.

Policies and guidelines

Wikipatterns.com, a toolbox of patterns & anti-patterns, lists a large variety of "people and anti-people patterns" and "adoption and and anti-adoption patterns.

Quality in a wiki can be determined by user evaluation (including self evaluation). In this Wiki, we use a simple templates like "stub" and "incomplete" to convey self-assessed quality statements to the reader.

“Wikipedia has developed several user-driven approaches for evaluating the articles. High quality articles can be marked as “Good Articles” or “Featured Articles” whereas poor quality articles can be marked as “Articles for Deletion”” (Wöhner and Peters, 2009). Features and processes are documented in Wikipedia's Good article reassessment and also Good article criteria

Software

Mediawiki extensions

  • The Collaboration diagram extension using the Graphviz extension dynamically creates graphs that show user contributions for an article, a list of articles or a category of articles. (Click on "authors" in this wiki to see how it works).

External analysis software

Below we only list software found in the wiki-related literature. See also:

  • BCFinder is a free open-source software for detecing communities in bipartite networks. The program is written in Java (platform independant).
  • SoniVis has the mission to create, as a community, a leading network analysis and network mining software that will run on all major platforms: SONIVIS:Tool.
  • Cognitive Computation Group (University of Illinois) created a series of natural language processing tools. Used by Arazy et al. (2009).
  • Wikitracer is a web service providing platform-independent analytics and comparative growth statistics for wikis.
  • Tinaweb A web application to explore online or off line bipartite graphs.
  • Tinasoft desktop A desktop application to text-mine corpora, build maps and explore the resulting networks.

Links

Conferences, organizations and journals

Conferencences
Organizations / Groups
Research projects

Policies and strategies

Metrics

Other good resources

  • snurblog run by Axel Bruns (Queensland University of Technology) includes some postings/news about wikis (among other things).

Bibliography

Collaboration and productivity metrics

  • Almeida, R.; B. Mozafari, and J. Cho. On the evolution of Wikipedia. In Proceedings of the International Conference on Weblogs and Social Media (ICWSM '07), Boulder, March 2007.
  • Arazy, Ofer and Eleni Stroulia, A Utility for Estimating the Relative Contributions of Wiki Authors, Proceedings of the Third International ICWSM Conference (2009). PDF
“we introduce an algorithm for assessing the contributions of wiki authors that is based on the notion of sentence ownership. The results of an empirical evaluation comparing the algorithm’s output to manual evaluations reveal the type of contributions captured by our algorithm.”
  • Buriol, L.S., Castillo, C., Donato , D., Leonardi , S., and Millozzi , S. (2006): Temporal Analysis of the Wikigraph. To appear in Proceedings of the Web Intelligence Conference (WI), Hong Kong 2006. Published by IEEE CS Press Abstract, PDF
  • K. M. Carley: “Dynamic Network Analysis”; In: R. Brelger K. Carley, P. Pattison, (eds.): “Dynamic Social Network Modeling and Analysis: Workshop Summary and Papers” National Academy Press, (2004), 133-145.
  • Ding, X., Danis, C., Erickson, T., & Kellogg, W. A. (2007). Visualizing an enterprise wiki. In Proceedings of ACM CHI '07, 2189-2194, San Jose.
  • Ehmann K., Large A., and Beheshti J., 2008, Collaboration in context, First Monday, 13:10, 6 October 2008.
  • Gabrilovich, Evgeniy; Markovitch, Shaul (2007): Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis. Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, January 2007. PDF reprint
  • Joachim Kimmerle, Johannes Moskaliuk, and Ulrike Cress, Understanding Learning - the Wiki Way, Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym '09). PDF
  • Hoisl, B., Aigner, W., & Miksch, S. (2006). Social rewarding in wiki systems - motivating the community. Lecture Notes in Computer Science , 4564, 362-371.
  • Jesus, Rut; Martin Schwartz and Sune Lehmann, Bipartite Networks of Wikipedia's Articles and Authors: a Meso-level Approach, Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym '09). PDF
“we use the articles in the categories (to depth three) of Physics and Philosophy and extract and focus on significant editors (at least 7 or 10 edits per each article). We construct a bipartite network, and from it, overlapping cliques of densely connected articles and editors. We cluster these densely connected cliques into larger modules to study examples of larger groups that display how volunteer editors flock around articles driven by interest, real-world controversies, or the result of coordination in WikiProjects.”
  • Jesus, Rut (2010). Cooperation and Cognition in Wikipedia Articles, A data-driven, philosophical and exploratory study, PhD thesis, Faculty Of Science University Of Copenhagen, PDF
E.g. read pages 61ff. for the methodology used
  • Harrer, Andreas; Johannes Moskaliuk, Joachim Kimmerle and Ulrike Cress(2008): Visualizing Wiki-Supported Knowledge Building: Co-Evolution of Individual and Collective Knowledge. Proceedings of the International Symposium on Wikis 2008 (Wikisym). New York, NY: ACM Press. PDF
  • Lehmann, S., M. Schwartz, and L. K. Hansen, “Biclique communities,” Physical Review E (2008), arxiv.org.
“We present a novel method for detecting communities in bipartite networks. Based on an extension of the k-clique community detection algorithm, we demonstrate how modular structure in bipartite networks presents itself as overlapping bicliques.”
  • Moskaliuk Johannes, Joachim Kimmerle and Ulrike Cress (2008): Learning and Knowledge Building with Wikis: The Impact of Incongruity between People’s Knowledge and a Wiki’s Information. In G. Kanselaar, V. Jonker, P.A. Kirschner, & F.J. Prins (Eds.), International Perspectives in the Learning Sciences: Cre8ing a learning world. Proceedings of the International Conference of the Learning Sciences 2008, vol. 2. (pp. 99-106). Utrecht, The Netherlands: International Society of the Learning Sciences, Inc. PDF reprint
  • Meishar-Tal, H. & Tal-Elhasid, E. (2008) Measuring collaboration in educational wikis - a methodological discussion, International Journal of Emerging Technologies in Learning, http://dx.doi.org/10.3991/ijet.v3i1.750
  • Müller Claudia and Benedikt Meuthrath, Analyzing Wiki-based Networks to Improve Knowledge Processes in Organizations (2007). Proceedings of I-KNOW ’07 Graz, Austria, September 5-7, 2007. PDF
“Four perspectives on Wiki networks are introduced to investigate all dynamic processes and their interrelationships in a Wiki information space. The Social Network Analysis (SNA) is used to uncover existing structures and temporal changes. [...] The collaboration network shows the nature of cooperation in a Wiki and reveals special roles, like the Wiki-Champion.”
  • Ortega, Felipe; Gonzalez-Barahona, Jesus M.; Robles, Gregorio (2007): The Top Ten Wikipedias: A quantitative analysis using WikiXRay. ICSOFT 2007 PDF
  • de Pedro Puente, Xavier, (2007) New method using Wikis and forums to assess individual contributions, WikiSym '07, PDF
  • Pfeil U., Zaphiris P., and Ang C.S., 2008, Cultural Differences in Collaborative Authoring of Wikipedia, J. of Computer-Mediated Communication, 12:1, pp. 88-113.
  • Roth, C., Taraborelli, D., Gilbert, N. (2008) Measuring wiki viability. An empirical assessment of the social dynamics of a large sample of wikis. Proceedings of the 2008 international symposium on Wikis, Porto, September 2008. (or Proceedings of the 4th International Symposium on Wikis - WikiSym 2008, New York, NY: ACM Press). PDF Draft Preprint - PDF - PDF (at Wikitracer).
  • Taraborelli, D,; Roth, C and N. Gilbert (2009). Measuring wiki viability (II). Towards a standard framework for tracking content-based online communities. PDF
  • Sabel, M. (2007). Structuring wiki revision history. Proceedings of the 2007 international symposium on wikis (pp. 125-130). Montréal: ACM.
  • Víegas, F., Wattenberg, M. and Dave, K. 2004. Studying cooperation and conflict between authors with history flow visualizations. In Proceedings of the SIGCHI Conf. on Human Factors in Computing Systems, 575–582, (April, 2004), Vienna, Austria.
  • Voss J. 2005. Measuring Wikipedia, In Proceedings International Conference of the International Society for Scientometrics and Informetrics ISSI : 10th, Stockholm (Sweden July 24-28, 2005). http://eprints.rclis.org/archive/00003610/
  • Wasserman, S; K. Faust: “Social network analysis: methods and applications“ Cambridge Univ. Press, Cambridge (1997)
  • D. Wilkinson and B. Huberman. Assessing the value of cooperation in Wikipedia. First Monday, 12(4), 2007.
  • V. Zlatic, M. Bozicevic, H. Stefancic, and M. Domazet. Wikipedias: Collaborative web-based encyclopedias as complex networks. Physical Review E, 74(1):016115, 2006.

Trust and quality, in particular metrics

  • Adler, B.T. and de Alfaro, L. 2007. A Content-Driven Reputation System for the Wikipedia. In Proceedings of the 16th International Conference on the World Wide Web. 261-270, (May, 2007), Banff, Canada.
  • Adler, B.T., Chatterjee, K., de Alfaro, L., Faella, M., Pye, I. and Raman, V. 2008. Assigning Trust To Wikipedia Content. In Proceedings of the 2008 International Symposium on Wikis. (September, 2008), Porto, Portugal.
  • Athenikos, Sofia J. and Xia Lin (2009), Visualizing Intellectual Connections among Philosophers Using the Hyperlink & Semantic Data from Wikipedia?, Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym '09). PDF
  • Blumenstock, J.E. 2008. Size Matters: Word Count as a Measure of Quality on Wikipedia. In Proceedings of the 17th international conference on World Wide Web. 1095-1096, (April, 2008). Beijing, China.
  • Dondio, P. and Barrett, S. 2007. Computational Trust in Web Content Quality: A Comparative Evalutation on the Wikipedia Project. In Informatica – An International Journal of Computing and Informatics, 31/2, 151-160.
  • Mark Kramer, Andy Gregorowicz, and Bala Iyer. 2008. Wiki trust metrics based on phrasal analysis. In Proceedings of the 4th International Symposium on Wikis (WikiSym '08). ACM, New York, NY, USA, , Article 24 , 10 pages. DOI=10.1145/1822258.1822291, PDF from wikisym.org
“Wiki users receive very little guidance on the trustworthiness of the information they find. It is difficult for them to determine how long the text in a page has existed, or who originally authored the text. It is also difficult to assess the reliability of authors contributing to a wiki page. In this paper, we create a set of trust indicators and metrics derived from phrasal analysis of the article revision history. These metrics include author attribution, author reputation, expertise ratings, article evolution, and text trustworthiness. We also propose a new technique for collecting and maintaining explicit article ratings across multiple revisions.”
  • Aniket Kittur, Bongwon Suh, and Ed H. Chi. 2008. Can you ever trust a wiki?: impacting perceived trustworthiness in wikipedia. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (CSCW '08). ACM, New York, NY, USA, 477-480. DOI=10.1145/1460563.1460639 http://doi.acm.org/10.1145/1460563.1460639, PDF from psu.edu
“Wikipedia has become one of the most important information resources on the Web by promoting peer collaboration and enabling virtually anyone to edit anything. However, this mutability also leads many to distrust it as a reliable source of information. Although there have been many attempts at developing metrics to help users judge the trustworthiness of content, it is unknown how much impact such measures can have on a system that is perceived as inherently unstable. Here we examine whether a visualization that exposes hidden article information can impact readers' perceptions of trustworthiness in a wiki environment. Our results suggest that surfacing information relevant to the stability of the article and the patterns of editor behavior can have a significant impact on users' trust across a variety of page types.”
  • Andrew Lih, Paper for the 5th International Symposium on Online Journalism (April 16-17, 2004) University of Texas at Austin, PDF CiteSeerX
“This study examines the growth of Wikipedia and analyzes the crucial technologies and community policies that have enabled the project to prosper. It also analyzes Wikipedia’s articles that have been cited in the news media, and establishes a set of metrics based on established encyclopedia taxonomies and analyzes the trends in Wikipedia being used as a source.”
  • Lim, E.P., Vuong, B.Q., Lauw, H.W. and Sun, A. 2006. Measuring Qualities of Articles Contributed by OnlineCommunities. In Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence. 81-87, (December, 2006), Hong Kong.
  • Thomas Wöhner and Ralf Peters, Assessing the Quality of Wikipedia Articles with Lifecycle Based Metrics, Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym '09).
“[...] quality assessment has been becoming a high active research field. In this paper we offer new metrics for an efficient quality measurement. The metrics are based on the lifecycles of low and high quality articles, which refer to the changes of the persistent and transient contributions throughout the entire life span.”
  • Stvilia, B., Twidale, M.B., Smith, L.C. and Gasser, L. 2005. Assessing information quality of a community-based encyclopedia. In Proceedings of the International Conference on Information Quality, 442–454, (November, 2005), Cambridge, USA

Various subjects

Such as embedded Wiki tools, wiki policies, passive use, community of users

  • Dekel, U, A Framework for Studying the Use of Wikis in Knowledge Work Using Client-Side Access Data, WikiSym '07, PDF
  • Hoisl, B., Aigner, W. and Miksch, S. 2007. Social Rewarding in Wiki Systems – Motivating the Community. In Proceedings of the second Online Communities and Social Computing. 362-371, (July, 2007), Beijing, China.
  • J. F. Ortega & J. G. Gonzalez-Barahona. Quantitative Analysis of the Wikipedia Community of Users, WikiSym '07, PDF
  • Sabel, M. 2007. Structuring wiki revision history. In Proceedings of the 2007 International Symposium on Wikis. 125-130. (October, 2007), Montreal, Canada.

Wiki pedagogics and analysis of collaboration in learning

This focuses on articles that are interested in assessment, rubrics, metrics and analytics. See also other articles in the Wikis category, e.g.

  • Bruns, A. and Humphreys, S. 2005. Wikis in Teaching and Assessment: The M/Cyclopedia Project, PDF (dead link)
  • Axel Bruns and Sal Humphreys (2007). Building Collaborative Capacities in Learners: The M/Cyclopedia Project, Revisited (WikiSym 2007) Abstract - Full paper (PDF)
  • MacDonald J. (2003) Assessing online collaborative learning: process and product, Computers & Education, Vol. 40, Issue 4, PP. 337-391
  • Swan, K. Shen, J. & Hiltz, R. 2006. Assessment and collaboration in online learning. Journal of Asynchronous Learning Networks, 10 (1), 45-62.
Retrieved from "https://edutechwiki.unige.ch/mediawiki/index.php?title=Wiki_metrics,_rubrics_and_collaboration_tools&oldid=35954"