Methodology tutorial - qualitative data analysis
This is part of the methodology tutorial (see its table of contents).
- Learning goals
- Understanding the purpose of code data and creating code books
- Learn about some descriptive analysis of variables techniques (including situations and roles)
- Learn about some causal analysis techniques
- Methodology tutorial - theory-finding research designs
- Methodology tutorial - qualitative data acquisition methods
- It aims at beginners. As with any method, you will have to practise and you likely will need some (human) guidance or at least serious additional reading. The purpose of this tutorial is to help you understand what qualitative data analysis could mean, not to present ready-made solutions.
- Slide style, should be expanded
In short, qualitative data analysis usually implies to related and iterative steps. See Methodology tutorial - theory-finding research designs for the general principle.
(1) Data needs to be coded and indexed so that you can find it for data analysis. More particularly:
- Information coding allows to identify variables and values, therefore allows for systematic analysis of data (and therefore reliability)
- ensures enhanced construction validity, i.e. that you look at things that will allow you to measure concept
(2) You then can do visualizations, matrices, grammars, etc.
(3) Interpret these
- Before we start - Keep your documents and ideas safe !
(1) Write memos (conservation of your thoughts). It is useful to write short memos (vignettes) when an interesting idea pops up, when you looked at something and want to remember your thoughts
(2) Create contact sheets that allow you remember your field work. After each contact (telephone, interviews, observations, etc.), make a short data sheet that should include:
- A clear tag for indexing purposes (filename or tag on paper), e.g. CONTACT_senteni_2005_3_25.doc.
- Type of contact, date, place, and a link to the interview notes, transcripts.
- Principal topics discussed and research variables addressed (or pointer to the interview sheet).
- Initial interpretative remarks, new speculations, things to discuss next time.
(3) Index your interview notes:
- Put your transcription (or audio/video files or audio tapes) in a safe place
- Assign a code to each "text", e.g. INT-1 or INTERVIEW_senteni_3_28-1. The same code could be used as filename.
- You also may insert the contact sheet (see above)
- Number the pages if you take notes manually (they can fall down ...)
(4) Do not trust your hard-disk !
2 Codes and categories
First step in qualitative data analysis is coding.
2.1 The coding principle
- A code is assigned to each (sub)category, i.e. theoretical variable you work on
- In other words: you must identify variable names
- In addition, you can for each code assign a set of possible values (e.g.: “positive”/”neutral/”negative)
- You then will systematically scan all your texts (documents, interview transcripts, dialogue captures, etc.) and tag all occurrences of variables.
- Three very different coding strategies exist as we shall see later:
- Code-book creation according to theory
- Coding by induction (according to “grounded theory”)
- Coding by ontological categories
- Benefit of coding
- Coding will allow you to find all informations regarding variables of interest to your research
- Reliability will be improved
- The procedure with a picture
2.2 Technical Aspects of coding
- The safest and most reliable way to code is to use some specialized software
- e.g. Atlas or Nvivo (NuDist),
- however, this takes a lot of time !
- For a smaller piece (e.g. a master thesis), we suggest to simply tag the text on paper
- you can make a reduced photocopy of the texts to gain some space in the margins
- overline or circle the text elements you can match to a variable
- make sure to distinguish between codes and other marks you may leave.
- Don’t use "flat" and long code-books, introduce hierarchy (according to dimensions identified)
- Each code should be short but also mnemonic (optimize)
- e.g. to code according to a schema “principal category” - “sub-category” (“value”) use a code like:
external_context -climate (positive)
- Don’t start coding before you have good idea on your coding strategy !
- either your code book is determined by you research questions and associated theories, frameworks, analysis grids
- or you really learn how to use an inductive strategy like "grounded theory".
2.3 Coding reliability
Assigning a code to a "text" segment is not always obvious and coding similar passages exactly the same way even less. In other words, we have a reliability problem.
There are two ways of improving reliability:
- Use clear and operational categories
- Use two or three coders (yourself and a friend) and compute intercoding index. If it is low, you will have to revise your coding scheme.
There exist several formula to compute intercoder (inter-rater) reliability. The most simple one is:
reliability = number of agreements (same coding) / total codes (agreements plus disagreements)
Read this for a very good introduction.
3 Code-book creation and management
3.1 Code-book creation according to theory
- analytical frameworks, analysis grids
- concepts found in the list of research questions and/or hypothesis
Example from an innovation study (about 100 codes):
|properties of the innovation||PI||
....(fill for your own code book).....
|support for the reform||CE-S|
|dynamics of the studied site||DS|
|external and internal assistance||EIA|
Before you start think about your own code book, you really should go through the relevant literature and try to find existing code books (that you then may adapt). E.g. below is an an example of codes use to analyze types of job-related problems of Turkish computer teachers (Deryakulu & Olkun, 2007).
- Role Conflict
- Inadequate Teacher Induction Policies
- Lack of Required Technological Infrastructure and Technical Support
- The Status of Computer Subject in School Curriculum
- Lack of Appreciation and Positive Feedback from Colleagues
- Unsupportive Administrators
- Rapidly Changing Nature of Content Knowledge in Computer Education
- Lack of Cohesive Computer Curriculum
- Insufficiency of Pre-Service Teacher Training Programs
- Large Class Sizes
- Indifferent Students
- Inadequate Supervision and Inspection
3.2 Coding by induction (according to “grounded theory”)
Grounded theory (Glaser, Strauss) refers to a set of approaches that that focus on interpretation and theory building, i.e. it is a fully inductive approach. The researcher starts by coding a small data set and then increases the sample in function of emerging theoretical questions. Categories (codes) can be revised at any time.
“Grounded theory begins with a research situation. Within that situation, your task as researcher is to understand what is happening there, and how the players manage their roles. You will mostly do this through observation, conversation and interview. After each bout of data collection you note down the key issues: this I have labelled "note-taking".
Constant comparison is the heart of the process. At first you compare interview (or other data) to interview (or other data). Theory emerges quickly. When it has begun to emerge you compare data to theory.
The results of this comparison are written in the margin of the note-taking as coding. Your task is to identify categories (roughly equivalent to themes or variables) and their properties (in effect their sub-categories). Grounded theory: a thumbnail sketch, retrieved 13:40, 15 October 2008 (UTC))”
Typically, you'd both code phenomena in isolation and relations (so-called axial coding). On starting point for axial coding could be these big abstract observation categories:
- conditions (causes of a perceived phenomenon)
- interactions between actors
- strategies and tactics used by actors
- consequences of actions
Read more in [Introduction to Grounded Theory].
To use this approach you really should document yourself, as beginner you'd likely fall into various traps, in particular selection and confirmation biases, i.e. you only look at things that will interest you for one or another reason.
3.3 Coding by ontological categories
Instead of initially creating a code book from variables found in your research questions or "inductive" coding à la grounded theory, you also start by creating or using a vocabulary for a given domain. This strategy is a compromise between “grounded theory” and “theory driven” approaches.
Below is a table that lists things you could observe in an organization (Bogdan and Biklen, cited by Miles & Huberman:1994 61)
|Context/Situation||information on the context|
|Definition of the situation||interpretation of the analyzed situation by people|
|Perspectives||global views of the situation|
|Ways to look at people and objects||detailed perceptions of certain elements|
|Processes||sequences of events, flow, transitions, turning points, etc.|
|Activities||structures of regular behaviors|
|Events||specific activities (non regular ones)|
|Strategies||ways of tackling a problem (strategies, methods, techniques)|
|Relations and social structure||informal links|
|Methods||comments (annotations) of the researcher|
In the literature, you may find several other such "accounting schemes". In educational technology, for example, there is a range of relatively simple code books for conversation and asynchronous discussion groups (forum) analysis. De Wever et al. (2006) provide a good overview. Some coding schemes are fairly simple. E.g. Cobos and Pifarré (2008) analyzed "collaborative knowledge construction in the web" with the following coding scheme:
|Explanation||Asks for clarifying some parts of the document||“The following link, which appears in your document, doesn't work now, but it worked a week ago”|
|Support||Express explicit agreement with the document's ideas or information organisation||“In my opinion this document is very useful and is easy to read”|
|Addition||Suggests additions to the document: ideas, opinions or information organisation||“I think that an index of the sections of the article should be added”|
|Delete||Suggests deletions from the document: ideas, opinions or information organisation||“In the summary there are some examples, which perhaps were not necessary”|
|Correction||Suggests changes to the document. They refer to ideas, opinions or information organisation||“I think that there is an error in the first paragraph in the conclusion section, where it says 'motor' it should be 'motivation'”|
Pena and Nichols (2004) used the following categories:
- Consensus Building
There exist more complex code books: As an example we produce a summary of students' messages code book by Eilon and Kliachko (2004).
|"Level A" Group||These categories indicate knowledge construction and a significant contribution to peer learning|
|Comprehension||Shows written evidence about the comprehension of the subjects studied by the following categories:|
|Reproducing- 1||Reproduces the main points, ideas, arguments or messages found in the incoming information with reference to its source and with critical evaluation.|
|Directing||Directs others to relevant sources for the subjects studied (both printed and online).|
|Clarifying by questions||Locates ambiguous, difficult or problematic areas in the new material. Describes the context of the question or the reason for asking it.|
|Clarifying by answers||Gives correct, relevant and comprehensive explanations as answers. Bases the answers on retrieved information while citing its origin.|
|Reflection||Shows written evidence of metacognitive processes that the learner applies when studying new subjects by the following categories:|
|Linking/extending||Links the new information with his or her own previous knowledge. Extends the new knowledge to other domains, especially to STS issues.|
|Critical evaluation||Evaluates the new information critically.|
|Transformation||Applies the new information in an original and creative way, draws inferences, and gives original examples.|
|"Level B" Group||These categories indicate a probable contribution to peer learning, by the following categories:|
|Documenting||Documents l earning exper iences or indivi dual contributions to the group.|
|Reproducing- 2||Reproduces the main points, ideas, arguments or messages found in the incoming information without any evaluation or original input to it.|
|Learning outcomes||Presents group and individual learning outcomes.|
|Technical questions/answers||Questions or remarks about any subject that does not relate directly to understanding the subjects studied.|
|Personal knowledge||Presents personal knowledge or daily life experiences.|
|"Level C" Group||These categories indicate no contr ibution (or adverse contribution) to peer learning, by the following categories:|
|Irrelevant/unexplained questions||Poses questions without giving the context or reason for asking them.|
|Casual quotations||Includes quotations without their context and without further explanation.|
|Irrelevant/incorrect answers||Offers Irrelevant or incorrect answers to questions sent by other students.|
|Emotional/personal comments||Includes personal comments, which should have been sent by e-mail as instructed by the teacher .|
3.4 Pattern codes
Some researchers also code patterns (relationships). Simple encoding (above) breaks data down to atoms, categories), whereas pattern coding identifies relationships between atoms. Pattern coding is also one the steps in the inductive grounded theory approach.
The ultimate goal is to detect (and code) regularities, but also variations and singularities.
Some suggested operations:
- Detection of co-presence between two values of two variables
- E.g. people in favor of a new technology (e.g. ICT in the classroom) have a tendency to use it.
- Detection of exceptions
- e.g. technology-friendly teachers who don’t use it in the classroom
- In this case you may introduce new variable to explain the exception, e.g. the attitude of the superior., of the group culture, the administration, etc.
- Exceptions also may provoke a change of analysis level (e.g. from individual to organization)
Attention: a co-presence does not prove causality
4 Descriptive matrices and graphics
In short: Analysis = visualization
There exist 2 popular types of analysis:
- A matrix is a tabulation engaging at least one variable, e.g.
- Tabulations of central variables by case (equivalent to simple descriptive statistics like histograms)
- Crosstabulations allowing to analyze how 2 variables interact
- Graphs ( networks ) allow to visualize links:
- temporal links between events
- causal links between several variables
- Some advice
- When you use these techniques always keep a link to the source (coded data)
- Try to fit each matrix or graph on a single page (or make sure that you can print things made by computer on a A3 page)
- you have to favor synthetic vision, but still preserve enough detail to make your artifact interpretable
- Consult specialized manuals e.g. Miles & Huberman, 1994 for recipes or get inspirations from qualitative research in the same domain as yours.
In this tutorial we can not cover all possible types of analysis, but just provide a few examples of what can be done. Before you start doing any sort of analysis, think about what you need to answer your research questions !
4.1 The “context chart”,Miles & Huberman (1994:102)
This technique allows to visualize relations and information flows between rôles and groups
- Example - Work flow for a "new pedagogies" program at some university
There exist codified "languages" for this type of analysis, e.g. UML or OSSAD
Once you clearly identified and clarified formal relations, you can use the graph to make annotations (like below)
4.2 Check-lists, Miles & Huberman (1994:105)
Check lists allow to make detailed summary for an analysis of an important variable.
Example: "external support is important for succeeding a reform project"
|Examples for external support||At counselor level||At teacher level|
|Analysis of deficiencies||Fill in each cell as below|
|Group dynamics||adequate: “we have met an organizer 3 times and it has helped us” (ENT-12:10)||not adequate: “we just have informed” (ENT-13:20)|
Such a table displays various dimensions of and important variable (external support). E.g. in the example above the values of the variable "external support" are listed in the left column
In the other columns we insert summarized facts as reported by different roles.
Review Question: Imagine how you would build such a grid to summarize teacher’s, student’s and assistant’s opinion about technical support for an e-learning platform
4.3 Chronological tables Miles & Huberman (1994:110)
- Can summarize a studied object’s most important events in time
Example: Task assignments for a blended project-oriented class
imposed tools (products)
Get familiar with the subject
links, wiki, blog
project ideas, Q&R
Students formulate project ideas
news engine, blog
Start project definition
Finish provisional research plan
Finish research plan
links, blog, annotation
Finish paper and product
Presentation of work
- This type of table is useful to identify important events.
- You can add other information, e.g. tools used in this example
4.4 Matrices for roles (function in an organization or program)
Miles & Huberman (1994:124)
- Crossing social roles with one or more variables
The abstract principle can be summarized as follows (see below for an example):
|roles||persons||variable 1||variable 2||variable 3|
|role 1||person 1||cells are filled in with values |
(pointing to the source)
|rôle 2||person 9|
|role n||person n|
- Crossing roles with roles
|role 1||...||role 3|
|role 1||fill in all sorts of informations about interactions|
Example: Evaluation of the implementation of a help desk software
|Actor||Evaluation||assistance provided||Assistance received||Immediate effects||Long term effects||Explanation of the researcher|
|Manager||-||-||-||demotivating||threatened the program||Felt threatened by new procedures|
|Consultant||+||help choosing the right soft. involved himself||-||contributed to the start of the experiment||-||....|
|“Help-desk worker”||+/-||debugging of machines, little help with software||better job satisfaction because of the tool||slight improvement of throughput||is still overloaded with work|
|Users||+/-||A few users provided help to peers with the tool||debugging of machines, little help with software||Were made aware of the high amount of unanswered questions||slight improvement of work performance||....|
Crossing between roles to visualize relations:
|role 1||trainers||role 3|
|trainers||“don’t coordinate very much” (1)||dosn’t receive all the information (2)|
5 Techniques to study relations
Often qualitative analysis stops with simple descriptive analysis (Daniel K. Schneider believes that this is the case with popular forum analysis). However, you also may use qualitative data to do some kind of "correlational analysis" as you typically do in quantitative data analysis.
5.1 Matrices ordered according to concepts (variables)
- Clusters (co-variances of variables, case typologies)
- An idea that certain values should "go together": Hunt co-occurrences in cells
- E.g.: “Can we observe a correlation between expressed needs for support and
expressed needs for training for a new collaborative platform (data from teachers’s interviews)?
|case||var 1||need for support||need for training||need for directives|
|case 2||not important||not important||not important|
|case 4||yyy||not important||not important||not important|
|case 6....||important||not important||not important|
- This table shows e.g. that need for support and need for training seem to go together,
e.g. cases 1,3,5 have association of "important", cases 2 and 4 have association of "not important".
- See next page how we can summarize this sort of information in a crosstab
- Co-variance expressed in a corresponding crosstab
|training needs * support needs||need for support|
We can observer a correlation here: "blue cells" (symmetry) is stronger than "magenta"!
You should check the data above to see if we did this right ...
5.2 Example typology with the same data:
We can observe emergence of 3 types to which we assign "labels"
Note: for more than 3 variables use a cluster analysis program
- Additional example
The table shows co-occurrence between values of 2 variables. The idea is to find out what effect different types of pressure have on ICT strategies adopted by a school.
|Strategies of a school|
|Type of pressure||strategy 1:no reaction||strategy 2:a task force is created||strategy 3:internal training programs are created||strategy 4:resources are reallocated||strategy 5: .....|
|Letters written by parents||(N=4)(p=0.8)||(N=1)(p=0.2)|
|Letters written by supervisory boards||(N=2)(p=0.4)||(N=3)(p=0.6)|
5.3 Recall: Interpretation of crosstabulation
See also: Methodology tutorial - quantitative data analysis (Cross-tabulation)
We would like to estimate the probability that a given value of the independent (explaining) variable entails a given value of the dependent (explained) variable.
- The procedure
- calculate the % for each value of the independent variable
- Note: this can be either the line or the column depending on how you orient your table
- compute the % in the other direction
|Variable y to explain = Strategies of action|
|Explaining variable x
||do nothing||send a mail||write a short tutorial||Total|
|Students making indirect suggestion||4 (80%)||1 (20%)||5 (100 %)|
|2 (40%)||3 (60%)||5 (100%)|
Interpretation: “... if students explicitly complain, the tutor will react more strongly and engage in more helpful activities.”
- See: quantitative data analysis.
6 Typology and causality graphs
There are no limits of what you can draw. Basically such analysis just use a more or less precise language to draw concept maps.
Below we just show two examples.
6.1 Typology graphs
- Display attributes of types in a tree-based manner
Example: Perception of a new program by different implementation agencies (e.g. schools) and its actors (e.g. teachers)
6.2 Dialog and message structures
Message structure can be rendered with "message maps" (e.g. Pena-Shaff and Nicholls, 2004).
Of course, you may use another language, such as UML activity diagrams to draw such maps.
6.3 Subjective causality graphs
A simple causality graphs relates variables (concepts) with directed arrows.
There exist many variants. One older method is “operational coding” (Axelrod, 1976) and is somewhat popular in political science. It allows to compute outcomes of reasoning chains
Example: Teacher talking about active pedagogies, ICT connections, Forums
Depending on your discipline of reference, you may be familiar with different software families that help drawing graphs:
- For concept mapping software, see Concept map
- For UML-supporting diagram software, see Unified modeling language
We also can recommend a good general purpose free diagram software:
Finally, for people who hate to draw, there exist useful free visualization software, in particular:
- Ground theory
- Bob Dick, Grounded theory: a thumbnail sketch
- Steve Borgatti, Introduction to Grounded Theory
- Glaser's Home page
- Grounded theory, Grounded theory (Glaser), Grounded theory (Strauss) (Wikipedia)
9.1 Introductions and Cookbooks
- Denzin, Norman K. and Yvonna S. Lincoln (eds) (). The SAGE Handbook of Qualitative Research, Sage Publications. ISBN 0761927573
- Dey, I. (1993). Qualitative Data Analysis. Routledge, London.
- Marshall, C. & Rossman, G. B. (1995) , Designing Qualitative Research, second edition, Sage, London
- Miles, M. B. & Huberman, A. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Sage. ISBN 0803955405 (This is still the cookbook for structured qualitative analysis.)
- Amanda Coffey, A., Holbrook,B. Atkinson, P., Qualitative Data Analysis: Technologies and Representations, School of Social and Administrative Studies, University of Wales, Cardiff HTML. (This is an overview of the literature without any practical advice)
- Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2), 77-101. Retrieved from http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa
- De Wever, B.; T. Schellens, M. Valcke and H. Van Keer. Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers & Education 46 (2006) 6–28 Abstract/PDF (Access restricted).
- Glaser, Barney G. (1992) Basics of grounded theory analysis: emergence vs forcing. Mill Valley, Ca.: Sociology Press.
- Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing. The Najadan Papers (pp. 117–136). London: Springer-Verlag.
- Lombard, Matthew; Jennifer Snyder-Duch and Cheryl Campanella Bracken (2008). Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects, Retrieved from http://www.temple.edu/sct/mmc/reliability/, accessed 20:59, 7 October 2008 (UTC).
- Strauss, Anselm, and Corbin, Juliet (1990) Basics of qualitative research: grounded theory procedures and techniques. Newbury Park: Sage.
- Cobos, Ruth and Manoli Pifarre, Collaborative knowledge construction in the web supported by the KnowCat system, Computers & Education, Volume 50, Issue 3, , April 2008, Pages 962-978. HTML/PDF (Access restricted).
- Deryakulu, Deniz & Olkun, Sinan (2007). Analysis of Computer Teachers’ Online Discussion Forum Messages about their Occupational Problems. Educational Technology & Society, 10 (4), 131-142. PDF
- Eilon, Batia and Kliachko, Sarah (2004). The Contribution of a Substance-Oriented Forum to the Study of Human Biology in Science Teacher Education, Journal of Technology and Teacher Education (2004) 12(1), 5-24.
- Lockhorst, D., Admiraal, W., Pilot, A., & Veen, W. (2003). Analysis of electronic communication using 5 different perspectives. Paper presented at ORD 2003 (in Heerlen).
- Pena-Shaff and C. Nicholls (2004). Analyzing student interactions and meaning construction in computer bulletin board discussions, Computers & Education 42 (2004), pp. 243–265. doi:10.1016/j.compedu.2003.08.003