View source for PISA

{{Stub}}
== Definition ==

PISA = Programme for International Student Assessment

{{quotation | PISA assesses how far students near the end of compulsory education have acquired some of the knowledge and skills that are essential for full participation in society. In all cycles, the domains of reading, mathematical and scientific literacy are covered not merely in terms of mastery of the school curriculum, but in terms of important knowledge and skills needed in adult life. [..] n the PISA 2003 cycle, an additional domain of problem solving was introduced to continue the examination of cross-curriculum competencies.}} ([http://www.pisa.oecd.org/pages/0,2966,en_32252351_32235918_1_1_1_1_1,00.html], retrieved 18:31, 30 September 2006 (MEST)).

PISA is often quoted in discussions about educational policy. Most people do not really know what is measured and just quote rankings.

=== The achievement studies ===

PISA 2000 achievement studies usually included three modules and that measured applicable skills in three areas: {{quotation|Are students well prepared to meet the challenges of the future? Are they able to analyse, reason and communicate their ideas effectively? Do they have the capacity to continue learning throughout life?}} (Adams & Vu, 2002: Introduction)
* Mathematics: 32 items, representing approximately 60 minutes of testing time
* Sciences: 35 items, representing approximately 60 minutes of testing time for each.
* Reading: 141 items representing approximately 270 minutes of testing time for each.

{{quotation|PISA 2000 was a paper-and-pencil test, with each student undertaking two hours of testing (i.e., answering one of the nine booklets). Pencils, erasers, rulers, and, in some cases, calculators, were provided. [...] The 141 main study reading items were organised into nine separate clusters, each with an estimated administration time of 30 minutes. The 32 mathematics items and the 35 science items were organised into four 15-minute mathematics clusters and four 15-minute science clusters respectively. These clusters were then combined in various groupings to produce nine linked two hour test booklets.}} (Adams & Vu, 2002: 23).

These booklets (also called '''cognitive booklets''' (as opposed to the questionnaires) tested different modules. {{quotation|Reading items occur in all nine booklets, and there are linkages between the reading in all booklets. This permits all sampled students to be assigned reading scores on common scales. Mathematics items occur in five of the nine booklets, and there are links between the five booklets, allowing mathematics scores to be reported on a common scale for five-ninths of the sampled students. Similarly, science material occurs in five linked booklets, allowing science scores to be reported on a common scale for five-ninths of the sampled students.}} (Adams & Vu, 2002: 23).

PISA 2003 added:
* Problem solving (cross-curicular competencies)

Options:
* Countries can choose and add their own tests and surveys. E.g. in Switzerland there was a survey of 9th graders (in addition to 15 year olds).

=== The survey questionnaires ===

In addition to tests, PISA also administers questionnaires to both individual learners and schools. Datasets that combine questionnaire results and test results are available. {{quotation|A Student and a School Questionnaire were used in PISA 2000 to collect data that could be used in constructing indicators pointing to social, cultural, economic and educational factors that are thought to influence, or to be associated with, student achievement.}} (Adams & Vu, 2002: 33)

The student questionnaire included items for:
* Basic demographics
* Family background and measures of socioeconomic status
* Student description of school/instructional processes
* Student attitudes towards reading and reading habits
* Student access to educational resources outside school
* Institutional patterns of participation and programme orientation
* Student career and educational expectations

The school questionnaire included:
* Basic school characteristics
* School policies and practices
* School climate
* School resources

Optional:
* The Information Technology or Computer Familiarity Instrument ([[computer literacy]])

== Mathematics skills ==

=== Design of the tests ===

According to quotations from PISA (2003):

{{quotationbox | 
In total, 85 mathematics items were used in PISA 2003. These tasks, and also those in reading, science and problem solving, were arranged into half-hour clusters. Each student was given a test booklet with four clusters of items - resulting in two hours of individual assessment time.

These clusters were rotated
in combinations that ensured that each mathematics item appeared in the same
number of test booklets, and that each cluster appeared in each of the four
possible positions in the booklets.

Such a design makes it possible to construct a scale of mathematical performance,
to associate each assessment item with a point score on this scale according to its
difficulty and to assign each student a point score on the same scale representing
his or her estimated ability.

The relative ability of students taking a particular test can be estimated by
considering the proportion of test items they answer correctly. The relative
difficulty of items in a test can be estimated by considering the proportion
of test takers getting each item correct.

Once the difficulty of individual items was given a rating on the scale, student
performance could be described by giving each student a score according to
Each student was given a
subset from a broad pool
of mathematics tasks...
...and their performance
was established on
a scale...
A Profile of Student Performance in Mathematics
the hardest task that they could be predicted to perform.
}}

=== The six proficiency levels of the PISA 2003 mathematical skills ===

PISA (2003:47):

{{quotationbox |
At Level 6, students can conceptualise, generalise, and utilise information based on their
investigations and modelling of complex problem situations. They can link different information
sources and representations and flexibly translate among them. Students at this level are capable
of advanced mathematical thinking and reasoning. These students can apply this insight and
understanding, along with a mastery of symbolic and formal mathematical operations and
relationships, to develop new approaches and strategies for attacking novel situations. Students at
this level can formulate and precisely communicate their actions and reflections regarding their
findings, interpretations, arguments, and the appropriateness of these to the original situations.

At Level 5, students can develop and work with models for complex situations, identifying
constraints and specifying assumptions. They can select, compare, and evaluate appropriate
problem-solving strategies for dealing with complex problems related to these models.
Students at this level can work strategically using broad, well-developed thinking and
reasoning skills, appropriately linked representations, symbolic and formal characterisations,
and insight pertaining to these situations. They can reflect on their actions and can formulate
and communicate their interpretations and reasoning.

At Level 4, students can work effectively with explicit models for complex concrete situations
that may involve constraints or call for making assumptions. They can select and integrate
different representations, including symbolic ones, linking them directly to aspects of realworld
situations. Students at this level can utilise well-developed skills and reason flexibly,
with some insight, in these contexts. They can construct and communicate explanations and
arguments based on their interpretations, arguments and actions.

At Level 3, students can execute clearly described procedures, including those that require
sequential decisions. They can select and apply simple problem-solving strategies. Students
at this level can interpret and use representations based on different information sources
and reason directly from them. They can develop short communications reporting their
interpretations, results and reasoning.

At Level 2, students can interpret and recognise situations in contexts that require no more
than direct inference. They can extract relevant information from a single source and make
use of a single representational mode. Students at this level can employ basic algorithms,
formulae, procedures or conventions. They are capable of direct reasoning and making literal
interpretations of the results.

At Level 1, students can answer questions involving familiar contexts where all relevant
information is present and the questions are clearly defined. They are able to identify information
and to carry out routine procedures according to direct instructions in explicit situations. They
can perform actions that are obvious and follow immediately from the given stimuli.
}}

See also [[learning level]]

== The six proficiency levels of the PISA 2006 science skills ==

According to the PISA 2006 technical manual (OECD 2009), the six proficiency levels on the science scale are the following:
{{quotationbox|
At Level 6 (1.3%), students can consistently identify, explain and apply scientific
knowledge and knowledge about science in a variety of complex life
situations. They can link different information sources and explanations
and use evidence from those sources to justify decisions. They clearly and
consistently demonstrate advanced scientific thinking and reasoning, and
they demonstrate willingness to use their scientific understanding in support
of solutions to unfamiliar scientific and technological situations. Students at
this level can use scientific knowledge and develop arguments in support
of recommendations and decisions that centre on personal, social or global
situations.

At Level 5 (9%), students can identify the scientific components of many
complex life situations, apply both scientific concepts and knowledge
about science to these situations, and can compare, select and evaluate
appropriate scientific evidence for responding to life situations. Students
at this level can use well-developed inquiry abilities, link knowledge
appropriately and bring critical insights to situations. They can construct
explanations based on evidence and arguments based on their critical
analysis.

At Level 4 (29.3%), students can work effectively with situations and issues that may
involve explicit phenomena requiring them to make inferences about the role
of science or technology. They can select and integrate explanations from
different disciplines of science or technology and link those explanations
directly to aspects of life situations. Students at this level can reflect on their
actions and they can communicate decisions using scientific knowledge and
evidence.

At Level 3 (56.7%), students can identify clearly described scientific issues in a range
of contexts. They can select facts and knowledge to explain phenomena
and apply simple models or inquiry strategies. Students at this level can
interpret and use scientific concepts from different disciplines and can
apply them directly. They can develop short statements using facts and
make decisions based on scientific knowledge.

At Level 2 (80.8%), students have adequate scientific knowledge to provide
possible explanations in familiar contexts or draw conclusions based on
simple investigations. They are capable of direct reasoning and making
literal interpretations of the results of scientific inquiry or technological
problem solving.

At Level 1 (94.8%), students have such a limited scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific
explanations that are obvious and that follow explicitly from given
evidence.
}}

These levels can then be described at a further operational level with other scales, e.g. the summary descriptions of the six proficiency levels for '''using science''' (OECD 2009: 300) defines level 5 perfomance with these terms:
{{quotationbox|
'''General proficiencies students should have''': Students at this level are able to interpret data from related datasets presented in various formats. They can identify and explain differences and similarities in the datasets and draw conclusions based on the combined evidence presented in those datasets.

'''Tasks a student should be able to do''': 
* Compare and discuss the characteristics of different datasets graphed on the one set of axes.
* Recognise and discuss relationships between datasets (graphical and otherwise) in which the measured variable differs.
* Based on an analysis of the sufficiency of the data, make judgements about the validity of conclusions.

'''Test item example''': GREENHOUSE Question 4
}}

== Setting up SPSS files ==

=== PISA 2012 ===

You can download the data sets from [http://pisa2012.acer.edu.au/downloads.php Database - PISA 2012]. You then can use SPSS or SAS control files to get nice ready-to-go data files.

; Dealing with the student questionnaire (tested on Jan 2014 with SPSS 22)

* Download [http://pisa2012.acer.edu.au/downloads/INT_STU12_DEC03.zip Student questionnaire data file] (about 230 MB compressed)
* Uncompress. This will give a 1.17GB INT_STU12_DEC03.txt file (or similar). We suggest to uncompress to a simple file location on Windows (e.g. c:\pisa\INT_STU12_DEC03.txt). Do not copy to "My Documents". On Linux, it doesn't matter since you probably know how to use a computer ...
* Download [http://pisa2012.acer.edu.au/downloads/INT_STU12_SPSS.sps SPSS syntax to read in student questionnaire data file]
* Open the *.SPS file in SPSS. It should open in the Syntax Editor
* Fix the second line (specify the location of the *.txt file, e.g. c:\pisa\INT_STU12_DEC03.txt) !!!
* Since the data uses a English <code>.</code> separator, you may have to keep the first line, but I had to remove it when I used the french speaking interface of SPSS. I finally switched SPSS to English, something I do anyhow after installing a new version.
* Run the SPS file. Make sure to run '''all the commands to the bitter end''' or you only will import partial information. On a decent computer this won't last for very long, max. a few minutes on a multi-core system with enough memory and an SSD card.
 Click the Green button
: If SPSS seems to be happy, do the following:
 Menu: Run -> To End
: If SPSS is unhappy, then just try to get the first two lines right. E.g. if you can't switch to English, learn how to tell SPSS how to use local ways (e.g. search for "SPSS set decimal dot PISA")
 Menu File -> Save

To do country-specific analysis, we suggest to create new data files, e.g. just consider splitting the file (see the ''data'' menu) with the first variable:
 Menu: Data -> Split into Files
 Add Country Code
 Define an output directory (optional)
 Click ok
: The file names will use: ISO-name.sav, e.g. CHE.sav (but you can change that)

... Ready to go :)

== Links ==

=== General ===

* [http://www.pisa.oecd.org/ OECD PISA .org Home page] (stable URL)
** From here you can download some free reports and some (huge) datasets. Manuals (tests, surveys, etc.) and data sets are quite difficult to find. Click on "What PISA produces". the year and then "Manuals" etc...
** [http://www.oecd.org/pages/0,3417,en_32252351_32235731_1_1_1_1_1,00.html OECD Pisa Home Page] (direct link)
** [http://www.pisa.oecd.org/findDocument/0,3354,en_32252351_32235731_1_119669_1_1_1,00.html PISA 2006 manuals]
* [http://www.oecdbookshop.org/ http://www.oecdbookshop.org]

; Australian websites - have additional stuff like sample questions or a mirror for the datasets
* [http://pisa-sq.acer.edu.au/ PISA Sample Questions]
* [http://pisaweb.acer.edu.au/oecd/oecd_pisa_data.html http://pisaweb.acer.edu.au] (PISA 2000 Data Archive)
* [http://pisaweb.acer.edu.au/oecd_2003/oecd_pisa_data.html http://pisaweb.acer.edu.au] (PISA 2003 Data Archive)
* [http://pisa2012.acer.edu.au/downloads.php PISA 2012 Data]
* [https://mypisa.acer.edu.au/index.php?option=com_content&task=view&id=69&Itemid=445 PISA Instruments]

=== PISA 2012 ===

; PISA 2012 Assessment and Analytical Framework
* [http://www.oecd.org/pisa/pisaproducts/PISA%202012%20framework%20e-book_final.pdf ISA 2012 Assessment and Analytical Framework] Glance at this first !

; PISA 2012 test items
* [https://mypisa.acer.edu.au/images/mypisadoc/pisa2012_relitems_maths_eng1.pdf Released PISA Items - Maths - PISA 2012]

; PISA 2012 Student questionnaires
: PISA 2012 included one main questionnaiare and several other ones. As of Jan 2014, we did not find any single documentation
* All the questionnaires are available from the [http://pisa2012.acer.edu.au/downloads.php Database - PISA 2012]] page. There are four variants of the student questionnaire. However, we found that [https://mypisa.acer.edu.au/images/mypisadoc/questionnaires/ms12_stq_eng.pdf Student Questionnaire] (main survey) found on mypisa is the best starting point. But you do need in addition the ICT and Educational career (EC) ones in addition.
* Watch out for the questions codes (in grey on top of each question)
* If you plan to conduct data analysis you also might download the code book (i.e. a dump of information that you will have in your SPSS/SAS file)

; PISA 2012 Student Questionnaires (alternative)
* [https://mypisa.acer.edu.au/images/mypisadoc/questionnaires/ms12_stq_eng.pdf Student Questionnaire] (probably the most complete one)
* [https://mypisa.acer.edu.au/images/mypisadoc/questionnaires/ms12_ict_eng.pdf ICT Familiarity component for Student Questionnaire]
* [https://mypisa.acer.edu.au/images/mypisadoc/questionnaires/ms12_ec_eng.pdf Educational Career component for Student Questionnaire]

; PISA 2012 Data
* These are available as compressed txt file plus SPSS/SAS Control files that you will have to run
* [http://pisa2012.acer.edu.au/downloads.php Database - PISA 2012]
* The questionnaire file includes about 500'000 entries. 

; PISA 2012 Other questionnaires
* [https://mypisa.acer.edu.au/images/mypisadoc/questionnaires/ms12_scq_eng.pdf School Questionnaire]
* [https://mypisa.acer.edu.au/images/mypisadoc/questionnaires/ms12_paq_eng.pdf Parent Questionnaire]

=== PISA 2015 ===

; DataSet for download (including SAS / SPSS files)

There are several separate datasets for:
* Student questionnaire
* School questionnaire
* Education career questionnaire for students (optional for countries)
* ICT familiarity questionnaire for students (optional for countries)
* Parents questionnaire (optional for countries)
* Teacher questionnaire (optional for countries)

Download page:
* http://www.oecd.org/pisa/data/2015database/

; Questionnaires
* The questionnaires are appended to [http://www.oecd-ilibrary.org/education/pisa-2015-assessment-and-analytical-framework_9789264255425-en PISA 2015 Assessment and Analytical Framework]
* Direct link to chapter: DOI 10.1787/9789264255425-8-en

=== Country specific ===

; Swiss pisa data
* [http://forscenter.ch/en/ FORS] Swiss foundation for research in social sciences. You can get some of the swiss datasets from here (but takes some time to click through).
* [http://www.pisa.admin.ch/ Swiss PISA website]
* [http://www.pisa.admin.ch/bfs/pisa/fr/index/05/04/03.html Swiss Public Use Sample PISA 2003] (used in the MALTT methodology class)

; Canada
* [http://www.statcan.gc.ca/pub/81-004-x/def/4068718-eng.htm Programme for International Student Assessment (PISA)]

== Bibliography ==

To do (there is a whole lot of literature).

*  PISA test items and school textbooks related to science: A textual comparison (2008). PISA test items and school textbooks related to science: A textual comparison, ''Science Education'', [http://www3.interscience.wiley.com/journal/117928236/abstract?CRETRY=1&SRETRY=0 Abstract]

* McGaw, B. (2002, October). Raising the bar and reducing failures: A possible dream. Invited paper given at the ACER conference “Providing world-class education: What can Australia learn from international achievement studies?”, Sydney.

* Martin Brunner, Cordula Artelt, Stefan Krauss, Jurgen Baumert, Coaching for the PISA test, Learning and Instruction, Volume 17, Issue 2, April 2007, Pages 111-122, ISSN 0959-4752, DOI:[http://dx.doi.org/10.1016/j.learninstruc.2007.01.002. http://dx.doi.org/10.1016/j.learninstruc.2007.01.002.]

* Raymond J. Adams, Margaret Wu, Programme for International Student Assessment, Organisation for Economic Co-operation and Development. OECD Publishing, 2002, ISBN 9264199519. This publication is also avaible as: PISA (2000) Technical Report ([http://www.pisa.oecd.org/dataoecd/53/19/33688233.pdf PDF/English version])

* PISA (2003), Learning for Tomorrow's World. [http://www.pisa.oecd.org/dataoecd/1/60/34002216.pdf PDF]

* Organisation for Economic Co-Operation and Development (2001). Knowledge and Skills for Life: First Results from PISA 2000. Paris: OECD Publications.

* Organisation for Economic Co-Operation and Development (2002a). Manual for the PISA 2000 Database. Paris OECD Publications.

* Organisation for Economic Co-Operation and Development (2002b). Sample Tasks from the PISA 2000 Assessment. Paris OECD Publications.

* OECED (Organisation for Economic Co-Operation and Development) (2009, PISA 2006 TECHNICAL REPORT. ISBN 978-92-64-04808-9
[[Category:organization and management]]
[[Category: Program, course and teacher evaluation]]
[[fr:PISA]]