The puzzling results of the PISA test in “digital reading performance”

The last week the OECD released its most updated report with the results of the students “digital reading performance”. This assessment was applied to 470 000 students in 2009, and taken by 26 million 15-year-olds in the schools of the 65 participating countries. Before reviewing some of the most relevant results, please take a look of this curious chart (VI.6.10a)

Figure VI.6.10a shows that, across OECD countries, “students who spend no time using a computer during school lessons perform the best, and the more time students spend using a computer during school lessons, the lower their scores in all three core subjects”.

Intensity of computer use in school lessons,
and digital reading performance [OECD average-15]

[Source: OECD PISA 2009 Database, Tables VI.6.8c-f.]

A possible explanations given by the OECD document suggests that the “computer use in school lessons have not effectively integrated ICT in a pedagogically meaningful way”. Whichever is the cause, the results are surprisingly unexpected. The relationship between students’ computer use at school and performance in digital reading tends to be negative with a slight curve. (Previously we provided details of this phenomenon in early assessments, see ppt in previous post). If that trend would be representative of many schools, the question that rises is: What are schools doing wrong? Probably that question has several answers.

Just a few months ago, Intellect Technology Association, warned to the Department for Education (DfE) in the UK: “The current system is too focused on teaching students how to use specific software packages and fails to allow for development of more advanced computer skills“. Intellect also highlighted the necessity to embrace a wider ICT skills and greater creativity, where interactive content and multimedia technology should be used across all lessons {Zdnet}. From our perspective Intellect is providing a very significant critique that should be analysed with more detail (not only in the UK).

Lets come back to the PISA study. Enclosed some of the most interesting results that worth a detailed review:

  • NAVIGATION: “After accounting for students’ performance in print reading, the relationship between digital reading performance and the frequency of browsing the Internet at home for schoolwork is close to linear” (p.189).”Navigation is a key component of digital reading, as readers “construct” their text through navigation. Thus, navigational choices directly influence what kind of text is eventually processed. Stronger readers tend to choose strategies that are suited to the demands of the individual tasks. Better readers tend to minimise their visits to irrelevant pages.” (p.20)
  • SEARCHING FOR INFORMATION: “The more frequently students search for information on line, the better their performance in digital reading. Being unfamiliar with online social practices, such as e-mailing and chatting, seems to be associated with low digital reading proficiency”
  • LACK OF INTEGRATION: “It is likely that the low level of ICT use at school indicates that ICT has not yet been fully integrated into pedagogical practices” (p.153)
  • MODERATE FREQUENCIES OF USE: “The relationship between the frequency of computer use at home for leisure and for schoolwork and digital reading performance is not linear, but rather mountain-shaped: in other words, moderate users attain higher scores in digital reading than both rare and intensive users”.
  • POSITIVE EFFECT OF ICT AT HOME: “The frequency of computer use at home for leisure is positively related to navigation skills, which is an essential and unique part of digital reading, while the frequency of computer use at school is not. These findings suggest that students are developing digital reading literacy mainly by using computers at home to pursue their interests […] it is important to encourage students to develop navigation skills and to foster self-confidence through using computers at home, while providing guidance on how to balance the amount of time students spend using computers with time for other activities.”
  • NEGATIVE EFFECT OF THE SCHOOL: “Computer use at school is not positively associated with digital reading performance […] access to computers at school is not the sole determinant of performance; students who use computers at school must also develop the knowledge and skills needed to locate and use the range of information available through the computer.”

Other thing that we found interesting was the following definition of high digital reading skills offered in that document:

Capability to evaluate information from several web-based sources, assessing the credibility and utility of what they read using criteria that they have generated themselves. Ability to work out a pathway across multiple sites to find information without explicit direction: that is  autonomous and efficiently. These two capabilities – critical evaluation and expertise in locating relevant information – are key skills in a medium in which there is virtually unlimited material available, and in which the integrity of the sources is often dubious.”

Source: PISA 2009 Results: Students On Line: Digital Technologies and Performance (Volume VI).

Noteworthy that Futurelab, short time ago also released a publication exploring the relationship between the use of ICT in the schools and at home. Enclosed some excerpts of this study developed by Lyndsay Grant (2010) Connecting digital literacy between home and school. Futurelab. Here some excerpts of that study (which provide a good link to the OECD report):


  • There were fewer instances of children’s home digital literacy practices coming into the school environment, with one child summing up: “we do different stuff in school like new documents”, emphasising the functional skills and more ‘work-focused’ technology use at school.
  • [At home] Students were thus developing some cultural and social understanding as well as critical reflection in the context of their own digital literacy practices.


  • “In some lessons at school, students were frustrated about not being able to suggest different or better ways of using technologies, saying they got told off. Students also said that some teachers did not welcome students’ out-of-school knowledge more generally into the classroom: “if you try to link it [out of school activities] with something in lessons, it’s always wrong and they’ve got to be always right.”
  • Bringing in out-of-school knowledge into the classroom can be seen as undermining teachers’ authority when it is framed as a question of who is ‘right’, or which knowledge is ‘legitimate’, but for other teachers it is simply a case of working with whatever students bring to a particular task.

After identifying the gap between the ICT practices at the school and at home, Grant states a question that we think is quite important: “How schools could foster particular skills and components of digital literacy?, rather than How they [teachers-students] could build connections between home and school digital literacy practices?“.

Some of these results bring us to fundamental questions of curriculum and how we place relative value on different forms of knowledge associated with different social and cultural groups. “It can seem impossible to completely bridge this gap, and it may not be desirable to completely erode the boundaries between knowledge and digital literacy practices at home and at school”.


My footnote:

Its quite evident that we are still (after decades of studies in this field) in a very early stage. We see that new literacies are adopted and developed in different (and sometimes unpredictable) ways. The PISA results show us that the impact of ICT in learning is demanding a broader strategy (in terms of skills, environments, outcomes and contexts to consider). Grant is suggesting not to force the connection and synergies between the use of digital technologies in the schools and at home. But he also proposed to: “looking in more depth at the complex and diverse reality of children’s digital literacy practices to better understand the skills, knowledge and understanding they are developing”. Something that links quite well with Intellect’s perspective to “embrace a wider ICT skills and greater creativity, where interactive content and multimedia technology should be used across all lessons”.