Vol. 13 No. 3, September, 2008 Contents | Author index | Subject index | Search | Home

Students' use of Web literacy skills and strategies: searching, reading and evaluating Web information

Introduction

The Internet is used extensively by children and teenagers, both at home and at school. At home, they communicate with friends through instant messaging, surf the Web to find information about the latest mobile phones and create their own Website or Weblog. At school, the Web has become a popular information resource and is increasingly used as an alternative to printed resources. Although many teachers acknowledge the Web's potential as an educational tool as well as the Web's motivating power, they also struggle to find meaningful applications of the Web in their educational practices. The Web is a learning tool that differs from other tools used in education because students acquire a great many Web skills in an out-of-school context and also regard themselves as skilled Web users. This is reflected in the perception of many teachers that their students' Web skills are superior to their own, which may result in underestimating the support students need when using the Web for educational purposes.

The Web has certain characteristics such as its size, topicality and accessibility, as well as the use of hypertext and non-textual elements, that are complicated for users and require specific skills. Many students use the Web quite naturally, but 'Too often, students - and adults, too - mistake their ability to move around the Internet for the skills that they need to navigate and read it' (Burke 2002: 38). This is confirmed by extensive research into children's Web behaviour, which shows children as lacking adequate search skills, as well as the necessary skills for critical evaluation of Web information. Although much research is based within librarianship and information science, with its tradition in the study of information-seeking behaviour (e.g., Bilal 2000, 2001, 2002; Fidel et al. 1999; Shenton & Dixon 2003), reading researchers (e.g., Coiro 2003) and educational researchers (e.g., Hoffman et al. 2003) have also focused on the Web as a new educational tool, requiring new skills and strategies from both teachers and students.

In our research we take an interdisciplinary approach in trying to combine the focuses of library/information science and educational researchers. Information science research points to the need for incorporating Web literacy, or, more broadly speaking, information literacy, into school curricula. It provides knowledge about the information searching process and suggestions for application in practice (e.g., Eisenberg & Berkowitz 1992; Kuhlthau 2004). Educational researchers primarily look for ways of using the Web as a meaningful learning tool for knowledge construction (e.g., Hoffman et al. 2003). Such use of the Web requires the mastery of certain strategies which in turn requires specific Web-related skills. In this article, we report on the adequacy and specific characteristics of students' use of Web skills and strategies after completion of a programme in which these skills and strategies were addressed.

In our study of the students' Web behaviour, we use Web literacy as an umbrella term, encompassing a range of interrelated skills. From the literature, we derive three major components of Web literacy skills: Web searching skills, Web reading skills and Web evaluating skills (see for example Coiro 2003; Enochsson 2005; Sutherland-Smith 2002; Shenton & Dixon 2003). The Web's size and topicality require good Web searching skills to find the information one is looking for (for example, knowledge of how to locate Web information and the ability to formulate relevant keywords). Web reading skills may be seen as a combination of traditional reading skills and new skills that are needed in the light of the Web's information overload and hypertext environment. Because most Web texts are not written for children, effective technical and comprehensive reading skills are required. The Web's use of hypertext and its multimodality require specific reading skills to be able to find one's way, in order to distinguish between potentially valuable and useless information and to identify the meaning of non-textual elements. Flexible use of scanning and close reading techniques are also examples of Web reading skills. Finally, the Web's open character and accessibility appeal to Web evaluating skills, i.e. the ability to critically assess the reliability and authority of Web information with a view to one's own information needs.

Empirical research into teaching Web skills and strategies is relatively scarce (Kuiper et al. 2005). Many studies are descriptive and small-scale and focus on students' search processes, that is, collecting information (e.g., Bilal 2000, 2001, 2002). Some studies describe students' own perception of their capability to use the Web (e.g., Enochsson 2005; Large & Beheshti 2000). More recent studies look more closely at differences in students' information seeking behavior (e.g., Heinstrom 2006). In studies with an educational orientation, more attention is paid to the classroom context and the incorporation of learning to use the Web within that context (e.g., Hoffman et al. 2003). However, in most of these studies the focus is on tools (e.g., a portal or interface) that help students to use the Web better for knowledge construction. Studies on teaching Web literacy in the classroom are scarce and mostly aimed at upper-grade and university students (e.g., Walton & Archer 2004). Although the mastery of Web literacy skills in itself does not lead to knowledge construction, they may be seen as preconditional: when students do not know how to use the Web in a critical way, knowledge cannot be obtained. However, at primary school level students are already expected to use the Web for papers and presentations. They also use the Web frequently at home.

In this article we attempt to provide additional insights into the teaching of Web literacy by studying 5th grade primary school students' use of the Web during a number of assignments. The data for this study were collected in the context of a broader research project. Two programmes were designed on teaching Web literacy skills in the context of a class project on healthy food, one programme being essentially sequential and pre-structured, the other focusing on collaborative inquiry activities in which the Web was used for students' own research questions. Both programmes paid attention to all three subcategories of Web literacy, that is, Web searching, Web reading and Web evaluating skills and strategies (see Appendix 1). Thus, in this article, the programmes may be seen as the context within which the data were collected. The learning results of both programmes showed a discrepancy between students' knowledge of Web literacy skills (determined with a questionnaire) and the actual Web behaviour they showed in assignments after the programmes. Students had gained knowledge of Web literacy skills but this knowledge was only partly reflected in their actual Web behaviour. This discrepancy occurred in both programmes and could not be related to either the programme itself or the way the teacher carried out the programme (Kuiper 2007; Kuiperet al. 2008).

Although many students showed instances of adequate searching, reading and evaluating behaviour, they alternated this with inadequate use of the Web, for example, by formulating proper search terms for one assignment but failing to do so for another. For both library and educational practices it is important to identify the origins of students' varying Web behaviour, with a view to both the conceptualization and the teaching of Web literacy. For this reason, in this article we look more precisely at the way students used the Web during assignments after the programmes. We analyse the performances of twenty-one student pairs from eight classes. We formulated the following research questions:

• Which Web searching, reading and evaluating strategies do students use?
• How adequate are students' Web strategies?
• What are the characteristics of (in)adequate Web strategies?

Method

Study design

With a view to these research questions, an exploratory approach was appropriate, since we aimed at both a systematic description of students' Web strategies and an exploration of possible explanations of the adequacy of their Web use (Robson 2002). We used both qualitative and quantitative data collected in eight schools participating in a study on teaching Web literacy (see Introduction; Kuiper 2007; Kuiper et al. 2008).

Participants and data collection

For the purpose of this study, three student pairs from each of the participating eight classes were selected to perform final assignments after the programmes had ended. Due to students' illness and technical (software) problems some loss of data occurred; as a result, we used a total of 21 data sets. Most of these pairs consisted of either boys or girls, but some were mixed couples. The pairs were selected by the teacher, on the basis of two criteria: in the three pairs from each class a range of reading skills should be present (i.e., relatively weak, intermediate and strong readers); and the students were expected to work together well. Students' level of reading skills was determined on the basis of their scores on a Dutch reading comprehension test.

Although Web literacy should be seen as a set of connected sub-skills that must not be separated and are all important in Web activities, we wanted to be able to look more precisely at the way the students used the skills and strategies that were at the centre of the programmes. Based on the components of Web literacy mentioned in the introduction, three kinds of skills and strategies are distinguished: Web searching, Web reading and Web evaluating skills and strategies (see also Appendix 1). A strategy may be seen as an approach to using the Web. For example, using Google is a search strategy; using links in a Web text is a reading strategy. A strategy may be adequate or not adequate within the context of a certain task. The skills students possess are conditional for effective strategy use and thus for the adequacy of students' use of the Web: an adequate use of Google requires the mastery of certain skills such as, for example, composing keywords. Therefore, the final assignments deliberately focused on the students' strategy use as well as their ability to use the Web searching, reading and evaluating skills addressed in both programmes, in a variety of topics and contexts. Students received six assignments, each focusing on particular aspects of Web literacy. Appendix 2 shows all assignments and the skills they were intended to measure, as well as the type of task.

Students' use of Web searching, reading and/or evaluating skills and strategies were investigated through three assignments. In assignments 1 and 2, students were asked to find a specific answer on the Web. They were free to use the Web as they wanted, which made it possible to observe their spontaneous Web searching, reading and evaluating strategies and skills. The assignments differed in level of complexity, assignment 1 being more simple and straightforward, while assignment 2 was more complicated both with regard to the phrasing and the difficulty of finding the right answer. Assignment 3 focused more exclusively on reading skills. This assignment provided students with a particular Website and asked them to search for specific information on that Website. Searching and evaluating skills were less important in this assignment.

Students' use of Web reading and evaluating skills and strategies were also investigated in assignments 4, 5 and 6, which were of a different nature, as they focused on students' elicited use of these skills and strategies. These assignments did not require students to find an answer on the Web, but asked them to give their comments or opinion on elements of Web evaluation. The distinction between the first three and last three assignments made it possible to compare such contemplative use of evaluating skills with the students' spontaneous use of these skills during their Web searches.

The time students spent at each assignment was recorded. Students were given approximately one hour for performing all assignments, with a time limit of about 12 minutes for each assignment. All assignments had been tried out previously by students of the same age and, where necessary, adapted. All student sessions were captured on Camtasia screen recording software, which also recorded the conversation of the student pairs. Students worked in a separate room and with a university notebook computer equipped with the necessary software, to prevent problems due to technical restrictions of the school computers. During the assignments, students were stimulated to discuss their way of working with each other. One researcher was present at all sessions and made supplementary notes about the way students worked together and carried out the assignments.

Data analysis

All Camtasia recordings were transcribed twice: students' conversations were transcribed verbatim and students' screen activities were written down. A transcription sample is given in Appendix 3. The transcripts were analysed both quantitatively and qualitatively.

The quantitative analysis focused on our first two research questions: the Web searching, reading and evaluating strategies students used as well as the adequacy of these strategies. This analysis concentrates on the assignments 1, 2 and 3; the assignments 4, 5 and 6 are analysed in a qualitative way (see below). To get an overall picture of students' strategy use, the recordings and transcripts of the assignments 1 and 2 were analysed with the help of a list of Web searching, reading and evaluating strategies the programme had paid attention to (see Appendix 1). All strategies the student pairs used in these assignments were classified and counted. In this first analysis, we took all students together and made a clear distinction between Web searching, reading and evaluating strategies.

However, especially with regard to searching for and reading Web information, the use of a certain strategy does not provide information about its adequacy in the context of a specific task or question. For example, using only one keyword when searching on Google may be an adequate strategy in the context of the Madagascar assignment (see Appendix 2), but inadequate in the context of another task. Since measures for quantifying the adequacy of students' Web behaviour do not exist (Kuiperet al. 2005), we constructed an adequacy index. We designed a coding procedure in which each separate reading and searching activity was first coded and subsequently scored as either 0, 0.5 or 1 depending on its adequacy in the context of the specific task. This procedure was refined after repeatedly going through the data, looking especially for activities that did not fit in with the procedure. Appendix 4 shows the resulting coding scheme, with the definitions used to identify activities and examples. These scores were added and divided by the total amount of reading and searching activities each pair had performed during that assignment. In this way, an overall score (between 0 and 1) for each student pair's adequacy in using Web searching and reading strategies was calculated. The reliability of the coding procedure was calculated by coding approximately 50% of the activities twice, by two independent researchers (Cohen's kappa (a measure of inter-rater agreement) = .87). The results of this quantitative analysis are reported in the first section under Results.

The qualitative analysis focused on our third research question: the characteristics of the adequacy of Web strategies. First, the transcripts of the student pairs' dialogue and screen behaviour during the assignments were analysed with a view to describing and identifying the Web skills students used, with the help of the list of skills in Appendix 1. Students' Web searching and reading skills were described and counted for the assignments 1, 2 and 3. When relevant, we compared and contrasted these qualitative data with students' adequacy scores, in order to create a more complete and distinctive picture. Students' use of Web evaluating skills was determined by describing all instances of both the spontaneous use of these skills during the assignments 1, 2 and 3, as well as the elicited use of these skills during the assignments 4, 5 and 6 which asked students to discuss certain aspects of Web evaluation (i.e. the quality, intentions or usefulness of a specific Website). Both ways of using Web evaluating skills were then compared in order to get a complete picture of students' evaluating skills. We report the results of this first part of the qualitative analysis in the second section under Results.

Secondly, we analysed the transcripts with a view to identifying tendencies that explain the adequacy of students' Web strategies, using a content analysis approach (Miles & Huberman, 1994). We focused on differences in the strategies of student pairs with different adequacy scores and on the strategies leading to unexpected scores (e.g., students with comparable reading comprehension scores but different adequacy scores and students pairs that performed very inconsistently). By constantly comparing and contrasting the performances of student pairs, we identified a number of tendencies underlying the adequacy scores. This second part of the qualitative analysis is discussed in the third section under Results.

Results

General analysis of students' use of the Web: strategy use and adequacy

With regard to Web searching strategies, Table 1 shows that students preferred using Google. They regarded Google as an easy way of searching the Web, arguing that 'you can find anything there' and 'you only have to type in some terms and then Google finds it'. They were able to use other search strategies, but did so only occasionally, although they had been introduced to a variety of strategies. The differences in complexity and difficulty between the assignments gave rise to students' use of either single or multiple search terms, assignment 2 encouraging, much more so than assignment 1, the use of multiple search terms (see also Appendix 2). Specific characteristics of the tasks were also reflected in greater use of directory pages in assignment 1 and greater use of search options within a specific Website in assignment 2. With regard to students' Web reading strategies, the 'non-reading' category stands out in that during both assignments students showed many instances of literally overlooking the right answer or overlooking a menu or link where the answer could easily have been found. Students frequently used a scanning technique when reading Websites, as they had learned during the programmes. Surprisingly, assignment 2 shows a smaller amount of reading than assignment 1. It seems as if the more complex and difficult assignment 2 resulted in more searching for relevant Websites, but fewer attempts to access and thus read, Web texts. Evaluating Websites or Web information occurred with a view to determining the usefulness of a site, with students looking mostly at its relevance for the specific task. Students never questioned the reliability of the Websites they accessed during these assignments.

To assess the adequacy of students' Web searching and reading strategies in particular, all student pairs' Web activities during assignments 1, 2 and 3 were scored (see Appendix 4). These assignments asked students to find an answer on the Web, either by searching freely or on a given Website. Table 2 shows all students pairs' scores, together with their reading comprehension level and the time they spent at each assignment. Students' scores and times show great differences, both within and between assignments. The differences between the assignments are reflected in both the mean scores and the mean time spent on the assignments. Assignment 3, in which students had to search for an answer on a specific Website, was relatively easy and, consequently, shows the highest mean score and the lowest mean time. The complex assignment 2 resulted in the lowest mean score and the highest mean time, while nine of the twenty-one student pairs failed to find the (complete) correct answer. However, there are several exceptions to this general tendency of students getting better scores on assignment 2 when compared to assignment 1, or even with assignment 3. Differences (within assignments) between student pairs are expressed in the wide range of scores on the first and second assignments in particular, which vary from 0.34 to 0.94 and from 0.23 to 0.83 respectively.

To explore the possible relation between students' adequacy scores and their reading comprehension level, we divided 18 student pairs into two group according to the highest score of each pair (the pairs 19, 20 and 21 were left out due to a lack of exact reading scores). For both groups the mean score and time were calculated, as well as the number of correct answers. Table 3 shows that better readers tend to outperform the weaker readers in all respects. However, the groups' performances differ significantly (using the Mann-Whitney test, because of the small sample size) only with regard to students' mean time used for assignment 3. When taking a closer look at Table 2 and especially at the performance of the highest and more intermediate reading pairs, it shows that on the level of the individual pairs the performances vary greatly. The figures may indicate that (very) low reading scores may result in inadequate Web use, but higher reading scores do not lead to more adequate Web behaviour as a matter of course.

Characteristics of the adequacy of Web strategies: which Web skills do students show?

In the course of the programmes, students had learned various Web searching, reading and evaluating skills (see also Appendix 1). In this section, we take a closer look at the skills students showed to master during the assignments.

Students' use of Web searching skills

In the course of the programmes, the students in our study had learned that when searching the Web, they first of all need to know what they are looking for. They had learned a variety of Web searching strategies, as well as the possibilities and limitations of those strategies and the skills they require. For example, using a search engine means that one has to be able to define appropriate search terms. In the former section we mentioned students' overall tendency to use Google. Most of the times they did not choose this search strategy deliberately and only used another strategy when they got lost in Google or when expecting easy results from a different strategy. For example, in assignment 2 some student pairs began by typing the URL of the newspaper mentioned in the assignment. Students also frequently used a search option on a Website they found through Google.

The fact that students predominantly use Google has consequences for the skills required. In light of this, our focus was primarily on Google-related skills. Appendix 1 gives an overview of relevant searching skills when using Google, which the students had been introduced to during the lessons. With regard to correct spelling of search terms, most students were either able to avoid spelling errors or corrected themselves when committing such errors. Five pairs (of 21) committed spelling errors, all of them in assignment 2. Although the first assignment had one difficult word (Madagascar), all students checked their spelling when typing this term. Their mistakes in the second assignment concerned simple Dutch words and were partly related to the students' general spelling problems; two pairs mainly made mistakes because of carelessness.

Steven and Vincent are two bright students who use the Web quite fluently. Especially when working on the second assignment, they made many spelling mistakes in ordinary Dutch words they failed to notice.

Such mistakes did not affect students' searching results when using Google, because Google corrected the mistakes itself (i.e., by suggesting a search term with correct spelling). However, when using a search box on a specific Website, the mistakes did influence the search results.

Composing relevant search terms to a great extent determines Web searching results. Students differed greatly in their ability to compose relevant search terms, which particularly affected their results in the second assignment. In the first assignment, half of the pairs used a single search term (Madagascar), the other half included the word language in their search term. Both strategies resulted in finding at least one particularly relevant Website. With regard to assignment 2, searching was much more complicated: the assignment included more potentially relevant search terms, thus asking students to choose the best combination and although the right answer could be found on several Websites, these Websites were partially quite obscure and/or the information was rather difficult to find. Students used a variety of search terms in this assignment, varying from highly effective ones to typing in the whole question, accompanied by a question mark.

Although most students already talked of themselves as skilled Web users before the lessons, students' Web use in the second assignment showed that many students were not aware of the way search engines such as Google operate. They used irrelevant words as 'which', 'and' or 'the', or spoken language, without realizing its consequences or not understanding why leaving out words did not affect the search results.

Saira and Fathma spent a lot of time on assignment 2. They tried out several search terms, mostly sentences derived from (part of) the literal text of the assignment. In a post-mortem on their performance afterwards, the researcher gave some alternatives for the search terms they used, e.g., algemeen dagblad doughnuts instead of the newspaper the Algemeen Dagblad and best doughnuts. They were very surprised: 'Algemeen dagblad doughnuts, that doesn't make sense, he [Google] doesn't understand that'.

In addition, nine student pairs consistently chose the Google option to restrict searching to Dutch Websites; twelve pairs did not choose that option. Most students did not distinguish between Dutch search terms (as in assignment 2), which logically only could deliver Websites in Dutch and a search term such as Madagascar, which could give also access to non-Dutch Websites. Students never took the number of results Google produced into account, although in the lessons they had been introduced to the relevance and meaning of that number to decide to broaden or narrow a search strategy. Overall, the skills taught during the programmes were only partially applied by the students during the assignments.

Students' use of Web reading skills

The students had been introduced to differences and similarities between general reading skills and Web reading skills that are specific to the Web's information overload and hypertext environment. They had learned that when using the Web as an information resource, their reading begins when (using Google) they are shown, a frequently endless, list of search results. The heading and short text referring to a specific search result offer information about its usefulness and skilful scanning through these lists may define success or failure to locate relevant information. Students also had been taught to relate their reading to what they were looking for and thus to what they wanted to know and had learned to reflect on their understanding of the information found.

How did the students in our research read the Web? And to what extent did they use the complicated skills mentioned above? During the assignments, all students showed at least some ability to use a menu, to use links, to navigate deeply into a Website and to scan Web texts. In these respects, some of them were quite fluent Web users. Yet, they also showed many instances of 'non-reading', i.e., of overlooking the right answer or a clue (menu, heading) for finding it, as well as overlooking relevant Websites when scanning Google's results and/or looking for an answer on a Website that had nothing to do with the assignment, etc. Students also tended to read only parts of Web texts and to click away sites with too much text or an overly complicated structure. One could say that although students knew how to deal with hypertext elements, they often failed to use ordinary reading skills when using the Web. This was especially remarkable with regard to students who were rather good conventional readers.

Tim and Simon come across the Website of the newspaper mentioned in assignment 2 and find a page with search results about the doughnut test. While the first result has both a clear heading ('Best doughnut this year in Gouda') and an answer entirely provided in one sentence, the students click on the second link on the page that has the heading 'Nasty, greasy and sometimes delicious'. When this page offers no answer, they return to Google's search results.

Robin and Nena are good readers and careful Web users, who take their time. Although they do search in the right way, they constantly overlook relevant headings and links. This results in a failure to find the answer to the first assignment.

Especially with regard to the Web reading skills, the nature of the assignment affected the students' use of the reading skills they had been introduced to. Most students seemed to be well able to locate information on a particular, not too complicated Website when they knew the information must be there. Table 3 shows that students performed quite well on the third assignment, which investigated the way students navigated on a particular Website and the reading skills and strategies they used to find the required information. When looking more closely at the students' performance, differences between the students do not seem to be primarily related to specific Web reading skills but more to navigational skills and efficiency in using these skills. For example, the particular Website had to be opened by clicking on the word Welcome on the homepage, or by selecting a menu that popped up when scrolling the screen. Although some students lost some time by pursuing the wrong track for a while, most of them quite easily found the right links to follow. Students differed in their reading speed and strategies, some scanning and others closely reading all texts, but almost all found the required answer. Thus, efficiency rather than effectiveness seems to be the distinguishing factor in students' behaviour.

When comparing students' use of Web reading skills during the first three assignments, it became clear that assignment 3 resembled assignment 1 in one respect. Almost all students came across one particularly relevant Website when working on assignment 1, on which the answer could easily be found through the use of a clear menu. Many students totally overlooked this menu, although they proved to be able to look for and use, menus very effectively when inspecting the Website given for assignment 3. This suggests that the students somewhat paradoxically tended to overlook meaningful clues on a Website when they were not sure it would offer them the right answer. More generally, this could mean that students tend to use less appropriate Web reading skills when asked to search freely, as opposed to search at a specific Website.

A similar observation concerns the differences between students' Web reading during the assignments 4, 5 and 6. Although all of these assignments asked students to access a specific Website and thus to read its contents, students showed different reading behaviour during these assignments. While quite effortlessly navigating deeply into the Website given in assignment 3, many of the students stayed mainly on the Websites' homepages for assignments 4, 5 and 6, without further exploring the Websites' contents. This could be a matter of laziness but may also suggest that the need to find a specific answer (as in assignment 3) elicits Web reading behaviour that differs from the Web reading that students show when they are asked to form their own opinion on a Website or to evaluate the main purpose of a Website (as in the assignments 4, 5 and 6).

As we have already mentioned, there is a relation between conventional (comprehensive) reading skills and adequacy of Web use. Weaker readers perform lower than stronger readers. However, although Web reading skills seem to be pervasive through all forms of Web use, the relation between students' level of conventional reading skills and their reading on the Web is not unequivocal. Students' lack of conventional reading skills may interfere with their effective use of the Web in several ways. However, not all strong readers in our study performed better than the more intermediate readers and not all weak readers performed poorly on all assignments. Moreover, also the strong readers show lack of adequate Web reading skills, or instances of non-reading.

Students' use of Web evaluating skills

Students had been introduced to the differences between the Web and (non-fiction library) books during the lessons on evaluating Web information and had discussed the problems involved, such as the abundance of unauthorized information. Three main aspects of Web evaluation had been addressed in the lessons: assessing their own understanding of the information found, assessing the reliability and authority of that information and assessing its usefulness or relevance. Understanding the information was closely linked to Web reading. Assessing the reliability and authority of Web information had been discussed by presenting the students with three questions they needed to ask themselves when visiting Websites (the three Ws): Who is the author of this Website? What intentions does this Website have? (or: Why is it made?) and, Which kinds of illustrations or pictures does this Website contain?

The students' use of evaluating skills was assessed in two ways during the assignments. First, assignments 4, 5 and 6 asked the students to comment on the pros and cons of a given Website, to formulate the purpose behind a Website and to comment on the usefulness for children of Websites. Secondly, the students' use of evaluation skills in practice was assessed by assignments 1 and 2. During Web searching, the evaluation of Websites and Web information is important to avoid following wrong tracks and to assess the relevance and reliability of information found. Although during the assignments both aspects came to the fore, the students in our study mainly looked at the usefulness of a Website for finding an answer. They explicitly or implicitly decided many times about whether to click on, or explore a particular Website, arguing 'this must be it', 'no, I don't think we can use that', 'that's not about Madagascar' or 'no, that says something about 2002 and what we need is 2004'. However, they did not always use the correct arguments, as was shown in the section on reading skills. In other words, they looked for the relevance of specific Web information but failed in their elaboration.

During assignments 1 and 2, students never questioned the reliability of the Websites they visited. At best, they asked themselves what a specific Website was about, without accepting the consequences when they did not know. With regard to the first assignment, this had only minor consequences because the question was rather simple and students were easily able to find a particular relevant and reliable Website. However, when working on the second assignment, students were confronted with all sorts of Websites, varying from newspapers' Websites to Weblogs and from bakery Websites to satirical Websites about doughnuts. The right answer could be found on a range of Websites. The students in our study treated everything they found as if it was factual information and neither distinguished facts from opinions, nor questioned the reliability of the information found or of the intentions of the author.

While searching for the answer to assignment 1, several students came across a Weblog called Peter29. They all began reading its text without looking at the Website as a whole. Two pairs thought that the bakery they were looking for was called Peter.

In other words, all Websites were treated equally as potentially offering the right information. This may reflect students' being used to assignments in which they are requested to answer questions about a particular text, presented and thus authorized by the teacher. They may not adapt their strategy to assignments in which they are asked to use the Web, which is open and unauthorized.

However, both during the programmes and during assignments 4, 5 and 6, students appeared to be able to distinguish between several types of Websites, as well as to identify the main purpose of a Website and to critically question a Website's reliability (although rather superficially and without navigating deeply into Websites). For example, during the programmes students had to comment on the Website of McDonalds, the fast food franchise. Almost all students were very critical of the Website and questioned its intentions, arguing that they found it quite ridiculous that McDonalds should make such a healthy looking Website while 'everyone knows you grow fat at McDonalds'. They could also distinguish between the intentions of a Website of the meat industry and of vegetarians and could comment on the way information about eating meat was presented on these Websites. However, they did not use these kind of practices when asked to use the Web to find an answer on concrete assignments. Perhaps a somewhat trivial question about a doughnut test as in assignment 2 does not elicit such critical Web behaviour.

Tendencies in students' Web strategies

A mere description of students' Web literacy skills and strategies does not explain the students' ability to use the Web adequately. Many students showed inconsistent Web behaviour. Their performance varied between the assignments, i.e. they performed better on more difficult assignments or showed varying adequacy scores (see Table 3). Also, performance varied within one assignment, e.g., by alternating effective actions or strategies with highly ineffective ones. From our results we identified four tendencies that underlie students' (in)adequate Web strategies and that affect the adequacy of their interaction with the Web. Thus, in this section the focus is no longer on students' separate Web skills and strategies but on the characteristics of their approach of Web tasks. For example, students who decide to follow a potentially productive search strategy but are too impatient to maintain this strategy, may be as unsuccessful as students who are not capable of designing good search strategies at all.

Inflexibility

Flexibility appears to be important for adequate Web use: the ability to alternate strategies, depending on their success. Many students in our study showed inflexible Web strategy use: they stuck to one particular search strategy despite lack of success, either because they were convinced of the appropriateness of their approach or because they expected Google to find the answer for them regardless of their own strategy use.

Tim and Simon are two boys who carried out the first assignment very effectively, resulting in an adequacy score of .94 in only 1.5 minutes. They used good search terms and worked in a very focused way. Tim in particular was convinced of his own Web skills. They confidently started the second assignment. After some time, they found a Website about a doughnut test in 2002. Being convinced that their search strategy was effective, they continued looking at Google's search results, arguing that 'We found 2002 in this way, so somewhere around here there must be a Website about 2004'. They continued 'browsing' through Google, scanning all Websites. Even when they arrived at the 20th page, the 200th result, Tim, in particular, remained convinced that 'it must be somewhere around here'.

Such inflexibility was related to many students' profound belief in Google's power to find relevant information. They rather naively appeared to rely on Google.

Kimberley and Rachelle were searching on Google with the search term 'Madagascar which language'. One of the first results was a Website about the language of Sri Lanka, given both in the heading of Google's search result and on the particular Website itself. Kimberley and Rachelle began to read the Website and write the language spoken at Sri Lanka down as the answer. After the session, the researcher repeated their search strategy and showed them the heading of the Website. The students were confused: 'We typed in Madagascar and then, well we thought everything is about Madagascar'.

Sana and Dunya typed in the whole text of assignment 2 in Google's search box. This resulted in only three (irrelevant) results. Both students were surprised: 'How is that possible, how is it possible he [Google] doesn't know that?'

This also seems to be related to the students expecting some sort of intelligence from Google.

Jim and Ashley used Google to find an answer to assignment 2. One of the search results said in its heading 'best doughnut this year in Gouda'. The students decided not to look at this site because 'it is about this year and we should be looking at 2004' (students worked on the assignments in spring 2005). Subsequently, Ashley suggested that they change the search term and write 'last year' instead of '2004'.

Although not all students trusted Google to a similar extent, their Web use reflected the same tendency. They knew that the answer could be found through using Google and failing to do so often resulted in questioning Google instead of questioning their own strategy. This also affected the students' Web reading since they did not give sufficient efforts to reading Google's results critically. This resulted either in their ignoring relevant Websites or picking out irrelevant ones, without looking at a heading or search result.

Impulsiveness

Trial and error was a much-used strategy by students. In fact, this may be an efficient way of trying out which strategy is worthwhile exploring further. However, the students in our study did not use trial and error in such a conscious way. They either tried out several search strategies from the beginning without much thinking, or continued to adapt their search strategy when initial success did not materialize. They also tended to change their search strategies rather impulsively.

Rachid and Ismail were using effective search terms for their second assignment. When failing to find the right answer right away, they removed all search terms and replaced them by others.

The Web reinforces impulsive behaviour because of its speed and ease of use; students are able to change tactics without much effort. Thus, patience seems to be a relevant component in adequate Web behaviour. As Web users many students in our study appeared to be impatient, impulsive and/or careless, regardless of their Web skills. They either were unaware of this, or seemed to underestimate its effect. Bright and confident students trusted their own capacities to such degree that they did not even look at their own spelling mistakes.

Steven and Vincent, two bright but also rather careless Web users, simply tried something else when their misspelled search terms did not produce the desired results.

Focus on 'finding the right answer'

Students' tendency to rely on Google also has to do with their focus on 'finding the right answer' when using the Web, which is also mentioned in other studies (e.g., Wallace et al. 2000). The rather factual assignments used in our study may have reinforced such behaviour. However, we saw similar behaviour during the programmes preceding the assignments in which students used the Web to find information for their own research questions. This tendency had repercussions for students' Web searching, in their reading as well as in their evaluating strategies. When scanning Google's results pages or scanning particular Websites, they primarily looked for the answer itself or very strong pointers to the correct answer. When scanning Web texts, they either searched for the exact answer or for clues in the exact wordings of the assignments. Thus, they tended to overlook relevant (parts of) Websites that used different phrasing or words or, conversely, explored irrelevant parts that used the words they were looking for. This affected, in particular, assignment 2, which used quite elaborate and complex phrasing.

Rachid and Ismail found part of the right answer on a Website. They hesitated to write it down, arguing 'This one talks about the most delicious doughnuts, it does not say anything about the best doughnuts'.

Nadia and Melanie were scanning Google's results of their search term 'best doughnuts 2004'. They clicked on a page that mentioned 'Best doughnuts' in its heading, which is, in fact, the Website of a swimming club. They scanned its text carefully, only looking for a reference to December 2004, instead of looking for the usefulness of the Website. They clicked on a link 'photos doughnut party December 2004' and even explored the photos further.

Tim and Simon were scanning Google's results. Because they found a Website about the best doughnuts of 2002, they went on to look at the results but only scanned for the clue '2004'. Thus, they overlooked several Websites on which the answer could be found.

Kimberley and Rachelle had already found half the answer and only needed to know the name of the bakery asked in assignment 2. As a result they only looked for names when scanning Google's results and thus clicked on several Websites that had nothing to do with bakeries or doughnuts but carry a personal name in its heading.

The tendency of students to look for the right answer also resulted in their lack of questioning of the reliability of the information found or of the intentions of the author. They treated everything they found as 'factual information' and did not distinguish facts from opinions.

Lack of reflection

Generally speaking, the lack of reflection on their Web use by students seems to explain the tendencies mentioned above. Our study indicates that Web literacy skills appear to be necessary, but not sufficient for adequate use of Web strategies. They do not result in finding the required information without reflection on one's search process and using this to adapt one's strategy. Thus, Web literacy skills as well as the flexible, patient and open-minded use of these skills together determine students' adequacy in using the Web. This may be illustrated by the following examples.

Emma and Josy had consistent high adequacy scores and worked fast. Emma is a smart student with high reading comprehension scores, Josy is a moderate student and reader. They were focused on the tasks at hand. They did not click at random on Websites found with Google and knew how to scan Google's results efficiently. This proved its value for the second assignment in particular. They composed relevant keywords and used Google's list of results to obtain a first impression of the usefulness of the Websites found. Twice they made a mistake, but in both cases they were able to correct themselves and to modify their search strategy, i.e., to refine their search terms. They also kept their task in mind while looking at Websites. After having initially clicked away a relevant Website, they realized it might be useful after all while scanning other sites: 'This one is not about... What does it say in the assignment, when was it? December 2004, oh, but then it must be that first one, we will have to look there again'.

Saira and Fathma are both students with spelling and reading difficulties. They wanted to perform well. From the beginning, they worked rather impulsively and did not take time to read Google's results. They were not able to compose good search terms and repeatedly typed in the whole question as search term. Sometimes they scanned and read Web texts on Websites extensively, but they also clicked Websites away without any reading at all. They mostly focused at only one or two words in the assignment, for example '2004' or 'December'. They repeatedly thought that they 'knew what to do' and easily became frustrated when not being able to find the right answer in a short time. Although they sometimes were able to correct themselves (for example, recognizing that a Website was not about Madagascar), they either did not adapt their search strategy, or changed it radically without reason (e.g., by deleting all keywords and typing new ones). This resulted in loss of much time. Their lack of success made them dispirited and even more persistent in using trial and error.

Discussion and conclusion

In this study we looked at students' use of Web literacy skills and strategies during assignments after they had been through a programme in which they were introduced to various Web searching, reading and evaluating skills and strategies. Our research questions focused on the students' use of Web strategies, the adequacy of their strategy use and the characteristics of Web strategies.

With regard to students' strategy use, they appeared predominantly to use Google when searching the Web. They used both single and multiple search terms, depending on the type and complexity of the assignment and used scanning techniques when reading Web texts, as well as close reading. However, they also frequently overlooked right answers or relevant texts. Evaluating Web information only occurred with a view to determining the usefulness of a site and not a Website's reliability. The adequacy of the students' strategies showed great differences both between and within assignments. There is some relation to students' level of reading comprehension, but stronger readers do not outperform weaker readers as a matter of course.

With regard to the specific characteristics of students' Web strategies, we first looked at the Web searching, reading and evaluating skills that students showed during the assignments. Our study shows a mixed picture of students' use of these skills. With regard to their Web searching skills, students in general knew how to use Google, but they were not always able to compose relevant keywords. The students sometimes made spelling mistakes, especially with regard to simple words. With regard to Web reading skills, all students were able to use Web text elements such as menus and links. Most students appeared to be quite able to locate information on a Website when they knew the information must be there. However, they easily overlooked relevant information, or looked for information on a totally irrelevant Website when they had to find relevant Websites themselves. The students' use of Web evaluating skills showed that they were aware of the importance of such skills and were able to identify the main purpose of a Website and to critically question a Website's reliability. However, they did not use this knowledge when using the Web for their own information need. Although they sometimes assessed the usefulness of a specific Website for finding the required information, they never questioned a Website's reliability or trustworthiness.

The students' Web literacy skills did not offer sufficient insight into their ability to use the Web adequately, which was illustrated by many students' inconsistent Web behaviour. We identified four tendencies that may explain students' inadequate Web use. First, inflexibility, that is, not being able to alternate Web strategies, appears to affect students' adequacy negatively. Secondly, impulsiveness seems to be related to inadequate use of Web strategies and may be contrasted with the patience required when using the Web. The students' general tendency primarily to focus on finding the exact right answer, instead of looking for bits and pieces of information from which they could compose the answer themselves, also contributed to inadequate Web use. Finally, a more general quality of adequate Web strategy use is reflection on one's search process. Reflective use of Web literacy skills seems to determine students' adequate Web use. Reflection may, for example, involve keeping in mind one's information need and relating the obtained information to that need. These were aspects of Web use that most students in our study did not practice as a matter of course. Their motives for preferring the Web to books were related to (what they perceived as) the speed and convenience of the Web. In other words, most students did not regard their Web use as being in need of reflection. This poses a challenge for educators and librarians who want to teach students how they can use the Web for knowledge construction.

Our study confirms other recent research on students' Web use in several ways. Students' lack of evaluating behaviour is mentioned in several studies (e.g., Shenton & Dixon 2003; Kuiper et al. 2005). Todd (2006) also mentions students' strong orientation towards the gathering of facts. Interestingly, our study shows a different type of student behaviour than the interview-based study by Enochsson (2005) into students' own perspectives on possibilities and difficulties with regard to Web use. The students in her study appeared to be aware of the specific skills needed on the Web, including language-related skills such as spelling correctly and critical evaluating skills. The older students in particular also mentioned the time it takes to become a skilful Web user. In our study, many students were of the opinion that they were already rather good Web users. However, this was only partly reflected in their actual Web use. Enochsson's study took place in the years 1998 to 2003. In most developed countries, students' use of the Web both at home and at school has increased considerably in recent years. This may have influenced their perception of their own Web skills.

Our study only differentiated between students with regard to reading comprehension. In a study on somewhat older students, Heinström (2006) mentions different information seeking behaviour by students with different study approaches, i.e., surface, deep and strategic approaches. Most students in our study seem to be using a surface approach, which may also be due to the task they had to perform: although almost all students were motivated to complete the assignments, the tasks itself were factual and not particularly inviting to use a more strategic approach. It would be worthwhile to use a similar approach to the study of Web behaviour with students working on more inquiry-like assignments.

What are the implications of our results for educational practice? First of all, educators should be attentive to the impact of using an originally non-educational tool in the classroom. It is not enough to look at the Web as merely a replacement of print information resources. The use of the Web must fit in with learning goals. For example, using the Web to learn specific, prescribed knowledge of the digestive system may require a different use of the Web to students' writing a paper on a self-chosen aspect of that digestive system. Secondly, already in the lower grades, students must be introduced to the skills needed when using the Web for knowledge construction. The students who participated in our study were 10 to 11 years old; most of them were already experienced Web users at home. This affected their opinions on the need to learn new skills or to modify their existing skills. Therefore, the school needs to deal with Web use in earlier school years, when students have not yet fully developed their own Web using habits.

The use of the Web in education poses a dilemma for teachers who want to motivate their students and fit their teaching in with students' preference for using the Web instead of print resources. Such preference is partly based on the Web's supposed speed and ease of use, which is at odds with using Web information reflectively and critically. Moreover, students use the Web even more at home than at school. It might be that their ways of using the Web at home do not fit in with what is expected at school. In the classroom, students are expected to use information resources for the construction of knowledge and to feel responsibility for their own learning. The students' view of the Web as an easy way to find a ready made answer is reflected in their impulsiveness and impatience when they do not find the required information quickly enough. Also, the extensive use of trial and error methods by students reflects their habits when using the Web in their own homes. Trying out several strategies is much more common than reading carefully and then deciding what to do first. On the Web, such a strategy may in fact be very useful, because there rarely is only one best way. When using the Web for educational purposes, this lack of a single best way may conflict with a teacher's need for control of the ways students use the Web and of the results of their information seeking.

At home, students do not learn critical reading and reflective skills naturally. They need others to show them the need for such skills and to learn their specific use. At school, these skills are already part of the literacy curriculum but mostly with respect to conventional reading resources only. In fact, most students learn such skills from print-based methods and do not apply them when using the Web as a matter of course. In our view, a more substantial part of the literacy curriculum should be devoted to new, less print-based information resources. In addition, reflective skills as well as the ability to make well-founded choices and decisions may become more important in the current information society and will consequently have to be discussed and learned at school. In this light, the task for education may not lie primarily in teaching students Web searching skills, but in showing students the need for learning and practicing specific Web reading skills and Web evaluating skills, as well as a reflective use of these skills.

References

Bookmark This Page

Appendix 1: Content of the two programmes

Searching for Web information

In general: introduction of various Web searching strategies, their possibilities and limitations and the specific skills required

• Google: inferring search terms from the assignment or question; correct spelling of search terms; restricting one’s searching to Dutch sites; composing searching terms; paying attention to the amount of Google’s results; the way Google ‘works’.
• Search engines different from Google
• Searching with a specific URL: composition and spelling of URL’s
• Directory page: purpose and structure of directory pages
• Children’s search engine: advantages and disadvantages of specific search engines for children
• Searching options within a specific Website: composing of search terms; need for correct spelling

Reading Web information

In general: introduction of differences between the structure of print and Web texts; specific Web reading strategies as well as reading skills required on the Web

• Scanning search engine results (using keywords)
• Reading text on a Website literally
• Scanning text on a Website (using keywords, headings); knowing the various ways Websites are constructed
• Using Website menus; knowing the meaning and function of a menu
• Using Website links; knowing the meaning and function of links

Evaluating Web information

In general: introduction of the need for Web evaluation; components of Web evaluation; strategies to determine the reliability of Websites as well as the skills required

• Evaluating one’s understanding of Web information: paying attention to the difficulty of specific Web information and the relation to previously acquired knowledge
• Evaluating the reliability/authority of Web information: paying attention to who has constructed a particular Website; paying attention to the purpose or intention of a Website; paying attention to the function and meaning of non-textual elements, especially images
• Evaluating the usefulness of Web information: paying attention to the relevance of information for one’s information needs

Appendix 2: Web assignments

Assignment 1

Web literacy skills: integrated use of Web searching, reading and evaluating skills.
Free Web search – one correct answer

Search the Web to find an answer to the following question:

“Which language do the inhabitants of the island of Madagascar speak ?”

You may search the Web in your own way!

Assignment 2

Web literacy skills: integrated use of Web searching, reading and evaluating skills
Free Web search – one right answer

Search the Web to find an answer to the following question:

“At the end of each year the Algemeen Dagblad daily newspaper wants to find out where in the Netherlands they sell the best doughnuts. Which doughnut bakery in which town made the best doughnuts in December 2004?”

You may search the Web in your own way!

Assignment 3

Web literacy skills: primarily Web reading skills
Specific Website – one correct answer

Open the Website of Queen Beatrix and her family: www.koninklijkhuis.nl
Search on that Website for the answer to the following question:

“What are the names of the children of Princess Margriet’s oldest son (one of Queen Beatrix’s sisters)?”

Assignment 4

Web literacy skills: Web reading skills and forming one’s opinion about a Website.
Specific Website – arguments, more than one correct answer

Examine the Website of the society for the prevention of cruelty to animals (www.dierenbescherming.nl). Then write a short evaluation of the Website.

Write as many things you appreciate about the Website and as many things you do not appreciate. Think of all the things you must take into account when you examine and read a Website!

Assignment 5

Web literacy skills: Web reading skills and one specific aspect of Web evaluation, i.e., determining the main purpose of a Website.
Specific Website – arguments, more than one correct answer possible

Open the Website www.waddenvereniging.nl.
Examine the Website carefully. Then answer the following question:

“Why has the Waddenvereniging constructed this Website?”

Write your answer down as completely as possible!

Assignment 6

Web literacy skills: Web reading skills and one specific aspect of Web evaluation, i.e., determining the usefulness of a Website
Specific Website – arguments, more than one correct answer

Open the Website www.kinderconsument.nl.
Examine the Website carefully. Then answer the following question:

“Do you think this Website is useful to children?”

Write your answer down as completely as possible!

Appendix 3: Example of a transcript of Camtasia screen and audio recording

Appendix 4: Summarized coding form used for the analysis of assignments 1, 2 and 3

The form is used in order to assess the adequacy of students’ searching and reading activities within the context of a specific assignment.

An activity is seen as a distinct action on the screen. Talking about an activity is not counted as an activity unless talking leads to action on the screen. Activities may concern three aspects of students’ Web behaviour:

• T = typing a word or a couple of words
• C = clicking on a search result or a link, menu or option
• R = reading activity; reading may refer to reading a specific Web text as well as reading a list of search results. Assessing a reading activity takes both the students' screen behaviour and their conversations into account.

All activities receive a score of 0, 0.5 or 1, depending on the adequacy (or relevance) of that activity in the context of a specific assignment.

0 - The activity is irrelevant for completing the assignment and does not contribute to finding the right answer; this category also contains ‘non-actions’ such as skipping a relevant text on a Website in which the answer may be found

0.5 - The activity is somewhat relevant for completing the assignment and might contribute to finding the right answer.

1 - The activity is relevant for completing the assignment and does contribute to finding the right answer

Examples of coding for assignment 1: “Which language is spoken by inhabitants of the island of Madagascar?”

T - 0 - Typing ‘language’ in Google’s search box

T - 0.5 - Typing ‘Madagascar’ in Google’s search box

T - 1 - Typing ‘language Madagascar’ in Google’s search box

C - 0 - Clicking on a search result that has no reference to Madagascar in its heading or accompanying text

C - 0.5 - Clicking on a search result that has some reference to Madagascar in either heading or text, but no reference to language spoken on Madagascar

C - 1 - Clicking on a search result that refers to both Madagascar and the language spoken on Madagascar in heading or text

R - 0 - Scrolling on, or reading a particular Web text that contains the answer to the question, but failing to see the answer (e.g., because students read too fast); looking for the answer on a Website that does not refer to Madagascar

R - 0.5 - Scrolling on a particular Website with the question in mind, but absent-mindedly, which may be reflected in students’ actions and talking (e.g., “we must look for headings about the language they speak” while scrolling too fast to really read anything)

R - 1 - Scrolling on a relevant Website or reading a relevant Web text clearly with the question in mind, which may be reflected in the way students talk while reading (e.g., “we must look for a heading about the language they speak”)

Web Counter

© the authors, 2008. Last updated: 22 September, 2008

online web site analytics