vol. 13 no. 4, December 2008 Contents | Author index | Subject index | Search | Home

Perceived difficulty as a determinant of Web search performance

Introduction

Today, the Web has become an integral part of people's lives: many people go online to conduct ordinary, day-to-day activities. Most people rely on search engines to discover and access content from the Web, but despite search engines' usefulness, their widespread use may activate a significant bias in people's perception of the Web. For instance, many people go to Google, type one or two keywords in an empty search box and then peruse the first few results. If they cannot find relevant pages after several iterations of keyword queries, they are likely to quit searching the Web. When using advanced search techniques, many make mistakes, as has been observed in many empirical studies on Web searching behaviour (e.g., Jansen et al. 1998). Nonetheless, these Web searchers felt confident, satisfied, and trusting of their particular search process and performance.

On the Web, a self-sufficient searching environment, people's perceived difficulty in searching for information is very low. Many Web searchers are confident about their searching abilities: they believe that Web searching is easy and fast (Griffiths and Brophy 2005) and their satisfaction with Web searching is high. Fast and Campbell (2004), in their study of university student perceptions of searching OPACs and the Web, confirmed that student satisfaction with Web searching was higher than with OPACs because of student expectations and confidence in Web searches through Google. They highlighted that this may reflect a mere perception of success and does not always lead to actual search success. Therefore, it seems meaningful to examine the role of searchers' perceived level of difficulty in search performance.

The purpose of this study is to understand perceived difficulty as a determinant of Web search performance. Specifically, the principal questions underlying this study are:

1. Do pre-search difficulty and post-search difficulty differ? Do perceptions of difficulty change after searchers have experienced actual searches?
2. Is there any difference in perceived difficulty between groups of searchers by their demographic characteristics? Do older searchers perceive more difficulty than younger searchers? Do female searchers perceive more difficulty than male searchers? Do soft science searchers perceive more difficulty than hard science searchers?
3. What is the relationship between perceived difficulty and Web search performance? When a search is perceived to be too difficult, are searchers less successful in their search performance?
4. What determines perceived difficulty in Web searching? What are the reasons for those perceptions?

Related research

Perceived difficulty

Perceived difficulty has been conceptualized as the perception of the ease or difficulty of performing a specific activity in many other areas, such as psychology, education, and management. Many psychological theories use the concept of perceived difficulty as a mediator of emotion, motivation, and behaviour. For instance, social psychology's Theory of Planned Behaviour (Ajzen, 1985) named perceived difficulty as a potent predictor of behavioural intention. Further, it has worked as a measure of the affective component of attitude as well as a measure of the evaluative component of attitude. Perceived probability of success has been inextricably linked to attribution of effort or ability. So if the task is perceived to be too difficult, then people expect that their chance of success is low and attribute failure to the perceived difficulty.

A few studies in information seeking and retrieval have investigated the complexity or difficulty of search or search tasks. These studies did not distinguish between complexity anddifficulty, but most of them explored the concept of complexity, the objective properties of a search, rather than difficulty, which refers more to the context of the individual searcher.

Earlier studies tried to capture quantitative characteristics intrinsic to a searcher's session, based on the assumption that query formation and expansion is an integral part of every effort to search for information, and used the number of search terms and/or search concepts involved in the query (Iivonen 1995; Saracevic et al. 1988) as the index of complexity. Recent studies (Byström and Järvelin 1995; Vakkari 1999) have given more attention to the concept of task complexity and have tried to connect it to information seeking and use. For instance, Byström (2002) investigated how task complexity affects searchers' needs and choices of different information types and sources. One consensus of these repeated investigations is that tasks can be categorized according to complexity. However, this task complexity has been seen as a priori rather than empirically observed or explicitly described by the searcher.

The searcher's perception of difficulty in a given search task has received less attention. White and Iivonen (2002) focused on assessing the difficulty of search questions for Web searching, but they did not observe its relationship with actual search performance. The recent study by Gwizdka and Spence (2006) found that subjective task difficulty correlated with many measures that characterize the searcher's activities, but they did not identify the reasoning behind such difficulty. All these studies also assumed that perception of difficulty is static, so the difficulty was measured only to predict the chance of success.

Search performance

There are quite a few different ways of measuring Web search performance for individual searchers. They generally fall into two groups: search behaviour and search outcomes. Some key indicators for search behaviour are time spent completing a search task (Aula and Nordhausen 2006; Bilal and Kirby 2002), the number of pages viewed or the number of pages saved (Kelly and Cool 2002), and the number of search queries submitted by a user (Su 2003). On the other hand, success (Lazonder et al. 2000), user satisfaction (Bruce 1998) and precision or recall (Palmquist and Kim 2000; Zhang et al. 2005) have been employed to assess the search outcome. These are the variables that could be collected either directly from users or without any intervention from users while they were engaged in information seeking tasks.

Of all measures, search task completion time proved to be the most frequently used variable and the best indicator of search interests or performance. It has also been understood as a measure of the effort the user has to expend. However, the obvious shortcoming in using task completion time alone is the need for a measure that would define the faster user as the more successful searcher (Aula and Nordhausen 2006). Therefore, many other variables should be included to consider both the efficiency and effectiveness of the search.

Research methods

Experiments

This study was conducted as an experimental study in which subjects were given search tasks and were observed as they searched the Web. Three tasks, factual, interpretative and exploratory, were assigned. Task scenarios were used to present realistic situations (Borlund and Ingwersen 1997), which allowed the searchers to identify what seemed to be relevant material, as shown in the Appendix. Thirty volunteers participated in this study. All were graduate students with backgrounds in library and information science.

Measures

A continuous five-point scale of searchers' perceived difficulty was developed. A value of 1 indicated a search with little difficulty. A value of 5 indicated a highly difficult search. Perceived difficulty was assessed before and after search completion because this study assumes that the perception of difficulty may change over time through increasing experience via searching. Thus, in the pre-search questionnaire, participants were asked, 'How difficult do you think this search will be?', and in the post-search questionnaire, they were asked, 'How difficult was it to complete this search?' In addition, participants were invited to articulate the reasons for their perception of difficulty in a post-search interview.

Search performance was assessed by search task completion time, the number of pages viewed, the number of pages saved, the number of search engines used, and the number of query reformulations. These were obtained from the stored search histories of each participant, so that they could be collected without any intervention from searchers while searchers were engaged in search tasks.

Three demographic characteristics for searchers, which were collected using a background questionnaire, were dichotomized as follows: sex (male, female), age (under 30 years old, over 30 years old), and academic background (soft science, hard science).

Results

Perceived difficulty: pre-search and post-search

As shown in Table 1, the overall post-search difficulty is lower than pre-search difficulty for all three tasks. This means that participants felt that their searches were much easier after they actually completed them. Among the three tasks, the exploratory task was most often reported to be difficult before the search was initiated (M = 3.28); however, the factual task was assessed to be most difficult after the search was complete (M = 2.62).

The perception of difficulty is not static, but may change with the search experience. A statistical analysis for the correlation between pre-search difficulty and post-search difficulty was conducted, to answer the question: If a search is expected as more difficult beforehand, will the search be also assessed as more difficult afterward? Only in the case of the factual task did participants who rated difficulty somewhat higher before the search tend to rate difficulty higher after the search (r = .40, p < .05). For the other two tasks, no relationship between pre-search difficulty and post-search difficulty was found.

Perceived difficulty by searcher's demographic characteristics

The levels of perceived difficulty were checked for differences between groups of searchers, with regard to the demographic variables of age, gender, and academic background. As shown in Table 2, searchers under 30 were overall more sensitive to the level of pre-search difficulty for the three search tasks than those over 30. A T-test was then conducted to determine whether there is a significant difference in the level of perceived difficulty between age groups. The statistical analysis shows that subjects under 30 expected the interpretative task to be rather hard before the search was executed (t(28) = 5.67, p< .05). Also, subjects under 30 experienced the exploratory task as having been harder after the search was completed than did subjects over 30 (t(28) = 12.80, p < .01).

Perceived difficulty and search performance

The Pearson coefficient of correlation was used to calculate for the relationship between perceived difficulty and search performance, as shown in Table 3. With regard to pre-search difficulty, there was a significant positive relationship between perceived difficulty and pages viewed (r = .49, p <.01), pages saved (r = .42, p < .05) and query reformulations (r = .59, p < .01), only in the case of the exploratory task. This means that participants who expected the assigned exploratory task to be difficult tended to view more pages, which resulted in more pages saved. In addition, this result indicates that participants who expected the exploratory task to be difficult sought information by reformulating queries multiple times. So, pre-search difficulty was a good predictor for search performance only in the case of the exploratory task.

However, post-search difficulty was significantly related to time spent (r = .66, p < .01), pages viewed (r = .60, p < .01), search engines used (r = .45, p < .05), and query reformulations (r = .39, p< .05) only in the case of the factual task. So the more participants spent time, viewed pages, and reformulated queries to complete the factual task, the more difficult those participants deemed the task. Post-search difficulty was a good indicator of search performance in the case of the factual task.

Reasoning for pre-search difficulty

The selective comments in Table 4 from post-search interviews with searchers show their reasons for the perception of pre-search difficulty. The participants were far more focused when giving reasons for ease than for difficulty, as confirmed in White and Iivonen's (2002) study. They reasoned somewhat similarly about ease and difficulty for perceived difficulty, and they represent opposite ends of the same scale. For example, participants ideas of how much they knew about the topic determined how easy or difficult they thought the search would be; that is, the more they knew about the topic, the less difficultly they assigned to it. When participants said that they knew what information source might include target information, or when they had searched before, or when they thought there would be considerable information available on the Web, they then expected that the search would be easy. On the other hand, pre-search difficulty was rated higher when participants said that they did not know where to obtain information, did not know much about the topic, had never searched that topic before, did not know what search terms to use, or when they thought the search task was unclear.

It is interesting to note that there are different opinions on the type of information needed, especially for the factual task. Some people thought that looking for the name of the oldest seafood restaurant would be easy because the needed information is specific. Subject 009 said, 'I think looking for the seafood restaurant is probably the simplest because it is very specific information (S009-T1)' and subject 014 said, 'It is easy to find information that is concrete (S014-T1).' On the other hand, others thought that it would be very difficult because needed information is too specific. Subject 008 reasoned, 'If you are looking for a very specific one on the Web, it is not easy to find specific information because there is so much information on the Web (S006-T1).'

Reasoning for post-search difficulty

Table 5 summarizes the subjects' responses to difficulty after the search. In general, participants engaged their preception of post-search difficulty only when they thought that the actual search was harder or easier than they had estimated before the search. Many students focused on the problems encountered during the search process, such as specifying a query in a certain site or accessing or navigating a certain site.

The amount of information available was identified as a source of ease or difficulty with searches. Many participants said that it was easy to complete the interpretative task or the exploratory task because there is so much information about those topics on the Web. However, selecting the right information from a large amount of information was difficult for other participants. This was especially observed in the case of the interpretative task which requires the searcher to configure an answer rather than simply locate one. For instance: 'There was lots of information. It took some time to choose the best site (S022-T2).'; 'I expected that there would be quite a bit of information. So, the challenge was finding the best source (S018-T2).'

A few participants mentioned search outcomes as the reason for difficulty, rather than direct reasons. For example, subject 022 responded that the factual task was 'somewhat difficult,' and, when the interviewer sought the reason for that assessment, he said, 'I am not convinced that I got a right answer (S022-T1).' So, it seems that if subjects complete a task, but are unsure about their search outcome, they are then likely to think that the search was actually difficult. Sometimes a lack of quality in information found could be the reason for post-search difficulty.

Discussion and conclusion

While there are many studies on various aspects related to Web search behaviour, this study focused on the perceived difficulty in Web searches. One limitation of this study was the representativeness of the study sample. Like most Web search studies, the present study used a sample from a specialized population, e.g., LIS students, which may not be a representative population of all Web searchers. Selecting this single expert group could control searchers' experiences, which may significantly influence search performance and also affect both external and internal validity. To demonstrate the validity of this study, it would be more meaningful to compare expert searchers with novice searchers in a future study.

Despite this limitation, the present study confirmed that while the perception of difficulty is a personal interpretation or judgment of the complexity of a task, it is not absolute for a given task. This might be interesting to note because in general searching for information about a particular subject has been assumed to be more complex than searching for a specific fact. However, participants in this study had different levels of perceived difficulty in searching for the name of the oldest seafood restaurant: some people said it would be easy because it is specific and it has clear search terms, while others said that it would be difficult because it is too specific to find the exact answer on the Web. It was also found that searchers' perceptions of difficulty are not constant over time. Furthermore, pre-search difficulty and post-search difficulty differ according to types of task.

This invites a subsequent question: What are the factors that contribute to perceived difficulty? This study found factors internal to the searcher, such as previous search experience and topic knowledge; factors internal to the search task, such as specificity of needed information and the quantiity of information sources available; and factors internal to the search, such as assessing or navigating a certain Website and locating a specific piece of information on a certain Web page. It is interesting to note that searchers have different perceptions about factors internal to the search task. For example, the specificity of information needed made some searchers think it would be easy and others think it would be difficult. Likewise, the quantity of information available was the reason of the difficulty to some people but reason for the easiness for others. On the other hand, the reasons of post-search perceived difficulty across all three types of task focused primarily on the problems encountered during the search process, which are identified as factors internal to the search. Surprisingly, some of the participants explicitly conveyed satisfaction with search results as a reason for difficulty.

These results support the proposition that perception of difficulty itself is complex, and, furthermore, that the perceived difficulty involves factors intrinsic to several contexts, and that the treatment of difficulty can be classified at several levels. However, they did not explicitly show Campbell's (1998) interaction between the task and the task performer's characteristics. Such interactions are hard to capture from real searchers' behaviour but they should be considered. For a better understanding of perceived difficulty, future studies should focus on analysing how difficulty is related to other factors, such as topic familiarity, self-efficacy, prior experience, and knowledge.

Acknowledgements

This work is a part of the author's doctoral dissertation and the preliminary results of quantitative analysis were reported earlier (Kim 2006). The author thanks Dr. Nick Belkin, her advisor, and Dr. Tefko Saracevic, Dr. Colleen Cool and Dr. Dan O'Connor, her committee members.

Bookmark This Page

Appendix

Task types	Task scenario
Factual task	You plan to visit San Francisco next week. One of your friends who has been there suggests that you visit the restaurant that is known as the oldest seafood restaurant in town. You want to know the name of the restaurant.
interpretative task	Your cousin, who is a typical teenage girl, said that one of her friends started to smoke. You fear your cousin might begin smoking in the near future. So, you decide to educate her. Specifically, you think that you have to find some information focusing on what could happen if she starts smoking.
Exploratory task	You have recently moved to Boston and you are interested in buying a home. You have heard that most homes built before 1978 have some lead paint, but that their paint status is often reported as "unknown." You think you should learn about lead paint and housing. The Web seems like a good place to locate this information.

---

© the author, 2008. Last updated 10 December, 2008

online Web site analytics

---

Contents | Author index | Subject index | Search | Home