Mapping the development of user constructs of relevance assessment as informed by topicality.

Theresa Anderson
Department of Information Studies
University of Technology
Sydney, Australia

Introduction

This paper reports on research examining the dynamism of a �user' view of topicality exhibited in the information retrieval experience of individual searchers. Experienced researchers presenting complex search requests to an electronic information retrieval system are observed. The information retrieval activities of an individual searcher are situated within the complex and constructive information seeking process. Viewed holistically within this process, that individual's relevance assessments are social constructions which are fluid, interactive and dynamic. To explore relevance and topicality within the broader context of information retrieval, the searcher's full experience must be recorded and evaluated. This paper discusses preliminary findings from pilot testing attempting to 'capture' this broader context. Within a qualitative framework, I will explore evolving views of 'topic' throughout the individual information seeking process.

Thesis summary

Relevance is a fundamental concept in information science which, despite more than forty years of debate, is still not fully understood. Notions like topicality, pertinence, or situational relevance have appeared in the literature over time, but support for one view has often meant excluding all others. For example, relevance is currently recognised as a multidimensional and dynamic concept extending beyond the traditional definition so closely associated with topic-matching and an identification of relevance as the relation between a document and a search question (Schamber et al, 1990). With this user-centred shift, topicality has been dispensed with even though criteria derived from users in real search situations show that subject content and related qualities remain important. I believe that the notion of topical relevance is more dynamic than thus far considered in much of the literature. Green and Bean(1995), for instance, contend every imaginable relationship

My thesis will focus on the interactive construction of meaning attributed to the 'topic' of an individual search. A 'symbolic interactionist' framework will allow me to explore the context and the process behind an action -- in this instance, relevance assessment. In this way I hope to get closer to the human factors associated with information retrieval and the search process. Symbolic interactionism recognises that people are constantly involved in processes of interpretation and definition as they move from one situation to another. Individuals shape the meaning and action associated with their relevance judgements through unique experiences and interpretations (Blumer, 1969; Denzin, 1989). Meanings of 'relevance' change throughout an information search as a result of their encounters with people, things and ideas (Cool, 1993). Using the symbolic interactionist framework, I can view information retrieval as social communication. Relevance will be defined as a "social object" and relevance

Relevance, information retrieval and the search process

Information retrieval is viewed as a communication process in which judgements of relevance are made. The judgements upon which my research intends to focus are those made by the individual searcher performing information searches in an electronic environment. Individual views of 'topic' are shaped through the information seeking process, making the search a learning process (Blumer, 1969; Kuhlthau, 1991). The notion of 'topic matching' may be a useful method of identifying documents held within a retrieval system. However, seeking out the human meaning of 'topical relevance' requires a better understanding of the learning process associated with individual relevance assessments. The articulation of what is 'on the topic' for a particular search and the selection of 'relevant' information are influenced by interactions during the information retrieval process as well as the broader information search process. This thesis aims to develop a thorough exploration of those user constructs in the context of

Previous research of user-derived relevance criteria suggests that all the criteria derived from the user perspective are inter-related (Barry, 1994; Barry & Schamber, 1998; Park, 1992; Schamber, 1991). These earlier inquiries asked individual searchers to make judgements of criteria directly associated with a document. Recognising the interrelationships of relevance criteria, however, means relevance must be explored by charting the whole information retrieval experience. From a searcher's perspective, 'topicality' is more about the articulation of a particular topic and the information answering the specific inquiry. Thus, my inquiry examines those interrelationships by focussing on user constructs of 'topic' without reference to a particular document.

My research applies an ethnographic approach to study information retrieval and record the 'experience' using human-computer-interaction facilities. Given the unique character of each person and their individual search, any attempt to understand how an individual derives topicality priorities must explore all phenomena influencing that person's relevance assessments during their information search. A naturalistic inquiry is required to study the complex interplay of all elements in a real�not contrived�context. Recognising that human activity is not context free, a naturalistic inquiry seeks out all factors involved in interaction (Guba & Lincoln, 1985; Erlandson et al, 1993; Mellon, 1990). Understanding the meaning of 'topical' for a user will require a thorough understanding of the context within which these individual decisions are made. In my research, developing an understanding of changing topicality priorities involves three elements:

detailed explanations about each individual's approach to information evaluation in a real (personal) situation (to elicit relevance assessment criteria from users);
direct observation throughout each subject's search process; and
observation of personalised information seeking sequences within each individual's framework of evaluation and use.

This research attempts to study relevance by absorbing and integrating theories from associated disciplines and bringing to the foreground the notions of human behaviour which have been shown to be so significant in the user-centred view of information retrieval (Barry, 1994; Park, 1994; Schamber, 1994; Sugar, 1995). Notions from computer science, ethnography and sociology, for instance, enrich the exploration of human factors in electronic information environments (Monk & Gilbert, 1995). Such an approach suggests a bridge between research on the information search process and user-centred relevance criteria. By applying other frameworks, a new means for evaluating and critiquing relevance criteria can be derived.

Work plan and progress to date

My candidature began in early 1997 and I am due to complete the degree by December 1999. I recently completed pilot tests of my data collection method. My work plan uses six-monthly reports as progress milestones, to ensure that I can complete my research on time. The stages of research and target dates for completion which I have hitherto met are: literature review and research question (April 1997 - January 1998); establishment of research framework (April - November 1997); pilot testing of method (December 1997 - April 1998); data collection (Phase One, November 1998; Phase Two, early 1999); preparation of transcripts, summarising data, coding (November 1998 - March 1999); interpretation and analysis (November 1998 -October 1999); final revisions (November - December 1999).

I recently completed method testing which involved observing the search behaviour of two individuals. This experience clarified the data collection strategy, which is the focus of my work for the next few months. Asking questions and engaging in conversation with participants about the individual's information retrieval practices, decisions, thoughts and reactions during searching proved enlightening and in fact essential to understanding the context of the searches.

The first stage of data collection will involve detailed observation of three participants. An emerging theory about user constructs of topicality will be tested in the second phase of data collection with three or more additional participants later this year. I am presently arranging access to a group of experienced researchers who regularly conduct searches in electronic environments. Their search process will be recorded from first interactions with an electronic database through to decisions to take actual items or documents away for further review or study. The interaction will be recorded using video and audio facilities in a human-computer-interaction laboratory. Screen activity will be recorded electronically. Discussions with participants during their searching will be used to develop the case studies about individual views of 'topicality.' Questions will revolve around each person's criteria for selecting and rejecting items judged 'relevant' by the retrieval system in use.

Detailed case studies in the first phase of the research will describe not only the criteria used to select items but also assumptions (as expressed by the research participant) that influence individual decisions to select or reject an item. All changes in the search query and evaluation criteria will be recorded and explored with each participant. Observation of a search will finish when the subject decides to take a book out of the library, photocopy an article, or download a document identified as 'relevant' to the search. The language of each individual searcher's interaction with an information retrieval system will be used to describe the process. By encouraging participants to 'think aloud' during their searches, I hope to develop a better understanding of the individual context of relevance judgements.

Applying naturalistic inquiry methods

A significant stage in the development of my research design has involved establishing the 'natural' context of the phenomenon under investigation and determining how and where data can be collected. In their discussion of relevance, Sperber & Wilson (1986) note that communication involves inferring meaning from the context in which it occurs. Human-computer-interaction research and usability studies suggest that observations of the system-searcher interaction in a computer facility can capture a 'natural' context (Blomberg, 1995). While the individual interactions with information on the screen and not the system interface are the focus of my inquiry, attempting to 'capture' the full information retrieval experience can benefit greatly by following the example of usability research to conduct research in a computer search facility. A workable replication of the 'natural' context of the individual search experience can be created if:

searcher-system environment is similar to the individual's usual environment;
search problem is a 'real' question for the individual, not imposed by researcher;
the task(a computer search) is not performed in isolation from the process of information retrieval.

While conceding that this environment cannot be considered the 'natural' environment of the individual searcher, use of a computer search facility appears to preserve the main elements of the searcher-system interaction. It is particularly important that the individual is still able to pursue a 'real' search problem � posing search questions and using databases of her own choosing. Moreover, much of the physical context of the individual's search can also be preserved in such a facility. Computer equipment can be arranged to mimic speed and interface, for instance. Observing the information search in this environment facilitates recording of search requests and all possible modifications from the first request through to the selection of 'suitable' information for searcher. Recording interactions using the video and audio facilities of such a facility also simplifies analysis of screen activity as well as discussions with participants during their searching. This approach thus offers a greater capacity

Pilot testing

The current focus of my research has been devising a method for observing how people interpret 'topic' during information retrieval. Instead of exploring evaluation criteria directly associated with a document, this inquiry aims to:

examine the whole process of shaping the topic;
describe the communication between system and searcher regarding that 'topic';
relate user constructs to a system description of 'topic'.

In a pilot test, conducted in June 1998, the data collection method that seemed to address these aims was evaluated for suitability.

Attempting to 'capture' the full retrieval experience of the participant involves discussions with a participant during the individual search. The pilot was divided into four sections:

interview just prior to the search
search session
post-test interview
subsequent review of retrieved items.

The interview with the participant was used to learn what was 'known' about the subject of the search and to gain insight into his experience, expectations, and attitudes to electronic searching. The search was conducted in a computer facility � recording all activity on screen and all discussions with the participant. The searcher was encouraged to discuss thoughts, reactions and actions during the search process. In addition to pursuing the individual's search process, the post-test interview explored the searcher's reactions to the test. The review of retrieved items involved asking about the items related to the desired topic. The first three stages were conducted consecutively over a period of four hours. The final stage was conducted at a later date.

Recording of this search session was linked to a usability software package known as DRUM- Diagnostic Recorder for Usability Measurement. The tool helps organise and analyse activities during the search session, by facilitating the logging of significant 'events' during the test. Preliminary evaluation of pilot test results suggests a valuable link between naturalistic inquiry of information retrieval and established usability testing methods.

Preliminary findings also indicated there are indeed significant benefits to observing searches of individuals in this environment, namely a computer search facility. The pilot participant, an experienced researcher who is a frequent searcher of electronic databases, appeared comfortable with conducting his search in the facility. The databases and interface were no different from that of his office workstation. He was eager to begin his search and appeared unfazed by the video and audio recording equipment. In the interview and discussion after the search session, he made no comment about feelings of discomfort or unease.

During the computer search session, the participant voiced his thoughts and described his actions at stages of the search. In addition, he was asked to explain individual actions throughout the session. Despite this potential for disrupting the 'natural flow' it was also evident that the dialogue between researcher and participant was necessary for identifying the individual's thoughts, reactions and actions throughout the search. During the search session, there were many occasions when the searcher's evaluations and selection decisions were markedly changed. To better understand this dynamism, it became imperative to ask the searcher to explain actions and choices. This provided a valuable discussion about the participant's search behaviour and selection decisions.

Capturing relevance assessment in the broader information retrieval experience

In my research, relevance assessment is related to contextual factors of information seeking in electronic environments. My exploration of the interaction of an information seeker with an electronic information retrieval system extends beyond the initial interaction to include the broader context of information retrieval � one which situates the activities observed within a search process involving much more than electronic information retrieval of information. My preliminary findings suggest that studies of user-centred relevance which address evaluation criteria only after a particular information retrieval system has been selected are not adequately addressing the concept. Many decisions about 'relevance' are made even before the search begins. To explore relevance and topicality within the broader context of information retrieval, the searcher's full experience must be recorded and evaluated. Recognising the significance of context also suggests the importance of situating individual information re

References

Barry, C. L. (1994) "User-defined relevance criteria: an exploratory study." Journal of the American Society for Information Science 45, 149-159.
Barry, C. L. & Schamber, L. (1998) "Users' criteria for relevance evaluation: a cross-situational comparison." Information Processing and Management 34, 219-236.
Blomberg, J. L. "Ethnography: aligning field studies of work and system design." In A. F. Monk & G. Nigel Gilbert (eds) Perspectives on HCI: Diverse Approaches. London: Academic Press. pp.175-197
Blumer, H. (1969) Symbolic Interactionism: Perspectives and Method. Englewood Cliffs, NJ: Prentice-Hall.
Bogdan, R.C. & Biklen, S. K. (1982) Qualitative Research for Education: An introduction to theory and methods. Boston: Allyn & Bacon.
Cool, C. (1993) "Information Retrieval as Symbolic Interaction: Examples from Humanities Scholars." 56^th ASIS Annual Meeting: 274-277.
Denzin, N. K. (1989) The Research Act. Englewood Cliffs, NJ: Prentice-Hall.
Eisenberg, M. & Schamber, L. (1988) "Relevance: the search for a definition." In ASIS '88: Information and Technology: Planning for the Second Fifty Years. Proceedings of the 51st ASIS Annual Meeting held in Atlanta, Georgia on October 23-27, 1988. Volume 25: 164-168.
Erlandson, D.A., Harris, E.L., Skipper, B.L. & Allen, S.D. (1993) Doing naturalistic inquiry. A guide to methods. Newbury Park, CA: Sage.
Froehlich, T.J. (1994) "Relevance reconsidered - towards an agenda for the 21st Century: Introduction to Special Topic Issue on relevance research." Journal of the American Society for Information Science 45(3): 124-133.
Gluck, M. (1995) "Understanding performance in information systems: blending relevance and competence." Journal of the American Society for Information Science 46(6): 446-460.
Green, R. & Bean, C.A. (1995) "Topical relevance relationships. II. An exploratory study and preliminary typology." Journal of the American Society for Information Science 46(9): 654-662.
Kuhlthau, C. C. (1991) Inside the search process: Information seeking from the user's perspective. Journal of American Society of Information Science 42(5): 361-371.
Lincoln, Y. S. & Guba, E.G. (1985) Naturalistic inquiry. London: Sage.
Mellon, C. A. (1990) Naturalistic Inquiry for Library Science: Methods and applications for research, evaluation, and teaching. Contributions in Librarianship and Information Science, Number 64. New York: Greenwood Press.
Mizzaro, S. (1997) "Relevance: the whole history." Journal of the American Society for Information Science, 48, 810-832.
Monk, A. F. & Gilbert, G. N. (eds) (1995) Perspectives on HCI: Diverse Approaches. London: Academic Press.
Park, T.K. (1994) "Toward a theory of user-based relevance: a call for a new paradigm of inquiry." Journal of the American Society for Information Science, 45, 135-141.
Park, T.K. (1992) The Nature of Relevance in Information Retrieval: An Empirical Study. Doctoral dissertation, School of Library and Information Science, Indiana University, Bloomington, IN.
Saracevic, T. (1996) "Relevance reconsidered '96." In P. Ingwersen, N.O. Pors (Eds) Information Science: Integration in Perspective � Proceedings of CoLIS2. Copenhagen: Royal School of Librarianship.
Schamber, L. (1994) "Relevance and Information Behavior." Annual Review of Information Science and Technology 29: 3-48.
Schamber, L.(1991) Users' Criteria for Evaluation in Multimedia Information Seeking and Use Situations. Doctoral dissertation, Graduate School of Syracuse University, Syracuse, NY.
Schamber, L., Eisenberg, M.B., Nilan, M.S. (1990) "A re-examination of relevance: toward a dynamic, situational definition." Information Processing & Management 26: 755-776.
Sperber, D. & Wilson, D. (1986) Relevance: Communication and Cognition. Oxford: Basil Blackwell.
Sugar, W. (1995) "User-centred perspective of information retrieval research and analysis methods." Annual Review of Information Science and Technology 30: 77-109.

Information Research, Volume 4 No. 2 October 1998
Mapping the development of user constructs of relevance assessment as informed by topicality, by Theresa Anderson
Location: http://InformationR.net/ir/4-2/isic/anderson.html © the author, 1998.
Last updated: 25th September 1998