Information Research, Vol. 6 No. 2, January 2001
Information Research, Vol. 6 No. 2, January 2001 | ||||
|
|
|
|
|
The aim of the paper is to compare the philosophies of the vignette and critical incident techniques in information behaviour research, with the methodologies used in object oriented analysis such as use case scenarios and CRC (class, responsibility, collaboration) cards. The principles of object oriented analysis are outlined, noting the emphasis on obtaining the "storyline" or "scripts" for information requirements analysis through use cases and CRC cards. The critical incident technique and vignettes are used to obtain valid interpretations of users" information behaviour, using a storyline approach for data collection (and analysis) which is similar to that of object oriented analysis. Some examples illustrate how techniques developed in object oriented analysis could be used for data display in information behaviour studies. Concludes that the methods developed by software engineering could be adapted usefully for information behaviour research.
With the growing emphasis on qualitative research in studies of information behaviour, any relationship between information behaviour analysis and the structured approach to information requirements analysis of conventional systems analysis seems remote. Information behaviour research is concerned with a deeper understanding of information needs and use, through sense making as proposed by Dervin (1986) or, for example, features, as proposed by Ellis (1989).A later integration by Wilson (1999) of the research in the field proposes a problem solving model which considers uncertainty, though in a slightly different manner from the phenomenological viewpoint of Kulhthau"s view of uncertainty (1991). Conventional systems analysis of the hard structured type (for example SSADM, Structured Systems Analysis and Design Methodology) approaches information requirements analysis with the aim of providing three views of the data. There is a formal role for the "user" in SSADM, but this "user involvement" may in fact be limited. Structured methodologies can be criticised for their failure to involve the "user", or fully comprehend their possible role. Inevitably, possibly the "user" role is often that of a nominal user representative. It might appear that these apparently very different schools of thought have very little to learn from each other in the methods that are used to investigate, analyse and "display" (Miles & Huberman, 1994) the data collected in their investigations of user behaviour. New developments in object oriented analysis methods have similarities to the methods that might be used to investigate information needs and use in information behaviour studies. For example, object oriented techniques such as use case analysis and the development of use case scenarios are similar to the "critical incidents" and vignettes used in studies of information behaviour (Urquhart & Hepworth, 1995) (Davies et al, 1997) (Urquhart, 1999). The underlying philosophy is similar though the physical manifestation may be different. For those using the techniques there may be opportunities to learn from the other school of thought, particularly in the way the analysis of the data can be displayed - sometimes a difficulty for qualitative researchers when attempting to explain their analysis convincingly to those who are more attracted to quick visual overview rather than reading a long textual account. The quality of qualitative research is often judged on the extent to which the results are credible. Enough raw data should be provided to give that evidence but the number of words produced in most qualitative research means that data should be condensed in a way which makes sense to those reading the research reports. Charts and diagrams often help to focus a reader on the main themes which emerge from the data.
Object oriented analysis aims to overcome some of the problems of the structured methodologies (such as SSADM), which can become a rigid straitjacket on systems development. Efforts to reduce problems and redesign often appear to have the effect of discouraging the re-investigation of changing user needs. Soft systems methodology, as developed by Checkland (1981) provides a general approach to the real, and messy problems of management in many organisations. Checkland & Holwell (1998) argue that information systems are more than "data manipulation" systems and that the study of the human activity systems is vital to an understanding of what an effective information system might look like.
For systems developers, communication between analyst and the system users is important and "sharing a language" (Crowe et al., 1996) more than just a common understanding of the meaning of technical terms within the organisational domain. There are subtleties of meaning and interpretations of situations which need to be considered, and there are likely to be many differences between the "formal rules", and what people actually do, and between what they think they are expected to do, and what they would prefer to do. A study (Davies et al., 1992) of an academic library illustrated the tensions that governed communication. Information systems designers who ignore these disparities do so at their peril.
Understanding the "stories" that are told within the human activity systems is thus an important part of systems analysis though the extent to which this analysts formally document this varies. Often the possible "scenarios" are treated informally rather than formally as more effort is put into formal data analysis.
One approach to object oriented analysis uses the concepts of roles and interactions. Object Requirements Capture and Analysis, ORCA, (McLean et al.. 1994) is based on much the same premise as Checkland"s Soft Systems Methodology (1981) in that analysis should not be based on a statement of initial requirements. The specifications require an understanding of the Old World, identification of problems and their causes, before proceeding to proposing justifiable changes for a New World.
Jacobson (1992) developed a more formal approach to scenarios with his Objectory method for object oriented technology. In this, the "use case" is introduced as a more formal method for requirements capture and initial planning. Use cases are usually associated with use case diagrams, and use case scenarios (and almost inevitably there are some variations in the ways these terms are interpreted and applied in practice). Fundamentally, they provide a way of depicting and structuring the interaction of a real world Actor (person, organisation or external system) with the application being studied. The Actors are users of the system, and they have goals, or what could be viewed as "things that must be done". For a retail system the goals or use cases concern the "ordering of goods", "delivery of goods" and "payment for goods". The scenarios can be divided into the success scenarios, when all goes according to plan, and the extensions deal with use cases where there is a problem. The company may be out of stock, or the customer may wish to return the goods, or the payment method may be different. Sometimes it is important to distinguish between system interactions and actual user goals, and deal with goals first, as suggested by Fowler (1997). For information service users the goal may be to obtain a document: the system interaction may concern the inter-library loan procedures. There should be a sequence of interactions in a compound system interaction (and requesting an inter-library loan would be a compound interaction) and that sequence would be the basis of a scenario. For Cockburn (1997) the key feature of a scenario is the absence of alternatives, and this definition of a scenario appears to relate closely to the Object Behaviour Analysis "script" as described by Rubin & Goldberg (1992).
One similar structured approach to user involvement in the process of object-oriented development is the use of CRC cards - Class, Responsibility and Collaboration cards. The technique uses small teams, including users who are "domain experts", with the aim of discovering the objects, the classes that the system comprises, their responsibilities (what they do) and their collaboration, how they interact with other objects in the system. The technique is aimed at a better, deeper understanding of the problem situation so that the information system developed provides a proper solution. Brainstorming is used to derive provisional classes and behaviour and role-playing is then used to work through some scenarios, "living the system" according to Bellin & Simone (1997). The use case methods have been developed in various other ways, e.g. task scripts, by Graham (1996), textual use case specifications in natural language as formulated by Rolland & Ben Achour (1998) and flowgraphs to illustrate control flow semantics, proposed by Van den Berg & Simon (1999).
What all these ideas on object oriented technology share is an emphasis on the storyline - seeing the information system in terms of the events and interactions, and the sequences which tell a meaningful story to the system users. There is an emphasis also on identifying, and discovering what happens, or should happen when co-operations are faulty or the use cases are "extended" or become "abuse cases".
There is an emphasis on object oriented analysis on what Schank & Abelson (1977) term "scripts", usual patterns of interaction, and Graham (1996) has developed the idea of task scripts (for completion of goals) and side scripts (the stereotypical exceptions). Information retrieval research has also used the "script" concept in development of the user interface. In the model of information retrieval design proposed by Belkin et al. (1995)scripts are proposed as prototypical interactions. The scripts are construed as a pattern of moves, or "abstract" dialogues, the purpose of this research being the production of a better interactive information retrieval system which will respond appropriately to the stage the user is at in the search. There are four proposed dimensions to any information seeking strategy: method of interaction (scanning/searching), goal of interaction (learning/selecting), mode of retrieval (recognition/specification) and resource considered (information/meta-information). Information seeking strategies are then represented as example scripts, a series of moves required to achieve desired results, and for each strategy there will be a arrangement of goals which organise the set of moves. The problem, of course, is to predict the likely combination of scripts which will correspond to the switches made by real users from one information seeking strategy to another.
Much of information systems analysis in organisations is based on the premise that users will routinely be using the information system in their work, and that these systems will be "operative" rather than "supportive" as defined by Malmsjo (1999). Many information systems of interest to library and information science professionals are "supportive" for users rather then "operative" and this distinction affects the approaches that have to be taken to understanding user behaviour. We choose whether or not to use such "supportive" systems and often we choose whether or not to make use of the search results obtained, for example. Actual system use may thus be a small proportion of potential use, and the emphasis in information behaviour research may be on the patterns of information need and use and the perceived preferences for certain information sources as opposed to others. Similarly, we may obtain information, in the form of the output of a database search, but the actual impact or use in practice may be hard to assess, as the actual use may be some time in the future.
Mapping possible patterns of use is difficult with so many variables affecting use or potential use. Smithson (1989) examined the contribution of cited references in students" dissertations of an online search found that a small (and individually variable) percentage of documents retrieved actually contributed to the students" dissertations. Some of the problems concerned the users" inability to find the documents to make a final judgment. Wang (1996) obtained similar findings in a study of research students and faculty working on a research project, and suggests that subtle changes in information needs while working on a project can be mapped by changes in cognitive structures. Needs and estimations of relevance alter in some way and the success of a search, whether done by an intermediary or by the user, will be hard to judge in terms of the contribution made to a list of references. On the other hand, such objective evidence of effective system use (or ineffective system use) may be an indicator of the success (or not) of a training programme, and more meaningful when training is delivered informally, rather than in formal classroom settings. In that case it may be necessary to track trends in time and consider the behaviour of a cohort, rather than the behaviour of individuals within a cohort. One aim of the GIVTS project (1999) was the evaluation of the effectiveness of Internet training on evidence-based sources through analysis of the references cited in audit project reports, which form part of the summative assessment required for general (family) practitioner trainees in the UK. It proved impossible to evaluate the training itself, but investigation of the data provided (titles of projects plus references) did show variations in the emphasis placed on quality sources of information. There are many factors involved - the formulation of questions around the clinical problem topic, the emphasis placed on the need for a systematic review of the literature, accessibility of databases, development of an appropriate search strategy, skills for searching, and critical appraisal skills. It is somewhat paradoxical that systems which are "supportive" may be anything but supportive in practice for the inexperienced and infrequent user. If the system is supportive, and all the external conditions are right, then the user can negotiate an effective route for information seeking, assessment and use. The problem is that of co-ordination and Spink (2000) emphasises the importance of the information feedback loop as the basis of human information co-ordinating behaviour.
Studying feedback implies that studies of information behaviour should include longitudinal studies, such as the "Uncertainty" project (Wilson, 1999).The problem for longitudinal studies is the time required, and, related to that, the risks of a high drop-out rate. For example, of 34 video-recorded (with concomitant field observations) users of OPAC machines in one study, only four of the subjects for the recorded cases took part in the longitudinal study (Zeitlyn et al., 1999).
Identifying the longitudinal development of information seeking behaviour is not, apparently, an easy task, though there have been some studies which track the behaviour of an individual (e.g. a case study by Kuhlthau (1997) of an individual, starting from secondary school, through university and to work as a securities analyst). Those developing and funding information services would like to predict with more accuracy the likely take-up of new services, the barriers which might have to be overcome to make the service truly supportive and the opportunities which the users perceive, and learn to perceive, from their perspective, of the services. From the marketing point of view, information managers need to assess the likely segments of the market for services.
In view of the difficulty of studying information behaviour over a long period of time, or what might be termed the whole story, other techniques have been used to capture brief episodes in the story. This section discusses some examples of these techniques and their applications, showing some similarities between the general philosophy of this approach and the use case scenario ideas of object oriented analysis.
One technique which may be used to help identify patterns of use or potential use, is the critical incident technique, which examines a brief, but memorable information seeking episode. It is therefore useful for examination of the occasional, or at any rate non-compulsory, use of "supportive" information systems. Often the incident is explored through an interview, but it is possible to use a questionnaire format. In the Value project (Urquhart & Hepworth, 1995) the technique was used to examine the patterns of information seeking behaviour among groups of doctors. A critical incident questionnaire was sent once a week for four weeks to the sample (just under 300 medical staff (hospital staff and general practitioners) at 11 sites) asking, for one incident in the previous week, the purposes for which information was sought, the sources used, and the degree of success obtained. A similar technique was used in the EVINCE project (Davies et al., 1997) to analyse the patterns of information behaviour among nursing, midwifery and health visiting staff. The type of questions which could be answered through this application of the critical incident technique were:
What are the purposes associated with use of colleagues (informal sources of information)?
Results for medical staff indicated clearly that personal education and updating was often associated with the need for information on a rare condition or specific patient problem. Nursing staff appeared to use colleagues as information sources more than the doctors did, with informal updating linked in this case more usually to drug therapy queries. The nurses" pattern was more complex than that of the doctors, reflecting their needs for information for teaching and mentoring colleagues, patient education and patient care administration.
The results could thus be analysed to show the patterns of purposes associated with particular sources of information, and, similarly the patterns of sources associated with particular purposes of information need. Both surveys demonstrated convincingly that the library use was closely associated with formal and informal continuing education needs, and research, rather than patient care problems on their own. For doctors, use of a database such as MEDLINE was primarily associated with personal research needs, while for nurses computer based resources such as the CINAHL database was primarily for formal course work.
The critical incident technique may also be used to profile groups of users, as was possible with the Value project results. For EVINCE, various groupings were attempted but there were few significant differences between the groups, suggesting either that the sample sizes were too small to show an effect (possible) or that differences in information behaviour among nursing staff are likely to be at the individual level, not at group level. One could not therefore predict the likely patterns of information behaviour on the basis of the assigned task role and place of work (hospital versus community, for example).
Critical incidents, to some extent, represent information use "stories" which are complete or near complete. Replication of the techniques in other setting provides a means of tracking changes in information behaviour of the groups in general. Bradshaw (1997), for example, adapted the Value and EVINCE methodology for a study of networked CD-ROM services in one hospital setting, and extended the methodology to cover social workers, and clinical psychologists. The findings suggested, for example, that networking did affect the patterns of use, but that the motivation for use (education (course work for nurses) and research) remained similar to the Value and EVINCE results.
Vignettes are short scenarios presented to subjects, who are then asked to describe their possible actions in those circumstances (or else to describe how they think others would react in the circumstances). Vignettes have been used to prompt information users into revealing their information seeking strategies and choice of information sources (Urquhart, 1999), when direct questioning might be less productive. Van de Wijngaert (1999) has used a similar technique, policy capture analysis, to examine media choices among students. The aim of vignettes used in this way is to present a realistic situation with which the subjects can identify. Similar aims appear to have led to the inclusion of a simulated work task situation as one of the sub-components investigated for an interactive information retrieval framework by Borlund (2000). The effectiveness of the simulated work task situation was compared with the simple indicative request and the subjects" own information needs. Results suggest that the simulated work task situation produces the same behaviour as the real information needs, although subjects spent longer on their own needs. In comparing the simulated work task situations with the indicative requests, a more mixed picture emerges, and this reflects the observations that some EVINCE vignettes worked better than others, generally when the subjects had some experience of the situation, or the topic was of universal interest (Urquhart, 1999), a finding which Borlund confirmed for their evaluation setting.
Vignettes may also be used to present data analysis, an example in the information studies field being the narrative "week in the life of" a social services department in Project INISS (Wilson & Streatfield, 1980).The aim is to condense the mass of textual information into a meaningful display which should fulfil several functions, but notably one of allowing the reader an insight into how the data was analysed in a particular research context. Vignettes may be used iteratively to help the researcher and subject agree on a common understanding of the situation (e.g., Greenhalgh et al., 1998) and for use cases in systems development that common understanding is the aim. To develop an information system, however, the interest is not simply in the "average" or "representative behaviour as might be expressed in the primary of "happy day" scenario but also in the secondary or alternative scenarios which deal with the exceptions to the primary scenario. Use case diagrams have rules for how to display the extensions to the primary use case, and in some complex conditions it may be better to draw the secondary scenario separately (see Schneider & Winters, 1998, for example).
Formal scenario writing, with the amount of detail required to document the flow of events, is possibly not a technique that can be used in its entirety in studies of information behaviour but there are ideas that might be borrowed and extended in a different way. Documenting and displaying information seeking pathways in use case type diagrams, showing optional sequences of events, can help debate these issues. Too often users" information seeking problems are seen solely in terms of "access to resources" but users" reasons for seeking new information (or not), and their estimation of the benefits to be obtained are often far more complex (Urquhart, 1999). Where there are choices to be made, with a plethora of electronic information sources available, influencing events and optional sequences can be mapped using use case diagram ideas, and some of the search strategies used by students in the JUSTEIS project, First Cycle (2000, Section 6.6)
The figures (Figures 1, 2, 3) illustrate some of the alternative and supplementary routes chosen by students when their search route demands that. In Figure 1 "update using NASA web site" is a regular search scenario which supplements the primary scenario of finding course notes on the Web to complete a laboratory report. The student also uses the library for background reading and books, to obtain alternative references that may be unique (that other students would not obtain).
Figure 2 illustrates that the process of searching for recent journal reference, using BIDS ISI was prompted by an IT course given to the student on BIDS. The student scans, and reviews the results of the search, and if the selected journals are not available, repeats the search by scanning using different keywords. Occasionally a search of electronic journals (full-text) contributes to the review stage.
In Figure 3, the failure of a link led the student to revert to emailing personal contacts.
These Figures represent preliminary work on the data display, and would be refined in the light of further data analysis and discussion. Such diagrams can be used to illustrate graphically to those involved in training in use of electronic information services just what students do in the process of searching, the ways in which they might revise their searching strategies (JUSTEIS 2000, Section 6.2.3), and the way in which advice from academic staff and LIS staff is sought, and acted on, in the process of the search. Diagrams which show by the "extend" notation which other 'scripts' are used in the searching process help to make sense of a vast mass of textual data from interview transcripts. Such real-life examples provide a valuable feedback into training needs analysis for library and information services, as well as showing graphically the actual search strategies that students adopt. It is a more informative way of representing findings than merely reporting the relative popularity of various sources. There are some limitations, of course, in that some students' searching strategies are (apparently) rather muddled and constraints on their time limit the alternative routes they are willing to try. Even so, the strategies are worthy of some more formal analysis.
Focus groups are now frequently used in LIS research, often with the aim of encouraging participation and stimulating thought to providing unexpected insights and contributions that might be less easy to obtain in a one-to-one interview (Goulding, 1997). One assumption of focus groups is that the group effect will produce more details about information behaviour than might be obtained from one-to-one interviewing. Similarly, the brainstorming used in the CRC card technique is used on the basis of the benefits of group problem solving. It is debatable whether brainstorming does in fact produce more ideas about a problem than might be obtained from the group working individually (Buchanan & Huczynski, 1997) but for the purposes of information needs or information requirements discussions, CRC cards or focus groups may encourage a shared understanding of the concepts, the events and procedures that may be involved in an information system or situation. Using a modified CRC card technique may help, for example, in structuring focus group data to illustrate the responsibilities and roles of the stakeholders in an information seeking situation. The CRC technique is aimed, of course, at system development and therefore make use of documentation that states system requirements, reports generated, as well as interviews with users. The main function of CRC technique is to examine the system behaviour and ensure that the developers do discover and define the right classes required for software development. The technique makes extensive use of brainstorming and role play. One aim of the role play is to find the false assumptions, the bits of the system that do not work together in quite the way some people might assume. At this stage of the JUSTEIS (2000) work one can only make speculate what the results of the second cycle will add to the first cycle findings, but there are indications from the data that there is some degree of mismatch between expectations of patterns of service use and actual use of information services. The structured role play that CRC involves could be adapted to the training situation, so that focus groups could first investigate the situation generally, and then in a second round of focus groups analyse exactly what might be happening (or not happening) through structured role play.
Techniques such as the critical incident technique and vignettes are used at various stages in the data collection and analysis of information behaviour. Such techniques share, to a greater or lesser extent, the idea of a "story" or "sense-making" storyline, which is useful in either eliciting details about information behaviour from users or displaying data meaningfully. Asking users for details of their experience in an information seeking incident, or presenting them with a realistic scenario, seems more likely to elicit a more valid interpretation of information behaviour than might be obtained from transaction logging or asking about frequency of information source use. Similarly, analysing observational data using the vignette provides a meaningful method of data display which can be discussed with users and then refined if necessary.
More recent developments in information requirements analysis in software development in object oriented technology are working on parallel lines, though there is little sign of any cross-fertilisation between ideas and techniques used by researchers working in information behaviour and and those used by systems analysts. This paper has shown the similarities in the approaches, despite the differing provenance (software engineering and information science). The CRC card techniques, and the use case diagram techniques used in object oriented analysis may be useful in helping to structure and present data in information behaviour studies, and may help communication with the research subjects. Such structured techniques from the field of systems analysis might help overcome some of the problems of data display for qualitative research in information seeking and use.
How to cite this paper:
Urquhart, Christine (2001) "Bridging information requirements and information needs assessment: do scenarios and vignettes provide a link?". Information Research, 6(2) Available at: http://InformationR.net/ir/6-2/paper102.html
© the author, 2001. Updated: 19th December 2000
Contents |
|
Home |