Information Research, Vol. 3 No. 4, April 1998
Information Research, Vol. 3 No. 4, April 1998 | ||||
|
|
|
|
|
This paper reviews the public reception of the Research Assessment Exercise 1996 (RAE) from its announcement in December 1996 to the decline of discussion at end May 1997. A model for diffusion of the RAE is established which distinguishes extra-communal (or exoteric) from intra-communal (or esoteric) media. The different characteristics of each medium and the changing nature of the discussion over time are considered. Different themes are distinguished in the public reception of the RAE: the spatial distribution of research; the organisation of universities; disciplinary differences in understanding; a perceived conflict between research and teaching; the development of a culture of accountability; and analogies with the organisation of professional football. In conclusion, it is suggested that the RAE and its effects can be more fully considered from the perspective of scholarly communication and understandings of the development of knowledge than it has been by previous contributions in information science, which have concentrated on the possibility of more efficient implementation of existing processes. A fundamental responsibility for funding councils is also identified: to promote the overall health of university education and research, while establishing meaningful differentiations between units.
The passion with which disputes are pursued within a community can sometimes only be matched by the indifference with which they are viewed from outside. For instance, Conan Doyle created a fictional character whose immersion in criminal milieux rendered him indifferent to the preoccupations of significant sections of late Victorian society:
'I didn't think there was a soul in England who didn't know Godfrey Staunton, the crack three-quarter, Cambridge, Blackheath, and five Internationals. Good Lord! Mr. Holmes, where have you lived?'
Holmes laughed at the young giant's naive astonishment.
'You live in a different world to me, Mr. Overton, a sweeter and healthier one.' (Conan Doyle 1980: 389)
At a further extreme, there can be a simple lack of awareness of issues in the enveloping society. Approaching closer to the empirical focus of this article, Samuel Johnson commented on the bitterness with which scholarly disputes were pursued despite their lack of practical implications for communal welfare:
'It is not easy to discover from what cause the acrimony of the scholiast can naturally proceed. The subjects to be discussed by him are of very small importance; they involve neither property nor liberty; nor favour the interest of sect or party. ... But ... there is often found ... a spontaneous strain of invective and contempt, more eager and venomous than is vented by the most furious controvertist in politics against those whom he is hired to defame.' (Johnson 1984: 448)
Such might be the fate of the Research Assessment Exercise 1996 (RAE), of acute interest within academia but perceived as marginally relevant or simply not known in a broader public context.
The initial reception of the RAE (from December 1996 to end May 1997) in widely publicly circulated media can be studied to investigate whether this pattern of intra-communal passion and wider public indifference is revealed and in order to reflect on the process of research assessment. A distinction between types of media and the communities to which they are primarily addressed can be made which corresponds to the contrast of intra-communal interest and extra-communal indifference. Esoteric media and communities, located centrally within the milieu of university education and research, can be distinguished from exoteric media and communities in the wider public domain.
A further significant distinction can be made between dissemination and diffusion. Dissemination would refer to the processes of transmission of the message or expression (which can be a physical product such as a journal or monograph or a sequence of electronic signals) from sender to receiver. Diffusion would cover the processes of understanding, including adoption, modification or rejection by scholarly or other communities. Dissemination is an essential precondition for diffusion but should not be identified with it. Diffusion could be expected to be more complex and less amenable to centralised control than dissemination. Aspects of diffusion, such as the level of understanding or the extent of rejection or adoption by receiving communities, may change over time. Development over time will therefore also form an important dimension for this study.
This study briefly discusses the dissemination of the RAE results in December 1996 and then concentrates on their diffusion over time through significant exoteric and esoteric media during the period to end May 1997. The exoteric media covered are principally the national and regional newspapers. Esoteric media studied are primarily those shared across university and research communities (principally The Times Higher Education Supplement (here referred to by its short title, The Times Higher)). The educational supplements published with national newspapers can be regarded as esoteric elements placed within exoteric contexts. A further level of granularity within esoteric media and communities would be intra-disciplinary communication and some attention is given to the journal literature of library and information studies. The focus of attention throughout is dual: both on the processes of dissemination and diffusion and on the topics discussed.
Data for the study was acquired by relatively informal and more formal methods. Relevant media were scanned at critical dates, although not all newspapers, particularly regional newspapers, were directly seen. A retrospective literature search was conducted through the UKNEWS and SCITECH groups made available by FT Profile (Note 1.), using a query equivalent to 'research assessment exercise OR (rae AND research)' across the full text of records. The files in these groups, with the notable exception of The Financial Times, receive limited, although increasing, assigned indexing and no assigned term directly descriptive of the RAE was found. Comparison of results obtained by relatively informal and more formal methods suggested that the formal retrieval results were relatively, although not entirely, comprehensive and that there was a degree of complementarity between the methods used. The principal anticipated fora for discussion were, then, covered by the processes of retrieval. Radio and television broadcasting, for which there are more limited facilities for retrieval and more restricted possibilities for obtaining items after their time of dissemination, were not covered by the formal literature search, although concurrent attention indicated limited discussion of the RAE.
The initial public dissemination of the RAE shows evidence of careful central coordination and control. The results were made available through the Higher Education Funding Council for England's (HEFCE) website (Joint Funding Councils of the United Kingdom 1996) from 11am Thursday 19 December 1996. National newspapers (Guardian 20 December 1996; Independent 20 December 1996 ; Times 20 December 1996; Times Education 20 December 1996 ) reported on these results, without necessarily giving a full listing but presenting analyses derived from the results, in their editions of Friday 20 December. The Times Higher for that day gave a full listing together with fuller comparative tables and graphics (Times Higher 20 December 1996a ). Both The Guardian (20 December 1996) and The Times Higher (20 December 1996b) also refer the reader from their to printed versions to fuller data sets available on their institutional websites. Electronic media were, then, critical to the initial process of dissemination - the HEFCE website reportedly received 330 hits on Christmas Day (Times Higher 10 January 1997a ) - and have remained readily available over time. Printed media were secondary in rapidity of dissemination and in wide publicly availability over time, although they can be deliberately retrieved. For processes of diffusion, particularly the development of understanding through public discussion, printed media reassumed their historically dominant role.
Diffusion of information about the RAE into exoteric media has been stratified. Data and discussions reach such papers as The Guardian (20 December 1996), The Times (20 December 1996 ), The Independent (20 December 1996), The Observer (22 December 1996; 29 December 1996; 5 January 1997; 12 January 1997 ) and The Sunday Times (30 March 1997), but no trace was discovered in the larger circulation daily and Sunday newspapers. The discussions tend to be deliberately explicatory, carefully contextualising the RAE and giving definitions before grades. Regionalism and nationalism emerge as highly significant contexts to which the RAE results are assimilated, most explicitly in newspapers from Scotland and Northern Ireland. The number of articles dealing with the RAE declines sharply over time from December 1996, reflecting the press's known strong preference for currency, although institutional disputes can still obtain attention.
The nature of the diffusion of the RAE into exoteric media can be read to imply a limited public awareness of the economic or social value of learning. The initial reception tends to present a set of institutions competing for prestige and funds, without giving attention to the use rather than the exchange value of their research activities. The Financial Times, despite its generally extensive concerns, concentrates upon the performance of business schools (Financial Times 23 December 1996). When discussion of the RAE does reach The Economist, it is presented in terms of the financial rewards for research not the value of learning (Economist 1 March 1997) and is not connected with another, more prominent, discussion of the economic benefits of education (Economist 29 March 1997a, b). Yet the extent of the diffusion of the RAE into public discourse is reflected in its repeated citation with regard to individual institutions in evaluations intended to inform student choice.
Members of the esoteric community have recognised the significance of exoteric media for communicating with a wider public and for political persuasion. The most notable example of this is an article in The Observer by the vice-chancellors or equivalent of four major research universities arguing for a clearly differentiated elite group of universities (Observer 22 December 1996). The institutions (Warwick University, Cambridge University, The London School of Economics, and Edinburgh University) were subsequently labeled the 'fratricidal four' (Guardian Higher 14 January 1997a ). A pattern of dual publication, with communications adapted to the different demands of exoteric and esoteric media, is also discernible. The local significance of some universities is reflected in the space made available in regional newspapers for articles produced by their members. At a national level, after the period of initial reception, the concerns of the esoteric community tend to emerge in the exoteric media only when they reflect the difficulties academics share with other citizens, such as continuity of employment and of their employing institution.
The literature search of exoteric media also served to remind that research is a wide, not necessarily institutionally supported, endeavour not limited to the restricted set of activities acceptable for the RAE. The most engaging example of unintended recall (recalled by the search statement `rae AND research') came from Scotland on Sunday and embodies both nationalism and the diversity of research: `Did Englishmen eat each other? In the 150 anniversary year of Sir John Franklin's death, Tom Pow considers why his failed endeavour overshadowed the greater achievement of the Orcadian explorer John Rae'. (Scotland on Sunday 26 January 1997) Considering the fratricidal four, Voltaire might have concluded that they still would if they could ('pour encourager les autres') - and that Edinburgh had acquired this English characteristic.
Discussion of the RAE in shared esoteric media contrasts with its treatment in exoteric fora. Rather than being deliberately explicatory, a degree of prior knowledge of the procedures of the RAE and its immediate institutional context is assumed. The discussion can even become cabalistic in the sense that grades can be cited apart and begin to detach themselves from the activities to which they refer. A further influence on academic discourse is revealed by reference to the RAE in reviews which have no immediate connection with it: Isaiah Berlin's work was 'emphatically not an oeuvre for the Research Assessment Exercise' (Times Higher 31 January 1997d). An absence of false or unintended recall was noted and this could be explained by the preliminary filtering effect of the educational context of discussion.
In esoteric sections of exoteric media, the topics discussed, such as the organisation of higher education, tend to be those present in other esoteric media. However, the level of analysis is higher, rarely straying below an institutional level. The mode of presentation, although not heavily explicatory, is closer to the enveloping exoteric medium and seems intended to ensure intelligibility within wider public fora.
A further level of granularity within esoteric media would be intra-disciplinary forms of public communication, predominantly journal literature. Informal conversations among members of disciplinary communities may over time be transformed into journal articles. There are traces of this in the shared esoteric literature, in the form of reference to specific future publications (Times Higher 28 February 1997e). The process of peer review (if this is applied), editorial selection and production and dissemination of the journal is likely to ensure that such articles appear at a later date than discussions in shared esoteric media. A strong focus upon the discipline of assessment could also be anticipated and is apparent from those which have appeared (for instance, with regard to accountancy (Whittington 1997) and library and information management (Law 1997)). This form of esoteric communication could be expected to be in the early stages of development and does not form part of the primary focus of this study, although it can be assimilated to the model of diffusion proposed here.
A model for the public diffusion of the RAE can, then, be formalised. Two major types of public communication, exoteric and esoteric, are distinguished. Further distinctions can be made within these categories, critically between trans-disciplinary, or shared, and intra-disciplinary esoteric literature. Each form of discourse has specific characteristics. If a topic migrates between or is simultaneously diffused in exoteric and esoteric discourse, it will tend to be treated in a manner characteristic of the immediately enveloping medium. This tendency can be ascribed to the participants in the public conversation holding a model of communication, although possibly only informally or partially, similar to that formalised here: authors may adapt their communication to the intended forum of publication and editors, in their gatekeeping role, may filter out inappropriate communications or modify them towards their publication's style of discourse. With these dimensions in mind, of an exoteric: esoteric distinction and of change of over time, the relevant literature can be reviewed.
Some common themes emerge in both esoteric and exoteric media, particularly in the early stages of diffusion (to c. end January 1997). Strategic use of data to press institutional or departmental cases occurs in both, although the level of focus tends to be finer in esoteric fora, to the department not the institution. Even where the nominal issue is to general issues and procedures, a contribution can be read with reference to the writer's own situation: for instance, there is a direct correspondence between the numbers of staff and the grades obtained by the departments in the fictional example of two 2 rated departments, one with 'three researchers of international excellence and three who are young and full of promise' and the other with 'three complete woodentops and three worthy hacks squeezing out a little dull work of national quality' (Times Higher 3 January 1997c) and the contributor's own department. While seeking political influence, scholars may betray political naiveté: some even seem to have forgotten Denis Healey's First Law of Holes - when you are in one, stop digging (Healey 1996). There is an unsurprising absence of departments proclaiming their undeserved good fortune. Common to both esoteric and exoteric fora is an uncritical misuse of citation data (Guardian 23 December 1996 ) and a strong tendency to see issues in institutional not structural terms.
The early diffusion of the RAE betrays some of the characteristics of a spoken conversation. Contributions can be brief and in immediate response to other contributions (for instance, articles and letters published in one edition of a Sunday newspaper may receive responses in the next (Observer 22 December 1996; 29 December 1996; 5 January 1997; 12 January 1997)); indiscreet remarks can be made, without the saving impermanence of informal oral utterance; influence is sought, and there is an absence of systematic reflection. A further stage of development is that of anecdote, which appeals to the exoteric media's known preference for the personalisation of issues, and is also present in esoteric media. The recurrent implication of these anecdotes is the RAE has had the undesirable effect of encouraging short term, functional research and discouraging ambitious and valuable projects. Another theme is that of talented individuals in research weak institutional contexts. Although less intemperate than the immediate responses, systematic reflection, for instance as a development of the insights yielded by the episodes recounted, is still absent - one is reminded of Doonesbury's defense of the Marlborough man: anecdotally, you'll live to a 100.
The character of the public discussion of the RAE changes after the period of initial reception. It becomes less intemperate and more considered, although no synthesis or comprehensive understanding of the RAE and its effects was discovered. The number of contributions also declines over time, reaching a low level from the end of May 1997 onwards. Themes announced in the early stages of diffusion are adopted and transformed. Reflection over time occurs predominantly, although not exclusively, in the esoteric literature and in esoteric sections of exoteric media. The discussion can usefully be considered under a number of themes.
A number of issues connected with the geopolitical distribution of research arise. More technical considerations to do with interpretation of critical terms, particularly of 'international excellence', used in the definitions of grades occur primarily in the esoteric literature. Some contributions argue that the process of peer evaluation has been too inward-looking, without input from real foreign referees, and that a weak understanding of international has therefore emerged (Times Higher 10 January 1997b; 21 February 1997a ; Guardian Higher 15 April 1997). A bibliometric study intended to provide fuller international comparisons with France, Germany Australia and the United States is reported, although the known Anglo-American bias of the data to be used (the ISI indexes) is not mentioned as a methodological reservation (Times Higher 21 February 1997b). Disciplines, with the notable exception of philosophy, are not usually conceived as being significantly geospatially distributed. Reflection on these contributions and directly on the process of research assessment would suggest: that 'international excellence' has been generally understood as international participation in scholarly communities, particularly through journals and conferences; that international has often been received as Anglo-American or, with a broader reference, as English language communication; and that contrasts in understandings of disciplines from different locations and different local cultures may not have been fully explored.
References to the regional role of universities and to the distribution of funding decisions to the constituent nations of the United Kingdom emerge in both the exoteric and esoteric literature. The importance of particular universities to a region is revealed by the reporting of their research standing in locally produced newspapers, particularly in Scotland and Northern Ireland (this qualification may be an artifact of the sources studied) (Herald 17 January 1997; Belfast Telegraph 20 December 1996; consider also Irish Times 20 December 1996). The relevance of research to regional economies, and not only as a direct consequence of investment, is claimed, although this is seldom demonstrated at the level of specific links. The distribution of funding decisions to Councils in Wales, Scotland and England, and the arrangements for Northern Ireland, are reported, although the combination of imposed uniformity and partly nominal regional or national autonomy is not fully remarked or related to the broader political context (Guardian Higher 25 March 1997). Absent from both exoteric and esoteric media is any reference to the pleasures of location as a resource for attracting or retaining research active staff.
Issues connected with the organisation of universities, both their relation to one another within a national framework and their internal governance, particularly their internal procedures for the distribution of funds, arise. These feature in the exoteric literature, in esoteric sections of the exoteric media, and in esoteric discussions. The level of interest characteristically becomes finer, descending from a national to an institutional and departmental level, as topic moves from the exoteric to the esoteric literature, with esoteric portions of the exoteric media occupying a transitional position. Anecdote can cross the exoteric:esoteric boundary and topics connected with academics as employees emerge in both types of literature.
The article in The Observer received as arguing for a clearly distinguished elite group of universities can be regarded as a members of the esoteric community deliberately addressing the exoteric community. It generates a debate which migrates to the esoteric literature and which gains the closer and deeper focus characteristic of those fora. As the discussion develops, its specific origins are disguised or forgotten, a process characteristic of the transmission of ideas (Kuhn 1970) (consider also the citation of secondary sources), and it becomes transformed into a recognisable theme of an elite group of universities.
Different aspects to the discussion of an elite group can be distinguished, its existence, its possible existence, and the desirability of its existence, although these are not necessarily clearly differentiated by contributions. Strategic intentions can be detected in the institutional affiliations of contributors. Arguments for formal differentiation of an elite group tend to come from members of the intended elite and to point to strong existing informal differentiations, particularly those revealed by orderings of universities derived from aggregations of RAE data. Arguments against differentiation originate mostly from institutions who would not be included in the proposed elite and tend to elide or play down the strength of the existing distinctions (Times Higher 3 January 1997a; 10 January 1997d). The history of universities (Times Higher 31 January 1997a), particularly the strong research development of some universities created in the 1960s, is used as an argument against imposing rigid formal distinctions between universities which would inhibit future developments (Times Higher 17 January 1997a ; with a response Times Higher 31 January 1997b ). Some contributions take a more radical stance, arguing that the idea of uniformity among universities is a fiction which only commands conviction among members of the university system itself (Times Higher 3 January 1997b). In a departure from the established pattern, one member of the intended non-elite does argue for formal differentiation of the functions of universities and the extent to which they should engage in research (Times Higher 3 January 1997b).
The consensus of academic sentiment, as indicated by a survey conducted before the RAE results were announced, is against a formal elite (Times Higher 10 January 1997c) and this could be connected with the numerical predominance of the intended non-elite. A more subtle response to calls for an elite, again emerging the from the intended non-elite, is an advocacy of diversity, coupled with an insistence that a differentiation of functions, for instance between teaching and research or regional, national and international roles, need not be equated with a hierarchy (Times Higher 7 February 1997b). The implied distinction between description and value may be valuable, particularly for considering future development, but, in the immediate context of competitive allocation of resources, does not command full conviction - Aesop's fox was convinced the grapes were sour.
Covert elitism is sometimes detected in the decisions taken on the distribution of funds (Times Higher 18 April 1997a). Neither the ratios assigned to RAE grades, nor the factors to be applied to subject groups (Band A: high cost laboratory or clinical subjects; Band B: intermediate cost subjects with a technical, experimental or practice-based element; and Band C, for all other subjects), were known prior to the announcement of the RAE results. There was a general understanding, which proved incorrect, that 5* (Research quality that equates to attainable levels of international excellence in a majority of sub-areas of activity and attainable levels of national excellence in all others) was reserved for recognition and would not carry extra funding (Times Higher 18 April 1997a). The publicly stated reason given by HEFCE for not specifying differentials in advance was to protect the objectivity of the assessment process (Times Higher 18 April 1997a). Comments elicited from institutions favoured by funding decisions tend to view them as just whereas those from disadvantaged institutions regard them as inhibiting future developments which would be valuable to the system as a whole (Times Higher 31 January 1997c). The acknowledged need for compensatory mechanisms to ensure system stability during transition points to the increased sharpness of the differentials (Times Higher 28 February 1997a, b, c ). In this context, we are dealing with processes of dissemination and diffusion secondary to those following on the RAE results themselves, with the messages giving funding decisions (reported in (Times Higher 24 January 1997)) and the slower development of understanding of their implications.
Funding decisions are policy instruments which descend to the level of institutions but not to research groups within or across institutions. Research Councils form the other component of the dual system of public research funding operative in the United Kingdom and their focus of interest is the research group or project not the institution. Concerns have been expressed about the lack of full support for research infrastructure (including library and information services) offered by Research Councils (Times Higher 28 March 1997a; Guardian Higher 8 April 1997). Universities also have autonomy in their internal distribution of RAE funds. These factors which might enable research group or intra-institutional autonomy and limit the reach of RAE funding differentials as a policy instrument are constrained by the prestige conferred by high RAE gradings. Prestige acquires an economic value by its role in attracting research funding from public, and from private, agencies (Times Higher 21 February 1997c). It can also influence intra-institutional allocation of funds, although conflict is still apparent, particularly in connection with strongly rated research groups within weaker institutional contexts (Guardian Higher 18 February 1997a). Yet the overall effect appears analogous to the Matthew effect ('unto every one that hath shall be given'), known to studies of scholarly communication in the form of the accumulation of citations by the already highly cited.
The department represents a significant level of concern immediately below that of the institution. More technical issues, such as the research grade and staff:student ratio required to avoid intra-institutional cross-subsidy, are confined to the esoteric literature. Departments closed, or threatened with closure, on the grounds of low grades obtained in the RAE are also reported in esoteric sections of the exoteric literature (Guardian Higher 4 February 1997). In one instance, an appeal is made through an exoteric forum on behalf of a department threatened with closure despite good RAE performance (Guardian 18 February 1997).
A final level of concern is that of the individual employed by a university. The idea of the individual or cluster of individuals as the locus of research emerges in the esoteric and exoteric media. In esoteric media, there is discussion of the possibility of assessments of individual researchers by assessment panels and internally by universities (Times Higher 28 February 1997e ). It should be noted that this represents a transformation of the definitions assigned to grades: these refer to relative proportions of sub-areas of research (which may be identified with individuals) at levels of international and national excellence not directly to individuals. Informal conversations suggest that this transformation, which enables finer distinctions than international/national (a 7 point scale is in use), has permeated the shared language of academic discourse.
The transformation of the interpretation of the grades can be understood from classic perspectives on language. Wittgenstein urged, 'Let the use of words teach you their meaning', and further observed that, 'For a large class of cases - though not for all - in which we employ the word meaning it can be defined thus; the meaning of a word is its use in the language.' (Wittgenstein 1958 : 43 and 220) From this perspective, we should recognise that original definitions have not fully prescribed the use of grades, although they have been strong influences, and examine the use of grades to discover their meaning in use. The structuralist principal that meaning is a product of mutual differences between items in a system (Saussure 1983) would account for the differential interpretation of grades. A highly specific analogy is provided by Saussure's brief discussion of artificial languages: the creator of an artificial language only controls it so long as it is not in circulation; once in general use it develops according to influences which have nothing in common with its logical interpretation (Saussure 1983:76). Influences apparent from the development of the artificial language of the grades are a desire for finer differentiation than the binary contrast national/international and a tendency towards individualism. Disciplinary cultures will emerge as a further influence.
In esoteric and exoteric media, particularly in the early stages of diffusion, the individual as the locus of research emerges in anecdotes which take the talented individual trapped in an indifferent or declining context as their theme (one senses a transformation of the Christian and romantic motif, present in popular histories of science, of a prophet without honour in his own country). Reports of research assessments of individuals, characteristically cast in terms of RAE grades, being used to determine conditions of work, including the distribution of duties across research, teaching and administration, promotion decisions, salaries paid, and the continuity of employment, emerge first in esoteric media (Times Higher 7 February 1997a, c; 28 February 1997d ; 28 March 1997a, b). The issue of continuity of employment may then be diffused, after some delay, to exoteric media (Herald 18 April 1997; Sunday Times 30 March 1997 ) and their esoteric sections (Guardian Higher 18 February 1997b ; Independent Education 24 April 1997). Non-academic employees, and other implications of changes in university funding for local economies, are characteristically neglected. Hints of a more systematic perspective, not limited to anecdote or reports of current events, are contained in observations of a convergence with United States patterns of employment for university academic staff (Sunday Times 30 March 1997) and of the dangers of reinforcing effects on research differentiations over time, particularly with regard to patterns of recruitment of individuals.
In summary, then, although a debate on the organisation of universities is signally initiated in an exoteric forum, the subsequent discussion largely migrates to esoteric fora. More technical issues are confined to the esoteric literature while employment issues, which reflect concerns academics share with other citizens, permeate to the exoteric literature. The concatenation of effects from national decisions to individual academics, particularly those associated with funding differentials, ensure that traditional scholarly isolation, itself misleading as an image of actuality, has been invaded by effects induced by a competitive external environment.
Disciplinary cultures can be discerned as influences on the process of assessment and upon the reception of grades obtained. For instance, philosophy, particularly analytic philosophy, might insist on strictly logical and numerically precise interpretation of the definitions attached to RAE grades and this is strongly implied in the intemperate protest already noted (Times Higher 3 January 1997c) (and is supported by informal discussions with members of the philosophy assessment panel). A managerial understanding of the value of strategic development over time can be sensed in Warwick Business School's guarded expression of regret at not obtaining a 5* (Times Higher 27 December 1996). From history, the process, although not the broader context, of assessment is examined with attention to narrative detail (Times Higher 10 January 1997c).
Contrasts within and between disciplines are asserted to have had critical effects on the value attached to different forms of research by assessment panels, in apparently independent contributions. Within philosophy, Anglo-American analytic philosophy (whose dominance in the process of assessment has been suggested) is accused of being unsympathetic to continental European philosophy and of undervaluing contributions in this tradition (Times Higher 28 February 1997e). The different understandings of a discipline revealed with regard to philosophy would be characteristic of the human and social sciences where contrasting paradigms may be held simultaneously, rather than successively, by different groups, and, where, to a lesser extent, some members of a paradigm may disparage work produced in another. Within the natural sciences, where successive paradigms are the expected pattern (Kuhn 1970), there are traces of similar conflicts, although groups are marked by more clearly differentiated labels. For instance, environmental sciences consider that they have been disadvantaged by a failure of understanding (or strategic group self-protection) by assessors from the earth sciences (Times Higher 10 January 1997f). The extent of understandings shared between the parties involved seems, then, strongly to influence the process of assessment.
A contrast which emerges across a number of disciplines is between theoretically and practically oriented research, with reports of a higher value being placed upon theoretical research. Comments suggest that this creates particular difficulties for areas of study allied to professional practice, where, in addition to an ethos not oriented towards research, a dominant mode of investigation may be strongly empirical (Times Higher 3 January 1997d; 7 February 1997d; 7 March 1997a). It is also perceived as a discriminating factor between groups of institutions, disadvantaging the new universities which are developing their research activities within an inherited and continuing practical and possibly local orientation (Times Higher 31 January 1997e).
Contrasts between disciplines point to the social natures of the processes of communication and negotiation by which RAE gradings are constructed. Different panels interpret assessment criteria in distinctive ways which can plausibly be connected with their disciplinary culture. Cultural dissonance between parties can result in outcomes perceived as unsatisfactory by the disadvantaged party. A critical feature of inter-panel contrasts would be that grades obtained, despite their nominal equivalence, do not refer to commensurate levels of research across disciplines. Any further study based upon statistical analysis of grades, conceived as data, must recognise this, although a standing danger for such studies is to neglect reservations originally made on the commensurability of data. An analogy can be made with citation analysis, where the specific significance of particular citations can be lost when they are manipulated in aggregate form. Although grades are not necessarily fully comparable as indicators of the level of research, they are made commensurate when they are re-interpreted for their exchange value, as factors for the allocation of funds.
Conflict between the demands of research and teaching is frequently reported. This is primarily seen at the level of individual motivation and the distribution of time and energies encouraged by the extrinsic rewards, such intra-institutional career development and the possibility of inter-institutional mobility, regarded as primarily associated with research. Teaching is seen as a localised activity which may, and, even then only uncertainly, attract intra-institutional rewards (Times Higher 31 January 1997f). The aggregate tendency of individual motivations is regarded as encouraging an institutional focus on research rather than teaching (Times Higher 7 March 1997c; 18 April 1997b, c). Some contributors recognise this and argue for extrinsic rewards for teaching which would change institutional foci (Guardian Higher 22 April 1997 ), possibly with systematic differentiations between the degree of institutional orientation towards teaching or research (Times Higher 17 January 1997c). Elements of strategic positioning, with arguments for recognition for teaching emerging from institutions with low research ratings and an inherited emphasis on teaching, can be discerned.
The value of the products of research to the teaching process is seldom discussed. A significant, if familiar, distinction is made between textbooks intended to assist teaching and research published in refereed journals. The low value attached to textbooks in the RAE is regretted (Times Higher 18 April 1997c). Contributions reveal, in a rather unreflexive way, the patterns which have been discerned in the transmission of scientific knowledge; textbooks are central to the initial stages of induction into a scientific community; the primary scientific literature is studied at a later stage (Kuhn 1970). A more sophisticated understanding of the processes of diffusion from research to instructional literature is absent.
In summary, then, the frequency of references to a tension between research and teaching points to the depth and pervasiveness of the experienced conflict. It is perceived primarily in at the level of the individual (even where it is claimed that good researchers tend to be good teachers (Times Higher 18 April 1997c)) and only secondarily in systemic terms. Intra-professional issues, such as career development, are largely confined to the esoteric literature; the exoteric literature is primarily concerned with the quality of students' educational experience, although the motivation of staff towards teaching is acknowledged as one factor influencing this.
The RAE could be regarded as part of the contemporary emphasis on evaluation and assessment, with numerically manipulable outcomes being preferred. A few contributions recognise this (Times Higher 21 February 1997d; Herald 26 February 1997), although the RAE is not always fully contextualised in this respect. Where the influence of a culture of accountability is recognised, the unproductive labour involved in reporting and the absence of a holistic view of a university's activities can be regretted (Herald 26 February 1997). Increased government control of research is also denounced (Times Higher 7 March 1997b).
A critical, and widely recognised (HEFCE 1997, paragraph 36), feature associated with evaluative measures is change in the behaviour of the objects evaluated in response to known or anticipated assessment criteria. Nominal improvements can occur without substantive developments. Features of research identified which could be regarded as examples of responsive changes include: changes in patterns of publication towards refereed journals and monographs from academic publishers (Times Higher 28 February 1997e); a reduction in communication with other disciplines and a wider public (Guardian 2 April 1997); a change in academic culture when compared to the 1950s and 1960s (Times Higher 21 February 1997e ); and, perhaps most disturbingly, a redistribution rather than increase in research activity. When theorised, these are regarded as 'inadvertent effects' (Times Higher 17 January 1997d ) rather than directly connected with understandings of accountability.
The results of evaluations, when made public, can be adapted to purposes for which they were not originally intended. RAE results appear, together with Teaching Quality assessments, in articles in the exoteric media intended to inform student choice (Times 28 January 1997; Independent 6 February 1997). In this respect, students would seem to be conceived as potential consumers of educational products and are provided with measures to sharpen their capacity for discrimination between products.
Reflection on the RAE from the perspective of a culture of accountability suggests some strong, possibly conflicting, tendencies, which may have significant future implications. A continuing stress upon monitoring and directing the use of public funds and upon the quality of research products can be anticipated. Within this broad context, and particularly from the perspective of funding councils, there might be a temptation to change methods and criteria in order to prevent nominal adaptation being pursued at the expense of substantive developments. Yet the unproductive labour associated with the RAE and the wariness of unintended, and unwelcome, effects indicate resistance to assessment and to changes which would increase its burden. Finally, the literature betrays a disturbing lack of support for the positive functionality of the RAE.
An analogy is repeatedly made between the transfer patterns and league tables of professional football and those associated with the RAE (for instance, Guardian Higher 7 January 1997, 14 January 1997a, b). The analogy may have an unexplored depth. The concept of place was crucial to the development of professional football, so fundamental that is critical role was only fully noticed when disturbed. Labour rapidly became mobile and the significance of this has recently increased. The relative importance of place has been further eroded by the power of entrepreneurial capital. History may only remain an influence for maintaining status when the circumstances which assisted development are renewed.
Analogous developments can be detected in the organisation of universities. Labour has long been mobile but the relation of an academic to their institution is increasingly being laid bare as one of selling the products of intellectual labour. Place is still significant although the pleasure of location could be regarded as a surrogate for salary differentials. History seems most significant to those institutions, the new universities of the 1960s and 1990s, which inherited cultures not fully adapted to a research environment. Long duration does not by itself guarantee high current significance.
The deeper analogies between the organisation of professional football and the distribution of university research are read from, rather than stated or even implied by, the contributions which indicate similarities in transfer patterns and league tables. As such, further exploration is beyond the scope of this article. Yet it is disturbing that potentially highly revealing aspects of an analogy are left below the level of deliberate implication and does again suggest the rather unreflexive and unsophisticated character of the public discussion of the RAE.
The approach taken here contrast with other treatments of the research assessment exercises in the information science literature. In summary, they have taken the research assessment exercises as a given and concentrated on adapting established techniques, particularly, although not exclusively (Hodges et al. 1996), citation analysis (Oppenheim 1995; Seng and Willett 1995), to the evaluation of research. The use of these techniques in assessment is then advocated as more economic than the procedures which have been used in the research assessment exercises.
A broader responsibility to inform discussion of the research assessment exercises and their effects from understandings of scholarly communication and of the development of knowledge has not yet been accepted. Some possible contributions can be indicated. The idea of discourse communities and of scholarly communication as a public conversation (Wilson 1983) could help identify the changed interval between public contributions required, particularly in the humanities, by research assessment procedures as a critical development. The value of a continuous public conversation to the development of eminent researchers could also be explored. Most critically, the known productivity of margins and transitions could be used to counter calls for an elite of institutions and to point to the dangers of rigidity and stagnation associated with restrictions on entry to scholarly communities. In some respects, for instance, in their attention to the processes of peer review, the studies sponsored by the Higher Education Funding Council for England (HEFCE) (HEFCE 1997; McNay 1997) have been more sophisticated and comprehensive in their understanding of scholarly communication than those emanating directly from information science.
The approach adopted in this article has been informed by understandings of scholarly, and other, forms of communication. The rather heterogeneous voices emerging in the public reception of the RAE have been assimilated to a model of communication which makes a crucial distinction between exoteric and esoteric, or extra- and intra-communal, media. Forms of communication not directly studied, such as intra-disciplinary considerations of the process of assessment for that discipline, could also be understood in terms of this model. In contrast to the HEFCE sponsored studies, which are inevitably historical in character (a study (McNay 1997) deals with the impact of the 1992 Research Assessment Exercise), the data offered by public discussions of the RAE is more current. It also give the impression of being richer, fuller and more diverse than that gathered by questionnaires or elicited from focus groups, where possible responses are, to some extent, constrained by the categories brought to the study by researchers.
The analysis and categorisation offered here may have given an appearance of systematicity to a diverse set of contributions. Reflection on contributions does identify a fundamental responsibility for funding councils: to ensure that evaluative methods, while producing meaningful distinctions between units assessed, contribute to the health of the system as a whole. The tenor of the public discourse studied here suggests that this responsibility is unlikely to be assumed by individual institutions. The individual rational economic self-interest of members of a system may not maximise group or wider communal benefit. Councils may wish to work towards ensuring the coincidence of institutional interest and social benefit. A fuller understanding of the forces influencing research systems can assist the construction of policies towards that desirable end. The value of this study would be in contributing towards the development of that understanding.
Finally, it could be suggested that while an institutional context can encourage an appropriate motivation towards research, it cannot impose it. Roland Barthes' exhortation, although addressed to the young in research, remains inescapable:
The task (of research) must be perceived in desire. If this perception does not occur, the work is morose, functional, alienated, impelled solely by the necessity of passing an examination, of obtaining a diploma, of insuring a career promotion. For desire to be insinuated into my work, that work must be demanded of me not by a collectivity seeking to guarantee my labor and to gain a return on the loans it grants me, but by a living collection of readers.' (Barthes 1986: 69-70)
And, in every new project, we may remain young.
1. In June 1997 the group UKNEWS consisted of the following files: The Daily Telegraph and The Sunday Telegraph; The Financial Times; The Guardian; Today; The Times and The Sunday Times; The Independent and The Independent on Sunday; The Northern Echo; The Observer; Lloyd's List; The European; The Scotsman and Scotland on Sunday; The Economist; Today's Financial Times; The Herald; Daily Mail and The Mail on Sunday; Evening Standard; The Irish Times; The Mirror and Sunday Mirror; Daily Record and The Sunday Mail; The People; The Times Educational Supplement; The Times Higher Education Supplement; Press Association; Regulatory News Service.
The group SCITECH consisted of the following files: New Scientist; The Economist; The Guardian; Financial Times; Computergram; US Telecommunications; Flightline; McGraw-Hill Aerospace; McGraw-Hill Computing Cable, Satellite & TV News.
Printed equivalents are adequately revealed by file titles.
Early drafts of this article were discussed with Dr Elisabeth Davenport. I am grateful for the comments and encouragement received.
How to cite this paper:
Warner, Julian (1998) "The public reception of the Research Assessment Exercise 1996." Information Research, 3(4) Available at: http://informationr.net/ir/3-4/paper45.html
© the author, 1998. Updated 12th March 1998
Contents |
Home |