Information Research, Vol. 2 No. 1, August 1996

Electronic journals and scholarly communication: a citation and reference study

Stephen P. Harter and Hak Joon Kim
School of Library and Information Science
Indiana University, Bloomington, Indiana, USA

The journal is fundamental to formal scholarly communication. This research reports highlights and preliminary findings from an empirical study of scholarly electronic journals. The purpose of the research is to assess the impact of electronic journals (e-journals) on scholarly communication, by measuring the extent to which they are being cited in the literature, both print and electronic. The intent is to provide a snapshot of the impact e-journals were having on scholarly communication at a given point in time, roughly the end of 1995. This study provides one measure of that impact, specifically on the formal, as opposed to informal, communication process. The study also examines the forms in which scholars cite e-journals, the accuracy and completeness of citations to e-journals, and practical difficulties faced by scholars and researchers who wish to retrieve e-journals through the networks.

Introduction

The first scholarly journal, Journal des Scavans, was published as a new medium of communication in 1665, and was soon followed by the Philosophical Transactions of the Royal Society (Osburn, 1984). For more than three centuries the journal has played a pivotal role in the creation and transmission of knowledge by serving as the primary medium of scholarly communication, and has remained essentially unchanged in form and function over its lifetime. Science as we know it is scarcely imaginable without the scholarly journal.

Despite its benefits to science and scholarship, the paper journal system has been subject to much criticism. Deficiencies noted by some authors include perceived problems with the peer review process (that it suppresses new ideas, favors authors from prestigious institutions, and causes undue delays in the publication process), high costs that are escalating faster than the rate of inflation, and lack of selectivity. Spiraling costs and long publication delays are perhaps the most serious of these criticisms.

At the same time that the costs of producing the paper journal have increased sharply, developments in computer and communications technology have accelerated. And of course we now have the dramatic explosion of the World Wide Web. Technology increasingly offers the possibility of using computers and communication networks to create alternative electronic forms of the conventional paper journal. It is possible that these new forms of computer-based communication will transform the scholarly communication system. There is much debate and discussion concerning how the roles of the various participants in the scholarly communication process, including libraries, may be redefined in the process (Meyer, 1993).

Although electronic journals (or e-journals) have been under development since 1976 (Turoff and Hiltz, 1982), e-journals in their non-experimental phase did not begin until the 1990s, with a few exceptions. The first peer-reviewed electronic, full-text e-journal including graphics was Online Journal of Current Clinical Trials (OJCCT) (Keyhani, 1993). In the most recently published list, and one of the few research projects involving e-journals, Hitchcock, Carr, & Hall (1996) identified 115 scholarly, peer-reviewed e-journals in science and technology, and there are many more in the social science and humanities. These constitute the first wave of what are likely to be many more scholarly e-journals to come.

Ann Okerson wrote, "One can fantasize endlessly about electronic 'journals,' but without active authorship and readership there is nothing" (quoted in Collins and Berge (1994)). We would add "use" to authorship and readership. If e-journals are not used, they cannot play an important role in scholarly communication. There has been much discussion and speculation in the literature about perceived advantages and disadvantages of e-journals, and about problems and issues related to their development and use. Such speculation is necessary and important. The present study is aimed at gathering hard data regarding the readership, use, and impact of e-journals. What kinds of e-journals are being published, and how often? What impact are e-journals having on the scholarly communities they are serving? To what extent are scholars and researchers aware of, influenced by, using, and building their own work on research published in e-journals?

The present study assesses the impact of scholarly e-journals by examining the artifacts of scholarly communication--the journal article and its references. In bibliometric studies, citation analysis is a well-known technique that has a long history in studies of scholarly communication (Borgman, 1990). As an artifact of scholarly communication, citations reveal formal communication patterns and scholarly impact. The major advantages of citation analysis are its high reliability and unobtrusiveness. Several standards for citing electronic publications have been promulgated (Li & Crane, 1993; Walker, 1995; Brown, 1995).

This research provides hard citation data concerning the impact of e-journals on the conduct of research and scholarship. At another level, publishers of e-journals can gain some information regarding use by recording the number of subscriptions to an e-journal or the number of times articles are accessed or downloaded from host servers. While these kinds of data provide useful indicators of one type of use, they do not measure the extent to which e-journal articles are playing a role in the scholarly and research process, that is, in the advancement of knowledge. While the meaning of citations can certainly be debated, what is clear is that they reflect an influence or impact of some kind on the author of the citing article.

The purposes of this study are to:

1. assess the accuracy and completeness of citations to e-journals
2. identify the extent to which scholars publishing in both print and electronic journals cite e-journals and other e-publications
3. identify fields in which researchers actively use scholarly e-journals
4. identify highly cited e-journals and articles in e-journals in these fields
5. record and analyze interesting demographic characteristics and access problems of e-journals.

The last stated purpose deserves further comment. Although it was not originally part of our study to investigate demographic data, we found as the research progressed that we were learning information of great interest to us that had little to do with the referencing of e-journals. As part of our original research plan, in order to investigate the references of articles in e-journals it was necessary that each e-journal in the population be accessed and articles and other information about the journal to be downloaded and printed for analysis. A useful byproduct of the study has resulted from a study of these data, which revealed several interesting demographic characteristics and access problems associated with the population of e-journals. Because of space limitations, only a few of these can be discussed here.

Summary of methods

The method followed in the first part of this study is one of defining and drawing appropriate samples of e-journals, obtaining data from printed documentation and e-journal articles, conducting descriptive analyses, and reporting the findings in tables and prose. In the second part of the study, tools published by the Institute for Scientific Information (ISI) are used to conduct cited reference searches and other citation analyses. The following section describes how the e-journal samples were drawn for study.

Our interest in this research was on scholarly electronic journals that publish articles reporting scholarship or research that are refereed or peer-reviewed. Other types of publications--newsletters, zines, and the like, were not subjects of study. We chose to use two published directories that include data on electronic journals: Mecklermedia's Internet World's on Internet 94 (Mecklermedia, 1994) [Note: the 1995 edition was not available at the time of the study], and the Association of Research Libraries' Directory of electronic journals, newsletters and academic discussion lists (Okerson, 1995). E-journals selected from the ARL directory were taken from the section entitled "Electronic journals, magazines, and zines" and identified there as "peer-reviewed." e-journals selected from the Mecklermedia directory were taken from the section entitled "Electronic journals and newsletters" and identified there as "refereed." All e-journals meeting these criteria became members of the sample. The two lists were combined and duplicates were removed. There were 134 e-journals in the final list. Four titles were eliminated from the sample for various reasons, and one title was added, for a refined initial sample of 131 e-journals.

Next, recent articles and other documentation (journal description, instructions for potential authors, statement of scope, etc.) from each of the e-journals were downloaded or otherwise obtained and printed for examination. The access and demographic study was conducted on all 131 e-journals in the refined initial sample.

For the reference study, we had two criteria for selecting the journals to be studied from the 131 e-journals in the refined sample. First, the e-journal must have published one or more articles that reported the results of research or scholarship. Second, the e-journal be peer-reviewed or refereed articles, as described in the e-journal documentation. Many e-journals were eliminated because they failed to meet the first criterion. A few were removed because of the absence of any statement in the e-journal documentation indicating that it was peer-reviewed or refereed. This characteristic, as well as all the other demographic information related to the e-journals, was taken directly from the e-journals. There were several instances where the data in the two directories (both providing self-reported survey data) did not agree with the information provided in the e-journals themselves.

To be included in the reference study, an article must, in our judgment, have reported the results of research or scholarship and must have cited at least one reference. Our sample of articles for the reference study was defined as the last four available scholarly, peer-reviewed articles of each of the 77 scholarly e-journals. If one or more issues or articles of the year 1995 were not available, one or more issues from 1994 and, if necessary, 1993, were used.

For the cited work study, the population of seventy-seven e-journals used in the reference study defined the initial sample. However, in order for an e-journal article to be cited, especially in a print publication, the lag time in conventional print publishing must be considered. One must provide some time for articles to be read, to influence a researcher or scholar in some way and thus become part of a study in progress, and eventually to be cited in the published article reporting the results of that study. For this reason, the cited work study was done only on scholarly electronic journals that commenced publication in 1993 or earlier. There were thirty-nine such journals. Table 1 summarizes the characteristics of the three e-journal samples used in the study.

Demographic characteristics of e-journals

Subjects covered

We assigned the 131 e-journals in the sample (consisting of all known e-journals at the time the sample was selected) to broad subject categories that coincided in most cases with university departments or professional schools. Subject categories could not be determined for seven e-journals. Although we were fortunate to have access to some fee-based journals, thanks to temporary passwords supplied by OCLC, Inc., we did not have access to three other fee-based e-journals in the sample. Also, three e-journals could not be accessed at all, either by using the information in the two directories or by using other means, such as Internet search tools. Finally, the language of one e-journal was Polish, and we did not attempt to obtain a translation. Since we required all of our demographic data to come from the e-journals themselves, rather than secondary sources, these seven journals were eliminated from the sample. Save for a very few exceptions, then, the remaining demographic data are for the 124 e-journals to which we were able to gain access.

Table 2 shows the subject categories with highest frequencies. Education, literature, and mathematics were the top three categories, with library and information science a somewhat surprising fourth, with seven e-journals. There were five e-journals under the heading of religious studies. We were also somewhat surprised at some of the subject categories with relatively low frequencies -- for example, physics, biology, engineering, and sociology with only one or two representatives each. Of course, since the publication of the ARL and Mecklermedia directories we used to select our sample, many more e-journals in these subject areas have appeared, coinciding with the explosion of the World Wide Web. Our demographic data represent the state of affairs as of June, 1995, and can be taken as a benchmark, against which subsequent studies of electronic journals can be compared.

Many new e-journals have evidently arisen from an introspective consideration of the new communication media themselves, and with a concern with how they will affect society and traditional disciplines and communication patterns. Computer science, communication, and the broad subject we called "information technology, media, and society" are examples.

A final way to characterize the subjects of the e-journals in the sample is to classify them into the traditional four broad areas of scholarship and research in the university: sciences, social sciences, humanities, and professional. Table 3 summarizes a breakdown of subjects into these four main categories. While we might have guessed that the sciences and possibly the professions would be most heavily represented, this was not the case. The four categories are essentially equal in size. Whatever the motivation may be for launching an e-journal, it is apparently held by scientists, social scientists, humanist scholars, and professionals alike.

Scholarly, peer-reviewed e-journals

The major purpose of this research was to study the effects of scholarly, peer-reviewed e-journals on formal scholarly and scientific communication, as measured by cited references. For the reference study and cited work study it was necessary to obtain and examine individual issues of the e-journals in the sample to identify which met our selection criteria. These criteria were that the e-journal must have published at least one article reporting the results of research or scholarship must be refereed or peer-reviewed, as stated in the documentation provided by the e-journal (corresponding to what is usually the front-matter in a print journal).

Table 4 reports the results of our analysis of the scholarly nature of the 131 e-journals in the initial sample. Only 77 e-journals were judged to be scholarly and peer-reviewed. Twenty-five e-journals in the initial sample were in our judgment neither scholarly nor peer-reviewed. These were zines and newsletters for the most part. Six other e-journals did not meet our selection criteria for various other reasons.

Media of publication

Scientific and scholarly journals are today being published in many media: CD-ROM, floppy disk, on computer networks such as the Internet, and of course, print. The literature has tended to use the term e-journal -- short for electronic journal -- to refer to networked journals but not other forms of electronic journals (on floppy disk, etc.). We have used the same terminology in this report, though technically it is at best potentially misleading and at worst incorrect.

Most (nearly three quarters) of the e-journals in the sample started as an original publication – they began their lives in electronic form. Exactly one-fourth of the sample began and continue as print journals. That is, they exist in two parallel forms, print and electronic. And in three known cases – Modal Analysis, [Note: the rights to Modal Analysis were bought by Sage Science Publishers and the journal ceased publication at the end of 1996. Ed.]Essays in History , and Journal of Extension, the print version has been replaced by an electronic version. Table 5 summarizes these findings.

Several models have been developed for the relationship between print journals and their electronic counterparts. (See Table 6.) The Chicago Journal of Theoretical Computer Science and other e-journals will provide individual articles in paper form, though the journals themselves are not published in a paper version. The Journal of Financial and Strategic Decisions calls its e-journal "secondary to" the print journal, by which is meant that the electronic version is the ASCII format and does not accurately represent the print version. [Note: the Journal ceased publication with the Fall 2000 issue. Ed.] Slavic Review publishes its electronic version a few months after the print version has appeared (called a "postprint edition"). Public-Access Computer Systems Review (PACS Review) has published at least two volumes in book form, many months after the appearance of the electronic version. [Note: the Review ceased publication in 1998. Ed.] Several journals publish some but not all of the print journal electronically. Finally, both an electronic and a print version of a journal may exist, but with different pricing arrangement.

Pricing Arrangements

Whether there exists a charge for a given e-journal is not as simple a question as it may first appear. Questions and distinctions not always made for print journals become important for an interpretation of the meaning of cost figures. For example:

• Is there a cost distinction between providing access to current issues versus archival issues?
• Is there a print counterpart of the e-journal?
• Is the e-journal included with membership in a society?

Clearly the answers to these questions will be important information needed to interpret the cost, if any, of an e-journal, as well as the meaning of pricing trends. We identified eight different categories of e-journals in the sample, depending on the answers to these questions. Table 7 condenses the eight categories into just two: e-journals in which all issues are free, and e-journals for which some form of fee exists. For nearly nine out of ten e-journals in the sample, all issues are free. And in some of the remaining e-journals, there is no fee for some kinds of access. For example, JAC Online, a journal of composition theory supported by the Association of Teachers of Advanced Composition and the University of South Florida, charges a fee to view the current issue, but there is no charge to examine archival issues. A chi-square test of the hypothesis that there is no relationship between pricing arrangement and whether the e-journal is peer-reviewed could not be rejected (p=.487).

Reference study

The purpose of the reference study was to analyze the references from a sample of articles in scholarly e-journals, to measure the extent to which authors of e-journal articles are currently citing e-journals and other online sources. Such authors can be assumed to be more knowledgeable about and sympathetic to the electronic media than typical authors. Thus the results of the reference study should provide a kind of upper bound on the current influence of e-journals and other electronic publications on scholarly communication.

Among the 77 scholarly and peer-reviewed e-journals in the sample, two were fee-based e-journals for which we had no subscription[4]. These e-journals were eliminated from the sample. In addition, Radioscientist ON-LINE, published by the New Zealand National Committee for the International Union of Radio Science, included only one article, which had no reference, so it was also eliminated from the sample. Therefore, the final sample size for the reference study was reduced to 74.

To examine their references, we retrieved and printed the last four available scholarly, peer-reviewed articles published by each of the 74 e-journals. Because the e-journals in the sample were distributed in different ways, a variety of means had to be employed to do this, ranging from email, listserv, and ftp commands to using Web browsers. This part of the study was quite illuminating as to the difficulties involved in accessing and using e-journals. Six of the 74 e-journals had published fewer than four articles as of September, 1995, when the data collection took place. In these cases we studied the articles that were available.

Table 8 summarizes the data relating to the samples in the reference study. We took the first twenty references in each e-journal article to build our set of references. If an article had fewer than twenty references, we studied all the references in that article. For this portion of the study there were a total of 4,317 references from the most recent 279 articles published in 74 scholarly and peer-reviewed e-journals.

Each of the 4317 references was classified as to its format (book, serial, book chapter, online source, etc.). Online sources were further subclassified (web page, email, e-journal article, etc.). Finally, the accessibility of the references to online sources was tested by actually trying to retrieve the cited item.

Table 9 shows the formats and frequencies of the most frequent reference sources cited by the 279 e-journal articles. Print serial (43.3%) was the format most frequently cited, with books (26.9%) and book chapters and other parts (16.0%) occupying the second and third ranks respectively. This distribution is roughly what is found in reference studies conducted using print journals in various fields, e.g., library and information science (Harter, Nisonger, & Weng, 1993).

References were classified as "online sources" in Table 9 if they were of a variety of electronic subtypes: web pages, electronic personal papers, email messages, e-journal articles, news group postings, listserv postings, and others. Only 83 online sources (1.9% of the total number of references) were cited by the 74 e-journals (279 articles). Among the 74 e-journals, only twelve had one or more reference to an online source and the other 62 e-journals had none at all.

The distribution of online and e-journal references in the twelve e-journals that have at least one reference to an online source is highly skewed. The e-journal Public-Access Computer Systems Review had 34 online references (40.9%) and Electronic Journal of Virtual Culture and e-journal accounted for a further 25 (30.1%) and 9 (10.8%) respectively from among the total number of 83 online references to online sources. Thus online references are distributed among only a few e-journals; 68 (81.8%) of the total come from these three.

Table 10 shows the distribution of the online references of each four reviewed articles in the top three e-journals with online references. This distribution is also skewed, with most references to online sources appearing in only three articles. Thus the number of online references also depends on the characteristics of individual articles within the same e-journal, although the degree of skewness is not so large as for the overall population of e-journals.

Table 11 shows the types and frequencies of references to online sources. "Web Page" was the most frequently cited document type, with "Electronic Personal Paper" ranking second and "Email Messages" and "e-journals" jointly occupying the third rank. We could not determine the types of twelve of the 83 online references because they contained incomplete bibliographic information and the online sources were not accessible.

One of the important findings of the reference study is that the citation styles of online references are frequently inconsistent, incomplete, and/or are inaccessible -- that is, they do not lead to the wanted online resource, in contrast to the citation styles of print references.

Each of the 83 references to online sources was examined and attempts were made to access the material cited. Obviously one of the potential advantages of citations to online resources is that if the reader can retrieve such resources quickly, the communication process has obviously been enhanced. The third column of Table 11 shows the results of the accessibility tests of the 83 references to online resources. Email messages and newsgroup postings were especially inaccessible, and none of the twelve resources that could not be classified could be accessed. Overall, only about half of the online resources were accessible online.

Table 12 reports accessibility findings in another way, by type of access protocol. Online information resources are accessible in several ways, and the 83 online references we found are analyzed according to these. Over half of the cited references included a Uniform Resource Locator (URL). We tried to access the online resources using a Web browser. However, only two-thirds of these led to the desired resource (see Table 12). Instead, we obtained the browser message "The server does not have a DNS entry" or the infamous server error message "404 URL Not Found". Several types of online references, such as Web pages, electronic personal papers, SGML encoded documents, software programs, and local file references, were mainly cited through URLs, often only with a URL.

Only four of the thirteen references including email addresses led to the desired online source. In seven cases, we received the message "User Unknown" in response to our email message. In two other cases, no error message was received, but there was no response and the online resource could not be obtained.

Nine of the online references were to listserv postings, with six (66.7%) being accessible. Accessible in this case meant that we could, through the given listserv access information such as name and discussion date, we could retrieve the original sources using listserv commands. However, three out of nine listserv references were not accessible. For instance, one listserv reference cited the 1994 file of CYBERIA-L. Although we could access the CYBERIA-L archive, we could not find the original source because the archive of CYBERIA-L did not include archives for 1994.

Seven Newsgroup postings were cited, from two different Newsgroups. We could access the archives of one of them but not the other, obtaining the message, "No such group." The Newsgroup that we could access maintained postings only from 1996. Finally, two newspaper articles that can be accessed through commercial online providers such as Nexis/Lexis were cited, and we were able to access these articles through the same providers.

We labeled five online references as "incomplete access information provided." These were cases in which no information was provided about how to access the source, or the information that was provided was insufficient for us to gain access to the resource.

In summary, except for commercially-provided electronic newspapers (to which not all Internet users will have access), none of the access methods led to perfect retrieval. The overall rate of 51.8% must be considered to be extremely low, considering that the online resources cited were apparently important enough for an author to cite them and yet in just one or two years, nearly half were no longer accessible, given the information provided in the citation.

Recall that the reference study was conducted to fix a kind of upper bound on the current influence of e-journals on scholarly communication. We argued that authors of e-journal articles would probably be more inclined than other authors to cite e-journals and other electronic sources. This still seems like a reasonable assumption to us.

We examined 4317 references in 279 scholarly, peer-reviewed articles appearing in 74 e-journals. Among these, only 83, or 1.9%, were to online sources, and only nine, or 0.2%, were to e-journals. Only twelve of the 74 e-journals in the sample cited one or more online sources. Only eight of the 74 e-journals cited another e-journal even once, and only one e-journal (not surprisingly, it was the e-journal e-journal) made two or more references to other e-journals. References to online sources are concentrated in a few e-journals, with PACS-R, Electronic Journal on Virtual Culture, [Note: ceased publication with Volume 4 No. 1, 1996. Ed.] and E-journal [Note: this journal appears to have ceased publication. I can find no trac of it. Ed.] accounting for more than 80% of all online references. And among this group, most are concentrated in just a few articles.

It is worth noting the identity of the e-journals in which the great majority of the references to online sources are made. The top three have as their central subjects of interest topics related to electronic communication. Among the twelve e-journals citing at least one online reference, the five e-journals dealing with online topics (PACS-R, Electronic Journal on Virtual Culture (EJVC), e-journal, Interpersonal Computing and Technology, Journal of Computer-Mediated Communication) account for 71, or 85.5 % of the references to online sources. This suggests that referencing online sources, and e-journals in particular, is a function mostly of academic discipline. One would expect a journal with a name like Electronic Journal on Virtual Culture to cite electronic sources, by virtue of the subject matter treated in the journal. Indeed, as subjects of academic treatment, the e-journals listed above focus on new topical areas that have arisen as a result of the new electronic environment.

On the other hand, the seven e-journals that have as their subject matter traditional academic specializations (sociology, education, library and information science, psychology, cognitive science, and mathematics), cite only twelve online sources between them, and five e-journal articles.

All of this taken together seems clear evidence that e-journals presently play almost no role in scholarly communication, as measured by references cited. Of course there are other ways of measuring impact; we do not claim that counting references and citations should be relied on exclusively. Interviews with scholars, for example, might well lead to a different conclusion -- but we doubt it. Certainly informal communication is being changed in a major way by the Internet -- especially by listservs, email, and web pages -- even if such communications are not being cited in large numbers. But the reference study suggests that a significant effect on the formal communication system is yet to happen. Obviously, this conclusion may well be very different in two or three year's time.

Tables 11 and 12 show that a high percentage (almost 50%) of online resources are not directly accessible from the information provided in references. Direct online accessibility to such resources is obviously potentially very convenient to readers, much more so than for print references. However, if readers cannot gain access to the original sources of the cited material, the references that link the citing and cited works are much less useful than references to print sources. Note that the references we studied appeared in recently published e-journal articles. One might reasonably expect such references to be largely correct and complete and to lead to the full online information source. This was not the case. It is worth contemplating what percentage of these same references will lead to the original information source in two or three years additional time -- say in the spring of 1998 -- it will very likely be much lower than 50%. Clearly, the accessibility of cited online resources is potentially a very serious problem in the conduct of research and scholarship, especially if the percentage of references to such sources increases beyond its current very small size.

Cited work study

The reference study reported in the previous section examined the cited references of 74 scholarly, peer-reviewed e-journals. In the cited work study, it is the references of thousands of print (and a very few electronic) journals that are of interest. The Institute of Scientific Information (ISI) publishes three citation indexes -- Science Citation Index (SCI), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) -- that allow searches of journal names as cited works. When a cited work search on an e-journal such as PACS Review is conducted on SSCI, one retrieves bibliographic citations to all the articles in journals indexed by SSCI that have cited PACS Review in one or more reference.

Since ISI indexes thousands of scholarly journals, which it represents as the most significant journals in the various fields of scholarship, a cited reference search should retrieve the majority of all citations made in scholarly journals to the cited work. Such a search will not reveal citations to a work made in less significant journals or in books and other materials. In the context of the World Wide Web, a cited work search will not reveal the hypertext links (analogous to citations) to works made in Web pages. However, these limitations notwithstanding, the ISI indexes are widely recognized as providing a valuable measure of the scholarly impact of articles, journals, and authors -- that is, the formal impact. Not the only, or even necessarily the best, measure, but valuable nonetheless.

Eugene Garfield, the founder of the Institute for Scientific Information, addressed the issue of the meaning of citation for individuals in a section of his book entitled "What Do Citation Counts Measure":

The only responsible claim made for citation counts as an aid in evaluating individuals is that they provide a measure of the utility or impact of scientific work. They say nothing about the nature of the work, nothing about the reason for its utility or impact. Those factors can be dealt with only by content analysis of the cited material and the exercise of knowledgeable peer judgment. Citation analysis is not meant to replace such judgment, but to make it more objective and astute. (Garfield, 1979, p. 246).

Much the same comment applies to the application of citation-based measures to the evaluation of journals. A number of studies have examined the meaning of citation as an evaluation tool for journals (McAllister, Anderson, & Narin, 1980) and research performance (Narin, 1976; Lawani, 1986 ). Bibliometric measures have also been used as methods for the evaluation of departments, universities, and published works.

This portion of the research reports the results of cited work searches of all three ISI citation indexes for the scholarly, peer-reviewed e-journals in the sample that have had a reasonable chance of being cited. Since the time lag in the publication schedules of print journals is substantial, we reasoned that a minimum of two years should be necessary to provide time for an e-journal article to be read, become part of an author's research, and eventually be cited in a print journal. For this reason, we limited the cited work searches to those scholarly, peer-reviewed e-journals that began publishing in 1993 or earlier. There were thirty-nine such e-journals in the sample.

The cited work study is in progress. Results from this portion of the research will be presented at the conference.

Acknowledgements

Note

This paper was originally delivered at the Midyear Meeting of the American Society for Information Science, San Diego, California, May 20-22, 1996 and is available in the conference Proceedings, pp. 299-315.

References

Borgman, C. L. (Ed.). (1990). Scholarly communication and bibliometrics. Newbury Park, CA: Sage Publications.

Brown, H. (1995). ASCII citation of electronic documents. URL http://library.ccsu.ctstateu.edu/~history/docs/cite.html.{Document no longer available 12.2.2002}

Collins, M. P., & Berge, Z. L. (1994). IPCT Journal: A Case Study of an Electronic Journal on the Internet. Journal of the American Society for Information Science, 45, 771-776.

Garfield, E. E. (1979). Citation indexing: its theory and application in science, technology, and humanities. New York: Wiley.

Harter, S. P., Nisonger, T. E., & Weng, A. (1993). Semantic relationships between cited and citing articles in library and information science journals. Journal of the American Society for Information Science, 44, 543-552.

Hitchcock, S., Carr, L., & Hall, W. (1996). A survey of STM online journals 1990-95: The calm before the storm. January 15, 1996 (Updated February 14, 1996). Available at URL http://journals.ecs.soton.ac.uk/survey/survey.html.

Keyhani, A. (1993). The Online Journal of Current Clinical Trials: An innovation in electronic journal publishing. Database, 16, 14-23.

Lawani, S. M. (1986). Some Bibliometric Correlates of Quality in Scientific Research. Scientometrics, 9, 13-25.

Li, X., & Crane, N. B. (1993). Electronic style: A guide to citing electronic information. Westport, CT: Meckler.

McAllister, P. R., Anderson, R. C., & Narin, F. (1980). Comparison of peer and citation assessment of the influence of scientific journals. Journal of the American Society for Information Science, 31, 147-152.

Mecklermedia. (1994). Internet World's on Internet 94: An international guide to electronic journals, newsletters, texts, discussion lists, and other resources on the Internet. Westport: Mecklermedia.

Meyer, R. W. (1993). Roles in scholarly communication under an electronic model. [email to Steve Harter], [Online]. Available email: [email protected].

Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Cherry Hill, New Jersey: Computer Horizons, Inc.

Okerson, A. (Ed.). (1994). Directory of electronic journals, newsletters and academic discussion lists (4th ed.). Washington, D.C.: Association of Research Libraries.

Osburn, C. B. (1984). The place of the journal in the scholarly communications system. Library Resources and Technical Services, 28, 315-324.

Schauder, D. (1994). Electronic publishing of professional articles: Attitudes of academics and implications for the scholarly communication industry. Journal of the American Society for Information Science, 45, 73-100.

Smith, L. C. (1981). Citation analysis. Library Trends, 30, 83-106.

Turoff, M., & Hiltz, S. R. (1982). The electronic journal: A progress report. Journal of the American Society for Information Science, 33, 195-202.

Walker, J. R. (1995). MLA-Style Citations of Electronic Sources. January 1995 (Rev. 4/95). URL http://www.columbia.edu/cu/cup/cgos/idx_basic.html.

How to cite this paper:

Harter, Stephen P. & Kim, Hak Joon (1996)   "Electronic journals and scholarly communication: a citation and reference study".  Information Research, 2(1) Available at: http://InformationR.net/ir/2-1/paper9a.html

© the authors, 1996.