Vol. 9 No. 4, July 2004 Contents | Author index | Subject index | Search | Home

An empirical study of the effect of information technology expenditures on student achievement

Alan R. Peslak
Information Sciences and Technology
Pennsylvania State University, Worthington-Scranton
Dunmore, Pennsylvania, USA

The impact of information technology on productivity in the private sector has been extensively researched. But the study of the impact of information technology expenditures in schools has been limited. This study of 1090 California schools and including over 6,000,000 students, attempts to address this issue through an analysis of IT expenditures at the school level and the effect on standardized reading and mathematics test scores. Thirteen other factors were also included in this analysis of the 2001-2002 academic year. Included are public school grades two through eleven. The results indicate that socio-economic status as measured by the percentage of students receiving free or reduced meals was the most significant factor in determining test scores. Also significant was percentage of fully qualified teachers. Information technology as measured by a number of factors did not show significant and positive effects on student performance.

Introduction

Robert Solow in 1987 started the debate on whether IT expenditures have actually increased productivity in his observance that computers could be seen everywhere but in the productivity statistics. (Solow, 1987). His reference was to the fact that modern industry was spending significant sums on computers, information systems, and related technology, and yet productivity in the private sector did not seem to be displaying corresponding gains in output. This topic has become a major research issue since that time and no conclusion has been reached on whether information technology does positively impact productivity in the private sector. One of the leading economic productivity researchers found in 1993 in his article "The Productivity Paradox of Information Technology" that at a firm level, IT expenditures did not correlate with increased output per hour. (Brynjolfsson, 1993) Educational institutions including public K-12 [1] schools have spent significant sums over the past several years on information technology as well. One of the goals of these expenditures was improved education of their students. But little work has been done determining whether these expenditures have led to improved performance on the part of their students. This study is an attempt to determine whether there is a Productivity Paradox among educational institutions, similar to the perceived Productivity Paradox in the private sector. In other words, do information technology expenditures at the K-12 level improve student performance and achievement?

Most of the school age children in the United States are educated through state tax funded public schools. There are thirteen grade levels, kindergarten and one through twelve grades. Educational institutions in the United States including public schools have spent significant sums over the past several years on information technology as well. One of the goals of these expenditures was improved education of their students. But little work has been done determining whether these expenditures have led to improved performance on the part of their students. This study is an attempt to determine whether there is a Productivity Paradox among educational institutions, similar to the perceived Productivity Paradox in the private sector. In other words, do information technology expenditures at the K-12 level improve student performance and achievement?

Over the past decade, billions of dollars have been spent on K-12 information technology for student use. Three instances of major expenditures have been detailed by Rowe (1998), including $32 million in Cobb County Georgia, $26 million in Austin, Texas and $44 million in Plano Texas. These were all specifically identified as IT expenditures due to floating of separate and itemized bond issues. Extrapolating just these few instances leads to a multi-billion price tag to provide information technology resources to K-12 public institutions.

Background of the Productivity Paradox

The Productivity Paradox in industry has been well studied and many of these studies have shown a lack of correlation between IT and productivity. Berndt and Morrison in 1995 studied data from a variety of governmental agencies and concluded that costs of adjusting the business to accommodate technology outweighed any benefits. Specifically productivity decreased and labor rose after information technology investments.

One of the first studies on the relationship between IT and productivity was performed by Cron and Sobel in 1983. The authors studied 138 wholesalers of the size of 1 -- 10 million dollars and found that firms using computers extensively were either very high or very low performers. If used ineffectively computers can actually increase costs and reduce productivity. There was no consistent trend showing positive returns from information technology spending and productivity, however.

In Paradox lost? Firm Level Evidence on the Returns to Information Systems Spending, Brynjolfsson and Hitt (1996) performed a firm level study that did suggest increases in firm level output with increased information technology spending. One of the main reasons suggested for past negative correlations was mismanagement of IT operations and expenditures. Done properly they suggest IT can and does increase productivity according to the authors.

Rai, Patnayakuni, and Patnayakuni (1997) analyzed one year of InformationWeek data supplemented by Compustat and ended up with mixed information technology productivity results over a series of output measures. They note that IT expenditures positively correlated with labor and output productivity but return on assets and administrative productivity did not show consistent positive results.

Paul Strassmann (1999) sees little productivity gains from IT and suggests three main reasons why though information technology has not increased productivity namely, bloated software, negligent software engineering, and high support costs. Any gains have more than been reduced by higher operating and support costs.

Tam in Information System Research (1998) studied the impact of information technology investment on firm performance for non-US firms . The hypothesis was that information technology correlates positively with return on equity, return on assets, return on sales, total shareholder return and market value of the firm.(used as productivity measures. The results of the study were that none of the productivity measures showed consistent positive significant returns based on information technology spending.

Educational studies

The concept of an Educational Productivity paradox, though no less important, has not been as frequently analyzed. A few researchers however have studied the importance of other factors on educational productivity as measured by student performance.

A study performed by Stern (1989) using California students in grades 3 and 6 uses California assessment program (CAP) test scores as the dependent variable. The independent variables include years of teacher experience, teacher salary, teacher/student ratio, and overall school size. The significant and positive factors on student achievement were overall school size and years of teacher experience. Expenditures or class size did not have statistically significant results.

Berger and Toma (1994) find that requiring a master's degree for high school teachers actually had a negative impact on student achievement as measured by SAT scores. This is an example where just money will not necessary bring improvements.

Montmarquette and Mahseredjian (1989) review the concept of whether schools themselves have a significant and positive impact on student performance. Their review uses a two way nested-error components specification as well as the more common multiple regression technique. They analyzed 39 first and fourth grade classes in Montreal public elementary schools in 1979-1980. They utilized a total of 19 variables split into four major categories: personal characteristics (such as self-concept and IQ), socioeconomic variables, class variables (such as teacher experience), and finally school variable (such as school size). The regression study found that personal and socioeconomic variables are the factors affecting academic achievement as measured by uniform test scores. Class variables and school variables were found to be unimportant. The two way nested approach did not improve the importance of either class or social variables in explaining test scores. The impact of class and school variables remain negligible.

Battle (1997) studies academic achievement as measured by standardized composite of math and reading tests administered to students. The sample population was the result of the National Educational Longitudinal Study of 1988. The study reviewed the impact of one versus two parent homes on the composite test scores of Hispanic students. Independent variables used included sex, socioeconomic status, urbanicity, school size, free lunch, family size, birth year as well as one versus two parent families. The study found that two parent families had an effect, but once socioeconomic status was entered into the equation the effect was eliminated. Thus, it was the income level of the two parent family that had the positive effect not the number of parents.

Card and Krueger (1992) use a different dependent variable and find positive monetary returns to quality schools. They note that many past studies have failed to show a relationship between quality of schools and standardized test scores. Card and Kruger find positive correlation between quality schools and income levels of the graduates.

Harnisch (1987) studied effective public high schools using a 1980 and 1982 comprehensive sophomore survey of nearly 19,000 students. After studying nearly two dozen factors, he found through multiple linear regression that school factors can make a difference in student test scores in verbal, math, and science. Social and economic status, number of school hours in the day, percentage of black and Hispanic students, and locus of control (the belief that success or failure are under the student's control) all are significantly associated with test scores. All are positively affected except the percentage of black and Hispanic students. Harnisch argues that the number of hours in the day and locus of control are able to be controlled by the school and positively affect achievement.

There have also been a few researchers who have started to explore the relationship between information technology and student performance.

Attewell and Battle (1999) reviewed the effects of home computers on standardized test scores in reading, mathematics. The study used a simultaneous approach to multiple linear regression using all the variables to predict dependent variables. Other independent variables used in the study included family size, urbanism, family structure, gender, ethnicity, social, and cultural support, and geographic region. The study used the National Longitudinal Study of 1988 produced by the National Center for Educational Statistics in the Department of Education. There was shown to be a 3-5% gain for students who did have home computers even after controlling for socio-economic status which was the most influential factor in standardized test scores.

Macmillan, Liu, and Timmons (1997) prepared a study that suggests that rapid implementation of technology in education needs to be accompanied by proper integration, technical support, parental support, and training. Just the purchasing of equipment and the resulting implementation will not by itself improve education. There needs to be a full integration framework and environment for the technology to be successful.

Hokanson and Hooper (2000) report that technology use in education generally has poor results. Specifically technology has been used only to automate existing educational processes and thus has shortchanged its potential. The computer should not be used as a mathematics machine. Only by thinking broader can the full potential of technology be realized leading to improved educational quality and higher student test scores and productivity.

Pelgrum and Plomp (1993) study the use of computers in 18 countries and find problems with the support and infrastructure in the classroom. They note general lack of teacher time, poor software, and insufficient training as several of the reasons why technology is not as successful as it can be.

Hapson, Simms, and Knezek (2001-2002) sampled a group of fifth and sixth grade students in a North Central school district. They provided a technology rich environment for a classroom magnet group. They used the Ross Test of Higher Cognitive Process as a measure of effectiveness of higher order thinking and analytical skills. The technology rich environment included increased computers, teacher training, and specific courses in Office type applications, Internet access, and electronic resources. The data analysis showed a significant positive effect on higher order thinking skills. But the positive effect was very small and was not significant in every category. In addition there is the potential for a bias since the students were specifically selected and measured and also attended select magnet schools.

Page (2002) studied 211 students in 10 classes in five Louisiana elementary schools. Five of the classes included significant technology in the classrooms whereas five did not. The five technology enriched classrooms included items such as computers, Internet access, digital cameras, and printers. All of the classes were from low socioeconomic backgrounds and included third and fifth grade. The technology was incorporated into the instruction. In this particular situation Page found a statistically significant improvement in mathematics achievement, and self-esteem among the technologically enriched.

Background of the study

In order to determine the existence of a Productivity paradox in education, a comprehensive review of actual expenditures was required. The state of California was chosen for this review due to its large population and the existence of excellent recent data on information technology in public schools as well as comprehensive and consistent testing to measure performance. The state of California is by far the largest of the fifty states in the United States representing more than 12% of the total population. California was also chosen because it exhibits a diverse ethnic and socio-economic population. The study measures the relationship of a variety of independent socioeconomic, demographic, school and teacher specific statistics, as well as information technology variables against dependent variables as measured by the Academic Performance Index, a standardized test in reading and mathematics administered each year in grades 2 through 12. Each school and grade is analyzed. The data are from the California Department of Education.

The study represents 2001-2002 statistics for California 2-11 grades and represents 1090 schools and 6,149,253 students.

According to the California Department of Education website "The API is the centerpiece of the statewide accountability system in California public education. The Public Schools Accountability Act (PSAA) of 1999 (Chapter 3, Statutes of 1999) requires that the California Department of Education (CDE) annually calculate APIs for California public schools, including charter schools, and publish school rankings based on these APIs. The PSAA also requires the establishment of a minimum five-percent annual API growth target for each school as well as an overall statewide API performance target for all schools."

The dependent and independent variables used in the study for each of the regression analyses are shown in Table 1. The choice of independent variables was based on past studies and available data. Past studies have suggested the importance of school size and classroom environment in student performance (Montmarquette and Mahseredjian, 1989). Available data was culled from California state sources that specifically dealt with these areas including class size, overall enrollment, and pupil teacher ratio. In addition many studies have found a link between socio-economic status and student performance (Battle,1997 ). Data relating to this issue were found and incorporated including free and reduced meals and aid to families with dependent children. To these suggested factors were added all data available relating to technology and technology expenditures in a school. These factors include data on numbers of computers, WAN (wide area network), and Internet enabling. Data which was not supported by past surveys was not included such as geographic information. The data were analyzed using SPSS 10.0.5 using multiple linear regression.

Results

Table 2 shows a summary of the statistically significant independent variables for both the reading and math achievement scores. In all there were 10 separate regression analyses performed for both the reading and math analyses, one for each of the grades 2 through 11. Table 2 shows the number of studies where the factor was found to be statistically significant at the p<.05 level.

The percentage of fully credentialed teachers and number of students receiving free or reduced cost meals (an indicator of low economic status) were statistically significant in all 10 grade reading analyses. Likewise both these factors significantly affected math scores as well. No other variable affected reading test scores more than four grades. This was pupil teacher ratio which was only significant in four of ten grades. None of the technology factors including number of computers, number of students per computer, computers with CDROM, classrooms with Internet, classrooms connected to WAN, %computers with CD, and students per class with Internet access had significant correlation with reading scores in more than a few classes.

A similar situation occurred in math achievement. Two additional factors correlated with math achievement in a majority of classes: average class size and students per computer with CDROM. But when the analysis of these factors is taken further, both yield unusual results.

Math Proficiency

Table 3 shows the factors that affected math scores in the California study where the number of significant grades was greater than 50% (6 out of 10). The factor most influencing math proficiency was the socioeconomic factor (% free or reduced meals). In all 10 grades the factor was negative and was the highest. Generally, for each 1 percent increase in students with free or reduced meals, there was a corresponding .36% decrease in the percentage of students achieving above 50% in math proficiency compared to the national average. The percentage of fully credentialed teachers was also significant in all ten grades but its influence was less marked. For each one percent increase in fully credentialed teachers, there was a .004 percent increase in math test percentage. Interestingly the affect of fully credentialed teachers seems more important in succeeding years with only a .00146 affect in second grade but a .00639 affect in tenth grade. The factor of average class size, though significantly correlated in six of the ten grades (3, 4, 5, 9, 10, and 11), showed a positive effect in all cases. In other words, the larger the average class size, the higher the math achievement. This seems counterintuitive and as noted four of the classes did not show a significant correlation. The factor of number of students per computer with CDROM also was significant in six of ten classes (2, 3, 4, 7, 8 and 10). But in second grade the relationship was positive. In only half the classes was the relationship was negative where more students per computer reduced math achievement. The influence was also relatively low with an average of .003 percent increase in the percentage of students achieving above 50% in math proficiency, compared to the national average percentage for every 1 percent decrease in number of students per computer.

The final analysis of math scores is shown in Table 4 which presents both coefficients and p values for all factors analyzed for the selected grades of second, sixth, and eleventh. As stated, there were many factors included in the analysis but only the noted few were statistically significantly correlated with math achievement test scores. None of the myriad technology factors showed consistent positive contributions and surprisingly, pupil teacher ratio was not a significant factor.

Reading Proficiency

As described, none of the technology factors showed consistent statistically significant correlation with reading achievement in grades two through eleven in California. Table 5 shows the two factors which did consistently correlate with reading test scores: percentage receiving free and reduced meals and percentage of fully credentialed teachers. There was found an average of .5 % decrease in the percentage of students achieving above 50% national average on reading scores for every one percent increase in students receiving free and reduced meals. An interesting finding is that this effect reduces somewhat over the 10 school years, with a .5-.6 impact in early grades and only a .335 impact in eleventh grade. This is opposite the effect of percent of fully credentialed teachers which has an increasing impact (though considerably less overall) from early grades to higher grades. The average impact of percentage of fully credentialed teachers is a .004 percent increase in achievement with every one percent increase in fully credentialed teachers.

The final analysis of math scores is shown in Table 6 which presents both coefficients and p values for all factors analyzed for the selected grades of second, sixth, and eleventh. As in math, there were many factors included in the analysis but only the noted few were statistically significantly correlated with reading achievement test scores. None of the technology factors showed consistent positive contributions and neither pupil teacher ratio nor average class size were significant factors.

Conclusion

This study reviewed 1090 California schools and over 6,000,000 students and found no consistent positive correlation between information technology and higher reading or math scores in any of the grades 2 through 11 as measured by the California standardized test scores. The study confirmed the strong negative correlation between socioeconomic status and student performance and also found that the percent of fully credentialed teachers promoted higher test scores in all grades. The data surprisingly reflected significant and somewhat positive correlation between average class size and math scores in 6 of the 11 grades. Further study is warranted to confirm these findings in other states. A working paper by the author has found similar overall results in Pennsylvania with relation to information technology expenditures. (Peslak, 2003). Detailed analysis needs to be performed to discover why IT expenditures do not translate into more effective tools to further reading and writing skills.

Notes

1. 'K-12': signifies the 'Kindergarden to Grade 12' levels of education in the USA. Grade is not directly related to age, but Grade 12 would typically be 16 years of age.

• Attewell, P. & Battle, J. (1999). Home computers and school performance. The Information Society, 15(1), 1-10.
• Battle, J. (1997). Academic achievement among Hispanic students from one versus two-parent households. Hispanic Journal of Behavioral Sciences, 19(2), 156-170.
• Berger, M. & Toma, E. (1994). Variation in state education policies and effects on student performance. Journal of Policy Analysis and Management, 13 (3), 477-491.
• Berndt, E. & Morrison, C. (1995). High-tech capital formation and economic performance in US manufacturing industries: an exploratory analysis. Journal of Econometrics, 65(1), 9-43.
• Brynjolfsson, E. (1993). The productivity paradox of information technology. Communications of the ACM, 36(12), 67-77.
• Brynjolfsson, E. & Hitt, L. (1996). Paradox lost? Firm-level evidence on the returns to information systems spending. Management Science, 42(4), 541-558.
• Card, D. & Krueger, A. (1992). Does school quality matter? Returns to education and the characteristics of public schools in the United States. The Journal of Political Economy, 100(1), 1-40.
• Cron, W., &Sobol, M. (1983). The relationship between computerization and performance. Information and Management, 6(3), 171-181.
• Hapson, M., Simms, R., & Knezek, G. (2001-2002). Using a technology-enriched environment to improve higher order thinking skills. Journal of Research on Technology in Education, 34(2), 109-119.
• Harnisch, D. (1987). Characteristics associated with effective public schools. Journal of Educational Research, 80(4), 233-241.
• Hokanson, B. &Hooper, S. (2000). Computers as cognitive media: examining the potential of computers in education. Computers in Human Behavior, 16, 537-552.
• Lehr, B., &Lichtenberg, F. (1999). Information technology and its impact on productivity: firm-level evidence from government and private data sources, 1977-1993. The Canadian Journal of Economics, 32(2), 335-362.
• Macmillan, R., Liu, X., &Timmons, V. (1997). Teachers, computers, and the Internet: The first stage of a community-initiated project for the integration of technology into the curriculum. The Alberta Journal of Educational Research, 53(4), 222-234.
• Montmarquette, C. &Mahseredijan, S. (1989). Does school matter for educational achievement? A two-way nested error components analysis. Journal of Applied Econometrics, 4(2), 181-193.
• Page, M. (2002). Technology-enriched classrooms: effects on students of low socioeconomic status. Journal of Research on Technology in Education, 34(4), 389-409.
• Pelgrum, W. &Plomp, T. (1993). The use of computers in education in 18 countries. Studies in Educational Evaluation, 19, 101-125.
• Peslak, A. (2003). The educational productivity paradox. Unpublished manuscript.
• Rai, A., Patnayakuni, R., &Patnayakuni, N. (1997). Technology investment and business performance. Communications of the ACM, 40(7), 89-97.
• Rowe, S. (1998). Utopia or a scary monster: a discussion of the effectiveness of computer technology. Contemporary Education, 69(3), 144-9.
• Stern, D. (1989). Educational cost factors and student achievement in grades 3 and 6: Some new evidence. Economics of Education Review, 8(2), 149-158.
• Solow, Robert. (1987, July 12). We'd better watch out. The New York Times, p.36.
• Strasssmann, P. (1999). IT paradox number. Computerworld, 33(18), 44.
• Tam, K. (1998). The impact of information technology investments on firm performance and evaluation: evidence from newly industrialized economies. Information Systems Research, 9(1), 85-98.

Web Counter

© the author, 2004. Last updated: 12 February, 2004