The Review Project

Led by John Hilton III

Additional Contributor: Stacie Mason

Sept. 2016 Updates in Red

This review provides a summary of all known empirical research on the impacts of OER adoption (including our own). The version below will be periodically updated with new articles as we become aware of them. If you know of an empirical research study on the impacts of OER adoption that is not included in this review, please leave a comment below. An in-depth article focusing on empirical research relating to perceptions and efficacy at the college level was published by the journal Educational Technology Research and Development in February of 2016. Please access the open-access version of this article. You can also access overview slides that summarize this research. 

Introduction

Open Educational Resources are teaching and learning materials that provide users with (1) free and unfettered access and (2) 5R legal permissions to retain, reuse, revise, remix, and redistribute them, that can be used to replace traditional expensive learning resources (such as textbooks). A recent nationally representative survey of 2,144 faculty members in the United States found that “most faculty remain unaware of OER” (Babson Survey, 2014 ).

This same survey found that college professors rate “proven efficacy” and “trusted quality” as the two most important criteria for selecting teaching resources. Thus we believe that for OER to gain traction it is important to gather empirical research demonstrating its efficacy and quality.

To this end, we have gathered articles that focus on the efficacy of OER or teacher/student perceptions of such resources in actual practice. We have limited our studies to those in which OER were the primary learning resource(s) and were compared against traditional learning resources; in addition, the study needed to include at least 50 participants. We are also gathering smaller studies focused on efficacy or perceptions and posting them on this page.

We originally only included research published by a peer-reviewed journal. We are currently in the process of also including white papers and/or completed theses and dissertations based on the recommendations of Polanin, Tanner-Smith, and Hennessy (2016).  Please let us know of any studies we are missing.

Studies that included data on both efficacy and perception

Research by Feldstein et al. (2012) took place at Virginia State University, where OER were implemented across nine different courses in the business department. 1,393 students took courses utilizing OER. Researchers found that students in courses that used OER more frequently had better grades and lower failure and withdrawal rates than their counterparts in courses that did not use OER. While their results had statistical significance, because of a new core curriculum employed at Virginia State University’s Business school, the two sets of courses were not identical. Thus while these data are highly interesting, we should not generalize them too far. 315 students completed a survey regarding their perspective on the shift to the OER, and almost 95% of responding students strongly agreed or agreed that the OER were “easy to use” and 78% of respondents felt that the OER “provided access to more up-to-date material that is available in my print textbooks.” Approximately two-thirds of students strongly agreed or agreed that the digital OER were more useful than traditional textbooks and that they preferred the OER digital content to traditional textbooks.

Hilton et al. (2013) chronicled a study that took place at Scottsdale Community College (SCC) in Arizona. In the fall of 2012, OER were employed throughout five different math courses at SCC, affecting 1,400 students. Issues with the initial placement tests made it so that only four of the courses could be compared; nevertheless, the results of Fall 2012 (when OER was used) compared to Fall 2011 and 2010 showed that student results on department exams were approximately the same before and after the OER implementation. Surveys completed by 910 students showed that 78% said they would recommend the OER to their classmates. Similarly, 83% of students agreed with the statement that “Overall, the materials adequately supported the work I did outside of class” (only 5% of students disagreed with this statement). Faculty members were likewise positive about the open materials. Of the 18 faculty members who reported on their view of the OER, 50% said that it was of the same quality as traditional textbooks, 33% said it was better, and 17% said it was worse.

Gil et al. (2013) (OA preprint, final published version), surveyed students over a five year time, asking them to compare blogs that featured OER resources versus other blogs (all blogs were directly related to student coursework). Approximately 500 students were surveyed; on average, 40% of students said that the blogs featuring OER were of equal quality to the blogs that did not feature OER, 45% of students said that the blogs with OER were superior and 15% said they were inferior. They also found a correlation between increasing access to OER and improving student grades; however, these results were tentative as they were not able to compare the results from individual students; rather this was a trend that was observed over time and may possibly have been due to other factors.

Studies that focused on efficacy

Lovett et al. (2008) measured the result of an implementation of an online, OER component of Carnegie Mellon University’s Open Learning Initiative (OLI). Over two semesters, forty-four students utilized the OER as part of this study. Researchers examined test scores (three midterm and one final exam) of those students who took the traditional course versus those who utilized the OER materials. They found no significant difference between the two groups.

Bowen et al. (2012) completed a rigorous study comparing the use of a traditional statistics textbook with Carnegie Mellon’s OLI at six different institutions. Participating students were randomly assigned to either the face-to-face class with a traditional textbook, or a “hybrid” class that used the OER resource. Both groups took the same standardized test at the beginning and end of the semester, as well as a final examination. 605 students took the OER version of the course, while 2,439 took the traditional version. Students who utilized OER performed slightly better on the standardized exam than those who did not. The difference in outcomes was not statistically significant. See also Bowen et al. (2014)

Hilton and Laman (2012) (Taylor & Francis Version, Open Repository Preprint), focused on introductory Psychology courses taught at Houston Community College (HCC). In the fall of 2011, twenty-three sections composed of 690 students used an open psychology textbook. The introduction of an open textbook was correlated with an increase in class grade point average, an increase of the average score on the departmental final examination, and a lower course withdrawal rate. No causation was claimed. 

Wiley et al. (2012) examined the standardized test scores of students using open textbooks in secondary science classes in three different school districts. Approximately 1,200 students used open textbooks during this study. Researchers examined their end-of-year standardized test results and found no apparent differences between the results of students who used traditional and open textbooks.

Pawlyshyn et al. (2013) reported on the adoption of OER at Mercy College in New York. In the fall of 2012, 695 students utilized OER in Mercy’s basic math course, their pass rates were compared with those of the fall of 2011, in which no OER were utilized. Researchers found that the pass rates increased from 63.6% in fall 2011 to 68.9% in fall 2012 when all courses were taught with OER. Similarly, students who were enrolled in OER versions of a reading course performed better than their peers who enrolled in the same course using non-OER materials.

Robinson et al. (2014) examined the use of open science textbooks in three secondary science subjects across several schools in a suburban school district. This rigorous study used propensity score matched groups in order to control for teacher effect, socioeconomic status, and eight other potentially confounding variables. There were 1,274 students in each condition (treatment and control). In examining the results of the end-of-year state standardized test there were very small, but statistically significant difference between the two groups, favoring those who utilized OER.

Fischer et al (2015) performed follow-up research on the second year of implementation at the schools studied by Robinson (2014). Their original sample consisted of 16,727 students (11,818 control and 4,909 treatment).  From this sample, there were fifteen courses for which some students enrolled in both treatment (n=1,087) and control (n=9,264) sections (the remaining students enrolled in a course which had either all treatment or all control sections and were therefore excluded). While this represents a large sample size, students in treatment conditions were only compared with students in control conditions who were taking the same class in which they were enrolled. For example, students enrolled in a section of Biology 111 that used OER were only compared with students in Biology 111 sections that used commercial textbooks (not students enrolled in a different course). Thus when diffused across fifteen classes, there was an insufficient number of treatment students to do propensity score matching for the grade and completion analyses. The researchers found that in two of the fifteen classes, students in the treatment group were significantly more likely to complete the course (there were no differences in the remaining thirteen). In five of the treatment classes, students were significantly more likely to receive a C- or better. In nine of the classes there were no significant differences and in one study control students were more likely to receive a C- or better. Similarly, in terms of the overall course grade, students in four of the treatment classes received higher grades, ten of the classes had no significant differences, and students in one control class received higher grades than the corresponding treatment class. Researchers utilized propensity score matching before examining the number of credits students took in each of the semesters as this matching could be done across the different courses. Drawing on their original sample of 16,727 students, the researchers matched 4,147 treatment subjects with 4,147 controls. There was a statistically significant difference in enrollment intensity between the groups. Students in fall 2013 who enrolled in courses that utilized OER took on average two credit hours more than those in the control group, even after controlling for demographic covariates.  ANCOVA was then used to control for differences in fall enrollment and to estimate differences in winter enrollment. Again, there was a significant difference between the groups, with treatment subjects enrolling in approximately 1.5 credits more than controls.

Allen et. al (2015) designed an experiment in which an experimental class of 478 students used the OER ChemWiki as its primary textbook, while the control class of 448 utilized a commercial textbook. The two sections were taught the same semester at back-to-back times using the same faculty member and teaching assistants. Students in both classes were given the same midterm and final exams. Researchers found no significant differences between the two groups both with overall exam results and item-specific questions. Beginning of the semester pre-tests, combined with final exams showed no significant differences in individual learning gains between the two groups. Student surveys regarding time spent on the class found that students in both groups spent approximately the same amount of time preparing for class.

Robinson (2015) devised a quasi-experimental study involving seven courses at seven colleges involved in Project Kaleidoscope. Outcomes of students using open textbooks (n=3,254) were compared with outcomes of students using traditional textbooks (n=10,819) in comparable courses. Controlling for covariates, Robinson found that students using open textbooks in the business and psychology courses earned lower average grades and were less likely than students using traditional textbooks to pass the course with a C- or better. Students in other five courses showed no significant difference in average grades or successful pass rates. In the biology courses, students using OER were more likely to complete the course than students using traditional textbooks. Across all courses, students using open textbooks took slightly more credits than students using traditional textbooks. This study is important in that it contains “the first finding of a negative effect associated with OER adoption” (p.59).

Wiley et al. (2016) focuses on open textbook adoption at Tidewater Community College. Across two pilot semesters (fall 2013 and spring 2014) 23,232 students enrolled in traditonal versions of selected courses. An additonal 753 students enrolled in “Z” versions of these same courses (“Z” standing for “Zero textbook costs”). 3.57% of students dropped the course in the traditional courses versus 2.79% in the Z courses – a statistically significant difference. The authors hypothesize that money kept by universities from lower drop rates could support open textbook initatives.

Hilton et al. (in press) focuses on an additional two semesters at Tidewater (their results include fall 2013, spring 2014, fall 2014, and spring 2015). They studied the combined the drop, withdrawal, and C or better grade analyses to estimate the throughput rates of students taking traditional verses Z courses. In the face-to-face courses (Control n = 36,223 Treatment n = 1,151) 59.8% of students in non-Z courses made it through the successive hurdles of drop, withdrawal and passing the class, compared with 66.4% of students in the Z courses, for a difference of 6.6%. Combined the drop, withdrawal, and C or better grade analyses to estimate the differences between the groups in the overall success rate from students’ registration to final grade. In the hybrid/online courses: (Control n = 7,000, Treatment n = 863) 54.2% of students who started in non-z courses successfully made it through the course with a C or better, compared with 59.8% of students in the Z courses, for a difference of 5.6%.

 

Studies that focused on perceptions

Petrides et al. (2011), administered surveys to instructors and students who utilized an open statistics textbook called Collaborative Statistics. In total, 31 instructors and 45 students participated in oral interviews or focus groups that explored their perceptions of the OER which they had utilized. The authors found that “Cost reduction for students was the most significant factor influencing faculty adoption of open textbooks” (p. 43), partly because it increased student access. 65% of students surveyed reported a preference for using open textbooks in the future because they are generally easier to use.

Bliss et al. (2013a) reported on surveys completed by 11 instructors and 132 students at seven different colleges. 60 percent of instructors stated their students were equally prepared when OER replaced traditional texts and 30 percent said their students were more prepared. One of the eleven instructors felt students were less prepared. All 11 instructors surveyed stated they would be very likely to use open texts in future courses. The students were also very positive regarding the OER materials. When asked to compare the OER texts to traditional texts, only 3% felt that PK texts were worse than their typical textbooks, 56% said they were the same quality, and 41% said they were better than typical textbooks.

In an extension of the previous study, Bliss et al. (2013b), studied OER adoption at eight different institutions of higher education. They surveyed an additional fifty-eight teachers and 490 students across the eight colleges regarding their experiences in utilizing OER. Approximately 50% of students said that the OER materials had the same quality as traditional textbooks and nearly 40% said that they were better. Students focused on several benefits of the open textbooks. Many cited technical advantages of the digital texts. In addition, the free cost of their open texts was important to many students. 55% of teachers reported that the open materials were of the same quality as the materials that had previously been used, and 35% felt that they were better.

Lindshield and Adhikari (2013) studied the perceptions of students who utilized a digital OER textbook in a Human Nutrition class. One hundred and ninety-eight students completed a survey in which they shared their perceptions of the OER text. “Students favorably rated their level of satisfaction, liking the idea of the [digital OER], ease of [digital OER] use, not having to buy a textbook, and preferring the [digital OER] versus buying a textbook for the course.” Moreover they found that students disagreed or somewhat disagreed with statements to the effect that they would like to have a traditional textbook in addition to OER.

Allen and Seaman in their Babson Survey (2014) surveyed 2,144 college professors regarding their opinions on OER. Of the 34% (729) who expressed awareness of OER, 61.5% of respondents said that OER materials had about the same “trusted quality” as traditional resources, 26.3% said that traditional resources were superior, 12.1% said that OER were superior. 68.2% said that the “proven efficacy” were about the same 16.5% said that OER had superior efficacy and 15.3% said that traditional resources had superior efficacy. It is important to note that the professors surveyed in this study expressed awareness of OER, but had not necessarily utilized OER. In contrast, in the above-mentioned four studies the professors had actually utilized OER, perhaps having a greater basis on which to judge the quality of OER resources.

Pitt (2015) reported results of two surveys of 126 educators, conducted 2013 and 2014-2015, as part of a collaboration between OER Research Hub (OERRH) and OpenStax College (OSC), a provider of open textbooks.  Around 65% of respondents reported that using OSC helped them meet diverse learners’ needs, while a minority of respondents said that the OSC materials made teaching easier, enabled innovation or changed their pedagogical approach. More than 65% of respondents perceived greater learner satisfaction for their students using OER.  Nearly all respondents said that  having using OSC materials increased the likelihood that they would recommend OSC materials to peers.

Jhangiani, Pitt, Hendricks, Key, and Lalonde (2016) examined awareness, usage, outcomes, and perceptions of OER among British Columbia post-secondary faculty. As part of a collaborative project between the BCcampus-led Open Textbook Project and OER Hub, researchers surveyed post-secondary educators through an online survey disseminated via email and social media. Of the 78 respondents, 77% had used OER. Most respondents rated OER quality as comparable or superior to that of traditional materials; educators who had adopted OER rated the quality significantly higher than educators who had not. Respondents reported that the top two barriers to using OER were finding relevant and high quality OER. Faculty at research-intensive universities reported significantly lower barriers to finding high-quality OER than did faculty at teaching-intensive universities or colleges/institutes, and were the most likely group to create or adapt OER.

The California OER Council (2016) released a white paper focused on OER adoption in CA higher education. 351 students completed a survey about their use of OER. When students were asked if the OER textbook chapter(s) were better than the traditional, 42% said the OER textbook as better, 39% said they were about the same, 11% rated the textbook as worse than the traditional textbook and 8% declined to answer. Of the 351 students in the survey, 71 printed the textbook and 209 used a PDF. 16% of students wanted to have the option to purchase a printed copy of the textbook from the bookstore for a small fee (10% of students wanted to print the textbook themselves). The predominant platform for reading e-textbooks is a laptop computer (only 89 of 351 students reported reading from their cell phones). Perhaps most importantly, 100% of the students in the study wanted to use OER textbooks in the future and would recommend the use of OER to friends. Sixteen faculty shared their perceptions about their use of OER. Seven faculty of sixteen felt that the OER textbook was superior to the traditional textbook for the course. Five faculty rated the OER as equivalent to the traditional textbook. Faculty were not as positive about the support materials (PowerPoints, Test banks) available with the OER textbooks. Half of the faculty felt that the support materials lacked quality. 25% of faculty felt that implementing the support materials took a significant amount of time. In their comments, the biggest comment made by faculty was about the need for support materials or the amount of time they spent in developing them for this adoption.

Delimont et al. (2016) surveyed 524 students in thirteen different courses at Kansas State University regarding their use of OER. They state, “Students indicated that they were somewhat satisfied taking courses using [OER] and used them somewhat more to more than a normal textbook. Students rated the [OER] as good quality and indicated that they were somewhat easy to use. Students agreed that they preferred using [OER] instead of buying textbooks for their courses.” Thirteen instructors were interviewed as well; all but one said they “preferred teaching their course with [OER] instead of a traditional textbook. Several (11/13) indicated that customization was a reason for this preference.” Similarly, all but one faculty member planned to keep using OER.

Illowsky, B. S., Hilton III, J., Whiting, J., & Ackerman, J. D. (2016) surveyed 325 students who used various versions of an open statistics textbook. In the first survey (n=126) students were surveyed about an earlier version of the textbook. “When students were asked to imagine a future course in which there were two sections, one offering traditional printed texts and the other offering texts such as the one they used in this course, 50% of students said they would choose the class with texts like those offered in this course. Only 19% said they would enroll in the course with the traditional printed text, and the remaining 32% said they would have no preference.” In this first survey, “When specifically asked how they would rate the quality of this text as compared to other textbooks they have used, 143 (62%) said that it was the same as books in their other courses, 57 (25%) rated it as better than other texts and 31 (13%) rated it as worse than other texts they have used.” 

The second survey (n=94) surveyed students who used the later OpenStax version. “In answer to the question, “How would you rate the quality of the texts used for this course?” 70% said it was about the same as the quality of the texts in their other courses, 23% said it had better quality and 7% said that it was worse.” There is a clear drop in the number of students who say that the OER is worse than traditional texts between the two studies. While the study does not point this out, it is noteworthy that the OpenStax textbooks has been adopted at a tremendously faster rate than previous iterations of the textbook.  

Summary

Over 5,000 students and faculty members have shared their perceptions across a dozen studies that have focused on perceptions of OER. In no instance did a majority of students or teachers report that the OER were of inferior quality. Across multiple studies in various settings, students consistently reported that they faced financial difficulties and that OER provided a financial benefit to them. A general finding seems to be that roughly half of teachers and students find OER to be comparable to traditional resources, a sizeable minority believe they are superior, and a smaller minority find them inferior.

In total, roughly 17,500 students have utilized OER materials across the studies that attempted to measure results pertaining to student efficacy. These students results were compared with approximately 100,000 students using traditonal textooks. While causality was not claimed by any researcher, the use of OER was sometimes correlated with higher test scores, lower failure, or withdrawal rates. In only one efficacy study did more students do worse than did better, and even in that study the majority of students achieved the same results as their peers using traditional textbooks.

Even if the use of OER materials do not significantly increase student learning outcomes, this is a very important finding. Given that (1) students and teachers generally find OER to be as good or better than traditional textbooks, and (2) students do not perform worse when utilizing OER, then (3) students, parents and taxpayers stand to save literally billions of dollars without any negative impact on learning through the adoption of OER.

As Hilton (2016) asks: “Because students and faculty members generally find that OER are comparable in quality to traditional learning resources, and that the use of OER does not appear to negatively influence student learning, one must question the value of traditional textbooks. If the average college student spends approximately $1,000 per year on textbooks and yet performs scholastically no better than the student who utilizes free OER, what exactly is being purchased with that $1,000?

Future Directions

Once adopted, OER provide the permissions necessary for faculty to engage in a wide range of pedagogical innovations. In each of the studies reported above, OER were used in manner very similar to the traditional textbooks they replaced. We look forward to reviewing empirical articles describing the learning impacts of open pedagogies.

If you are aware of a peer-reviewed efficacy or perceptions study that we have not mentioned, please let us know in the comments or by contacting us directly.

References

Allen, G., Guzman-Alvarez, A., Smith, A., Gamage, A., Molinaro, M., & Larsen, D. S. (2015). Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for traditional general chemistry textbooks. Chemistry Education Research and Practice, 16(4), 939-948. See also Allen, G., Guzman-Alvarez, A., Molinaro, M., Larsen, D. (2015). Assessing the Impact and Efficacy of the Open-Access ChemWiki Textbook Project. Educause Learning Initiative Brief, January 2015, and this newsletter.

Allen, I., Seaman, J. (2014). Opening the Curriculum: Open Educational Resources in U.S. Higher Education, 2014.

Bliss, T., Robinson, T. J., Hilton, J., & Wiley, D. (2013). An OER COUP: College teacher and student perceptions of Open Educational Resources. Journal of Interactive Media in Education, 1–25.

Bliss, T., Hilton, J., Wiley, D., Thanos, K. (2013). The cost and quality of open textbooks: Perceptions of community college faculty and students. First Monday, 18:1.

Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2012). Interactive Learning Online at Public Universities: Evidence from Randomized Trials. Ithaka S+R.

Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2014). Interactive Learning Online at Public Universities: Evidence from a Six‐Campus Randomized Trial. Journal of Policy Analysis and Management, 33(1), 94-111.

The California OER Council (2016). OER Adoption Study: Using Open Educational Resources in the College Classroom.

Delimont, N., Turtle, E. C., Bennett, A., Adhikari, K., & Lindshield, B. L. (2016). University students and faculty have positive perceptions of open/alternative resources and their utilization in a textbook replacement initiative. Research in Learning Technology, 24.

Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open, Distance and E-Learning.

Fischer, L., Hilton III, J., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159-172.

Gil, P., Candelas, F., Jara, C., Garcia, G., Torres, F (2013). Web-based OERs in Computer Networks. International Journal of Engineering Education, 29(6), 1537-1550. (OA preprint)

Hilton, J., Gaudet, D., Clark, P., Robinson, J., & Wiley, D. (2013). The adoption of open educational resources by one community college math department. The International Review of Research in Open and Distance Learning, 14(4), 37–50.

Hilton, J., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning: The Journal of Open and Distance Learning, 27(3), 201–217. (Open Repository Preprint).

Illowsky, B. S., Hilton III, J., Whiting, J., & Ackerman, J. D. (2016). Examining Student Perception of an Open Statistics Book. Open Praxis,8(3), 265-276.

Jhangiani, R. S., Pitt, R., Hendricks, C., Key, J., & Lalonde, C. (2016). Exploring faculty use of open educational resources at British Columbia post-secondary institutions. BCcampus Research Report. Victoria, BC: BCcampus.

Lindshield, B., & Adhikari, K. (2013). Online and campus college students like using an open educational resource instead of a traditional textbook. Journal of Online Learning & Teaching, 9(1), 1–7.

Lovett, M., Meyer, O., & Thille, C. (2008). The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education2008 (1).

Pawlyshyn, Braddlee, Casper and Miller (2013). Adopting OER: A Case Study of Cross-Institutional Collaboration and Innovation. Educause Review.

Petrides, L., Jimes, C., Middleton‐Detzner, C., Walling, J., & Weiss, S. (2011). Open textbook adoption and use: Implications for teachers and learners. Open learning26(1), 39-49.

Pitt, R. (2015). Mainstreaming Open Textbooks: Educator Perspectives on the Impact of OpenStax College open textbooks. The International Review of Research in Open And Distributed Learning, 16(4).

Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes a meta-review. Review of Educational Research, 86(1), 207-236.

Robinson, T. J. (2015). The effects of open educational resource adoption on measures of post-secondary student success. (Unpublished doctoral dissertation). Brigham Young University, Provo, Utah.

Robinson T. J., Fischer, L., Wiley, D. A., & Hilton, J. (2014). The impact of open textbooks on secondary science learning outcomes. Educational Researcher, 43(7): 341-351.

Wiley, D., Hilton, J. Ellington, S., and Hall, T. (2012). “A preliminary examination of the cost savings and learning impacts of using open textbooks in middle and high school science classes.” International Review of Research in Open and Distance Learning. 13 (3), pp. 261-276.

Wiley,D., Williams, L., DeMarte, D., and Hilton, J. (2016). “The Tidewater Z-Degree and the INTRO Model for Sustaining OER Adoption.” Education Policy Analysis Archives, 24(41), pp.1-12.