The Review Project
Led by John Hilton III, with additional contributions from Stacie Mason
This review provides a summary of all known empirical research on the impacts of OER adoption (including our own). The version below will be periodically updated with new articles as we become aware of them. If you know of an empirical research study on the impacts of OER adoption that is not included in this review, please leave a comment below. An in-depth article focusing on empirical research relating to perceptions and efficacy at the college level was published by the journal Educational Technology Research and Development in February of 2016. Please access the open-access version of this article or a summary video. You can also access overview slides that summarize this research.
Open Educational Resources are teaching and learning materials that provide users with (1) free and unfettered access and (2) 5R legal permissions to retain, reuse, revise, remix, and redistribute them, that can be used to replace traditional expensive learning resources (such as textbooks). A recent nationally representative survey of 2,144 faculty members in the United States found that “most faculty remain unaware of OER” (Babson Survey, 2014 ).
This same survey found that college professors rate “proven efficacy” and “trusted quality” as the two most important criteria for selecting teaching resources. Thus we believe that for OER to gain traction it is important to gather empirical research demonstrating its efficacy and quality.
To this end, we have gathered articles that focus on the efficacy of OER or teacher/student perceptions of such resources in actual practice. We have limited our studies to those in which OER were the primary learning resource(s) and were compared against traditional learning resources; in addition, the study needed to include at least 50 participants. We are also gathering smaller studies focused on efficacy or perceptions and posting them on this page.
We originally only included research published by a peer-reviewed journal. We are currently in the process of also including white papers and/or completed theses and dissertations based on the recommendations of Polanin, Tanner-Smith, and Hennessy (2016). Please let us know of any studies we are missing.
Studies that included data on both efficacy and perception
Research by Feldstein et al. (2012) took place at Virginia State University, where OER were implemented across nine different courses in the business department. 1,393 students took courses utilizing OER. Researchers found that students in courses that used OER more frequently had better grades and lower failure and withdrawal rates than their counterparts in courses that did not use OER. While their results had statistical significance, because of a new core curriculum employed at Virginia State University’s Business school, the two sets of courses were not identical. Thus while these data are highly interesting, we should not generalize them too far. 315 students completed a survey regarding their perspective on the shift to the OER, and almost 95% of responding students strongly agreed or agreed that the OER were “easy to use” and 78% of respondents felt that the OER “provided access to more up-to-date material that is available in my print textbooks.” Approximately two-thirds of students strongly agreed or agreed that the digital OER were more useful than traditional textbooks and that they preferred the OER digital content to traditional textbooks.
Hilton et al. (2013) chronicled a study that took place at Scottsdale Community College (SCC) in Arizona. In the fall of 2012, OER were employed throughout five different math courses at SCC, affecting 1,400 students. Issues with the initial placement tests made it so that only four of the courses could be compared; nevertheless, the results of Fall 2012 (when OER was used) compared to Fall 2011 and 2010 showed that student results on department exams were approximately the same before and after the OER implementation. Surveys completed by 910 students showed that 78% said they would recommend the OER to their classmates. Similarly, 83% of students agreed with the statement that “Overall, the materials adequately supported the work I did outside of class” (only 5% of students disagreed with this statement). Faculty members were likewise positive about the open materials. Of the 18 faculty members who reported on their view of the OER, 50% said that it was of the same quality as traditional textbooks, 33% said it was better, and 17% said it was worse.
Gil et al. (2013) (OA preprint, final published version), surveyed students over a five year time, asking them to compare blogs that featured OER resources versus other blogs (all blogs were directly related to student coursework). Approximately 500 students were surveyed; on average, 40% of students said that the blogs featuring OER were of equal quality to the blogs that did not feature OER, 45% of students said that the blogs with OER were superior and 15% said they were inferior. They also found a correlation between increasing access to OER and improving student grades; however, these results were tentative as they were not able to compare the results from individual students; rather this was a trend that was observed over time and may possibly have been due to other factors.
Ozdemir and Hendricks (2017) examine over 51 e-portfolios written by faculty in the state of California about their use of open textbooks. For the 55% of the 51 faculty who assessed the impact of adopting an open textbook on student learning outcomes, all reported that they remained the same or improved. None reported that student learning declined. The vast majority of faculty also reported that the quality of the textbooks was as good or better than that of traditional textbooks. 40 of the 51 portfolios contained data about students’ attitudes towards the open textbooks used in their classes; the overwhelming majority of students reported positive experiences with the open textbooks, only 15% of the e-portfolios reported any negative comments by students.
Hendricks, Reinsberg and Rieger (2017) examine the use of OER in a physics course at the University of British Columbia. 143 students completed surveys about the OER; students reported using the open textbook with the same frequency as their traditional textbooks. Moreover, 93% of respondents said it was the same or better than textbooks in other courses. The researchers also found that in one year of OER adoption students saved approximately $85,000 (Canadian dollars) and that student final exam scores and grade distributions remained the same after OER adoption as they had been previously.
Gurung (2017) created a Qualtrics survey and sent it to the course instructors at seven schools who in turn forwarded it to their students. In the first study, 569 students from five institutions who used an electronic version of the NOBA Discovering Psychology open textbook in their introductory psychology class responded to the survey. At the other two institutions, 530 students who used hard copies of one of two different commercial textbooks responded to the survey. Students reported on their evaluations of the textbooks, the behaviors demonstrated by their teachers, their study habits, ACT scores and then answered fifteen psychology questions drawn from the 2007 AP Psychology exam. When controlling for ACT scores, students who used OER scored 1.2 points lower than those who used traditional textbooks, a difference of 13%. In a second study, published in the same article, Gurung included an additional ten quiz questions from the NOBA test bank (in addition to the fifteen AP test questions used in the first study). The results were largely the same; when comparing total quiz scores, there was an overall significant effect of the book used, favoring those who used the commercial textbook. These results are nuanced by the fact that when only the NOBA test bank items were used there were no significant differences between the two groups. This indicates that alignment may be an important part of the reason for the discrepancy between groups.
With respect to perceptions, in his first study, he also found that while OER users rated the total quality of their textbook to be lower than those using a commercial textbook. Further analyses showed this was occurred because of differences in ratings on figures, photos and visual appeal. There was no significant difference in terms of how students rated the helpfulness of the textbook nor in terms of how enjoyable and easy to read students perceived the books as being. However, students using OER found the material more applicable to their lives. The results in the second study were slightly different. Overall, commercial textbook users rated their books higher than OER users, and rated them as being more helpful.
Lawrence and Lester (2018) focus on the use on an open textbook in an introductory American Government course. They surveyed students in both 2014 and 2015 and asked them questions about the textbook they were using (traditional commercial textbook in 2014, and an open textbook in 2015). Contrary to many OER research studies, they found that students were more positive about the commercial textbook that the open one. 74% of the 162 students who used the traditional textbook said that they were “overall satisfied with the book” versus 57% of the 117 people who used the open, a difference of 17% (279 total survey respondents). They attribute this to problems related to the specifically open text used and believe that their study would be different had a more robust textbook (like this one) been available. Students who used the open textbook did perform better for one of the two teachers studied; however authors attribute this change to policy changes regarding online classes (the teacher that saw significant changes teaches online). They conclude that their “findings do not support the notion that OERs represent a dramatic improvement over commercial texts, nor do they indicate that students perform substantially worse when using open content texts either. ”
Clinton found that students saved significant amounts of money with the OpenStax textbook. An extremely commendable aspect of Clinton’s research was that she accounted for prior academic achievement by including students’ high school GPAs when considering class performance. She found that while students who used the OpenStax textbook outperformed those in who used the traditional textbook, the difference was small and may be related to small differences in the student population. Ultimately she concludes, “There was likely no real effect of open-source textbook adoption on course grades.” However, “The number of students who withdrew from the course was substantially higher in the semester with a commercial textbook compared to the semester with an open-source textbook.”
Jhangiani, Dastur, Le Grand, and Penner (2018) examined seven sections of an introductory psychology class taught in Canada across two separate semesters. Two sections were assigned a digital open textbook, two were assigned the same open textbook, but in hardcopy format, and three were assigned a commercial textbook. Three different instructors taught the seven courses; notably one instructor taught back-to-back semesters, first with the open print textbook, and then with the print commercial textbook. The other two instructors taught with either open or commercial, but not both.Students in all conditions had similar demographic variables and previous psychology knowledge. Throughout the semester, students took three exams, each of which consisted of approximately sixty multiple-choice questions. When all sections were analyzed in a MANOVA, students assigned the digital open textbook performed significantly better than those who used the commercial textbook on one of the three exams. When only the two sections taught by the same teacher were analyzed (to control for teacher bias), a second MANOVA was performed just on those two sections. Similarly, this indicated that in one of the three exams students using the open textbook (this time in both print and digital) outperformed students using the commercial textbook.
They also found in their survey of 178 students that the OER textbooks were rated more favorably than commercial textbooks.
Studies that focused on efficacy
Lovett et al. (2008) measured the result of an implementation of an online, OER component of Carnegie Mellon University’s Open Learning Initiative (OLI). Over two semesters, forty-four students utilized the OER as part of this study. Researchers examined test scores (three midterm and one final exam) of those students who took the traditional course versus those who utilized the OER materials. They found no significant difference between the two groups.
Bowen et al. (2012) completed a rigorous study comparing the use of a traditional statistics textbook with Carnegie Mellon’s OLI at six different institutions. Participating students were randomly assigned to either the face-to-face class with a traditional textbook, or a “hybrid” class that used the OER resource. Both groups took the same standardized test at the beginning and end of the semester, as well as a final examination. 605 students took the OER version of the course, while 2,439 took the traditional version. Students who utilized OER performed slightly better on the standardized exam than those who did not. The difference in outcomes was not statistically significant. See also Bowen et al. (2014)
Hilton and Laman (2012) (Taylor & Francis Version, Open Repository Preprint), focused on introductory Psychology courses taught at Houston Community College (HCC). In the fall of 2011, twenty-three sections composed of 690 students used an open psychology textbook. The introduction of an open textbook was correlated with an increase in class grade point average, an increase of the average score on the departmental final examination, and a lower course withdrawal rate. No causation was claimed.
Wiley et al. (2012) examined the standardized test scores of students using open textbooks in secondary science classes in three different school districts. Approximately 1,200 students used open textbooks during this study. Researchers examined their end-of-year standardized test results and found no apparent differences between the results of students who used traditional and open textbooks.
Pawlyshyn et al. (2013) reported on the adoption of OER at Mercy College in New York. In the fall of 2012, 695 students utilized OER in Mercy’s basic math course, their pass rates were compared with those of the fall of 2011, in which no OER were utilized. Researchers found that the pass rates increased from 63.6% in fall 2011 to 68.9% in fall 2012 when all courses were taught with OER. Similarly, students who were enrolled in OER versions of a reading course performed better than their peers who enrolled in the same course using non-OER materials.
Robinson et al. (2014) examined the use of open science textbooks in three secondary science subjects across several schools in a suburban school district. This rigorous study used propensity score matched groups in order to control for teacher effect, socioeconomic status, and eight other potentially confounding variables. There were 1,274 students in each condition (treatment and control). In examining the results of the end-of-year state standardized test there were very small, but statistically significant difference between the two groups, favoring those who utilized OER.
Fischer et al (2015) performed follow-up research on the second year of implementation at the schools studied by Robinson (2014). Their original sample consisted of 16,727 students (11,818 control and 4,909 treatment). From this sample, there were fifteen courses for which some students enrolled in both treatment (n=1,087) and control (n=9,264) sections (the remaining students enrolled in a course which had either all treatment or all control sections and were therefore excluded). While this represents a large sample size, students in treatment conditions were only compared with students in control conditions who were taking the same class in which they were enrolled. For example, students enrolled in a section of Biology 111 that used OER were only compared with students in Biology 111 sections that used commercial textbooks (not students enrolled in a different course). Thus when diffused across fifteen classes, there was an insufficient number of treatment students to do propensity score matching for the grade and completion analyses. The researchers found that in two of the fifteen classes, students in the treatment group were significantly more likely to complete the course (there were no differences in the remaining thirteen). In five of the treatment classes, students were significantly more likely to receive a C- or better. In nine of the classes there were no significant differences and in one study control students were more likely to receive a C- or better. Similarly, in terms of the overall course grade, students in four of the treatment classes received higher grades, ten of the classes had no significant differences, and students in one control class received higher grades than the corresponding treatment class. Researchers utilized propensity score matching before examining the number of credits students took in each of the semesters as this matching could be done across the different courses. Drawing on their original sample of 16,727 students, the researchers matched 4,147 treatment subjects with 4,147 controls. There was a statistically significant difference in enrollment intensity between the groups. Students in fall 2013 who enrolled in courses that utilized OER took on average two credit hours more than those in the control group, even after controlling for demographic covariates. ANCOVA was then used to control for differences in fall enrollment and to estimate differences in winter enrollment. Again, there was a significant difference between the groups, with treatment subjects enrolling in approximately 1.5 credits more than controls.
Allen et. al (2015) designed an experiment in which an experimental class of 478 students used the OER ChemWiki as its primary textbook, while the control class of 448 utilized a commercial textbook. The two sections were taught the same semester at back-to-back times using the same faculty member and teaching assistants. Students in both classes were given the same midterm and final exams. Researchers found no significant differences between the two groups both with overall exam results and item-specific questions. Beginning of the semester pre-tests, combined with final exams showed no significant differences in individual learning gains between the two groups. Student surveys regarding time spent on the class found that students in both groups spent approximately the same amount of time preparing for class.
Robinson (2015) devised a quasi-experimental study involving seven courses at seven colleges involved in Project Kaleidoscope. Outcomes of students using open textbooks (n=3,254) were compared with outcomes of students using traditional textbooks (n=10,819) in comparable courses. Controlling for covariates, Robinson found that students using open textbooks in the business and psychology courses earned lower average grades and were less likely than students using traditional textbooks to pass the course with a C- or better. Students in other five courses showed no significant difference in average grades or successful pass rates. In the biology courses, students using OER were more likely to complete the course than students using traditional textbooks. Across all courses, students using open textbooks took slightly more credits than students using traditional textbooks. This study is important in that it contains “the first finding of a negative effect associated with OER adoption” (p.59).
Wiley et al. (2016) focuses on open textbook adoption at Tidewater Community College. Across two pilot semesters (fall 2013 and spring 2014) 23,232 students enrolled in traditonal versions of selected courses. An additonal 753 students enrolled in “Z” versions of these same courses (“Z” standing for “Zero textbook costs”). 3.57% of students dropped the course in the traditional courses versus 2.79% in the Z courses – a statistically significant difference. The authors hypothesize that money kept by universities from lower drop rates could support open textbook initatives.
Hilton et al. (2016) focus on an additional two semesters at Tidewater (their results include fall 2013, spring 2014, fall 2014, and spring 2015). They studied the combined the drop, withdrawal, and C or better grade analyses to estimate the throughput rates of students taking traditional verses Z courses. In the face-to-face courses (Control n = 36,223 Treatment n = 1,151) 59.8% of students in non-Z courses made it through the successive hurdles of drop, withdrawal and passing the class, compared with 66.4% of students in the Z courses, for a difference of 6.6%. Combined the drop, withdrawal, and C or better grade analyses to estimate the differences between the groups in the overall success rate from students’ registration to final grade. In the hybrid/online courses: (Control n = 7,000, Treatment n = 863) 54.2% of students who started in non-z courses successfully made it through the course with a C or better, compared with 59.8% of students in the Z courses, for a difference of 5.6%.
Croteau (2017) synthesizes reports written by faculty members as part of the Affordable Learning Georgia OER adoption program. Across twenty-four separate data sets involving 3,847 students using OER, Croteau found no significant differences in student pass-rate before and after implementing OER. Additional analyses of relevant data sets likewise showed no significant differences in completion rates or final exam scores.
Winitzky-Stephens and Pickavance (2017) examine a large-scale OER adoption across 37 different courses across several different general education subjects. Over a four-year period of time 26,538 students enrolled in non-OER versions of these courses while 7,588 students enrolled in the OER versions. The multilevel models used by the authors find no significant difference between courses using OER and traditional textbooks for continuing students, and a small benefit for new students.
Chiorescu (2017) explores OER adoption in a college algebra class and found that by adopting an open textbook, each enrolled student could save $86.00. Across four semesters, 447 students used commercial resources and 159 students used OER. Depending on which semesters were compared, students were either as likely or more likely to pass the class when OER was used; moreover, significantly fewer students withdrew when OER were implemented.
Wiley, Tonks, Weston, and Webb (2017) study the use of student-created OER as a means to improving student performance. Students created tutorial videos, chapter summaries, and review games for a particular topic, some of which were placed into later versions of the course. The average grade on student assignments across several semesters (181 students) rose significantly as more student-created OER were added to the course.
Kelly and Rutherford (2017) compare the academic results of 114 seventh graders studying mathematics. All students received regular instruction, and for supplemental instruction (30 minutes a day, over four weeks) were placed into groups that (1) used Khan Academy videos, (2) received other math instruction, or (3) studied English Language Arts. At the conclusion of the study, all students were given a mathematics test; there were no significant differences between the three groups.
Grewe and Davis (2017) finds a positive relationship between the use of OER and student academic achievement in an online history course. She studies 146 students who enrolled in the fall of 2013 and spring of 2014; these courses had an identical instructional design (but different content and different instructors). In examining the final course grades of 146 students, Grewe demonstrates that OER adoption can be associated with higher student performance.
Colvard, Watson, and Park (2018) examined eight undergraduate courses that switched from commercial to OpenStax textbooks between fall 2010-fall 2016 at the University of Georgia. When comparing the final grade distributions for each group of students, the researchers found statistically significant differences favoring OER. There was a 5.5% increase in A grades after OER adoption, a 7.7% increase in A- grades, and those receiving a D, F or W grade decreased by 2.7% after OER were introduced. The differences were even larger for certain sub-populations. In overall GPA, there was a 6.90% increase for non-Pell recipients and an 11.0% increase for Pell recipients. Furthermore, OER adoption resulted in a 2.1% reduction in DFW grades for non-Pell eligible students versus a 4.4% reduction for Pell-eligible students. Similarly, White students had a 7.1% increase in average grade, while non-White students had a 13.1% increase. Collectively the DFW rates for White student dropped by 1.5 percentage points, but non-white students DFW rates dropped by 5.2 percentage points. There was an even larger gap with respect to full and part time students. Course grades improved by 3.2% for full-time students but jumped 28.1% for part-time students. The DFW rate for full-time students actually increased from 6.3% to 7.4%; however the rate from part-time students dropped from 34.3% to 24.2%.
Studies that focused on perceptions
Petrides et al. (2011), administered surveys to instructors and students who utilized an open statistics textbook called Collaborative Statistics. In total, 31 instructors and 45 students participated in oral interviews or focus groups that explored their perceptions of the OER which they had utilized. The authors found that “Cost reduction for students was the most significant factor influencing faculty adoption of open textbooks” (p. 43), partly because it increased student access. 65% of students surveyed reported a preference for using open textbooks in the future because they are generally easier to use.
Bliss et al. (2013a) reported on surveys completed by 11 instructors and 132 students at seven different colleges. 60 percent of instructors stated their students were equally prepared when OER replaced traditional texts and 30 percent said their students were more prepared. One of the eleven instructors felt students were less prepared. All 11 instructors surveyed stated they would be very likely to use open texts in future courses. The students were also very positive regarding the OER materials. When asked to compare the OER texts to traditional texts, only 3% felt that PK texts were worse than their typical textbooks, 56% said they were the same quality, and 41% said they were better than typical textbooks.
In an extension of the previous study, Bliss et al. (2013b), studied OER adoption at eight different institutions of higher education. They surveyed an additional fifty-eight teachers and 490 students across the eight colleges regarding their experiences in utilizing OER. Approximately 50% of students said that the OER materials had the same quality as traditional textbooks and nearly 40% said that they were better. Students focused on several benefits of the open textbooks. Many cited technical advantages of the digital texts. In addition, the free cost of their open texts was important to many students. 55% of teachers reported that the open materials were of the same quality as the materials that had previously been used, and 35% felt that they were better.
Lindshield and Adhikari (2013) studied the perceptions of students who utilized a digital OER textbook in a Human Nutrition class. One hundred and ninety-eight students completed a survey in which they shared their perceptions of the OER text. “Students favorably rated their level of satisfaction, liking the idea of the [digital OER], ease of [digital OER] use, not having to buy a textbook, and preferring the [digital OER] versus buying a textbook for the course.” Moreover they found that students disagreed or somewhat disagreed with statements to the effect that they would like to have a traditional textbook in addition to OER.
Allen and Seaman in their Babson Survey (2014) surveyed 2,144 college professors regarding their opinions on OER. Of the 34% (729) who expressed awareness of OER, 61.5% of respondents said that OER materials had about the same “trusted quality” as traditional resources, 26.3% said that traditional resources were superior, 12.1% said that OER were superior. 68.2% said that the “proven efficacy” were about the same 16.5% said that OER had superior efficacy and 15.3% said that traditional resources had superior efficacy. It is important to note that the professors surveyed in this study expressed awareness of OER, but had not necessarily utilized OER. In contrast, in the above-mentioned four studies the professors had actually utilized OER, perhaps having a greater basis on which to judge the quality of OER resources.
Pitt (2015) reported results of two surveys of 126 educators, conducted 2013 and 2014-2015, as part of a collaboration between OER Research Hub (OERRH) and OpenStax College (OSC), a provider of open textbooks. Around 65% of respondents reported that using OSC helped them meet diverse learners’ needs, while a minority of respondents said that the OSC materials made teaching easier, enabled innovation or changed their pedagogical approach. More than 65% of respondents perceived greater learner satisfaction for their students using OER. Nearly all respondents said that having using OSC materials increased the likelihood that they would recommend OSC materials to peers.
Jhangiani, Pitt, Hendricks, Key, and Lalonde (2016) examined awareness, usage, outcomes, and perceptions of OER among British Columbia post-secondary faculty. As part of a collaborative project between the BCcampus-led Open Textbook Project and OER Hub, researchers surveyed post-secondary educators through an online survey disseminated via email and social media. Of the 78 respondents, 77% had used OER. Most respondents rated OER quality as comparable or superior to that of traditional materials; educators who had adopted OER rated the quality significantly higher than educators who had not. Respondents reported that the top two barriers to using OER were finding relevant and high quality OER. Faculty at research-intensive universities reported significantly lower barriers to finding high-quality OER than did faculty at teaching-intensive universities or colleges/institutes, and were the most likely group to create or adapt OER.
The California OER Council (2016) released a white paper focused on OER adoption in CA higher education. 351 students completed a survey about their use of OER. When students were asked if the OER textbook chapter(s) were better than the traditional, 42% said the OER textbook as better, 39% said they were about the same, 11% rated the textbook as worse than the traditional textbook and 8% declined to answer. Of the 351 students in the survey, 71 printed the textbook and 209 used a PDF. 16% of students wanted to have the option to purchase a printed copy of the textbook from the bookstore for a small fee (10% of students wanted to print the textbook themselves). The predominant platform for reading e-textbooks is a laptop computer (only 89 of 351 students reported reading from their cell phones). Perhaps most importantly, 100% of the students in the study wanted to use OER textbooks in the future and would recommend the use of OER to friends. Sixteen faculty shared their perceptions about their use of OER. Seven faculty of sixteen felt that the OER textbook was superior to the traditional textbook for the course. Five faculty rated the OER as equivalent to the traditional textbook. Faculty were not as positive about the support materials (PowerPoints, Test banks) available with the OER textbooks. Half of the faculty felt that the support materials lacked quality. 25% of faculty felt that implementing the support materials took a significant amount of time. In their comments, the biggest comment made by faculty was about the need for support materials or the amount of time they spent in developing them for this adoption.
Delimont et al. (2016) surveyed 524 students in thirteen different courses at Kansas State University regarding their use of OER. They state, “Students indicated that they were somewhat satisfied taking courses using [OER] and used them somewhat more to more than a normal textbook. Students rated the [OER] as good quality and indicated that they were somewhat easy to use. Students agreed that they preferred using [OER] instead of buying textbooks for their courses.” Thirteen instructors were interviewed as well; all but one said they “preferred teaching their course with [OER] instead of a traditional textbook. Several (11/13) indicated that customization was a reason for this preference.” Similarly, all but one faculty member planned to keep using OER.
Illowsky, B. S., Hilton III, J., Whiting, J., & Ackerman, J. D. (2016) surveyed 325 students who used various versions of an open statistics textbook. In the first survey (n=126) students were surveyed about an earlier version of the textbook. “When students were asked to imagine a future course in which there were two sections, one offering traditional printed texts and the other offering texts such as the one they used in this course, 50% of students said they would choose the class with texts like those offered in this course. Only 19% said they would enroll in the course with the traditional printed text, and the remaining 32% said they would have no preference.” In this first survey, “When specifically asked how they would rate the quality of this text as compared to other textbooks they have used, 143 (62%) said that it was the same as books in their other courses, 57 (25%) rated it as better than other texts and 31 (13%) rated it as worse than other texts they have used.” The second survey (n=94) surveyed students who used the later OpenStax version. “In answer to the question, “How would you rate the quality of the texts used for this course?” 70% said it was about the same as the quality of the texts in their other courses, 23% said it had better quality and 7% said that it was worse.” There is a clear drop in the number of students who say that the OER is worse than traditional texts between the two studies. While the study does not point this out, it is noteworthy that the OpenStax textbooks has been adopted at a tremendously faster rate than previous iterations of the textbook.
Jung, Bauer and Heaps (2017) report on the perceptions of faculty members who used OpenStax textbooks. This study adds significant value by focusing not simply on “OER in general,” but on a specific set of OER. They survey 137 faculty and find that only 16% report taking more time preparing to teach a course using OpenStax textbooks, and that of these, 78% felt that the benefits of OpenStax were worth the additional time. In terms of quality; 81% thought that OpenStax textbooks have same or higher quality as commercial textbooks. Thematic analysis revealed faculty perceptions of what constitutes high-quality open textbooks.
Fischer, Ernst, and Mason (2017) examine the rich corpus of the reviews hosted by the Open Textbook Network (OTN). The OTN’s Open Textbook Library has been an extremely beneficial way to identify open textbooks and includes a very useful feature of allowing faculty members at participating universities to review open textbooks. This study examines 416 reviews of 121 textbooks and finds that reviewers generally gave open textbooks high ratings (a median of 4.5/5 overall rating).
Cooney (2017) studies three sections of a Health Psychology course that replaced a traditional textbook with OER. Six students were interviewed and 67 surveyed regarding their perceptions of the OER; findings indicate that students were generally positive about the OER. Just over 80% of 67 students surveyed rated the OER as being better than a traditional textbook, with an additional 16% saying it was of the same quality.
Jhangiani and Jhangiani (2017) survey 320 post-secondary students in British Columbia who were enrolled in courses that used an open textbook. They find that a majority of students report not purchasing at least one of their required textbooks and that a substantial portion of students report taking fewer courses and/or dropping courses based on textbook costs. Students favorably rated open textbooks, with 96% of survey participants stating that they were at or above average.
Watson, Domizi, and Clouser (2017) surveyed 1,299 students at the University of Georgia who used the OpenStax biology textbook. These students were directly asked to “rate the quality of the OpenStax textbook as compared to other textbooks they had used. Sixty-four percent said the quality was about the same, and 22% said it was higher or much higher than other textbooks. With regards to the readability of the OpenStax textbook, 64% of the students said it was “good” or “very good,” 31% said it was “fair,” and 5% said it was “poor” or “very poor.””
Ikahihifo, Spring, Rosencrans, and Watson (2017) received responses from 206 college students who had used OER materials and compared it to typical textbooks. “A majority (54.9%, 113 students) of the participants rated the open material as excellent (Figure 1). Roughly 39% (81 students) considered the quality as good as a traditional text or slightly favored the quality of the OER material. Less than 6% (12 students) considered the quality of the OER material to be less than that of a traditional textbook.” They also asked students what they did with the money saved by not spending textbooks. They report, “Of the 206 responses, 87 students (42.2%) said they reinvested in their education, 63 students (30.5%) indicated they applied it towards daily expenses, and 42 students (20.3%) said they saved the money. Additional codes such as “Spent It Anyway,” “Do Not Purchase Own Textbooks (Receive Money or Books from Other Sources),” and “Leisure,” were used, but these categories were relatively small (6.3%, 2.9%, and 0.97%, respectively).
Morales and Baker (2018) chronicle the efforts of a large suburban public school district that adopted and piloted open secondary science textbooks. They surveyed students with respect to their learning before and during the use of the open textbook. They found that “there were qualitative and quantitative indications that students’ perceptions of an open textbook in place of a standard textbook are improving students’ attitudes and behaviors toward learning.”
Abramovich and McBride (2018) surveyed 35 college instructors and 662 students at seven colleges. They were involved in several courses, including chemistry, first-year college experience, psychology, history, writing, business and teacher education course. In total, 86% of students rated OER as either as useful or more useful than materials used in their other courses. Only 6% of students stated that the open textbooks rarely or never helped them meet their course objectives. Faculty members were also very favorable with respect to OER. Nearly every instructor rated the OER as being either equal (40%) a little more useful (23%) or much more useful (34%) than materials they had previously used. This left only 3% of instructors who felt that the OER were less useful than other materials.
Several thousand students and faculty members have shared their perceptions across more than a dozen studies that have focused on perceptions of OER. In no instance did a majority of students or teachers report that the OER were of inferior quality. Across multiple studies in various settings, students consistently reported that they faced financial difficulties and that OER provided a financial benefit to them. A general finding seems to be that roughly half of teachers and students find OER to be comparable to traditional resources, a sizeable minority believe they are superior, and a smaller minority find them inferior.
In total, more than 25,000 students have utilized OER materials across the studies that attempted to measure results pertaining to student efficacy. These students results were compared with approximately 100,000 students using traditional textbooks. While causality was not claimed by any researcher, the use of OER was sometimes correlated with higher test scores, lower failure, or withdrawal rates. In only one efficacy study did more students do worse than did better, and even in that study the majority of students achieved the same results as their peers using traditional textbooks.
Even if the use of OER materials do not significantly increase student learning outcomes, this is a very important finding. Given that (1) students and teachers generally find OER to be as good or better than traditional textbooks, and (2) students do not perform worse when utilizing OER, then (3) students, parents and taxpayers stand to save literally billions of dollars without any negative impact on learning through the adoption of OER.
As Hilton (2016) asks: “Because students and faculty members generally find that OER are comparable in quality to traditional learning resources, and that the use of OER does not appear to negatively influence student learning, one must question the value of traditional textbooks. If the average college student spends approximately $1,000 per year on textbooks and yet performs scholastically no better than the student who utilizes free OER, what exactly is being purchased with that $1,000?”
Once adopted, OER provide the permissions necessary for faculty to engage in a wide range of pedagogical innovations. In each of the studies reported above, OER were used in manner very similar to the traditional textbooks they replaced. We look forward to reviewing empirical articles describing the learning impacts of open pedagogies.
If you are aware of a peer-reviewed efficacy or perceptions study that we have not mentioned, please let us know in the comments or by contacting us directly.
Allen, G., Guzman-Alvarez, A., Smith, A., Gamage, A., Molinaro, M., & Larsen, D. S. (2015). Evaluating the effectiveness of the open-access ChemWiki resource as a replacement for traditional general chemistry textbooks. Chemistry Education Research and Practice, 16(4), 939-948. See also Allen, G., Guzman-Alvarez, A., Molinaro, M., Larsen, D. (2015). Assessing the Impact and Efficacy of the Open-Access ChemWiki Textbook Project. Educause Learning Initiative Brief, January 2015, and this newsletter.
Allen, I., Seaman, J. (2014). Opening the Curriculum: Open Educational Resources in U.S. Higher Education, 2014.
Bliss, T., Robinson, T. J., Hilton, J., & Wiley, D. (2013). An OER COUP: College teacher and student perceptions of Open Educational Resources. Journal of Interactive Media in Education, 1–25.
Bliss, T., Hilton, J., Wiley, D., Thanos, K. (2013). The cost and quality of open textbooks: Perceptions of community college faculty and students. First Monday, 18:1.
Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2012). Interactive Learning Online at Public Universities: Evidence from Randomized Trials. Ithaka S+R.
Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2014). Interactive Learning Online at Public Universities: Evidence from a Six‐Campus Randomized Trial. Journal of Policy Analysis and Management, 33(1), 94-111.
The California OER Council (2016). OER Adoption Study: Using Open Educational Resources in the College Classroom.
Chiorescu, M. (2017). Exploring Open Educational Resources for College Algebra. The International Review of Research in Open and Distributed Learning, 18(4).
Cooney, C. (2017). What Impacts do OER Have on Students? Students Share Their Experiences with a Health Psychology OER at New York City College of Technology. The International Review of Research in Open and Distributed Learning, 18(4).
Croteau, E. (2017). Measures of student success with textbook transformations: the Affordable Learning Georgia Initiative. Open Praxis, 9(1), 93-108.
Delimont, N., Turtle, E. C., Bennett, A., Adhikari, K., & Lindshield, B. L. (2016). University students and faculty have positive perceptions of open/alternative resources and their utilization in a textbook replacement initiative. Research in Learning Technology, 24.
Feldstein, A., Martin, M., Hudson, A., Warren, K., Hilton, J., & Wiley, D. (2012). Open textbooks and increased student access and outcomes. European Journal of Open, Distance and E-Learning.
Fischer, L., Ernst, D., & Mason, S. L. (2017). Rating the Quality of Open Textbooks: How Reviewer and Text Characteristics Predict Ratings. The International Review of Research in Open and Distributed Learning, 18(4).
Fischer, L., Hilton III, J., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159-172.
Gil, P., Candelas, F., Jara, C., Garcia, G., Torres, F (2013). Web-based OERs in Computer Networks. International Journal of Engineering Education, 29(6), 1537-1550. (OA preprint)
Grewe, K., & Davis, W. P. (2017). The Impact of Enrollment in an OER Course on Student Learning Outcomes. The International Review of Research in Open and Distributed Learning, 18(4).
Hendricks, C., Reinsberg, S. A., & Rieger, G. W. (2017). The Adoption of an Open Textbook in a Large Physics Course: An Analysis of Cost, Outcomes, Use, and Perceptions. The International Review of Research in Open and Distributed Learning, 18(4).
Hilton, J. Fischer, L., Wiley, D., Williams, L. (2016). “Maintaining Momentum Toward Graduation: OER and the Course Throughput Rate,” International Review of Research on Distance and Open Learning, 17(6): 1-10 (2016).
Hilton, J., Gaudet, D., Clark, P., Robinson, J., & Wiley, D. (2013). The adoption of open educational resources by one community college math department. The International Review of Research in Open and Distance Learning, 14(4), 37–50.
Ikahihifo, T. K., Spring, K. J., Rosecrans, J., & Watson, J. (2017). Assessing the Savings from Open Educational Resources on Student Academic Goals. The International Review of Research in Open and Distributed Learning, 18(7).
Illowsky, B. S., Hilton III, J., Whiting, J., & Ackerman, J. D. (2016). Examining Student Perception of an Open Statistics Book. Open Praxis,8(3), 265-276.
Jhangiani, R. S., & Jhangiani, S. (2017). Investigating the perceptions, use, and impact of open textbooks: A survey of post-secondary students in British Columbia. The International Review of Research in Open and Distributed Learning, 18(4).
Jhangiani, R. S., Pitt, R., Hendricks, C., Key, J., & Lalonde, C. (2016). Exploring faculty use of open educational resources at British Columbia post-secondary institutions. BCcampus Research Report. Victoria, BC: BCcampus.
Jung, E., Bauer, C., & Heaps, A. (2017). Higher Education Faculty Perceptions of Open Textbook Adoption. The International Review of Research in Open and Distributed Learning, 18(4).
Kelly, D. P., & Rutherford, T. (2017). Khan Academy as Supplemental Instruction: A Controlled Study of a Computer-Based Mathematics Intervention. The International Review of Research in Open and Distributed Learning, 18(4).
Lawrence and Lester (2018). Evaluating the Effectiveness of Adopting Open Educational Resources in an Introductory American Government Course. Journal of Political Science Education.
Lindshield, B., & Adhikari, K. (2013). Online and campus college students like using an open educational resource instead of a traditional textbook. Journal of Online Learning & Teaching, 9(1), 1–7.
Lovett, M., Meyer, O., & Thille, C. (2008). The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education, 2008 (1).
Morales, R., & Baker, A. (2018). Secondary Students’ Perceptions of Open Science Textbooks. Journal of Interactive Media in Education, 2018(1).
Ozdemir, O., & Hendricks, C. (2017). Instructor and student experiences with open textbooks, from the California open online library for education (Cool4Ed). Journal of Computing in Higher Education, 29(1), 98-113.
Pawlyshyn, Braddlee, Casper and Miller (2013). Adopting OER: A Case Study of Cross-Institutional Collaboration and Innovation. Educause Review.
Petrides, L., Jimes, C., Middleton‐Detzner, C., Walling, J., & Weiss, S. (2011). Open textbook adoption and use: Implications for teachers and learners. Open learning, 26(1), 39-49.
Pitt, R. (2015). Mainstreaming Open Textbooks: Educator Perspectives on the Impact of OpenStax College open textbooks. The International Review of Research in Open And Distributed Learning, 16(4).
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes a meta-review. Review of Educational Research, 86(1), 207-236.
Robinson, T. J. (2015). The effects of open educational resource adoption on measures of post-secondary student success. (Unpublished doctoral dissertation). Brigham Young University, Provo, Utah.
Robinson T. J., Fischer, L., Wiley, D. A., & Hilton, J. (2014). The impact of open textbooks on secondary science learning outcomes. Educational Researcher, 43(7): 341-351.
Watson, C. E., Domizi, D. P., & Clouser, S. A. (2017). Student and Faculty Perceptions of OpenStax in High Enrollment Courses. The International Review of Research in Open and Distributed Learning, 18(5).
Wiley, D., Hilton, J. Ellington, S., and Hall, T. (2012). “A preliminary examination of the cost savings and learning impacts of using open textbooks in middle and high school science classes.” International Review of Research in Open and Distance Learning. 13 (3), pp. 261-276.
Wiley, D., Webb, A., Weston, S., & Tonks, D. (2017). A Preliminary Exploration of the Relationships Between Student-Created OER, Sustainability, and Students Success. The International Review of Research in Open and Distributed Learning, 18(4).
Wiley,D., Williams, L., DeMarte, D., and Hilton, J. (2016). “The Tidewater Z-Degree and the INTRO Model for Sustaining OER Adoption.” Education Policy Analysis Archives, 24(41), pp.1-12.
Winitzky-Stephens, J. R., & Pickavance, J. (2017). Open Educational Resources and Student Course Outcomes: A Multilevel Analysis. The International Review of Research in Open and Distributed Learning, 18(4).