The Review Project Launch

At #OpenEd14 John Hilton presented a summary of all the empirical research on the impact of OER adoption that we could find. The presentation was extremely well received, and John has turned it into a nice journal article which is currently under review at a journal which shall remain nameless (for the time being).

By the time the article appears, however, it will almost certainly be out of date. As a service to the field, we’ve published an abstracted version of John’s article on the OEG website under the label The Review Project. We’ll keep this page up-to-date as we come across additional articles that actually take a solid empirical look at the impacts of OER adoption. Do you know of one we’ve missed? Head over to The Review Project and leave a reference in the comments.

Ed Researcher Article is Live!

We’re extremely excited about our newest publication:

Robinson T. J., Fischer, L., Wiley, D. A., & Hilton, J. (2014). The Impact of Open Textbooks on Secondary Science Learning Outcomes. Educational Researcher. DOI: 10.3102/0013189X14550275

Putting something in Ed Researcher has been a “bucket list” item for many of us. It is, arguably, the highest ranking education journal which accepts this kind of research. And yes Jared, the first author on this paper, is a student member of the OEG!

We’re just completing another paper like this one, but with post-secondary data. It’s going to be even more awesome than this article was! Which journal should we target next? PLOS One? Nature?

Gates Grant is a Go

The Bill and Melinda Gates Foundation has approved a grant to the Open Education Group to apply our COUP framework to post-secondary adopters of open textbooks and open educational resources. This study will examine each of the following questions:

1. What is the relationship between OER-adoption by teachers of post-secondary courses and their students’:

a. rates of course completion?
b. rates of course success (i.e., completing a course with a C or better grade)?
c. enrollment intensity (i.e., the number of credit hours they take during the semester they are taking the OER course)?
d. rates of persistence (i.e., registering for classes again in the semester following the OER course)?

2. What is the relationship between the total cost of required instructional materials on course syllabi (whether the course uses OER or not) and the outcomes listed in question 1? Do very inexpensive but non-OER materials result in similar changes in academic success?

3. Do the answers to questions 1 and 2 vary according to students’ Pell eligibility?

4. Do the answers to questions 1 and 2 vary according to the degree to which teachers exercise the reuse, revise, remix, and redistribute (4Rs) permissions granted by OER licenses?

We will initially conduct this research with Project Kaleidoscope participants and Open Course Library materials adopters. If you’re using open textbooks or OER in your course in place of traditional textbooks and would like to participate in this study, contact David Wiley.

The $5 Textbook – Now For Less Than $5

From time to time I re-review the costs from various print-on-demand vendors to see if we can’t get the cost down a little further on our just-as-effective-as-$100-big-publisher-books $5 textbooks. Good news! CreateSpace, one of the first sites I reviewed this time around, can print our books (8.5×11, B&W, paperback, 250 pages) for $3.85. Add $0.40 per book for shipping, and the $5 dollar textbook is now a $4.25 textbook, delivered. What else will we find as we continue to review vendors in preparation for 2012-2013?

If you’re not familiar with the $5 textbook, these are digital textbooks that use an open license (a Creative Commons license, to be more specific) originally published by CK12. These books are free to adapt, revise, improve, and use in any format. The Open Education Group at BYU works with districts to help them develop their own custom versions of these high quality high school science textbooks (and, starting Winter 2012, high school math textbooks) that are individualized for their students’ specific needs. Where infrastructure allows, these books can be used digitally for no cost. For other settings, we work with print-on-demand vendors to provide printed versions of these books that cost about $5 each.

When we compare the Utah state CRT scores of students who use the $5 custom books with students who use traditional, expensive textbooks, we find no difference in the percentage of students who are proficient at end of year. That’s a savings of over 50% of textbook costs (when you buy one per student each year and give it to the student to keep forever, highlight in, take notes in, etc. – things they aren’t allowed to do in their traditional textbooks) for the same amount of learning. We have anecdotal evidence supporting the conclusion that when students use these books proactively – taking notes, highlighting, etc., the amount they learn increases. While this finding would agree with previous research, we need more data to be able to make this claim with certainty.

And what can a district do when it saves 50% on textbook costs? Well, textbook money can only be used for textbooks, unfortunately, so districts can’t turn these savings into increases in teaching staff or teacher salary. But there are still interesting things that can be done. As one example, these savings can be used to purchase hardware (like iPads) on which digital versions of open textbooks could be read. The digital versions of books can still be annotated, highlighted, etc., and students can keep their digital versions forever. And the digital versions of these books are completely free. In other words, one way to view the financial savings gained from using open textbooks is as a completely self-funded way of acquiring the hardware needed to transition an entire district from static print material to interactive, digital content. And that’s just one example – a district is only limited by its imagination.

So far about 4000 Utah high school students have used these open high school science textbooks, with the support of the Open Education Group at BYU, CK12, and the William and Flora Hewlett Foundation. This year almost the entire Nebo district is participating. If every high school science class in the state were to save 50% for the same (or better) learning outcomes, the annual statewide savings would approach $1.5M.

We’re hoping to “greatly expand” the program for next year, and should be able to make a specific announcement soon. Keep watching!

Efficacy Data Are In!

The Big Take Away:

Schools can save over 50% on textbook costs without negatively impacting student learning.

The Short Version: Simply substituting open textbooks for proprietary textbooks does not impact learning outcomes.

The Longer Version:
If there’s one thing this project has taught me about data, it’s that they’re messy. I knew this already, but I assumed that with our relatively small year one pilot group (n=7 teachers) things would be easier and cleaner. Instead, what we ended up with was the 2011 CRT (state standardized test) scores for each teacher (these scores are a percentage indicating what proportion of their students demonstrated proficiency on the exam as judged by the state), the 2010 scores for each teacher, and the 2009 scores for only four of the teachers. We had hoped for 2011 plus three years back for every teacher, but some of our teachers are new (no data beyond 2010), some have moved schools (“difficult” to get data beyond 2010), etc. Fortunately there doesn’t seem to be any hidden systematicity to our missing data.

So what did we find? Table 1 gives raw scores.

Teacher T Teacher U Teacher V Teacher W Teacher X Teacher Y Teacher Z
2009 64 N/A 54 59 100 N/A N/A
2010 69 62 44 59 99 88 89
2011 61 61 58 82 100 83 85

There are two straightforward ways of asking these data about the impact on student learning of substituting open textbooks for proprietary ones.

The first, noisier way is just to subtract the 2010 scores from the 2011 scores (remember, these scores are the percentage of students achieving proficiency). When we do that, we get a distribution that looks like this: -8% (i.e., 8% fewer students achieving proficiency), -5%, -4%, -1%, 1%, +14%, +23%. The mean of this distribution is +2.86% and the mode is -1%. By either measure of central tendency, there is almost nothing happening in this data.

The second, slightly more stable way of looking for the impact on student learning is to subtract either the average of the 2009 and 2010 scores (when both are available) or the 2010 scores otherwise from the 2011 scores. This gives a slightly better picture of what the “true” previous scores were. This method provides the following distribution: (-5.5%, -5%, -4%, -1%, +0.5%, +9%, +23%), which has a mean of +2.43% and a mode of -1%. Again, there is nothing to see here folks. These aren’t the droids you’re looking for. You can go about your business. Move along.

As a side note, I should point out that we’re keenly interested in what the teachers who saw 23% and 14% jumps did with their open textbooks last year. One of these teachers told me, “the betters students write in their textbooks more.” If this casual observation turns out to be true, and this particular change in pedagogy can be propagated broadly, perhaps we can see wide increases in proficiency scores. We’re looking at just what exactly students are doing with their books more closely this year (with 20+ teachers in this year’s group).

We’ll be running more sophisticated analysis next year with the larger data set, and collecting data on this for a few years to come to improve the stability of the findings, but for the first year pilot this is a fabulous outcome. The implications for students, schools, and districts are “large” indeed. A more formal writeup to come.

Binder Stuffing

Today we stuffed 255 binders full of open science textbook material to deliver to some very happy teachers.

School is Starting and We’re Still Printing!

For some of our teachers school has just started and for others, it will begin in the next few days. Over the last few weeks we’ve been frantically trying to get everything printed on time but alas it hasn’t happened quite like we imagined. Several of our books are still out to the printers and most of them are already printed, sitting, and waiting for binders to arrive. It has been interesting to try and juggle all of the different printing options that we made available. We’ve printed and delivered “starter kits” for teachers to get going, and will be delivering completed books soon.

Some of the teachers decided to go with regular paper 3-hole punched and loaded into a 3-ring binder, and others chose to print a more traditional textbook-look by printing it paperback with a ‘perfect bind.’ Unfortunately there was some confusion about the technical terms in book binding and what exactly they meant. There were also unforeseen delays in ordering binders and in getting the manuscripts completely prepared. In the future, having fewer options (perhaps even dictating what kind of printing a teacher will get) could eliminate confusion and make the process go better. Furthermore, misunderstandings and disappointment is also less likely with fewer choices.

On the other hand, teachers make the choices they make for pedagogical reasons. It would be interesting to look at questions surrounding learning, pedagogy, and book binding. Do teachers and students use their textbooks differently depending on what form they take? Do they use a loose-leaf 3-ring binder textbook differently from one that is perfect-bound? What pedagogical goals are teachers considering when they make the decisions about how they want their books bound? What difference do they think it will make to the students and will it make a difference to the students in the end? –All interesting questions for further research.

Second Teacher Meeting

On August 9, 2010 we convened the second meeting of participating teachers. Having had a month to dig into their textbooks and the adaptation process, we were excited to see the progress teachers had made. It was incredible!

For example, the teachers working on adapting the Chemistry textbook reported on their work. They began with the stock CK-12 chemistry book which is an extremely thorough 1200 pages long! They then went through the book, pulling out material they didn’t need. They next searched the web for other open educational resources, inserting supplemental material directly into the textbook as they found what they were looking for. The end result: a 100% Utah-specific textbook, tailored precisely to the needs of these teachers’ students, that is less than 250 pages long! It looks like printing this book in paperback form will cost just over $5.

What happens when you empower teachers with open educational resources? First, you end up with a highly customized textbook. Second, you end up with a teacher who has read every word of the textbook she is using. Third, you save a huge amount of money. How will student learning be affected? That’s what we’re going to find out…

Launch!

Neeru Training Participating Teachers
Neeru Khosla, CK-12 Executive Director, trains participating teachers.
On July 8-9, 2010, we held the first teacher training sessions for the project. Neeru Khosla and other members of the CK-12 team came to the BYU campus for the two days. David Wiley of BYU opened the training by teaching about open educational resources, what they are, what they allow us to do, and why they’re important, and provided an overview of the project. Tiffany Hall (also of BYU) provided training on active learning strategies that can be employed with textbooks. Neeru Khosla and her team then provided nuts-and-bolts training on the CK-12 system. Teachers spent the second day digging into the CK-12 books and beginning the adaptation process, with CK-12 folks on hand to provide support when we ran into technical issues.

Group Photo
Some of the CK-12 staff, BYU staff, and participating teachers pause for a group photo.
Overall, the first training was a great success. Everyone left energized about the project, empowered by the idea of actually being in control of their textbook, and with a good understanding of the CK-12 system.