Assessing the Impact of Digital Media on Learning and Teaching

Elizabeth Pierson is a highly experienced researcher and evaluator. In her work for EDC’s Center for Children & Technology, she has contributed to multiple studies that provide new insights into effective STEM education and the role of technology in supporting 21st century school and system reform in the U.S. and around the world. She played a key role in evaluating Cisco’s 21st Century Schools Initiative, researching Intel’s one-to-one laptop work, and examining Teachers’ Domain NASA Physics and Engineering Collection online professional development. Most recently, she managed an impact study in which EDC investigated how teachers are using PBS LearningMedia digital resources and what impacts the usage is having in classrooms. In this post, Elizabeth shares findings from this impact study that she presented at the 2015 International Society for Technology in Education (ISTE) conference.

In June 2015, our team—myself, PI Babette Moeller, Pilar Gonzalez, Loulou Momoh, Ashley Lewis Presser, Angela Pazzaglia, and Jessica Wille—completed an impact study of PBS LearningMedia, a free media-on-demand service that offers educators access to the best of public media. Previously known as PBS Teachers and TeachersDomain, PBS LearningMedia is a partnership between PBS and WGBH Educational Foundation that delivers research-based, classroom-ready digital learning experiences to engage students in exploring curriculum concepts that align with national education standards and Common Core State Standards. Nationwide, 1.6+ million teachers have registered to access the services’ over 100,000 digital resources. 

Research Questions, Design, and Participants
To help PBS better understand PBS LearningMedia’s potential as a teaching and learning tool, our study addressed four research questions:

  1.  How do teachers integrate PBS LearningMedia resources into the curriculum?
  2. What is the impact of PBS LearningMedia resources on teachers and on the quality of their instruction?
  3. What is the impact of PBS LearningMedia resources on students’ learning, school engagement, and critical-thinking skills?
  4. What conditions enhance or impede the use of PBS LearningMedia resources?

Using a case-study design, we explored the potential impacts of the sustained and structured use of PBS LearningMedia resources on middle school teachers’ classroom practices and student learning outcomes in English language arts (ELA), mathematics, science, and social studies. We conducted the study during the 2014–2015 school year in three school districts in New Jersey and California that volunteered to participate. Study participants included 47 teachers and 2,211 students across the four subject areas. The samples for all subject areas comprised teachers and students from grade 8. The ELA sample also included teachers and students from grades 6 and 7.

Collaborative Development of Curriculum Units
To help ensure that teachers and students used PBS LearningMedia resources with sufficient intensity and consistency across classrooms, our study team and PBS staff collaborated with a small group of educators from each subject area to develop adapted curriculum units that integrated PBS LearningMedia resources. Each subject-area team created between 9 and 25 lesson plans that incorporated one or more PBS LearningMedia resources into a 90-minute class period. We shared these adapted curriculum units with a larger group of teachers from each subject area, who also received an orientation to the materials during a professional development day. All lessons and accompanying materials were available on simple websites (visit the study’s Science and Social Studies Teacher Curriculum and Math Teacher Curriculum sites), which made accessing the resources easier for most teachers. The Science and Social Studies site is shown below.

Overview of Methods
Measuring Impact on Teachers: We collected teacher data through pre- and post-surveys, weekly instructional logs, classroom observations, and informal interviews.

Measuring Student Learning Outcomes: We designed subject-specific assessments to examine students’ content knowledge before and after their teachers engaged them in the PBS LearningMedia-enhanced curricula. Then, we compared this data with student achievement data from state (e.g., New Jersey Assessment of Skills and Knowledge—NJ ASK) and national (e.g., National Assessment of Education Progress—NAEP) assessments. We created the subject-specific assessments because there were no existing assessments to measure just the content knowledge and skills targeted in the PBS curricula, but we were careful not to align the assessments directly to specific activities within the curricula. If we had done so, it would have given the students participating in our study an unfair advantage over the national and state groups whose responses we were using as comparison data.

To ensure that both the assessment and the curricula were grounded in the learning goals but not directly related to the curricular activities, we created a “learning blueprint,” or matrix, that we used to match teacher-identified learning objectives to the existing state and national assessment questions. We also administered a survey—partially derived from the School Engagement Scale (SES) and Critical Thinking in Everyday Life Scale—that asked students to report on their critical thinking practices and evaluate their experiences with PBS LearningMedia. We administered the assessments and surveys twice, once before and once after the PBS LearningMedia-enhanced curriculum was implemented.  

Snapshot of Our Findings
The data we collected through this impact study do not allow for making causal inferences about the connection between the use of PBS LearningMedia resources and the observed outcomes (a randomized control study would be required to demonstrate causality). However, the findings are consistent with such a connection and provide evidence of the promise of the PBS LearningMedia website and its resources for improving outcomes for teachers and students. Below, I summarize some of our key findings related to increased student learning, increased frequency of critical thinking practices, and changes in classroom use.

Increased Student Learning: On average, and across subject areas, students’ performance on content assessments increased by 8 percentage points over the course of the study. There were significant increases in students’ performance on the content assessment for science, social studies, and math, but not for ELA. We found no significant change in ELA students’ content knowledge in grades 6 and 7, and a significant decrease in their scores on the content assessment in grade 8. The decrease in test scores for grade 8 may be due to a series of guest speaker lectures unrelated to the adapted curriculum that resulted in a lower number of regular instructional days during the six-week implementation period for that grade.

Participating students outperformed national and state norms for the content assessments by an average of 10 and 11 percentage points at the end of the study, respectively. As an example, the graphs below show students’ pre- and post-test scores for certain test items as compared to students’ scores nationally. The ELA data are compared at the state level because national level items were not used in the assessment.

Increase in Frequency of Students’ Engagement in Critical Thinking Practices: Across subject areas, over half of the 2,211 students who took our survey reported increases in the frequency with which they engaged in critical-thinking practices (56%). There was a significant increase in the frequency with which science, social studies, and ELA students reported engaging in critical-thinking practices, but not for the math students.

Impacts on Instructional Practices: Teachers in all four subject areas indicated that the PBS LearningMedia resources had an effect on their classroom and lesson-planning practices, though the effect on ELA teachers’ practices was only moderate. Teachers reported that they spent less time lecturing and were more likely to integrate digital media. They also noted that they had more resources to draw from and were able to design more student-centered and collaborative activities when planning lessons. Three-quarters of teachers in all four subject areas would recommend the use of PBS LearningMedia resources to their colleagues.

Across subject areas, participating educators were most likely to use PBS LearningMedia resources to teach or reinforce content, concepts, and skills. Within subject areas, we found differences in the purposes for which teachers used the PBS LearningMedia resources, the types of resources used, the frequency with which the resources were used, and the types of activity formats into which teachers integrated the resources. Teachers frequently used resources in math and ELA curricular units to engage students in practicing skills, while teachers often used the resources in the science and social studies curriculum units to extend student thinking and to assess student knowledge, respectively. Social studies and math teachers were most likely to use PBS LearningMedia resources in a small-group setting, while teachers in science and ELA used a variety of different activity formats, including small-group, whole-class, and individual activities, and combinations thereof.

Final Reflections/Looking Ahead
The results of this study suggest that PBS LearningMedia can have a significant impact on middle school students’ learning outcomes when digital content is embedded in the curriculum in sustained and deliberate ways. Our findings also suggest that the sustained and structured use of PBS LearningMedia resources over a 6- to 10-week period can have an important effect on teachers’ instructional practices. In addition to the session that my PBS colleagues and I presented at the 2015 ISTE Conference, PBS  hopes to present these and related findings at upcoming conferences such as the  2016 National Science Teacher Association Conference, the Future of Education Technology Conference, and the Texas Computer Education Association. Check this website for news about our upcoming presentations and publications.