At Leanlab, we often speak about research as a journey. Unlike a journey, though, there is no end point. There’s no point in the life-cycle of an edtech company where research loses its relevance. The journey metaphor is used to illustrate that edtech companies can and should be using research to codesign and evaluate their products with the input of educators, students, and parents at every stage of development—from using focus groups and interviews to understand if a product solves an actual problem, to engaging users throughout the semester, to measuring the impact on student outcomes.
The "journey" also describes the cumulative effects of conducting research in a certain order. Before we can measure student outcomes in a quasi-experimental study, we have to ensure that teachers and students are using the tools as intended and there’s high user engagement through usability, feasibility, and implementation studies.
“We consider research not a one-and-done, but an ongoing process. We care about the results, even if they're not positive, because that's what enables continuous improvement in our platform development.”
This process can, at times, seem slow or incremental but there are moments within the journey that reaffirm that the idea is worthwhile and meaningful.
We recently finished a quasi-experimental study with Speak Agent—an academic language learning platform for K-12 public schools—that yielded some very exciting results. Our analysis found significant improvements in math scores for middle grade students and a significant, positive effect on multilingual learners' English language proficiency scores as measured by third-party assessments.
However, the process they used to get to those results embodies the idea of the research journey. “We consider research not a one-and-done, but an ongoing process. We care about the results, even if they're not positive, because that's what enables continuous improvement in our platform development,” said CEO Ben Grimley.
We first worked with Speak Agent, Inc. in 2019 to implement a pilot of the academic language learning platform at KIPP KC. This first study we undertook with them used a mixed methods approach to understand the effects of Speak Agent on reading performance, social-emotional outcomes, and attitudes towards reading and learning.
Speak Agent has continually used this approach to incorporate both design and instructional changes in their products. Chief Learning Officer Dan LaFountain notes, “We’ve added in the option for students to speak their responses in many of our learning activities—in addition to typing them—a change that came directly from teachers at our partner districts.”
Dan LaFountain, Chief Learning Officer at Speak Agent and Skylar Hurst, teacher at KIPP:KC, during Leanlab Education’s K12 Fellowship in 2019
After the promising results from this pilot study, we partnered with Speak Agent again to conduct two more data analyses. This time we teamed up with a long-time partner of theirs, Prince George's County Public Schools (PGCPS).
The initial data analysis explored the effect of using the platform in isolation, while the second study incorporated the effect of Speak Agent on standardized test scores. “The first analysis laid the foundation and allowed us to make changes in our data collection,” shared Grimley. The second analysis incorporates student Math Benchmark assessments from a sample of nearly 1,600 middle school classrooms.
The study evaluated the effect of Speak Agent Math+LanguageSM usage on student outcomes in math and English language proficiency across grades 6 to 8 within PGCPS. The nation’s 20th largest school district, PGCPS serves a diverse student population from urban, suburban, and rural communities. All of the classes and students in the sample had full-year access to the platform, with usage based on teacher and student choice in real-world classroom environments.
Data analysis showed that students who used Math+Language outperformed their peers who did not use the program. Students who completed ten Speak Agent learning activities experienced a 10% higher mean score than their peers on the PGCPS math benchmark assessments. Multilingual learners experienced a 5% higher mean score on the WIDA ACCESS for ELs test, in addition to a higher math score.
This is the first large-scale study to find a positive, independent effect at a 99.9% confidence level for both Math and English language proficiency scores, as measured by third-party assessments, caused by a single education technology program.
A student who completes one activity on Speak Agent is expected to get a score of about 329.54, a student who completes two activities is expected to get a score of 331.25, while a student who completes 10 activities on Speak Agent is predicted to get a score of 344.91.
Dr. Michelle Dyson, PGCPS Mathematics Instructional Supervisor for Grades 6-8, said of the findings, “This research shows that our implementation—which closely integrates with our curriculum—helped students make strong gains on math and ELD assessments."
At PGCPS, in addition to assessment and platform data, students participated in a short survey. Eighty percent of the student respondents agreed or strongly agreed that they enjoyed Speak Agent, and 86% agreed or strongly agreed that it helped them learn math.
Speak Agent embodies what we look for at Leanlab in edtech partners: enthusiasm for codesign research, appreciation for data, and willingness to keep iterating to find the best edtech solution. It’s exciting to see a commitment to research-driven product development result in such positive student impacts.
