Try It. Use It. Study It.
A Research Primer for Educators
Educators and students are the source of truth on the classroom experience, and they hold invaluable knowledge as the end users of many edtech products. Leanlab Education’s AGILE Network and Codesign Collective tap into their expertise by bringing together innovative educators and school districts to collaborate with edtech companies seeking to solve real problems in education.
Members of the Network and Collective have opportunities to engage with edtech product research in a variety of capacities, sharing honest feedback and influencing product decisions. Leanlab conducts different types of research studies in partnership with schools and edtech companies—studies developed using the Codesign Framework, which emphasizes the co-creation of research objectives, methodologies, data collection, and dissemination among stakeholders.
Each type of research study asks different commitments of educator and school partners, and compensation is based on the scope of the study. Schools and/or individual educators are matched with studies that best fit their availability and capacity, and address their unique community’s challenges.
Try It.
The shortest-term research opportunities include concept validation and product validation studies. These studies are often conducted with edtech companies early in their product development process—the perfect time to incorporate formative research. Typically running only a few weeks, concept validation and product validation studies require smaller time commitments from educators and students, and they help set edtech companies in the right direction.
Concept Validation
The first step in the research journey, Concept Validation studies seek feedback on a product concept to validate if a product is needed by schools.
Concept Validation research questions
To what extent is this tool needed in schools? How would it add value?
Would this tool address a critical pain point for schools?
How do users believe the product compares to similar products?
This feedback is typically collected through focus groups or surveys. Sometimes, Leanlab may need to diversify the sample and recruit different administrators and teachers from across the country, while other times, participants are recruited from a single school.
Product Validation
Product Validation research studies tools farther along in the development process to understand how intended users would use it given the realities of school environments, and how the product might add value.
Product Validation research questions
To what extent is this tool needed in schools? How would it add value?
Can students see themselves using the tool? What about teachers? How does it compare to similar products?
What might keep schools from implementing this product?
Feedback is typically collected through focus groups or surveys from an educator or student sample that varies depending on the study’s needs.
In a recent codesigned study, teachers and students from Kansas City International Academy and Clinton County R-3 Schools received a product demonstration from CodeAlgo—an edtech platform designed to teach early learners the basics of programming and computer science. Elementary and middle school students shared ideas and suggestions for the user experience, while teachers also explored and gave feedback on the instructor interface. Their collective input helped ensure the product would address school needs for computer programming and informed CodeAlgo’s curriculum development.
“There’s a lot of learning that happens in the process of piloting new ideas, and being a part of that process is both exciting and enlightening. For us, involving our students in the CodeAlgo study gave them instant agency in their own learning, and helped them see the real-world applications of the technology they’re learning. It’s always good to engage students in the process and empower them to take ownership of their learning. It was definitely a neat experience all around.”
Brandon Burns
Principal, Plattsburg High School
Use It.
Usability
The next step in the research journey is a Usability study. Usability studies, which run for 3-4 weeks, explore and improve the ease of use and appeal of an edtech tool with actual end users. During a Usability study, Leanlab researchers will present the edtech tool to a teacher, administrator, or student and ask them to perform a series of actions on the platform (this is called a task analysis).
Researchers will note which actions the participants struggle with and which actions can be completed with ease. After the task analysis, more qualitative data is gathered by interviewing the research participants.
Usability research questions
How well can students in the targeted age range independently navigate the product? How well can any user independently navigate the product?
Are users able to find and use features? Could they access those features more easily?
What are the most valuable features of the tool?
While still lighter on time commitment for school partners, and open to individual Codesign Collective participants, Usability studies offer users more exploration of the edtech tool and helps researchers gather first impressions of the product and its functionality. Feedback generated helps edtech companies create a platform that is simple to use and, hopefully, saves teachers time and frustration.
Rapid Cycle Evaluation
Rapid Cycle Evaluation studies allow teachers the opportunity to see the impact of their research participation in real time. Over the course of several cycles—often two or three cycles that each last four weeks, for a total of 8-12 weeks—educators implement a tool, provide feedback, and developers iterate on the product before beginning the next cycle. There is considerable variation in the types of tools teachers work with in Rapid Cycle Evaluation studies; in one recent study with SpeakAgent, teachers and students helped design a new language and math product in the first rapid cycle, provided feedback on the product plan in the second design cycle, and then interacted with the prototype by the end of the study. In another Rapid Cycle Evaluation study, elementary teachers and their students used an on-market AI literacy tool consistently of the course of three cycles, with their feedback leading to product improvements after each cycle.
Rapid Cycle Evaluation research questions
To what extent is the tool needed, and how would it add value?
How well can end users successfully use the product in the classroom or other authentic learning environment?
What barriers or challenges might keep people from using this product?
What are the most valuable features of the tool?
Feasibility
Feasibility studies are similar to Usability studies, except that during a Feasibility study, teachers use the product for a longer period of time and have greater opportunity to explore the platform on their own.
Feasibility studies collect data to improve a product’s ability to be used within a classroom setting. These studies collect data over 4-8 weeks, allowing for regular use by teachers and students. To collect data on how educators feel about the product, Leanlab asks teachers to keep a user journals and regularly jot down their impressions, ideas, and feedback. Teachers are also asked to participate in one or two focus groups or interviews throughout the study.
Feasibility research questions
How successfully can this tool be used within a classroom?
What are the barriers to use in the classroom?
What is the value added by the tool for teachers and/or students?
In fall 2022, Leanlab codesigned a Feasibility research study with charter middle school IDEA Round Rock Tech and Forged Ed—an advisory curriculum intended to help educators build the shared knowledge, vision, and collective capacity to forge schools where students of color thrive. Over a five-week period, teachers implemented Forged Ed’s advisory curriculum, with each teacher completing a list of tasks that covered the full range of the tool’s features. In addition to participating in focus groups after the pilot period, teachers also kept journals of their insights and experiences in real time.
Study It.
Implementation and Correlational studies require the most time commitment from educators and coordination with schools/districts, but they rarely last more than a semester.
Implementation Research
Implementation studies are designed to investigate the practicalities of using an edtech tool. A company can create the most usable and impactful edtech tool but if it doesn’t integrate well with other student data systems or the onboarding instructions are lackluster, it will never get used in the classroom.
During an Implementation study, teachers will use the tool for up to a semester. Researchers pay special attention to the first few weeks of usage, when teachers are learning how to use the platform and integrating it into their daily teaching routines.
Implementation research collects data over a full semester across multiple classrooms and/or schools.
Implementation research questions
When integrating a tool into a classroom, which techniques are most effective?
How do contextual factors influence implementation success or failure?
How can these contextual factors be modified to increase chances of success?
Correlational Research
Correlational studies may feel similar in the amount of time required by the participants; however, the research goals are different. Correlational studies seek to understand if there is a relationship between a tool’s use and the intended outcomes. Leanlab’s most intensive level of research, Correlational studies take place over a full semester or longer. Typically, this will include a pre- and post-survey to measure changes in qualitative and quantitative outcomes. For example, do teachers report that students seem more engaged (qualitative) and do student’s test scores increase after using the product (quantitative).
During a Correlational study, teachers implement the edtech tool into their curriculum and use it as intended for a semester or longer. Leanlab researchers will periodically check in with the teachers through interviews and focus groups to understand if there are any challenges with using the product. Feedback is then relayed to the edtech company, so they can make product revisions.
Correlational research questions
What is the relationship between the use of the product and the intended outcomes?
Is there a correlation between use of the product and improved outcomes?
Over the course of the 2021-2022 school year, Leanlab conducted a Correlational study with Speak Agent, creators of content+language programming, and Prince George’s County Public Schools. Nearly 1,600 middle school classrooms had access to Speak Agent’s Math+Language platform, the effects of which were evaluated through math and English language proficiency outcomes.
Dr. Michelle Dyson, PGCPS Mathematics Instructional Supervisor for Grades 6-8 shared, “We could see students making progress, so we collaborated with our ESOL department to commission this study to quantify the impact. This research shows that our implementation–which closely integrates with our curriculum–helped students make strong gains on math and ELD assessments.”
There is a need for a broad range of educator perspectives and expertise across edtech research. With different types of studies and varying goals, there are also research opportunities to fit educator and school priorities and constraints.
Leanlab’s goal with the Codesign Collective and the AGILE Network is to expose educators and students to new edtech tools that are attempting to solve their specific challenges and give them a voice in the development of those solutions.