Many educational publishers and technology providers are uncertain about what constitutes an acceptable study of product effectiveness. While ESSA (Every Student Succeeds Act) seeks to unravel this confusion, publishers and tech providers still have many questions. To help answer some of the key questions you have, SEG Measurement has published this special edition of our SEGway newsletter.
We felt one of the best ways to demonstrate what a strong efficacy study should look like is to share an example of an efficacy study that reflects the high efficacy research standards for which SEG Measurement has come to be known.
We are sharing some of the highlights of the efficacy study SEG Measurement completed for LearnBop in one of the nation's largest and most diverse school districts (Hillsborough, Florida). LearnBop is an interactive, instructional mathematics application designed to help students improve their mathematics skills. SEG was able to assist LearnBop with crucial evidence as outlined in the ESSA.
SEG Measurement conducted a controlled, matched-group, quasi-experimental study of the impact of LearnBop use on middle-school students' mathematics skills. A pool of thousands of students were examined to create matched groups of LearnBop users and non-users of LearnBop using propensity score matching--students were matched based on their ability prior to the start of the study and other key variables.
The matched groups of users' (treatment) and nonusers' (control) mathematics skills growth were compared, by examining their end-of-year, Florida state assessment test scores. The two groups (independent/predictor variable) were compared using Analysis of Covariance (ANCOVA) with the end of year test scores serving as the outcome/dependent variable adjusting for the prior year test scores (covariate).
The findings clearly showed that students using LearnBop showed much greater improvement in their mathematics skills. Students in classes using LearnBop achieved 11-17 percentile points more growth on end-of-year state assessments in mathematics as compared to students in classes that did not use LearnBop.
This study effectively supported Hillsborough's need to know if the product they were choosing was effective. At the same time, the study effectively supported LearnBop's need to prove that LearnBop works to support their marketing and sales.
This is not the first time a well designed and executed SEG study has supported LearnBop's efforts to support its marketing and sales. Dr. Pamela Haimer, Assistant Superintendent of Curriculum & Instruction, Freehold Township School District, NJ. captures this best: "We were so pleased with the results that we decided to deploy LearnBop district-wide."
Please visit our website to read more about the LearnBop study:
School buyers want to spend their precious school resources on products that are supported by technically-sound proof of effectiveness research. And, educational publishers and technology providers realize they are losing sales if they do not have sound proof of efficacy.
