SEGway May 2016
Research and Assessment News from SEG Measurement
|
|
|
31 Pheasant Run
New Hope Pennsylvania 18938
800.254.7670
267.759.0617
|
|
|
|
Greetings!
We are wrapping up quite a few effectiveness studies from from kindergarten to post-secondary, in science, history, math and other subject areas.
We are pleased to see that educational publishers are gaining an early understanding of the Every Student Succeeds Act (ESSA), and preparing now for the fall 2016-2017 school year. As you'll see from the first article in this issue of SEGway, educational product providers can also take advantage of summer 2016 programs to conduct an efficacy study at a reduced cost; a summer 2016 study will allow you to boost fall marketing and sales efforts and provide early information for product development.
In this issue of SEGway, we also discuss assessments which are the cornerstone of the instructional process for teachers, administrators, and educational publishers. Strong assessment and reporting components within online educational products are critical for diagnosing, monitoring student performance and for clearly showing student outcomes.
We encourage you to learn more about our work in assessment and efficacy research by connecting with us at conferences. We look forward to continuing the discussion with you through the newsletter, on Twitter (@segmeasure) and on LinkedIn (SEG Measurement).
Sincerely,
Scott Elliot
SEG Measurement
|
|
|
 |
Make Summer Count For Kids
Does your summer intervention truly help students learn?
"Students who attend summer programs have better outcomes than similar peers who do not attend these programs. Not all summer learning programs result in positive outcomes for enrollees. Programming needs to be high-quality..." according to a 2011 publication by RAND Corporation.
Is your product one of those that leads to positive outcomes for students attending summer programs? How do you know? Do you have evidence? Your customers want to know.
With an SEG Measurement summer effectiveness study, you will be able to: 
-
Have results for fall 2016 marketing
-
Provide tangible proof that your product increases student achievement
-
Have your product qualify and be ready for school buyers to purchase under ESSA requirements
-
Stop summer learning loss by engaging students
-
Assist students with higher learning outcomes
-
Conduct a quick pilot study to discover any issues and make needed improvements prior to fall implementation
-
Measure student and teacher engagement
Conducting a summer efficacy study can be a very cost effective way to obtain proof of effectiveness. A short four to eight week summer 2016 study of your educational program can be conducted at a lower cost than studies conducted during the regular school year. And, the results can be ready for fall marketing.
Join our growing list of clients at SEG Measurement that have benefited from a summer efficacy study. Increase your customer base with more sales, while helping students gain critical knowledge and skills during the summer months. Be ESSA ready!
Call or email us today at 800-254-7670 ext. 102 or hrickert@segmeasurement.com for a cost-effective summer efficacy study by SEG Measurement.
|
|
Does Your Program Have the High-Quality Formative and Summative Assessments that Teachers are Asking For?
Weak Assessment and Reporting Features Severely Limit the Value of Your Educational Product
|
More than 4 out of 5 educational buyers say that the quality of the assessment component of a product is an important part of their purchasing decision, according to a study by KnowledgeQuest. How strong is the assessment component of your product?
Most of the educational programs we see attempt to align their instruction with learning theory principles, help keep students engaged with the material, and are standards-aligned. Many of these programs even have evidence that they help students to learn more than alternative instructional products. Yet, all too often we find that digital learning programs are missing rigorous, reliable and valid assessments to diagnose, monitor and measure outcomes.
The instructional innovation offered by many of the online educational programs may not be enough to achieve the level of outcomes that educators are seeking. For your product--digital or otherwise-- to be effective, the product needs to provide quality assessments that can be used formatively and summatively. Teachers need to be able to quickly identify the skills for which students need extra assistance and additional instruction. A program without an effective assessment component limits the usefulness of the program. Simply providing test items for teacher use is helpful, but is a far cry from meeting the expectations for quality assessment. It is critical to have assessments that are technically sound, clearly fit with the curriculum, fall naturally in the learning path, and that have results that are actionable, easy to view and can support decision making.
One of the most common failings we see of instructional tool providers is failing to take the time to carefully think out and design the assessment component. The purpose(s) of the assessment needs to be defined first in order to define the test specifications to best meet the purpose(s). Once the purpose of each assessment is determined, additional details such as skill coverage, types of items, length of test, scoring rules, and recommendations based on performance can be determined. Considerations such as scaling to allow monitoring of progress over time and establishing standards for competency need to be addressed.
The assessments are only of value if they can easily be used to make instructional decisions. Effective, easy to use reporting features for administrators, teachers, and students are essential in order to for the assessments to be of value. Assessment data that is unclear, hard to find, or incomplete works against the value of the product.
Field testing should be conducted to evaluate the item and test quality and to verify the appropriate use of assessment data within the instructional components of the product. User feedback should be collected regarding the assessment reporting features to inform future product development.
We can help you to plan for, develop, and evaluate assessments and reporting that are integrated within your educational product.
We offer a complete suite of psychometric services. Please contact us to help you plan and execute this work at info@segmeasurement.com or 800-254-7670. For a sampling of our assessment services, please visit
|
|
|
|
Recent Questions Asked by Educators and Publishers Considering Conducting an Efficacy Study
|
|
Here is the latest installment of this popular section of the newsletter with common questions and answers. If you have specific questions that are not covered here or if you are interested in exploring any of these questions further, please contact us. We are here to help.
What is the role of a teacher survey in a product effectiveness study?
Including a teacher survey allows for the collection of qualitative feedback regarding your product, particular usage information in the classroom, and teacher perceptions of effectiveness. We recommend that both treatment (users) and control (non-users) complete surveys with some overlapping/shared items and some unique items relevant to their study group. Data collected in the teacher survey lends more perspective and supporting information to the quantitative evaluation of student performance. It is also a great opportunity for you to collect additional information from teachers about their use of the product to help inform product enhancements.
We understand we need to use a pre/post test design for our effectiveness study. Yet, we are not sure how to select the proper assessment to use. How do we decide among a custom test, a third-party test, and using available state test data?
The goals of the study help to drive this critical decision regarding assessments. State test data allows for you to have pre and post data without any additional testing requirements for the students, yet this data may not measure the outcomes of greatest interest to you or may not be closely related to your product. A third-party assessment allows you to administer the assessments as part of the study in very specific windows and allows you greater control over the data collection process. A custom assessment also allows for control over administration and data collection, yet it also allows you to target very specific skills and outcomes that may not be available through a third-party or state assessment. Custom assessments need to be carefully crafted and evaluated to ensure that they do not advantage users of your program. We can help in determining the most appropriate assessment plan for your study while balancing psychometric, logistic, and research design needs.
How much time is typically needed for a teacher to participate in a study?
For teachers in the treatment group, they will need to use your product with fidelity according to your recommendations. In many cases, the teachers are already doing this and would not need to change their instruction. In other cases, the teachers may need to participate in virtual training and may need to spend some time planning for the use of your program. In addition to the instructional component, the treatment teachers may need to complete a survey that could take anywhere from 5 - 20 minutes to complete depending on complexity. Teachers may also need to administer a pre and post assessment, which could take one class period each.
For teachers in the control/comparison group, they will need to continue with their traditional instruction and not make any changes to their teaching. In addition, they may also need to complete a brief survey and administer the pre and post assessments, if applicable to the design.
Studies can be designed that take no to very little teacher or class time to complete. Other study designs could include more involvement of the teachers and students. Once we finalize the design with you, we can provide a more detailed estimate of the level of effort for the participants.
What questions do you have? Let us know what questions you want answered. Contactinfo@segmeasurement.com to share your feedback.
|
Contact SEG today to find out how we can help you establish the effectiveness of your education product or service:
|
|
SEG At Upcoming Conferences
Let's Meet!
We are looking forward to seeing our colleagues and meeting new friends at the upcoming conferences. We are participating in several sessions & we invite you to join us.
|
|
Look for us at these upcoming conferences:
-
AAP Content in Context, June 6 - 8, Philadelphia, PA
-
CCSSO National Conference on Student Assessment, June 20 - 22, Philadelphia, PA
-
AACE EdMedia 2016, June 27 - June 30, Vancouver, Canada
-
NJEA Convention, October 10, Atlantic City, NJ
We would love to meet with you and discuss how we can help you build strong assessments and get the proof of effectiveness you need for success.
If you would like to meet with a representative from SEG Measurement to discuss how we might help you with your assessment and research needs, please contact us at info@segmeasurement.com. |
About SEG Measurement
Building Better Assessments and Evaluating Product Efficacy |
SEG Measurement conducts technically sound product efficacy research for educational publishers, technology providers, government agencies and other educational organizations, and helps organizations build better assessments. We have been meeting the research and assessment needs of organizations since 1979. SEG Measurement is located in New Hope, Pennsylvania and can be accessed on the web at
|
|
|
|
