January 2011

SEGway
An update from SEG Measurement
News and information about educational research and assessment

January 2011
In This Issue
Improving Test Quality: Why Field Test Assessments?
Pressure Mounts for Proof of Effectiveness
Featured Study: 3D Library Effectiveness
SEG at Upcoming Conferences
SEG is Named to Who's Who in Testing


Improving Test Quality: Why Field Test Assessments?

Most educational products are offered with some form of assessment.  All too often, however, the assessment component receives less attention during development than the instructional components.

The quality of the assessments accompanying educational products is often the source of dissatisfaction among teachers and school administrators.  Schools increasingly rely on these assessments to target instruction and monitor student progress.  
 

In many cases, educational publishers develop the assessments without pilot testing or field testing the assessments.  And, even when they do, they fail to analyze the results using sound psychometric procedures.
 

Editorial review, without field or pilot testing, is not sufficient to guarantee high-quality assessments. I realize that this statement will not make me popular with editors working on assessments.  But study after study has shown that editors are less than 100% reliable in judging assessment quality and even less so in judging assessment difficulty.

Field Testing provides a basis for evaluating the quality of the tests and individual items.  Administering the draft tests/items to a group of students comparable to those that will be using the product allows you to achieve several goals.

Evaluating item quality. First, the results of the field test can be used to evaluate the quality of the test items.  A sound item analysis performed by a competent psychometrician can tell you if the item is "working"-- whether or not the difficulty is in an acceptable range and whether or not it is being answered in a systematic fashion as expected.  For multiple-choice questions with several response options, the item analysis can tell you if one or more of the response options are not "working." 

Evaluating overall test performance.  Second, a psychometric analysis of the field test results can tell you whether the test is reliable (measures the underlying skill consistently) and whether or not it is well-targeted in difficulty for the population of students for which it is designed.  It can also tell you how different groups of students from different population groups respond to the test and whether any differences are the result of expected differences or test bias.
 

Calibration/Equating. Third, if you are intending on using the assessments to measure student growth over time, then the field test can be used to place the test scores on a common scale.  Calibrating or Equating the tests allows the tests to be placed on a common scale so that student results can be compared accurately over time--across versions of the tests, within school years and over multiple school years.

In summary, developing high-quality assessments to accompany educational products is very important.  Delivering high quality assessments requires field testing followed by a sound psychometric analysis of the results. Taking these steps will help ensure better assessments and ultimately more satisfied customers and improved product sales.

If you currently provide assessments or are developing new ones, please contact us to help with field testing and analysis. 
Greetings!

Welcome back! Schools are back in session and Publishers and Technology Providers are trying to figure out what schools need and how they can best get them to adopt their products. This issue examines several topics that should help you in your quest to build better assessments and obtain evidence of product effectiveness.

Publishers often wonder whether or not to expend the resources to pilot or field test their assessments.  In our technical corner, Dr. Elliot, SEG's president, explains some important reasons why field testing is essential. 

Also in this issue, we explore the growing pressure on publishers and technology providers to provide proof of effectiveness, with an intriguing discussion of Academic ROI.

Our featured study this month is a very interesting one looking at the integration of print products with computer delivered animation. The 3D Library product is among the most interesting and effective we have seen in a long time.

SEG will be attending several conferences and events in February and March.  We look forward to seeing you there.  Please let us know if you will be attending and if you would like to meet with us.

As always, please feel free to learn more about us at www.segmeasurement.com.

Sincerely,

Melissa Garza
Editor
 

Pressure Mounts for Publishers and Technology Providers to Provide Proof of Effectiveness: Academic ROI takes center stage

 Educational publishers, technology providers and service providers are under increasing pressure to deliver proof that their products and services are effective.  Faced with limited resources, schools are increasingly making purchasing decisions based on the extent to which products and services have been shown to increase learning and improve other educational outcomes.  And, many federal and state funding decisions are tied to scientifically-based research evidence.

"Academic ROI" is taking center stage in this evidence-based purchasing environment.  Corporate America has long relied on the concept of ROI (Return on Investment).  In short, this is a measure of the amount of money made (Return) for a given amount of money invested (Investment). 

Academic ROI is a translation of this concept into the education world as a way to measure the amount of academic outcome achieved for a given amount of investment made. 

The Academic Return on Investment (Academic ROI) can be measured as the amount of learning or other educational outcome achieved per dollar spent on a school product or service; in the true vernacular, "Bang for the Buck."

For example, let's assume that a school is considering purchase of a particular supplemental product and we want to compute the Academic ROI.  Assuming that a sound research study was available that listed the amount of gains achieved on the state test we can compare the cost per student to the gains achieved to determine the ROI. If the study reported that students using the product in a one year study gained 10 points and the cost per student of this product is $20, we can calculate the Academic ROI as 10/20, or .5.  This means that for each dollar spent, we can expect to see, on average a 1/2 point  increase in student scores.
 


If we use a an indicator of outcomes that is consistent across studies, such as the field-standard Effect Size, we can use this method to compare several competitive products.  The Academic ROI for each can be determined and we can consider this information along with other factors to make more informed purchasing decisions.   

Academic ROI is likely to take on a growing role in school decision making as limited resources continue to be a problem and as school administrators are faced with an ever increasing set of purchasing options from which to choose. Educational publishers, technology providers and service providers would be wise to provide sound effectiveness research to support buying decisions.

 

 

Featured Study: Augmented Reality Enhanced Books Shown to Significantly Increase Student Learning
 

Augmented reality, a technology that augments real-world environments with computer-generated sound or graphics, is one of the most exciting new technologies that we have seen introduced in education in a very long time.  SEG has been fortunate to work with one of the first providers in this area, 3D Library. In the fall of 2010 SEG Measurement evaluated the effectiveness of the 3D Library program in 17, 4th grade classrooms in 5 states.
 

The 3D Library program consists of two components: 1) a set of books, each containing age-appropriate content aligned to the book content and includes extensive illustrations and photographs where appropriate, and  2)  a computer equipped with a webcam (or dedicated device) with the 3D Magic Library software, animations, voiceovers, and music.

 

To use 3D Library, the student reads the book directly in front of the computer/device where the webcam can recognize the page content.  As each two-page spread is opened, the 3D Magic Library software automatically recognizes the page and launches the aligned animation on the computer screen along with narration and music.  The animations are presented using augmented reality, which presents a visual image along with an actual image (in this case, a 3D animation with the actual book and the student reading the book). 

 

Students can move the book from left to right, turn it and tilt it, and the image will move and change position with the book.  For example, an image of an animated dinosaur can be seen from the front and side by turning the book.

 

The tool is designed to support educators and engage students in learning science and social studies content and can be used in both one-on-one and group settings in numerous ways, from introducing a new lesson or topic to illustrating complex subject matter to learning a body of newly introduced content.

Study Design and Results
 

The quasi-experimental study compared learning growth among a group of students using both the 3D Magic Library computer-based animations and books, a group of students who used only the 3D Magic Library books, and a control group who used none of the 3D Library Program components.  The learning growth among the three groups was compared using Analysis of Covariance (ANCOVA). We compared the average (mean) score for the 3D Magicbooks Users Group to the average (mean) score for the Control Group (adjusted for any initial differences in student ability).  The results indicate that students who used the Program learned significantly more than comparable groups of students who used only the books or who used none of the program components.

 

The Effect Size for the comparison of students using both the 3D Library program books and computers to students using only the books for the 3D Library program was .47. The Effect Size for the comparison of students using both the 3D Library program books and computers to Control Group students who did not use any of the 3D Library program components was .56. This is a large effect; for a student at the 50th percentile, an effect size of .47 reflects and increase to the 64th percentile and an effect size of .56 reflects and increase to the 71st percentile. 
 

 

SEG at Upcoming Conferences


SEG Named in "Who's Who in Testing" in leading industry analysis report