Assessment Projects

Validate a Placement Test for English Language Learner Program Placement

Client: 

National Geographic School Publishing (NGSP)

The Challenge: 

A growing number of students speak a language other than English as their primary language. It is critical that students be placed in the appropriate instructional level to have the best opportunity for success. NGSP, a provider of English language instruction and assessment, recognized the importance of proper placement and asked SEG to validate the Placement Test they had developed. They needed to ensure that the tests were psychometrically sound and that the scores and recommended placements led to proper placement decisions. SEG was asked to conduct a validity study for the assessment.

The Solution: 

SEG conducted a psychometric analysis of the measures developed and conducted a validity study. Using both classical and Item Response Theory, SEG evaluated test and item performance. SEG verified that the items and tests were psychometrically sound. In order to validate the assessments, SEG conducted several analyses including: 1) a construct validity study; the new measures were compared to established standardized measures and were factor analyzed to evaluate the internal consistency of the tests, and 2) a study of the accuracy of placements based on the Placement Test Scores.

Provide Systems Analysis and Vendor Management for the Implementation of an Online Instruction and Assessment Data Management System

Client: 

Red Wings Learning

The Challenge: 

Red Wings Learning offers an innovative reading comprehension curriculum, Spotlight on Comprehension, for use in schools at the grade 1 through 6 levels. Recognizing the importance of a centralized instruction and assessment management system, Red Wings Learning asked SEG to manage the process of the implementation of an online system to accompany Spotlight on Comprehension and other Red Wings Learning products.

The Solution: 

SEG provided a complete set of services to Red Wings Learning in support of the implementation of the Red Wings Learning Engage! Instructional Improvement Platform. SEG worked with RWL to develop the system specifications, review proposals from potential vendors, and to serve as the liaison to the selected vendor throughout the software development lifecycle. During the software planning, design, and development, SEG served as the main point of contact to resolve questions and concerns, to provide clarity on functional specifications, and to conduct all system testing. SEG has also developed internal training and documentation. SEG’s leadership in the software development process has allowed RWL staff to focus on further instructional program development and marketing efforts.

Provide Business Systems Analysis and Quality Assurance Services to a Certification Board

Client: 

American Board of Internal Medicine

The Challenge: 

The American Board of Internal Medicine (ABIM) has a distinguished history of providing certification services for physicians in the field of Internal Medicine and its subspecialties. ABIM utilizes multiple systems and vendors for item authoring, test publishing, test delivery, scoring, and reporting and needed assistance with systems development efforts.

The Solution: 

SEG provides ongoing services to ABIM in support of examination scoring and psychometric analyses. SEG shadows examination processing and provides support with regards to complex system enhancements. SEG also assists with the integration of third-party statistical tools into the core psychometric system. In addition, SEG provides project support as needed including system specification development, needs analysis, risk analysis, and software testing. SEG provides an independent resource to assist in the examination scoring efforts and focus on quality control.

Provide a Math Diagnostic Testing System

Client: 

Virginia Department of Education

The Challenge: 

Many students lack fundamental mathematics skills and, by middle school, encounter substantial challenges with math. Recognizing this problem, The Virginia Department of Education launched the Algebra Readiness Initiative (ARI) to provide additional mathematics instruction to students in grades 5-8. Virginia needed an accurate way to diagnose student mathematics ability and to identify student strengths and weaknesses to help guide instruction and to monitor student progress towards algebra readiness. SEG, in partnership with Internet Testing Systems (ITS), was selected through a competitive bidding process to develop and implement an Algebra Readiness Diagnostic Test (ARDT) to support the ARI.

The Solution: 

SEG and ITS responded with a complete online assessment system using both computer adaptive and traditional assessments. The ARDT system measures student math skills at the beginning and end of the instructional program and at several points during the year. The assessment system also provides grade-specific strand tests that provide a more surgical analysis of specific skill deficiencies. Detailed score reports are available to students, parents, teachers and schools immediately after testing. The ARDT provides information about student math skills on a developmental scale ranging from third to eighth grade. Because the assessment is adaptive, the ARDT provides an accurate reading of the student's skill level, regardless of what grade in which the student is enrolled. The reports provide six strand scores and indicate specific standards to be targeted for instruction.

Field Test and Evaluation of an Assessment for English Language Learners in Latin America

Client: 

Latin American Education Services

The Challenge: 

SEG was asked to evaluate the psychometric performance of a new assessment to be used for measuring the English language skills (specifically Listening, Reading, and Writing) of students and teachers in Latin America. Latin American Education Services wanted to be sure that the schools who used their assessments were receiving reliable and valid data regarding English language abilities of the student and teachers.

The Solution: 

SEG designed a field test administration to collect item performance data, and test reliability and validity data. Students and teachers were administered the field test as well as the Stanford 10 Achievement Test. The use of the SAT10 was to evaluate the validity of the new assessment. Using both classical and Item Response Theory, SEG evaluated the item performance and calibrated the test items. Using the performance data from the field test, SEG defined final test forms to be used for future administrations. Evidence for the reliability and validity of the assessment was provided in a technical report.

Update a Suite of Assessments Used as Part of an Anti-Bullying Program

Client: 

Students in the Know

The Challenge: 

Students in the Know offers an anti-bullying program for use by students in grades 4 through 8. Recognizing the importance of high quality assessment to effective instruction, SEG was asked to help develop psychometrically sound assessments that can be used as part the Bullying Academy Program. Students in the Know wanted to be sure that the schools who used their program had a reliable way to assess whether the students had learned the material after participating in the program.

The Solution: 

SEG reviewed and edited the current set of items that were used in the pre and posttests that were a part of the Bullying Academy Program. SEG then determined the content areas where additional items were needed to best cover the entire set of information included in the instructional program. SEG developed and reviewed new test questions. Field test forms were developed according to the test specifications. These forms were field tested with over 1500 students in multiple schools. Using both classical and Item Response Theory, SEG evaluated the item performance and calibrated the bank of test items. In addition, SEG analyzed timing data to investigate how quickly students were answering the items and how this related to their performance. Students in the Know implemented the new assessments as part of their Bullying Academy Program.

Evaluate the Effectiveness of an Online Algebra 1 Course

Client: 

Educational Development Center (EDC)

The Challenge: 

Many middle school and high school students may not have access to a traditional in-school Algebra 1 course or do not choose to take such a course in a traditional setting. Students are increasingly being offered the opportunity to take the course online as an alternative. EDC, a research organization, was asked to evaluate the effectiveness of an online Algebra 1 course offered to students in the Northeastern United States. EDC asked SEG and its partner Internet Testing Systems to develop and administer a mathematics assessment for use in conducting this evaluation.

The Solution: 

SEG designed and implemented a computer adaptive (CAT) mathematics assessment that could be used by EDC to measure student mathematics skills at the beginning of the project and at the conclusion of the project. The assessment provided more than a thousand four response option multiple choice questions covering the content typically encountered in Algebra 1 as well as the prerequisite knowledge and skills expected. The assessments were calibrated using a one-parameter RASCH model. The resulting 20 item test had a reliability ranging from about .80 to .90. EDC designed and conducted a scientific effectiveness study relying on the SEG assessment. A quasi-experimental design was used to investigate program effectiveness.

Evaluate a Web-Based Language Proficiency Assessment System

Client: 

Department of Defense, DELTAB

The Challenge: 

The ability to speak a language other than English is particularly important in an increasingly global society. Along with this development has come a growing need for language proficiency assessment. SEG’s client, a provider of language proficiency testing services, saw that its volume of test takers were increasing rapidly; they planned to move to a web-based delivery system to provide increased flexibility in administration. They wanted to ensure that the delivery platform was up to the task and conducted a search for an organization to review the technology. As a leading provider of online assessment services, SEG was selected to conduct the study.

The Solution: 

SEG conducted an in depth study of both the test delivery application and the technical environment supporting the application. SEG conducted numerous interviews, reviewed documentation and conducted test simulations to review the delivery application and the hosting environment. SEG provided several critical recommendations to ensure that the move to web-based testing was successful. SEG summarized the study in a written report and presented its findings to the client.

Develop Response to Intervention Tests to Accompany a K-12 Online Curriculum System

Client: 

Curriculum Advantage

The Challenge: 

Curriculum Advantage offers an innovative online curriculum, ClassWorks, for use in schools at the K-12 level. Recognizing the importance of high quality assessment to effective instruction, Curriculum Advantage asked SEG to help develop psychometrically sound brief assessments that can be used as part of Response to Intervention initiatives.

The Solution: 

SEG developed an assessment plan that provided detailed specifications for the short assessments (“probes”) which would accompany the ClassWorks system and be effective alongside current ClassWorks assessments including Universal Screeners that are used to initially screen students and determine their instructional needs and Benchmark-Progress Assessments that are used for periodic benchmarking of student proficiency and progress. Multiple field test forms of the probes for reading and math in grades K – 8 were developed according to the test specifications. The new forms were field tested with a population of more than 2,000 students in K-12 schools throughout the United States. Using both classical and Item Response Theory, SEG evaluated test and item performance and calibrated the bank of test items. Multiple parallel forms were developed and standards for proficiency were established.

Develop and Validate an Effective Reading Assessment

Client: 

National Geographic School Publishing

The Challenge: 

Student attitudes toward reading can play a significant role in the development of reading skills. SEG’s client, a large publisher of reading instructional and assessment programs, recognized the importance of student effect in the reading instructional process and asked SEG to help develop a measure of student attitudes toward reading. SEG was asked to develop and validate an effective measure of reading.

The Solution: 

SEG developed and validated an effective reading measure. SEG developed approximately 30 Likert-type statements based on the reading instructional research. Approximately 2,000 students were asked to indicate the extent to which they agreed or disagreed with each statement on a four point scale. The results were psychometrically analyzed. Only psychometrically sound items were retained. The instrument was factor analyzed to determine the dimensionality of the instrument. To validate the instrument, SEG conducted a construct validity study; the effective measure was compared to existing standardized measures of the same and different constructs.