Assessment Projects

Develop and Implement an Operational Plan for a Large-Scale Higher Education Assessment Program

The Challenge: 

Institutions of Higher Education have an increasing need to measure the value added as the result of a college education. There is increasing pressure from stakeholders in the higher education community to measure the outcomes of the collegiate experience. The Collegiate Learning Assessment provides an innovative and effective solution to this challenge. The program grew rapidly from a small research project to a high volume assessment program. The CLA sought out SEG to help plan for and respond to this growth.

The Solution: 

SEG designed and helped implement operational systems to accommodate growth in the program. SEG developed constructed response scoring systems integrating both human and electronic scoring as well as new procedures for test development. SEG also assisted in strategic planning, advising on new products, pricing and marketing.

Develop a Wellness Measure (Well Scale™) for K-12 Students

Client: 

Vitae Wellness

The Challenge: 

Obesity and poor fitness have reached epidemic proportions. Vitae Wellness recognized that proper education at an early age is the key to wellness and that effective wellness education begins with proper Vitae Wellness asked SEG to help develop psychometrically a sound measure of wellness. .

The Solution: 

SEG defined the domain of wellness with subject matter experts and developed assessment questions to correspond to the definition. The items were field tested with a population of more than 1,000 students in k-12 schools in Florida, Pennsylvania and Arizona. Using both classical and Item Response Theory, SEG evaluated test and item performance and calibrated the bank of test items. Multiple parallel forms were developed and standards for wellness were identified

Develop a Universal Screener and Benchmark/Progress Monitoring Tests to Accompany a K-12 Online Curriculum System

Client: 

Curriculum Advantage

The Challenge: 

Curriculum Advantage offers an innovative online curriculum, ClassWorks, for use in schools at the K-12 level. Recognizing the importance of high quality assessment to effective instruction, Curriculum Advantage asked SEG to help develop psychometrically assessment for use in the initial screening of students and ongoing benchmarking and progress assessment.

The Solution: 

SEG developed an assessment plan that provided detailed specifications for two types of assessments to accompany the ClassWorks system: 1) Universal Screeners that could be used to initially screen students and determine their instructional needs and 2) Benchmark-Progress Assessments used for the ongoing evaluation of student proficiency and progress. The test items included in the ClassWorks software for both types of assessments were field tested with a population of more than 2,000 students in k-12 schools throughout the United States. Using both classical and Item Response Theory, SEG evaluated test and item performance and calibrated the bank of test items. Multiple parallel forms were developed and standards for proficiency were established.

Develop a Model for Predicting Student Proficiency on Statewide Tests based on Local Benchmark Results

Client: 

Confidential

The Challenge: 

School administrators and classroom teachers are increasingly interested in predicting student performance on statewide high-stakes examinations. Obtaining information about students that are at risk as early as possible in the school year is critical to ensure that instruction in properly targeted. Moreover, with the consequences associated with failure to achieve annual yearly progress targets under NCLB, school administrators need accurate information for decision making. While schools often have local Benchmark test results available, they typically do not have an effective way to use this information. This provider of school management systems sought out SEG to help develop a model for using local Benchmark results to predict student proficiency on statewide examinations

The Solution: 

SEG worked with this client to develop a methodology for accurately predicting whether students were at risk for failing to achieve proficiency on statewide high stakes examinations. Using a combination of test linking/equating, Item Response Theory and Multiple Discriminant Analysis, SEG assisted the client in developing a predictive model.

Develop a Diagnostic Mathematics Assessment for “Stalled Students”

Client: 

Teachers for a New Era/ Florida A & M

The Challenge: 

Many Institutions of Higher Education offer developmental instruction for entering students without the necessary mathematics skills to be successful in college. However, many of the students taking developmental courses to address mathematics weaknesses are not successful after multiple attempts. In order to determine an instructional program for these “stalled students” a diagnosis of their mathematics skills is required.

The Solution: 

SEG worked with faculty and staff from the Teachers for a New Era Project to develop an intensive diagnostic mathematics assessment for use in diagnosing stalled students. This diagnostic assessment used a talk aloud method to identify the specific areas where students were encountering difficulty in solving math problems. Students were presented with a series of math problems in algebra, geometry and other math domains and asked to talk aloud as they solved the problem and respond to additional queries from trained administrators. The sessions were recorded and the areas of weakness were coded by two independent reviewers using a checklist and rating scale approach. The date were analyzed using Rasch Analysis to create a meaningful scale and to support the creation of the final instrument. The diagnostic assessment will be sued operationally beginning in the Fall of 2010.

Design and Develop an Online K-12 Assessment Platform

Client: 

America's Choice

The Challenge: 

A successful provider of online testing services was seeking to expand its presence in the K-12 market. They recognized that the technical requirements, uses of assessment and culture differed considerably from their traditional markets. SEG was called in to help with the design and implementation of the new platform.

The Solution: 

SEG helped design a new k-12 online assessment platform and participated in the ongoing development of the application. SEG advised on the technical and educational requirements and development of specifications for the new platform. SEG staff monitored the application development and reviewed interim versions of the application functionality and interface.

Design and Develop an Online Constructed-Response Scoring System

Client: 

Collegiate Learning Assessment

The Challenge: 

While the value of including written performance assessments as a basis for evaluating student skills is widely recognized, scoring those written assessments presents a challenge. This is particularly true as the volume of test takers grows. The success of the CLA presented a growing challenge for the scoring of the open ended assessments. CLA asked SEG to help automate and design an online scoring system that would allow for the scoring of large volumes of student responses.

The Solution: 

SEG assisted the CLA and their online provider in designing an online system that reduced the scoring burden, reduced turnaround time and reduced costs. The final system also provided real time monitoring of scorer performance to ensure that scorers were scoring accurately

Design an Online Consumer-Based Arithmetic Assessment

Client: 

AccountabilityWorks

The Challenge: 

Arithmetic skills are critical to later success in mathematics. Yet, many students lack the necessary arithmetic skills needed to be successful. Families are increasingly turning to resources outside the school to help address this problem. SEG's client recognized that an accurate assessment of the child's arithmetic skills is the first step in providing supplemental arithmetic instruction. SEG was asked to help address this largely untapped need by supporting the design, development and delivery of a web-based arithmetic assessment.

The Solution: 

SEG helped design an online assessment system and develop specifications for the development of the web-based system.. SEG developed an RFP and helped evaluate potential vendors. SEG staff monitored the application development and implementation to support a successful product launch.

Develop a Suite of Assessments to Accompany a Grade 1 through 6 Reading Comprehension Educational Program

Client: 

Red Wings Learning

The Challenge: 

Red Wings Learning offers an innovative reading comprehension curriculum, Spotlight on Comprehension, for use in schools at the grade 1 through 6 levels.   Recognizing the importance of high quality assessment to effective instruction, Red Wings Learning asked SEG to help develop psychometrically sound assessments that can be used as part the Spotlight on Comprehension program.

The Solution: 

SEG developed an assessment plan that provided detailed specifications for the assessments which would accompany the Spotlight on Comprehension program including (1) Diagnostic assessments that can be used in the beginning of the year to identify strengths and weaknesses, (2) Benchmark assessments that can be used periodically throughout the year for benchmarking purposes, and (3) Level assessments that are used in conjunction with the completion of instructional levels within the program.  Items were developed to measure each of the Spotlight on Comprehension core reading comprehension skills.  Test forms were developed according to the test specifications.  These forms will be field tested in a new online system at schools across the country.  Using both classical and Item Response Theory, SEG will evaluate test and item performance and calibrate the bank of test items.  SEG will develop vertical scales for the use in reporting as students progress through the years.  Standards for proficiency will be established.

Conduct a Psychometric Analysis and Validation Study for an Assessment of English Language Learners

Client: 

National Geographic School Publishing

The Challenge: 

 A growing number of students speak a language other than English as their primary language.  This increase combined with federal testing requirements has led to a rise in the assessment of English Language Learners.  SEG’s Client, a provider of English language instruction and assessment, saw this build up in ELL assessment and asked SEG to help develop sound assessments for benchmarking and end of course assessment. They needed to ensure that the tests were psychometrically sound and that the forms were both valid and statistically parallel.  They called upon SEG to conduct a psychometric study of the assessment.

The Solution: 

SEG conducted a psychometric analysis of the measures developed and conducted a validity study. Using both classical and Item Response Theory, SEG evaluated test and item performance. SEG calibrated the bank of test items using IRT and created multiple parallel forms for operational use.

In order to validate the assessments, SEG conducted a construct validity study; the new measures were compared to established standardized measures and were factor analyzed to evaluate the internal consistency of the tests.