Matthew W. Ohland, Clemson University, workshop facilitator

Demonstration of model using Clemson's FIRST CLASS learning community as an example

The FIRST CLASS learning community at Clemson University
The goal of the First-year Community for Learning And Student Success (FIRST CLASS) project at Clemson University was to create a learning community of engineering students. By arranging for a common residence facility and common sections of certain courses, FIRST CLASS intends to improve the persistence of engineering students. Further information about the FIRST CLASS program can be found at http://www.clemson.edu/FIRSTCLASS/

Step 1 Example: the approach to the evaluation of the Clemson FIRST CLASS project
When designing the FIRST CLASS project, the investigators addressed the “Why Evaluate” question by organizing their evaluation activities into three stages. In stage one, the evaluation was designed to identify which elements of the intervention were contributing to the development of the desired characteristics, providing information that could be used to improve the implementation of the program and to improve its efficacy. In stage two, evaluation efforts focus on determining whether the improved implementation of the learning community is leading to the expected improvements in learning and in retaining the students as engineers. Since learning communities are widely found to be beneficial, it is anticipated that after sufficient formative evaluation, the results from stage two will indicate scaling up the FIRST CLASS program, at which point stage three evaluation will begin in order to confirm the program’s effectiveness on when implemented on a larger scale.

Step 2 Example: sample objectives of the Clemson FIRST CLASS project

Impact Objectives
• Increase the proportion of engineering majors participating in the FIRST CLASS program at Clemson who graduate from 50% (recent historical average) to 75%.
• Increase the proportion of students enrolled in Calculus I receiving a passing grade from 50% in 1999 to 70% in 2001.

Outcome Objectives
• At least 25% of participants will be in study groups with members of their FIRST CLASS cohort at the end of the sophomore year (one year after program ends).
• Increase student confidence in engineering from 3 to 4 on a 5-point Likert scale
• At least 75% of participants will have a favorable opinion of group work at the end of the freshman year (when the program ends)

Process Objectives
• Secure dormitory and classroom space for FIRST CLASS participants
• Enroll approximately 100 participants in the FIRST CLASS program
• Conduct studies to investigate the efficacy and effectiveness of FIRST CLASS, and improve the program based on the results of the investigations.

Step 3 Example: model of change for the Clemson FIRST CLASS project
A. Students who share a common residence and classes will tend to affiliate with one another;
B. Students who develop affiliations receive support in times of academic difficulty
C. Timely support leads to improved performance on course quizzes and course grades
D. Improved performance in freshman courses will lead to improved performance in later courses building on the same concepts
E. Improved course performance will lead to an increased number of students who graduate in an engineering discipline.

Step 4 Example: selected criteria and indicators used in Clemson’s FIRST CLASS project
• Graduation rate (Impact)
o Number of FIRST CLASS students who graduate DIVIDED BY
o Number of FIRST CLASS students
• Calculus I pass rate (Impact)
o Number of FIRST CLASS students who pass Calculus I DIVIDED BY
o Number of FIRST CLASS students enrolled in Calculus I
• Study group affiliation (Outcome)
o Number of FIRST CLASS students in a study group with other
FIRST CLASS students at the end of the sophomore year DIVIDED BY
o Number of FIRST CLASS students
• Dormitory and classroom space for FIRST CLASS participants secured (Process)
• Number of students enrolled in the FIRST CLASS program (Process)
• Total direct costs (excluding developmental costs) of FIRST CLASS per student contact hour

Step 5 Example: sample data sources and periodicity for Clemson’s FIRST CLASS project
The key to successful evaluation is careful organization. PT project investigators identified the data they would need to measure each of their priority indicators, and the source from which that data could be obtained. "Project records" reflects the need for a careful management information system that includes the data points needed for all monitoring and evaluation. Special evaluation studies require additional data bases, but the summary results should always be maintained in project records. As is clear in the table below, tracking (by name or ID number) of individual students who have participated in PT instruction will be essential for later follow-up studies of graduation and retention rates.

Indicator Data Source(s) Frequency
Graduation rate Institutional Research / Project records Annual
Calculus performance Math department / Project records End of semester
Overall GPR Institutional Research / Project records End of semester
Dorm space allocated Housing / Project records Monitor until achieved

Step 6 Example: using qualitative interviews in Clemson’s FIRST CLASS project
After the first semester of the FIRST CLASS program, confidential (but not anonymous) surveys were used to identify both issues of interest and to select those that would be interviewed for further information. Individuals were interviewed by a project investigator. Results obtained through the interviews were used to modify the FIRST CLASS program, as well as the way in which it was presented to prospective participants and their families. Examples of findings and resulting modifications include

• FIRST CLASS participants had inferred that the program would provide them access to special resources beyond the scope of the program. As a result, the way in which the program was advertised was modified for the 2001-2002 academic year.
• A common reason offered for why students in the FIRST CLASS program did not make use of certain standard Clemson University resources was, “I was lazy.” As surprising as we find the honesty of this response, we are considering various approaches to motivating attendance.

Step 6 Example: sample study design used in Clemson’s FIRST CLASS project

Efficacy Evaluation of the FIRST CLASS program
The purpose of this study was to determine whether participation in the FIRST CLASS program was associated with improved performance in pre-engineering courses. The study uses a non-equivalent control group design. After a brief explanation of the FIRST CLASS program sent as a brochure by mail, 110 of the students accepted to the General Engineering program at Clemson volunteered to participate (approximately 750 students matriculated out of 1500 accepted). To test selection bias, we will test for significant differences in Predicted Grade Point Ratio, which includes SAT scores, high school grade point average, high school rank, and high school quality as factors. The dependent measures include grades in Calculus I and other courses as well as overall grade point ratio.