Evaluation

BART IGERT Evaluation Tools

In the spirit of collaboration, we offer the following information about efforts to evaluate the BART IGERT program.  Below, you will find descriptions of the evaluation activities as well as sample surveys and interviews.  Most of these have been developed and implemented by me, as external evaluator for the program.  Some tools have already been modified or shelved in the Archives section, as both the program and our thoughts about the evaluation needed to assess it continually evolve.

I encourage questions, conversations, and feedback at any point.  Until then, best wishes in the work.

Kristin N. Kusmierek    
BART IGERT Program Evaluator  
kriskus@umich.edu    

EVALUATION EXPERIENCE FOR UNDERGRADUATES (EEU)

In 2006-2007, the BART IGERT program implemented a small pilot program—an Evaluation Experience for Undergraduates (EEU)—to introduce STEM undergraduates to professional program evaluation activities. This capacity-building pilot was funded by a supplemental grant from the National Science Foundation and supported two undergraduate engineering students as interns to BART’s professional program evaluator. The internship experience included readings about and seminar-style discussions of evaluation topics, mentoring with a professional evaluator, informational interviews with professional evaluators from varying fields, and hands-on activities within the context of BART’s ongoing program evaluation.

EEU Program Announcement

EEU Completion Survey

BACKGROUND INFORMATION

Evaluation Overview
Describes the goals of the evaluation, methods used, ethical assumptions, and the evaluator's background.  I use this document when I introduce myself and explain the program evaluation to faculty and students new to the program. 

Schedule of Evaluation Activities
Provides a chronology and brief description of evaluation activities.

General Consent Form
Although this evaluation research has been deemed “exempt” from annual review by the Institutional Review Board (IRB), I use this general consent form to facilitate the granting of consent through signature, particularly in case the program would like to use the evaluation data to address any external audience. An additional explanatory consent document (no signature required) accompanies most surveys.

EVALUATION TOOLS

Introductory Survey
Used to collect information from incoming students within the first few days of formal participation in the program.

Workshop Evaluation Form
A brief online form, allowing participants to provide feedback on program workshops, courses, and other activities. A generic standard can be tailored to suit the activity or presenter’s interests. The system is password-protected, and respondents are prompted to reply to those activities they may have attended.

End-of-Summer Survey (first-year sample and second-year sample)
Written survey to collect information about students’ perceptions of their experiences in the program, including the curriculum, research, mentoring, and evolving relationships.

Graduates' Interview Guide
Used to conduct one-on-one interviews with former BART students, closely following doctoral degree attainment. Topics include graduate experience, program impacts, and career and employment pursuits and plans.

Faculty-of-Graduates Interview Guide
Used to conduct one-on-one interviews with the mentors of former BART students, closely following the students’ graduation from their doctoral programs. Topics include perceptions of student experience, program impact, mentoring, and students’ degree of interdisciplinarity.

Update Survey
A very brief written survey, emailed annually, to track all students’ whereabouts and relevant activities.

Departure Survey/Interview
Used as either a written survey or face-to-face interview tool to collect information from students who withdraw prematurely from the program.

SURVEY ARCHIVES

These surveys were created and implemented at some point, but are no longer used by the BART program. As the program and evaluation have evolved, some surveys have become less relevant or useful—and I have become increasingly sensitive to the burden of evaluation on respondents. Thus, despite innumerable ongoing questions, the questioning at some point must cease.

Academic Year Experience Survey
A written survey distributed by email to rising second-year student participants, exploring the student experience of the preceding academic year at home institutions and expectations for the second year of BART participation on-site.

Faculty Survey
An intermittent survey, used to collect targeted information from the faculty who participate in numerous ways and to varying extents in the program.