Periodic program review and assessment provides members of an academic program with the opportunity to reflect formally on areas of instruction, scholarship, service, advising, and mentorship. It also enables the members to develop plans for the future in light of changes in internal and external conditions which affect the program. The primary purpose of periodic program review is for members to become aware of how they might enhance both the learning environment for students and their own professional development, while still contributing towards the mission of the institution. Periodic review and assessment helps to validate programmatic strengths, and helps identify those shortcomings towards which effort and resources might be allocated in the future. Review and assessment of academic programs should rely upon multiple measures, be comprehensive, be non-punitive, and prove useful to the community.

All program reviews should begin with the institution’s mission, explaining how a particular program reflects, enhances, and implements this mission. Next, programmatic mission and course or event content should be considered (how does the latter reflect and implement the former?). Thus, all courses or events designed for this program should be justified by their specific function vis-à-vis implementing the mission of the college and the operational area (as affirmed by both quantitative and qualitative assessment measures).

Fellowship Advising: The Importance of Holistic Development
The primary “outcome” of most fellowship advising programs, namely the number of awards made, is actually one of the least important aspects of assessing a scholarship enhancement or application assistance program. In this article, I have tried to identify examples of outcomes that we can use to measure the success of a fellowship advising program. The examples below are ones that we use at Marist, so they may not be appropriate or available at other institutions. Student development occurs at many levels, so outcomes can include measures for the institution (such as increased number of applicants and attendees at information sessions, and increased number of successful graduate school applications) or more individual measures—measures for the students and faculty involved in the process (such as overall student development, participation in study abroad, community service, or other programs).

Quantitative Data
Number of Applications (by category, by year)
Number of Awards (by category, by year)
Number of Students Identified as Likely Candidates (by category, by year)
• Retention Rate
• Graduation Rate
Number of Group Advising Sessions Held, with attendance figures
Number of Individual Advising Sessions Held
Number of Students Participating in Preparation Events (state purpose of event)
Faculty Attendance at Advising Sessions and Events (by department or division, attendance figures)
Student Satisfaction Surveys from Advising Sessions and Events
Alumni Survey Data (graduate school, employment, fellowship preparation)
Budgetary Allocation for Fellowship Program Initiatives (compared to other operational budgets for student support services, such as academic advising, career services, opportunity programs, athletics – indicates level of institutional support and need for services)

Qualitative Data
Identify faculty who submit nominations (culture and support)
Quality of faculty involved (measured through course evaluations and/or other evaluations of faculty
Involvement of trustees, outside stakeholders
For those identified as potential applicants, report on their plans and activities upon graduation.
• Graduate Schools (even if not awarded funding)
• Employment Opportunities (even if not awarded funding)
Identify student (applicant) participation in research projects, student publications, other peripheral achievements or activities

The Assessment Report
You should prepare a multi-dimensional process for assessing your fellowship preparation program. The various assessment components might include:
• Success in achieving annual operational plans/goals (include comparison to previous years and external benchmarks, if available)
• Progress made toward the attainment of assigned goals and objectives from the strategic plan/mission of the institution
• Feedback from faculty and staff, board of trustees and outside constituents if used
• Student satisfaction data for courses, events, or sessions used to promote programming
• Fellowship application statistics

Student Learning Outcomes: Incorporating this assessment aspect into our fellowship programs
Student learning outcomes have become a significant focus of regional accrediting bodies when considering the learning environment offered at university/college level institutions. Student learning outcomes, briefly defined, are the competencies and skills that students should have obtained upon the completion of a course, program, degree, or other activity. In the area of fellowship advising, student learning outcomes may focus on:
• Increased oral communication skills, particularly interviewing techniques
• Increased written communication skills
• Increased awareness of ethical standards, modes of behavior, and professional work environments
• Improved understanding of global events and issues

Other specific award applications might involve increased skills in research and publication, critical analysis, public service awareness, understanding of political change and environments or other discipline-specific areas, etc.

Assessment of such learning outcomes is a somewhat tricky consideration. The first step is establishing that an opportunity for acquisition of these skills/objectives exists within your program (e.g., a committee to conduct mock interviews to build oral communication skills) and then to assess whether the individual student successfully acquired these skills (e.g., improvement in interview techniques). Lastly, and most importantly, student learning outcomes should be evaluated to determine if the process for teaching these skills is adequate and serving the student population as effectively as it can (e.g., the addition of a journalism faculty member to the committee in order to improve interview technique training). In fellowship programs, this is done mostly at a co-curricular level whereas most student learning objectives are established at the individual course level. Nevertheless, individual programs can certainly be assessed from a student learning objectives perspective using the logic described above.

Strategic and Operational Planning: How to link your office to institutional goals and objectives
Planning at Marist, for instance, is composed of four linked and mutually supportive processes. These include the Mission Statement, Five-Year Strategic Plan, Annual Operational Plans, and the Process for Developing Annual Budget Priorities. The College’s Mission Statement provides the overall context that drives and directs the entire planning process. Marist College employs an annual operational planning process in which line executives, as well as selected supervisors, are asked to formulate annual goals for their areas. This process pertains to academic as well as administrative support areas. Goals are submitted to the supervisor who incorporates them into her/his annual plan. Goals are discussed at the cabinet level and/or between the appropriate vice president and the president, at which time they may be refined. Progress towards these goals is assessed at mid-year and at the end of the year, and effectiveness is taken into consideration during annual performance appraisals of the individuals charged with attaining these goals. In some areas, managers and directors submit action plans based on the “management by objectives” model to their supervisors. The “action plan” format includes identifying objectives, responsible person(s), strategies used to accomplish the action, an anticipated date of completion, and criteria for outcomes assessment. In some cases, end of year reports are compared to qualitative and/or quantitative benchmarks as a foundation for the subsequent year’s goals.

SAMPLE
The goals of the fellowship area at Marist College are designed under the umbrella of Academic Affairs Support Initiatives, and are as follows:

The Primary Goal: To mentor students and help them obtain fellowships, scholarships and entrance to quality graduate programs and/or attractive positions in high quality companies or corporations.

I. College wide support and mentorship activities – Goal: Define your planned events and activities along with their development purpose. Specify, if applicable, your goals for attendance, # of applications, etc. Assessment: For each event, record attendance, number of interested students/appointments made, survey information if administered for individual events, and other assessment materials as necessary.

Examples of recently specified goals in this category:
• Enrichment events to cultural locations, such as Hudson River Environmental Science sailing trip, a Broadway show, etc.
• Special sessions with authors, such as poet Adrienne Rich or neurologist Dr. Robert Sapolsky, for students with high achievement and interest in fellowship applications
• Award/Recognition events

II. Community service projects – Goal: Define how you will increase opportunities for community service involvement for your students, if applicable. Describe events or support networking activities with development purpose and/or population intended. Assessment: For each event or activity sponsored, indicate the amount of support (both staffing and financial) given by the fellowships office, record attendance, number of interested students/appointments made, survey information if administered for individual events, and other assessment materials as necessary.

Examples of recently specified goals in this category:
• Giftwrapping parties for Early Development Childhood Center
• Book drives for local children’s libraries
• Habitat for Humanity

III. Graduate school preparation – Goal: Define your graduate school assistance activities, plans to increase # of applicants for specific types of institutions, funding opportunities, etc. Describe any large events and purpose you will plan throughout the year for institution-wide information or exposure. Assessment: In general, prepare logs of appointments, institutions contacted, successful applicants (alumni surveys/statistics), and contact faculty department chairs and program directors at high-traffic times to survey interest level and # of applicants; prepare graduate school forum, improve profile of institutions attending, highlight college strengths.

IV. Scholarship preparation – Goal: Describe your fellowship and scholarship assistance activities, plans to increase # of applicants for specific awards, if applicable. Describe plans and purpose for informational workshops on undergraduate scholarships throughout the year. Assessment: Record attendance and follow up with interested students; keep logs on appointments with interested students; maintain contacts with national organizations, funding boards, and agencies, increase professional development and job competencies by attending workshops, conferences, etc.; work with faculty to develop application review process.

V. Faculty Outreach – Goal: Describe goals for involving faculty in fellowship and graduate school process, including specific workshops and events, if applicable. Describe your support of training and information for faculty regarding review and preparation process. Other projects might include work with faculty on writing letters of recommendation, preparing applications, identifying candidates, reviewing and interviewing. Assessment: Survey faculty involved in process both for effectiveness and possible assistance with personal professional development goals.

The Process
It is important to set feasible goals for your staffing capacity. Once the operational planning goals are defined, you will review them throughout the year. At the year end report, you will provide all supporting documentation, including statistics and reports, along with your determination of whether or not goals have been met for the year. This is also a good opportunity to analyze staffing and budget adequacies and to make requests, if possible, for improved services in the coming year.

For accreditation review years, you will sometimes be asked to supply an accompanying narrative regarding the goals and outcomes of your operational area. Because fellowship advising is a highly qualitative endeavor, this narrative will generally not be data-intensive.

An excerpted example follows:

…The Deans’ Circle Program at Marist College is designed to mentor students and help them obtain fellowships, scholarships, and placements in quality graduate programs and/or quality companies or corporations. The Deans’ Circle is in its sixth year and has recruited approximately 25-35 new students every year. Recruitment usually occurs during the freshman year, when candidates are invited to join the program based on academic strengths and leadership skills. Upperclassmen are also eligible and are invited on a rolling basis, based on the recommendation of School Deans. Currently, there are over 100 students involved in the program, and admissions have become highly competitive with many students actively seeking admission. Deans’ Circle students are invited to attend special events throughout the year. A 300% increase in the number of scholarship applications has been noted in the past three years, outcomes include the awarding of 4 Fulbright student grants, 1 Goldwater, 1 NSEP, 1 Pickering, among numerous honorable mentions, finalists (including 2 for the Truman Scholarship in 2005-2006) and other awards. Most notably, however, is the sharp increase in the number of students attending information sessions, workshops, and lectures concerning competitive applications, with nearly 250 students attending events annually (compared to ~50 students in 2002-2003) – which has indicated a desired culture shift among the student body at Marist. This has enhanced the overall profile and reputation of the institution, which in turn is at least partly responsible for the increase in the number of employers and graduate schools that are seeking a presence during forums and expositions. This program has also contributed to the strategic goal of internationalizing the campus, by promoting study abroad through awards like the Gilman and Fulbright...

Click here to view the Operational Planning Process
Click here for a sample Student Questionnaire, or download the pdf