Assessment can often be a challenge for scholarship foundations. To appear neutral, we must remain somewhat removed from our applicant pool. Those students to whom we have the most access, our scholarship recipients, tend to be understandably biased in favor of the current processes. Constraints of staff time and resources can also dictate both the ability and depth with which foundations can conduct assessments.

These difficulties notwithstanding, self-assessments are critical to provide an equitable and valuable experience for every participant in the scholarship process. Meeting the challenges of assessment while attempting to provide a valuable process is the subject of this piece.

The Scholar Perspective
The Truman Foundation is fortunate in that fully one-half of the staff are former Truman Scholars. Our Executive Secretary is a member of the first class of Truman Scholars, selected in 1977. The Deputy Executive Secretary, the author of this piece, was selected as a Truman Scholar in 1993. Our Program Manager was selected as a Scholar in 2004.

While simple mathematics highlights a separate problem (a full staff of only six), staff members who have gone through the scholarship process provide a unique assessment opportunity. These Scholars give perspective not only on the selection cycle but also on the way that the Scholarship has developed over the years.

THE CLASS OF 1977

In the first year of the Truman competition, the Foundation selected 53 Scholars – one from each state and one each from the District of Columbia, Puerto Rico, and American Samoa. Students were selected as sophomores in college. Approximately 10% of those students selected were enrolled at community colleges at the time of selection.

We do not have a record of the proposed career interests of this first class of Scholars. But a quick view of those Scholars with whom we still have contact shows a list of educational professionals, attorneys, civil servants, and one governor. Indeed, the original plan to promote the scholarship centered around departments or programs of political science, history, education, and pre-law. The memos and other documents that surround the first years of the Foundation show that early recruitment efforts were rather narrowly tailored around the idea of government service.

With the narrowly tailored recruitment strategy, it was fairly simple for the Executive Secretary to travel to each nominating institution to discuss the program with the Faculty Representative. Through these discussions, the Foundation was able to gather first-hand assessments of our program. Early Foundation documents make reference to a variety of Faculty Representative concerns that are still present today. During a 1976 visit to California, our Executive Secretary noted that many Faculty Representatives were concerned that they would not have enough time to recruit for the Truman in addition to their other duties.

Using input from Faculty Representatives, the Foundation began to adapt its program to meet the needs of students. After receiving input that public service should be more broadly defined, the Foundation realized that changes would have to be made. The Foundation began recruiting from a wider variety of departments and institutions. As increasing interest in the program rendered this level of personal attention untenable, the Foundation began to rely on more standardized methods of assessment.

THE CLASS OF 1993

In 1993, the Foundation selected 83 students as Truman Scholars. These students were selected as juniors. Six of the 83 students were selected directly from a community college or other two-year institution. Several others were transfer students from community colleges nominated by their new institution. The total number of transfer students for the class of 1993 was around 10%.

The proposed majors of this class were somewhat broader than their predecessors: There were a number of students interested in environmental sciences, public health, and one prescient individual who wanted to study the Middle East. The overall class breakdown was as follows:

International Affairs 15%
Public Health 13%
Education 13%
Human Rights 12%
Environment 12%
Development 11%
Urban/Domestic Affairs 10%
Children/Youth Services 8%
Other 6%

The Foundation had already instituted a series of annual assessments, most geared toward understanding the individual candidate’s impressions of the selection process. During this time, the Foundation began to realize the emergence of several “new types” of candidates – students with hard science backgrounds, students interested in public health, and students with increasingly varied academic backgrounds. The Foundation responded by instituting additional assessments to ensure both that these students were fairly evaluated by our process and that these students found the process valuable.

The Class of 2007

In 2007, the Foundation selected 65 students as Truman Scholars. No Scholar was nominated directly by a two-year institution, but approximately 10% were transfer students at the time of selection.
Although the proposed majors of this class are similar to those of the 1993 class – international affairs, the environment, development and public health all make a strong showing – at first glance, the breakdown of majors is dramatically different from the 1993 class:

International Affairs 14% (v1%)
Public Health 31% (u18%)
Education 15% (u2%)
Human Rights 7% (v5%)
Environment 9% (v3%)
Development 12% (u1%)
Urban/Domestic Affairs 8% (v2%)
Children/Youth Services 2% (v6%)
Other 3% (v3%)

The most significant change is in the area of public health – a dramatic 18% increase over the 1993 class. This breakdown is also representative of the 2007 applicant pool – a pool that saw a dramatic increase in applications from students with a stated interest in public health.

In contrast, these figures, like those for our applicant pool, potentially show a decreased interest in issues relating to Children/Youth Services, Human Rights and the Environment.

Initially, these numbers caused some concern around the Foundation. Such a dramatic change in our applicant pool would necessitate a number of adjustments to both the selection cycle and Foundation programs. We would, for example, need to dramatically change the make up of our reviewing panels to ensure adequate representation by individuals who were able to evaluate these applications. Additionally, we would need to make adjustments to many of our programs to assure that these Scholars received appropriate mentorship, information regarding graduate school, and internship opportunities.

But these numbers present an incomplete picture. A quick review of the applications of those students who declared an interest in public health shows that many students view public health as a means to an end. A quarter of the public health Scholars want to focus on issues that relate to children’s health; a third want to focus on health education; a number of the others have a strong focus on health disparity as well as the environmental causes of public health problems. Quite simply, these Scholars do not fit neatly into one box.

Assessment Outside the Box
The class of 2007 is not the first time the Foundation has had to adapt to a change in student interest – or at least student vernacular. The mid-1990s saw similar spikes in environmental studies and public health, to correspond with the Earth Day movement and the AIDS crisis, respectively. Likewise, Scholar classes selected in the immediate aftermath of 9/11 had unusually high proportions of Scholars interested in international affairs.

The Foundation has also adapted to more subtle shifts in student interest. The mid-1990s saw several Scholars interested in deficit reduction, a topic that has generated considerably less interest of late. Mid-millennial classes have included a number of students interested in micro-finance and venture-philanthropy, topics unheard of just ten years ago. While these Scholars might not be as numerous as those interested in public health, such changes in the Scholar class do require adjustments by the Foundation.

Just how to respond to these shifts, without either creating new problems for those Scholars with more traditional interests or over-correcting for a temporary change, is determined through a variety of assessment methods employed by the Foundation.

The Goals of Assessment
The Foundation uses both formal and informal assessment tools throughout the year. These tools help the Foundation to determine how well we are meeting the following goals:
1. Having an equitable and unbiased selection process;
2. Selecting the strongest applications as Finalists and selecting the strongest interviews as Scholars;
3. Ensuring a worthwhile pedagogical exercise for all students no matter the outcome;
4. Encouraging all participants to continue in their stated public service paths;
5. Working with Faculty Representatives to both ensure their comfort with our process as well as encourage public service on their campus; and
6. Developing a simple and seamless application process for all participants.
The first goal of an equitable selection process must come before anything else; we cannot expect to be able to identify the best candidates with a flawed process. We define this goal quite broadly. We are concerned not only with eliminating the more traditional notions of bias based on gender, race, sexual identity or economics but also eliminating any bias that might exist either for or against private institutions, certain majors, and political philosophies.

With this paramount goal in mind, the remaining goals are much easier to accomplish and often shift in terms of priority. When the Foundation made the shift from selecting sophomores to selecting juniors in the late 1980s and early 1990s, we focused quite heavily on the quality of the applications we were receiving as well as the worth of the application as a pedagogical exercise. We determined that for most students, allowing the application to be made a year later would not only increase the value of the application process but also result in a higher quality application. When we developed the on-line application in 2005, we were far more concerned both with the comfort of Faculty Representatives and the simplicity of the process. Our assessment of the on-line process centered on these goals.

Outside of special projects, however, we often tend to focus these goals generally and attempt to give consideration to each. Our method for doing so is a battery of annual self-assessments, some formal and some informal.

Formal Assessments
Perhaps it is our public policy background showing, but the Foundation loves a well-crafted survey. We have had a number of instruments in place to evaluate our programs since the 1980s. The main instruments for the selection process are a survey of all applicants and a survey of Faculty Representatives.

Both surveys are important as baseline tools to evaluate the selection process. We inquire about the student’s reasons for applying, the amount of time he or she spent on the application, how he or she became aware of the program, and whether the student has any suggestions for the process in the future. We review the survey results thoroughly each year and make an effort to post the results on our website.

These results have alerted us to a number of issues with the application process that might have gone unnoticed. Students were the first to point out a lack of clarity in the on-line application instructions that led to a number of unsatisfactory responses to our question regarding Summer Institute participation. Suggestions from students helped to shape much of the format of our current website so that they might better access information.

The Faculty Representative survey responses were also helpful in developing our current website. Faculty Representatives also volunteered to participate in a beta test of an early version of the on-line application that allowed the Foundation to troubleshoot a number of issues prior to launching the site.

These surveys are not without their limitations. We often have difficulty getting responses, particularly from Faculty Representatives. We also find that some of the open-ended questions tend to engender responses that are more related to the school competition than our national level competition. The main limitation, however, is that the information obtained from the survey is quite general.

To obtain more specific information about our applicant pool, we also conduct formal assessments during our Finalist Selection Committee. When our Finalist Selection Committee meets to read the applications, we begin with a formal calibration process designed to ensure fairness in selection. At that calibration, we alert the committee to any trends that we have noticed in the applications. After the weekend, we formally debrief the Finalist Selection Committee to discuss both the process as well as any observed trends in the application.

This formal debriefing process often yields specific information regarding our applicants. From this process we learn which states or regions had a strong showing, which institutions had difficulty adequately preparing applicants, and even which policy topics are popular in a given year. One of the most significant changes to come from this debriefing began three years ago with feedback from several readers regarding the relative emphasis of GPA in our scoring forms. These readers felt that our current scoring form allowed students with very strong GPAs to be advanced as finalists over those students who had superior records of public service with less stellar GPAs. The Foundation agreed that such an emphasis on GPA over service was inconsistent with the Foundation’s overall mission and the score sheets were changed.
The Foundation also has formal debriefing sessions with the staff several times a year to evaluate the selection process from a practical standpoint. These assessments allow us to notice problems in the application process and, hopefully, deal with them before they arise. While these issues may not be the most glamorous of the cycle, problems with mail submission, formatting, and notification can often be the most frustrating of the cycle. If one of the Foundation’s goals is to provide a valuable experience for all applicants, we cannot achieve that goal with a difficult submission process.

Our last area of formal assessment might come as a bit of a surprise, our feedback to Faculty Representatives regarding non-selected nominees. Providing feedback is a formal part of our program, and we keep track of the feedback we provide. We use this information to determine where there may be problems in our application or our instructions. We are also mindful of particular issues that may arise at different institutions or in different regions. Based on the feedback that we provided this year, we are making significant changes to clarify the guidance on our website for our leadership questions. We have also adapted our FAQs to include additional information that frequently comes up in our feedback discussions.

Informal Processes
The Foundation receives a good deal of information through informal assessment channels during the year. We receive a number of website inquires, phone calls, emails and the occasional letter in response to our application process. We also spend a good deal of time working with our Scholar network and taking advantage of the strengths of our staff. While these items may not seem like traditional assessment methods, the information that results from these channels is every bit as valuable as those gained by traditional assessment instruments.

These instant assessments require a bit of finesse to interpret. Because of the instant nature of the internet, individuals may provide feedback to the Foundation without thoroughly considering their message. These messages can range from incoherent to outright hostility, but the burden falls to the Foundation to evaluate each one and determine how best to address the concerns of the writer.

The benefit of this informal assessment is that we can be alerted immediately to problems in our application process and fix them quickly. Keeping these channels open and monitored allow us to both adapt to changes and respond accordingly. Since the nature of this feedback is often very specific, these assessments allow us to see a level of detail that might otherwise be overlooked in a larger, more formal assessment.
We also make good use of our Scholar network in conducting assessments and responding to suggestions. Since our classes of Scholars are relatively small and we spent a fair amount of time with each Scholar, we do hear a good deal about our process from our Scholars. A number of our Scholars have gone on to serve as Faculty Representatives. Several Scholars volunteer with the Truman programs at their current or former institutions. While we do not conduct formal assessments of these Scholars, our informal assessments provide us with useful information.

The 2007 class of Truman Scholars were a wonderful example of this type of information assessment. When we looked over our final list of Scholars, we realized that the program for our Truman Scholars Leadership Week would need to be changed; we had not planned to have a third of all Scholars interested in public health. Based on informal feedback that we received from both our newly selected Scholars as well as alumni, we were able to better understand the needs of these students and adapt our program to better suit their interests. While we were not able to get a perfect fit in time, we were able to call upon a number of our Scholar alumni with similar interests to participate in the program as mentors.

Finally, the Foundation staff conducts a number of informal assessment activities throughout the year. The staff spends a great deal of time staying aware of developments on college campuses and in youth culture. Understanding the importance of Second Life or realizing how much time is required to fully manage an on-line protest are critical to our application process. While the staff is multi-generational, an effort is made to “think young” to better understand the temperament of our applicants.
Does Assessment Practice Make Assessment Perfect?

While the Foundation has invested a good deal of time and thought into both creating these assessment instruments and managing the data they create, the need to reevaluate our assessment strategy is constant. Even if we become comfortable with our assessment strategy, there is always room to improve how we use the data we collect. It is axiomatic that we will never please everyone involved in our process, but in all fairness, we are our own harshest critics.

Perhaps the nature of the work we do – identifying future public service leaders – is too important for us not to worry over every detail. Maybe our mission statement and commitment to service attracts a staff who shares those ideals. But it is just as likely that the three Scholars who work for the Foundation are too invested in the process not to strive for perfection. While Scholar interests might change from year to year, the values of service, excellence, and leadership never change.