The National Research Council's Assessment of Research Doctorate Programs

September 27, 2010

The National Research Council (NRC) will release its third Assessment of Research Doctorate programs in the United States on September 28, 2010 at http://www.nap.edu/rdp. This study is intended to provide a more comprehensive assessment of doctoral program quality than can be found in the popular ranking systems available today. To coincide with the release, NRC has scheduled a webcast to be aired at noon (Iowa time) on the 28th at http://national-academies.org.

The NRC data came from over 5,000 doctoral programs in 62 fields from 212 universities, and were gathered during the 2005-06 academic year. Information was submitted to NRC in 2007. Based on the criteria and taxonomy established by NRC, 50 of the University of Iowa’s PhD programs participated in the assessment.

The data report on 20 variables categorized under three “dimensions” — 1) research activity, 2) student support and outcomes, and 3) diversity of the academic environment. The variables include such things as faculty publications, student funding, completion, time to degree, and diversity of faculty and students.

A range of rankings has been produced based on the following:

  1. S-rankings - Quantified, weighted measures based on faculty selections of variables they felt most important to the quality of a doctoral program.
  2. R-rankings – Faculty ratings for a sample of programs in their field which were then related to the 20 program variables through a regression.
  3. D-rankings – Based on the three dimensions of doctoral programs:
    1. Research activity
    2. Student support and outcomes
    3. Diversity of the academic environment.

Initially, as reported in the Methodology Guide, the first two methods (S and R) of obtaining weights for program characteristics were combined to give an overall ranking. However, after reviewing the results, NRC decided to provide ranges of rankings for each of these individually.

Rankings of programs are not tied to literal positions, but rather to broad ranges. For example, the NRC assessment does not identify one program as #1 and another as #2. Instead, it places each program within a relative range of rankings based on a 90% confidence level. For example, the report might say that there is a 90 percent chance that a program is between 15th and 35th best in the nation. This avoids what the project committee calls a “spurious precision problem” that could falsely offer an exact ranking of one program in relation to others.

Data also are available in downloadable Excel spreadsheets. Users of the NRC data can thus select measures based on their own values about “excellent” graduate programs to produce customized ranking results or comparisons.

Because the data are somewhat dated, some changes may already have taken place that would affect actual rankings. The value of the NRC data is therefore less related to the rankings and more to the ability to compare programs over several variables key to a specific discipline.

It is important to understand that while we are appreciative of the NRC assessment, this information should be reviewed in the proper context, and its value may be limited. This view is based, in part, on a number of factors. For example:

  • Much of the information gathered for the assessment focuses on data for 2005-06. Thus the information is a “snapshot” in time and is not reflective of program changes since that time.
  • Since the submission of the data, there has been an 18% change in the faculty included in the assessment. Changes in program faculty vary from under 5% to greater than 30% in some cases. Thus, the reputation of the program faculty since 2005-06 has likely changed as well.
  • Because of recommendations made by the Graduate Education Task Force or by individual colleges, some actions are already being taken to restructure or phase out some programs.
  • Graduate programs are benefiting from sharing “best practices.” These efforts, which started in earnest a number of years ago, are beginning to show benefits. This year, every strategic indicator target for graduate education was met or was exceeded in the annual report to the Board of Regents. Improvements include increased financial aid, shortened time to degree, and increased attainment of national awards and recognitions.
  • University of Iowa graduate students have been awarded five national dissertation prizes from the Council of Graduate Schools, more than any other institution in the United States. Of those five, two were awarded in the past 3 years and another student was recognized as a finalist.

As with any rating or ranking system, there are benefits and flaws to the study. The NRC assessment is no different. The Graduate College encourages you to explore the information that is available to gain further perspectives on the assessment and its place in understanding the research doctoral programs that participated in the assessment from The University of Iowa.

Questions/suggestions may be directed to John Keller or Caroline Mast.

Quick Links