Malcolm Gladwell & the US News Rankings
Context for Gladwell's harshly critical podcast on the rankings
The well-known writer Malcolm Gladwell teamed up with statisticians from Reed College to examine the prominent US News & World Report college rankings in a recent podcast titled Lord of the Rankings and presents a highly critical view of the rankings. While we share Gladwell’s opinion, listeners not familiar with some of the data questions may misunderstand his argument. We’ll take this as a chance to add context to this issue, an issue which is both vexed and at the same time silly.
How US News & World Report calculates the rankings
US News is quite transparent with its formula. US News’ Robert Morse and Eric Brooks outline the formula and what goes into the calculation:
Graduation & Retention rates: 22%
Undergraduate Academic Reputation (Peer Assessment): 20%. This is the element Gladwell hones in on.
Faculty Resources: 20% (class size, faculty/student ratios, faculty pay)
Financial resources per student: 10%
Graduation rate performance (historical tracking of graduation metrics): 8%
Entering students selectivity: 7%
Indebtedness of graduates: 5%
Pell Grants recipients performance: 5%
Giving by alumni: 3%
Readers familiar with higher ed data will immediately see the cross relationships and note how almost all tie back to selectivity and the socioeconomic base of a college’s students. Before we isolate the issues here, note that really the only element that actually is related to academic performance - how well a college educates its students - is assembled through the Undergraduate Academic Reputation survey, which Gladwell tears into. All other measures are indirect proxy measurements. And to use a stats term, they are confounded - the different variables are related and interact.
Besides being proxies, the elements are biased:
Graduation & Retention rates: Favors selective schools with well-prepared students and those with a higher proportion of full-time students, both positively related to student socioeconomic levels.
Faculty Resources: Favors schools with large resources, rich endowments and higher prices. All related to socioeconomic levels of incoming students & graduates.
Financial resources per student: Favors schools with large resources, rich endowments and higher tuition. All related to socioeconomic levels of incoming students & graduates.
“Graduation rate performance”: Tracks change in the graduation metric, basically. This is one metric that in practice favors less selective schools which have been improving their graduation rates.
Student preparation/selectivity: Admissions rates as of 2021 did NOT directly factor into the rankings. SAT/ACT and class rank made up the bulk of this element, which must present a major data problem for US News right now given the test optional shift.
Indebtedness of graduates: Favors socioeconomic levels of incoming students.
Giving by alumni: Favors socioeconomic levels of graduates.
The broad fundamental drivers of the rankings:
Summary: 67% of the US News rankings results from selectivity and financial resources measures, all well-correlated with the socioeconomic levels of incoming students and graduates of the school.
What does “best” mean?
US News labels its rankings as "the “best” colleges. That word ”best” is part of a motte-and-bailey act. Parents and students will often interpret it to mean the schools offering the best education. But looking at the analytic factors, “best” looks more like “highest socioeconomic status.” The most cynical readers of the rankings probably have a more accurate view than the more casual, ordinary readers: it’s a proxy for class status.
Gladwell and the Peer Assessment
The one element of the US News formula that claims to measure “Undergraduate Academic Reputation” is the Peer Assessment. US News is forthright about what is being measured: Reputation. The issue is that the Peer Assessment can’t possibly measure reputation well and that “reputation” isn’t equivalent to “academic quality.” As the podcast lasts over 40 minutes with some fluff, summarizing Gladwell’s points is worthwhile, because he makes substantive criticisms of the Peer Assessment.
Students working with statistician Kelly McConville at Reed College in Oregon reverse engineered US News rankings to identify drivers of the 2019 scores. The reverse engineering reveals that the Peer Assessment correlates well with the index overall, about 2/3, a solid social science relationship. (The Reed student interviewed about the correlation isn’t exact but the figure is around 0.65.) We showed above how 67% of the rankings are very dependent on student socio-economic drivers. But after that 67%, the second largest driver is correlated with the other elements. More confounding is at work - almost all the different contributing factors in the rankings are related.
The head of data at US News & World Report explains to Gladwell how the reputational survey is conducted: 3 administrators at all US colleges - the president, provost and head of enrollment - are emailed and asked to provide a numerical ranking of academic quality at comparable undergraduate programs. These comparable institutions can number in the hundreds and include schools of all sizes across the country. Gladwell is astounded by this, commenting that in his naïveté he had expected ”some sort of rigor” from the survey.
“Some sort of rigor”: US News assigns a 17.6% weighting to a college’s graduation rate (over 6 years). Sure that’s not 17.4% or 17.8%, people? The justification for the weightings is unclear.
Gladwell has great fun exposing the reputational survey and how it “disguises mythology as methodology”.
Noting that Gonzaga in Washington, BYU in Utah and Yeshiva in New York are three denominational schools which have very similar rankings, he asks how an administrator at one of the schools is supposed to accurately evaluate the other two schools’ academics, given their distance and dissimilarity.
He interviews an enrollment head at a prominent school - both the person and school remain anonymous to protect the innocent - who freely admits that he has no idea how to rank even large comparable institutions. One example they use is Syracuse University - the enrollment head had coffee there once and says that he is more qualified to vote on Major League Baseball’s Hall of Fame entrants than on the US News academic assessment.
The fact that this enrollment head feels the need to remain anonymous is a testament to the power US News’ rankings have and shows he doesn’t believe the rankings are put together strictly objectively. If the rankings were just pure calculations from data such as faculty salaries, he could say whatever he wants and it would have no impact. He feels he can’t speak freely, ergo…
Gladwell says that, for the college administrators being surveyed, the only real source of information for these rankings are prior years’ rankings.
Gladwell is a big fan of Rowan University in New Jersey, which he thinks is a wonderful school but one ranked low in the US News rankings. (We had earlier singled out Rowan for its excellent retention scores, suggesting that Gladwell has good reasons for his view.) Gladwell asks the president of Rowan, Ali Houshmand, how many college presidents have visited Rowan in recent years. Houshmand can think of only three, and one of them came from abroad. Houshmand reveals that as part of his campaign to raise Rowan’s US News rankings, he had sent each and every college president in the US a sample of his own home-brewed hot sauce. Revelry ensues.
Reed College stopped supplying US News with data for its rankings, the only US college to do so. In the wake of Reed’s decision, the reverse engineering conducted by Prof. McConville and her students, Huaying Qiu and Lauren Rabe, succeeded in closely replicating the rankings of each US school — with the exception of Reed. They calculated Reed’s ranking as #36 (all schools); US News & World Report ranked Reed as #90 in 2019. The size of the difference suggests Reed was punished for noncompliance, adding credibility to the anonymous enrollment head’s behavior.
Gladwell then introduces the subject of his next podcast on higher ed, Dillard University, an HBCU in Louisiana, to examine how implicit bias affects reputation. Gladwell says that the US News survey methodology represents a good example of implicit bias (although a mostly un-racialized one).
The threadbare and unconvincing approach used to compile the Peer Assessment is exposed fully by an unintimidated Gladwell. College enrollment offices as we are aware take elaborate steps to optimize their rankings, including encouraging students with little chance of admission to apply, wasting everyone’s time and money. The impact of selectivity though is statistically minor compared to the Petri dish of US News’ reputational survey, a survey which Gladwell persuasively argues is very flawed.
If you don’t know by now whether to laugh or to cry about the rankings, you’re not alone.
See also Talia Schaffer’s wise take on Gladwell’s podcasts.
Read this post and others at the CTAS site.