Ezra Klein: Samuelson vs. The Rankings

« Racist | Main | The GOP and World War III »

June 28, 2007

Samuelson vs. The Rankings

"What's so shameful about this campaign against the [US News and World Report college] rankings," Robert Samuelson writes, "is its anti-intellectualism." No. What's so shameful is the Samuelson column is its anti-intellectualism, and speed to recast a serious attack against an idiosyncratic, toweringly powerful rankings system as a mere example of sour grapes.

Does Samuelson think it relevant, for instance, how much money alumni give to the university every year? Are financial resources per student really so telling? Should US News be allowed to simply make up SAT scores for non-participating schools? We don't know. Not only does Samuelson neglect to engage such critiques, he never even mentions them. Instead, we get this cunning bit of divination: "Unsurprisingly, many complaining schools don't rank high. Some seem further down the list of colleges than their old-line reputations imply. Barnard is at 26; Kenyon at 32."

Unsurprisingly indeed. Sadly, Samuelson never seems to realize that this is utterly unilluminating. If poorly-ranked schools lose their authority to complain, so too do highly-ranked institutions forfeit the credibility to defend. Indeed, given the self-interest of all the actors involved, the only viable method of settling the dispute would be to examine the actual claims being made. Which Samuelson never does. What we do learn, though, is that "Brian Kelly, U.S. News's editor, is a friend," and also that students of the non-participating schools "will learn...a life lesson in cynicism: how eminent authorities cloak their self-interest in high-sounding, deceptive rhetoric."

You don't say.

June 28, 2007 in Education | Permalink

Comments

Yes, how much alumni support is absolutely relevant to to students. I don't know about it being anti-intellectual, but it certainly is something that can tell you about the alumni network available to you to know that they are donating money and supporting the school. That means they are the people you will have a better shoot of getting a job with when trying to pay off loan debt.You may be right about Samuelson, but the info is needed, and that's only one example of how students should be readingg the tea leaves of whether a school will be of any use to them or not.

Posted by: akaison | Jun 28, 2007 3:31:21 PM

Ezra,

I think you're barking up the wrong tree-- and it actually relates to the cost-effectiveness in health care discussion in an odd way.

I've discussed the use of cost-effectiveness in health with a number of government officials and industry executives-- those who are uncomfortable with the data have a similar criticism as the US college rankings-- the give an artificially precise answer because of their quantitative nature, the underlying data has a number of important gaps and they question the methodology.

My response to them-- and to you-- is that the "rankings" are happening in people's minds anyways and to rule out the use of quantitative measures just ensures the process will be even less exact than would otherwise be the case. Attempts in determining the right algorithms for rankings are actually helpful in itself-- your thought on the relevance of almuni donation rates is an interesting one-- but is a thought that would be less likely discussed if the rankings algorithm didn't include it.

Hence, future rankings could be better than current ones. The college presidents aren't looking to improve them (I recognize US News may not either), they are looking to get rid of them. I think that's Samuelson's anti-intellectualism point.

I think there are two other points that bear resemblance to the cost-effectiveness analogy. The first is that quantitative measures more easily allow for unexpected outcomes-- i.e. schools with less traditional reputations like Pomona-- there are similar examples among treatments as well. In a pure qualitative process, these "diamonds in the rough" more typically get lost.

The other point is that quantitative rankings should not be the end-all/be-all in decision-making. They are a helpful input into a larger discussion-- but to make decisions based purely on the quantitative assessments is foolish. (Something I've heard directly from experts that you'd love to staff your cost-effectiveness institute concept) I think Sameulson's three lessons based on his experience fit that bill as well.

Overall, I'd say his rationale is similar to those who do want quantitative cost-effectiveness assessments in health-care: recognize they are imperfect, continue a dialogue to improve them, they are just one tool in decision-making, they shed light in some unexpected ways-- but definitely don't suppress them.

Posted by: wisewon | Jun 28, 2007 3:41:39 PM

"Does Samuelson think it relevant, for instance, how much money alumni give to the university every year? Are financial resources per student really so telling? Should US News be allowed to simply make up SAT scores for non-participating schools"

Yes, yes, no.

Rankings are essential, but many schools simply supply false data to US News, and this should be a bigger bone of contention.

Posted by: Steve | Jun 28, 2007 4:27:11 PM

It's not always clear why one school has more alumni giving than another. Many of the graduates of the college I attended (I'm not going to name it because I don't want to get into a pissing contest) are very high-achieving, but in non-profits and other areas where the financial rewards are often not so great. (If you define success as how much money you make we probably disagree on many things.) Percentage of alumni who give something might be more meaningful, since it could reflect the sentiment that their time was well spent there.

When I was in grad school there was another iteration of this general rankings argument, and the National Survey of Student Engagement seemed to me a better way to compare colleges. Unfortunately, it can be hard to get at the rankings or the data -- the goals of the survey seem to be more aimed at showing colleges how they can improve.

Posted by: Karla | Jun 28, 2007 4:52:35 PM

> Rankings are essential, but many schools
> simply supply false data to US News, and
> this should be a bigger bone of contention.

Is inducing hordes of unqualified high schoolers to apply so that the school can increase its rejection ratio in the US News ranking providing "false data"? Why or why not?

Also, you skimmed right over US News actually /making up numbers out of whole cloth/ to punish schools that had displeased it, which would seem to invalidate the entire exercise right there.

Cranky

Posted by: Cranky Observer | Jun 28, 2007 4:54:00 PM

While we're at it: Given that the schools that have chosen not to require SAT scores (resulting in numbers being made up by USNWR) have found zero difference in performance between students who do and don't turn in scores (despite a presumed difference in scoring) shouldn't we question not just how colleges are ranked, but how students are?

Posted by: Sam L. | Jun 28, 2007 5:34:29 PM

As others note, you may be right to criticize the overall power of the US News rankings, but the specifics you pick are, well odd: school endowments and the opportunities for financial aid matter greatly in almost everyone's consideration set, and I can tell you I wound up at my school (and I'll happily share it: Hamilton College) because of their alumini-based wealth (Hamilton has an absurdly large endowment for a school of its size, and a hardcore alumni effort is key to it) and the financial package it afforded me while I was there. As with other discussions here, there's nothing wrong with being rich; and in an educational institution, it seems an especially useful trait.

But I have to say I watch the supposed importance conferred on the US News rankings with some bemusement: this stuff has power only because people seem to buy into the myths; our values systems are so out of whack that we've stopped looking at colleges as a special, rare time for kids to thrive and grow in an educational setting, and instead focus our green eyeshades on bottom-line assessments of what value a degree from a certain school can mean to future employment. That approach seems very ass-backwards to me (I based most of my decision making on The Preppy Handbook, where Hamilton, among a number of my other choices, was in the top 10; that too may not be the best approach, but it was at least kind of about the experience I would have at school, not the job I would hold after that). I'd love it if the US News rankings had less power. But achieving that has to do with confronting a larger question about the American values system, and I'm not sure anyone's ready to do that just yet.

Posted by: weboy | Jun 28, 2007 5:39:10 PM

ditto what weboy said. I forgot to mentioned that while the real numbers (not the fake ones) are important to know, including I might add real employment rates and doing what (ie, what type of work for how much etc), I do think that the rankings are stupid. With this the real answer for each person should be - it depends. Why? because its all individual specific as to what their goals are and what their situation is.

Posted by: akaison | Jun 28, 2007 5:45:28 PM

(I'm not going to name it because I don't want to get into a pissing contest)

Ha ha! Not Harvard!

Posted by: Stephen | Jun 28, 2007 6:41:40 PM

Rankings are essential

Says who? How did we ever manage in the dark ages before the college ranking business brought light to the world? Way back in 1974, when I graduated from high school, I don't recall there being any such thing, and I somehow made an informed decision, anyway.

Posted by: Donna Queen | Jun 28, 2007 6:45:36 PM

Well, rankings wouldn't have, and reputation didn't, warn me against the University of Chicago (motto: "Hell does freeze over"). I'll be for any system that ranks that place at the bottom.

Posted by: calling all toasters | Jun 28, 2007 8:21:23 PM

"How did we ever manage in the dark ages before the college ranking business brought light to the world? Way back in 1974, when I graduated from high school, I don't recall there being any such thing, and I somehow made an informed decision, anyway."

An informed decision based on what information source? My grandfather picked my school for me. I knew I was a lot smarter than most of the people going there, but it wasn't until years later that I found out how low most people's GPAs were, or that many hadn't taken college prep courses.

All I had for information sources were the silly brochures that the colleges sent out, which with one exception didn't report any hard data about grades or SATs. On the last day of school one of my teachers said: "we really thought you were going someplace better that State U."

I thought about transferring for several years, so I read through a lot of college guidebooks that only said vague things like: "the people are really nice at this school," "there's not much parking space," "the faculty are helpful." So I just stayed where I was, because the courses offered at my school were pretty much the same as those offered anywhere else.

Posted by: Steve | Jun 28, 2007 9:15:47 PM

If we like schools with large endowments because that means more financial aid, couldn't we just look at stats on financial aid?

Posted by: Consumatopia | Jun 28, 2007 9:35:02 PM

For a critique of the US News rankings, see
http://www.educationsector.org/research/research_show.htm?doc_id=404230

“[T]he U.S. News ranking system is deeply flawed. Instead of focusing on the fundamental issues of how well colleges and universities educate their students and how well they prepare them to be successful after college, the magazine's rankings are almost entirely a function of three factors: fame, wealth, and exclusivity. They directly or indirectly account for 95 percent of a school's ranking...

As a result, the influential rankings have led colleges and universities to focus their energies on becoming wealthier, more famous, and more exclusive, often at the expense of what matters most—educating their students well...

The failure of the U.S. News rankings to provide colleges with incentives to improve the quality of their teaching is one reason why studies have found that many American collegians aren't learning what they need to know...”

Posted by: Bloix | Jun 28, 2007 9:52:11 PM

calling all toasters,

I believe U. of Chicago is popularly known as "the place where fun went to die."

Of course I went to a law school, Catholic University, that is becoming the preserve of right wing nut jobs like Doug Kmiec. I wouldn't give those bastards a dime that I filched from the collection plate.

Posted by: Klein's tiny left nut | Jun 28, 2007 10:21:40 PM

My own undergraduate alma mater (Pomona College: Сecil Sagehen chirps!)assembled one of the nation's largest endowments not by particularly high alumni giving, but by investment policies that stretch back to the 1920s. I'm not certain that that really says much useful about the school for current potential students ("we were early investors in distressed debt" and "we stayed out of media/entertainment venture capital in the late 1990s"?). The school is actually comparatively substantively worse in placing students in the job market after college than other California schools (including Claremont McKenna, Stanford, USC, and some UCs - as well as having a worse placement record than competing East Coast liberal arts colleges) and doesn't have many super-wealthy alumni.

Posted by: burritoboy | Jun 29, 2007 2:45:11 AM

That acutally surprises me, burritoboy, because when I was applying to colleges in the early 90s, the people who chose to go to Pomona were people I'd probably place on the same caliber as the top UCs and certainly higher than people who went to CMc and USC. I would have assumed it would have attracted basically everyone in California who wanted to attend a small liberal arts college like Williams or Amherst but didn't actually want to leave the west coast.

Which I guess goes to show you that solid quantitative data about job placement (and graduate school placement and grades and scores) is worth more than data about endowments. Though endowment size does give a certain amount of indirect data-- after all, schools with a large endowment can offer better and more consistent financial aid packages to attract good students rather than merely students who are able to pay.

The US News rankings show up in a way that turns out to be "about what you'd expect" in many ways, and there's a reason for this-- when the list was first compiled, they used a methodology to rank the schools. When doing so, this methodology placed Harvard somewhere towards the top, but not THE top-- maybe somewhere in the middle of the top 10 or 15. It was decided that, since Harvard/Yale weren't being ranked at the very top, there must have been something wrong with the methodology, so they changed their equations and ratings until they got the desired result and decided that this was the "correct" methodology.

Posted by: Constantine | Jun 29, 2007 9:54:57 AM

All the raw data is interesting and helpfull for applicats, but does US News invite any outside experts to peer review their particular rankings. We can say what we want about the relative value of the factors, but for most people I suspect it's the Number that exerts the most influence. School administrators certainly seem to believe that, as evidenced by their campaigns to increase applicant pools of even unqualified student, just so they can reject them and increase their selectivity rating.

Posted by: AJ | Jun 29, 2007 11:59:14 AM

Constantine,

It's the pecularities of the immediate post-college job markets in California (and the West Coast generally) versus the East Coast. Both of the leading universities in early California were engineering focused (UC Berkeley and Stanford) at their foundings. And the major employers in California have always been technology focused (thus primarily seeking engineering majors), even into the early twentieth century, if not before. Add in that the traditional demand for liberal arts grads - banking, consulting, publishing and the federal government - are all not much present on the West Coast, and you have a problem. (CMC and USC both have undergraduate business majors).

Not that I would discourage potential students from going to Pomona - the school has an unparalled record of gaining graduate school admittance for its students (more Sagehens go to Harvard Law than any other school except Harvard for instance) - but the job market straight out of Pomona isn't as stellar as the rest of the school.

Posted by: burritoboy | Jun 29, 2007 12:28:32 PM

I am with wisewon. Partial data is better than no data.

Posted by: joeo | Jun 29, 2007 6:50:07 PM

托盘
 托盘
 钢托盘
 钢制托盘
 塑料托盘
 木托盘
 木制托盘
 纸托盘
 木塑托盘