18 June 2009

Gaming the rankings

Among the many issues with ranking schools, one of the most glaring is incorporating the input of those who are impacted by the result. Students reporting on MBA programs or University presidents ranking schools all put people influenced by the result in a position to influence the results. This creates quite the incentive problem.

Recent evidence comes from the rankings of schools (pdf) provided by University of Florida President Bernie Machen. The surveyed rankings are an integral part of the U.S. News ranking formula, and were obtained by the Gainesville Sun in a public records request. Other Florida university presidents were shrewd enough to "lose" theirs.

See Machen game the rankings

U.S. News treats the surveys as anonymous, meaning that a university president's ranking of his own school carries equal weight as others' rankings. On Machen's survey, the University of Florida was given the highest possible ranking, one that he granted several generally well-regarded schools only after some revision.

University of Florida President Bernie Machen games U.S. News rankings
(excerpt of Machen's rankings)

More telling is the rankings Machen gave to other Florida public schools which are competitors for State funds. Machen rated more Florida schools as "marginal" (the lowest possible category) than schools from all other states combined.

Editors responsible for the ratings claim that "statistical methods" are used to adjust for such biases. The reality, of course, is that no statistical test can divine thoughts separate from incentives. If you asked me to rate myself as a "good person" on a scale of 1 to 10, a period of reflection would follow. If you added that my results would be anonymous, unverifiable, and come with a million dollar payment if I circled "10," you would learn nothing about me from the exercise except my responsiveness to incentives. So why would U.S. News editors contend that as-yet uninvented statistical methods protect the integrity of their results? Perhaps they, like President Machen, have a stake in the results.

UPDATE: I am not suggesting that UF does not deserve to be ranked highly along several dimensions. For example, one reader reports that UF must be at the top of its peer group in criminology, with over 4% of its students arrested annually.

09 June 2009

Circular reasoning and the debasement of science

Ranking journals is a popular pastime among academics. Each of us has a favorite ranking, largely chosen by the results fitting with our favorite publication outlets. There are more debates over the methodology of journal rankings than of ranking business schools. There may be no universal agreement on the right method but there certainly is a wrong one.

Continue Reading

Kristie Engemann and Howard Wall have published a new ranking of economics journals. Their method consists of "a simple rule that considers citations only from a short list of top general-interest journals in economics." In short, they arbitrarily select the "top" journals, count the number of citations from these to other journals, add an adjustment here and there for effect, and presto! We determine the top journals by counting citations from top journals. Seems a bit circular.

If you walk into a random high school and want to know who the popular kids are, the Engemann and Wall method would have you identify them by seeing with whom the popular kids choose to hang out. The procedure might produce slightly different results if you started with the debate team than if you started with the cheerleading squad. It might not be a surprise, then, the top five journals in their results are included in the list of top journals by assumption. I don't disagree with the list, intuitively, but science should perhaps take a more objective path.

A more objective path does indeed exist. A commonly-used recursive algorithm initially assigns all journals an equal value. Each iteration of the algorithm assigns value from one journal to another based on citations. The iterative procedure, by the way, is at the heart of Google search results (replace "citations" with "links"). From the Google founders' monumental paper:

PageRank or PR(A) can be calculated using a simple iterative algorithm, and corresponds to the principal eigenvector of the normalized link matrix of the web.

The authors of the new ranking poo-poo this mathy stuff:

[The iterative] procedure is largely a black box: It is not possible to see how sensitive the weights (and therefore the rankings) are to a variety of factors. The obvious objection to our rule is its blatant subjectivity. Our counter to this objection is to point out that the [iterative] procedure, despite its sheen of objectivity, contains technical features that make it implicitly subjective.

Ummm... Sensitivity analysis even has its own Wikipedia page.

If Engemann and Wall were to start their own search engine, the Google formula would presumably be replaced with "pages with links from pages we like."

This is not to say that cheerleaders don't often overlap with the debate team. But, seriously, they don't.

Hat tip, Mankiw

10 May 2009

I blame the public schools

Preparing for my upcoming Canada trip, I initiated an online chat with a Sprint rep to find out the roaming voice and data rates. Sending an average-length email would, according to the agent, cost somewhere between a few cents and a few hundred dollars. Transcript below the jump.

Read the transcript

Me: Hi, I will be going to Canada this week, and wanted to know what the voice and data rates are with and without the Canada package.

Lakisha: Hi, my name is Lakisha. Thank you for your chat request. Please wait while I review your information.

Lakisha: I will be more than happy to assist you today.

Lakisha: While roaming in Canada with your Sprint device all calls will be billed $0.59/minute, data service is $0.002/KB. Sprint does offer a Canada Roaming plan for $2.99 per month; this plan reduces the voice rate to only $0.20/minute.

Me: But not the data?

Lakisha: Correct.

Me: And data is 2/10 of one cent per KB, is that right?

Lakisha: Data is $2.00 per kb.

Me: $0.002/KB is very different from $2.00/kb - could you please confirm the rate

Lakisha: $0.002/KB is the same as $2 per kb

Me: $0.002 is 2/1000th, right? Which is very different from 2.

Lakisha: The data rate if you were to use it will be $2 per kb.

Me: Above, you wrote $0.002/KB.

Lakisha: Which is the same.

Me: You're kidding, right? So, a 5KB email is $100 or 10 cents?

Lakisha: It will not be 10 cents because you will pay $2 per kb.

Me: Can you please review the transcript above. The first thing you said is $0.002/KB.

Lakisha: I do understand

Lakisha: The data rate within Canada will be $0.002kb which is compatible to $2 per kb.

Lakisha: Would you be activating the Canada reduce rate plan today?

Me: 1 cent = $0.01 ! There's a difference between using 1000KB and being billed $2 and $2000

Lakisha: You can always call our toll number which is 8882267212

Lakisha: Thanks again for choosing Sprint Worldwide chat. Have a great day.

Lakisha: has disconnected.

So, $1=$0.001 (=1/10 of one cent), and 1 kb = 1 kilobit = 1 KB = 1 kilobyte (=8 kilobits). Using Lakisha's "is compatible to" operator (you'll learn about it in higher-level math classes), a 5 KB email costs 1 cent, which is compatible to $80.

Before you go mocking Sprint, note Verizon isn't much better.

19 January 2009

If "English Only" passes

This Thursday, Nashville votes whether to prohibit public business from being conducted in any language other than English.

In an impassioned speech, Councilman Eric Crafton contends that newcomers to foreign lands must learn the local language. Strangely, this sentiment was not expressed in Cherokee, Iroquois, or Choctaw! Exhibiting his penchant for irony, Crafton delivered the speech in a recently adulterated dialect of the imperialist powers that, for effect, I adopt in this blog post.

Crafton, fresh off a three year crusade affirming Jesus Christ legislatively, is still not content wasting his time in Metro Council on such mundane issues as schools or zoning.

But there's a positive side to this bill if it passes. If English is to be mandated, every time Crafton begins a sentence with "If I was" he can be held in contempt, and required to attend a lecture on subjunctive mood.

24 September 2008

Rules are for sissies, not MBAs

At an online forum for aspiring MBA students, participants are discussing an application essay (400 word limit) for a top ten program:

Applicant 1: Quick question guys! How stringent is the word limit? I am at 423 words.

Applicant 2: I wouldn't worry about it. I've been following the +10% max rule.

Applicant 3: mmm... it's not like they count the words right? I'm thinking if you don't push it too much, they won't even notice.

A suggested essay topic for the above applicants: "Discuss the importance of corporate ethics, respect for the law, and sound editing skills. Compare the above applicant statements to the principles displayed in recent accounting and financial scandals."

The actual essay topic in question: "What is your greatest example of leadership and what personal qualities helped you succeed in that role? (400 word limit)." How about a 500 word essay on thinking outside the box?

23 September 2008

Sunspots in Nashville

Last weekend, Nashville ran out of gas. This was not because of significant shortages, but because of a belief that there were significant shortages. So, people rushed to get gas. And we ran out.

This is a demonstration of sunspot equilibria, one of the items on my still- incomplete list of the five useful things I learned in macroeconomics. What if people believed that sunspots cause the populace to turn into violent beasts who, behind their smiles, "good morning"s, and "bless your heart"s, secretly plot our demise; they appear to act normal in every way but wait for their chance to attack us. In what Charles Gibson incorrectly labeled the "Bush Doctrine," we may all contemplate preemptive self-defense by attacking first. Then, of course, sunspots did cause the populace to turn violent.

A feature of these self-fulfilling prophecies is that there are multiple equilibria; usually one very good one where we expect calm and act calmly, and another very bad one where we expect the worst and, by our reaction to it, cause it.

One simple role of government is to help coordinate the populace on the better outcome. Neither our local Nashville government, nor our presidential candidates, seem to grasp that.

04 September 2008

What Erica Gilmore can learn from the Soviets

Burning grapes

Nashville Councilwoman Erica Gilmore has resurrected a bill banning single-bottle sales of beer in a misguided attempt to curb drinking and littering. To understand the unintended consequences of hair-trigger paternalism, we turn to the Soviets.

Continue Reading

To curb alcohol and vagrancy, the Soviet Union tried everything from burning some of the oldest grape vines in Europe (fine wine is often the alcoholic's cheap fix) to banning sales of vodka in containers smaller than one liter. Since Ms. Gilmore has not yet suggested a torching of wine stocks, it is the latter experience that is instructive.

Vagrants of the social-drinker ilk would certainly consider a liter excessive, as three is the optimal number of people for splitting such a volume (really!). A simple social convention was born. The first thirsty citizen arriving at the store would stand outside with three extended fingers held against his chest. The symbol conveys an attempt to create a troika, or group of three held by the common interest of securing the appropriate measure of the beverage. A second would arrive and assume a similar loitering stance. Upon a third compatriot's arrival, a bottle would be purchased and shared.

Result: more loitering, more nuisance, more litter, and a slight uptick in violence, partly resolved by bringing a 1/3 liter measure.

Burning the grapes didn't do much, either, except hinder the economic growth of modern Moldova and Georgia.

Hat tip thinktrain, though one who asks "who really needs just one beer" has probably never heard of Trappist ales, imperial stouts, doppelbocks, barley wine, and, well, beer.