Design Review of the National Resident Matching Program


First posted: 12/12/95.
Revised: 3/11/96 (added interim report #1)
5/7/96 (added interim report #2, and some background links)
most recent revision 12/6/96: added Report on the design and testing of an applicant proposing matching algorithm, and comparison with the existing NRMP algorithm

visits since 1/5/96.

Welcome...

My name is Al Roth, I am a professor of economics, and I was retained in the Fall of 1995 by the Board of Directors of the National Resident Matching Program (NRMP) to direct a study intended to evaluate the present NRMP algorithm, design one or more alternative algorithms, and compare these in various ways. The purpose of this page is to both publicize the study in the medical community, and to solicit suggestions (for which my email address is provided at the bottom of the page).

The design and comparison phase of the study was concluded in late 1996, and the final report is now available here: Report on the design and testing of an applicant proposing matching algorithm, and comparison with the existing NRMP algorithm

For those unfamiliar with the market and the questions which were to be answered, a little background may be helpful. The progress of the study can be followed by looking at the initial proposal, and at the Interim Report #1: Evaluation of the current NRMP algorithm, and preliminary design of an applicant-processing algorithm, and Interim Report #2: Some Proposed Comparisons of Matching Algorithms

Background

The National Resident Matching Program is the computerized market through which the vast majority of new physicians in the United States find their first jobs, as residents. Graduating medical students (and others seeking residency positions) interview at residency programs, and then both students and programs fill out rank order preference lists and submit these to the NRMP, which produces a match by computer. Recently there has been a good deal of discussion in the medical community about how this market is organized and conducted, focusing particularly on the matching algorithm, and whether it should be changed.

This discussion took specific form in four articles published in the June 1995 issue of the journal Academic Medicine. On the web, the American Medical Students Association (AMSA) has posted both a "call to arms" urging changes in the NRMP, and (together with Public Citizen, Ralph Nader's organization) has distributed a more detailed Report on the NRMP explaining their position. See also the subsequent Statement of the American Medical Association Council on Medical Education, Section on Medical Schools, Resident Physicians Section and Medical Student Section Concerning the National Resident Matching Program

In the early 1980's I conducted a study of the NRMP as it was then organized, starting with an account of the history which led to its organization in the early 1950's. The introductory parts of that paper are up on the web, here:
Roth, A.E. "The Evolution of the Labor Market for Medical Interns and Residents: A Case Study in Game Theory", Journal of Political Economy, 92, 1984, 991-1016.

That paper showed that the matching algorithm used by the NRMP produced a matching that is stable in a certain sense. In a subsequent paper, I studied markets for new physicians in the U.K., which helped make a very strong empirical case that this kind of stability is important for the success of computerized markets of this kind. An electronic 'reprint' of that paper is available in its entirety on the web, here:
Roth, A.E. "New Physicians: A Natural Experiment in Market Organization," Science, 250, 1990, 1524-1528.

It now seems uncontroversial that, to continue to organize the market successfully, the NRMP must continue to avoid producing the kinds of instabilities which have caused some other attempts at organizing computerized markets to fail. However there can be more than one stable matching, and so the controversy surrounding the NRMP has focused on which of perhaps many stable matchings should be chosen, and whether the algorithm should be organized in a way which, when there is a choice, tends to favor hospital programs, or students.

Some further background material, which puts the NRMP in the context of many other entry level labor markets (e.g. lawyers, graduates of elite Japanese universities, etc.) can be found in a short paper
Roth, A.E. "The NRMP as a Labor Market," Journal of the American Medical Association, 275, April 3, 1996, 1054-1056. (this paper is reproduced here in its entirety).
Much more detail (at the cost of much greater length) can be found in
Roth, A.E. and X. Xing "Jumping the Gun: Imperfections and Institutions Related to the Timing of Market Transactions," American Economic Review, 84, September, 1994, 992-1044.

In the Fall of 1995, the NRMP Board invited me to propose a plan for analyzing the effects of changes which might be considered in the matching algorithm. The plan I proposed is reproduced below. It consists of three phases, and the NRMP Board has decided to proceed with the first of these, which will involve an analysis of the current market algorithm and organization (which has changed considerably since my 1984 paper), design of alternative algorithms, and computational experiments based on the preference orders submitted in recent matches.


Consultant's Report

Proposed research program:
Evaluation of changes to be considered in the NRMP algorithm


by Alvin E. Roth

October 24, 1995
Executive Summary:

This document outlines a preliminary research program for the evaluation of the current NRMP algorithm and of changes to be considered in its operation and description. A principal objective will be to design a student-proposing stable (or approximately stable) match algorithm and to assess what would be the effect on the match of substituting such an algorithm for the present NRMP algorithm. Attention will need to be paid both to the mathematical/computational properties of the algorithm, and to the behavior of market participants and how this might be influenced by a change in algorithms. In addition, attention will have to be paid to how any changes would be monitored and evaluated.

The following table of contents gives an overview of the issues to be addressed and tasks to be undertaken.

Table of Contents

Introduction

Task List and Tentative Timeline


Introduction:

This document outlines a preliminary research program for the evaluation of the current NRMP algorithm and of changes to be considered in its operation and description. The matters considered are primarily those raised in the June 1995 issue of Academic Medicine. A principal objective will be to design a student-proposing stable (or approximately stable) match algorithm and to assess what would be the effect on the match of substituting such an algorithm for the present NRMP algorithm. Attention will need to be paid both to the mathematical/computational properties of the algorithm, and to the behavior of market participants and how this might be influenced by a change in algorithms. A related issue is how the match should be described to participants, and what advice should be given.

This research program can be usefully thought of as consisting of three overlapping but largely distinct parts, namely:

To design an alternative (student-proposing) algorithm, and compare it with the existing algorithm, in terms of both output matchings and opportunities for strategic behavior;

To assess and analyse the behavior and perceptions of participants in the market (including behavior prior to filling out rank order lists--e.g. in the interviewing stage), and consider how these might be affected by a change in algorithms;

To consider the effect of changes in market demographics-- i.e. the numbers and kinds of positions and applicants--on the above.

Studies by this author of other entry level professional labor markets and matching processes, both medical and non-medical, have now made it noncontroversial that centralized matching mechanisms like the NRMP arise in response to certain kinds of market failures, which deprive all participants of much of the benefit that an orderly market offers 1. In the case of the NRMP, these market failures involved both very early appointments, and a chaotic and unreliable process of making and confirming these appointments, which existed in virulent form in the decade prior to the establishment of the NRMP. These same kinds of market failures persist today in other markets, such as the American market for Federal court law clerks. One concern therefore is to evaluate the danger that these market failures could re-emerge, as market conditions change, both under the current operation of the market and under changes in its operation which might be considered. In this respect, economists as well as physicians should seek to work to Hippocrates' standard:

"The physician must be able to tell the antecedents, know the present, and foretell the future- must mediate these things, and have two special objects in view with regard to disease, namely, to do good or to do no harm." [Hippocrates 2]

In particular, decisions about the future operation of the market should be informed by some assessment of the likelihood of a recurrence of the behavior which led in the past to market failure. Thus if, following the analysis of the results of this proposed study, a policy decision is made to change the algorithm, then it would be prudent to design the change, and subsequently monitor and evaluate it, with the aid of a further program of monitoring and experiments, akin to clinical trials for a new treatment of a disease.

The sections which follow outline particular questions to be pursued, and methods of pursuing them.

I. Design of a student-proposing algorithm, and comparison with the current NRMP algorithm