Scientific Reviews of Medical Journals
Author: Wallace Sampson
By the time this editorial is read, the Scientific Review of Alternative Medicinewill have been evaluated by the National Library of Medicine (NLM) for Medline/PubMed indexing-for the fourth time. Indexing on Medline is important to the exposure and survival a medical journal for obvious reasons. Systematic reviews and reviews of the literature on any subject include reviews of the titles and abstracts on Medline. Medline is the most widely used biomedical database. It is the derivative of the Index Medicus-the printed volumes of which were the major source for references in years before the Internet. Medline's presence and universal availability has revolutionized research and education and the ease with which even the average person can, within minutes, survey the latest and presumably the most accurate information on a biomedical subject. Our experience with Medline and the workings of the NLM raises questions about those statements-at least as applied to information about "alternative" medicine.
The Scientific Review of Alternative Medicine (SRAM) was formed in 1997 and first published in 1998. The NLM first reviewed SRAM for indexing and abstracting on Medline the same year. This was unknown to SRAM's editors and publisher, who at the time were unaware of the workings of Medline and assumed that the NLM abstracted most journals, and we did not give the matter much more thought. We did not know there were standards and guidelines or consider that an application to the NLM might be necessary. We applied for listing in Medline in late 1998/early 1999 and were asked to send several issues of SRAM for review. We were at the time also unaware of the existence of the NLM's formal standards or how to access them. The NLM reconsidered SRAM and rejected it again.
We were not told of any deficiencies upon rejection. We knew of other "alternative" medicine journals indexed on Medline, and we knew that their contents were ideologically biased and often erroneous, and that many of their editorials and commentaries contained false information. Suspecting that our review had been biased, we requested copies of all the evaluations as well as names of all reviewers, under the Freedom of Information Act. Pleading confidentiality, the NLM refused to release them.
We then requested that the NLM repeat the evaluation with independent outside reviewers. The NLM consented, and again rejected SRAM. We had exchanged letters with Mr Sheldon Kotzin, executive head of the NLM, and Donald Lindberg, MD, the director. We suggested that, considering that advocacy journals were indexed by Medline, we believed the NLM review procedure was probably faulty or biased. Dr Lindberg maintained that the review process was fair.
Dr Lindberg revealed in his letter that SRAM's first review had been conducted by 14 organizations and individuals devoted to "alternative" medicine. This did not seem to us to be objective, given that the evaluators were likely to be people whose occupations and methods were being analyzed critically in SRAM. The most that the NLM would reveal about the third review was that it was conducted by independent reviewers from within the National Institutes of Health. This was not what we had in mind when requesting outside review, as there was no assurance that the reviewers were either objective or authorities in pseudoscience. Nevertheless, according to NLM/Medline policies, SRAM would not be eligible for another review for four years.
In fairness to the NLM, its job is ponderous at least. Of some 7000-10 000 or more biomedical journals in print, the NLM has the space and personnel to handle only about 4000. Some journals are unworthy of indexing, for instance, because of a lack of relevance to biomedicine; some do not make it financially and fold after only a few years; some have a demographic so small that indexing them would be a waste of resources. The editor of one OB-GYN journal told me that her publication waited 9 years before being indexed on Medline. Another said his journal had been accepted after its first issue.
The NLM has a hierarchy for selection. Highest priority goes to journals carrying original basic research, especially in cutting-edge subjects, and next to those publishing clinical trials. Editorial policy, peer review, presentation of "balanced views," and other internal policies are considered. Review and commentary journals are apparently felt to be of lesser importance. A review committee of outside experts reviews more than 100 journals at least 3 times per year.
On rejection, we were not informed of Medline policies, told where to obtain them, or informed how our journal failed to fulfill them. And we were still troubled by the fact that Medline indexes a number of complementary and "alternative" medicine (CAM) journals that print erroneous, ideologically oriented material.
A 90-minute survey of Medline CAM references under only three headings-" acupuncture," "homeopathy," and "alternative medicine"-resulted in a surprising abundance of advocacy journals in the NLM database:
- AAOHN Journal
- Advances in Mind Body Medicine
- Advances in Nursing Practice
- Alternative Medicine Review
- Alternative Therapies in Health and Medicine
- American Journal of Holistic Nursing
- British Homeopathic Journal
- Chinese Journal of Integrative Traditional and Western Medicine
(Now Chinese Journal Integrative Medicine)
- Complementary Medicine and Nurse Midwifery
- Complementary Therapies in Medicine
- Ecological Medicine
- Forschung Komplementar Medizin (German)
- Holistic Nursing Practice
- Integrative Medicine
- Journal of Advanced Nursing
- Journal of Alternative and Complementary Medicine
- Journal of Holistic Nursing
- Med. Ges. Gesch. (German)
- Medical Hypotheses
- Singapore Medical Journal
The following, found during the same search, are magazines and other non-peer reviewed publications that are occasionally abstracted on Medline:
- Nursing Standard
- Nursing Times
- US News & World Report
- Women's Health
A more recent brief search for "acupuncture" produced the following:
- AAOHN Journal
- Acupuncture Medicine
- American Journal of Chinese Medicine
- Beijing Da Xue Xue Bao
- Di Yi Sun Yi Da Xue Xue Bao
- Journal of Traditional Chinese Medicine
- Sichuan Da Xue Xue Bao Yi Xue Ban
- Zuong guo Zhong Yi Jie He Za Zhi
While researching chelation therapy, we searched the records of a 1998 House of Representatives Governmental Reform Committee hearing and found that a number of those journals had been recommended by Wayne Jonas, then director of the Office of Alternative Medicine (which would later become the National Center for Complementary and Alternative Medicine [NCCAM]), himself a homeopath. The recommendations had come at the request of Dr Lindberg. No request was made to any organization outside the NCCAM.
At that same congressional hearing, Dr Lindberg listed the 14 organizations and individuals used to evaluate "alternative" journals, which the NLM refused to release to us under the FOIA:
- The AIDS Research Center at Bastyr University (Bothell, Washington)
- The Center for Addiction and Alternative Medicine Research (Minneapolis)
- The Center for Alternative Medical Research at Beth Israel Deaconess Medical Center (Boston)
- The Center for Alternative Medicine Research (Houston)
- The Center for CAM Research in Women's Health at Columbia University (New York)
- The Center for Complementary and Alternative Medicine Research in Immunology at the University of California, Davis
- The Center for the Studies of Complementary and Alternative
- Therapies in Nursing at the University of Virginia The Complementary Medicine Program at the University of Maryland
- The Institute of Information on Traditional Chinese Medicine (Beijing)
- Kessler Research (West Orange, New Jersey)
- Project Munchner Model at Ludwig-Maximullism-Universitat (Munich)
- The Research Council for Complementary Medicine (London)
- An unnamed specialist in chiropractic medicine
- The Stanford Center for Research in Disease Prevention
All of these are known advocacy organizations, many of which had previously heard of SRAM. The head of he University of Maryland's program was aware of SRAM's orientation; he and the editor had met in 1996. The Stanford center, an NCCAM grant recipient, had consistently ignored the editor's ongoing critique course in "alternative" medicine at Stanford. The Beth Israel Deaconess research center is that of David Eisenberg, MD, who knows of and has corresponded with the editor and who is aware of SRAM's objective, scientific orientation and its published critique of his noted survey. The Columbia University center personnel were likely aware of our previous activities, and had previously listed the editor's Stanford course in its "alternative" medicine medical school course list, but has omitted it in recent listings.
The Minneapolis center published a paper claiming acupuncture efficacy for alcohol recidivism in which 90% of controls dropped out.
A specialist in chiropractic medicine-the NCCAM accepts the inappropriate occupational label. Chiropractic is not a form of medicine.
Bastyr University is an academically unaccredited naturopathy school.
Following the last rejection by the NLM, I requested a meeting with Mr Kotzin at an NLM-sponsored conference on scientific misconduct for science editors in Leesburg, Virginia, held in November 2003. Mr Kotzin met with me for an hour and a half, during which time we both aired our concerns. He recently wrote that he is willing to reassess the situation since our talk and his receipt of the following letter. Because of our recent change in publisher, he kindly offered to review SRAM again ahead of schedule, as a new journal's first review.
Sectarian medicine, pseudoscience, and their ideologues have made extraordinary inroads into the inner structure of science and medicine. This has gone almost unnoticed except by a few scientists and professionals in the skeptical and antifraud organizations. So pervasive are these inroads that the perception of what is "normal" and "balanced" has shifted. Mr Kotzin stated that some criticism of SRAM is that it is too polar and one-sided. We find it frustrating that what we perceive to be objective analysis is now viewed by people in such major institutions as the NIH and the NLM as "polar."
* * *
The following letter, dated November 28, 2003, was the editor's follow-up analysis of the SRAM/NLM problem, along with speculation as to how the distorted NLM database may have come about.
Dear Mr. Kotzin,
Thank you for the time spent and courtesy shown to me in discussing the problems that "alternative medicine"-sectarian systems and implausible methods-present to the scientific community, and for access to accurate information. This long letter will amplify on our perception of the unbalanced "CAM" material on Medline, and the National Library of Medicine's inheritance of a problem of which its personnel may not be aware. Our journal, the Scientific Review of Alternative Medicine, has a special position in this controversial situation.
I understand the policies as you clarified for me; the NLM has responded to the NIH, under which it serves as a major source of information; NLM operates within its own and NIH policies in selecting journals for indexing, including procedures for peer review. I assume also that NLM responds to Congress, to which it owes its existence and funding.
However, there are two issues I hope to introduce and ask be considered as I request our journal's indexing. These are:
- the fact that the NLM mission and service to science has been altered by background activities of which NLM may not be aware, and for which there may be no governing policy, and,
- NLM has an implied responsibility also to the general medical scientific community to supply truly balanced and valid information.
The NLM policies, although seemingly fair and consistent, are based on perhaps unrecognized assumptions that they can be relied upon without review of their outcomes, and that those outcomes have not been altered by unrecognized forces. However, the "alternative medicine" ("CAM") movement has brought about changes in biomedical literature advantageous to sectarian advocates and practitioners, and detrimental to the scientific enterprise. My series of letters and Freedom of Information requests were intended to investigate how Medline searches came to result in a lopsided fund of information favorable to unscientific material, and for clues as to how at the same time SRAM was rejected. This letter outlines in lengthy detail justifications for the two above issues to be considered.
I obtained the journal list below in a 1.5-hour Medline search, taking the first 50 citations under each of just three headings "alternative medicine," "homeopathy," and "acupuncture." I did not use "complementary," "integrative," "chiropractic," "traditional Chinese medicine," "herbs," "naturopathy," or scores of other subjects. The search produced over 30 periodicals devoted largely to positive "CAM" material, and none of which was critical.
[The list of journals is the same as the one in the body of the editorial above.-EDS.]
All of the above journals are advocacy journals. None of them prints balanced views or more than rare critical statements. One questions how and why magazine articles are indexed on Medline when journals are theoretically held to peer review and editorial standards. There is reason to question the listing of many journals from Mainland China and Taiwan-especially written in Chinese-since almost all journal articles from there claim positive results (vs. 75-80% of those from North America and UK.) This implies at least positive publication bias, and probably lack of objectivity in research and reporting. There is also reason to question the listing of all acupuncture journals because acupuncture, being only a counter-irritant and placebo, does not deserve more detailed clinical investigation or discussion-certainly not more than one journal.
Yet to be determined is how some journals became accepted for indexing during the same past 7-year period. For example, Integrative Medicine, a polar and advocacy organ, previously edited by Andrew Weil, now staffed by naturopaths. The journal was apparently dropped, but had been approved for two years. Alternative Medicine Review is the commercial organ of Thorne Research-a supplement distributor-and contains only ads of that company. Its editorial staff is largely naturopaths. A search under "energy medicine" produced an article in a magazine named Director. It is not listed as a medical journal. Medical Hypotheses is a periodical devoted to explorations of implausible theories. Laetrile-a scientific fraud-was first proposed in this journal.
I also obtained a list from both our own medical librarian and an NLM librarian of over 100 journals that publish significant amounts of "CAM" material. I am familiar with many of these journals, have subscribed to several, and have read through issues of many others. They are mostly advocacy oriented. This situation speaks to the first principle above-that the "CAM" community has essentially flooded the journal market with new advocacy journals and has packed their contents with exclusively positive reports and surveys that are irrelevant or self-serving, false, or highly suspect. For some reason, this has escaped the notice of NLM.
Another consideration is the position that "alternative" thinking has obtained in editorial staffs of some major journals. "Alternative" thinking refers to a type of postmodern, complex, and poorly understood ideological movement in literature and the social sciences. However, it profoundly affects the orientation of people and organizations in evaluation of scientific information in standard journals.
In the past thirty years it has become journalistic bad form to use "negative" terms in describing misrepresentation, error, irrationality, and ideological influence. This shift results from relativistic and postmodern thinking, now taught in medical anthropology and sociology, medical schools, journalism, and other courses. The principles are now embedded in some recent editorial policies. The most concerning of these is the demand for a "neutral" style in critique, and the diminished importance of plausibility, physicochemical laws, and basic science. Style now trumps content.
Changes in language use and editorial policies have transformed "quackery" into "alternative" and have replaced constructive scientific criticism with a policy of demand neutrality. Manuscripts containing the words "quackery," "misrepresentation," etc. are generally rejected.
These characteristics are based on postmodern principles: there is no objective "truth"; attempts to reach reality are fruitless; all healing systems are therefore equally plausible; scientific medicine is just one way of looking at nature; science and its laws are primarily products of social interactions and thus are culturally determined; error and misrepresentation reflect only the different views of authors; and that language and personal views are the ultimate "reality." Reality itself is challenged as a concept. At the same time, any challenge to these principles, or any manuscript criticizing implausible claims, stands to be rejected as "too negative."
The recent 2-year series on "CAM" in the Annals of Internal Medicine, authored largely by sectarian advocates, is a classic example. Attached to an e-mail version is a draft article to be in a future issue of SRAM from a distinguished medical historian from Creighton University. His experience in the field of history of medicine is similar to those in other fields. The volume Flight from Science and Reason (Baltimore: Johns Hopkins University Press; 1997) records similar experiences in fields of biology, physics, sociology, and medicine.
The above changes, the list of advocacy journals, and our access to the history of the situation (demands of Congress, referrals from Office of Alternative Medicine) yield a better understanding of how the Medline data base came to be distorted.
In summary, Medline has a database distorted by a flood of journals that contain largely false or biased material. The situation has been accelerated by intentional changes in language usage and editorial policies. This all results from an ideological movement originating in academic and editorial circles authorizing the manipulation and distortion of language-declaring inaccuracies to be "alternative," "complementary," "socially determined," "unconventional," "unorthodox," etc. In other words, implausible claims are acceptable, but criticism of them is "biased" and "polar." These latter terms were apparently used to describe SRAM. The NLM itself used language change when it reclassified acupuncture and chiropractic from previous classifications including "medical cults"-a term which was abandoned.
Regarding the history of "CAM" journals, a set of "alternative medicine" journals was recommended by the director of what was then the Office of Alternative Medicine, who was also a practicing homeopath. Homeopathy is known scientifically as a pseudoscience and its practice as quackery. The NLM's error here was not to have sought outside opinion, because the OAM was, and the NCCAM still is, a politically and ideologically biased office devoted to the ideas of a few members of Congress. The only other governmental organization specifically assigned to evaluation of sectarian systems is the Cancer Advisory Council of the State of California (of which I am chairman pro tem). The only nongovernmental organizations devoted to analysis of sectarian systems and methods are the National Council against Health Fraud (of which I was a cofounder and Board Chairman), the Committee for Scientific Investigation of Claims of the Paranormal (of which I am a fellow), and service organizations such as the American Cancer Society's former Unproven Methods Committee (on which I served for 2 years). None of these organizations was, to my knowledge, consulted. I mention my role only to document and emphasize my long experience with these matters.
In addition, a new class of NLM reviewers was apparently established to review journals covering these subjects. SRAM's first reviews were carried out under that system and by individuals who were themselves biased. The journals requested by Congress (Rep Dan Burton) and the OAM passed this review procedure.
This situation has resulted in an advocacy-dominated fund of material that biases research reports, systematic reviews, and meta-analyses; alters student research projects and medical school teaching; and biases patient and physician searches. It is also altering public policy. SRAM was formed partly because of this surprising abundance of "alternative" journals with inaccurate information carried on Medline. Its other function was to serve as a reference for the community of scientists interested in developing a systematic inquiry into implausible claims and into scientific, sociological, and public health consequences of such claims.
We hope you will see indexing of SRAM as the first move in an attempt to rebalance the material in Medline searches. SRAM does not encourage clinical trial and other primary research submissions, preferring to concentrate on reviews, investigations into anomalous claims, and new methodologies for investigation. I request consideration of convening a meeting of biomedical researchers, clinicians, and other scientists, exclusive of "CAM" advocates and researchers, to review the problem and the present database, and to consider dropping from Medline those periodicals not containing valid material. Perhaps a separate database could be opened for such journals. A reassessment and rearrangement would be in the interests of accuracy.