Is Peer Review Honest?

Jyllian Kemsley

AN ASSISTANT PROFESSOR at a state university submitted a paper to a well-known chemistry journal. The paper quickly received two positive reviews, then a very tardy negative review. After a revision, the submission again languished in the hands of the negative reviewer.

That reviewer, the professor suspects, was a senior colleague who was doing experiments similar to those chronicled in the paper. The professor had requested samples from the colleague to further the work but never received them. When the frustrated assistant professor contacted the journal editor and asked whether the colleague was indeed the tardy reviewer, the paper was accepted nine hours later.



Seizo Terasaki/Image Bank

That wasn’t this assistant professor’s only experience with the sharp end of what appears to be conflict-of-interest impropriety. The professor submitted to a federal agency a grant application that didn’t get funded. A year later the professor noticed that someone from the agency’s study section was working on the same topic. “I don’t think it was that novel an idea—it was an obvious thing to try,” the professor acknowledges of the proposal. But in light of a seeming conflict of interest, “it’s a tough thing to swallow.”

Separately, another associate professor had run into some troubling behavior on the part of his senior colleagues. In a grant review study section that he was part of, he witnessed a friend failing to recuse himself from reviewing an application of a competitor. The friend gave the application a poor review and the grant didn’t get funded. “He may have had reasonable criticisms,” the associate professor says, “but he shouldn’t have been reviewing that grant.”

Neither scientist felt comfortable being named for fear of professional repercussions. But when younger faculty members are asked about their experiences with peer review, stories such as these are common. Many have similar anecdotes of their own or have heard them from others.

In some cases, such as the grant review study group situation, there is fairly good evidence of wrongdoing. But more often the evidence is circumstantial, such as the assistant professor’s suspicion about why her paper was held up in review and why her grant wasn’t funded. Researchers say the vague nature of such cases makes it difficult to make accusations. There’s also often a power imbalance, in that a junior researcher will often be accusing an older, more established colleague—someone who might be asked to review future papers or grant applications or weigh in on a tenure decision.

WHEN JOURNALS and funding agencies are queried about whether there is a problem with peer reviewer integrity, the response is often a firm “no.” Peter J. Stang, editor of the Journal of the American Chemical Society (JACS) and a chemistry professor at the University of Utah, says that in six years he hasn’t seen any significant problems with reviewers, although he notes that any peer review system will have some people who abuse it.

Peter Gölitz, editor-in-chief of Angewandte Chemie, says his journal has ways to prevent and address problems with reviewers. Authors can exclude up to five possible reviewers when they submit a paper, and “we honor such requests unless there are compelling reasons not to do so,” Gölitz says. Papers generally go out to three reviewers; if their reviews are divergent, the editor can adjudicate or choose to send the paper to an additional referee.

Biochemistry Editor Richard N. Armstrong, a professor of biochemistry and chemistry at Vanderbilt University, in Nashville, says the procedures for his journal are similar, although authors may exclude only up to three potential reviewers. Both JACS and Biochemistry are published by the American Chemical Society, as is C&EN. Armstrong also says that journal editors may enter notes about specific reviewers in the ACS electronic manuscript tracking system. Such notes can include potential conflicts of interest. When asked whether Biochemistry has had any problems with reviewer behavior, Armstrong says he occasionally gets a complaint from a suspicious author but has never found any evidence of impropriety.

Some funding agencies also say they don’t see a problem with reviewer integrity. Antonio Scarpa, director of the National Institutes of Health‘s Center for Scientific Review, says that in more than two years he’s received perhaps two or three vague and unsubstantiated complaints about reviewer behavior.

The rosters of NIH grant study sections are public, and researchers applying for grant applications can ask that particular scientists not review their grants. The scientific review administrator decides whether to honor the request. After review, if applicants believe their grants were judged unfairly, they can appeal to an advisory council for re-review by a different study section.

The National Science Foundation receives about 45,000 grant applications annually. Complaints of grant-related misconduct are sent to the Office of the Inspector General (OIG) for investigation. James T. Kroll, head of administrative investigations at OIG, says that about once a year NSF has a substantiated case in which a reviewer plagiarizes material from a grant application that he or she reviewed. In the seven years Kroll has been at OIG, the office has investigated two or three cases in which reviewers have allegedly not excused themselves from reviewing grant proposals with which they had competitive conflicts of interest. None of those cases was substantiated, he says.

Yet the perception of reviewer wrongdoing persists, especially among younger researchers. Older faculty, when queried by C&EN, say they’ve never experienced anything like the stories told by their junior colleagues. “In all the years I’ve been publishing science, I’ve never encountered an instance where I thought I was being treated unfairly by a reviewer or editor,” says Vanderbilt’s Armstrong. “Sometimes I’ve disagreed, but I never came to the conclusion that someone was out to get me.”

Because researchers are often afraid to report suspicions, many cases of potential wrongdoing are never investigated and bad feelings can linger. Such under-reporting also makes it less likely that journals or funding agencies can learn the full extent of ethical problems.

The current difficult funding climate also tends to enhance perceptions of wrongdoing. Gregory A. Petsko, a professor of biochemistry and chemistry at Brandeis University, in Waltham, Mass., notes that the source of such perceptions may be mistrust stemming from increased anxiety, rather than increased wrongdoing. “People are very anxious and very frightened,” Petsko says, noting that the current funding climate is the worst he’s seen in a 35-year career.

The anonymity of peer review may also play a role. “The fact that peer reviewers are basically anonymous gives them cover,” says Janet D. Stemwedel, an assistant professor of philosophy at San José State University, in California, and an expert in scientific ethics. At the same time, she says that outright reviewer misconduct may be less of a problem than unconscious actions to protect one’s advantage at the expense of a competitor. “I’m not really optimistic about individuals’ abilities to be aware of their own conflicts of interest,” she says.

However, reviewer wrongdoing is almost surely a relatively uncommon event in the chemical research enterprise. “The community of chemists I know has a very, very high level of integrity,” says Nobel Laureate Elias J. Corey, an emeritus professor of chemistry at Harvard University.

Nevertheless, it may take only a few to harm the overall reputation of many. Also, the perception, particularly among less established scientists, of more widespread reviewer impropriety hurts morale at a time when young researchers already may be discouraged. Corey would advise his colleagues to be mindful of the Golden Rule—to treat others as you would like to be treated—and emphasizes that trust is critical in chemical research if it is “to maintain its vigor, value to our society, and intellectual excellence.”

About these ads

16 Responses to Is Peer Review Honest?

  1. Somenath Chowdhury says:

    Sometimes, some reviewers review a manuscript based on the authors name, not based on the quality of work.
    I think, when a manuscript is sent for review, the author’s names and addresses should not be disclosed to the reviewers but a code number which should be setup by the editorial office should be sent to the reviewer.

  2. Purnendu Dasgupta says:

    It is not a new idea. The trouble is that a great many manuscripts rely on previous work and then we have to prohibit people from saying “our” previous work. Not that it cannot be done but it will be difficult to implement.
    The reality is that an Editor’s task is to get the ms to the reviewers most knowledgeable in the area and more often that reviewer can recognize whose work it is.

    Also credibility sometimes is an issue. Brand name recognition is here to stay in the marketplace, to think that a parallel would not exist in other arena may be unrealistic.

  3. Ray Clement says:

    Let’s not forget the other side of the coin: poor quality research being funded and poor quality manuscripts being accepted because the reviewer is biased in favor of the researcher.

  4. Philip Warner says:

    After 37 years in this business, I have seen only 2 cases of what I would call misbehavior involving manuscripts. In one case I was a reviewer, and another an author, and both cases involved the same associate editor of a well-known ACS journal. The journal’s senior editor had a chance to intervene, but chose not to. So yes, these issues are sometimes a problem, albeit rarely.

  5. Lauren Murata says:

    The idea of “blinding” reviewers as to authors’ names is attractive, but it wouldn’t work. Not only would there be difficulties in citing one’s own previous work, as mentioned above, but also, most researchers in a particular area know exactly what their competitors are doing, and can spot their competitors’ manuscripts a mile away, regardless of whether the authors are listed.

  6. Robert Blackledge says:

    Early in my career I submitted an application for a small research grant having to do with chemical education. It was turned down. About two years later the professor I had submitted the proposal to published in a prestigious journal the results of a study that was almost identical to the one I had proposed. No, I didn’t file a protest. I knew that “resistance is futile.” However, I got out of the education dodge, went into forensic science and never looked back. In publishing papers in forensic science on two occasions out of approximately 30, one of the three reviewers absolutely had nothing positive to say about the ms while the other two liked it and suggested only very minor changes. By dealing with the journal editor I was able to bypass the curmudgeon reviewers in both instances. To me, much more problematic is the review of forensic science grant proposals by the National Institute of Justice (NIJ). First, only those proposals that are almost sure to come up with positive results are funded. To me, that isn’t research. True research has a high chance of failure, but if successful will provide a significant advance. The second problem is that the forensic science research grant proposals to the NIJ are reviewed by a secret old boy/old girl committee with an “I’ll scratch your back if you’ll scratch mine” philosophy. Lots of luck if you’re not a member of this clique!

  7. hedwig says:

    Here are a couple of occurrences:

    1) I reviewed a paper for a well-known European biochemistry journal, and rejected it first time. It came back revised less than 2 weeks later, and I rejected it a second time. I received an e-mail from the editor telling me of the decision to reject. The other reviewer, in the online system, was simply listed as “discontinued”, i.e. they probably didn’t want to review the revised version. Thus, I was the sole reviewer for the revision. Well, 3 months later the paper is in print, with an acceptance date 5 days after the editor’s e-mail telling me of the final decision to reject. All attempts to get the editor to explain exactly what happened during those 5 days have been fruitless. Notably, an additional author appears on the revised accepted paper – I seriously suspect that he was the other reviewer. He wangled his way onto the paper, then did a behind the scenes effort to get the paper in. Anyway, in the end the paper was retracted 1 year later, so I was right. No apology. Stone-walled by the journal and the editors.

    2) A paper came out in a cardiovascular journal recently. There was an editorial accompanying it, written by a very well respected scientist in the field. I decided to write a letter to the editors of the journal, protesting the paper (it contains some serious flaws), but before doing that I wrote to the scientist who did the editorial, to get his opinion – after all, as a younger investigator I don’t want to antagonize one of the big-shots in the field, because if the paper ends up being retracted then he will look like an idiot for endorsing it in an editorial. Anyway, the response I got is that basically he didn’t write the editorial, but it was written by the author of the article itself. I am still deciding what do do about this one, because the ramifications are huge.

  8. Jim Tarr says:

    All “peer review” should be required to be signed and dated by the person who submits it. To allow otherwise is to encourage a variety of abuses such as undisclosed conflicts of interest, dishonesty, etc.

    Those who would submit a “peer review” without signing their name to it are intellectual cowards.

    Jim Tarr
    2-14-08

  9. hedwig says:

    @Jim Tarr
    What you speak of IS required, in that most peer-review nowadays is done over the internet, with dedicated systems, so the editor knows who it is and can trace the reviewers easily. BUT, it is one thing for the editor to know who it is, and a completely different affair to pass that information on to the authors. If the authors find out, it can compromise the whole affair – they can for example make an erroneous claim that the reviewer is a competitor (very difficult for the editor to verify), and that the review is biased. The editor would then spend 90% of his/her time involved in “he said/she said” arguments, and would do a worse job of actually filtering the science. Alternatively if a reviewer knew their identity would be revealed to the authors, they might not be so “free” in their opinions, especially if the paper is from a senior person in the field who would crush you like a bug if they found out you’d trashed their paper. I got a paper from a senior person in the field a couple of weeks ago, and did not feel one bit of guilt about trashing it (it was really really bad). I may have been less harsh if I knew he would find out and call me on the phone, and threaten to prevent my grants getting funded (yes, there are people like that out there). It is essential to keep the reviewer’s identity anonymous from the authors, to avoid all sorts of problems. As long as the editor can trace the reviewers (which they ALWAYS can because they hand select them), there’s not a problem.

  10. Sylvester Adejo says:

    The whole thing is issue of trust. If one is review another person’ s work he/she should note that his/her integrity is at stake. The guiding principle is one’ s conscence. Simply apply the golden rule or else same may happen to you one day.

  11. Oleg Matveev says:

    The real problem is not honest or unhonest the existing peer review system. It is evident that precision of modern peer review system in selection of the most valuable scientific results and proposals is very poor. Real problem is how to evaluate, rank them much more precisely. One of the approach is described at my web site http://www.rankyourideas.com.

  12. Michelle Ulrich says:

    It is very difficult without the proper “handshake” to get proposal ideas funded, much less get quality manuscripts published. There are many obstacles, but amongst them are mendacious reviewers who either a.) do not want to be “scooped” on an important subject or b.) want young investigators to “pay their dues” rather than fund a proposal on the first try.

    Certainly reviewers have a bias that well-established researchers get the benefit of the doubt, which is one of the manifold reasons why there is so much crap out in the literature nowadays. The second reason is sloth. Too often I see materials where apparently the only work done in the field is by the PI. Those papers get published because of the ‘scratch your back’ system that does not allow for much in the way of solid research to be pursued.

    As for the ‘double blind’ review process (apparently which confused Dr. Maureta), that is a good idea in theory, but in practice would be nearly impossible because of the facilities that need to be listed in most grant proposals, and experimental sections of manuscripts.

    I believe the system of peer-review is broken, especially with the perception that federal funding is tightening. Too often the haves (who normally are an aged, conservative lot) are preventing the have-nots from their success. Perhaps when (if?) the top-heavy academic departments start seeing retirements will the situation (temporarily) change.

    Case in point: I have written entire proposals to funding agencies as a post-doc, but was not a PI. About half were funded. I took a junior faculty position and did not had the same “luck” – none were funded. I am now a research scientist who has gotten federal funding (with the same proposals, and in one case, the same exact proposal), but again, not as PI. Perhaps I should have been more collegial…

    The bottom line is a untenured faculty (who doesn’t have the secret handshake) has to prostrate herself for her “peers” for both proposal and manuscript acceptance. Her peers are capricious group who are not willing or able to accept newcomers to the community.

  13. John S says:

    We only needed to wait a week for the article “A Massive Case Of Fraud” to prove that even if the peer review system is honest, it is not effective.

    Another blow for journals justifying their high prices by the value of peer review. Put it all on the web and anyone can see the similarities.

  14. Lauren Murata says:

    “As for the ‘double blind’ review process (apparently which confused Dr. Maureta), that is a good idea in theory, but in practice would be nearly impossible because of the facilities that need to be listed in most grant proposals, and experimental sections of manuscripts.”

    Michelle,

    What exactly am I confused about? I know that authors are blinded to reviewers’ identities. I also know that reviewers are NOT blinded to authors’ identities. As I stated above, blinding reviewers to author’s identities would NOT be feasible because manuscripts are likely to contain other clues to the authors’ identities. One such clue, which I described above, is the described/proposed research itself, which is probably easily attributed by the other researchers in that field. Your example (the location of facilities) only strengthens my point.

  15. Dear colleagues,
    I’d like to offer you to visit Web site of our project “The Project of the Revision of the System of Publication and Estimation of Scientific Papers.” (http://electrochemist2.narod.ru/).
    I hope that the site materials will be interesting for you and your colleagues.
    It is to be regretted that materials of the project were not discussed at the C&EN.

  16. Dear colleagues,
    My name is Alexander Shagaev. I am Russian electrochemist. This paper is very interesting for me. I think that materials of our project ” Project of Revision of the Present System of the Publication And Estimation of Scientific Papers.” (http://electrochemist2.narod.ru/index.html) will be interesting for you also. With best regards Alexander Shagaev (Russia)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: