On 2016 Apr 03, Lydia Maniatis commented:

I don't see why this discussion is still going on. Anyone who has read PubPeer's blog posts will understand that a. they have solid arguments based on the public interest and thus that b. they have no reason to back down on their publishing model which c. is very popular for users and d. no one can make them do it against their will. End of story. MB's claims, on the other hand, turn a blind eye to important facts.

Given MB's general hostility to anonymity in the context of scientific discourse, I'm having trouble imagining how he rationalises the anonymity of reviewers of submissions for publication. Why isn't it a problem that the potential critic of the submission is cravenly hiding (as he might put it) behind anonymity? What's the danger of being up front?

On 2016 Apr 02, Boris Barbour commented:

None

On 2016 Apr 02, Boris Barbour commented:

The important issue here is the reluctance/refusal of Michael Blatt to engage in a substantive analysis of the pros as well as the cons of anonymous commenting, a recurring theme in this thread.

The questions about the negotiations to publish a reply to Michael's original editorial in Plant Physiology represent a distraction from the more fundamental issues. However, because he is creating the impression that we have been untruthful and have something to hide, I reluctantly respond again on this point.

Michael, as I said, we felt your initial suggestions were unfair. They did improve when we pushed back, as I have been happy to confirm. However, as I did not spread misinformation, I'm not apologising for it. Specifically, that you attempted to impose constraints that (at least we felt) were unfair is true, so I'm not apologising for having said that either.

You requested permission to publish our email exchange. We do so below.

Some context will be helpful in understanding why we did not reach agreement. Michael had just published a 3-page editorial in which he deployed a combination of insinuation and plausible deniability to associate us with notions such as voyeurism (peeping at published articles...), going through dirty laundry and money grabbing. A completely neutral editor-in-chief covering a controversial issue might have considered allowing us a reply of the same length, published at the same time (we must have missed the invitation to do so...) or as soon as possible afterwards. But Michael was in the conflicted position of also being chief prosecutor, having turned Plant Physiology into his personal propaganda vehicle (three editorials attacking PubPeer so far). Although there was a degree of mistrust and we were skeptical that he would be able to dissociate his conflicted roles, we decided to explore the possibility of replying to the same audience. As Michael was well aware, speed was of the essence, with any delay affording him the comfort of monopolising the "news cycle". From our point of view, truth was struggling to get her boots on, and any delay would reduce the effectiveness of our response.

I have edited email adresses, boilerplate (signatures and embedded emails) and some whitespace in the chain below (posted in one or more comments below because of PMC size limits). Emphasis and text in square brackets have been added by me.

On 2016 Apr 02, Michael R Blatt commented:

Boris

I shall take this as your apology for propagating misinformation and suggesting that I “tried to impose various unfair constraints” on any response from your and your PubPeer colleagues.

Thank you for your (eventual) candour.

Mike

On 2016 Mar 31, Boris Barbour commented:

Dear Michael,

There is no real contradiction on the format negotiations. After some to-and-fro, your final offers were indeed relatively generous given "journal constraints". But by that time we had come to realise that we didn't need to satisfy ourselves with the "halfway" we were working towards. As anybody who has tried to correspond with a journal knows, the process can feel extremely restrictive compared to the freedom and immediacy of a blog post.

Anyway, the point of the above comment was to correct rapidly three possible implications ambiguously left open (and predictably seized upon by a twitter denizen): i) that you'd offered to give us equal airtime, ii) spontaneously, and iii) that we hadn't felt able to counter your arguments. That's why I gave a bit more background about the process.

On 2016 Mar 31, Michael R Blatt commented:

Boris

Again, I think you do me a disservice. Given the constraints of publishing in a scientific journal, I did my utmost to meet you halfway and not limit your effectiveness (for example, engineering a way around the time lag between submission, acceptance, and final publication so that your response might be published instantly). The email thread I refer to above bares this out. Once more, I am happy to share it here with your approval (yes, vetos can work both ways).

As for any mis-reading of the latest editorial (or any of the others, for that matter), I can only say that it is always possible to take a statement out of context and twist it into someting altogether different, no matter how precise the text. The context in this case was of an offer to respond in Plant Physiology, nothing more or less. If this was misconstrued to imply that you declined to make any response whatsoever (which, clearly, is not the case as you've noted), I can only say that this was not my intention.

On 2016 Mar 31, Boris Barbour commented:

Dear Michael,

You made several suggestions ("tried to impose constraints") that would, coincidentally of course, have limited the effectiveness of our reply to your editorial: shorter, later, hobbled, elsewhere. I didn't invent the list of issues I gave (and the problem was of course your veto not ours). Sure, the restrictions weren't untypical of journal correspondence and the power that editors are accustomed to wielding. And, yes, we might have been able to work something out. But we decided it was just not worth the struggle when we could post instantly in our desired format. In one way we acknowledge that was a mistake, because it has proven exceptionally difficult to engage you in any discussion of specifics.

We didn't say that the statement about us declining to publish in Plant Physiology was wrong, we just felt that it might mislead people (just as it misled Leonid Schneider) into believing that we had avoided the debate. Those tempted by that interpretation are invited to read this thread, and, of course, our replies to your editorials.

On 2016 Mar 31, Michael R Blatt commented:

None

On 2016 Mar 31, Boris Barbour commented:

Michael Blatt has published yet another editorial attacking PubPeer, containing an incomplete and potentially misleading statement:

"An offer to respond had been made to Brandon Stell of PubPeer, who ultimately declined."

We at PubPeer requested the opportunity to put our case to the readers of Michael's original editorial. He agreed in principle but tried to impose various unfair constraints ("no more than 3 points", "limit on text", interleaved rebuttals, publication veto, etc). In addition, as the timing of the new piece shows, we might have had to wait 5 months and Michael's decision for our reply to appear. The process reminded us why journal correspondence sucks so much and indeed why PubPeer was created in the first place. So we decided to publish our response immediately as a blog post.

Readers of this thread can follow our largely unsuccessful attempt to draw Michael into a joint, open and even-handed evaluation of the pros and cons of anonymous post-publication peer review. Given his preference for preaching (several times) to a captive and passive audience, we shall just have to wait and see how scientists in general and the plant community in particular, which is by no means united on this matter, votes with its feet.

On 2016 Mar 23, Jaime A. Teixeira da Silva commented:

From my personal experience a PubPeer from what I have observed is that there are all kinds of anonymous: those with a desire to hold an academic discussion, as if in a journal club; those with valid, succinct claims; those with wild, but plausible claims; those with wild, and sometimes unsubstantiated claims; those with simple observations or concerns; those who have come to troll; those who have come to abuse, make libelous comments, or harass.

Comments by the last group tend to be flagged and removed by the moderator(s), who are likely Boris Barbour and the other two PubPeer management figures. But all others remain, which is what makes PubPeer so conflictual, because it has attractive and highly unattractive aspects.

One will never know the identity of all these types of anonymous commentators, and except for the use of extremely bad language, slang, or downright libelous name-calling (e.g. calling someone a fraud), we need this type of platform to allow a free level of discussion that is never possible with any journal's comment platform. Most scientists will know how to differentiate the wheat from the chaff, and can discern valid criticisms or concerns from noise, evasion or deflection. The most important thing is if what is written, either as a bounce from PubMed Commons, or directly here at PubPeer, has any value, and to whom?

In my opinion, PubPeer serves for me as a platform to begin to show how sad the state of affairs is in plant science. Comments might not always be perfect, or tone-perfect, and you will find that will ultimately always create enemies or irritate those who oppose you, or your ideas. But this is a risk that comes with using an anonymous tool. Those who use PubPeer should know that these risks exist.

I think the anonymous vs named argument is a dead horse. It is quite obvious that there are three groups: those who understand, and appreciate, anonymity; those who will always be skeptical and critical of it, and ultimately shun it; and those who see some benefit, and also some risk, but who would likely never venture to use it, either because they are of a traditional class of scientists/editors, or because they fear.

I think that ultimately that what is lacking is the respect and recognition of one of these groups of the other two. And because there is a lack of recognition and/or respect, there will always be frustration and passionate defense of the home turf opinions. That is so evident in the responses by select members of the public or scientific community to Prof. Blatt's two editorials.

I can personally see where Prof. Blatt's fears and concerns are coming from, and I respect his opinion and point of view, because that's all the editorials represent. I might not necessarily agree with his views in their entirety, but I understand that we need to respect his position, or at worst, respect his position in a civil way. Ultimately, one has to ask: has Blatt been a valuable asset to the plant science community, even if within his own restricted niche at Plant Physiology, and has something positive come from these two editorials?

The answers to these two questions are more than evident.

I thus suggest a new trajectory, at least for plant science. PubPeer has shown, in already hundreds of cases, that there are problems with the plant science literature. Problems that neither leaders like Blatt, Kamoun, or Zipfel knew or detected. But problems that ultimately drew them into the conflict that is, broadly speaking, a literature that is problematic, even in the top level plant science journals.

We only need two things to make this recipe of correcting the literature work: a) the recognition that there are problems and that they need to be corrected; b) action, i.e., getting editors and publishers to recognize these errors formally, and correcting the ills of the traditional peer review process.

Unless a) and b) take place, this whole discussion surrounding the anonymous voice is meaningless.

In closing, I should add that not all anonymous commentators are the same, and that not all necessarily agree with the position, or choice of words, employed by Boris Barbour or PubPeer.

On 2016 Mar 28, Jens Sommer commented:

Dear Michael,

I know about the complications of double blind review and disclosure of reviewers. As the reviewers see the reference list and self reference is part of scientific writing it is almost imposible to hide author's identity, so most authors don't care.

About disclosure of the reviewers: It is not essential to insist on disclosure. Let the reviewers decide. In addition allow the authors to rate the quality of the reviews (anonymously?).

As long as we want to improve our knowledge (and scientific progress) we will need skilled reviewers, not just many reviewers. But again, this has been discussed before.

On 2016 Mar 22, Michael R Blatt commented:

Dear Jens,

Forgive me for not responding to your last paragraph. These issues have been addressed time and again (e.g. in my editorial and in the discussion below with Boris Barbour).

As for your two, numbered points, you may be aware that several journals have tried and/or do offer a double-blind review process, including several of the Nature journals. Only a very tiny percentage of authors ever take up this option, however, and realistically it is often difficult to hide the authors' identities (see the editorial from Chris Surridge in Nature Plants last September for more information).

Complete disclosure, as you propose in your second point (and if I understand you correctly), is also problematic. I think most editors would argue that, were they to insist on such disclosure, then it would be very difficult indeed to secure reviewers. Of course, editors are generally acknowledged; all journals publish the list of their editors on the journal masthead and some journals include the names of the handling editor with each published article (e.g. PNAS). As for the social contract involved in considering a manuscript for publication, I have commented on this in my editorial of last October.

On 2016 Mar 21, Jens Sommer commented:

Thank you for the editorials and thanks to all comments. This gives hope for the future of the scientific community and scientific progress.

Maybe it is just the point in time, when we need or accept anonymous comments.

Is it important to have

1) a double blind review process (authors, reviewers) and 2) a complete disclosure (authors, reviewers and editors) after rejection or acceptance?

While the first is essential to get an unbiased review, I expect the second to improve the quality of reviews and thus the quality of articles. At least my idea of reviewing an article is to improve it, and the communication with the authors is more like an anonymous discussion.

As the review process includes more than one reviewer it is interesting to see that the process sometimes fails completely (false acceptance). So if it takes time to review an article properly, why shouldn't we see the names of the reviewers, who supported the authors in getting their work published?

Finally, when the article is published and a discussion starts any reasonable comment will be welcomed - anonymous or named. Do we really need regulation if there is more than one free platform with high-quality comments and good usability? Why don't we let people figure out what suits their needs best?

On 2016 Mar 23, Daniel Corcos commented:

I would be grateful if you could show me potentially toxic comments. As for the toxicity of anonymous reviewers and bad editor choice, I know too many examples, but I only have to mention the case of CRISPR role discovery by Francisco Mojica, which has been rejected by many high impact journals for 2 years (http://www.cell.com/cell/pdf/S0092-8674(15)01705-5.pdf). You may say that two years delay for a basic paper does not harm much, but when it comes to medicine, it can be terribly harmful.

On 2016 Mar 18, Michael R Blatt commented:

Daniel, I guess we will have to disagree on the toxicity potential of anonymous comments.

On 2016 Mar 18, Daniel Corcos commented:

Mike, I agree that PubPeer comments can be wrong and misleading, but their advantage is that they can be seen by everybody. I prefer anonymous comments that I can read to hidden criticism. Rejecting a paper for spurious motives makes certainly more harm than comments in PubPeer. I must say that I am not in favor of anonymity and I hope that this debate will lead to openness of review. With time, this would allow a full evaluation of the harm done by some renowned scientists to the progress of knowledge.

On 2016 Mar 18, Michael R Blatt commented:

Daniel, I agree that overtly offensive comments are usually obvious as such to the reader. What is much more worrying about anonymous commenting is its potential to spread untruths and to do so without accountability. So I cannot agree with you that anonymous commenting is always bland and harmless. Subtle rumours can have just as serious consequences for an individual as an undue rejection.

On 2016 Mar 17, Daniel Corcos commented:

Mike, we are often aware that major breakthrough papers had been initially rejected by many journals, especially when the authors were not renowned. It would be interesting to know who were the experts and if the editor has ceased to ask them to review papers after considering that rejection was undue but for ordinary people like us, reviewers remain anonymous. On the other side, offensive anonymous comments have no great consequences because readers can judge by themselves, whereas undue rejection has much more consequences for the authors and for science.

On 2016 Mar 17, Michael R Blatt commented:

Daniel, it is a common mistake to equate blind (confidential) peer review with anonymity. There is a world of difference here. In assessing the potential of a manuscript for publication, an editor will often turn to one or more known experts in the field for their opinions. The editor will know the identity and expertise of these individuals, so their advice is most certainly not anonymous. Please have a read of my editorial from October 2015.

Of course, we might discuss the pros and cons of open (non-confidential) review; however, this is not the same discussion as that of anonymity. Mike

On 2016 Mar 14, Daniel Corcos commented:

If anonymity "has no place in scientific critique" as Blatt argues, then it should have no place in peer review. For this reason, Blatt's position is untenable.

On 2016 Mar 22, Michael R Blatt commented:

As you will have noted, I too am in favour of open (non-anonymous) debate.

On 2016 Mar 19, Lydia Maniatis commented:

I'm against artificial and unnecessary obstruction of open scientific debate, either via selection by a self-proclaimed "meritocracy" or any other means. Open debate is messy but the alternatives, arguably, are messier.

On 2016 Mar 17, Michael R Blatt commented:

Hello Lydia. Do I understand you correctly, then, that you are in favour of meritocracy in science?

Mike

On 2016 Mar 17, Michael R Blatt commented:

None

On 2016 Mar 08, Lydia Maniatis commented:

I would like for the moment to single out the following argument/counterargument from the article, because it argues that science is not/should not be democratic.

View attributed to those in favor of anonymity: “Anonymity is essential to protect fundamental rights and free speech in a global democratic society.”

Blatt's rebuttal: “Yes, science is a “massively cooperative undertaking,” to quote one of my PubPeer commenters,1 but that does not mean it is democratic. Science requires substantial training; its foundations are logic and reasoning; it builds on the merits of knowledge and expertise; it is not a ‘one man, one vote’ endeavor with universal enfranchisement. To argue otherwise is manifestly absurd. “

First, arguments appealing to “manifest absurdity” are not worthy of scientific debate, whether signed or anonymous. Second, logic and reasoning are the property of every human being, and their use is very often and very demonstrably absent from the scientific literature (which is the reason editors so fiercely protect the published literature from dangerous “Letters to the Editor.”) Third, there is no degree that can confer infallibility to any individual; likewise, no individual should be denied the right to have their arguments evaluated ON THEIR MERITS (something very different from the misleading "one man one vote" argument.)

People who don't feel comfortable with the responsibility to defend their positions (scientific and otherwise) by argument and not on the basis of membership in a closed society of initiates do not understand how science progresses, and how it stalls.

On 2016 Mar 08, Lydia Maniatis commented:

I think that the “warped worldview” arguments made by Dr. Blatt and others in opposition to anonymous critiques could more appropriately be levelled at them. Their main concern seems to be that the material welfare and (relatedly) personal reputation of individuals within the scientific community will be threatened by anonymous trolls whose only aim is to sully reputations by suggestive but ill-founded attacks. (The critics of anonymity seem less concerned about the benefits to the public interest that PubPeer's editors have demonstrated and documented to have followed from the enabling of anonymous posting).

Let's assume that such trolls exist and are even in the majority (though I don't believe this to be the case). How, in the context of a healthy, intellectually sharp, critically-minded scientific community will their efforts have an influence? Why would the targets' astute colleagues, grant reviewers, academic employers, etc. allow specious, unmerited criticism to influence their views or choices? If, on the other hand, decision-makers in the community are not equipped to separate the wheat from the chaff (whether we are referring to criticism of scientists or the scientists academic productions), then this is indeed a warped world, and in such a world the documented public interest value of anonymous criticism surely outweighs any nuisance value to individuals. Relatedly, Dr. Blatt states early on in his editorial that he is “Putting aside the issues of policing for fraud and whistleblowing for the moment....” I would be interested if he would come back to this issue, and particularly in his plan for creating a system where anonymity could be automatically enabled for comments falling in the category of “policing for fraud and whistleblowing” while denying it for silly comments. Who would decide, a priori, which commenters/comments get the privilege?

On 2016 Mar 08, Lydia Maniatis commented:

Blatt says: "Ultimately, it is a warped worldview, indeed, in which scientists are so fearful of engaging that they never challenge others’ research and ideas openly, whether online or in publication."

Barbour notes that: "Plant Physiology has no functional feedback mechanism..."

Perhaps Dr. Blatt should consider helping to unwarp the world by enabling signed, open publication of criticism in his journal...

On 2016 Mar 28, Michael R Blatt commented:

Dear Boris:-

Thank you, but no apologies are necessary.

However, I think we are now going around in circles. You continue to use the ‘volume’ argument which I am not prepared to accept and, if you think about it, I suspect neither are you. As devil’s advocate, I could point out that the volume of good research still overwhelming outweighs the bad (see the references in my first editorial and further discussion in the editorial to be published next week), just as you argue in favour of the “overwhelming majority” of PubPeer comments that you claim are useful compared to the “tiny minority” that are antisocial, ethically unsound and/or defamatory.

You will see my point, I hope. So let’s not beat this one to death. We are not going to resolve the problem by defending corners or looking for the lowest common denominator. I am convinced that to find a solution it will be necessary to look outside the box, so to speak.

I’m happy to continue our discussion, but I don’t think anyone is particularly interested in following this thread much further. So I suggest we now do so by email. If we do come to a solution, then of course we will want to share this with the community, either through PubMed Commons or some other way.

Mike

On 2016 Mar 27, Boris Barbour commented:

Dear Michael,

I apologise if I have misconstrued your position, which I thought was a good deal more negative.

Anyway, let's work on the common ground a little.

In favour of anonymous comments, some disseminate useful information that: 1) reduces wasted research based on unreliable research, 2) diminishes errors in clinical trials and medical guidelines arising from unreliable publications, 3) therefore saves careers, taxpayers' money and lives.

Against anonymous comments ("abuse, misrepresentation, sock-puppetry, and other antisocial or ethically unsound behaviours"): 4) unjustified denigration of reputations, 5) no declaration of conflicts of interest, 6) no information about commenter's status, 7) no discussion of equals

Let's weigh the "costs" and "benefits" of the anonymous comments on PubPeer as they are; we'll worry about how to influence their nature later. So do 4-7 outweigh 1-3, taking into account their relative frequencies?

I would say that the most extreme negative outcome is damage to somebody's reputation. But how much damage can be done without convincing ammunition? Remember also that researchers can always defend themselves by explaining, showing data etc, in the case of a truly unfortunate misunderstanding. So, even if reputations probably can be damaged slightly by the ill-intentioned, I would contend that it is difficult to cause severe unjustified damage to somebody's reputation on PubPeer.

In contrast, it is highly likely that rapid dissemination of information can save a PhD or post-doc from wasting 6 months to a year trying to build on some exciting but unreproducible result. In today's competitive environment that unproductive time may spell the end of a young career, and taxpayer's money will have been poorly spent. There are no doubt clinical trials in progress based upon flawed research - deeply unethical - and there are - hopefully rare - cases of flawed research causing erroneous medical decisions. In these cases, rapid dissemination of information could save lives. Examples where one wishes information had been made public (and acted upon) earlier include the Poldermans case mentioned in our blog and the Wakefield MMR vaccination scandal.

So in terms of extreme outcomes, do you agree that saving lives, taxpayers' money and research careers outweighs slight damage to researchers' reputations?

What about frequencies of the different types of comments? From having read nearly all of the ~50000 comments that have appeared on PubPeer over 3.5 years, I am happy to report that the overwhelming majority report valid signs of low-quality research or misconduct - the sort of comment that could lead to benefits 1-3. Only a tiny minority might be suspected of trying to run down the reputations of other researchers unfairly.

Based upon the importance of disseminating information to readers and the observed low frequency of comments appearing to abuse the system, we have concluded that the anonymous comments appearing on PubPeer are very clearly beneficial on average. Therefore they should be encouraged. Do you agree?

A question that I would like to keep separate and analyse next is what can be done to tilt the balance further towards beneficial comments (including your desire to convert anonymous to nonanonymous comments).

On 2016 Mar 25, Michael R Blatt commented:

Dear Boris,

There’s nothing grand in these statements nor are they exempt from a cost-benefit analysis. It just happens that my measures of cost and benefit are (obviously) different from yours. Of course, it may be that we can still find common ground, and I would hope this is the case.

As to your question “Is it a good thing to alert readers … to possible problems?”, clearly the answer is yes. I have said so repeatedly in my editorials and here on PubMed Commons. However, in my opinion, this needs to be done in a way that does not open the door to abuse, misrepresentation, sock-puppetry, and other antisocial or ethically unsound behaviours. I don’t think this is a particularly difficult concept, even if its solution is more complex in practice.

On 2016 Mar 22, Boris Barbour commented:

Dear Michael,

You appeal to "foundational arguments" and "principles", but sounding grand doesn't exempt them from a cost-benefit analysis.

I'll ask you just one question, the one you have avoided answering over 2 editorials and all the discussion here: is it a good thing to alert readers of publications to possible problems?

We could call it the foundational principle of PubPeer...

COI statement: see my original post.

On 2016 Mar 22, Michael R Blatt commented:

Dear Boris,

I really do not think that we are so far apart in our views. We both are dismayed by some of what we see in scientific publishing and communication today, and we both want the same for the scientific community as a whole. Where we differ is only in some details of the means to this end.

The point you raise is of measures, ‘averages’, and quantity, rather than of principle. I do not doubt that there are many comments on PubPeer that are thoughtful and constructive. I certainly never suggested that all comments on PubPeer “abuse the system” (nor did I ever suggest coersion, so let us not confuse the issue here). The point on which we differ is whether the quantitative argument for anonymity that you pose outweighs the foundational arguments I have set out against it. I think not.

You raise the analogy to the utility of cars and whether these should be banned. Of course analogies are poor vehicles (pun intended) for ideas, but let’s follow it for a moment. It would be virtually impossible to ban anonymous commenting from social media, just as it is impossible to ban reckless driving (I recall you had this discussion with Philip Moriarty previously). However, this is not to say that either should be actively encouraged. There are norms for interpersonal interaction that we generally follow and that protect civil society (e.g. accountability), just as there are rules of the road and legal requirements (e.g. the need for a driving license) that are there to protect us when we are on the road.

I think it is always important to look for other ways to a solution. Answers sometimes come from taking an entirely different perspective rather than looking for the common denominator. So, to follow your analogy one step further, rather than banning cars (and anonymous commenting), would it not be better to make them less attractive as a whole while making the use of public transport (and of open, accountable commenting) more attractive? Are we not both in a position to influence the process of PPPR?

I alluded, at the end of my March 2016 editorial, to what I hope will be an approach to such a ‘third solution.’ It comes straight out of discussions with Leonid Schneider who, I think you will recall, was originally one of my fiercest critics last October. I am convinced this alternative is worth a try and, at this point, have a number of my opposite numbers from other publishers on board. You may be convinced as well in due course. Again, I hope that I will have much more to say on this matter later this year.

On 2016 Mar 19, Boris Barbour commented:

The key issue is whether or not anonymous comments are beneficial ON AVERAGE. If they are or can be made so (PubPeer implements guidelines to favour useful comments), then such comments should be encouraged. Your arguments focus purely on the negatives and you systematically avoid consideration of the benefits of comments that happen to be anonymous. We can agree that it is possible to abuse anonymous commenting. And anonymity does enable commenters to forego in-depth discussion with meritorious professors. However, a balance needs to be struck and you have still made little attempt to do that. Thus, even if abuse is possible, that doesn't mean that all comments do abuse the system. In fact, the great majority of anonymous comments on PubPeer are perfectly factual and some highlight matters of genuine importance to readers, disseminating that information without delay. At PubPeer we have weighed both the advantages and the disadvantages of the anonymous comments we publish. We are convinced that their overall effect is overwhelmingly beneficial, despite a small number of awkward cases. So we shall continue to enable anonymous commenting.

Having been around the houses of this argument a few times without making much progress, maybe an analogy will be helpful. Would you ban cars because sometimes people get run over? Or would you take into consideration the fact that they are a useful means of transportation? I'd like to see you take into consideration the potential benefits of the content of anonymous comments.

We agree that we should all strive to create a system in which researchers feel able to comment freely and transparently. But we don't have a magic wand to create that environment. You at least are in a position of power to implement some changes, but that will require supportive and constructive action, not coercion. The coercive approach has failed in the past: our direct experience on PubPeer has shown that many useful comments will only be made if anonymity is available. In other words, there is no way to make all useful commenting non-anonymous, you can only suppress the majority of comments, including many useful ones, by (hypothetically) forbidding anonymity.

You continue to confuse research that contains known flaws (including overinterpretation) when produced with that which doesn't. Although all research is indeed potentially, eventually falsifiable, the use of small sample sizes, inappropriate statistics, unverified cancer cell lines etc, etc (the list is long) is known today to generate unreliable research. You can't expect researchers to predict the future, but it's not unreasonable to ask them to avoid known mistakes (respecting the "state of the art"). Moreover, isn't it precisely your job as a journal editor to draw this line? Do you really not recognise this distinction? In any case, PubPeer simply allows comments and questions; the site makes no judgement.

Regarding the arsenic life paper, I'll leave you, as a practising editor-in-chief, to interpret the (admittedly inconsistent) COPE guidelines on the matter. Here are a couple of key quotes:

"Journal editors should consider retracting a publication if ... they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)."

"Retraction should usually be reserved for publications that are so seriously flawed (for whatever reason) that their findings or conclusions should not be relied upon."

COPE guidelines

Finally, we obviously agree that "science does not end with publication". Nobody at PubPeer has ever said otherwise. Indeed, the whole raison d'être of the site is to enable science to continue after publication, something that traditional journals have not always embraced.

COI statement: see my original post.

On 2016 Mar 18, Michael R Blatt commented:

I’ll reply to both your comments here, Boris. I did address all of the standard, conceptual arguments around anonymity in my second editorial, and I will be discussing some of these and other aspects of anonymity again next month with Jaime Teixeira da Silva.

I believe you wish to point out, as your central argument, that the traffic on PubPeer is far greater than on PubMed Commons, for example, and you ascribe this to encouraging anonymous comments. Your numbers may be correct – I am not in a position to comment one way or the other – but I do dispute your underlying assumption that traffic volume equates with scientific value. I raised this point in my October 2015 editorial, as did Philip Moriarty both in the PubPeer threads that followed and in his discussion with you in the Times Higher Education in December 2015.

I maintain, furthermore, that anonymous commenting encourages grubby comments and nefarious behaviours that undermine the very scientific community you want to build. So, in my opinion, encouraging anonymous commenting is counterproductive and, ultimately, self-defeating. Again, you will find all of the arguments in my editorials, so I’ll not revisit them here.

As for my misinterpreting your definition of ‘ultimately unreliable’ “in the sense of ‘unreproducible’, ‘low-quality’, ‘known to be wrong’, [and] ‘overinterpreted’”, I agree there is such a thing as ‘bad science.’ Ben Goldacre has much to say about this. However, I think you need to be more cautious in calling for sweeping retractions on the basis of your definitions. Can you issue a blanket statement of unreproducibility without first seeking to reproduce each set of data and explaining why it is unreproducible, for example? Where do you draw the line between interpretation and overinterpretation? And when does overinterpretation become grounds for vilification?

Again as a specific example, I agree that the Science paper on arsenic-based life was overinterpreted and included experimental methods that were insufficient to meet the exacting standards expected for such a claim. On this basis alone there is a strong argument to say that it should never have found its way into the journal. However, my understanding is that the results were not low-quality per se; they were demonstrably reproducible; and they did lead (ultimately!) to detailed knowledge of a transporter with a remarkable selectivity for phosphate over the structurally similar arsenate anion. As I noted before, “science does not end with publication. Publication is only the beginning of scientific debate. Progress often arises from what, in hindsight, is ‘ultimately unreliable’, and its cornerstone is open debate.”

On 2016 Mar 07, Boris Barbour commented:

The words 'ultimately unreliable' in the title of this editorial are a quote from the text of our blog Vigilant scientists. By accident or by design, Blatt deforms their meaning completely. We meant "unreliable" in the sense of "unreproducible", "low-quality", "known to be wrong", "overinterpreted", while "ultimately" simply added emphasis (meaning something like "most importantly"). In contrast, Blatt interprets the word pair to include the meaning "eventually improved upon". In other words, we were discussing research that is of low-quality or wrong according to the state of the art at the time of publication, while he lumps such poor work with outstanding research containing no known defects at publication but upon which even greater discoveries are subsequently built. Thus, he gives the example of Hodgkin and Huxley's explanation of the action potential building upon Cole and Curtis' measurements of axonal impedance, characterising the latter authors' work as "ultimately unreliable". Nothing could be further from our intended meaning, which should have been abundantly clear from the context of the blog. In particular, we used the terms "unreliable" and "unreproducible" interchangeably, gave numerous examples and references relating to unreproducible and low-quality research, and gave no examples of the sort Blatt mentions.

So there is a clear criterion of reliability that Blatt did not consider: does a paper contain known problems according to the state of the art at the time of publication? Papers that fail this test are "unreliable" and the relevant information should be disseminated to the readers. Applied to examples in the editorial, this test would classify the arsenic life paper as unreliable and Cole and Curtis as reliable, while Blatt considers both to be "ultimately unreliable". Coming from the editor-in-chief of a high-quality journal, this seems to be questionable relativism.

COI: see previous post.

On 2016 Mar 06, Boris Barbour commented:

This editorial by Michael Blatt, editor-in-chief of Plant Physiology, follows up a previous one, Vigilante Science; both attack the anonymous commenting enabled by PubPeer (see "COI" below). PubPeer has responded to both editorials at Vigilant scientists.

In his follow-up, Blatt completely avoids addressing our central argument in favour of anonymity, which is that our priority as a community should be to disseminate information about publications to readers and users as rapidly and as widely as possible, a process encouraged by anonymity. As Plant Physiology has no functional feedback mechanism and because Blatt has refused to join any discussions on PubPeer, maybe he would like to respond here, at least to address our principal argument in favour of anonymous commenting?

Potential conflicts of interest: I am a co-organiser of PubPeer and wrote most of their two blogs on this subject. These views are expressed in a personal capacity, not as an official PubPeer position.

On 2016 Mar 06, Boris Barbour commented:

This editorial by Michael Blatt, editor-in-chief of Plant Physiology, follows up a previous one, Vigilante Science; both attack the anonymous commenting enabled by PubPeer (see "COI" below). PubPeer has responded to both editorials at Vigilant scientists.

In his follow-up, Blatt completely avoids addressing our central argument in favour of anonymity, which is that our priority as a community should be to disseminate information about publications to readers and users as rapidly and as widely as possible, a process encouraged by anonymity. As Plant Physiology has no functional feedback mechanism and because Blatt has refused to join any discussions on PubPeer, maybe he would like to respond here, at least to address our principal argument in favour of anonymous commenting?

Potential conflicts of interest: I am a co-organiser of PubPeer and wrote most of their two blogs on this subject. These views are expressed in a personal capacity, not as an official PubPeer position.

On 2016 Mar 07, Boris Barbour commented:

The words 'ultimately unreliable' in the title of this editorial are a quote from the text of our blog Vigilant scientists. By accident or by design, Blatt deforms their meaning completely. We meant "unreliable" in the sense of "unreproducible", "low-quality", "known to be wrong", "overinterpreted", while "ultimately" simply added emphasis (meaning something like "most importantly"). In contrast, Blatt interprets the word pair to include the meaning "eventually improved upon". In other words, we were discussing research that is of low-quality or wrong according to the state of the art at the time of publication, while he lumps such poor work with outstanding research containing no known defects at publication but upon which even greater discoveries are subsequently built. Thus, he gives the example of Hodgkin and Huxley's explanation of the action potential building upon Cole and Curtis' measurements of axonal impedance, characterising the latter authors' work as "ultimately unreliable". Nothing could be further from our intended meaning, which should have been abundantly clear from the context of the blog. In particular, we used the terms "unreliable" and "unreproducible" interchangeably, gave numerous examples and references relating to unreproducible and low-quality research, and gave no examples of the sort Blatt mentions.

So there is a clear criterion of reliability that Blatt did not consider: does a paper contain known problems according to the state of the art at the time of publication? Papers that fail this test are "unreliable" and the relevant information should be disseminated to the readers. Applied to examples in the editorial, this test would classify the arsenic life paper as unreliable and Cole and Curtis as reliable, while Blatt considers both to be "ultimately unreliable". Coming from the editor-in-chief of a high-quality journal, this seems to be questionable relativism.

COI: see previous post.

On 2016 Mar 08, Lydia Maniatis commented:

Blatt says: "Ultimately, it is a warped worldview, indeed, in which scientists are so fearful of engaging that they never challenge others’ research and ideas openly, whether online or in publication."

Barbour notes that: "Plant Physiology has no functional feedback mechanism..."

Perhaps Dr. Blatt should consider helping to unwarp the world by enabling signed, open publication of criticism in his journal...

On 2016 Mar 08, Lydia Maniatis commented:

I think that the “warped worldview” arguments made by Dr. Blatt and others in opposition to anonymous critiques could more appropriately be levelled at them. Their main concern seems to be that the material welfare and (relatedly) personal reputation of individuals within the scientific community will be threatened by anonymous trolls whose only aim is to sully reputations by suggestive but ill-founded attacks. (The critics of anonymity seem less concerned about the benefits to the public interest that PubPeer's editors have demonstrated and documented to have followed from the enabling of anonymous posting).

Let's assume that such trolls exist and are even in the majority (though I don't believe this to be the case). How, in the context of a healthy, intellectually sharp, critically-minded scientific community will their efforts have an influence? Why would the targets' astute colleagues, grant reviewers, academic employers, etc. allow specious, unmerited criticism to influence their views or choices? If, on the other hand, decision-makers in the community are not equipped to separate the wheat from the chaff (whether we are referring to criticism of scientists or the scientists academic productions), then this is indeed a warped world, and in such a world the documented public interest value of anonymous criticism surely outweighs any nuisance value to individuals. Relatedly, Dr. Blatt states early on in his editorial that he is “Putting aside the issues of policing for fraud and whistleblowing for the moment....” I would be interested if he would come back to this issue, and particularly in his plan for creating a system where anonymity could be automatically enabled for comments falling in the category of “policing for fraud and whistleblowing” while denying it for silly comments. Who would decide, a priori, which commenters/comments get the privilege?

On 2016 Mar 08, Lydia Maniatis commented:

I would like for the moment to single out the following argument/counterargument from the article, because it argues that science is not/should not be democratic.

View attributed to those in favor of anonymity: “Anonymity is essential to protect fundamental rights and free speech in a global democratic society.”

Blatt's rebuttal: “Yes, science is a “massively cooperative undertaking,” to quote one of my PubPeer commenters,1 but that does not mean it is democratic. Science requires substantial training; its foundations are logic and reasoning; it builds on the merits of knowledge and expertise; it is not a ‘one man, one vote’ endeavor with universal enfranchisement. To argue otherwise is manifestly absurd. “

First, arguments appealing to “manifest absurdity” are not worthy of scientific debate, whether signed or anonymous. Second, logic and reasoning are the property of every human being, and their use is very often and very demonstrably absent from the scientific literature (which is the reason editors so fiercely protect the published literature from dangerous “Letters to the Editor.”) Third, there is no degree that can confer infallibility to any individual; likewise, no individual should be denied the right to have their arguments evaluated ON THEIR MERITS (something very different from the misleading "one man one vote" argument.)

People who don't feel comfortable with the responsibility to defend their positions (scientific and otherwise) by argument and not on the basis of membership in a closed society of initiates do not understand how science progresses, and how it stalls.

On 2016 Mar 14, Daniel Corcos commented:

If anonymity "has no place in scientific critique" as Blatt argues, then it should have no place in peer review. For this reason, Blatt's position is untenable.

On 2016 Mar 21, Jens Sommer commented:

Thank you for the editorials and thanks to all comments. This gives hope for the future of the scientific community and scientific progress.

Maybe it is just the point in time, when we need or accept anonymous comments.

Is it important to have

1) a double blind review process (authors, reviewers) and 2) a complete disclosure (authors, reviewers and editors) after rejection or acceptance?

While the first is essential to get an unbiased review, I expect the second to improve the quality of reviews and thus the quality of articles. At least my idea of reviewing an article is to improve it, and the communication with the authors is more like an anonymous discussion.

As the review process includes more than one reviewer it is interesting to see that the process sometimes fails completely (false acceptance). So if it takes time to review an article properly, why shouldn't we see the names of the reviewers, who supported the authors in getting their work published?

Finally, when the article is published and a discussion starts any reasonable comment will be welcomed - anonymous or named. Do we really need regulation if there is more than one free platform with high-quality comments and good usability? Why don't we let people figure out what suits their needs best?

On 2016 Mar 23, Jaime A. Teixeira da Silva commented:

From my personal experience a PubPeer from what I have observed is that there are all kinds of anonymous: those with a desire to hold an academic discussion, as if in a journal club; those with valid, succinct claims; those with wild, but plausible claims; those with wild, and sometimes unsubstantiated claims; those with simple observations or concerns; those who have come to troll; those who have come to abuse, make libelous comments, or harass.

Comments by the last group tend to be flagged and removed by the moderator(s), who are likely Boris Barbour and the other two PubPeer management figures. But all others remain, which is what makes PubPeer so conflictual, because it has attractive and highly unattractive aspects.

One will never know the identity of all these types of anonymous commentators, and except for the use of extremely bad language, slang, or downright libelous name-calling (e.g. calling someone a fraud), we need this type of platform to allow a free level of discussion that is never possible with any journal's comment platform. Most scientists will know how to differentiate the wheat from the chaff, and can discern valid criticisms or concerns from noise, evasion or deflection. The most important thing is if what is written, either as a bounce from PubMed Commons, or directly here at PubPeer, has any value, and to whom?

In my opinion, PubPeer serves for me as a platform to begin to show how sad the state of affairs is in plant science. Comments might not always be perfect, or tone-perfect, and you will find that will ultimately always create enemies or irritate those who oppose you, or your ideas. But this is a risk that comes with using an anonymous tool. Those who use PubPeer should know that these risks exist.

I think the anonymous vs named argument is a dead horse. It is quite obvious that there are three groups: those who understand, and appreciate, anonymity; those who will always be skeptical and critical of it, and ultimately shun it; and those who see some benefit, and also some risk, but who would likely never venture to use it, either because they are of a traditional class of scientists/editors, or because they fear.

I think that ultimately that what is lacking is the respect and recognition of one of these groups of the other two. And because there is a lack of recognition and/or respect, there will always be frustration and passionate defense of the home turf opinions. That is so evident in the responses by select members of the public or scientific community to Prof. Blatt's two editorials.

I can personally see where Prof. Blatt's fears and concerns are coming from, and I respect his opinion and point of view, because that's all the editorials represent. I might not necessarily agree with his views in their entirety, but I understand that we need to respect his position, or at worst, respect his position in a civil way. Ultimately, one has to ask: has Blatt been a valuable asset to the plant science community, even if within his own restricted niche at Plant Physiology, and has something positive come from these two editorials?

The answers to these two questions are more than evident.

I thus suggest a new trajectory, at least for plant science. PubPeer has shown, in already hundreds of cases, that there are problems with the plant science literature. Problems that neither leaders like Blatt, Kamoun, or Zipfel knew or detected. But problems that ultimately drew them into the conflict that is, broadly speaking, a literature that is problematic, even in the top level plant science journals.

We only need two things to make this recipe of correcting the literature work: a) the recognition that there are problems and that they need to be corrected; b) action, i.e., getting editors and publishers to recognize these errors formally, and correcting the ills of the traditional peer review process.

Unless a) and b) take place, this whole discussion surrounding the anonymous voice is meaningless.

In closing, I should add that not all anonymous commentators are the same, and that not all necessarily agree with the position, or choice of words, employed by Boris Barbour or PubPeer.

On 2016 Mar 31, Boris Barbour commented:

Michael Blatt has published yet another editorial attacking PubPeer, containing an incomplete and potentially misleading statement:

"An offer to respond had been made to Brandon Stell of PubPeer, who ultimately declined."

We at PubPeer requested the opportunity to put our case to the readers of Michael's original editorial. He agreed in principle but tried to impose various unfair constraints ("no more than 3 points", "limit on text", interleaved rebuttals, publication veto, etc). In addition, as the timing of the new piece shows, we might have had to wait 5 months and Michael's decision for our reply to appear. The process reminded us why journal correspondence sucks so much and indeed why PubPeer was created in the first place. So we decided to publish our response immediately as a blog post.

Readers of this thread can follow our largely unsuccessful attempt to draw Michael into a joint, open and even-handed evaluation of the pros and cons of anonymous post-publication peer review. Given his preference for preaching (several times) to a captive and passive audience, we shall just have to wait and see how scientists in general and the plant community in particular, which is by no means united on this matter, votes with its feet.