Peer-review is probably the most obscure part of the publication of scientific results. In this blog post, I would like to make the point that the best way to learn about it – except by being directly involved – is to read public peer review files. In addition, I will recommend some interesting or surprising peer review files, mostly for systems neuroscience, but also for some optics papers.

Public peer review files should be the standard

Peer review takes place before official “publication” in a peer-reviewed journal and is therefore usually not accessible to the reader. During the last years, this practice has changed, and more and more journals are now offering the peer review and rebuttal letters as supplementary files. This is done for example by the journal eLife, but also Nature Communications, sometimes for the SfN journals, and Nature. For some journals, like Nature, the authors can opt out of the peer review file publication (here is Nature’s policy). But, honestly, if you are not willing to share the reviews publicly, what do you want to hide? I think it should become a mandatory standard to share the reviews, with only specific reasons justifying an opt-out. (Update 2022-02-14: a Twitter-discussion on this topic.)

What to learn from peer review files

As a young researcher like a PhD student, it is rare to be involved in the review process, which, therefore, remains a black box. For me, it was fascinating to read my first peer review files on eLife maybe 5 years ago. It felt like the entire smooth surface of this paper started to crumble and give way to a more rich and more nuanced point of view. Looking back at this paper, this nuanced view was also somewhat included in the paper, but in such a smooth manner that it was difficult to extract without absolute expert knowledge.

Nowadays, I rarely read eLife papers until also the peer review section is online (which comes a couple days after the preliminary pdf). The reviewer comments provide an additional point of view that is very helpful in particular when I cannot fully judge the paper myself.

Additionally, reading such review files helps to write both better manuscripts, better rebuttal letters, and also better reviews. Also, gaining experience from these files prepares a bit for the sometimes very difficult experiences when receiving the reviews. Plus, reading those reviews makes the entire process a bit more transparent. When I wrote my first reviews for journals, I had seen only three or four reviews for my own co-author papers; but I had seen many more as public review files.

Let’s look into some examples that show what can happen during peer review. Paper links are in the title, links to the review files/sections thereafter.

Toroidal topology of grid cell activity (Review file)

When the preprint of this paper by Gardner et al. from the Moser lab appeared, I put it on my reading list, but never actually read it fully because I did not feel on top of the vast literature about continuous attractor models, and I did not even know whether a “toroidal” geometry of the activity space would be a surprising result or not.

Checking the peer review files after publication at Nature provided exactly this entry point that I had been missing before. In retrospect, all the information needed is also in the paper, but the more direct and less formal language in the reviews took me by the hand and showed me directly what the paper is about. The summaries at the beginning of each review provide a very useful second perspective on the main messages of the paper.

Deep physical neural networks (Review file)

A few weeks ago, I noticed an interesting article published at Nature on “deep physical neural networks”. I checked the author’s Twitter thread on the paper and found the topic intriguing, but slightly beyond my area of expertise. The review file provided me exactly with the lacking context and critical external opinion that I needed to form a somewhat less vague opinion of the specific advance made by the paper. Really helpful!

Place cells respond to moving bars in immobile rats (Review file)

This review file contains an entire story by itself. In the first round of review, reviewer #1 was rather critical, while the other two reviewers were almost convinced already. The authors, in a 8-month revision period finally managed to very decently address most points brought up by the reviewers. They do this, as has become apparently common practice for rebuttal letters to high-impact journals, in a very long and detailed letter (the entire review file is 44 pages).

But after this round of reviews and rebuttals, suddenly, reviewer #3 changes his/her opinion entirely:

“Unfortunately my evaluation of this paper has changed since the first submission due to comments from the other reviewers and because of literature I have discovered since then. This paper presents a set of reasonably well performed and analyzed experiments, but I no longer think the main results are novel or surprising. I therefore do not recommend publication in Nature and think this paper is better suited for a specialist journal.”

This is just the beginning of a long comment on why the reviewer thinks the paper is not novel enough any more. This is of course a nightmare for the authors. In the end, the authors do their best to address these concerns of novelty. Reviewer #3 remains unconvinced. The editor decides to ignore these concerns and goes with the recommendation of the other reviewers to publish the manuscript. Check it out yourself to form your own opinion on this discussion!

Non-telecentric two-photon microscopy (Review file)

A much wilder story is hidden in the peer review file of this optics paper from Tom Baden’s lab, which was finally published at Nature Communications. The first set of reviews at Nat Comm is still pretty normal, but then something goes wrong. Reviewer #4, who is – very obviously – highly competent but also a bit obsessed with details and annoyed by imprecision, has a few things to complain about, mostly about a few relatively small unsubstantiated claims and some minor errors that do not affect the main idea of the manuscript. However, the authors do not agree with this opinion, and a long and slowly escalating discussion between an annoyed reviewer and frustrated authors evolves. Check it out yourself. If you don’t get stomach pain while reading, you have my full admiration. At some point, reviewer #4 writes:

“In the last round of review, I wrote detailed comments, color-rebuttal, in a 16-page PDF, with the hope that they would be of help to the authors to make the manuscript better. The response I received on this round, surprisingly, is only 5 pages, and the authors reluctantly chose, on purpose or randomly, 5 comments to address, and ignored all other comments I curated. PLEASE RE-WRITE your response by addressing all comments I gave last time.”

Upon “editorial guidance”, the authors refrain from doing so. It all ends with mutual passive aggression (“No further comment” from both sides) – and acceptance for publication as an article.

Read it yourself to see how crazy and tiring peer review can be, and consider yourself how this situation could have been avoided by the authors or the reviewers. However, in the end, this review file is also a contribution to the scientific discussion (e.g., about proper PSF measurements) and therefore valuable by itself. It is a painful document, but also useful and full of insights.

Three-photon microscopy (Paper 1, Reviews; Paper 2, Reviews)

Three-photon microscopy is a relatively new field still, and when these two papers came out in Nature Communications and eLife, respectively, I was very happy to be provided with the additional context and details in the peer review files. I found especially the discussion in the Nat Comm paper (paper 1) about potential concerns for three-photon imaging very interesting.

Somato-dendritic coupling of V1 L5 neurons (Review file)

A few years ago, I had covered a preprint by Francioni et al. from the Rochefort lab on my blog. This study was later published on eLife, and since I liked the work of this paper already, I was very curious about the comments of the reviewers, their concerns, and the author’s replies. It is nice to get additional insights into such interesting studies!

Juxtacellular opto-tagging of CA1 neurons (Review file)

The review file of this beautiful methods paper from the Burgalossi lab tells an interesting story. Apparently, the authors had included an additional experimental sub-study based on cfos in the paper, but the reviewers were not convinced by the results. They therefore suggested – very surprising to me! – the acceptance of the paper for publication, but only after deletion of this subsection. I would not have guessed that such a unexpected and helpful consensus can be reached during the review process. This was probably helped by the fact that at eLife, it is common practice that editors and reviewers discuss their reviews with one another.

Nonlinear transient amplification (Review file)

Purely theoretical (neuroscience) papers are often challenging because it is difficult to fully judge the novelty, even whenthe concepts and ideas and equations are transparent. This paper by Wu and Zenke is conceptually close to what I have studied experimentally during my PhD (paper link in case you’re interested), so I was happy that this paper got published at eLife, with the review publicly available. A very useful secondary perspective!

In vivo calcium imaging in CA3 (Review file)

This is – so far – the only paper with public review file where I have been involved as an author. I wrote some sections of the rebuttal letter, actually without knowing that the reviews and rebuttals would be openly available afterwards. Unfortunately, the journal (eNeuro) messed up the formatting in such a horrible way that the review file becomes almost unreadable (which is a pity, because our rebuttal letter was very nicely written). This mess-up shows that there is still some progress to make, also in terms of infrastructure.

In general, I hope that public review files will become more common in the future, to the extent that non-public review files will be a thing of the past entirely. Public reviews make the editorial process more transparent, they open up the discussion behind the paper, lower the barriers for junior researchers with less peer review experience, and do not have, to my understanding, any major negative side-effects.