Who owns the reviews generated during the peer review process? This is not a rhetorical question.

I’ve been wanting to make a post recently to discuss reviews I’ve recieved over the past year or so from various conferences. In particular, I’d like to discuss the value I’ve gotten from each review and what each tells me about the nature of the conference and the values of the conference communitities. However, I don’t even know whether I can publicly post reviews, in part or entirety!

ACM Policy

One of the nice things about the ACM is that they put so much stuff online! However, the only link to anything about reviewing I’d found is for the Social Computing Journal (TSC):

Reviews are single blind: reviewers can see the names of the authors but not vice versa. By policy, ACM reviewers must remain anonymous to authors of a manuscript. Ensure that there is no identifying information in the content or the properties of any review or any uploaded documents.

Reviewers must protect the confidentiality of their involvement in the review process, of the reviews themselves and of the ideas represented in the submitted papers.

There is no discussion of who retains the rights to the reviews. There is another, separate page, about reviewer anonymity, but that hasn’t been updated since Bush II’s first term.

Google Search Results

I didn’t spend too much time on this, because I have many other things I ought to be writing right now! However, the best first-page google results for a variety of search queries I’ve already forgotten turned up had the best discussion of this topic I’ve seen so far. The tl;dr – it depends. One of the other queries I used turned up a discussion of who owns Yelp reviews, and the answer is the person who wrote them! This is what John had suggested was the case when it came up in conversation previously. There’s a discussion here stating that the reviewer owns the content, but that the content provider can do whatever they want with it. I suppose in this case, the content provider would be the conference organization, so it seems like they are the ones who should ultimately decide. Review platforms like OpenReview seem much closer to the Yelp model of things. Non-public reviews would seem to make things more complicated.

Dangers of Sharing Reviews

In a small community like SIGPLAN, the main danger I see is that someone would learn to identify the tone of reviewers. I doubt that there would be enough labeled data to build a language classifier for reviewers, but I also doubt you’d need to – if you yourself review for long enough, you will come to recognize the tone of reviewers. This will happen when you review with others, learn their style, and then later recieve reviews from them. This pool will be very small if people do not routinely share their reviews, but will become much larger if they do.

Sharing in Practice

In my training as a PhD student, it has generally been common practice to share reviews within the lab. This helps junior students build perspective on what to expect from reviews, reduces the shame some students feel when they receive bad reviews, builds camaraderie, and gives a valuable perspective to students when soliciting feedback (that is, if your peers know what critics have said, they can potentially offer more useful advice in the future).

I’ve also shared reviews with potential future collaborators, especially on rejected papers, and when the collaborator is unfamiliar with the research community.

Neither of these cases amount to public posting. Furthermore, I feel like there are qualitative differences between the reviews of papers that have been accepted and those that have been rejected – the rejected ones tend to include information about making the paper stronger, but the reviews for the accepted papers often contain ideas about future work. It’s exciting to see reviewers who are enthusiastic about the work, but it feels wrong to share them without obtaining explicit permission to do so.

Reviewer Agreements?

Of course, we could make all of this explicit by having reviewer agreements. I am embarressed to say that I do not recall ever signing one. Maybe I did, and it’s like terms of serve that everyone skims through and doesn’t read. In any case, if I did sign one, it’s clear that we need to have something like a quiz before being approved as a reviewer, in order to ensure that reviewers know the rules and best practices. If I have never signed one, maybe we should start by introducing them?