Crowd-Sourced Peer Review in Philosophy

A Dynamic Prospectus

(unadorned links display/hide further information)

What Is Populus?

Populus is a curated archive and a new kind of journal for work in philosophy, with a committment to transparent, crowd-sourced  peer review.


A Curated Archive

Populus members can submit their unpublished and published (where consistent with their publisher's policies) work to the site as archive records, which will be made freely available to the public. Members can also read and review each other's work. Reviews are both quantitative ('upvote' or 'downvote') and qualitative, via a wiki. Votes are used to generate various ordered displays of archive records—e.g., by general popularity (net votes), popularity within subdisciplines, percentage of upvotes vs. downvotes, etc. The archive is intended to help philosophers find those works most relevant to their research or interests, and members are therefore expected to vote on the basis of the quality and contributory value of the work in question. By default, all reviews, both quantitative and qualitative, are anonymous. (Editors can tell that two reviews come from the same member, but not who that member is.) Anonymity will be breached only with the consent of the author or in extreme circumstances, such as for legal reasons.


Membership Requirements

To be eligible for membership, and therefore to submit works and vote, one must meet one or more of the following criteria:
  1. Be a faculty member (including non-tenure-track and emeritus) or graduate student in philosophy.
  2. Have published work in a recognized venue of philosophical publication.
  3. Receive approval for an exception in the absence of (1) and (2).
When applying for membership, one may either provide proof that one meets criteria (1) or (2), or apply for an exception. Those applying for exceptions will be asked to submit a sample of work for anonymous review. An appropriate reviewer will be selected and asked to determine whether on the basis of the work whether its author merits membership.


Housing Policy

  • Unpublished archive records will be housed on Populus servers, at least initially. As noted above, for the sake of transparency Populus flags changes to archive records, and there is currently no way to track this for unpublished work archived elsewhere. However, Populus is fully open to coordination with other archives, either to allow for easy linking of files on Populus servers or to allow for links on Populus to archive records on other servers, conditional on transparency in file-alteration.
  • Archive records of publications in other venues may be housed on other servers, on the condition that they are publicly accessible. Thus, for instance, a member may add a link to a paper published in an open source journal, such as Philosophers' Imprint . By contrast, if a member wishes to archive a paper published in a journal published by Springer, they must upload a penultimate draft, in keeping with Springer's self-archiving policy , since a link to the published work on Springer's servers would be unavailable to many people.
  • Journal submissions and publications in Populus will be housed on Populus servers. Authors retain copyright and are therefore also welcome to archive elsewhere.


Review Wikis

Just like on Wikipedia, rather than one person's writing a review and others' responding to it, once an initial review is written all subsequent reviewers will edit the initial review. Edits will be logged and visitors will be able to view previous versions and see how things have changed. If necessary, other features of Wikipedia will be adopted, such as editorial oversight to flag controversy (e.g., if a wiki is being edited back and forth in certain ways). The benefit of this system is that reviewers, authors and editors will have a single, manageably sized piece of written feedback, whereas traditional comment threads run the risk of becoming unmanageably bloated.


A New Kind of Journal

Populus will periodically publish a volume of its journal (also called Populus). The projected publication schedule is quarterly or biannual. Unlike a traditional journal, conditional acceptance (think 'pending final review' for an edited volume) is a direct function of the quantitative review system. The function is expected to evolve over time. However, initially, it will simply output the submissions with the highest ratio of upvotes to downvotes, beyond a minimum number of total votes and a minimum vote ratio. Conditionally accepted works will then go through an expedited version of a more traditional review process. An appropriate referee will be asked to review the work and make a recommendation as to whether it should be published as-is or revised. Reviewers can also reject papers, but the expectation is that this will be rare, as there is a presumption in favor of publication on the basis of member votes. It should go without saying that members are also expected to be both charitable and respectful in editing the review wiki.


Submitting for Publication

When uploading an archive record to Populus, members will have the option to flag their work as a journal submission. Journal submissions are anonymous (whereas archive records may or may not be) and authors must confirm that the work has not been published elsewhere (though simultaneous submission is permitted). Members are encouraged to give priority to reviewing submissions, so as to aid in the review process. When reviewing submissions, members are expected to use the same criteria they would if acting as referees for any other journal: upvoting only if they believe the work warrants publication as-is or with a reasonable amount of editing, downvoting only if they believe that the paper does not warrant publication in its current or any similar form.


Simultaneous Submission

Populus is pleased to accept submissions simultaneously with other publication venues. However, members should be aware that once they have submitted work for consideration for publication with Populus, it remains under consideration until the author changes its status. Since most other journals do not allow simultaneous submission, members should be sure to check their work's status before submitting elsewhere. It is also members' responsibility to ensure that any archive records published elsewhere are in keeping with their publisher's policies on archiving.


Editing Uploads

At any time, an author may choose to upload a new version of their work or remove it from the archive (with the exception of works published in the journal). Authors may also choose to change an archive record to a journal submission or vice versa (in the former case, this may involve a fee). If a new version is uploaded or the submission status is changed, all votes attached to the original upload remain and a flag will be added indicating that the file or its status has been altered.



Access to the site and electronic subscription to the journal are free to the public. Membership and archiving are free to those who qualify. There will likely be a minimal charge for journal submission and/or publication, to help defray costs, though this will probably not be instituted until Populus is more established.

Why Populus?

There are many well-recognized problems with traditional publication venues. Some of these concern things like price and access. Others concern the integrity of the traditional review process and its ability to track quality. Many of the former and some of the latter issues have been addressed by the recent development of open source, online journals. However, for the most part, these have simply exported traditional publishing models to the Internet. Populus seeks to reimagine the publication process to take fuller advantage of the resources afforded by the Internet to provide a better experience in publishing, reviewing, and archiving.


A Better Publishing Experience

The Populus submission and review process offers a welcome change from that with other publications. For one, there is less uncertainty regarding the timeline for review. Works are eligible for conditional acceptance at the deadline for each journal issue after submission, so long as they are flagged as journal submissions. What's more, authors have access to real-time feedback on their work. They are able to see at all times whether their work is being read and how it is being received, both in terms of votes and feedback on the wiki. Is your work getting less upvotes than you'd like? Are there comments in the wiki you believe you can address? Simply address them and upload a new version.


A Better Reviewing Experience

The hope is that philosophers will see reviewing works on Populus as a duty of citizenship within the philosophical community, much as many do now when asked to referee for other publication venues. However, because the review process relies most heavily on votes, reviewing for Populus is less arduous than reviewing for other journals. If there is a paper you are interested in reading, there is little cost to reading it through Populus and voting. Populus will also incorporate game-like incentives, such as status boosts (e.g., 'top reviewer'). It is TBD whether these will be purely symbolic or will come with tangible incentives, such as additional voting power.


A Better Archiving Experience

As Populus membership grows, having work on Populus will make it more visible to the philosophical community. But unlike with other archives, authors also gain insight into their peers' assessment of their work. As time goes on, further information may also be available. For example, information may become available on what other works appeal to the same population as yours. Citation indexing is also planned, and thus eventually Populus plans to offer real-time information on impact factor.

How will Populus avoid...

...favoring popular subdisciplines?




Won't Populus favor work in popular areas?

Possibly. This is less of a concern when it comes to archive records, since part of Populus' purpose is explicitly to track popularity. However, it is a serious concern for journal submissions. One possibility is that the minimum threshold number of votes can be lowered for such areas. This will be kept in mind and addressed as necessary.


Won't some members just vote for their friends' work, or against their enemies', or otherwise use problematic criteria in voting?

The hope is that this will be the exception, rather than the rule. One reason for hope is that the desirability of being published in Populus depends on the journal's reputation. If members vote as worried here, without regard to quality, that reputation is unlikely to be good, thus undercutting the incentive to vote this way in the first place. There is therefore good reason to vote properly not only for the sake of personal integrity, but also to preserve the integrity of Populus itself. Of course, there will probably be some who vote as worried here. But the same is true in the traditional peer review system. One of the hopes behind crowd-sourcing is that the overall results will remain reflect the better nature of the majority. There are doubtless many who leave insincere or self-serving reviews on Netflix, Amazon, etc. But in the best cases, the ratings systems on such sites nevertheless provide useful information to users.


Is this going to work?

That depends on a lot of things including, most importantly, you. Participation from the philosophical community will be indispensable to Populus' success. There are plenty of other potential worries, many surrounding how well crowd-sourced reviews will track quality. But as suggested above, the relevant question is not whether the Populus model is ideal, but whether it is better than the status quo and other feasible alternatives. Whether that is the case can only be determined through experimentation. Crucially, Populus is not a static experiment. Members are encouraged to offer suggestions regarding changes in editorial policy, and in the long run, the data itself may make recommendations. Suppose, for example, a correlation is found between the works upvoted by reviewers with certain profiles and those works' impact factor. Perhaps reviewers with those profiles will be given additional voting power, either permanently or for a special issue.

Where can I find out more?

Populus' founder, managing editor and managing web developer is David Faraci . If you have a question, or are interesting in getting involved with Populus, you can email him at populusjournal@gmail.com . If you'd like to stay up-to-date on further developments, you can join the Populus mailing list here .

September 5, 2016