Elijah Millgram

Copyright © 2018 Elijah Millgram.
All Rights Reserved.

Sausage, Legislation, and Journal Reviews

Bismarck said (or anyway is said to have said) that you don't want to know how your sausage or legislation is made; that goes for the contents of our journals as well. But if you're starting off in philosophy, you'll need to publish, and to do that effectively, you have to understand how the system functions (or rather, doesn't quite function). At the early stages, getting into print usually means submitting to professional journals.

The most important thing to understand is the reviewers' incentive structure. Journals make decisions on the basis of reviews provided by volunteers. Now -- and this is something you'll have to come to terms with from the other side, once you've entered professional life -- reviewing is a thankless job. Doing it right is time consuming, and you get practically nothing in the way of professional rewards for it.

Of course you're not paid for it; journal editors expect you to do it for free. And while presses will often give you a token honorarium for advising them on a book manuscript, it won't come close to the minimum wage, given how much time such a job consumes. But since philosophers aren't in it for the money, it's more important that the professional signals you get tell you not to spend a lot of time and effort on it. As a reviewer, you're anonymous, and while you can put your reviewing on your CV and your annual Faculty Activity Report, no one cares: not your department, not your administration, and not anybody else. Reviewing won't help you get tenure, won't get you a raise, and in fact, it won't get you anything.

On top of that, the task itself is unpleasant, because most of the papers you'll get sent are genuinely bad: badly written, and typically, rehashes of something you've seen dozens of times before. If you're an academic, you probably spend a lot of time grading already; reviewing is volunteering to do additional grading. And if you're an academic, what you have to do pushes out what you'd like to do. So the amazing thing is that anyone ever reviews anything at all.

(There are exceptions: a couple of journals are run by philosophy departments, and they do most of the editorial work in-house; in those cases, reviewing journal submissions is part of someone's job description. And Social Philosophy and Policy has recently started to pay its reviewers. What would it look like to provide effective incentives? Constantine Sandis suggests: "Perhaps the answer is to give people submission credits for every two articles they referee (with exemptions for submissions from graduate students and early career researchers)." ["How Finch Stole the Creative Commons," Oxford Magazine])

The incentive structure shapes what the product turns out to be. In a world of very busy people, the reviewing will be disproportionately done by people who have don't have anything else they have to do with their time. Often that means people who don't have much in the way of an active research program; in any case, it's not infrequently hard for an editor to find someone who is really up to speed on the subject of a given manuscript. When reviewing is done by busy people, it is likely to be done as quickly as possible; thus reviews tend to be hastily composed, and written on the basis of skimming rather than a careful reading. To be sure, that the review itself was written quickly doesn't mean that it was gotten to quickly; the same reviewers who have no incentive to do a careful job on your manuscript have no incentive to put it high on their task list, and turnaround times of six months to a year are normal, with two-year turnarounds not unheard of.

Reviewing will also be done disproportionately by people for whom the emotional rewards of reviewing make it worthwhile. Sometimes those emotional rewards are the knowledge that one is doing one's fair share of the work that makes the discipline go around. But all too often, it's the joy of getting to beat up somebody who can't fight back. (As I've mentioned, the process anonymizes reviewers, and many of them take full advantage of that fact.) The structural problem -- built into the profession as we now have it -- is that the incentives almost guarantee that when you submit papers for publication, the philosophers who determine whether your paper will get published -- and thus more generally what goes into the professional journals -- will, ironically, be as often as not unprofessional.

The first draft of this web page contained a couple of examples volunteered by a former student who had gotten a rejection from Mind, and which nicely exhibited the full range of problems I've been describing. The editor of Mind has asked me not reproduce them, and I'll discuss his letter (and his publisher's, which delivered a threatening legal opinion), as a way of posing a practical and ethical problem with which the dysfunctional institutions of our profession face us. (Here, because general worries about what's gone wrong with the business don't belong on an advice-to-students page.) For now, what you need to know is that the example came from what is widely regarded as one of the most prestigious journals in the business. The effects of the incentive structure are pervasive; they're not just found at the low end. Until we have a systematic procedure for publishing and reviewing journal reviews, the evidence will be anecdotal; my impression, however, is that low-quality reviews are typical.

What does this mean for you, as someone starting out?

First of all, you need to be aware that you're going into a business whose basic institutions are broken. If you're public spirited, you may want to devote some time to figuring out how they might be fixed. But in any case you're going to have to live with them until they are fixed. You should give some thought, now, to what your publication strategy is going to be; if you're a grad student at Utah, make an appointment, and I can review some of them with you. I haven't been covering the mechanics of journal submissions here (partly because it's too easy to get distracted by those, and lose sight of the larger process), but your strategy will have to be designed with those in mind.

Second, take the process by which journal reviews are generated into account in figuring out what to do with the ones you receive. In particular, don't treat the reviews as normally giving you a reliable assessment of your work. Remember, reviews are typically hasty, shoddy, often motivated in precisely the wrong way to serve as a reliable assessment, and often produced by precisely the people you wouldn't want to have doing them.

Third, get used to a diet of abuse, and develop a thick skin. You're going to have to submit many manuscripts to journals in order to have a career, and reviewers take advantage of anonymity to get their ya-yas out.

Fourth, when you see a paper published in a prestigious journal, you're likely to assume that it must be of pretty high quality. Time to unlearn that reaction. That's nontrivial: as the psych literature informs us, it's hard, almost impossible, to fully compensate for anchoring.

Nonetheless, remember that journal placement is produced by the process we've just been examining. The inputs to editorial decisions are -- for structural reasons -- of low quality. That means you should expect the decisions themselves by and large to be of low quality, which further means that you should expect publication in a journal to tell you very little about the quality of the articles in it. That expectation should apply even to the most prestigious journals.

For what it's worth, my own take is that a clear-eyed look at our journals confirms these expectations. Overall, quality is at best uneven, and often quite low. This is a structural problem in our profession. You should not assume that publication -- in any journal -- is a useful certification of philosophical quality.

Finally, if philosophy matters to you, you also need to develop your own standards, and figure out a way of holding yourself to them. You can't rely on the feedback you'll get from our degraded public quality control mechanisms. And if you want to have a career in the world of professionalized philosophy, you'll need to think through how you're going to stick to your standards, while nonetheless publishing regularly, in venues that this world considers respectable.