Over the years, a great deal has been written about the ontology of pizza (http://robertdavidstevens.wordpress.com/2010/01/22/why-the-pizza-ontology-tutorial/). It’s a good example, is easy to understand and works surprisingly well in a tutorial context. It is also comes up surprisingly commonly in the public sphere as it did last year on BBC News (http://www.bbc.co.uk/news/magazine-33542392). The key point of which is this: the pizza maker argues that you can’t have a marinara (tomato and garlic) with added mozzarella because a marinara is pizza rossa which can’t have mozzarella; a margherita (tomato and mozzarella) with garlic is fine though. Ha, those crazy Italians. I paraphrase, of course.

Of course, the right to comment first on this article rests with Robert Stevens, my colleague and world acknowledged authority on the ontology of pizza, and indeed he has done so (http://robertdavidstevens.wordpress.com/2015/08/13/where-a-pizza-ontology-may-help/).

I would like to take a slightly different approach to the question though. How do we know what our ontology should be? I’ll start by ontological reading of the article, look at some issues with it, and then consider how we might gather the knowledge to resolve them. As with many parts of ontology construction, there is not a perfect answer and it turns out to be more complex than it might appear at first sight.


A Literate Approach

I start by building an ontology. As (both) regular readers of this journal might expect, I am going to use Tawny-OWL (http://www.russet.org.uk/blog/3088) to do this, so the syntax is a little different than for Robert’s post.

I’m starting from a slightly different place from Robert. One of his concerns was to fit within the context of the existing pizza ontology. Now, I use to be a firm believer in the idea that ontologies were all about a shared conceptualisation of the world, but over time, I have become less sure that this should always be a consideration. In this case, the purpose of building the ontology is to allow me to formally and explicitly describe something with the context of this blogpost, to clarify my own understanding. Sharing is irrelevant for this use case and I am going to build my ontology rapidly from scratch.

I am a big fan of literate ontologies (1512.04250) and I want do that with blogs as well. So the source code of this post (accessible from http://archive.org/download/phil-lord-journal/), is lenticular text (http://www.russet.org.uk/blog/3035) and the whole article can be parsed as a valid ontology. Perhaps no change for the reader, but a comfort for me as the author to know that I’ve evaluated every statement in this post.

(ns the-epistemology-of-pizza
  (:refer-clojure :only [])
  (:use [tawny owl english reasoner]))

(reasoner-factory :hermit)

As with all Tawny-OWL ontologies, we start with a namespace declaration. If you are reading this and use Clojure a lot, note the :refer-clojure and use of tawny.english: none of or, some or not are the clojure.core functions! We select a default reasoner also.

(defontology epistemology)

(defclass PizzaTopping)
(defoproperty hasTopping)

(as-subclasses
 PizzaTopping
 :disjoint
 (defclass Mozzarella)
 (defclass Tomato)
 (defclass Garlic))

We need a set of primitive terms on which to base the ontology, which we define here. Again, this is not an ontology for sharing: I have not added labels, formal textual definitions nor do I need to care about what the IRIs actual are. We only need three toppings and all we need to care about them is that they are different.

(defclass Pizza
  :super (some hasTopping (or Mozzarella Tomato)))

It turns out, I do not need to care about pizza bases either, so I am not going to talk about them explicitly. Rather unusually (and in a difference from the pizza ontology) I am going to insist that a Pizza have either Mozzarella or Tomato. I’ll come back to this later.

(defclass PizzaRossa
  :super
  Pizza
  (only hasTopping (not Mozzarella)))

(defclass PizzaBianca
  :super
  Pizza
  (only hasTopping (not Tomato)))

The definitions of PizzaRossa and PizzaBianca are a little surprising; they are defined negatively, but I quite like these definitions. They are sort of like the backward definitions of the fly geneticist — the “white” gene is responsible for the enzyme that causes red eyes. The gene is named after the mutant.

(defclass Marinara
  :super
  PizzaRossa
  (some hasTopping Garlic Tomato))

The definition of Marinara is now straight-forward enough, simply stating the ingredients, and that Marinara is a PizzaRossa.

(clojure.core/assert
 (with-probe-entities
   [WierdMarinara
    (owl-class "WierdMarinara"
               :super Marinara
               (some hasTopping Mozzarella))]
   (clojure.core/not (coherent?))))

And, so we get to the humourous crux of the story which is, indeed, you cannot have a Marinara with Mozzarella.


The Epistemological Questions

But this leaves us with a number of problems. One of which is that in explaining the joke, we have rather killed it; one sad reality that all ontologists have to face is that it pushes us toward pedantry, making us humourless, crushing bores. A much deeper problem though is that what we have produced is a computational data structure which we have queried and got an answer about that computational data structure. When what we really want to know about is pizza.

How do we know whether what we have modelled is correct. Is this ontology a good ontology, an accurate reflection of reality? And what does this mean anyway? In short, how do we know what we know?

Let’s consider the issue from a set of different perspectives.

The Software Engineer

As a software engineer, I’m rather fond of the ontology that I have produced, and in that sense it is a good ontology. I’ve already said that I quite like the “backward definitions”, and find them quite elegant. The ontology is also symmetrical: PizzaBianca and PizzaRossa are both defined in the same, quite equivalent way.

Now, of course, we might argue that this considerations do not tell us that we have a good ontology. But an elegant and symmetrical axiomatisation is useful; it’s easier to remember, there are no “special cases”, and it is easy to spot outliers. All of these things support maintainability of software in general and specifically in ontologies.

There are some issues with my ontology; we have not, for instance, following Alan Rector’s normalisation pattern (http://ontogenesis.knowledgeblog.org/49); the Margherita is neither white nor red in this schema.

Margherita presents a bigger problem, though, than not being normalised, which is simply that my knowledge of margherita tells me that it is normally considered a pizza rossa, while my ontology say it is not. My ontology is nice, but it is wrong.

The philosophical approach

We could also consider this ontology from a more philosophical point-of-view. Of course, I am perhaps not the ideal person to do this, but I would note that our ontology has a clear, single inheritance hierarchy. Most of our classes have clear differentiatia (I’ve just stopped at the toppings, but you can’t define everything, because that turtles (http://en.wikipedia.org/wiki/Turtles_all_the_way_down)). Even our definition of pizza could fit into a larger hierarchy: pizza without either mozzarella or tomato is either a focaccia or some other type of bread.

After that, I am rather stuck. It is hard to draw many more conclusions about pizza from first principles. From a realist point-of-view, we should model reality and universals, which sounds nice. But how do I determine that what that reality is?

Time to phone a friend.

The Expert Analysis Technique

One standard technique in ontology building is to consult with an expert. Indeed, that is often the main evidence and justification that is used to support an ontology, which is why many ontology papers have more authors than the human genome paper. So, let’s try it in this case. The BBC article quotes from quite a few experts.

“La marinara is a pizza rossa,” she states frostily. “A pizza rossa is made with tomato and without mozzarella. So you can’t have a marinara with mozzarella because there’s no such thing.”

Emanuela
— BBC

My ontology supports this because Marinara is a PizzaRossa so, indeed, cannot have Mozzarella.

“No, it’s not,” pipes up a customer who until now has been quietly consuming his pizza and beer on a stool behind me. “She’s right. A pizza rossa can’t have mozzarella.”

Customer
— BBC

Also!

The pizzaiola is right. A marinara is not a marinara if you add mozzarella. But she was wrong to say she would make you a margherita with garlic because margherita with garlic doesn’t exist.”

Friend
— BBC

Currently, my ontology says nothing about this one way or the other. But, we can add a closed definition for Margherita easily, and now, indeed, a margherita with garlic cannot exist.

(defclass Margherita
  :super Pizza
  (some-only hasTopping Mozzarella Tomato))

(clojure.core/assert
 (with-probe-entities
   [WierdMargherita
    (owl-class "WeirdMargherita"
               :super Margherita
               (some hasTopping Garlic))]
   (clojure.core/not (coherent?))))

I also tried extending my analysis further, with a novel technique; I tried it by asking my friends on facebook. One of them, replied as follows.

my take on this is that we must distinguish between pizzas made by bakeries (pizza bianca and rossa) which tipically (but theres no firm rules) do not have mozzarella, and pizza from a pizzeria (restaurant) which almost always has mozzarella (bianca and rossa). a pizza bianca without mozzarella is a focaccia ( with rosemary, onions, potatoes, etc)

— Ulisse Pizzi

It’s a disaster! Almost, none of this has been modelled in my ontology, although it supports my assertion that a pizza must have tomato or mozzarella. But worse, “there are no firm rules”; it’s enough to make a grown ontologist cry. But there are some deeper frustrations here. None of the experts have told me all the issues that I want to know. None have really answered what is a pizza.

We could ask some more experts? But what happens when they contradict? Do we take averages? And, more important, how do I know when to stop; I could carry on asking friends all day. And how do we avoid cherry-(tomato)-picking?

From the Definitive sauce (erm, source)

Having tried ontology by facebook, let’s try a research method with a much older and richer pedigree; we will look the answer up on wikipedia instead. First port of call is the main Pizza, which says many things, but none of them that useful in this case. Hunting around got me to Pizza al taglio (pizza by the slice) which says:

The simplest varieties include pizza Margherita (tomato sauce and cheese), pizza bianca (olive oil, rosemary and garlic),[4] and pizza rossa (tomato sauce only).

— Pizza al taglio

This is interesting becuase it brings in garlic and rosemary (which has not been mentioned before). In this version of the world, a margherita is not a pizza bianca. And there is a lack of symmetry between rossa (which is closed) and bianca (which is not).

Perhaps a better source of material would be to look on the Italian wikipedia for its definition of Pizza. This says:

Pizza marchigiana […] Le varianti tradizionali sono quattro: bianca[12] con il rosmarino, bianca[12] alla cipolla, rossa semplice[13] e rossa[13] con la mozzarella.

Pizza marchigiana
— Italian Wikipedia

So, more types of pizza, including with rosemary, with onion and simple. The footnotes are more informative still.

[12] Per “bianca” si intende una pizza senza pomodoro. [13] Per “rossa” si intende una pizza con il pomodoro.

Pizza
— Italian Wikipedia

So, pizza bianca must NOT have tomato, while rossa MUST have it. The definition of pizza rossa here is inconsistent with mine, but makes more sense to be honest — margherita finally becomes a pizza rossa. Strictly, this quote only really applies to “pizza marchigiana” — once we move out of Marche, all our definitions might be different!

Another useful thing that does come out of reading wikipedia, however, is the information that Pizza Neopolitana has regionally protected status in the EU. And that there is an official body for defining what is a pizza, namely the AVPN. Their document describing the pizza is called the disciplinare (also in English).

And, indeed it has definitions, although unfortunately only of two pizza, the Marinara and the Margherita. So, the answers we get from here are limited; but they look like this:

Marinara (tomato, oil, oregano, and garlic) Margherita (tomato, oil, mozzarella or fior di latte, grated cheese and basil)

Disciplinare
— Associazione Verace Pizza Napoletana

So, we have our answer? Well, it’s one I find surprising (grated hard cheese on margherita!). But, actually, the disciplinare is more specific. For instance, where they say tomato not just any tomato will do, they have to be one from a very specific type:

The following variations of fresh tomatoes can be used: “S.Marzano dell’Agro Sarnese-nocerino D.O.P”., “Pomodorini di Corbara (Corbarino)”, “Pomodorino del piennolo del Vesuvio” D.O.P.” (see attached appendices for suppliers and technical details)

Disciplinare
— Associazione Verace Pizza Napoletana

How can we encode this? It really just too complicated, and is unlikely to be useful ontologically at any point; worse, we only have answers for two pizza types and there many others. Looking further, there are also instructions for the flour, the proving, the stretching and much more besides. So, it turns out that the definitive answer is not that useful either: it is incomplete and overly complicated.


Conclusions

The BBC articles take on the whole process is this:

Pizza has taught me that logic can be subjective and that subjective logic can be cultural

— BBC

Of course, it’s not true; logic is not subjective at all, although there are many different forms of logic. But definitions can be. I have tried multiple different mechanisms of reaching a definition, and all of them have flaws:

  • software engineering approach — maintainable, nice but correct?
  • philosophical approach — limited in its applicability
  • find a friend (the facebook approach) — prone to cherry picking
  • literature review (the wikipedia approach) — lacks interactivity, so may not answre the question
  • A definitive source (the authoritarian approach) — over specified, under covering

Definitions are difficult, and there will be no universal answer. Having a clearly defined use case, and a mechanism to test your ontology against that use case remains key. But also having a clear awareness of the techniques that you are using for build your ontology and the flaws that exist with them.

It is also a good excuse to eat pizza, if you need one.


Updates

Minor spelling corrections!

Bibliography

My last blogpost was on logging (http://www.russet.org.uk/blog/3112) with the SLF4J library, its inclusion in the OWL API, and its impact on Tawny-OWL. It raised quite a response, not least because it suggested that people ignore the advice of the authors of the library.

Perhaps, unsurprisingly, the general feeling was against my position — something that I expected given that this had already happened on the SLF4J mailing list. Now, of course, it does have to be said that there is a significant sampling bias here: anyone on the mailing list, or who reads and responds to a post about a logging API, is likely to be interested in logging rather than a passive user (or non-user) of such a library. None the less, I think it is worth reflecting on the reasons for the comments.

Of course, I am aware, that as many people pointed out, the problem is not huge (a three line error message) and my solution is messy. Well, I agree, the solution is messy, but it is the only method that SLF4J has support for. There are alternatives; for example, this commit from the OWL API provides another solution. I have not decided yet whether this is the right way to go or not.

Several people complained that including the nop dependency will silently turn off all logging. Actually, this is not true: in applications which have logging already, including a nop dependency, will trigger an error message; whether logging is turned off or not is hard to determine — it depends on the classpath ordering. I am, however, clearly making life harder for those who want logging on. Specifically, I am requiring that instead of using:

["uk.org.russet/tawny-owl "1.5.0"]

It will require this:

["uk.org.russet/tawny-owl "1.5.0" :exclusions ["org.slf4/slf4f-nop"]]

(or the equivalent for maven). So, harder, but not much harder.

The other two main suggestions are that I split my code into library and application, even though this is artificial (the “application” would currently contain no code). But this approach would require most of my downstream dependencies to also be split into two for the same reason; it was even suggested that the Clojure REPL or Leiningen that should do the work, and that they should be split into two. Introducing such complexity for a logging API is surely not good engineering.

Finally, there were many suggestions that I leave the decision to the “end user”. But, as I have said, while I like allowing the “end user” to have a choice, I do not believe in imposing it on them. Choose a default where it is at all possible and do not sit on the fence.

During this process, though, I have thought a lot about logging, and I have come to the conclusion that the core of the problem is that logging is heavily overloaded. Quite a few people were frankly amazed that I would not want to see logging, asserting that most people would want to see it.

Now, I use a web server, and use its logs all the time. Logs are critical for many reasons, and many tools like webalyser or fail2ban work over them. I would never want to turn these off. But, here, though, the logging is effectily a part of the application: apache logs are part of the adminstration user interface, as well as forming an ad hoc API to downstream tools; part of the plumbing in git parlance. Why would you want to switch this off? I would not.

In this case, though, if logging is part of the application, why on earth are people so keen on libraries logging — the application should not be deciding the location of the logs only, but also how and what is logged. Libraries should just not be logging at all; they should be taking callbacks, returning values and leaving the outcome of this to the application.

What about “error messages” though? Surely libraries can produce these? The OWL API (which is the library that makes use of SLF4J) is an example of this. Most of the logging message are errors: for example, this situation:

if (loadCount.get() != importsLoadCount.get()) {
   LOGGER.warn(
               "Runtime Warning: Parsers should load imported ontologies using the makeImportLoadRequest method.");
}

which is effectively a form of active documentation or this:

try {
    return optional(new XZInputStream(new BufferedInputStream(new FileInputStream(file))));
} catch (IOException e) {
    LOGGER.error("File cannot be found or opened", e);
    failedOnStreams.set(true);
    return emptyOptional();
}

which is sort of an error but one that has actually been dealt with. These are mostly debugging statements that the authors think they may use again. That’s fine and an entirely reasonable use of a logging framework. Equally, and I feel entirely reasonably, I want this off by default.

The “error message” from SLF4J that started me down this path falls into the same category. It is not an error, or it would throw an exception (as it used to do in the past). It is a debugging statement that is there for a normal condition that the authors are, nonetheless, not entirely happy about. There are two reasonable defaults (be noisy, or be silent), and they do not know which to choose.

My previous trip into the field of logging was twenty years ago, when I wrote a fun Emacs packages for putting System.out.println statements into the code (then taking them out again); as one person said of me, I “simply lack the knowledge” of logging frameworks to comment. I am happy to accept that this is entirely correct. Despite this my conclusions are simple. I remain convinced that SLF4J should have the courage of its convictions; if there is an error, signal an error, otherwise remain silent. Until it does this, adding the nop dependency remains the right way forward for me. Take my advice with a distinct pinch of salt, certainly. At the same time, think how many people have been irritated by that error message and not taken the time to comment. If you are worried about these people, then using the nop dependency may be the right way forward for you to.

And there I shall leave the discussion and comment no more.

Bibliography

Shortly after the release of Tawny-OWL 1.5.0 (http://www.russet.org.uk/blog/3110), I noticed a strange message being printed to screen, which looks like this:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.

What strangeness is this? I had never heard of SLF4J before, and could not see why this was happening; so I googled around, and found out. The reason is that I had switched versions of the OWL API (http://dl.acm.org/citation.cfm?id=2019471), and this now uses the SLF4J API as a logging API. By default, it prints out this error message.

Slightly irritated by this, I tweeted

#slf4j I mean, who decided to print a message to tell me I am using a no-op logger. How is that no-op?

To which the author of SLF4J Ceki Gülcü kindly replied:

@phillord if you place slf4j-nop.jar on the class path the nop message should not appear.

But this seems unsatisfying to me. The reason for the message is that SLF4J is a facade, and needs a logging backend. It prints this error message when it cannot find one. The way to switch it off is to include the slf4j-nop dependency (which, ironically, does not contain the NOP logger — that’s in slf4j-api).

Normally, I would say that a library should not print to stderr in normal operation. But I can appreciate SLF4J logic here; it has to do something by default. For most libraries, my response would be “use a proper logger then!”, but, of course, SLF4J does not have that option — it cannot do this during bootstrap, and it either prints to stderr or just dumps all logs.

Okay, no worries, just include slf4j-nop in your project? Unfortunately, SLF4J say explicitly not to do this.

If you are packaging an application and you do not care about logging, then placing slf4j-nop.jar on the class path of your application will get rid of this warning message. Note that embedded components such as libraries or frameworks should not declare a dependency on any SLF4J binding but only depend on slf4j-api

Now, for a library/application/textual UI like Tawny-OWL (1303.0213), the distinction between “library” and “end-user” is not that clear, so this advice is not that useful. SLF4J justify this by saying:

When a library declares a compile-time dependency on a SLF4J binding, it imposes that binding on the end-user, thus negating SLF4J’s purpose.

Of course, imposing a binding of the end-user is a negative thing, but then is not forcing users to choose a binding also an imposition? It is made worse in my case, because I also get this message from ELK, another dependency of Tawny-OWL:

log4j:WARN No appenders could be found for logger
(org.semanticweb.elk.config.ConfigurationFactory).
log4j:WARN Please initialize the log4j system properly.

A similar error message for a similar reason. I now have to explicitly pacify the logging from two libraries in my project!

The curious thing about the whole situation is that everyone is behaving entirely reasonably. I can fully appreciate the logic of SLF4J, I can fully understand why the OWL API uses it, and I hope that others appreciate why I do not want print outs to stderr. It’s another one of those situations which show that software is difficult, and that often there are only compromises; whether something is a solution or a problem depends on where you sit.

For what it is worth, my own advice is that libraries and framework developers should always include the slf4j-nop if they use slf4j-api. If their end-users want logging, they can always use dependency exclusions to enable it.

I’d like to thank the members of the SLF4J users list, and Ceki for their helpful and polite responses to my questions. You can read the full thread on gmane. This article was, however, written without their endorsement.

  • Update

I have written a followup to this article.

Bibliography

I am pleased to annouce the 1.5.0 release of Tawny-OWL, the fully programmatic ontology construction environment.

As with the 1.4.0 release, there has been quite a long gap between releases. The main change for this release has been the move to the OWL API version 4.

The next release of Tawny will include more extensions to the patterns.

Tawny 1.5.0 is now available on Clojars.

I was entertained to see that Springer recently retracted a set of papers, having apparently detected a set of fake reviewers (http://retractionwatch.com/2015/08/17/64-more-papers-retracted-for-fake-reviews-this-time-from-springer-journals). The game seems to be that authors suggest reviewers who are real people but with fake emails owned by the authors. Allyson Lister, a colleague of mine, was twice the victim of this form of identity theft (http://themindwobbles.wordpress.com/2011/05/04/stolen-professional-identity/) (http://themindwobbles.wordpress.com/2011/07/19/a-new-journal-a-new-bogus-review-again-the-culprits-are-banned/) a while back so I am perhaps less surprised to hear that this is happening than some in the scientific community.

Now, of course, this is a form of fraud and is not something than I would condone. Ally lost some time chasing the situation down, but I do not think that there were any negative outcomes for her other than this. She also got to write some entertaining blog posts about the issue. But there is clearly a reputational risk as well. If the review apparently coming back from Ally had been of poor quality, then that, by itself, is an issue.

For the journals, my immediate response, though is to hope that they got the money back that they paid for these reviews. However, hidden behind the flippancy is, perhaps, a serious point.

The serious fraud office define fraud as “an act of deception intended for personal gain or to cause a loss to another party.” You can argue the former here, but they the last? The fake reviewers do not stand to gain any money from the publishers, so clearly they are not trying to cause loss to them. The best you can argue is that the authors using fake reviewers are trying to cause loss to their employers at some point in the future, gaining promotions to which they are not entitled.

The journals say that organising reviews and ensuring peer-review is a significant contribution that they are making; yet, to achieve this, they are dependent on people with who they have no existing business relationship. It’s all done on a freebie, a gentlemens agreement. Does this really make sense? If the whole system were moved toward a more formal basis, then suddenly, fake reviewers would clearly be in breach of contract, as well as clearly be defrauding the reviewers. Even the basic machinary of payment would help to prevent the fraud; instead of just requiring a name and email, journals would need to know bank accounts, tax details and so on.

All in all, I think, this supports provides another strong reason why we should pay reviewers (http://www.russet.org.uk/blog/3095).

I am not sure why, but while writing this it became clear to me that the argument would be much better supported if also represented as a sea shanty. Middle age hitting me no doubt, but here we go…

What shall we do with the fake reviewer?
What shall we do with the fake reviewer?
What shall we do with the fake reviewer?
Early in the Morning

Pay Reviewers, Sue the Fake Ones
Pay Reviewers, Sue the Fake Ones
Pay Reviewers, Sue the Fake Ones
Early in the Morning

Full Economic Costs are Rising
Full Economic Costs are Rising
Full Economic Costs are Rising
Time to Pay the Piper

What shall we do with the fake reviewer?
What shall we do with the fake reviewer?
What shall we do with the fake reviewer?
Early in the Morning

— Phillip Lord

Bibliography

I have discussed the issue of peer-review before, and my frustration at being asked to review for journals with high submissions fees (http://www.russet.org.uk/blog/2327), something for which we as academics gain very little credit for.

I recently fell across by an economist who acts as an editor for a journal (http://marcfbellemare.com/wordpress/10776), which has an interesting perspective. One comment he makes rings true.

When you receive a request to review, please respond, and please respond as quickly as possible. I am always baffled by how many people fail to ever respond to requests to review.

— Marc F Bellemare

I have always felt slightly guilty about this, although I do get a fairly large quantity of spam about reviewing, much of which is not particuarly directed at me and often with short deadlines. And much of which says “please log into our terrible system to accept or decline”. Which you need a password to do. Which you have to request from a different part of the terrible system. But, ultimately, I am aware that in many cases, there is a person at the end, sending a perfectly reasonable request. And I do not always reply. For me, this goes against the grain, as I try to be helpful and courteous whenever I can, as life is more pleasant this way. There are limits, of course; this is my job not my hobby, and if I gain little value from an activity, at some point the time cost is going to become unjustifiable.

Recently, there has been some interest in Publons, a new website which attempts to change this. With Publons, you send your review in and then later your send in the confirmation email from the journal about your review, and this is associated with your profile. It opens up the peer-review process because the reviews end up public, making a process that I have attempted to do for myself before (http://www.russet.org.uk/blog/2366) considerably easier. For that purpose, Publons seems like a good thing. But, also, it attempts to bring some credit to the process; over time, you build your profile up and the metrics associated with it.

Now, all of this seems like a nice idea. Reviewers gain credit. But, there again, what do these metrics mean? And how important are they to me, yet another metric from yet another social network? It’s nice to get a new Twitter follower, a facebook like or even “earn” some Eve-Online ISK. But, to justify the effort I need something more than nice.

Marc Bellemare offers a different perspective, that of an hard-working journal editor, during which he describes his feelings about the responses to requests for reviews.

Consider these quotes:

In this business, “I’m too busy” is a really bad excuse and often means “I don’t care.” Have you ever met a pre-tenure academic who was not busy? […] if you truly care about something, you will make time for it, even if it means getting up at 4 am to do it

— Marc F Bellemare

From my background, I like the idea that you should do reviewing because you care about it, because you care about the science (or economics), and you want to do a good job for yourself and your community. There again, if you have to get up at 4am to do pretty much anything, then yes, you are indeed too busy. There might be a few exceptions, I guess, mostly associated with travel which sometimes requires odd hours (I wrote some of this at 4am, and yes, it was travel), but I do not and will not work at 4am routinely. Perhaps, this makes me unusual. Maybe many of my papers have been reviewed by an exhausted colleague at 4am; it would explain the grumpy or just plain confused tone of some of them.

Interestingly, though, this rule appears not to apply to everyone.

That said, it is perfectly legitimate to decline a request to referee if you are senior in your field […] But if you are a recently minted PhD and you tell me “I just don’t have time for this right now,” […] chances are you are not going to impress anyone.

— Marc F Bellemare

Personally, I do not understand this at all. If you are a “senior” in your field, then you probably have a permanent position, it’s probably reasonably well-paid, and probably reasonable secure. If you are a “recently minted PhD”, you probably have none of these things, in addition to a significant debt that you are servicing. It’s even less clear to me why, under such circumstances, you would want to get up at 4am, however much you care. Surely, he has a stronger argument than this for why we should review? , the only other answer he gives is this:

[…] the journal’s editors will often be among the people asked to provide external letters for tenure and promotion cases, and good citizenship in your discipline tends to get rewarded.

— Marc F Bellemare

Frankly, this seems a pretty weak argument — good citizenship tends to get rewarded; just a tendency and you have to wait till the tenure committee stage of your career if you haven’t left before then? I also dislike the implicit threat (don’t review, and we’ll rubbish your tenure case), which explains why he excludes seniors, as I dislike the general tone of arrogance and entitlement in the whole article.

Putting that aside, the more important question is how much do tenure and promotion committees really care? In my experience, they are far more interested in high scoring papers for REF or that your research is sufficiently expensive (http://www.dcscience.net/2014/12/01/publish-and-perish-at-imperial-college-london-the-death-of-stefan-grimm/). The world has become metric-based (http://www.hefce.ac.uk/rsrch/metrics/), and good citizenship is a marginal issue at best.

I was entertained, therefore, to read that Nature is experimenting with actually paying reviewers for their reviews (http://blogs.nature.com/ofschemesandmemes/2015/03/27/further-experiments-in-peer-review). As the publishers say: “Authors submitting a manuscript to Scientific Reports can choose a fast-track peer-review service at an additional cost.” The costs are passed directly onto the authors of course. The practical upshot is that “Authors who opt-in to fast-track will receive an editorial decision (accept, reject or revise) with peer-review comments within three weeks”

Of course, the general feedback to this annoucement was not particularly positive (http://svpow.com/2015/04/01/pay-scientific-reports-extra-to-bypass-peer-review-altogether/), and it seems that Scientific Reports will not be carrying on with this process (http://www.nature.com/news/concern-raised-over-payment-for-fast-track-peer-review-1.17204). Concerns were expressed that it could affect the independence of review, that the reviewers would be no good (presumably because they are poor “newly-minted” PhDs), and that it would favour scientists with lots of money.

But I am less convinced that it is such a bad thing. The reviewers are not paid by the scientists directly; it’s unclear that the current system produces good reviews and the entire system favours scientists with lots of money; after all, they pay the oft-mentioned newly-minted PhDs to do the work and write their papers anyway.

Ultimately, I have come to the conclusion that the contribution of the reviewers need to be acknowledged in some way; the general way that we acknowledge work is money. The journals get paid, so why should the reviewers not do so also? And unlike publons metrics or unquantified brownie points from editors, I can spend money at the supermarket.

So, I think it is time that reviewing was put onto a reasonable footing, and the same footing that most of the rest of our lives run on. Don’t get me wrong here, I do enjoy my work: fundamentally, research retains its excitement for me, and teaching is a privilege. But I would probably do very little of either if I won the lottery and had enough to live on without. It’s my job, and I do it because I get paid.

I have, therefore, put my terms and conditions for reviewing up for everyone to see. I think that they are reasonable and proportionate. Of course, I have not worried about my reviews impressing anyone for years, and I hope that I will plan my workload so that I never have to get up at 4am. More likely, of course, the journals will just say no (one has already), and I will get asked to review less. But, either way, I will be able to respond as quickly as possible and that will remove the one piece of guilt that I feel about the whole process.

Comments welcome as always.

Bibliography