Rejection
Some fun rejection comments
Joshua Gans and George Shepherd were doctoral students in economics at Stanford University back in the 1990s. They wrote an interesting paper that I just came across titled, “How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists.” It grew into a 1995 book edited by Shepherd: Rejected: Leading Economists Ponder the Publication Process.
Today I want to discuss the same issue in our area of theory.
Ken and I have not had a chance to do a formal survey of papers that were rejected in our area. We also would not do exactly the same as Gans and Shepherd since it’s not what happens to the “mighty” that matters most but rather to the great band of those doing productive and creative and sporadically uneven work. Our point is rather that all of us who write articles for conferences and journals are subject to sporadically uneven reviews.
So we will today just offer a few things from personal experience to season the grill. We are mostly interested in negative comments from bad reviews. We could also touch on the opposite, heroic reviews that found subtle mistakes—or maybe mistakes missed by everyone including the referees.
Some Rejections
The authors assume incorrectly that the graph
has an even number of vertices in Lemma
The graph in question was a cubic graph. By what is sometimes called the First Theorem of graph theory, all cubic graphs have this property. Just double-count edge contributions and one gets that
where 
“What was missing? Was the problem not motivated? Was the proof unclear?” And so on.
The head simply replied: “we like longer papers.” I pushed and said I thought making a paper longer for no reason seemed wrong. He responded that the paper was now unaccepted.
I could not believe it. We quickly sent it off to an IEEE journal. Don Knuth handled it, and it soon was accepted with minor changes only. By the way the paper solved a simple question: what is the best way to store a triangular array in memory?
Leonid once submitted a paper to a journal and got back a negative review: It said that the paper was too short and also terms were used before they were defined. Leonid responded by taking two identical copies of his paper, stapling them together, and resubmitting the “new” paper. It was now twice as long, which answered the first issue, and clearly all terms were defined before they were used.
It is unclear what happened to the paper.
What is correct in your paper is known, and what is new is wrong.
I hope to never get this one.
Open Problems
We’d love to hear from you with your own examples of strange reviews.
I Get Rejection
I submitted the above post last week to my blog editor but didn’t hear back—I assumed he was overwhelmed with work. Then he replied and asked me to make the post “longer.” I asked back:
“What was missing? Was the issue not motivated? Was the evidence unclear?” And so on.
The editor simply replied, “it’s a bit thin.” I pushed and said I thought making a post longer for no reason seemed wrong. This editor at least gave some concrete suggestions:
Gans and Shepherd give one interesting kind of example where the delay caused by rejection enabled others with similar ideas to get ahead—not on purpose by the rejecter but just-so. They also give some self-revealing quotes including this one by the economist Paul Krugman:
The self-serving answer [to the “why me?” question] is that my stuff is so incredibly innovative that people don’t get the point. More likely, I somehow rub referees and editors the wrong way, maybe by claiming more originality than I really have. Whatever the cause, I still open return letters from journals with fear and trembling, and more often than not get bad news. I am having a terrible time with my current work on economic geography: referees tell me that it’s obvious, it’s wrong, and anyway they said it years ago.
Ken recently heard a true giant admit he gets rejections, “often because the referees don’t believe this work is really new.” Perhaps Krugman’s last clause means the same?
A famous case in our field was the number of times the first interactive proofs paper by Shafi Goldwasser and Silvio Micali was rejected from FOCS and STOC before finally appearing at STOC 1985 with Charles Rackoff as third author. Ken recalls people excitedly telling him and everyone about the work at the FCT conference in Sweden in August 1983. This could have become an example like in the Gans-Shepherd paper of others pipping ahead, but happily didn’t.
A version of it was used by Christopher Chabris and Joshua Hart at the end of their negative review in the New York Times last month of the book The Triple Package:
Our conclusion
In March, Ken took part in an online discussion with Chabris about sourcing it. Ken recalled hearing it in the early 1980s in this snarkier form:
“This paper has content that is novel and correct. However, the parts that are novel are not correct, and the parts that are correct are not novel.”
It was already then a widely-known math cliché. Someone else in the discussion sourced it to the slamming of John Keynes’ book, The General Theory of Employment, Interest, and Money, by Henry Hazlitt in the introduction to his 1960 book, The Critics of Keynsian Economics:
In spite of the incredible reputation of the book, I could not find in it a single important doctrine that was both true and original. What is original in the book is not true, and what is true is not original. In fact, even most of the major errors in the book are not original, but can be found in a score of previous writers.
We wonder if any of our readers can find an earlier source? When did “not true” become “not new”? In any event it was one tough review.
Open Problems
Was my referee right about lengthening the post?
[photo at top; format fixes]
Hi Dick
I’ve seen an early version of the quote attributed to Samuel Johnson in a review of a piece of writing. But see http://quoteinvestigator.com/2013/06/17/good-original/
Cheers
Alan
In terms of famous rejections, there is the story of Einstein’s reaction on a paper with Nathan Rosen he had sent to Physics Review. In “Einstein Wrote Back” John Moffat reports that Rosen had been there when the referee report arrived. “Upon reading the letter…Einstein leapt out of his chair and threw the envelope with the letter and the manuscript into his trash can, which he then kicked loudly around his office. He promptly wrote to the editor of Physics Review vowing never to submit a single paper to the journal again. In the letter, he criticized the editor for not warning him upon submitting the paper that it would be subject to anonymous refereeing.”
Rosen recovered the paper from the trash can, made the changes to answer the referee and published it elsewhere.
Moffat says that this probably was the only time Einstein ever had to deal with pre-publication peer review. In fact, Moffat says that the report suggested modifications and correction of errors and was not actually a rejection but, either being unfamiliar with peer review or simply upset at being questioned, Einstein treated it as a rejection.
Yes, your blog editor performed a great service–in concert with you. Bravo for the superb joint work. My rejection(s)?–: “Insufficient Postage” (prior to Internet submissions–now I get a “404” error … on EMAIL! ! ! ! !…)
Dear Professor,
I have received two rejections with the following messages:
“I regret to tell you that, after checking the contents of your paper, the member of the Editorial Board handling your paper has decided that publication of your article cannot be recommended, as it falls outside the scope of the journal.
Please accept our apologies for the time spent in the review process, which in this case has led to a negative decision, which is not motivated by a negative judgment of your work.
I advise you to submit your paper to another journal.”
This was a journal specialized in complexity….
And another journal said something similar:
“Your manuscript UP versus NP was evaluated by a member of Editorial Board. A preliminary screening proved that said paper did not receive high enough priority for us to proceed further with a detailed review.
Thank you for giving us the opportunity to consider your work.”
This was also a journal specialized in computational complexity too..
I don’t know if the cause is that am I a Cuban or am I an unknown researcher?
Maybe is both. I decided to send it to a Chinese journal, maybe there, my paper can have the revision that I need.
This is the preprint of the paper,
https://hal.archives-ouvertes.fr/hal-01304025
and the pdf file in
https://hal.archives-ouvertes.fr/hal-01304025v2/document
So you can take your own opinion about it.
Best Regards…
I’ve found this Leonid Levin’s solution of this “goofiness problem” (as well as its reincarnation in this post) just nice. Rejection because of “being too short, too simple” is something “devilish”. Something against THE Book holding by God (as Paul Erdos told us). Unfortunately, this trend goes though the entire TCS journals, “most prestige” journals (and conferences): you should spent more time to make your proofs more complicated, less understandable … This said, I am still for more stronger rejections. Each (witnessed) rejection saves time for lots of researches (potential readers). As it was in former USSR: publish only then when you cannot not publish. When you REALLY want to tell something to your colleagues. Not just to hiring committees …
I think it is over the top to say that there is any trend whatsoever, be it in conferences or journals, that encourages obfuscating proofs.
What you probably meant to say is, there is no incentive for authors to simplify their proofs; in fact the incentive is negative. If this is what you meant, then you are absolutely correct.
Yes, I mainly meant what you say: the incentive is, in fact, negative. Especially, for young people: simplifying proofs needs experience and time – things young people don’t have yet. So, having no incentive then actually means a discouragement.
I have right now a paper “on hold” at arXiv:
http://www.andrebarbosa.eti.br/A_Human_Checkable_Four_Color_Theorem_Proof.pdf
And what? Was this (nice) post a “call for papers”?
The hole in your proof is the first paragraph on page 16 and the corresponding Figure 2.22. If that statement were true, then the rest of the proof would work. Even if the statement is false, the proof could still succeed, because at this point it is still only a single case which needs to be investigated further.
Overall, the paper is an honest attempt at a human readable proof. It strives to be understandable despite the missing (English) language skills of the author, using many pictures and references to accessible external sources. The hole is delivered straight, and could be an honest mistake. (It could also be the understandable attempt to “finish” a proof that could otherwise go on and on with no end in sight.) However, the paper contains some serious formal mistakes (or misunderstandings) like “Mathematics Subject Classification (2016)”, “Keywords. …, 2-DSCC_M, 4-CM, …”, or “[1] From Wikipedia, the free encyclopedia, “Four Color Theorem”, unpublished, available: …”.
arXiv is not peer reviewed, I have no idea what “on hold” means in that context.
Dear Mr. Gentzen,
Thank you. I have a lot of papers sent to conferences and blogs, and your comment was [unexpectedly] very nice.
However, it’s common I receiving angry reviews and comments something like (as the defense of new ideas was a criminal behavior…):
“Not relevant to ********.
(Also, please stop spamming TCS conferences with the same submissions.)”
Clifford Truesdell was on the giving end of a classic rejection. It begins, “This paper gives wrong solutions to trivial problems. The basic error, however, is not new.”
It happens occasionally. I have a handful of examples out of many submissions, so I can’t really complain, but they do make for interesting anecdotes.
One of my top cited papers was rejected three times. It originally contained a solution to an important problem as well as a discussion on a heretofore completely missed variant. The accepted version removed the discussion of the second part. Someone rather famous pipped in on the second version and it’s one of his most cited papers.
Another one, also widely cited, was submitted to a prestigious journal, only to be rejected as uninteresting. A followup by others with medium improvements appeared in JACM. So much for an uninteresting piece of work.
A third one solved a long standing open problem, only for a referee to claim it wasn’t such. In the resubmission we quoted well known sources highlighting this problem as open and important to no avail. Finally we gave up and submitted to a very modest conference only to see a paper solving the same problem appear simultaneously in STOC.
The last one, which proposed a radical new approach (with much better results) to a well studied problem was rejected three times from big conferences, only to finally appear in a slightly more modest venue because by then “everyone knew that was the way to do things now”.
A colleague and I received, from a medical journal, a Chief Editor’s letter that read (in essence):
Perhaps medical journals are exceptionally autocratic? 🙂
Further evidence that (to paraphrase Churchill) “Peer review is the worst form of academic evaluation, except for all those other forms that have been tried from time to time.” 🙂
Much more common is “two reviewers loved your article, but we don’t want to publish it for reasons I will not specify, so it is rejected.”
It happens occasionally. I have a handful of examples out of many submissions, so I can’t really complain, but they do make for interesting anecdotes.
One of my top cited papers was rejected three times. It originally contained a solution to an important problem as well as a discussion on a heretofore completely missed variant. The accepted version removed the discussion of the second part. Someone rather famous pipped in on the second version and it’s one of his most cited papers.
Another one, also widely cited, was submitted to a prestigious journal, only to be rejected as uninteresting. A followup by others with medium improvements appeared in JACM. So much for an uninteresting piece of work.
A third one solved a long standing open problem, only for a referee to claim it wasn’t such. In the resubmission we quoted well known sources highlighting this problem as open and important to no avail. Finally we gave up and submitted to a very modest conference only to see a paper solving the same problem appear simultaneously in STOC.
The last one, which proposed a radical new approach to a well studied problem was rejected three times from major conferences, only to finally appear in a slightly more modest venue because by then “everyone knew that was the way to do things now”.
The most hilarious rejection I’ve ever gotten consisted primarily of one reviewer claiming that the results were too obvious to merit publication, and another reviewer claiming that the results must obviously be wrong. We told the editor something along the lines of “Judging by these contradictory reviews, the one obvious thing is that it’s not obvious.”
There is an article entitled “We Are Sorry to Inform You …” by Simone Santini about negative reviews for major researchers including Dijkstra, Code, Shannon and Turing. The article appears here: http://ieeexplore.ieee.org/xpls/icp.jsp?arnumber=1556500.
There is also this article, it refers to some negative reviews that some referees gave to some of the most important papers in the history of computer science
http://www.fang.ece.ufl.edu/reject.html
There are cases, hopefully not so many, where the result of the review of a paper is of the form
“I don’t understand what you are talking about, so it must be wrong. Therefor your paper is rejected”
It wouldn’t be bad to pass the review of a paper to some other peer-reviewer if you have trouble handling it.
And then what should be done with a rejection like this one:
” … nobody is willing to read your paper”.
The best (worst…) rejection I got: “this manuscript contains 11 different Greek characters in its equations and makes me unable to read it”.
Later on, it was published here:
http://link.springer.com/chapter/10.1007%2F11732990_18
About your folklore rejection, the version I heard had a bit more poetry to it:
“There is much that is new and true in this paper. Unfortunately, what is new is not true and what is true is not new”
It must be wonderful to live our fiction.
Dick seems to want to know what happened to the terse one with the length doubled. As a ‘referee’, I can oblige. I rejected it.
Let P be a paper submitted for stapling.
Let C be a concept and D be definition of C.
P has C used before D and PP has C before D before C before D. The convolution is nice but does not help. C still appears first before D. I am quite happy that I rejected it, not based on the possibility of loss of common sense but on the sole basis of this business, whatever the heck that was, of not knowing how to staple things properly.
But that is obvious and pure myth. In the reality we dream, the paper was actually accepted. Levin just put the self-same copy right back in the rejection letter envelop and wrote on it:
The round-trip envelop was given no additional postage and I read with hellish amazement. I had to accept. It now suddenly had D before C and I found not a single Greek letter that I could recognize in the entire thing!
The next morning, over breakfast, my young daughter was curious.
“Dear parent, I like it so much. The recent show of formal capabilities is stunning. But how did you develop the skill of reading the newspaper from the back, holding it upside down?”
I swallowed the words that rushed to just behind my lips and merely smiled. I could not say that it was not the contents but the sheer formidable name. I did not want to kill the innocence and aggravate the complexity of the problem of counting adults in this piece of reality that we call the world. Anyways, Dick, we probably should not issue the stealth complaint so daringly. The situation is of OUR own creation. It is always easier to hallucinate, but harder to measure, the height of our own moral ground. I nevertheless hope that, from now on, everybody knows how to properly staple in a protest.
As to the brevity of Levin, people may read The Longest Second to see the possible impact.
We sometimes try to be cute. But it is always prudent to first know what is cute before trying. Attempting to be cutely poetic can often turn out dearly lousy lines. The lyrical “What is new is not true and what is true is not new.” sounds infinitely stupid.
What is new, IMHO, can not be not true, whether by or not by PUC logic.
— Roielle Hugh
There is a classic example in the field of High Energy Physics of a rejection which allowed others to get ahead on Nobel prize winning work. It concerns the invention of dimensional regularization by G. ‘t Hooft and M. Veltman (1999 Nobel prize winners), who were the first to publish about this revolutionary technique after the paper by Argentine physicists C.G. Bollini and J.J. Giambiagi was rejected from the prestigious journal Physics Letters B, and only got published months later in the not-so-well-known Italian journal Il Nuovo Cimento. At the time, the idea of working in a complex number of dimensions d = 4-\epsilon was so unconventional that the referee ironically suggested Bollini and Giambiagi stop wasting their time and get back to work in four spatial dimensions (the original version of the paper did end up being published by Physics Letters B later on, after much discussion).
Although the original paper by ‘t Hooft and Veltman cites Bollini and Giambiagi’s preprint ( http://www.staff.science.uu.nl/~hooft101/gthpub/regularization_renormalization.pdf ), they were not mentioned by the Royal Swedish Academy of Sciences press release or ‘t Hooft’s Nobel acceptance speech. Most papers today refer only to ‘t Hooft and Veltman’s work, which has over 3500 citations to date (see http://inspirehep.net/record/74886), and neglect to mention Bollini and Giambiagi’s independent work (which has “only” about 1000 citations, see http://inspirehep.net/record/74881 and http://inspirehep.net/record/74400).
There are several detailed historial accounts of this story, one of the latest being http://arxiv.org/pdf/1211.1741.pdf (or https://sites.google.com/site/schaposnik/sobreelpremionobeldef%C3%ADsica1999 in Spanish). However, the whole issue did not develop into an actual dispute as ‘t Hooft and Veltman’s Nobel prize was more than well-deserved, Bollini and Giambiagi being the first ones to recognize it.