Critiques

The CFR does not just believe in replicability; it believes in actual replication.

It is not difficult to write a straightforward paper that has a good chance to be accepted into the CFR. Take the set of papers that are about 10-15 years old, that have not yet been shown to have serious issues, and that have gathered a lot of citations in the last 24 months. Reproduce the paper first with the old data to make sure that you fully understand it. Then think about what problems the paper might have. Clarify for the profession what the problems are—empirical papers are by nature never perfect. We just need to understand the shortcomings better. Be critical but fair. Do not accuse the authors of malfeasance.

Why?

Independent replication is the only thing that distinguishes science from religion. As of 2016, top academic economics and finance journals publish far too few replications. Ideally, each paper should be independently replicated 2-3 times before it is to be believed. (This means following the finding of the authors, not just following the same steps in exactly the same order with the same data.) Despite popular impression to the contrary, peer review des not usually replicate anything.

The CFR publishes many more critiques than other journals. It is tilted in favor of publishing critiques. It views critiques as "conversations" about economic subjects and academic papers, the same way we view academic seminars as venues to understand ideas and evidence better. Contradictory and controversial aspects are better discussed in open forums than in anonymous referee rejections.

Unlike in other journals, the null hypothesis at the CFR remains the "nothing really works" hypothesis. (In inferential statistics, the term "null hypothesis" usually refers to a general statement or default position that there is no relationship between two measured phenomena, or no association among groups.) A published original paper usually claims that it could reject the null, usually at the 95% level. It's as burden of proof is "beyond reasonable doubt" (not "beyond shadow of doubt"). However, this it is not enough to warrant that it should itself become the null hypothesis after publication. This is a good reason why we as a profession should have a higher bar for rejecting the null hypothesis than for rejecting the rejecting of the null hypothesis. Thus, at the CFR, a critique is following the lower "credible evidence" standard.

For most phenomena, the empirical evidence is often not as black and white as presented in an original paper (that presumably had to be surprising and novel to qualify for publication in top journals' rat-race editorial processes). The point of critiques is to learn more about the hypotheses and data, and do so under less pressure than original papers have at the journals. Even more obvious and less surprising issues are fair game.

Being generally very critique-friendly, the acceptance of a critique does not (necessarily) mean that the editor shares the views of the critique. Instead, critiques often raise interesting problems without necessarily invalidating the original paper. It is often the case that both the critique and the original authors have something interesting to say, even if (or perhaps because) they do not agree.

The CFR also plans to publish selected independent replications (with discussions and optional critiques) of exceptionally influential papers, even if they confirm original papers.

Advice on Writing A Critique

A critique should not be a paper assassination, much less a character assassination. The critique authors should make every effort to remove inappropriate language from their manuscript. This especially includes insinuation as to the motives of the authors. Such speculation is never necessary or interesting to readers. It is also petty and reflects badly on the writer. Readers are only interested in how to draw the best inference from the data today. In many cases, a critique presents another view of the evidence, and not necessarily the only view. Many a good critique and many a good rebuttal have rewarded readers with a better understanding of the phenomenon under investigation.

The guidelines for writing critiques boil down to a few simple rules:

1. first replicate the original paper as closely as humanly possible (data, sample period, coefficients, etc.) before you critique it. show that you understand and can do it.
2. be polite and gracious. never doubt the motives of the original author(s).
3. state in the most clear and obvious language where you think an inference was wrong. here you should be tough.
4. focus sharply on the clearest issues. if an original paper shows a phenomenon in two specifications, and your paper shows that only one is wrong, then say so. do not try to take issue with the other original specification that you believe to be correct. instead, state prominently that (after you have checked into it) that this other specification remains valid.
5. never let personal like or dislike of authors color your critique. avoid critiquing the same original author twice in your career.
6. a critique is primarily about an underlying economic phenomenon and secondarily about the paper that brought evidence to bear on the phenomenon. we need to understand the original paper only because we must reconcile the differences in conclusions.

In addition to any substantive points, both the original paper authors and critiquing authors should point out to the editor where the language of the other piece is inappropriate. The more specific the pointers, the more likely it will be that such language will disappear. Again, both critiques and rebuttals should not be filled with gratuitous remarks.

I will give the critiqued authors very wide latitude to answer the critique in print immediately following the paper. At this point, the critique authors get no more chance to respond. If a critique is bad, the critique authors will have made fools out of themselves. And, unlike ordinary journal publication, there really is someone with a stake eager to explain all problems right away. The original authors will find it!

However, I also expect original authors to point out mistakes in the critique early. The point is not to ambush either the critiqued paper or the critique.

Process (and Special CFR Handling) Procedures

Before circulating a critique (and definitely before submitting a critique to any journal), authors are strongly encouraged to send their critique to the original authors, with a polite request for comments. This can often help prevent a major embarrassment for the critique authors.

If you write a critique of a paper published in a top journal, always share it with the original authors and then submit to the original publishing journal first. Then try the CFR second. This is the CFR's special niche. It's the best journal for critiques.

Publishing critiques is a delicate process. The CFR is not in the assassination business. It is in the scientific findings business. The CFR has more experience in handling critiques, and it is constantly improving the process by which it is handling these types of submissions. This document explains the process, as it stands today. The editorial process for critiques takes longer than the editorial process for ordinary submissions. (The CFR does not publish turnaround statistics. It is focused on quality evaluation and reports, not turnaround times. However, it believes that most papers should be reviewed within six months.)

CFR Process Steps

1. An author submits a critique of one (or more) papers.
2. The original authors are asked to write a memo (often a rebuttal). This memo is never anonymous, neither to the real referee nor to submitting author. It is also not deciding. The original authors are given one week to decide whether they want to write a response. If they decide to do so, they have four weeks to do so, plus they will (usually) be permitted to author a rebuttal in the CFR immediately following their critique. (If they decide not to do so, the editor may or may not invite them to publish a response in the CFR, but sofar a response was always invited. Note that this memo is in essence and format simply a non-anonymous referee report.)
   • Advice for the Memo: The incentives are really quite simple. Be factual and neutral. A critique is not of authors but of a specific paper. If the critique author made an important simple mistake (such as a math or data error), the best choice is just to point it out. The real referee seeing the response will presumably agree early on and advise rejection. If the critique author made a conceptual mistake, this is also good to point out to the referee. If the critique author made a mistake in his assessment of importance, emphasis, etc.—and disagreements of this type are often appropriate (what kind of non-trivial paper would be void of such?)—the original authors get to write a response that will be published immediately after the critique and that can explain these kinds of objections quite fairly.
3. The critique and the (non-anonymous) memo are sent to the "real" referees. It is only these real referees that can recommend editorial decisions.
4. The editor makes a decision based on the paper, memo, and editorial suggestions of the real referees. The editor sends the non-anonymous author memo and the anonymous referee reports to the critiquing authors. If the paper is rejected, the process ends here.
5. The critique author is strongly encouraged to address objections raised not only by the referees but also by the original authors (in the rebuttal memo).
6. When the critique is accepted, the original authors are invited to write a response, usually up to half the length of the critique itself, that covers points they raised in their first response. The critique author does not get the chance to respond to comments. (S)he should have addressed this ex-ante in the revision of the critique itself. As a result, the original authors and not the critique author can always get the last word, but only after they have given the critique authors an opportunity to address the rebuttal points in the critique revision. This process may not always be ideal, but there must be closure to it in the journal and for the original authors. Any further points can be posted on the critiquing author's own website.

Publishing a Rebuttals Following a Critique

There is some limits on published rebuttals, however. Published rebuttals should also not introduce substantially new points, unless the critique revision itself did include new points, too. (The process should converge!) If the original (critique) authors raised new points in the revision, then the rebuttal authors can address them. If the to-be-published rebuttal raised new points not in the first draft of the rebuttal that the critique authors saw, then critique authors may be allowed to address these in an additional appendix, at the discretion of the editor.

Traditionally, the CFR gives the original authors a lot of latitude in their responses. We believe that even if the original authors ultimately choose not to publish any rebuttal, their option to do so will keep the critique authors honest.

The original critiqued authors are free to submit everything from a one-paragraph response ("the critique authors are wrong") to a full response ("the critique has a point, but in another analysis, our results happened to return"). The CFR critiques are of scientific findings, and presumably the original authors published their original paper out of interest in the subject, not out of interest in staking out a political position. Even if the critique itself is wrong, the critique author was a reader of the original paper; and the original authors should want to explain to the scientific community why their interpretation of the data are correct and the critique perspective is not. A solid response should not be difficult.

Usually, when the CFR does not publish a rebuttal, the original authors chose not to submit one. This can be interpreted in a variety of ways, not all of them flattering.

Lack of a Rebuttal

How should readers view lack of original authors' responses?

Papers published in the CFR are not careless cranks, but have been vetted by editors and referees. This does not mean that the critiques are correct, but it does mean that they contain questions and perspectives reasonable enough at least to warrant further explanations by the original authors.

Thus, the primary reason for original authors choosing not to respond seems to be

• The critique is correct.

Otherwise, why would original authors not be interested in discussing a reasonable published response on a topic which they considered important enough to publish on themselves in the first place?

There is on occasion one other reason:

• The original author(s) have moved on and become too busy to take an interest in their own earlier work.

IMHO, in such cases, the reader should again give the benefit of the doubt to the critique.

Replicability

Unless there are strong specific circumstances prohibiting it, critique authors must post data sets and computer program (Stata, R, Matlab, etc.) that allow the original authors (and subsequently CFR readers) to replicate the critique findings easily. The CFR will not publish critiques based on proprietary data. For commercial data, we require programs and instructions of how to obtain the data. Posting programs and data early on will eliminate forth-and-back whether critique findings are correct or not.

The CFR also expects the critique author to answer short specific clarifying questions by the original paper authors, relayed through the editor.

Time Extensions and Updates

Unlike other journals, the CFR will publish important "updates."

If a paper was published in a top journal 20 years ago and its findings continue to be widely cited, this paper still remains interesting.

For argument's sake, say this paper shows an effect with a T-statistic of 2.5. However, when one runs the analysis with exactly the same kind of data (not just similar data, such as from other countries) but over the last 20 years, if the T-statistic in the new data is negative significant, and/or the net T-statistic is now under 1.0, then this is definitely important news. It is important even if the update offers no methodological updates. Such an update is not critiquing what the original authors did. It simply points out that the evidence and inference have changed.

The time-change effects must be itself meaningful. If the overall T-statistic drops merely to 2.0, this is not enough and if the mean effect in the new data is still positive, then this is not enough. It has to be a negative (but not necessarily significant) mean effect in the new data and a reasonably large change in the inference in the overall sample. If just the mean has dropped, the new-era T-statistic is +1.0 and the updated overall T-statistic is now +2.0, this is not enough yet. We then need to wait and accumulate more data.

A Typical First Letter

Here is a typical letter that I send to original authors after receiving a critique:

The CFR also welcomes good and competent critiques of papers from this document.