What does a decision do to the person who receives it?

Treatment depends on the judge's leniency and on the case's own resistance. The effect is largest for those who most easily get treatment. With judges who flip for men, monotonicity fails. With a direct judge effect, exclusion fails. With learning, judges change their leniency when their decisions are reversed.

Explanation · terms, estimators and tests

How to read the table

The table for the design you chose, step by step

1. Calculating …

The basic idea

Treatment and outcome

The treatment is the decision, for example that a defendant is released before trial. The outcome is what we want to know the effect on, for example whether the case ends in a conviction. In the simulation they are called d and y.

Leniency

How often a judge gives the treatment, compared with other judges who get the same types of cases. A lenient judge releases more.

Instrument

Something that moves the treatment but does not affect the outcome in any other way. Here the instrument is the judge the case lands with. Assignment is random. Those who meet a lenient judge are therefore like those who meet a strict one. The difference in outcomes between them is then due to the difference in treatment.

Why not just compare?

Those who get treatment are not like those who do not. In the simulation, those with the lowest resistance most often get treatment, and they also have different outcomes. A simple comparison mixes the effect with who gets treated. That is why OLS misses by so much.

True effect and compliers

The instrument measures the effect for those whose decision actually depends on the judge. They are called compliers: they would have been treated by a lenient judge but not by a strict one. The true effect in the table is the average for them, between the strictest and the most lenient judge in each cell. Both potential outcomes are simulated, so it is known.

Cell and shift

Cases are assigned at random only within a cell, for example a court in a year. The estimates therefore control for the cell. A shift is a group of cases that go to the same judge on the same day. Cases in the same shift share a shock, for example a busy day.

Standard error and deviation

The standard error says how much the estimate would vary with new data. The deviation in the table is the estimate minus the true effect, measured in standard errors. More than two standard errors away is unusual if the estimator is right. Such rows are in bold.

A worked example: why the case must be left out of its own judge's rate

A judge has ten cases and releases four. The rate is 0.4. Take one of the cases where the defendant was released. If the rate is computed with this case, it is 0.4. Without the case it is 3 of 9, that is 0.33. The released case thus pulls its own judge's rate up. For a case where the defendant was not released, the rate without the case is 4 of 9, that is 0.44. That case pulls the rate down.

The instrument is therefore higher precisely for the cases that were treated. That is the same as using the treatment itself, and then we get the OLS error back. With ten cases per judge the error is large. With a thousand cases it is small. Jackknife means that the rate is always computed without the case itself, or without the shift the case is in. Then the case cannot affect its own instrument.

The estimators

OLS

Compares the outcome for treated and untreated within each cell. Does not use the judges at all. Biased when the treatment is related to other things that affect the outcome, as it is here.

2SLS with judge dummies

First estimates each judge's treatment rate, then uses the estimated rate as the instrument. The problem is that each case helps estimate its own judge's rate. With few cases per judge the rate then picks up the case's own treatment. The estimate is pulled towards OLS. Fine with few judges and many cases each.

IJIVE

Jackknife: the judge's rate is computed without the case itself, so the case cannot affect its own instrument. IJIVE first removes the cells from the data and then leaves out the case. Close to UJIVE in practice, but not entirely free of bias when there are many controls.

UJIVE

Also jackknife, but correct also with many cells and many judges (Kolesár 2013). The case is left out both when the judge's rate is computed and when the cell's average is computed. The instrument is the difference between the two. The recommended default in both guides. Assumes that cases are assigned one at a time.

UJIVE, leaving out the shift

Like UJIVE, but the whole shift is left out, not just the case. Cases in the same shift can share a shock. The other cases in the shift would otherwise leak that shock into the judge's rate. Correct when cases are assigned in shifts.

CJIVE

Cluster jackknife (Frandsen, Leslie and McIntyre 2025). The same idea as the previous line, as the JEL guide builds it in its example: remove the cells, then compute each judge's rate without the shift the case is in. Standard errors are clustered on shift. The guide's choice when cases are assigned in shifts.

Which estimator should you trust?

Few judges, many cases each

All the instrument estimators give about the same answer. 2SLS with judge dummies is fine. Choose a jackknife variant anyway. Then you need not assess whether there are enough cases.

Many judges, few cases each

UJIVE. 2SLS is pulled towards OLS, and its standard error is too small, so it looks precise without being so. Try the setup with many examiners in the simulation and see how far 2SLS lands from the true effect.

Cases assigned in shifts or batches

CJIVE or UJIVE leaving out the shift, with standard errors clustered on shift. Clustering on judge is wrong in both guides: the assignment of cases is what decides.

Doubt about the assumptions

No estimator rescues a design that violates monotonicity or exclusion. Check the tests, but remember that they can miss violations.

Uncertainty

Confidence interval

The estimate plus and minus about two standard errors. The interval covers the true effect in 95 of 100 new samples when the estimator is right. In the simulation you can see whether it does.

Clustered standard errors

When cases in the same shift share a shock, they are not independent observations. The standard error must then be computed as if each shift were one observation. Otherwise it is too small.

How much data is needed

Precision depends mostly on how different the judges are, and on how many cases are assigned in total. Small differences between judges give large standard errors even with many cases. That is why the standard error for OLS is much smaller than for the instrument estimators, even though OLS is wrong.

First-stage F

A measure of how strongly the judge moves the treatment. Both guides advise against rejecting a design because F is below 10. With UJIVE and many judges a small F is not a problem in itself.

The tests

Monotonicity

The assumption that a more lenient judge is more lenient in all cases. Suppose some judges are lenient towards women and strict towards men. Then it does not hold, and the instrument can give a biased average effect. In the simulation you control this with the slider for judges who flip for men.

Exclusion

The assumption that the judge affects the outcome only through the decision. If a strict judge also affects the outcome in other ways, it does not hold. The slider for a direct judge effect breaks it.

First stage

How much the treatment rate rises when the judge's leniency rises. It should be positive, also within subgroups such as men and women. A weak or negative first stage in a subgroup suggests a violation of monotonicity.

FLL test

Frandsen, Lefgren and Leslie (2023). If the assumptions hold, the judges' average outcomes should lie on a smooth curve when the judges are sorted by leniency. The curve cannot be steeper than the outcome allows. A low p-value means that something fails. The test requires few judges with many cases each.

GHK test

Goldsmith-Pinkham, Hull and Kolesár (2026). The share of women among the compliers can be estimated. If it is below 0 or above 1, the assumptions cannot hold. The test catches only gross violations.

Balance

If assignment is random, leniency should not be related to characteristics such as sex, age and prior cases. A high joint F-value with a low p-value means that assignment is unlikely to be random.

Learning after reversal

Bhuller and Sigstad (2025, Review of Economic Studies) show that Norwegian district court judges change practice after a ruling is reversed by the court of appeal. They become stricter after a lenient ruling is reversed, and more lenient after a strict ruling is reversed. In the simulation, a decision against the defendant is reversed with probability 0.3. The judge then shifts his or her leniency by the value on the slider. Later decisions then depend on earlier cases, and the judges' leniency changes over time. The leniency measure uses all of the judge's cases, including the later ones.

What learning does to the estimates

In this design, learning makes the judges more alike. A lenient judge who has releases reversed becomes stricter, and vice versa. The differences that the instrument relies on shrink. The first stage gets weaker and the standard errors larger. The estimate moves away from the true effect, but by less than the uncertainty grows. Reversals are rare, so each case has little effect on the judge's later decisions.

When the tests do not reject

Then the assumptions are not proven. With judges who flip for men, CJIVE becomes clearly biased without any of the tests reacting. Both guides say that the tests have little power against such violations.

The tests · random assignment and monotonicity

Would the tests have detected a problem?

The guides · JEL 2025 and JEP 2026

What the two recommend

They disagree on the balance test, on which monotonicity test is best, and on whether judges with few cases should be dropped. JEP advises against building the leniency measure by hand.

Real data · the guides' own examples

Reproduced with the same tools

The tools have been tested on the data behind the two examples. The data are not on this server.

All 24 numbers in JEP Table 3 are reproduced. In JEL Table 4, CJIVE with covariates differs most: −0.507 against −0.500.