PaperScope

Question

Can an AI paper evaluator become more useful by learning a venue's historical standards without leaking the final review decision into its input?

What I built

• Created forum-level datasets from OpenReview while keeping model inputs closed and final decisions private during prediction.
• Used disjoint calibration and evaluation sets, plus hash checks, to reduce accidental leakage between examples.
• Measured rating error, decision accuracy, and false accepts rather than relying on persuasive generated reviews alone.

Main result

rating MAE in the descriptive ICLR pilots

−21.8%

decision accuracy after calibration

62.9 → 77.1%

false accepts in the recorded comparison

9 → 2

Result figures

What the work showed

• Venue calibration improved each of the three recorded descriptive metrics in the ICLR pilot runs.
• The largest practical change was fewer false accepts, suggesting the calibrated evaluator became more conservative in the tested setting.
• Leakage controls were essential because a strong score is meaningless if review outcomes can reach the model input.

What it does not prove

• The pilot results are descriptive and were not reported as statistically significant.
• The evidence is not yet cross-venue; behaviour on other conferences or fields may differ.
• The system estimates historical review behaviour. It does not define scientific quality or replace expert review.