though it does have correlations to rating and citation.

clearly prove

use footnote sizeing

you can make the width smaller, this doesnt look proportional, and also the caption should be cnetered under the table.

you are using old formatting, please remove and use the new macro

i am confused, we never mention deepreviewer in the table. what are you referring to?

also the columns should be PaperLens-Vision and Best Baseline btw.

just say PaperLens (no need for 7B specificailly)

again you should use OR-ICLR in the table

Best Baseline

hanging line

OR-ICLR

hanging line

use the text and vision macros defined in the main table

i just realized at the very end we want a short hardware details subsection. where we mention that all of our SFT/RL experiments use 4 H200. we used liger-kernel @article{hsu2024liger, title={Liger kernel: Efficient triton kernels for llm training}, author={Hsu, Pin-Lun and Dai, Yun and Kothapalli, Vignesh and Song, Qingquan and Tang, Shao and Zhu, Siyu and Shimizu, Steven and Sahni, Shivam and Ning, Haowen and Chen, Yanning}, journal={arXiv preprint arXiv:2410.10989}, year={2024} } to reduce memory usage, allowing per device btach to be 4 (and grad accum is set appropriately, i.e., 32 batch, 4 per device batch, 2 grad accum).

i dont think we need the exact numbers. just qualitatively explain the trend that even tho reward does go up, as the model learns to format better, its not actually that much more accurate, and retains the initial accept bias.

delete this. simply state that though the models improve their reward, they retain their accept bias, and it even gets worse with RL training.

there is no checkpoint policy. just state that for 7b/14b training we use 2 epochs, for 3b we use 4 epochs.

you need to also put the conclusion lol. also be consistent iwth OR-ICLR macro

hanging line

we need more content here. Also ICLR should be OR-ICLR (italics) with macro. I think we want to say that the modes spend anywhere between ~2k to 4.5k thinking, with gemini ceiling noticeably higher than gpt.

hanging line

same here add system and user blocks (even if empty) also for all prompts, please fix the line wrapping

please add the system and user blocks here as well.

OR-ICLR

i think we want better labeling for the table here. 1) we dont want to mention that are testing on validation vs test, so just remove that. 2) remove the n= column, 3) we want a column that says forward-trained/reverse-trained. then another column thats text or vision. and then other column that is test-set ordering. got me?

can you add vspace between the tables and subcaptions? they are a bit too close rn.

this is hanging

hanging line

after this i think we want to say, despite the larger claim rate, the accuracy of the claim is less than 1% across all cues, showing that while llms might recall, they cant do it accurately.

isnt this italicised? or supposed to be?

we should specify the anonymization. Here we strip similarly all the identity-bearing spans directly from the latex source, and we also recompile the pdf under the [review] latex macro.

OR-ICLR (needs italics)

put in paranthesis (OR-ICLR), then refer to this throughout the rest of the text plase. (even in the tables)

hanging line

hanging line

i want the arxiv-m to be a right arrow, not down arrow.

if we use the latex symbol we should be consistent. id just use the latex (plain text) throughout

we should be consistent with this symbol. i think we should just always use plain text for it

again delete this

again delete this

delete this step

remove this

this could/should be a wrapfig, please add it in where mentioned

same with this

these need to be weaved into the text

for OR-ICLR

can these be updated, and can you state the numbers for arxix text/vision and OR-iclr?

remove this

the OR-ICLR test set

OR-ICLR

lets make sure we use OR-ICLR unless its necessary and we are tlaking about the dataset curation details where i think its appropriate to use OpenReview-ICLR there

this is the pipeline for OpenReview-ICLR AND arXiv, please fix. also where is the main boxy text?

OpenReview-ICLR

hmm, i think we should add in these prompt variants, otherwise its hard to know what stru tured json or PDR with advocate.critic, etc means

to check whether, with a strong base model (the qwen3.5 model is more likely to be strong than the qwen2.5 base), will scale help prompting?

(a full review with strenghts/weaknesses/etc.)

say 4.3%

these need to be mentioned before, also do you think we could put a and be next to each other and c underneath to save space?

this feels somewhat out of place. where is it referenced, i feel this should be placed earlier, before prompting. we should 1) have a preamble in the prompting and extended experiments. two we should mention this summary table in line in the preamble talking about how all settings are summarized here, notably the forntier models use high reasoning and temperature=1, we also replicate this in our RL prompt ( need to add temperature =0.7 there). then talk about how our sft after trained is greedily decoded? maybe then introduce the prompting we tested and how we did some experiments to correct open-source model bias

we didnt explain what the schema validator is, we didnt explain the predicitons.json or session.jsonl. please explain in the main text, the schema validator is a utility given to the agents to ensure their predictions (stored in the PRedictions.json) is of correct format. we record the entire agent trajectory in a session.jsonl file.

is this the same as the forntier reasoning prompt? if so we should indicate that

However, in our main results, we calibrated the model's decision using its rating and accuracy performance on the held-out validation set. This generally improved performance.

simply state: In this section we denote the prompts and decoding settings used during prompting, sft, and RL. we omit the paper body from the prompts for brevity. also please dont line wrap until you have hit the full width of the pormpt with. it seems like line wrapping is used too early rn

i like what was done in the base open-source prompting, please adopt that ehre by denoting whats in system, whats in user, and then the difference between iclr and arxiv.

same for this

i was thinking this should be its own subsection

please use something other than colon here

this sentence is trange and idt this is meant to be here. remove it. i think the key is that we should state that none of these design choices consistently improve performance, so we choose to simply use paper -> decision prediction

canw e increase the size of these tables to /footnotesize

i think stated this causally as: "we therefore study ..." is weird. i think we should say: "to improve the quality of the corpus, we tried to filter out borderline cases ..."

choose something else other than colon here

delete this

please dont abbreciate the columns, we have more than enoughs pace

we report acc in the main table as a %, id do the same here. Make sure this is clear by stating outcome accuracy when claimed (%)

delete

"contrib"

probably need "avail"

i think this could be shortneed. as well as the previous all we need to say is that we find the abstract content and then either count number of sentences or the length of each sentence (average)

we also need to state that the w_iclr, w_arxiv are derived from logistic regression models trained on all these features

these need to appear closer to where they are mentioned in the appendix

State: Decisions for both accepts and rejects are fully available, so the pipeline is simple: download openreview PDFS and metadata, extract markdown content/apply normalization, and subsample.

how is this balanced?? something is wrong with your metrics, this should be 50/50 accepts and rejects? with our 40k we do a stratified year balance, please recheck

lets remove the red text between the boxes. also lets increase the spacing between boxes

can we move this to previous page (before shortcuts in prior datasets?)

i dislike overuse of colon/semicolon, please clean this up,

Need setup. We attempted to recover a anonymized version of ICLR '24 by performing manual anonymization on the camera-ready PDF.

so we only source ICLR submissions from OpenReview.

we should add one last sentence stating: Our audit shows that using papers sourced from the stage of review (latest rebuttal), doenst suffer from any of these shortcuts.

this sentence seems a bit listy and not so coherent. State: "We tested out several signals denoted in Table ..." (add a table that adds a deinition to each signal. Markdown/LaTex source is mainly for deepreview, but basically we identify this for looking for \begin{equation} or \cite tags in the text, rather than rendered. author names visible is lookinf for names before abstract, same with affiliation before abstract. has ack is if acknowledgmenets ection header is present. and github in abstract searches fro a github url in the abstract.

i think we should say: "For instance in Deep Paper Gestalt \cite, they used accepted CVPR papers and used workshop papers as a reject pool. However, from formatting alone, if one can tell if something is workshop format (less pages, specific headers, etc) they can easily shortcut the core prediction problem.

what does this mean?

'14/'21

this should be called: "Correcting Model Biases with Prompting" and this should have a short preamble talking about how our base open-source models are heavily accept biased (accepting all), so we tried various prompt corrections and tested the effects over various opensource and a cheap frontier model (Gemini-2.5-Flash) on OpenReview-ICLR. then table F.3, F.4, and F.5 should be weaved in here. i am thinking we have one one paragraph for critical variant, one paragraph for multi-step peer review emulation. the experimental notes should be in the preamble. be results-driven, right now this reads wayy too much as describing setup.

this should be its own sub-section titled: "Frontier Model Reasoning Usage"

we can fully remove this and the prompt search, idt we are mentioning this anymore

most important thing to say, is temperature=0 (greedy evaluation) we dont talk about training here. Context length is fine too

Qwen Base Prompting

merge with above: Frontier Models (Gemini/GPT)

lets reorganize this into a subsection: Base Open-Source Prompting and then we have one paragraph for No Reasoning Prompt. then we have one for Reasoning Prompt. for the reasoning prompt id like you to be a bit more clear by showing the prompt more contiguously rather than split as it is now. then we have a subsection: "Frontier Model Prompting" and one para: "Frontier Reasoning Prompt", then one para: "Frontier Reasoning Agent Workspace. (no need for deepreview).

remove this because we arent mentioning arXiv search for RL anymore

fix the hanging line

this needs to be BEFORE SFT and RL Method Details

ok so we are movign the RL section from the previous to here. i think thats actually all we need, we dont need to put any other details, because they arent mentioned in the main text. lets alos remove the 4-step text+arXiv, lets support the main text only which is that we trained text/vision RL, just uses the single-turn settings.

remove this paragraph header, just do subsection

this should be moved E.2

add citation for rouge-l here actually

you need to qualitatively describe the result. the tldr for both tables should be that frontier mdoels cant adequately memorize or recall decision results, though its hihgest for the earlier years and for arXiv.

delete

also you should denote the metric (Label-Recall (%)) above the years, so the reader knows what you are referring to. also table caption should qualitatiively give us the result.

dont say memory probe: "Frontier memorization evaluation on ...

dont report max, report median

dont inlcude this, its useless, remvoe from table

needs citation, add to bibtex: @inproceedings{lin-2004-rouge, title = "{ROUGE}: A Package for Automatic Evaluation of Summaries", author = "Lin, Chin-Yew", booktitle = "Text Summarization Branches Out", month = jul, year = "2004", address = "Barcelona, Spain", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/W04-1013/", pages = "74--81" }

we should rename the recall rate, i think this seems like it is "correctly recalling", we want a better deinition.

this is confusing, what sentence prefix? like first sentence of the intro? are the tiutle + abstract incliuded?

this is confusing because whats the input? title + abstract?

stop using textttt, use the main text convention of calling this Gemini-3.1-Pro and GPT-5.4

lets 1) move this up before section D Training Variations. 2) i want this to be named: "Do Frontier Models Memorize the Training Data?"

i think we should call this "Ablating Training Distribution and Paper Representation", then we should setup that we tried various techniques on OpenReview-ICLR to improve accuracy. first we started with filtering high-quality accepts/rejects (ingoring borderline), then we looked at various suffix options to help the decision objective.

i think it would be highly prudent to add a new subsection titled "Our Data Pipeline" with two paragraphs for ICLR and arXiv. for arXiv, you can find the pipeline here: /scratch/gpfs/ZHUANGL/sk7524/AutoReviewer/plots/funnel(_openreview).html <- the first contains the pipeline for arXiv, the other for OpenReview. i think its important to maybe have a table that summarizes what we start with, how many sty we can recognize, how many are matched, how many are filtered out, and then the implied reject pool, then the filtereing. for openreview we can talk about start and the filtering? or perhaps a diagram would be nice? you decide, dont just verbatim copy the htmls, they are too dense. look at how we represented information throughout the paper, be wise clear succinct.

can you put subcaptions BELOW the tables please!

all headline PaperLens results use native decisions, unless stated otherwise (like for calibration). you shouldnt mention generated TSV's

i think instead of grouped bar charts, we could instead, on the table show the feature weight for logistic regression (as a column?), then the paragraph just discusses the table. no need for all these grouped bar charts.

just say "Abstract keyword", fix this throughout the table

i dont think we need this detail, stop before colon. also dont use texttt here, no need, just use english for body pages

also feel free to split this into two 5-8 paragraphs, that are full width no hanging lines

i almost feel this should be BEFORE the B.1 section, we reference this first in the related works, so it makes sense to put it first.

i dont think we need to state the numbers, we merely just need to state the effect, 24 has a notable gap gap

can we ensure that this plot maintains the existing pltotting conventions? i think the colors are font, and i mainly want to make sure font sizing on the labels/axes are consistent

delete this, this was legacy. we should state that nips, icml, colm from openreview only release accepts and no rejects, so we ignore it for our openreview set.

good but we are missing '23 notation here we just say 20-23. please fix that in the table

good section, no change needed

this should be Appendix B.2

i think this is part of the previous, it shouldnt be made its own paragraph.

i think we mean the lower-end of rating not just the extremes

i think we measure confidence intervals earlier, so we should efine the acronym there and use it here

the table has messed up its formatting, i think something happened with the numbers?

max context length

export. We

, ensuring uniformity

. For

delete

where accepts are only camera-ready.

delete

delete this, and just say form the last sentence: All datasets are balanced 50/50, and Figure 2 shows examples from both.

remove this

remove this

this hsould just be section 7 like the above reference

something we arent talking about is the fact that we measure gains in rating alignemnt as well. note that after the model's predictions flip from A -> R it starts spitting out weaknesses and novelty critiques at a higher rate, agreeing with human critique.

this sentence is way too long, you need to cut it down. also no one knows what newtowrk ties and halo effects are please fix that.

OR-ICLR

OR-ICLR!

OR-ICLR

hanging line

OR-ICLR

hanging line

please center the numbers in the columns also make delta Acc on a new line, can we decrease the width of this table, so that the figure B can be larger?

OR-ICLR

OR-ICLR

OR-ICLR here too

please read LLamaFactoryAutoReviewer/results/frontier_memory_probe/breakdown.md., please add the results to the appendix section that talks about frotneir models and then reference that here. show a summarized table view here that clearly shows that the frontier models consistently fail at the different recall tests. highest on 24. add a short paragraph here.

this should be OR-ICLR

OR-ICLR

OR-ICLR

lets create a new paragraph header called: "Baseline Correction Measures."

lets name this Baseline Prompting.

use period here

delete colon and replace with period

delete

here i think we can say OR-ICLR

because they are inherently calibrated to the test distribtuion, since our training data has the test years (or somethign like that).

instead of probes lets call these "models". also lets name the paragraph: "Tabular feature analysis."

for the figure 4 feature prob audit, you state full feature definitions are in table c.1, but really it should be appendix c like in the main text. also you were supposed to raise the legend [accept/reject] and change the naems to OR-ICLR and arXiv. please fix (in the right subfigure) you corrected the table

the table should state OR-ICLR and arXiv in italics, and then center the columns for T/V

hanging line

OR-ICLR in the dataset coverage image

datasets

add (OR-ICLR) here

hanging line please fix

In contrast with prior work (cite), reviews, ratings, and meta-reviews are not regressed on; they are only used for measuring rating correlation and review alignment.

add section reference here

i think we want more setup. I think we want to state: "As mentioned in the problem formulation, papers are transformed into reviews before reaching the decision. Given a paper (X), over decisions (D) and reviews (R) ( aka p(D,R|X)). While the flow indicates that reviews precede the decision, now that we have built a shortcut that predicts the decisionwithout the review, we question whether we can generate more \italics{decision-calibrated} reviews. This is valid by simply rearranging the joint distribution like so:

then the formula.

then the both factorizations are vlaid. then we can give a more conceptual understanding: if the conjecture is correct, that much of the decision can be determined by overall presentation quality, then perhaps reviews are simply a means-to-an-end to justify the reviewer's inital recaction.

heres more context about sources that we can cite for this behavior: The strongest terms to search are:

motivated reasoning, post hoc rationalization, biased assimilation, affect heuristic, halo effect, and in peer review specifically prestige bias, reviewer bias, or cognitive bias in peer review.

Here are the most relevant scientific articles.

1. Kunda — “The Case for Motivated Reasoning”

Ziva Kunda, 1990, Psychological Bulletin

This is probably the canonical article. Kunda argues that motivation can bias the cognitive processes people use to access, construct, and evaluate beliefs. In your case: a reviewer’s initial evaluation of a paper can shape which criticisms feel salient or persuasive.

Best use: cite this for the general mechanism: people reason toward a conclusion they are already inclined to reach.

1. Nisbett & Wilson — “Telling More Than We Can Know”

Richard Nisbett & Timothy Wilson, 1977, Psychological Review

This is highly relevant to the “reverse reasoning” part. They reviewed evidence that people often lack direct introspective access to the mental processes that caused their judgments, but still produce explanations for those judgments.

Best use: cite this for the idea that reviewers may sincerely report reasons for a judgment without accurately knowing what actually caused the judgment.

1. Haidt — “The Emotional Dog and Its Rational Tail”

Jonathan Haidt, 2001, Psychological Review

Haidt’s social intuitionist model is about moral judgment, not peer review, but the mechanism maps well: quick intuitive judgment is followed by slower post hoc reasoning. The article explicitly frames reasoning as often coming after intuition rather than causing judgment.

Best use: cite this when you want the phrase “intuition first, reasoning second.”

1. Lord, Ross & Lepper — “Biased Assimilation and Attitude Polarization”

Charles Lord, Lee Ross & Mark Lepper, 1979, Journal of Personality and Social Psychology

This is a classic study on how people evaluate evidence differently depending on their prior beliefs. People tend to accept congenial evidence more readily and scrutinize uncongenial evidence more harshly.

Best use: cite this for the reviewer behavior where evidence supporting the reviewer’s initial take gets treated as decisive, while contrary evidence gets nitpicked.

1. Slovic et al. — “The Affect Heuristic”

Paul Slovic, Melissa Finucane, Ellen Peters & Donald MacGregor, 2007, European Journal of Operational Research

This article argues that feelings of “goodness” or “badness” can rapidly guide judgments and decisions. That is very close to what people informally mean by a “paper gestalt”: an overall positive or negative feeling that then colors assessment of specific features.

Best use: cite this for the gut reaction / affective global impression component.

1. Tomkins, Zhang & Heavlin — “Reviewer Bias in Single- versus Double-Blind Peer Review”

Andrew Tomkins, Min Zhang & William D. Heavlin, 2017, PNAS

This is peer-review-specific. In a large conference-review setting, submissions were reviewed under single-blind and double-blind conditions. The study found that single-blind reviewing advantaged papers by famous authors and high-prestige institutions.

Best use: cite this to show that manuscript judgments are not purely about the paper’s intrinsic content; contextual cues can bias reviews.

1. Blank — “The Effects of Double-Blind versus Single-Blind Reviewing”

Rebecca Blank, 1991, American Economic Review

This was a randomized experiment at the American Economic Review. It found that reviewers were more critical when author identity was blinded, and acceptance rates were lower under double-blind review.

Best use: cite this as older experimental evidence that review outcomes shift when identity/prestige cues are removed.

1. Peters & Ceci — “Peer-Review Practices of Psychological Journals”

Douglas Peters & Stephen Ceci, 1982, Behavioral and Brain Sciences

Classic and brutal study: previously published psychology articles were resubmitted to the same journals with altered author/institution information. Most were not recognized, and many were rejected.

Best use: cite this for the unreliability/context-sensitivity of peer review.

1. Teplitskiy et al. — “The Social Structure of Consensus in Scientific Review”

Misha Teplitskiy et al., 2018

This study analyzed reviews of 7,981 neuroscience manuscripts submitted to PLOS ONE and found that reviewers favored authors closer to them in the co-authorship network. The authors interpret this not just as simple nepotism but as partly reflecting “schools of thought” and substantive evaluative differences.

Best use: cite this if your point is that reviewers’ intellectual/social position can shape what they see as valid or flawed.

1. Sordi et al. — “Halo Effect in Peer Review”

José Osvaldo De Sordi et al., 2020

This one is directly about the halo effect in peer review. It explores whether belonging to the same professional field or group can bias article evaluation during review.

Best use: cite this for the closest peer-review-specific phrase to “paper gestalt”: halo effect in peer review.

A good synthesis sentence would be:

The phenomenon can be described as an interaction of affective first impressions, motivated reasoning, and post hoc rationalization: reviewers may form an early global evaluation of a manuscript and then selectively construct or emphasize criticisms that make that evaluation appear analytically grounded.

For a paper or lit review, I’d probably cite Kunda 1990 + Nisbett & Wilson 1977 + Slovic et al. 2007 for the psychology mechanism, then Tomkins et al. 2017 + Peters & Ceci 1982 + Teplitskiy et al. 2018 for peer-review-specific evidence.

the key is that based off this realization, we sought to see whether conditioning frontier coding agents for reviews can 1) improve their decision-prediction and 2) improve their generated reviews' alignment to human reviews.

add a specific (review prompt in Appendix ...)

add specific apepndix reference here (Appendix ...)

remove

predicted decisions poorly track ground-truth outcomes.

Section

delete this

empirically proven:

i think we should add these back into the plot, but not state their sample contributions.

delete

arXiv-ICLR

just say R_A and R_R

OpenReview-ICLR

OpenReview-ICLR

OpenReview-ICLR -> OpenReview-ICLR

has human ground-truth ratings for each paper.

correct

only drops performance on OpenReview-ICLR by 2%