271 Matching Annotations
  1. Jun 2026
  2. Local file Local file
    Untitled document
    71
    7. skones 03 Jun 2026
      Holm–Bonferroni correc-tion

      we should add a cite here: @article{abdi2010holm, title={Holm’s sequential Bonferroni procedure}, author={Abdi, Herv{\'e}}, journal={Encyclopedia of research design}, volume={1}, number={8}, pages={1--8}, year={2010}, publisher={Thousand Oaks, California} }

    12. skones 03 Jun 2026
      o in Figure 6b the largest arXivAcc gain comes from data scale rather than parameter scale.

      this is insufficient. Lets make one more paragraph that talks about data scale. and how we varied the OpenReview-ICLR by 50% size to 75% size, but for arXiv, since we have the arXiv-M and arXiv sets, we plot a 3.3x multiple. We see that more data helps for both sets. Expand ont his a bit more. and talk about figure 6b

    13. skones 03 Jun 2026
      The 3B text checkpoints show the tradeoff: OpenReview-ICLR-trained text gainsICLR Acc but loses AUC and rating alignment, while the arXiv-medium text model trades a small Acc dropfor slightly stronger ranking.

      lets remove this and the delta AUC and delta rating columns they arent very important here.

    19. skones 03 Jun 2026
      Gemini-3.1-Pro (Vision)

      legend needs to be fixed its box isnt spanning all the items, additionally too much whitespace from legend to plots. also add more hsapce between plots.

    20. skones 03 Jun 2026
      additionally

      I think we want to have one finaly short paragraph that states that for all the reported Baselines, for fairness, we run validation tuning for their decision threshold. Some model's are biased out of box, for example GPT 5.4 is reject-biased, while Gemini is accept-biased. And yet, they still have relatively high AUC. Its possible that the predicted rating is well-ordered, but the mdoel's chosen decision threshold is biased. therefore, for all frontier models (and coding agents), and DeepReview, we calibrate the decision threshold to vlaidation data before reporting test results (ignoring the model's initial decision). This consistently helps baseline numbers, in agreement with the findings of \cite{AI Scientist v1}.

    22. skones 03 Jun 2026
      Table 3. Main results. For each baseline row × test set (frontier APIs, agents, DeepReview, and the Qwen base models),we run 10 random 50-sample tuning trials, average the resulting metrics, and adopt the tuned threshold only when itimproves balanced accuracy by ≥ 2pp; otherwise the native decision is kept. PaperLens SFT and RL rows use theirnative binary decisions throughout. AUC and ρR are threshold-free.

      revert the table caption to what we had before. i just wanted you to change the baseline results! also the shading on PaperLens needs to be reverted to only shade the top results in OpenReview-ICLR and arXiv!

    31. skones 03 Jun 2026
      Ordi-nary reviewing prompts were consistently accept-biased, so the main table reports base-model performanceon the exact answer-only and reasoning prompts used for SFT and RL. Workspace layouts, prompt ablations,and launch settings are in Appendix F.1 and Appendix F.2 for reproducibility and comparison

      lets change this. We want to state that: Vanilla prompts were consistently accept-biased, so it correct we tested using "critical" language; however, while the acceptance rate drew closer to 50%, the model's were still wildly inaccurate at 50% accuracy. Therefore, in our main results, we report the model behavior on the base prompt, the same ones we start with for SFT/RL training. These base prompts, as well as the critical-variants are defined in Appendix....

    36. skones 03 Jun 2026
      anuary 2025-era cutoffs.

      please check: /scratch/gpfs/ZHUANGL/sk7524/LLaMA-Factory-AutoReviewer/results/frontier_memory_probe/breakdown.md

      we actually did some analysis on the behavior of frotnier models and showed that they can't recall even ablated across many options like abstract completion, multiple choice guess whether the paper is accept/reject, remember the paper's venue etc.

    42. skones 03 Jun 2026
      The full feature inventory is listedin Table C.1, and Appendix C gives grouped distribution diagnostics.

      remove the grouped diagnostics, just state at the end of the last setnence that (the full feature inventory is captured in Appendix C)

    50. skones 03 Jun 2026
      ICLR is balanced within OpenReview year

      OpenReview-ICLR starts with 40K ICLR submissions, and subsamples the reject population while maintaining the annual distribution. However for arXiv, submission year is more nebulous because implied rejects have upload dates, while proceedings-matched accepts ahve conference years and postreview uploads. THerefore, we choose not to year balance, and instead subsample 101,659 filtered papers to forma. 83,556 balanced set, maintaining the starting venue distribution. This is further subsampled to achieve a 25K set, equivalent size to OpenReview-ICLR for scale-invariant training comparison.

    52. skones 03 Jun 2026
      OpenReview-ICLR is ICLR submissions fromOpenReview: 25K examples split 85/5/10 across train, validation, and test, with the held-out test focused on’25–’26. arXiv is source-derived proceedings papers, with a held-out test covering ’24–’26. Training uses twopools: a 25K proportional subset (arXiv-m) and the full 83K set (arXiv).

      both datasets are 85/5/10. lets state the following: OpenReview-ICLR consists of 25K ICLR submissions from OpenReview from '20-'26, whose test set spans '25 - '26. arXiv consists of LaTex source-derived proceedings papers, with a held out test-set covering '24-'26 (to include evaluation of conferences that are biannual) like ECCV). For arXiv, training utilizes two pools: a large 83K set (arXiv) and a smaller set proportional to OpenReview-ICLR (arXiv-m). All datasets contain equal numbers of accepts and rejects, and are split 85/5/10 for train,validation, and test sets.

      Because arXiv spans many venues, it also contains an ICLR subset (we call this \italics{arXiv-ICLR}), which is distinct from the OpenReview-sourced OpenReview-ICLR set. Figure 2 shows paired examples from both datasets.

    53. skones 03 Jun 2026
      Because our target metric is Acc, which averages accept andreject recall, a balanced test set directly measures the behavior we care abou

      I think we want to say: "Therefore, we use a balanced test-set, whereby accuracy equally weights accept and reject recall.

    54. skones 03 Jun 2026
      .Year-by-year coverage is documented in Appendix B.1;

      just say (Appendix B.1) after the aftifact itself, dont say this year-by-year coverage is documentd.

    61. skones 03 Jun 2026
      source-derived papers linked to proceedings after processing

      i think we want to say something like: arXiv examples are derived from LaTex source, recompiled with venue-specific anonymization macros. (or something like this much shorter)

    65. skones 03 Jun 2026
      Because public anonymous artifacts are most consistently available after rebutta

      i think we need a reference to the appendix section which talks about this availability over years.

    Annotators