_Self-reported score with custom Anthropic scaffold._ SWEPro were evaluated with the mini-swe-agent scaffold. However, we use the scores reported by Anthropic for Opus with the max thinking efforts due to frequent timeouts during our evaluation trials.
脚注2揭示了重要数据点：Opus 4.6的53.4分是Anthropic的自报分数，因为作者在评估过程中频繁遇到超时问题，无法自行验证。这表明性能比较中存在数据可靠性问题，特别是对于Opus的评估依赖于厂商自报数据，可能存在偏差。
Argument quality and fallacies. (n.d.). HackMD. Retrieved January 17, 2022, from https://hackmd.io/@scibehC19vax/argumentquality
Scientists try to pinpoint why rapid Covid tests are missing some cases. (2022, January 6). STAT. https://www.statnews.com/2022/01/06/scientists-try-to-pinpoint-why-rapid-covid-tests-are-missing-cases/
Taylor, L. (2021). The Venezuelan health-care workers secretly collecting COVID stats. Nature, 597(7874), 20–21. https://doi.org/10.1038/d41586-021-02276-1
I haven't met anyone who makes this argument who then says that a one stop convenient, reliable, private and secure online learning environment can’t be achieved using common every day online systems
Reliable: As a simple example, I'd trust Google to maintain data reliability over my institutional IT support.
And you'd also need to make the argument for why learning needs to be "private", etc.
Speelman, C., & McGann, M. (2020). Statements about the Pervasiveness of Behaviour Require Data about the Pervasiveness of Behaviour [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/bxzm4
Parsons, Sam. ‘Reliability Multiverse’, 26 June 2020. https://doi.org/10.31234/osf.io/y6tcz.
DaveLagnado on Reddit
Fenton, N., Hitman, G. A., Neil, M., Osman, M., & McLachlan, S. (2020). Causal explanations, error rates, and human judgment biases missing from the COVID-19 narrative and statistics [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/p39a4
Edelsbrunner, P. A., & Thurn, C. (2020, April 22). Improving the Utility of Non-Significant Results for Educational Research. https://doi.org/10.31234/osf.io/j93a2