Less reletad to the paper random thought: These evaluaiton practices are great but I wonder if what are the most popular ways to evaluate generalization of AI agents. Feels like interval evals (with non-contaminated data and complex problems from diverse domains) of frontier labs is still the standard. But this may become less and less scalable as people put everything into model training data, where data contamination is kind of unavoidable, semantically speaking.

I suspect the underlying reason is largely because those tasks where the agent took longer are simply much harder in terms of cognitive demands. Therefore, I believe increasing the test time inference budget can yield satisfactory performance returns if scaled up cleverly, especially in a CL / RSI loop

What about tasks for which we don't have environmental feedback that serve as guardrails? This is especially the case for non-verifiable tasks.

Is ReAct necessarily the best paradigm? Looks like most agentic work focuses on this. I wonder what are the competitive alternatives out there, and what are the pros and cons.

Can scale to large context size but how to most effectively manage the context such that critical information's loss rate is reduced as much as possible? Ideally, we should build AI agents that can be fine-tuned and select what it should rememeber selectively

How can we design the AI agents such that it can flexibly adapt to new repositories or any virtual information workflow to perform optimally given its available computational resources and inherent capabilities? Feels like self-evolving AI agent (in particular harnesses) is the only way to go as manual evaluation and adaption is not scalable, though occasional human feedback can help.