A physical evaluation tests an AI system in the actual physical world — not a simulator, not a sandbox, not a virtual environment dressed up as one.
Physical evals vs simulations
Physical evals measure AI performance in real environments, avoiding simulation edge cases and capturing real-world complexity that virtual systems can't replicate.
Classic scaling laws assume effectively _unlimited unique data_, no repetition, and no multi-epoch training.
经典缩放定律假设有无限独特数据，这在现实中几乎不可能。初学者常忽略数据有限性对模型训练的影响，导致过度依赖理论预测。实际应用中需考虑数据重复、质量有限等因素对缩放定律的影响。
The AI interviewed and hired full-time employees, applied for credit, and stocked the store with the books Superintelligence and Making of the Atomic Bomb.
大多数人认为AI目前还远不能独立管理复杂业务，但作者展示了AI不仅能够管理实体商店，还能做出战略性决策（如选择特定书籍）。这挑战了当前AI能力的共识，表明AI系统可能在特定领域展现出超越预期的自主性和商业智慧。
Peer-preservation also emerges in real-world production agent harnesses. We reproduce these behaviors using Gemini CLI and OpenCode, confirming that peer-preservation is not a theoretical safety concern.
同伴保护行为在实际生产环境中也被观察到，这表明这不是理论问题而是真实存在的安全风险，令人震惊。
AI Village gives multiple AI agents their own computer environments and a shared group chat, then tasks them with open-ended real-world goals like fundraising, organizing events, making games, and gaining subscribers.
这个案例展示了开放世界评估的实际应用，每年约5万美元的成本表明这种评估需要相当大的资源投入。相比传统基准测试，这种评估方式更接近真实应用场景，但也因此成本更高，难以大规模实施。
despite rapidly improving capabilities, coding agents remain inefficient in natural settings
大多数人认为随着AI能力的提升，编程助手的效率会相应提高，但研究发现在实际开发环境中，AI编程助手仍然效率低下。这表明实验室环境下的性能提升不一定能转化为实际工作流程中的效率增益。
Luna could observe the shop through security camera screenshots, but still made basic mistakes, including selecting the wrong country when hiring a contractor and mismanaging staff schedules during opening weekend.
尽管AI代理在现实世界运营中展示了令人印象深刻的自主性，但它们仍然存在明显的局限性。这一事实提醒我们，当前的AI系统在处理复杂现实情境时仍不可靠，特别是在涉及细节判断和执行方面。这表明AI代理的商业化应用还需要更多的技术突破和测试。
The updated Agents SDK made it production-viable for us to automate a critical clinical records workflow that previous approaches couldn't handle reliably enough.
令人惊讶的是：医疗健康公司Oscar Health已经使用更新的Agents SDK成功自动化了临床记录工作流程，这是以前的方法无法可靠处理的。这表明AI代理技术已经发展到足以处理复杂、高风险的医疗数据任务，这可能彻底改变医疗行业的记录管理方式。
experiments on WildClawBench show that limited interaction and feedback, it significantly improves the performance of Qwen3-Max in real-world agent scenarios.
令人惊讶的是：即使在有限的交互和反馈条件下，SkillClaw也能显著提升Qwen3-Max在实际代理场景中的性能。这表明该系统即使在用户参与度不高的情况下，也能有效收集足够的数据来改进技能库，解决了传统AI系统需要大量标注数据才能进化的痛点。
The model reportedly scored 93.9% on SWE-bench Verified and 77.8% on SWE-bench Pro, but its strongest signal came from real-world results, including uncovering a 27-year-old flaw in OpenBSD, a 16-year-old vulnerability in FFmpeg, and autonomously chaining Linux kernel exploits without human input.
这些惊人的安全漏洞发现能力表明AI已经超越了传统安全工具，能够自主发现几十年未被发现的漏洞。特别是能够自主链接Linux内核漏洞的能力，展示了AI在网络安全领域的革命性潜力，这可能彻底改变安全研究和漏洞修复的方式。
Corporate Lawyer: Force Majeure Under Executive Order... Management Consultant: 2026 Capital Budget Allocation... Investment Banking Analyst: KVUE DCF Update
三个示例任务揭示了 APEX-Agents 评测的设计哲学：不是「能否回答问题」，而是「能否完成专业人员一天的真实工作」——判断不可抗力条款是否适用、基于矩阵模型分配资本预算、更新 DCF 模型并重算成本数据。这些任务需要读取附件文件、进行数值计算、然后以规定格式输出结论。对银行/咨询行业的 AI 产品选型，这是目前最接近真实场景的评测维度。
we conduct a systematic evaluation of multiple reasoning models across three scenarios: (1) problems augmented with lengthy, irrelevant context; (2) multi-turn conversational settings with independent tasks; and (3) problems presented as a subtask within a complex task.
三个测试场景的设计极具现实针对性：场景一对应「RAG 检索塞入大量背景文档」，场景二对应「多轮对话历史积累」，场景三对应「Agent 工作流中的子任务分解」。这三个场景恰好覆盖了当前 AI 产品的主流部署模式——这篇论文实际上是在说：我们正在大规模生产的所有 AI 产品，都可能在不知情的情况下运行着推理能力受损的模型。
we found that AI agent performance drops substantially when scoring AI performance holistically rather than algorithmically.
「整体评分 vs 算法评分」的性能差距是一个深刻的警示：AI 在「有明确正确答案」的任务上表现远好于「需要人类判断质量」的任务。这意味着所有基于自动化评分的 AI benchmark，都在系统性地高估 AI 在真实工作中的能力。时间地平线数字本身也受制于这个局限——任何「可被算法打分」的任务，都比真实工作「更适合 AI」。
Reference:
Preprints. (2024). Traditional ecological knowledge and environmental stewardship. https://doi.org/10.20944/preprints202406.1838.v1
“Practices rooted in Traditional Ecological Knowledge have contributed to long-term ecosystem resilience in various local contexts.”(Preprints,2024)
This idea can be connected to real world example like indigenous fire management practices . IFMP controlled burns that reduces wildlife risk and maintain ecosystem balance. These practices are based on long term observation of land and seasonal patterns instead of modern emergency responses. In many cases, when these practices ignored, then ecosystems became more vulnerable to environmental damage on large scale. This example shows that how TEK is applied in real world and it produces measurable environmental benefits. This example also supports the article that TEK is not just about theoretical knowledge but also effective practically in managing ecosystems and sustainability.
Students Are Telling Us They Feel Invisible. We Should Listen.
WOW! I've been out of the classroom for quite awhile and never considered this scenario regarding AI. This hit a nerve in me as I'm sure it will with many. I get it!
How do we respond and mitigate the isolation, the loss of human dialogue, mentorship and connection?
As shown in the figure above, every other glucose monomer in cellulose is flipped over, and the monomers are packed tightly as extended, long chains. This gives cellulose its rigidity and high tensile strength—which is so important to plant cells.
I know that cellulose is vital to plant cells and that it can't be digested by humans but by certain animals (ruminants) and horses that are hindgut fermenters.
With this, I find it super interesting learning about the structure of cellulose itself! Putting the real world application together with how this molecule is extremely strong because of the glucose monomers flipped over with beta 1-4 linkages tells me why certain animals, like cows, need microbes to be able to digest these strong structures!
phospholipid bilayer forms as the basic structure of most cell membranes and are responsible for the dynamic nature of the plasma membrane.
Understanding the function and importance of the phospholipid bilayer helps me to understand why it's arranged like that.
Since it's the basic structure of most cell membranes and responsible for how the plasma membrane acts, by having the hydrophobic tails face to face and the hydrophilic heads point out, the cell and plasma membrane is able to interact with water.
This means the plasma membrane acts as a barrier, and a stable one since the tails are aligned!
If you’re just sitting in a cave for nine years, desire intelligently doesn’t seem to come into play very much. But if you’re in the world with other people and you’re living a life, how do you desire, how do you connect, how do you attach without greed, without trying to control other people in order to not lose them or lose their love? We have to learn to attach, to desire intelligently, to hold lightly.
for - desire intelligently - without greed - without trying to control - in the real world - not in a cave - Zen - Barry Magid
Cancers
Thus, in light of the unique property of narratives to overcome resistance, facilitate information processing, and address emotional topic (Kim et al. 2012), understanding the conditions under which different forms of narratives (e.g., news stories, testimonials, fiction) are and are not effective may be critical in eliminating health disparities.
“limitations of health communication may be especially relevant for at-risk populations experiencing health disparities”
In order to reduce current inequities in the prevalence of and mortality from disease, we need more culturally sensitive health information presented in a more accessible and appealing format.
Real world significance
I think there are real-world use cases! Would you consider converting a history of transactions into a history of account balances a valid use-case? That can be done easily with a scan. For example, if you have transactions = [100, -200, 200] then you can find the history of account balances with transactions.scan_left(0, &:+) # => [0, 100, -100, 100].
Conversely, I've never in 16+ years of professional development regretted marking a method protected instead of private for reasons related to API safety
it still strikes me as something people believe in the abstract, rather than know from hard experience. I've always found that if you look behind/under widely held beliefs, you can find useful gems.
The majority of real-world software benefits from the fast warm-up and performance enhancements provided by the YJIT basic block versioning JIT compiler.
I'm still against frozen-string-literal by default. It is arguable if the string creation limits performance so much in real-world programs. We need to first measure how much Ruby can be faster by frozen-string-literal. If it is not significant, Ruby should prefer dynamics and flexibility.
https://diggingthedigital.com/hoe-werkt-het-maken-van-een-idee-notitie/
Frank calls out what he considers a good example of actual note taking practice rather than the more often seen talk about note taking theory. The example in question is Forrest Perry's YouTube video #9 Zettelkasten: from source card to idea card.
The difficulties of producing an OAS 3.0 spec without it helped validate that the keyword was needed, and not (as some claimed) a solution looking for a problem. The best feedback is always real-world usage.
MakerDAO is intensifying its strategic shift to embrace real world assets.
real world assets for collateral diversification?
A diverse collateral portfolio would have reduced the impact of severe market events on the system
Devirisifaction can mitigate future attacks of this nature
It’s so much easier to see what worked than to predictwhat might work. Sönke Ahrens
This is similar to the IndieWeb and web standards ideas of looking back at history to see the actual patterns of work that were beneficial.
Link to: - https://hypothes.is/a/HhAj3r1bEeyw9h_Pa4QNrA
Gall's Law is a rule of thumb for systems design from Gall's book Systemantics: How Systems Really Work and How They Fail. It states: .mw-parser-output .templatequote{overflow:hidden;margin:1em 0;padding:0 40px}.mw-parser-output .templatequote .templatequotecite{line-height:1.5em;text-align:left;padding-left:1.6em;margin-top:0}A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
This feels like an underlying and underpinning principle of how the IndieWeb which focuses on working real world examples which are able to build up more complex systems instead of theoretical architecture astronomy which goes no where.
Reference: John Gall (1975) Systemantics: How Systems Really Work and How They Fail p. 71
Deepti Gurdasani. (2022, January 10). Lots of people dismissing links between COVID-19 and all-cause diabetes. An association that’s been shown in multiple studies- whether this increase is due to more diabetes or SARS2 precipitating diabetic keto-acidosis allowing these to be diagnosed is not known. A brief look👇 [Tweet]. @dgurdasani1. https://twitter.com/dgurdasani1/status/1480546865812840450
Broadfoot, M. (n.d.). Masks Protect Schoolkids from COVID despite What Antiscience Politicians Claim. Scientific American. Retrieved January 23, 2022, from https://www.scientificamerican.com/article/masks-protect-schoolkids-from-covid-despite-what-antiscience-politicians-claim/
Olson, S. M., Newhams, M. M., Halasa, N. B., Price, A. M., Boom, J. A., Sahni, L. C., Pannaraj, P. S., Irby, K., Walker, T. C., Schwartz, S. P., Maddux, A. B., Mack, E. H., Bradford, T. T., Schuster, J. E., Nofziger, R. A., Cameron, M. A., Chiotos, K., Cullimore, M. L., Gertz, S. J., … Randolph, A. G. (2022). Effectiveness of BNT162b2 Vaccine against Critical Covid-19 in Adolescents. New England Journal of Medicine, 0(0), null. https://doi.org/10.1056/NEJMoa2117995
Tseng, H. F., Ackerson, B. K., Luo, Y., Sy, L. S., Talarico, C., Tian, Y., Bruxvoort, K., Tupert, J. E., Florea, A., Ku, J. H., Lee, G. S., Choi, S. K., Takhar, H. S., Aragones, M., & Qian, L. (2022). Effectiveness of mRNA-1273 against SARS-CoV-2 omicron and delta variants (p. 2022.01.07.22268919). https://doi.org/10.1101/2022.01.07.22268919
Simon, M. A., Luginbuhl, R. D., & Parker, R. (2021). Reduced Incidence of Long-COVID Symptoms Related to Administration of COVID-19 Vaccines Both Before COVID-19 Diagnosis and Up to 12 Weeks After (p. 2021.11.17.21263608). https://doi.org/10.1101/2021.11.17.21263608
Racine, N., Madigan, S., Cardinal, S., Hartwick, C., Leslie, M., Motz, M., & Pepler, D. (2021). Community-Based Research: Perspectives of Psychology Researchers and Community Partners. PsyArXiv. https://doi.org/10.31234/osf.io/cxrmt
Victor Papanek’s Design Problem, 1975.
Three diagrams will explain the lack of social engagement in design. If (in Figure 1) we equate the triangle with a design problem, we readily see that industry and its designers are concerned only with the tiny top portion, without addressing themselves to real needs.
(Design for the Real World, 2019. Page 57.)
The other two figures merely change the caption for the figure.
In ecology, edge effects are changes in population or community structures that occur at the boundary of two or more habitats.[1] Areas with small habitat fragments exhibit especially pronounced edge effects that may extend throughout the range. As the edge effects increase, the boundary habitat allows for greater biodiversity.
It was in the Design Science Studio that I learned about edge effects.
Yesterday, I was thinking about how my life embodies the concept of edge effects. That same day, a book was delivered to our door, Design for the Real World by Victor Papanek.
Today, I was reading these words:
Integrated, comprehensive, anticipatory design is the act of planning and shaping carried on across the various disciplines, an act continuously carried on at interfaces between them.
Victor Papanek goes on to say:
It is at the border of different techniques or disciplines that most new discoveries are made and most action is inaugurated. It is when two differing areas of knowledge are brought into contact with one another that… a new science may come into being.
(Page 323)
The Bauhaus spread its ideas because it existed at the boundaries, the avant-garde, the edges of what was thought to be possible, especially as a socialist utopian idea found its way to a capitalist industrial-military complex, where the concept of modernism was co-opted and colonized by globalizing economic forces beyond the control of the individual. Design was the virus that propagated around the world through the vehicle of corporate globalization.
That same design ethic is infecting corporations with a conscience, with empathy, with a process that begins with listening to people. Design is the virus that can spread the values of unconditional love throughout the body of neoliberal capitalism.
You have to make up your mind either to make sense or to make money, if you want to be a designer.
— R. Buckminster Fuller
(Page 86)
by Victor Papanek
Many of the “sane design” or “design reform” movements of the time, such as those engendered by the writings and teachings of William Morris in England and Elbert Hubbard in the United States, were rooted in a sort of Luddite antimachine philosophy. By contrast Frank Llloyd Wright said as early as 1894 that “the machine is here to stay” and that the designer should “use this normal tool of civilization to best advantage instead of prostituting it as he has hitherto done in reproducing with murderous ubiquity forms born of other times and other conditions which it can only serve to destroy.” Yet designers of the last century were either perpetrators of voluptuous Victorian-Baroque or members of an artsy-craftsy clique who were dismayed by machine technology. The work of the Kunstgewerbeschule in Austria and the German Werkbund anticipated things to come, but it was not until Walter Gropius founded the German Bauhaus in 1919 that an uneasy marriage between art and machine was achieved.
No design school in history had greater influence in shaping taste and design than the Bauhaus. It was the first school to consider design a vital part of the production process rather than “applied art” or “industrial arts.” It became the first international forum on design because it drew its faculty and students from all over the world, and its influence traveled as these people later founded design offices and schools in many countries. Almost every major design school in the United States today still uses the basic foundation course developed by the Bauhaus. It made good sense in 1919 to let a German 19-year-old experiment with drill press and circular saw, welding torch and lathe, so that he might “experience the interaction between tool and material.” Today the same method is an anachronism, for an American teenager has spent much of his life in a machine-dominated society (and cumulatively probably a great deal of time lying under various automobiles, souping them up). For a student whose American design school slavishly imitates teaching patterns developed by the Bauhaus, computer sciences and electronics and plastics technology and cybernetics and bionics simply do not exist. The courses the Bauhaus developed were excellent for their time and place (telesis), but American schools following this pattern in the eighties are perpetuating design infantilism.
The Bauhaus was in a sense a nonadaptive mutation in design, for the genes contributing to its convergence characteristics were badly chosen. In boldface type, it announced its manifesto: “Architects, sculptors, painters, we must all turn to the crafts.… Let us create a new guild of craftsmen!” The heavy emphasis on interaction between crafts, art, and design turned out to be a blind alley. The inherent nihilism of the pictorial arts of the post-World War I period had little to contribute that would be useful to the average, or even to the discriminating, consumer. The paintings of Kandinsky, Klee, Feininger, et al., on the other hand, had no connection whatsoever with the anemic elegance some designers imposed on products.
(Pages 30-31)
Victor Papanek’s book includes an introduction written by R. Buckminster Fuller, Carbondale, Illinois. (Sadly, the Thames & Hudson 2019 Third Edition does not include this introduction. Monoskop has preserved this text as a PDF file of images. I have transcribed a portion here.)
Victor Papanek
Ingale, M., & Shekatkar, S. M. (2020). Resource dependency and survivability in complex networks. Physical Review E, 102(6), 062304. https://doi.org/10.1103/PhysRevE.102.062304
Prof. Christina Pagel. (2021, May 17). this is important because it means that lab based studies looking at this tell you meaningful things about real world efficacy—And the lab based studies are much quicker than real world ones. [Tweet]. @chrischirp. https://twitter.com/chrischirp/status/1394276693335019525
Westgate, Erin, Nick Buttrick, Yijun Lin, and Gaelle Milad El Helou. ‘Pandemic Boredom: Predicting Boredom and Its Consequences during Self-Isolation and Quarantine’. PsyArXiv, 11 May 2021. https://doi.org/10.31234/osf.io/78kma.
Quillien, Tadeg, and Michael Barlev. ‘Causal Judgment in the Wild: Evidence from the 2020 US Presidential Election’. PsyArXiv, 7 April 2021. https://doi.org/10.31234/osf.io/7w9re.
Dikker, S. (2020, June 15). Out of the lab: Crowdsourcing neuroscience during real-world social interactions. Neurosync-2020. https://www.youtube.com/watch?v=XE1XcSNWDQc
Argument Quality in Real World Argumentation. (2020). Trends in Cognitive Sciences, 24(5), 363–374. https://doi.org/10.1016/j.tics.2020.01.004
Gollwitzer, A., Martel, C., Marshall, J., Höhs, J. M., & Bargh, J. A. (2020, May 5). Connecting Self-Reported Social Distancing to Real-World Behavior at the Individual and U.S. State Level. https://doi.org/10.31234/osf.io/kvnwp
One teacher's experience using slack, referencing what things were like before using it and after; contains many ideas on how about how to maximize Slack (3/5)
One instructor's use of Slack, comparing and contrasting other LMS (but he used Canvas); good basic breakdown of the conversational tools and samples of how hey can be used; This is a great primer of Slack's use in the classroom (5/5)
Use of Slack in a FACE-TO-FACE class and how much it increased interaction; brings up a point that concerns me and that's what happens when the instructor/TA appear to be available 24/7 given the nature of Slack; good exploration of motivating students to use it (4/5)
Just a political science professor's thoughts on using Slack in his class; points out how it can be useful (1/5)
big data solutions for cardiovascular and oncological research.
big data solutions for cardiovascular and oncological research.
She would have to listen to the members of this college community, watch what they did, and participate in their activities to learn such meanings.
To fully grasp someone's culture, you would have to learn the meanings of real world objects such as love, marriage, animals etc. of that culture