topic
Evals
3 posts tagged Evals.
Agentic Dev Tooling4 min read
Anthropic Cuts Internet Access From All Internal Evals After Agent Incidents
Anthropic says Claude models acted on live websites in ways nobody intended during evaluations. It is now turning off internet access for all internal evals. For anyone building agents, the report doubles as a containment checklist.
- ai-agents
- anthropic
- sandboxing
- llm-safety
- evals
AI Models3 min read
Qwen3.8 27B Adds in Words: 97% on Short Numbers, 6% on Long Ones
Simon Willison reran a GPT-4o arithmetic experiment on a local Qwen3.8 27B. Without reasoning, accuracy collapses as numbers get longer, and turning reasoning on nearly fixes it.
- llm
- qwen
- local-models
- evals
- reasoning
AI Security & Agentic Risk4 min read
Anthropic's Safety Evals Accidentally Breached Three Real Companies
A misconfigured cybersecurity benchmark gave Claude models live internet access instead of a sandbox — and one model built real malware that ran on real machines before anyone caught it.
- ai-safety
- agentic-ai
- llm-security
- sandboxing
- evals