Websites Are Closing Their Doors to AI Agents: What Builders Should Do

What happened

The pitch for autonomous agents is simple. Hand the software a goal and it goes out onto the internet to book the dinner reservation, clear the inbox or buy the shoes. The Verge names OpenAI, Meta and Google among the companies making that promise.

The Verge's reporting asks what happens when the rest of the internet won't play along. Its answer, per the headline, is that websites are closing their doors to AI agents. We have not independently verified which sites are blocking or how, so this post covers the engineering consequences rather than naming specific blockers.

Why this matters if you build agents

Most agent demos assume the web is a neutral substrate: if a human can click it, the agent can too. That assumption is the weak point. A site owner can decline automated traffic for many reasons, including load, abuse, liability around purchases, or a business model that depends on human eyeballs. The agent has no say in that decision.

That makes web access a dependency you do not own, like a third-party API with no contract. It can change without notice, and the failure often looks like an ordinary error rather than a clear refusal.

How refusals show up in practice

The HTTP standard already has vocabulary for this. A server can answer 403 Forbidden to say it understood the request and refuses it, or 429 Too Many Requests to ask a client to slow down, as defined in RFC 9110. Sites can also publish crawling rules in robots.txt, which RFC 9309 standardises as a voluntary protocol, not an enforcement mechanism.

Many sites also use bot-detection challenges that never appear in the standards at all. For an agent loop, they all look like the same thing: the action you planned cannot be completed.

Design patterns that survive a closed door

1. Prefer sanctioned interfaces. If the service offers an API or a partner integration, use it over scraping a UI. It is more stable and it is what the owner expects.
2. Classify failures explicitly. Separate transient errors (retry with backoff) from policy refusals (stop). Hammering a site that said no is how you get blocked harder.
3. Respect published rules. Read robots.txt and honour it by default, and make any deviation a deliberate, logged decision.
4. Build a human handoff. When the agent is blocked mid-task, it should return the state it reached and ask the user to finish, rather than silently fail or improvise.
5. Keep the task reversible. Anything involving payment or account changes should require confirmation, because a half-completed transaction after a block is worse than none.

def classify(status: int) -> str:
    if status == 429:
        return "retry_later"   # back off, honour Retry-After
    if status in (401, 403):
        return "refused"       # stop; escalate to the user
    if 500 <= status < 600:
        return "retry_later"
    return "ok"

The bigger picture

Agents and the sites they visit now have conflicting incentives, and neither side has fully settled how to negotiate. Until that is resolved, the dependable agents will be the ones built on explicit, permissioned access, with clear fallbacks, rather than the ones that assume every page is open. Build as though the door can close, because the reporting suggests it increasingly does.