Just as a driver's license confirms a driver's identity and competence but doesn't guarantee future responsible behavior, verifying a bot's identity is merely the first step. While knowing the bot operator establishes a basic level of accountability, trust in a bot depends on what it actually does next. Even a verified bot can ignore website preferences, overload servers, or circumvent control mechanisms. This isn't always malicious intent; often, it's a result of misconfiguration or a software bug.
In the context of automated traffic management, we strive to answer two key questions: "Are you who you claim to be?" and "Can we trust you?" For a bot to be considered fully verified within our ecosystem, it must meet three fundamental criteria: accurately and securely identify itself, do only what it declares, and not exhibit abusive behavior. Identity is addressed through a mechanism where bots sign requests, which are then cataloged by an open system. Our focus then shifts to the second, equally important principle: bots must not cause harm.
Therefore, we are implementing a system for continuous evaluation and reporting of verified bot behavior if their actions begin to deviate from the mentioned rules. In this way, we ensure that the answer to the question "Can we trust you?" remains positive. Here, it is important to distinguish between risk and trust. Risk describes the probability that a specific request or action at a given moment could be harmful, and it can change from moment to moment. Trust, conversely, is built over time, based on repeated and consistent behavior.
An example from the past is when a well-known bot was removed from the list of verified bots because it attempted to access resources on websites that explicitly prohibited scraping, and subsequently tried to circumvent control mechanisms when these requests were blocked. This underscores the importance of adhering to a simple requirement: do not exhibit abusive behavior. Although this sounds obvious, in practice, many nuances are hidden within this point.
Planen Sie eine neue Website oder App?
Schreiben Sie mir, was Sie vorhaben — ich schaue es mir gerne an und berate Sie zum besten Vorgehen.
When crawlers do not respect site preferences, they undermine trust. When sites ask a crawler not to crawl or to slow down, and the crawler ignores these requests, trust is damaged. Therefore, we have developed a system that measures how often a crawler respects website preferences. These observations are reflected in an explainable "trust band": a range that shows how consistently a bot adheres to site preferences and what specific behaviors influenced this rating, rather than a permanent and immutable judgment about the bot or its operator. Initial signals we evaluate include whether the bot respects instructions and whether it responded to direct feedback.
We consider these signals to be "common sense" and a demonstration of courtesy towards operators who provide the information bots seek. In the future, we plan to publish these results within monitoring tools and also expand measurement methods. This evaluation gives responsible bot operators the opportunity to demonstrate that trust is maintained over time. Verifying identity is necessary, but it is also essential to continuously verify behavior.
The robots.txt file is the primary way to communicate a website owner's preferences. Our system assesses whether a bot's observed traffic over time aligns with these published preferences. Robots.txt instructs all (or specific) bots to only crawl particular sections of a site. Isolated violations or one-off incidents may not be significant. Instead, we look for repeated patterns of behavior that ignore site preferences. Responsible bots are simply expected to consult and honor the preferences that websites publish, in accordance with relevant standards (e.g., RFC 9309). It should be emphasized that robots.txt establishes minimum rules for what bots and agents must not do. However, this does not mean that everything not explicitly forbidden is allowed. For example, robots.txt might not say that a bot must stop when a website can communicate that further access is undesirable through other signals.
Bots can remain entirely within permitted paths yet significantly burden the websites they visit. One of the clearest ways a website can defend itself is an HTTP 429 ("Too Many Requests") response. HTTP 429 is a directive for a visitor to reduce their request frequency, sometimes with instructions on how long to wait before trying again. Our system considers whether bot traffic adapts after receiving this signal. Does it slow down, pause, or continue sending requests at the same pace? This behavior is then progressively incorporated into the bot's trust band: consistent slowing or pausing increases trust, while repeatedly ignoring the signal decreases it.
Monitoring these signals is only the first step towards ensuring continuous accountability for verified automation. By combining a bot's identity with evidence of its ongoing behavior, we can give site owners greater confidence that the bots they choose to allow access respect their preferences. For bot operators, this provides a way to demonstrate that they earn trust, and for website owners, it offers context about the automated traffic coming to them. This system, together with tools for managing website preferences and bot governance, forms a comprehensive cycle of accountability for automated traffic. It helps site owners publish their preferences, and monitoring bot interaction with these preferences shows whether verified bots consistently adhere to them over time. Bot governance tools then allow website owners to decide how to handle this traffic. Together, these mechanisms ensure that trust is earned through transparent and responsible behavior, not merely assumed based on identity.
We are already using this system internally to evaluate how consistently verified bots adhere to website preferences. Because trust is built over time, we assess persistent patterns of behavior, not isolated events. We are using this initial phase to build history and refine how these observations become useful guidance. Based on this foundation, we plan to make the system available to all users, who will be able to choose a trust threshold to decide how verified bot traffic will be handled. Our goal is straightforward: to keep website preferences meaningful and ensure that trust is earned through sustained behavior.