SilverWarden crawler This page is the address our User-Agent declares.
What it is. SilverWarden is a monitoring crawler run by Silver Data Lab on behalf of the owners of the websites it visits. It reads pages the way a search engine or an AI assistant would, to check technical SEO, structured data and answer-readiness, and reports the findings to the site owner in a private control panel. It does not collect personal data, does not fill in forms, and does not index or republish content.
How it identifies itself. Every request carries this User-Agent:
Mozilla/5.0 (compatible; SilverWarden/1.0; +https://silverwarden.com/bot; info@silverwarden.com)Some checks compare the page served to our declared identity with the page served to well-known crawlers (Googlebot, GPTBot and similar), because a site that serves an empty shell to AI crawlers is a finding. Those requests are made only on behalf of the site owner and stay within the same politeness limits.
How it behaves
- robots.txt is binding. The crawler reads
/robots.txtbefore anything else and never requests a URL it disallows. It reads the group forSilverWardenfirst and falls back to*. - Crawl-delay is honoured. If your robots.txt sets a
Crawl-delayhigher than our own pause, yours wins. - One connection at a time per host by default, with a pause between requests. A few large properties are crawled with two connections, never more than eight, and always from the same declared identity.
- A block is respected, not worked around. On a 403, a 429 or a challenge page the crawler stops for that host and reports it. It does not switch identity, does not use proxies and does not retry in a loop.
- Bounded reads. A page body is read up to a few megabytes; the rest is not downloaded. The nightly visit is capped in pages and rotates across the site, so the same pages are not re-read every night.
- Only the site it was asked to monitor. Links that lead outside the monitored domain are recorded as destinations, not crawled.
How to limit or block it
Use the SilverWarden token in your robots.txt. To slow it down:
User-agent: SilverWarden
Crawl-delay: 10To keep it out of a section:
User-agent: SilverWarden
Disallow: /private/To block it entirely:
User-agent: SilverWarden
Disallow: /Changes to robots.txt take effect on the next visit: the file is read once per host at the start of each run.
Contact
Questions, or a visit you did not expect: write to info@silverwarden.com with the hostname and, if you have it, a line from your access log. We answer every message.