Screaming Frog SEO Spider
A desktop site-auditing tool run by SEO practitioners against specific sites, not a centralized crawler.
Screaming Frog SEO SpiderOfficial documentation: www.screamingfrog.co.uk
See what Screaming Frog SEO Spider sees on your site → try Peekabot, free
What Screaming Frog SEO Spider does
Screaming Frog SEO Spider is desktop software that an individual runs from their own machine to audit a specific website - checking links, metadata, redirects, and technical SEO issues. There is no central bot fleet; each crawl is one person pointing the tool at one site, so traffic arrives from ordinary consumer or office IP addresses. By default it respects robots.txt and crawls plain HTML, but both are user-configurable: the operator can tell it to ignore robots.txt entirely, enable a JavaScript rendering mode backed by headless Chromium, or swap the user-agent to impersonate Googlebot or anything else. That configurability is why both robots compliance and JS rendering are honestly "it depends".
Why it visits your site
Someone - your own SEO team, an agency you hired, or a competitor doing recon - is auditing your site.
The bigger picture: SEO tool crawlers
SEO tool crawlers are commercial data harvesters with a twist: the product they build from your pages - link graphs, keyword data, authority metrics - is one your own marketing team may pay to use. They are consistently among the highest-volume non-search crawlers on the web, and they send you nothing directly. The trade is indirect: tolerate the crawl and the industry's measurement layer, including your own dashboards, stays complete.
These crawlers are generally well-behaved - documented, robots-compliant, honoring crawl-delay - so the decision is genuinely yours to make on resource and privacy grounds. Sites that block them typically do it to deny competitors easy backlink intelligence or to trim server load; sites that allow them are usually buying the completeness of their own tooling.
Should you block it?
If the default user-agent shows up, it is usually your own people or an agency doing legitimate work, so ask around before blocking. A robots.txt rule only deters operators who leave compliance switched on, and anyone determined can change the user-agent anyway. Rate limiting is a better defense than blocking if an audit is hammering your server.
Block via robots.txt
Add these lines to the robots.txt at your site root to ask Screaming Frog SEO Spider to stay away:
User-agent: Screaming Frog SEO Spider Disallow: /
Remember that robots.txt is a request, not a lock - compliance is voluntary, and enforcement happens at the server. Viz's AI Control pairs robots.txt rules with server-level 403 responses and then probes your site to verify the block is actually holding.
How to see Screaming Frog SEO Spider traffic on your site
You have three windows onto it, each with a catch. Raw hosting access logs contain every Screaming Frog SEO Spider request, but many managed WordPress hosts don't expose them, and when they do you are grepping text files by hand. A CDN dashboard sees the traffic at the edge, but it lives outside WordPress and speaks in totals, not in your site's terms. And JavaScript analytics - GA4 and friends - will never show it at all, because Screaming Frog SEO Spider doesn't run scripts.
Viz's answer is the request log: filter to Screaming Frog SEO Spider and read exactly which URLs it fetched and when, right inside wp-admin - then watch the same filter after any blocking decision to see whether the visits stopped, slowed, or kept coming.