Bot directory · Utility & developer tool

WordPress

WordPress itself making HTTP requests - pingbacks, cron, and loopbacks from millions of independently run sites.

Operator
CategoryUtility & developer tool
Respects robots.txtNo - documented as not applying
Executes JavaScriptNo - reads raw HTML
robots.txt tokenWordPress
User-agent stringWordPress/6.8; https://example.com

Official documentation: developer.wordpress.org

See what WordPress sees on your site → try Peekabot, free

What WordPress does

This UA is WordPress core's built-in HTTP client, and it helpfully includes the URL of the site that made the request. From your own server it's cron and REST loopbacks; from elsewhere it's another WordPress site sending a pingback because someone linked to you, or fetching your page to verify a link. These are application-to-application requests, so robots.txt is never read.

Why it visits your site

Usually a pingback: another WordPress site linked to your post and is notifying you. If the URL in the UA is your own domain, it's just your site talking to itself.

The bigger picture: utility & developer tools

This category is the honest miscellany of a server log: monitoring services, platform callbacks, and the default user agents of HTTP libraries. The unifying rule is that the string identifies software, not intent. A curl request might be your host's health check or a vulnerability probe; a python-requests hit might be a researcher or a scraper; a WordPress UA is one site talking to another. Reading these names as a threat list misses what they actually are - a census of the scripted web touching your site, most of it mundane, some of it yours.

robots.txt barely applies here, and not out of rudeness: monitors fetch the exact URL a customer configured, feed fetchers act on a user's subscription, and bare libraries simply ship with no robots logic at all. Blocking a default library UA filters the honest and the lazy while the malicious change one line and continue, so the effective levers are different - rate limiting by IP, watching which paths get requested, and reserving blocks for named services you can verify and have decided you don't want.

Should you block it?

Don't block the UA outright - you'd break your own loopback requests. The real nuisance is pingback spam and the occasional xmlrpc.php abuse wave, which fakes this UA anyway; disable pingbacks in Settings or restrict XML-RPC to address that directly. Check the URL in the string before judging any individual request.

Block via robots.txt

Add these lines to the robots.txt at your site root to ask WordPress to stay away:

User-agent: WordPress
Disallow: /

Remember that robots.txt is a request, not a lock - compliance is voluntary, and enforcement happens at the server. Viz's AI Control pairs robots.txt rules with server-level 403 responses and then probes your site to verify the block is actually holding.

How to see WordPress traffic on your site

You have three windows onto it, each with a catch. Raw hosting access logs contain every WordPress request, but many managed WordPress hosts don't expose them, and when they do you are grepping text files by hand. A CDN dashboard sees the traffic at the edge, but it lives outside WordPress and speaks in totals, not in your site's terms. And JavaScript analytics - GA4 and friends - will never show it at all, because WordPress doesn't run scripts.

Viz's answer is the request log: filter to WordPress and read exactly which URLs it fetched and when, right inside wp-admin - then watch the same filter after any blocking decision to see whether the visits stopped, slowed, or kept coming.

Measure first

Is WordPress on your site right now? Find out in two minutes.