Python Requests

What Python Requests does

Requests is the standard HTTP library for Python, and this UA appears whenever a script's author didn't set a custom one. That covers an enormous range: data pipelines, monitoring scripts, academic research, hobbyist scrapers, and vulnerability scanners all look the same. The library has no robots.txt logic of its own and doesn't execute JavaScript.

Why it visits your site

A Python script fetched your page. The UA tells you the tool, not the intent - polite scrapers usually set a descriptive UA, so a bare python-requests string often means a quick script or a lazy one.

The bigger picture: utility & developer tools

This category is the honest miscellany of a server log: monitoring services, platform callbacks, and the default user agents of HTTP libraries. The unifying rule is that the string identifies software, not intent. A curl request might be your host's health check or a vulnerability probe; a python-requests hit might be a researcher or a scraper; a WordPress UA is one site talking to another. Reading these names as a threat list misses what they actually are - a census of the scripted web touching your site, most of it mundane, some of it yours.

robots.txt barely applies here, and not out of rudeness: monitors fetch the exact URL a customer configured, feed fetchers act on a user's subscription, and bare libraries simply ship with no robots logic at all. Blocking a default library UA filters the honest and the lazy while the malicious change one line and continue, so the effective levers are different - rate limiting by IP, watching which paths get requested, and reserving blocks for named services you can verify and have decided you don't want.

Should you block it?

Some sites do block it, reasoning that well-behaved bots identify themselves. Fair enough - but anyone can change the string in one line, so you're filtering only the honest and the careless. If a python-requests client is hammering you, rate limiting by IP works better than a UA rule ever will.

Block via robots.txt

Add these lines to the robots.txt at your site root to ask Python Requests to stay away:

User-agent: python-requests
Disallow: /

Remember that robots.txt is a request, not a lock - compliance is voluntary, and enforcement happens at the server. Viz's AI Control pairs robots.txt rules with server-level 403 responses and then probes your site to verify the block is actually holding.

How to see Python Requests traffic on your site

You have three windows onto it, each with a catch. Raw hosting access logs contain every Python Requests request, but many managed WordPress hosts don't expose them, and when they do you are grepping text files by hand. A CDN dashboard sees the traffic at the edge, but it lives outside WordPress and speaks in totals, not in your site's terms. And JavaScript analytics - GA4 and friends - will never show it at all, because Python Requests doesn't run scripts.

Viz's answer is the request log: filter to Python Requests and read exactly which URLs it fetched and when, right inside wp-admin - then watch the same filter after any blocking decision to see whether the visits stopped, slowed, or kept coming.