Who's Visiting the Web? A User-Agent Analysis of 1.3 Million Requests — and Why Only a Quarter Are Human

Update — 2026-06-29: Refreshed against LLMSE's current traffic log of 1,317,348 classified requests across 14,828 unique user agents (up from 387,960 requests and 8,974 agents at first publication). The category mix inverted: HTTP libraries (32.7%) and crawlers (32.6%) now tie for the lead, where desktop browsers used to sit at 49.0%; desktop browsers fell to 21.9%, and the AI-crawler share roughly halved from 15.0% to 7.3% even though its raw volume grew. Googlebot's share of traditional crawlers collapsed from 46.6% to 16.0% as automation and social-monitoring bots flooded in. Every number below was recomputed from live data, and four charts were added. The core thesis from the original — most web traffic is not human — not only holds but strengthens: human browsers are now barely a quarter of requests.

Every HTTP request carries a user agent string — a self-declared identity that tells the server what is asking. Most analyses of this data chase one number: browser market share. That framing assumes the interesting actors are people choosing a browser. On the modern web, they are not.

The conventional mental model — a request equals a user — was already shaky when we first published this analysis on 388,000 requests. Refreshed against more than 1.3 million, it has collapsed entirely. The largest single source of requests is no longer any browser. It is HTTP libraries: curl, Go's standard client, Python's request stack, and the automation built on top of them. Crawlers are tied with them. Human beings clicking through pages in a browser are a minority that keeps shrinking.

We classified 1,317,348 requests across 14,828 unique user agent strings into eleven categories, then broke the data down by browser family, operating system, and crawler operator. Classification comes from a deterministic parsing pipeline (the user-agents library plus a category map), not a hand-labeled sample, and every figure here is a sum of per-agent hit counts pulled directly from the live traffic index.

The headline finding is sharper than it was: only 24.8% of requests come from human browsers. The other 75.2% is HTTP libraries, search and SEO crawlers, AI training bots, and a long tail of agents that report no recognizable identity at all.

The Data

Requests are counted by summing the hit count recorded for each unique user-agent string, then grouped by the category our parser assigns. The table reports both views — request volume and the number of distinct agents behind it — because they tell opposite stories.

The two columns disagree about who runs the web, and the disagreement is the story. By distinct agents, browsers dominate: 4,730 desktop and 2,843 mobile strings, plus 6,534 in the unidentifiable "Unknown" bucket. But by request volume, just 83 HTTP-library agents generate more traffic than 4,730 desktop browsers. A handful of automated clients, each hammering the server far harder than any person browses, outweigh thousands of individual humans. This is why request-weighted and agent-weighted analyses of the same log reach different conclusions — and why "browser market share" is the wrong lens for understanding load.

This traffic reaches LLMSE both organically and through programmatic API submissions, so the automation share here is higher than a typical content site would see. That caveat does not soften the thesis; it sharpens the mechanism. The web is increasingly a place where machines talk to machines, and a developer-facing endpoint simply sees that future earlier than a recipe blog does.

Methodology

This post makes quantitative claims about traffic composition, so the definitions and limits matter.

• Categories. Each unique user-agent string is parsed once and assigned to one of eleven categories. Browser (Desktop/Mobile/Tablet) are human-driven browsers by device class. HTTP Library covers programmatic clients (curl, Go-http-client, Python aiohttp/requests, axios, okhttp). Crawler / Bot covers declared search, SEO, monitoring, and social-listening crawlers. AI Crawler covers bots that crawl for LLM training or AI-answer retrieval (GPTBot, ClaudeBot, Bytespider, Amazonbot, CCBot, PerplexityBot). Unknown is any string the parser cannot match to a known agent. Social Media, Email Client, Media Player, and Feed Reader are the small remainder.
• "Human" defined. Throughout, "human" means the three browser device classes summed (Desktop + Mobile + Tablet) = 326,954 requests, 24.8%. This is a generous ceiling: a parser-level is_bot flag marks 37.8% of requests as declared bots, and HTTP libraries (another 32.7%) are not human either, so by elimination the non-human floor is firm. Two non-human agents — Applebot (a verified Apple crawler) and HeadlessChrome (browser automation) — are filed under the desktop-browser family by the parser, so the true human share is slightly below 24.8%.
• Counting basis. Request volume is the sum of per-agent hit counts stored in the traffic index. Category, browser-family, and OS-family breakdowns are computed by aggregating those hit counts over the per-agent metadata records — a reproducible sum, not a sample.
• Known limits. The dataset is a snapshot of traffic to one endpoint, not a representative crawl of the open web; absolute shares reflect this audience. Request-weighting means a few very active clients move the totals far more than thousands of occasional visitors — we report unique-agent counts alongside every request share to keep that visible. User-agent strings are self-declared and trivially spoofable, so any single classification can be wrong even when the aggregate is sound. Russian-language and Russia-origin agents are excluded from this analysis.
• Why these numbers differ from the original. The original covered 387,960 requests; this refresh covers 1,317,348 — a 3.4x larger log. The growth was not uniform. It was dominated by two clients that were minor or absent before: Go's standard HTTP client (now 30% of all traffic on its own) and a single brand-monitoring crawler. Because request share is volume-weighted, these heavy clients inverted the category ranking — desktop browsers did not collapse in absolute terms (190,145 → 288,059 requests, up 52%); they were simply outgrown by automation. Every share below is recomputed; no figure is carried over from the 2026-03 original.

The Traffic Mix: Automation Takes the Top Two Slots

The category distribution is the inversion in one picture.

HTTP libraries (32.7%) and traditional crawlers (32.6%) are now statistically tied for the lead, and together they are 65.4% of all traffic before a single AI bot or browser is counted. In the original, HTTP libraries were a 3.5% footnote and desktop browsers led at 49.0%. The reordering is not a story about browsers declining — it is a story about programmatic and crawler traffic growing far faster than human browsing, which is exactly the macro pattern the industry has been documenting. Cloudflare reports that global "user action" automated crawling jumped fifteenfold in 2025 while overall internet traffic grew 19%; automation is outpacing humans by an order of magnitude.

Collapsing the eleven categories into human versus machine makes the thesis unambiguous.

At 24.8% human, this traffic is more automated than the web at large — and the web at large already tipped past half. Imperva's 2025 Bad Bot Report (published by Thales) found that automated traffic surpassed human activity for the first time in a decade, reaching 51% of all web traffic, with bad bots alone at 37%. Our 75.2% non-human figure sits well above that global benchmark, consistent with a developer-and-API audience that draws disproportionate automation. If your analytics show mostly humans, that is because tools like Google Analytics 4 filter known bots by default — not because the bots are not there.

HTTP Libraries: The Automation Layer Took the Lead

The category that was a footnote is now the leader, and one client did most of the work.

Go's standard HTTP client is 91.9% of library traffic and 30.1% of all traffic on the server — the single largest agent family in the entire dataset. This is not a quirk of one endpoint. Cloudflare's 2025 Year in Review found that Go-based clients rose to 20% of all automated API requests, up from 12% a year earlier — the fastest-growing language among machine clients on its global network. Go has become the default language for backend automation, scrapers, and the API integrations that increasingly mediate the web, and its standard library's bare Go-http-client user agent is the visible exhaust of that shift. The pattern in our log is an amplified version of one Cloudflare sees across millions of sites.

curl, the previous leader, is now second at 7.5% — though its share of library traffic understates its real footprint. curl runs in more than 20 billion installations worldwide, embedded in phones, cars, game consoles, and servers; on a command-line-friendly API it remains the natural tool for a developer testing an endpoint. The broader point is structural: HTTP libraries are no longer a niche traffic source. They are the dominant way software now talks to the web, and distinguishing legitimate library traffic from malicious automation — both of which forge headers freely — is one of the hardest problems in modern traffic security.

Crawlers and Bots: Googlebot's Slice Shrinks to a Sixth

Traditional crawlers match HTTP libraries almost exactly in volume, but their internal makeup has shifted hard away from search.

Googlebot fell from 46.6% of crawler traffic in the original to 16.0% now — not because Googlebot slowed, but because everything else sped up. In absolute terms Googlebot's volume rose (45,879 → 68,868 requests). It was simply outgrown: a single brand- and social-monitoring crawler, AwarioBot, now generates 41.8% of all crawler traffic on its own. This is the same volume-weighting effect that reordered the top-level categories, applied one level down.

It also illustrates why crawler rankings are audience-dependent. On Cloudflare's global network, Googlebot expanded from 30% to 50% of crawler traffic between May 2024 and May 2025 and accounts for over 28% of all verified-bot traffic. Our 16.0% is far lower because our crawler population is dominated by a monitoring bot that Cloudflare's "verified bot" framing weights differently, and because a domain-intelligence endpoint attracts a heavier SEO-tooling mix than the average website. We flag the divergence rather than paper over it: the direction (Googlebot is a large but no-longer-majority crawler) agrees; the exact share depends on whose logs you read.

The SEO-tooling ecosystem is the other story in this table. AhrefsBot, SemrushBot, DataForSeoBot, DotBot, MJ12bot, and their peers collectively account for roughly 19.5% of crawler traffic — bots that continuously rebuild backlink and ranking databases. UptimeRobot's 11.0% represents the monitoring category: services pinging URLs every few minutes to check availability. Bingbot at 1.5% underscores Microsoft's comparatively modest crawling footprint.

The AI Crawl: Share Halved, Volume Climbed

The most counter-intuitive finding is that AI crawling grew and shrank at the same time.

AI crawlers' raw volume rose from 58,185 to 95,736 requests, yet their share of traffic roughly halved — from 15.0% to 7.3% — because total traffic grew faster. This is a crucial correction to the original's framing: AI crawling is not retreating; it is being diluted in a log increasingly dominated by HTTP libraries and monitoring bots. Read as a level, AI bots are more active than ever; read as a proportion, they are a smaller slice of a much bigger pie.

OpenAI is the most active operator: GPTBot (26.6%) plus OAI-SearchBot (2.4%) put it at 29.0% of AI-crawler traffic. ByteDance's Bytespider (17.7%), Amazon's Amazonbot (16.8%), and Anthropic's ClaudeBot (15.7%) follow in a tight cluster. The named operators are consistent with Cloudflare's global picture, where GPTBot rose from 2.2% to 7.7% of tracked crawlers (+305%) while Bytespider collapsed from 22.8% to 2.9% over the same window — our higher Bytespider share is consistent with a longer-tailed local sample. Perplexity, despite its consumer visibility, barely registers at 1.9%.

The economics behind these bots remain lopsided. Cloudflare's crawl-to-refer analysis found that in July 2025, ClaudeBot crawled roughly 38,000 pages for every visitor it referred back — an 86.7% improvement from January's 286,930:1, but still three orders of magnitude worse than Google's 5.4:1. OpenAI sat at about 1,091:1. The same analysis found training now drives about 79% of AI-bot activity, up from 72% a year earlier: these crawlers overwhelmingly take content to train models, not to send users back. That asymmetry — heavy extraction, negligible referral — is the structural tension every publisher now negotiates.

Browsers: A Developer Desktop

Among the 24.8% of traffic that is human, the browser mix is distinctly technical.

* Applebot and HeadlessChrome are non-human agents the parser files under the desktop-browser family; they are shown for completeness but are not human browsing.

Firefox runs at 14.2% here versus 4.1% worldwide on Statcounter — roughly 3.5x its global desktop share — while Edge collapses from 9.1% globally to 1.9%. Chrome (72.1% vs 76.4%) and Safari (5.8% vs 5.3%) track the global baseline closely. The deviations are the signal: a developer-heavy audience over-indexes on Firefox and all but abandons Edge, a pattern consistent with technical users who favor Firefox's privacy posture and avoid Microsoft's browser. The appearance of CCleaner and Avast Secure Browser in the top ranks points to a security-conscious sub-segment running hardened or bundled browsers.

The mobile picture (2.9% of all traffic) is led by Mobile Safari (41.0% of mobile) and Chrome Mobile (23.3%), with Google's supplementary crawler GoogleOther (11.6%) inflating the rest. Mobile's small absolute share is itself a finding: against a web where mobile is the majority of human traffic, a developer tool is emphatically desktop-first. It is worth remembering this is a browser-engine near-monoculture — Chrome, Edge, Opera, Brave, and CCleaner all run on Chromium/Blink, so the engine concentration behind the "browser" column is far higher than the brand diversity suggests.

Operating Systems: The Machine Majority

The OS distribution is dominated by traffic that reports no real operating system at all.

At 68.0%, "Other / none" is the largest OS bucket — and it is almost entirely HTTP libraries and crawlers that ship no OS token. This is the operating-system shadow of the category inversion: when two-thirds of traffic is automation, two-thirds of traffic has no OS to report. Restricting to actual desktop browsers tells a cleaner human story: Windows 69.1%, Mac OS X 21.7%, and the Linux family (Linux + Ubuntu + the rest) at roughly 8.9% — about double Linux's typical ~4-5% desktop share, again consistent with the developer audience. The raw, all-traffic OS table is dominated by machines; the human OS table looks like a workstation fleet.

What's at Stake

• A request is no longer a reader — plan capacity for the 75%. Three of every four requests here are automated. Server provisioning, rate limiting, caching, and abuse defenses must be sized for the machine majority, not the human quarter. Analytics that silently filter bots understate raw load by roughly 4x on a developer endpoint.
• The automation layer is the new front door, and it is hard to police. With Go-http-client alone at 30% of traffic and HTTP libraries the single largest category, the dominant client is a programmatic one that forges headers as easily as it sends them. Telling a legitimate integration from a scraper or a credential-stuffer increasingly requires signals below the user-agent string (TLS fingerprints, behavioral analysis), because the string itself is worthless as identity.
• AI crawling is an extraction economy whose share is masked by volume. AI bots grew in absolute terms even as their percentage halved. With ClaudeBot near 38,000:1 and training driving ~79% of AI crawling, the asymmetry between content taken and visitors returned is the unresolved economic question of the AI-search era — and a falling percentage does not mean falling pressure.
• Browser diversity is eroding even where humans remain. The human quarter runs on three engines, and most of the "browser" column is Chromium underneath. Firefox's over-representation in a technical audience (14.2% vs 4.1%) shows where engine diversity still lives, but for the broader web the monoculture deepens.
• User agents are becoming unreliable as identity. A self-declared, spoofable string now has to classify a population that is three-quarters machine, full of headless browsers blending in with humans and 6,534 distinct "Unknown" agents reporting nothing legible. The 2.5% Unknown share is a floor on that ambiguity, not a ceiling.

What Would Help

1. Site operators: instrument the machine half before you optimize the human quarter. Most analytics suites hide bot traffic by default, so capacity and abuse decisions get made on a quarter of the real load. Log raw user-agent categories, watch the HTTP-library and crawler shares, and size infrastructure for the 75% — the API traffic is where growth and risk both live.
2. Security and platform teams: stop trusting the user-agent string. When the single largest client is a generic Go-http-client token and headless browsers impersonate Chrome, header-based classification fails. Move identity decisions to TLS/JA-fingerprinting and behavioral signals, and treat the user-agent as a hint, not a fact.
3. SEO and content teams: account for AI crawlers as a permanent, growing cost center. Their share dipped only because other automation grew faster; their volume is rising and ~79% of it is training extraction with almost no referral return. Decide deliberately what to allow via robots.txt and crawler controls, and measure the bandwidth, not just the rankings.
4. Developers building integrations: send an honest, specific user agent. A descriptive agent string (product, version, contact URL) keeps legitimate automation out of the Unknown and bad-bot buckets and off the blocklists that header-forging scrapers are forcing operators to deploy. The default library token is increasingly a fast path to being throttled.
5. Analytics and tooling vendors: report request-weighted and agent-weighted views side by side. The same log says "browsers dominate" by distinct agents and "automation dominates" by volume. Presenting only one hides half the web. Surface both, as the user-agent database behind this post does.

This analysis was conducted using LLMSE, drawn from 1,317,348 HTTP requests across 14,828 unique user-agent strings observed via organic traffic and API submissions, classified with a deterministic parsing pipeline (the user-agents library plus a category map). All figures reflect the live traffic index as of June 2026; request shares are volume-weighted sums of per-agent hit counts. See the full User Agent database for the underlying dataset, or analyze your own site at llmse.ai/classify.