Every request tells a story.
I spent 7 years learning to read it.

The part most guides
conveniently skip

Bypassing detection is only half the question. The other half is whether you should, whether it is legal where you operate, and whether your approach survives contact with reality. This section comes first on purpose. It is deliberately honest about the limits of everything that follows.

This is not legal advice, but you need to think about these

The case law: who actually won

The line is moving: agents, robots.txt, and the end of a clean binary

The legislative response: transparency, not technical blocking

The three tiers of AI access, and where the pressure actually is

The other direction: sites that hand agents a sanctioned door (WebMCP)

The missing half of the handshake: proving who you are

Read this before you copy anything from this guide

The scraping
decision flow

Walk steps in order. Stop at the first win. Complexity and cost increase right. Most production scraping is solved at steps 1–3.

First, place the target on the difficulty ladder

Sweep the impersonation profile before you escalate the tool

Before you send a single byte,
you've already been judged.

The moment your scraper opens a TCP connection to a CDN, a fingerprinting pipeline triggers. By the time your HTTP request body arrives, four independent scoring systems have already assigned you a trust score. Here's exactly what each one measures, and why defeating just one is never enough.

Layer 1, TLS Fingerprinting: The Handshake That Betrays You

This fires before a single HTTP byte is exchanged. Understanding it is non-negotiable.

# Test your actual JA4 fingerprint against tls.browserleaks.com
import requests
from curl_cffi import requests as cffi

# ❌ requests, exposes Python/urllib3 JA4, blocked immediately
r1 = requests.get("https://tls.browserleaks.com/json")
print(r1.json()["ja4"])
# → t13d1516h2_8daaf6152771_b0da82dd1658  (Python fingerprint, catalogued, blocked)

# ✓ curl_cffi, emits Chrome 124's exact JA4 hash, HTTP/2 frames, cipher order
r2 = cffi.get(
    "https://tls.browserleaks.com/json"–
    impersonate="chrome124"  # also: chrome110, chrome107, safari17
)
print(r2.json()["ja4"])
# → t13d1517h2_c4b4b4b4b4b4_aaaaaaaaaa   (Chrome 124 fingerprint, passes)

# Also check HTTP/2 fingerprint
print(r2.json()["http2"])  # Chrome's exact SETTINGS frame values

Below the handshake: the layer most stealth work never reaches. TLS is where this section starts because it is where most practitioners start, but it is not actually the bottom. The TCP SYN packet that opens the connection carries its own signature: initial window size, maximum segment size, window scaling, the exact set and ordering of TCP options, and the initial time-to-live. Those values come from the operating system's network stack, not from your HTTP library and not from your TLS library, which is precisely what makes them useful to a defender. A request can present a flawless Chrome-on-Windows story at every layer you control and still contradict itself at layer 4, because the packet was assembled by a Linux server kernel. Passive OS fingerprinting from SYN packets is old technique, and it is now being applied to proxy egress specifically: the question stops being what your client claims and becomes what kind of machine actually sent the first packet. It is a small share of blocks compared with IP reputation and behaviour, but it is the residue that survives everything else, and it is invisible from inside the browser. A defender does not have to stop at the first packet, either. Rewriting only the SYN, at a firewall or with a packet-mangling rule, leaves every later segment, the timestamp option, window updates and retransmission timing, still produced by the real kernel, so a check that reads the whole connection sees the original stack. The fix that holds is a stack that behaves like the claimed OS from the first segment to the last, or a client persona whose OS matches the machine that actually sends the traffic. Proxy vendors began selling per-OS TCP emulation in beta in October 2026 with no published success rates, so test any such claim against a capture server you control before paying for it.

Layer 2, JavaScript Fingerprinting: The Page That Interrogates You

Once your TLS passes, the page loads its anti-bot script. This is a 500KB+ obfuscated interrogation that runs dozens of tests in parallel.

How an extension list gets built, and why it reads as evidence of a person. The anti-bot vendors that probe for browser extensions are exploiting a gap the browser deliberately left. Chrome exposes no API that reports which extensions are installed, and that is an intentional anti-fingerprinting decision. Individual extensions can still be detected one at a time, and enough of those checks in sequence reconstructs a list.

The main technique is resource probing. An extension declares web_accessible_resources, files it will serve to any page that asks, so a script can request a known file path under chrome-extension:// with the extension's fixed identifier and learn from success or failure whether it is present. Run that against a catalogue of popular extensions and you have an enumerated list.

The signal is unusually good for a defender on two counts. It carries real entropy, because the combination of a dozen installed extensions is close to unique, and it is stable, because people rarely add or remove them, so the same list returns tomorrow. The second count is the one that decides tool choice: an extension list is decent positive evidence of a human, since automated browsers almost never ship with any. That is the reasoning behind the vendor probe covered in the Akamai profile, and it is why the browsers that clear it do so by loading genuine password-manager profiles rather than by patching a response. An empty extension surface is not neutral. It is a statement.

Layer 2.6, Side Channels And State Leaks: The Signals Nobody Thinks To Patch

Layer 2.5, WebAssembly Fingerprinting: The Layer Below Your Stealth Browser

Layer 3, Network Identity: The Five Vectors That Must Agree

Layer 3.5, DOM Honeypots: The Trap Doesn't Care About Your Fingerprint

Layer 4, Behavioural ML: You Can't Fake Being Human

Six companies built the walls.
Here's every key.

Each vendor applies the detection layers differently, different weights, different signals, different architectures. The same fusion logic now shows up in published attempts to separate agents from people, and the signals that carry it are timing ones. A form that reveals each field only after the previous one is filled turns the agent’s perceive–reason–act loop into a measurable latency: in one October 2026 test, humans answered a newly revealed field in roughly 5.5 s, while two locally driven agents were faster at 1.7 s and 2.3 s and became far slower, 9.5 s and 21.2 s, once dropdowns appeared. Dwell time separated them more cleanly, with several agents showing millisecond key presses at near-zero variance against a human mean near 108 ms. Instructing an agent to type character by character made its score worse, not better, because added keypresses do not reproduce human variance. Timing signals only exist once the agent does something, so plan around the gap. One client-side detector’s November 2026 table named all 14 browsing agents it tested once they clicked or typed, but only 9 of them while they were just reading; the five it missed were four assistants running in the user’s own browser and one desktop app browsing from its vendor’s cloud. Server-side fetchers such as ChatGPT-User never execute a page script at all. If you defend a site, count those fetchers at the edge against the operators’ published IP lists rather than trusting their user agents. If you run an agent, expect acting, not reading, to be what gets it labelled. The table is the vendor’s own and comes with no accuracy or false-positive figures. One lab, one site, small human baseline. What bypasses Cloudflare has zero effect on Kasada. You need to know exactly which wall you're facing before you choose a tool.

Quick identification reference

How I approached real-world bypasses

The theory above tells you what anti-bots do. These notes tell you what I did when I hit them on a production job. Each is a full day or two of work distilled to: what I tried, why it failed, what finally worked, and the decision tree I'd use next time.

// One session, one proxy, one request
s := sensor.NewSession("chrome-148",
    sensor.WithSessionProxy("http://user:pass@comcast-ip:port"),
    sensor.WithSessionTimeout(30*time.Second),
)
resp, _ := s.Get(context.Background(), "https://target-site.com/")
// Status: 200, Protocol: h2, _abck: ~0~ (validated)

// Then GraphQL directly on the same session
gql, _ := s.DoWithBody(ctx, req, bytes.NewReader(payload))
// Status: 200, 30KB product data, zero blocks

Scrapy spider
  → GoProxyMiddleware (urllib, ~35ms round trip)
      → Go HTTP server :8765 (4-session pool)
          → Go TLS library sessions
              → ISP proxy (Comcast AS7015, static residential)
                  → Target site

Every tool built to fight
every wall we just described.

Now that you understand the detection stack and the six anti-bot vendors, every library below makes sense in context. curl_cffi exists because of JA4. Camoufox exists because of CDP leaks. PatchRight exists because of Kasada's toString() inspection. The arsenal wasn't built randomly, each tool is a direct countermeasure to a specific detection innovation. Treat any star count in the table below as a dated snapshot, not a live signal, and check a repo’s last push, licence and archived flag against the GitHub API before you adopt it. A star count can now move faster than anyone could read the code behind it, so weigh it least of all the signals. One reverse-engineering MCP server went from 136,505 stars on 12 October 2026 to 260,901 on 15 October, nearly doubling in three days, while its own commits celebrated each milestone and its stargazer list could not be enumerated to see who starred. The curve may be genuine attention, but nothing public lets you audit it. Before adopting a fast-rising tool, read its releases and security notes, pin an exact version, install with lifecycle scripts disabled, and try it offline on traffic you captured yourself. For a tool that reads apps and pages on your behalf, check what it keeps and what it passes to your model provider; no star count answers either question. A 28 Sep 2026 recheck of seven popular scraping repos found every one alive and none relicensed, but the table’s own figures had drifted badly: Scrapling was listed at 38k against a live 84.2k, ScrapeGraphAI at 18k against 31.4k, Crawlee at 15k against 25.9k and Katana at 8k against 17.6k. One widely recommended research tool, minet, has no row, and it ships under GPL-3.0, the only copyleft licence of the set, which matters if you embed it rather than run it. Stars measure attention, not whether a tool clears a given anti-bot, so use them to spot abandoned projects rather than to rank capable ones. Downloads separate them better, but less than you might hope for Python. In one 18 October 2026 snapshot of 39 scraping tools, star rank and 30-day download rank agreed closely for 10 npm packages, a rank correlation of about 0.85, and hardly at all for 28 Python ones, about 0.08: an archived Python crawler kept 16,756 stars with 350 downloads a month, while a library last released in February 2019 still drew about 370,000. Check the latest release date and the archived flag first, then downloads from pypistats with mirrors excluded, and remember that dependency pulls and CI reinstalls inflate the libraries underneath popular tools. It was a hand-picked list and a single snapshot, so read the gap as a pattern, not a law. Read the licence file and the commit graph as well as the star count, because a young stealth tool can be very popular and still rest on one maintainer. invisible_playwright had 3,009 stars less than five months after its 13 May 2026 creation, but one account wrote 663 of its 728 commits, PyPI shows 38 releases in the 30 days to 5 October, and the engine ships only for Windows and Linux. GitHub lists its licence as MIT, while its own pyproject.toml declares MIT AND Apache-2.0 because it bundles a fork of Playwright’s Python client, and the patched Firefox binary is MPL-2.0. Pin an exact version, keep the engine build you validated, and check the declared licence in the package metadata rather than the repository badge if you redistribute it. Pin Playwright itself as tightly as the stealth layer on top of it, because wrappers reach into its private modules and a minor release can break them. Playwright 1.65.0 reached PyPI on 9 Nov 2026, and by 10 Nov CloakBrowser 0.7.2 had capped its requirement below 1.65 after a fresh install failed to import playwright._impl._js_handle; Patchright’s newest release was still built on 1.64.1. An unpinned pip install in a container build would have pulled the incompatible pair. Lock both packages in the same file, upgrade them together, and run one smoke page through the wrapper in CI before the image ships. Pinning cuts both ways, so revalidate the pin when a detection change lands. Camoufox builds before the 12 Oct 2026 fix (PR #852, merge commit 83ebe0d) still register Juggler as a chrome:// package, which hCaptcha probes for from page script; on one site that validates tokens server-side, every token from such a build was refused, including ones a human solved, while stock Firefox passed on the same IP. Nothing in the page or the challenge says why, so the symptom looks like a broken solver. When a stealth engine starts failing a challenge it used to pass, check its issue tracker before you change proxies or solvers. The evidence is one reporter with small samples on one datacenter IP.

Master comparison table, all 91 libraries & tools

Browser engines, deep dive

from camoufox.sync_api import Firefox

# geoip=True: auto-aligns IP, timezone, locale, WebRTC simultaneously
with Firefox(
    geoip=True–        # align all 5 identity vectors to proxy exit country
    humanize=True–    # Gaussian mouse jitter
    proxy={"server": "http://proxy.provider.com:8011"–
           "username": "user"– "password": "pass"},
    screen={"width": 1920– "height": 1080}
) as browser:
    page = browser.new_page()
    # Warm up, never go directly to target URL
    page.goto("https://www.google.com")
    page.wait_for_timeout(2000)
    page.goto("https://cloudflare-protected.com")
    page.wait_for_load_state("networkidle")
    print(page.content()[:500])

Describe, don't
select

AI-native scraping replaces CSS selectors with natural language. A 2025 NEXT-EVAL benchmark showed an LLM hit F1 0.96 on record extraction when the page was fed as flattened JSON, and 0.10 when fed as slimmed HTML: the input format, not the model, decided the result. The 0.96 (exactly 0.9567) is Gemini 2.5 Pro on 164 real pages after each page was flattened into a map from XPath to text. Given the same pages as slimmed HTML it scored an F1 of 0.10 with a hallucination rate of 0.91, and on nested JSON 0.40. So convert the DOM into a flat, addressed representation before you ask a model for records, and have the model return paths that code then applies. Do not quote the 0.95 as what an LLM does on HTML. It is one model on one benchmark from 2025, and the authors designed both the benchmark and the input formats.

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel

# Define exactly what you want, LLM extracts it, no selectors needed
class Product(BaseModel):
    name: str
    price: float
    model_number: str
    brand: str

async def extract(url):
    strategy = LLMExtractionStrategy(
        provider="openai/gpt-4o-mini"–
        schema=Product.model_json_schema(),
        extraction_type="schema"–
        instruction="Extract all products with prices and model numbers"
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url=url– extraction_strategy=strategy)
        import json
        return json.loads(result.extracted_content)
# NEXT-EVAL 2025: F1 0.96 on flat-JSON input, 0.10 on slimmed HTML

From extraction to production-grade data

Crawl4AI and Firecrawl get you semantic understanding out of an LLM. But ask an LLM for a price across 10,000 articles and you will get $40, 40 dollars, 40 USD, "forty dollars", and occasionally null. Production pipelines cannot ingest that. The fix is to separate the two concerns LLMs conflate: semantic understanding and structural guarantees.

# pip install instructor pydantic anthropic
from pydantic import BaseModel, Field
import instructor, anthropic

class JobPosting(BaseModel):
    title: str
    company: str
    salary_min_usd: int | None = Field(description="Floor of salary range in USD")
    salary_max_usd: int | None
    years_experience_min: int
    location: str
    remote: bool

client = instructor.from_anthropic(anthropic.Anthropic())
result = client.messages.create(
    model="claude-sonnet-4",
    response_model=JobPosting,
    messages=[{"role": "user", "content": scraped_html}],
    max_retries=3,
)
# result is a validated JobPosting object, not a string
# If LLM hallucinates "competitive" for salary, Instructor retries

Making LLM extraction production-reliable: the parts that are engineering, not prompting

Serving scraped data to an agent: the naming problem replaces the scraping problem