Every request tells a story.
I spent 7 years learning to read it.

The part most guides
conveniently skip

Bypassing detection is only half the question. The other half is whether you should, whether it is legal where you operate, and whether your approach survives contact with reality. This section comes first on purpose. It is deliberately honest about the limits of everything that follows.

This is not legal advice, but you need to think about these

The case law: who actually won

The line is moving: agents, robots.txt, and the end of a clean binary

The legislative response: transparency, not technical blocking

The three tiers of AI access, and where the pressure actually is

The other direction: sites that hand agents a sanctioned door (WebMCP)

The missing half of the handshake: proving who you are

Read this before you copy anything from this guide

The scraping
decision flow

Walk steps in order. Stop at the first win. Complexity and cost increase right. Most production scraping is solved at steps 1–3.

First, place the target on the difficulty ladder

Sweep the impersonation profile before you escalate the tool

Before you send a single byte,
you've already been judged.

The moment your scraper opens a TCP connection to a CDN, a fingerprinting pipeline triggers. By the time your HTTP request body arrives, four independent scoring systems have already assigned you a trust score. Here's exactly what each one measures, and why defeating just one is never enough.

Layer 1, TLS Fingerprinting: The Handshake That Betrays You

This fires before a single HTTP byte is exchanged. Understanding it is non-negotiable.

# Test your actual JA4 fingerprint against tls.browserleaks.com
import requests
from curl_cffi import requests as cffi

# ❌ requests, exposes Python/urllib3 JA4, blocked immediately
r1 = requests.get("https://tls.browserleaks.com/json")
print(r1.json()["ja4"])
# → t13d1516h2_8daaf6152771_b0da82dd1658  (Python fingerprint, catalogued, blocked)

# ✓ curl_cffi, emits Chrome 124's exact JA4 hash, HTTP/2 frames, cipher order
r2 = cffi.get(
    "https://tls.browserleaks.com/json"–
    impersonate="chrome124"  # also: chrome110, chrome107, safari17
)
print(r2.json()["ja4"])
# → t13d1517h2_c4b4b4b4b4b4_aaaaaaaaaa   (Chrome 124 fingerprint, passes)

# Also check HTTP/2 fingerprint
print(r2.json()["http2"])  # Chrome's exact SETTINGS frame values

Below the handshake: the layer most stealth work never reaches. TLS is where this section starts because it is where most practitioners start, but it is not actually the bottom. The TCP SYN packet that opens the connection carries its own signature: initial window size, maximum segment size, window scaling, the exact set and ordering of TCP options, and the initial time-to-live. Those values come from the operating system's network stack, not from your HTTP library and not from your TLS library, which is precisely what makes them useful to a defender. A request can present a flawless Chrome-on-Windows story at every layer you control and still contradict itself at layer 4, because the packet was assembled by a Linux server kernel. Passive OS fingerprinting from SYN packets is old technique, and it is now being applied to proxy egress specifically: the question stops being what your client claims and becomes what kind of machine actually sent the first packet. It is a small share of blocks compared with IP reputation and behaviour, but it is the residue that survives everything else, and it is invisible from inside the browser.

Layer 2, JavaScript Fingerprinting: The Page That Interrogates You

Once your TLS passes, the page loads its anti-bot script. This is a 500KB+ obfuscated interrogation that runs dozens of tests in parallel.

How an extension list gets built, and why it reads as evidence of a person. The anti-bot vendors that probe for browser extensions are exploiting a gap the browser deliberately left. Chrome exposes no API that reports which extensions are installed, and that is an intentional anti-fingerprinting decision. Individual extensions can still be detected one at a time, and enough of those checks in sequence reconstructs a list.

The main technique is resource probing. An extension declares web_accessible_resources, files it will serve to any page that asks, so a script can request a known file path under chrome-extension:// with the extension's fixed identifier and learn from success or failure whether it is present. Run that against a catalogue of popular extensions and you have an enumerated list.

The signal is unusually good for a defender on two counts. It carries real entropy, because the combination of a dozen installed extensions is close to unique, and it is stable, because people rarely add or remove them, so the same list returns tomorrow. The second count is the one that decides tool choice: an extension list is decent positive evidence of a human, since automated browsers almost never ship with any. That is the reasoning behind the vendor probe covered in the Akamai profile, and it is why the browsers that clear it do so by loading genuine password-manager profiles rather than by patching a response. An empty extension surface is not neutral. It is a statement.

Layer 2.6, Side Channels And State Leaks: The Signals Nobody Thinks To Patch

Layer 2.5, WebAssembly Fingerprinting: The Layer Below Your Stealth Browser

Layer 3, Network Identity: The Five Vectors That Must Agree

Layer 3.5, DOM Honeypots: The Trap Doesn't Care About Your Fingerprint

Layer 4, Behavioural ML: You Can't Fake Being Human

Six companies built the walls.
Here's every key.

Each vendor applies the detection layers differently, different weights, different signals, different architectures. What bypasses Cloudflare has zero effect on Kasada. You need to know exactly which wall you're facing before you choose a tool.

Quick identification reference

How I approached real-world bypasses

The theory above tells you what anti-bots do. These notes tell you what I did when I hit them on a production job. Each is a full day or two of work distilled to: what I tried, why it failed, what finally worked, and the decision tree I'd use next time.

// One session, one proxy, one request
s := sensor.NewSession("chrome-148",
    sensor.WithSessionProxy("http://user:pass@comcast-ip:port"),
    sensor.WithSessionTimeout(30*time.Second),
)
resp, _ := s.Get(context.Background(), "https://target-site.com/")
// Status: 200, Protocol: h2, _abck: ~0~ (validated)

// Then GraphQL directly on the same session
gql, _ := s.DoWithBody(ctx, req, bytes.NewReader(payload))
// Status: 200, 30KB product data, zero blocks

Scrapy spider
  → GoProxyMiddleware (urllib, ~35ms round trip)
      → Go HTTP server :8765 (4-session pool)
          → Go TLS library sessions
              → ISP proxy (Comcast AS7015, static residential)
                  → Target site

Every tool built to fight
every wall we just described.

Now that you understand the detection stack and the six anti-bot vendors, every library below makes sense in context. curl_cffi exists because of JA4. Camoufox exists because of CDP leaks. PatchRight exists because of Kasada's toString() inspection. The arsenal wasn't built randomly, each tool is a direct countermeasure to a specific detection innovation.

Master comparison table, all 86 libraries & tools

Browser engines, deep dive

from camoufox.sync_api import Firefox

# geoip=True: auto-aligns IP, timezone, locale, WebRTC simultaneously
with Firefox(
    geoip=True–        # align all 5 identity vectors to proxy exit country
    humanize=True–    # Gaussian mouse jitter
    proxy={"server": "http://proxy.provider.com:8011"–
           "username": "user"– "password": "pass"},
    screen={"width": 1920– "height": 1080}
) as browser:
    page = browser.new_page()
    # Warm up, never go directly to target URL
    page.goto("https://www.google.com")
    page.wait_for_timeout(2000)
    page.goto("https://cloudflare-protected.com")
    page.wait_for_load_state("networkidle")
    print(page.content()[:500])

Describe, don't
select

AI-native scraping replaces CSS selectors with natural language. A 2025 NEXT-EVAL benchmark showed LLMs hit F1 > 0.95 on structured extraction when input is properly formatted.

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel

# Define exactly what you want, LLM extracts it, no selectors needed
class Product(BaseModel):
    name: str
    price: float
    model_number: str
    brand: str

async def extract(url):
    strategy = LLMExtractionStrategy(
        provider="openai/gpt-4o-mini"–
        schema=Product.model_json_schema(),
        extraction_type="schema"–
        instruction="Extract all products with prices and model numbers"
    )
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url=url– extraction_strategy=strategy)
        import json
        return json.loads(result.extracted_content)
# F1 > 0.95 on well-structured pages, NEXT-EVAL benchmark 2025

From extraction to production-grade data

Crawl4AI and Firecrawl get you semantic understanding out of an LLM. But ask an LLM for a price across 10,000 articles and you will get $40, 40 dollars, 40 USD, "forty dollars", and occasionally null. Production pipelines cannot ingest that. The fix is to separate the two concerns LLMs conflate: semantic understanding and structural guarantees.

# pip install instructor pydantic anthropic
from pydantic import BaseModel, Field
import instructor, anthropic

class JobPosting(BaseModel):
    title: str
    company: str
    salary_min_usd: int | None = Field(description="Floor of salary range in USD")
    salary_max_usd: int | None
    years_experience_min: int
    location: str
    remote: bool

client = instructor.from_anthropic(anthropic.Anthropic())
result = client.messages.create(
    model="claude-sonnet-4",
    response_model=JobPosting,
    messages=[{"role": "user", "content": scraped_html}],
    max_retries=3,
)
# result is a validated JobPosting object, not a string
# If LLM hallucinates "competitive" for salary, Instructor retries

Making LLM extraction production-reliable: the parts that are engineering, not prompting

Serving scraped data to an agent: the naming problem replaces the scraping problem

Agentic browsers
the visitor that reads, decides, then clicks

A category that did not exist in a serious way two years ago and now sits on both sides of this guide: it is the thing you might build with, and it is the thing arriving at your target. Worth its own section because the economics, the failure modes and the threat model are all different from headless automation, and most of what is written about it is either marketing or panic.

Four categories, and they are not competing with each other

People argue about these as if one wins. They serve genuinely different jobs, and the distinction that matters is who the browser is for and whose session it uses.

The cost is the tool surface, not the model

This is now the best-evidenced claim in agentic browsing, with five independent measurements from five teams using five methods. It is also the one most likely to save you money this week, because on most frameworks it is a configuration change rather than a rewrite.

The security problem is architectural, not a bug queue

The operational tax nobody scopes

Agentic browsing demos beautifully and degrades quietly, which is the worst combination for a project plan. Three failure modes that show up after the pilot.

What this means if you scrape rather than build agents

When DIY cost
exceeds platform cost

If spending more than 2 engineer-days/month on anti-bot maintenance, a managed platform is cheaper. Crossover typically hits when facing F5 Shape or Kasada at scale.

IP type matters
more than provider

Rotating proxies is table stakes. The real variable is IP type, datacenter IPs score near-zero on DataDome and PerimeterX regardless of fingerprint quality.

from curl_cffi import requests
import time– random

session = requests.Session(impersonate="chrome124")

# Crawlera/Zyte: BOTH keys use http://, never https://
PROXIES = {
    "http":  "http://apikey:@proxy.crawlera.com:8011"–
    "https": "http://apikey:@proxy.crawlera.com:8011"–  # http:// not https://
}

def fetch(url– retries=3):
    for i in range(retries):
        try:
            r = session.get(url– proxies=PROXIES–
                             timeout=30– verify=False)  # verify=False: proxy cert
            if r.status_code == 200: return r
            if r.status_code in (403–429):
                time.sleep(2**i + random.uniform(0–1))
        except Exception as e:
            print(f"Error: {e}")
    return None

Rotate sessions, not IP addresses: stickiness is the strategy

Know where your IPs come from, and remember you are a guest

What a compromised residential pool actually looks like from the inside

Verify the geography, do not buy it. Where a provider says an IP lives and where it answers from are different claims, and the industry lets the first stand in for the second. Location data for most networks comes from geofeeds the operators publish about themselves, which is a self-declaration with no verification step in front of it. Measure instead. Ping the addresses from a distributed measurement network and compare the latency to what the claimed location would require, and a familiar pattern appears: addresses advertised across dozens of countries resolve to a handful of large datacentre regions, with Northern Virginia, Amsterdam and Singapore absorbing most of it. Asia is the worst offender, where an address sold as Indonesian or Australian is very often answering from Singapore because the bandwidth is cheaper there.

This matters beyond truth in advertising. The five-vector coherence test at the top of the network layer assumes your exit is where you think it is. Set a timezone, an Accept-Language and a DNS resolver to match a country your traffic is not actually leaving from, and you have built a contradiction carefully and deliberately. Before you tune the four vectors you control, measure the one you bought.

The risk that runs the other way: your IP is liable for the whole pool

Walk this in order.
Stop at first win.

Each step adds complexity, cost, and maintenance. Most production scraping is solved at steps 1–3. Never start at step 5.

Quick reference cheat sheet

Disclosure · my company
If your target sits on a Sometimes, Rarely or No row, email me at scrapesync@gmail.com before you lose a weekend to it. Send the URL and a note on what you have already tried, and I will tell you which rung it is actually on and what I would try next. I usually reply within a couple of days, slower when I am mid-job. Most of the time the answer is a section further up this page and I will point you at it. Sometimes it is one of the managed APIs in the Best tool column, in which case I will name Zyte or Scrapfly rather than myself. The interest I should declare, since this is a table of vendors and I sell on the same rung: scraping is my day job, I run ScrapeSync (my company). The answer is free either way.

What practitioners are
actually shipping in 2026

Fresh insights from engineers actively solving these problems in production. Shared publicly on LinkedIn.