01Daily web scraping benchmarkTRACE (Transparent Real-world Assessment of Content Extraction)
How reliably can web scrapers read a page?
Every day we send a new fresh sample of about 2,500 web pages to context.dev, Exa, Firecrawl, and Keenable. We publish how often each one gets through, how fast it answers, whether it returns the right page, what it costs, and why it fails.
Key findings · Nov 6, 2026 sample
- context.dev returned useful content from the most pages, 87.0%. It led Keenable by 2.9 points across 2,527 shared pages, a statistically significant gap.
- context.dev leads in news. The top two are statistically tied in 8 categories.
- Exa answers fastest at the median, 1.20 s. context.dev has the shortest slow tail: 95% of its requests finish within 8.91 s.
- 223 pages defeated every provider. Most came from legal and government, jobs, and Wikipedia links pages.
- pages, each sent to all four providers
- 2,527
- categories, weighted equally
- 9
- daily samples scored since Nov 4
- 3
02Leaderboard
Pages with useful content
What counts as useful
How to read the ranges
context.dev runs this benchmark and is one of the providers tested. See the methodology for scoring rules and limitations.
03Head-to-head
Same pages, paired
Missed it →context.devKeenableFirecrawlExa
Pair verdicts
- context.dev vs Keenable148 to 71context.dev ahead
- context.dev vs Firecrawl160 to 35context.dev ahead
- context.dev vs Exa285 to 73context.dev ahead
- Keenable vs Firecrawl170 to 122Keenable ahead
- Keenable vs Exa246 to 111Keenable ahead
- Firecrawl vs Exa258 to 171Firecrawl ahead
A pair is called only when an exact McNemar test on the pages the two disagree on passes at the 95% level. Requests rejected by our own account limits are left out of both sides.
04Categories
Results by category
In the table, black marks the highest rate in each row. “No provider” counts sample pages that all four failed. A verdict compares the top two providers in that category with a paired test on the same pages; “Tied” means the gap is within the noise. Categories hold up to 300 pages each.
05Latency
How long a page takes
Requests that hit the deadline
How we measure
06Quality
From response to useful page
- Returned content
- The provider answered with text before the deadline.
- Page accessed
- The text is real page data, not a block or error screen (earlier samples need a page-access regrade).
- Right page
- It is the requested document, not a homepage or a different item.
- Substantive
- It holds the page's main content, not just a title or navigation.
| Provider | Right page | Substantive | Ended on another page | Median length | Length vs. other providers |
|---|---|---|---|---|---|
| context.dev | 93.6% | 86.8% | — | 8,205 chars | 1.20× |
| Keenable | 91.2% | 83.8% | — | 5,429 chars | 1.00× |
| Firecrawl | 90.5% | 81.9% | — | 11,897 chars | 1.65× |
| Exa | 83.8% | 78.4% | — | 5,252 chars | 1.00× |
“Ended on another page” compares the final URL with the requested one, ignoring the scheme, “www.”, query strings, and trailing slashes. Length compares each provider with the median length across providers that returned the same page. context.dev and Firecrawl request main content only while Exa returns full text, so length is a signal, not a score.
07Cost
What a page costs
Each provider's published monthly price for roughly 100,000 pages, billed monthly. Volume discounts, annual billing, and free tiers change the result. Prices checked Nov 5, 2026. context.dev and Keenable say they do not bill failed requests, so their cost per accessed page equals their request price. * Exa and Firecrawl do not say, so we assume failed requests are billed. Keenable's first 100,000 requests each month are free.
08Failures
Why pages fail
Bars show the share of all scored pages each provider failed to return useful content for. Hover a segment for the count. Provider-reported failures and missed deadlines count as failures, even when the provider did not return an error code. Thin pages and wrong documents count as failures under this scoring rule.
09History
Daily results over time
November 4, 2026 – November 6, 2026 3 sampled days
Showing the latest completed sample by default. Hover to preview a day; click, touch or use the arrow keys to keep it selected. Gaps are not estimated.
November 6, 2026 UTC
Useful content · 2,527 pages
- context.dev
- 87.0%
- Exa
- 79.1%
- Firecrawl
- 82.3%
- Keenable
- 84.2%
Reading the daily history
Same evidence, new rules
10Sample & sources
Where the pages come from
- Discover. On each source's schedule we poll publisher feeds, job boards, SEC and Federal Register listings, and sitemaps for new URLs. Wikipedia edits from the previous UTC day are checked once a day for new external links.
- Sample. At 00:15 UTC we draw up to 300 previously unseen pages per category, with at most 125 from one organization and 375 from one website.
- Freeze. The sample is locked and sent to all four providers with caching turned off for every one.
- Grade. A blind judge reads each response. Every URL, verdict, and timing is published in the page-level data.
| Category | Sources | Examples |
|---|---|---|
| News | 29 | BBC, NPR, Al Jazeera, CNBC |
| Commerce | 6 | Allbirds, Bombas, Gymshark, Vuori |
| Jobs | 23 | Stripe, Anthropic, Spotify, Cloudflare |
| Finance | 20 | SEC filings, Federal Reserve |
| Research | 17 | arXiv, PLOS, Nature, eLife |
| Legal & government | 37 | Federal Register, UK legislation, FTC |
| Software & docs | 31 | VS Code, Rust, Kubernetes, PostgreSQL |
| Community | 34 | Hacker News, Stack Overflow, Bluesky |
| Other | 1 | New external links on English Wikipedia |
Daily source examples by category
These feeds, APIs, and sitemaps supply the URLs for the daily sample. We follow those links and test whether each provider can recover content from the destination page. We assign each page the category of the source that supplied it. Pages are new to the benchmark, but may have been published earlier.
- News
- Publisher articles · BBC feed ↗
- Commerce
- Product pages · Allbirds sitemap ↗
- Jobs
- Individual job postings · Stripe job API ↗
- Finance
- Company filings and financial releases · Apple SEC filings ↗
- Research
- Papers and scholarly articles · Nature feed ↗
- Legal & government
- Rules, notices and official records · Federal Register API ↗
- Software & docs
- Release notes and documentation · VS Code releases API ↗
- Community
- Discussion threads and social posts · Stack Overflow API ↗
These are configured discovery sources, not a guarantee that every source contributes to every daily sample.
11Methodology
How the benchmark works
How scores are calculated
A page counts as useful when the response contains substantive content that matches the requested page. We calculate the percentage of useful pages within each category, then average those percentages, so every category has the same influence regardless of its page count. Failed requests, unresolved judgments, and pending results remain in the denominator and lower the score. Requests rejected by our own account limits (HTTP 401, 402, or 429, or exhausted credits) are excluded from that provider's score and listed separately; they are not retried, so no provider gets a second attempt at a page.
The ± range is a 95% confidence interval for the category-weighted score. Rankings use an exact McNemar test on the pages two providers disagree on; providers that do not differ at the 95% level share a place.
For the current selection, 248 provider results remain unresolved and 0 are pending. These results count against the score until useful content is confirmed.
The judge, and how we audit it
TypeSafe's Jev 1.13.0 model grades up to 12,000 characters taken from the beginning, middle, and end of each response. For the useful-content score it checks for substantive content matching the requested page, without seeing the provider name. Unclear evidence and judge failures stay unresolved rather than being guessed. Document and content grades for the Quality section use a separate judge call that also sees the requested URL, title, and category, still without the provider name.
Each week we hand-check a random sample of judged pages. Reviewers see the requested URL and the same evidence as the judge, but not the provider or the judge's verdict. No audit has been published for this scoring rule yet; agreement will appear here once it has.
Fresh content and cache checks
Every request asks for a fresh fetch, and every response is checked for the provider's own record of where the content came from. A response that a provider marks as served from cache is left out of the scores and latency instead of being counted as a success.
| Provider | Fresh-content setting | Response field checked |
|---|---|---|
| context.dev | maxAgeMs: 0 | cache_metadata.status, cache_metadata.age_ms |
| Exa | maxAgeHours: 0 | statuses[].source |
| Firecrawl | maxAge: 0, storeInCache: false | metadata.cacheState |
| Keenable | live=true | usage.sku must be fetch.live |
Request settings for each provider
Each provider's API is different, so these are the closest equivalent settings. Any change to them is logged in the repository with the date it took effect.
| Provider | Endpoint | Fresh content | Content | Deadline |
|---|---|---|---|---|
| context.dev | /v1/web/scrape | maxAgeMs: 0 | Main content only | 27 s |
| Exa | /contents | maxAgeHours: 0 | Full text | 27 s |
| Firecrawl | /v2/scrape | maxAge: 0, storeInCache: false | Main content only | 27 s |
| Keenable | /v1/fetch | live=true | Default | 30 s cutoff (no setting) |
All outputs are limited to 250,000 characters.
Deadlines, timeouts, and rate limits
Each request has a 30-second hard cutoff on our side. context.dev, Exa, and Firecrawl are also asked to give up at 27 seconds. Providers report a missed deadline in different ways: an HTTP 408, an empty failure, or our cutoff. All of them count as timeouts.
Scheduled collection checks every fifteen minutes, and workers continue through the queued sample. Limits are 4,000 requests per provider per UTC day, plus a separate grading budget. HTTP 401, 402, and 429 responses pause the affected provider across workers. Results reflect our own account plans, and requests those limits reject are excluded from scores. Interrupted requests are not automatically repeated, which avoids giving a provider extra attempts at the same page.
Reruns, duplicate days, and retention
Every sampled day has exactly one run.
Grading evidence (the excerpts the judge reads) is compressed and kept for 30 days so samples can be regraded and audited. Page-level records are kept for 180 days, and daily summaries are kept permanently. Raw responses are not published.
Daily collection and Wikipedia discovery
The selected daily run collected 2,527 of 2,700 targeted pages. In the unfiltered daily-only view, a missing category withholds the overall score.
Wikipedia discovery runs once before the daily sample is created, using up to 1,200 non-bot article edits spread across the previous UTC day. We compare the old and new revision text. Only new external URLs that have not been tested before are eligible. The daily sample includes up to 300 of these pages in the Other category, without inferring their subject from the citing article. This is a sample of edits, not every link added to Wikipedia. If discovery fails, the other sources can still form the daily sample, and Wikipedia is checked again the next day.
This group receives the same weight as each other category when present; missing Wikipedia pages do not suppress the eight-category daily score.
Known limitations
The sample is drawn from sources we chose; it is not a random sample of the web. The judge can make mistakes, which is why its agreement with human reviewers is published. Provider rate limits, request budgets, and the number of available pages can reduce a daily sample. Latency is measured from one region with caching turned off, which is slower than many real workloads.
186 of 198 sources last fetched successfully. Last collector update: 2026-11-06 18:00 UTC.
12Data & API
Check our work
- Filtered results for this view
- Every page-level record for 2026-11-06NDJSON
- Daily summaries with intervalsJSON
- Judge prompts and request settingsGitHub ↗
- Full source codeMIT license ↗
- Plain-text summary for AI assistantsllms.txt
curl -s https://trace.context.dev/api/report \ | jq '.latest.providers[] | {id, rate, ci, p50, p95}'13FAQ
Common questions
Which web scraping API has the highest success rate?
In the Nov 6, 2026 sample, context.dev returned useful content from 87.0% of 2,527 pages, followed by Keenable at 84.2%, Firecrawl at 82.3%, Exa at 79.1%. Its lead over the next provider is statistically significant.
Which web scraping API is fastest?
Exa had the fastest median response, 1.20 seconds. context.dev had the shortest slow tail: 95% of its requests finished within 8.91 seconds. Times are measured with caching turned off.
Does the benchmark use cached pages?
No. Every request asks for a fresh fetch, and we check each response for the provider's own cache marker. Responses served from cache are left out of the scores and latency.
How often is the benchmark updated?
A new sample of about 2,500 never-tested pages is drawn every day at 00:15 UTC and sent to all four providers. Its scores fill in over the same UTC day.
What does a success mean?
Each sample states its scoring rule. Under page access, a page succeeds when actual page data loaded rather than a block, login screen, error, or empty shell. Under useful content, the response must also contain substantive content from the requested page, a stricter test, so compare scores only between samples that use the same rule.
Can I reproduce the results?
Yes. Every page-level record is downloadable, and the collector, judge prompt, and request settings are open source.
14Citation
Cite the benchmark
Our approach is inspired by NEEDLE: A Live Open-Source Search Benchmark for AI Agents (2026), by Andrey Styskin, Matthias Petri, and Ilya Gusev. Its fresh-query methodology and transparent comparisons informed this benchmark, which evaluates scraping rather than search.
@misc{trace2026, author = {Bakour, Yahia and Ryaboy, Michael}, title = {{TRACE}: A Live Open-Source Scrape Benchmark for {AI} Agents}, year = {2026}, url = {https://github.com/context-dot-dev/trace-benchmark}, note = {Contact: yahia@context.dev,michael@context.dev}}