How reliably can web scrapers read a page?

A page counts as accessed when actual page data or file content loaded. A partial preview, populated listing, publication landing page, or raw file can pass. Blocks, mandatory login screens, main-page errors, and empty shells fail when they contain no page data. Relevance, document identity, completeness, and extraction quality are not part of the access score; they are reported separately under Quality. We calculate the percentage of accessed pages within each category, then average those percentages, so every category has the same influence regardless of its page count. Failed requests, unresolved judgments, and pending results remain in the denominator and lower the score. Requests rejected by our own account limits (HTTP 401, 402, or 429, or exhausted credits) are excluded from that provider's score and listed separately; they are not retried, so no provider gets a second attempt at a page.

The ± range is a 95% confidence interval for the category-weighted score. Rankings use an exact McNemar test on the pages two providers disagree on; providers that do not differ at the 95% level share a place.

For the current selection, 9 provider results remain unresolved and 0 are pending. These results count against the score until access is confirmed.

TypeSafe's Jev 1.13.0 model grades up to 12,000 characters taken from the beginning, middle, and end of each response. For the access score it sees only that text and whether it was sampled, never the provider name or the requested URL. Unclear evidence and judge failures stay unresolved rather than being guessed. Document and content grades for the Quality section use a separate judge call that also sees the requested URL, title, and category, still without the provider name.

Each week we hand-check a random sample of judged pages. Reviewers see the requested URL and the same evidence as the judge, but not the provider or the judge's verdict. No audit has been published for this scoring rule yet; agreement will appear here once it has.

Every request asks for a fresh fetch, and every response is checked for the provider's own record of where the content came from. A response that a provider marks as served from cache is left out of the scores and latency instead of being counted as a success.

How each provider's cache status is checked

Provider	Fresh-content setting	Response field checked
context.dev	maxAgeMs: 0	cache_metadata.status, cache_metadata.age_ms
Exa	maxAgeHours: 0	statuses[].source
Firecrawl	maxAge: 0, storeInCache: false	metadata.cacheState
Keenable	live=true	usage.sku must be fetch.live

Each provider's API is different, so these are the closest equivalent settings. Any change to them is logged in the repository with the date it took effect.

Request settings

Provider	Endpoint	Fresh content	Content	Deadline
context.dev	/v1/web/scrape	maxAgeMs: 0	Main content only	27 s
Exa	/contents	maxAgeHours: 0	Full text	27 s
Firecrawl	/v2/scrape	maxAge: 0, storeInCache: false	Main content only	27 s
Keenable	/v1/fetch	live=true	Default	30 s cutoff (no setting)

All outputs are limited to 250,000 characters.

Each request has a 30-second hard cutoff on our side. context.dev, Exa, and Firecrawl are also asked to give up at 27 seconds. Providers report a missed deadline in different ways: an HTTP 408, an empty failure, or our cutoff. All of them count as timeouts.

Scheduled collection starts every Friday at 00:15 UTC, and workers continue through the queued sample. Limits are 4,000 requests per provider per UTC day, plus a separate grading budget. HTTP 401, 402, and 429 responses pause the affected provider across workers. Results reflect our own account plans, and requests those limits reject are excluded from scores. Interrupted requests are not automatically repeated, which avoids giving a provider extra attempts at the same page.

Every sampled day has exactly one run.

Grading evidence (the excerpts the judge reads) is compressed and kept for 30 days so samples can be regraded and audited. Page-level records are kept for 180 days, and run summaries are kept permanently. Raw responses are not published.

The selected run collected 2,527 of 2,700 targeted pages. In the unfiltered view, a missing category withholds the overall score.

Wikipedia discovery runs once before the weekly sample is created, using up to 1,200 non-bot article edits spread across the previous UTC day. We compare the old and new revision text. Only new external URLs that have not been tested before are eligible. The weekly sample includes up to 300 of these pages in the Other category, without inferring their subject from the citing article. This is a sample of edits, not every link added to Wikipedia. If discovery fails, the other sources can still form the weekly sample, and Wikipedia is checked again during the next weekly collection.

This group receives the same weight as each other category when present; missing Wikipedia pages do not suppress the eight-category score.

The sample is drawn from sources we chose; it is not a random sample of the web. The judge can make mistakes, which is why its agreement with human reviewers is published. Provider rate limits, request budgets, and the number of available pages can reduce a sample. Latency is measured from one region with caching turned off, which is slower than many real workloads.

184 of 198 sources last fetched successfully. Last collector update: 2026-11-07 02:30 UTC.