What QUEST has collected, and what it is still chewing through.
Every URL QUEST knows about, by state.
| State | URLs | share |
|---|---|---|
| pending | 2,051,063 | 86.4% |
| blocked | 80,658 | 3.4% |
| failed | 243,436 | 10.2% |
Pages stored the way the current extractor wants them — text from v2 with HTML kept.
46,992 of the remainder are blocked or failed — unreachable, so progress legitimately plateaus short of 100%.
| Version | Pages | share |
|---|---|---|
| extractor v1 | 81,393 | 5.5% |
| extractor v2 | 1,411,801 | 94.5% |
Writing system of archived text, from a 3,000-page sample.
| Script | Pages | share |
|---|---|---|
| Latin | 2,603 | 89.1% |
| CJK | 255 | 8.7% |
| Cyrillic | 26 | 0.9% |
| Arabic | 23 | 0.8% |
| Devanagari | 16 | 0.5% |
Share of the queue by TLD.
| TLD | URLs | share |
|---|---|---|
| .com | 1,156,933 | 48.7% |
| .org | 529,100 | 22.3% |
| .edu | 83,612 | 3.5% |
| .io | 61,443 | 2.6% |
| .net | 57,952 | 2.4% |
| .other | 486,117 | 20.5% |
Registrable domain, so every *.wikipedia.org language edition counts once.
A tall bar here is a domain crowding out the rest of the crawl.
| Domain | Queued URLs | share |
|---|---|---|
| wikipedia.org | 90,175 | 18.4% |
| tumblr.com | 80,039 | 16.3% |
| blogspot.com | 60,951 | 12.4% |
| stackexchange.com | 58,270 | 11.9% |
| github.com | 36,390 | 7.4% |
| wiktionary.org | 26,483 | 5.4% |
| wikisource.org | 18,491 | 3.8% |
| diesel.com | 17,108 | 3.5% |
| wordpress.com | 16,967 | 3.5% |
| google.com | 15,043 | 3.1% |
| iana.org | 14,895 | 3.0% |
| stanford.edu | 14,309 | 2.9% |
| wikimedia.org | 14,044 | 2.9% |
| co.uk | 13,292 | 2.7% |
| wikiquote.org | 13,274 | 2.7% |
Characters of extracted text, from the sample.
| Size | Pages | share |
|---|---|---|
| 0 | 77 | 2.6% |
| <500 | 103 | 3.4% |
| 500–2k | 351 | 11.7% |
| 2k–5k | 784 | 26.1% |
| 5k–20k | 1,370 | 45.7% |
| 20k–100k | 287 | 9.6% |
| 100k+ | 28 | 0.9% |
Last recorded reason, across 200,000 flagged URLs.
| Reason | URLs | share |
|---|---|---|
| 4xx not found | 108,109 | 54.1% |
| timeout | 39,580 | 19.8% |
| link bomb | 19,554 | 9.8% |
| other | 7,398 | 3.7% |
| not HTML | 7,249 | 3.6% |
| 5xx server error | 5,914 | 3.0% |
| DNS failure | 4,983 | 2.5% |
| TLS error | 3,995 | 2.0% |
| connection refused/reset | 2,396 | 1.2% |
| tarpit / hard timeout | 822 | 0.4% |
One point per snapshot. All three series are URL counts on a single axis.
Averaging 5,885 pages/day archived across 31.0 days of snapshots.
| Snapshot | Archived | Queue | HTML | DB |
|---|---|---|---|---|
| 2026-11-15 04:15 | 1,493,179 | 2,051,063 | 1,411,783 | 64.4 GB |
| 2026-11-14 04:15 | 1,492,756 | 2,051,471 | 1,411,351 | 64.3 GB |
| 2026-11-13 04:15 | 1,492,386 | 2,051,829 | 1,410,973 | 64.3 GB |
| 2026-11-12 04:15 | 1,491,923 | 2,052,295 | 1,410,504 | 64.3 GB |
| 2026-11-11 04:15 | 1,491,486 | 2,052,717 | 1,410,065 | 64.2 GB |
| 2026-11-10 04:15 | 1,491,025 | 2,053,212 | 1,409,600 | 64.2 GB |
| 2026-11-09 04:15 | 1,490,538 | 2,053,719 | 1,409,107 | 64.1 GB |
| 2026-11-08 04:15 | 1,490,118 | 2,054,160 | 1,408,679 | 64.1 GB |
| 2026-11-07 04:15 | 1,489,621 | 2,054,672 | 1,408,168 | 64 GB |
| 2026-11-06 04:15 | 1,489,109 | 2,055,212 | 1,407,644 | 63.9 GB |
| 2026-11-05 04:15 | 1,488,782 | 2,055,647 | 1,407,311 | 63.9 GB |
| 2026-11-04 04:15 | 1,488,319 | 2,056,073 | 1,406,843 | 63.9 GB |
| 2026-11-03 04:15 | 1,487,972 | 2,056,422 | 1,406,490 | 63.8 GB |
| 2026-11-02 04:15 | 1,487,631 | 2,056,803 | 1,406,149 | 63.8 GB |
| 2026-11-01 04:15 | 1,487,271 | 2,057,167 | 1,405,786 | 63.7 GB |
| 2026-10-31 04:15 | 1,486,941 | 2,057,571 | 1,405,456 | 63.7 GB |
| 2026-10-30 04:15 | 1,486,643 | 2,057,934 | 1,405,157 | 63.6 GB |
| 2026-10-29 04:15 | 1,486,241 | 2,058,372 | 1,404,749 | 63.6 GB |
| 2026-10-28 04:15 | 1,486,008 | 2,058,843 | 1,404,513 | 63.5 GB |
| 2026-10-27 04:15 | 1,485,777 | 2,059,319 | 1,404,281 | 63.5 GB |
| 2026-10-26 04:15 | 1,485,579 | 2,059,650 | 1,404,082 | 63.5 GB |
| 2026-10-25 04:15 | 1,485,427 | 2,059,946 | 1,403,929 | 63.4 GB |
| 2026-10-24 04:15 | 1,485,066 | 2,060,234 | 1,403,568 | 63.4 GB |
| 2026-10-23 04:15 | 1,474,577 | 2,064,803 | 1,392,940 | 62.8 GB |
| 2026-10-21 04:15 | 1,440,388 | 2,072,190 | 1,358,262 | 60.9 GB |
| 2026-10-20 04:15 | 1,423,270 | 2,075,380 | 1,340,929 | 60 GB |
| 2026-10-19 04:15 | 1,405,053 | 2,078,114 | 1,322,659 | 59 GB |
| 2026-10-18 04:15 | 1,386,131 | 2,082,263 | 1,303,684 | 58 GB |
| 2026-10-17 04:15 | 1,368,790 | 2,000,080 | 1,286,052 | 57 GB |
| 2026-10-16 04:15 | 1,357,080 | 2,000,018 | 1,274,131 | 56.4 GB |
| 2026-10-15 04:15 | 1,310,749 | 2,000,101 | 1,227,318 | 54.1 GB |
Snapshot generated in 30,441,451 ms. Sampled figures use 3,000 random pages. Schema v5, extractor v2.