XIO

xStats

What QUEST has collected, and what it is still chewing through.

Crawler idle state: idle heartbeat 12.8 h ago 60 pages/min configured snapshot 21.2 h ago
Pages archived 1,493,179 1,493,172 in the search index
HTML archived 1,411,783 49.6 GB stored, 293.5 GB raw
Queue pending 2,051,063 76,712 hosts
Database size 64.4 GB WAL 1.1 GB
Mean page 10,588 chars of text (sampled)
Domains 71,333 459 hosts hold half the queue

Queue composition

Every URL QUEST knows about, by state.

pending: 2,051,063 (86.4%)blocked: 80,658 (3.4%)failed: 243,436 (10.2%)
  • pending2,051,063 86.4%
  • blocked80,658 3.4%
  • failed243,436 10.2%
Table view
StateURLsshare
pending2,051,06386.4%
blocked80,6583.4%
failed243,43610.2%

Archive coverage

Pages stored the way the current extractor wants them — text from v2 with HTML kept.

Current 1,411,783 (94.5%)Needs refetch 81,396

46,992 of the remainder are blocked or failed — unreachable, so progress legitimately plateaus short of 100%.

By extractor version
VersionPagesshare
extractor v181,3935.5%
extractor v21,411,80194.5%

Script mix

Writing system of archived text, from a 3,000-page sample.

Latin: 2,603 (89.1%)CJK: 255 (8.7%)Cyrillic: 26 (0.9%)Arabic: 23 (0.8%)Devanagari: 16 (0.5%)
  • Latin2,603 89.1%
  • CJK255 8.7%
  • Cyrillic26 0.9%
  • Arabic23 0.8%
  • Devanagari16 0.5%
Table view
ScriptPagesshare
Latin2,60389.1%
CJK2558.7%
Cyrillic260.9%
Arabic230.8%
Devanagari160.5%

Top-level domains

Share of the queue by TLD.

.com: 1,156,933 (48.7%).org: 529,100 (22.3%).edu: 83,612 (3.5%).io: 61,443 (2.6%).net: 57,952 (2.4%).other: 486,117 (20.5%)
  • .com1,156,933 48.7%
  • .org529,100 22.3%
  • .edu83,612 3.5%
  • .io61,443 2.6%
  • .net57,952 2.4%
  • .other486,117 20.5%
Table view
TLDURLsshare
.com1,156,93348.7%
.org529,10022.3%
.edu83,6123.5%
.io61,4432.6%
.net57,9522.4%
.other486,11720.5%

Biggest domains in the queue

Registrable domain, so every *.wikipedia.org language edition counts once. A tall bar here is a domain crowding out the rest of the crawl.

wikipedia.orgwikipedia.org: 90,175 — 3.8% of queue90,175tumblr.comtumblr.com: 80,039 — 3.4% of queue80,039blogspot.comblogspot.com: 60,951 — 2.6% of queue60,951stackexchange.comstackexchange.com: 58,270 — 2.5% of queue58,270github.comgithub.com: 36,390 — 1.5% of queue36,390wiktionary.orgwiktionary.org: 26,483 — 1.1% of queue26,483wikisource.orgwikisource.org: 18,491 — 0.8% of queue18,491diesel.comdiesel.com: 17,108 — 0.7% of queue17,108wordpress.comwordpress.com: 16,967 — 0.7% of queue16,967google.comgoogle.com: 15,043 — 0.6% of queue15,043iana.orgiana.org: 14,895 — 0.6% of queue14,895stanford.edustanford.edu: 14,309 — 0.6% of queue14,309wikimedia.orgwikimedia.org: 14,044 — 0.6% of queue14,044co.ukco.uk: 13,292 — 0.6% of queue13,292wikiquote.orgwikiquote.org: 13,274 — 0.6% of queue13,274
Table view
DomainQueued URLsshare
wikipedia.org90,17518.4%
tumblr.com80,03916.3%
blogspot.com60,95112.4%
stackexchange.com58,27011.9%
github.com36,3907.4%
wiktionary.org26,4835.4%
wikisource.org18,4913.8%
diesel.com17,1083.5%
wordpress.com16,9673.5%
google.com15,0433.1%
iana.org14,8953.0%
stanford.edu14,3092.9%
wikimedia.org14,0442.9%
co.uk13,2922.7%
wikiquote.org13,2742.7%

Page size distribution

Characters of extracted text, from the sample.

0: 77770<500: 103103<500500–2k: 351351500–2k2k–5k: 7847842k–5k5k–20k: 1,3701,3705k–20k20k–100k: 28728720k–100k100k+: 2828100k+
Table view
SizePagesshare
0772.6%
<5001033.4%
500–2k35111.7%
2k–5k78426.1%
5k–20k1,37045.7%
20k–100k2879.6%
100k+280.9%

Why fetches fail

Last recorded reason, across 200,000 flagged URLs.

4xx not found4xx not found: 108,109108,109timeouttimeout: 39,58039,580link bomblink bomb: 19,55419,554otherother: 7,3987,398not HTMLnot HTML: 7,2497,2495xx server error5xx server error: 5,9145,914DNS failureDNS failure: 4,9834,983TLS errorTLS error: 3,9953,995connection refused/resetconnection refused/reset: 2,3962,396tarpit / hard timeouttarpit / hard timeout: 822822
Table view
ReasonURLsshare
4xx not found108,10954.1%
timeout39,58019.8%
link bomb19,5549.8%
other7,3983.7%
not HTML7,2493.6%
5xx server error5,9143.0%
DNS failure4,9832.5%
TLS error3,9952.0%
connection refused/reset2,3961.2%
tarpit / hard timeout8220.4%

Growth over time

One point per snapshot. All three series are URL counts on a single axis.

0694.1k1.4M2.1MQueue pending: 2,051,063Pages archived: 1,493,179HTML archived: 1,411,783Oct 15Nov 15
  • Queue pending2,051,063 41.4%
  • Pages archived1,493,179 30.1%
  • HTML archived1,411,783 28.5%

Averaging 5,885 pages/day archived across 31.0 days of snapshots.

Table view
SnapshotArchivedQueueHTMLDB
2026-11-15 04:15 1,493,179 2,051,063 1,411,783 64.4 GB
2026-11-14 04:15 1,492,756 2,051,471 1,411,351 64.3 GB
2026-11-13 04:15 1,492,386 2,051,829 1,410,973 64.3 GB
2026-11-12 04:15 1,491,923 2,052,295 1,410,504 64.3 GB
2026-11-11 04:15 1,491,486 2,052,717 1,410,065 64.2 GB
2026-11-10 04:15 1,491,025 2,053,212 1,409,600 64.2 GB
2026-11-09 04:15 1,490,538 2,053,719 1,409,107 64.1 GB
2026-11-08 04:15 1,490,118 2,054,160 1,408,679 64.1 GB
2026-11-07 04:15 1,489,621 2,054,672 1,408,168 64 GB
2026-11-06 04:15 1,489,109 2,055,212 1,407,644 63.9 GB
2026-11-05 04:15 1,488,782 2,055,647 1,407,311 63.9 GB
2026-11-04 04:15 1,488,319 2,056,073 1,406,843 63.9 GB
2026-11-03 04:15 1,487,972 2,056,422 1,406,490 63.8 GB
2026-11-02 04:15 1,487,631 2,056,803 1,406,149 63.8 GB
2026-11-01 04:15 1,487,271 2,057,167 1,405,786 63.7 GB
2026-10-31 04:15 1,486,941 2,057,571 1,405,456 63.7 GB
2026-10-30 04:15 1,486,643 2,057,934 1,405,157 63.6 GB
2026-10-29 04:15 1,486,241 2,058,372 1,404,749 63.6 GB
2026-10-28 04:15 1,486,008 2,058,843 1,404,513 63.5 GB
2026-10-27 04:15 1,485,777 2,059,319 1,404,281 63.5 GB
2026-10-26 04:15 1,485,579 2,059,650 1,404,082 63.5 GB
2026-10-25 04:15 1,485,427 2,059,946 1,403,929 63.4 GB
2026-10-24 04:15 1,485,066 2,060,234 1,403,568 63.4 GB
2026-10-23 04:15 1,474,577 2,064,803 1,392,940 62.8 GB
2026-10-21 04:15 1,440,388 2,072,190 1,358,262 60.9 GB
2026-10-20 04:15 1,423,270 2,075,380 1,340,929 60 GB
2026-10-19 04:15 1,405,053 2,078,114 1,322,659 59 GB
2026-10-18 04:15 1,386,131 2,082,263 1,303,684 58 GB
2026-10-17 04:15 1,368,790 2,000,080 1,286,052 57 GB
2026-10-16 04:15 1,357,080 2,000,018 1,274,131 56.4 GB
2026-10-15 04:15 1,310,749 2,000,101 1,227,318 54.1 GB

Snapshot generated in 30,441,451 ms. Sampled figures use 3,000 random pages. Schema v5, extractor v2.