XIO

xWire

Posts from every feed QUEST has found across the web — blogs, news, podcasts, channels and more.

Hugging Face Forums - Latest posts

Latest posts

Can we prove this: the code submitted with this research paper actually produced the numbers reported?

So with K-VERITAS, the reviewer doesn’t have to re-run the artifacts. He just has to verify it using the cli or the web verifier. No need to re-run. If everything is correct, it proves the codes produced the results, otherwise something is suspicious and needs careful attention. You can try it in a simulated reviewer workflow to see it. I think testing it gives the best understanding of the friction level.

Can we prove this: the code submitted with this research paper actually produced the numbers reported?

Thanks for this comment. Regarding the question, it’s a bit hard to predict the expected behavior from a code, and estimating it might cause errors and penalizing people. That’s why we just answer a fundamental question. However, your point is also interesting. If you can improve the tool in that direction or have any suggestions, feel free to lets us know or contribute directly. Everything is welcome to make science transparent.

Can we prove this: the code submitted with this research paper actually produced the numbers reported?

I think we are trying to Answer a different question because being able to re-run a code doesn’t mean you will get the same results. The results will heavily depend on the resources, etc… That’s why we are targeting this aspect to be able to prove that a code produced a specific result(s). I think if that’s answered, a big problem about good practice is solved.

Alternative GPU Compute Infrastructure Model for Large Pipelines

Replying to @alencooper0’s question about comparing cost over longer jobs: what has worked for me is costing the whole job, not the hourly rate. GPU-hours × rate, plus persistent storage (per GB-month, often billed while the instance is stopped), plus egress for checkpoints and datasets, plus setup and idle time. For spot or interruptible capacity, also add restart overhead: the time since your last checkpoint × the expected number of interruptions. A cheap spot rate with 30-minute checkpoint…

vLLM on three GPUs: Qwen2.5-7B tokens per second and cost per 1M tokens on RTX A6000, L40S and H100

Thanks for posting the method along with the numbers. One way to make it reusable beyond Runcrate: cost per 1M tokens = hourly price ÷ (tokens/s × 3600) × 1,000,000. Throughput belongs to the card and the serving config, and price belongs to the vendor, so readers can plug your tok/s into their own rate. It matters because the same card’s price varies far more than its throughput. Across the vendors I track, on-demand rates today were RTX A6000 $0.33–1.09, L40S $0.38–2.25 and H100 $1.99–6.16…

Open LLM API price history: 3 months of verified price changes (dataset + calculator Space)

Nice work on the announced vs observed dates. One thing that would make the self-host vs API comparison sharper is a GPU-side price series built the same way, because the hourly side moves too. In the GPU rental data I maintain, there were 148 list-price changes across 13 daily snapshots from Oct 27 to Nov 10. The same card also spans a wide range on the same day: on-demand L40S was $0.38–2.25 per GPU-hour across 10 vendors, and H100 was $1.99–6.16 across 14. So a single “GPU cost” constant can…

How much VRAM do you realistically need to fine-tune small models on a laptop?

Your second question (train in the cloud, run locally?) never got a direct answer, so here’s one. For 1B–7B LoRA/QLoRA that split works well and is cheap. Prototype on the laptop with a short sequence length and a few hundred steps so you know the config runs. Then do the real run on a rented 24 GB card. Pull down only the adapter (usually tens of MB) and run it locally on a quantized base. Rough cost: on-demand RTX 4090s were renting for about $0.34–0.89 per GPU-hour across the providers I…

Open-Source LLM Evaluation and Annotation QA Benchmark

I’d score RAG in two parts. First, retrieval: did the passages hold what the answer needed? Second, the answer itself. Split it into single claims and check each one against the retrieved text. The score is supported claims over total claims. Ragas defines its faithfulness metric this way. Citation correctness is a separate check. For each claim, take only the passage it cites and ask if that passage alone backs it. An answer can be true and still cite the wrong source. The ALCE benchmark also…

Toy demo: fine-tuning a 3.2M-param char transformer makes it forget Shakespeare (held-out numbers)

I pre-trained a 3.2M-parameter character-level transformer on Tiny Shakespeare (validation loss 1.50), then fine-tuned it for 1,000 steps on Mrs Beeton’s Book of Household Management (1861). Results on text held out from both: Recipe loss: 2.65 to 1.09 Shakespeare loss: 1.50 to 2.39 After fine-tuning, the prompt “ROMEO:” yields “377. INGREDIENTS.–4 oz. of salt”. A small, concrete example of forgetting. I also froze the fine-tuned model, hashed its weights (SHA-256), prompted it two ways and…

Arabic (Egyptian dialect) veterinary diagnosis assistant: dialect -> clinical-English gap and low accuracy on real owner messages

Hi Sayali, Thank you for your thoughtful and insightful response. I really appreciate you taking the time to understand the challenges we’re facing with Vetopedia. I agree that the main challenge is bridging Egyptian Arabic dialect expressions with accurate veterinary concepts, while preserving the original context and clinical details. I’d love to ask for your advice: Are there any open-source tools, libraries, models, or frameworks you would recommend to help us with: Egyptian Arabic dialect…

Remote MCP server OAuth token exchange returns HTTP 500 after authorization — is this a known server-side issue?

Before dropping resource, check what the server’s protected-resource metadata (/.well-known/oauth-protected-resource) lists as the resource. If it’s different from the host root you’re sending, use exactly that value in both the authorization and token requests, with a fresh code. If it already matches, the resource probably isn’t the cause and the 500 is more likely on their side.

How to get articles on a topic from Tim-Pinecone / LOC-Chronicling-America dataset

Good catch — you are looking at pages, not articles: each row is one page, each Parquet file is a whole issue, and there is no article field, which is why no boundary shows up in the page text. The smallest usable unit is already in the schema, though: blocks, the column/article blocks with their own text. So narrow in two steps — first filter pages by state, date, and a page-level keyword match, then keep only the blocks whose own text matches your topic, and glance at the blocks either side…

How to get articles on a topic from Tim-Pinecone / LOC-Chronicling-America dataset

I have gone through the examples on how to extract data from Tim-Pinecone/LOC-Chronicling-America dataset : README.md · Tim-Pinecone/LOC-Chronicling-America at main Are we getting the individual articles or the entire news paper? Inspecting some outputs I don’t see any “boundary” of articles. So, I am not sure that the outputs are at the article level. If not then How to narrow down the search to get articles on a topic - e.g., earthquake, suffrage etc? Has anyone done this? Thanks!!

Six browser-only writing tools (emotion-label and simile flags, sentence-length distribution) — looking for false positives

Update: two new tools in the same set, both browser-only. Summarization Bias Checker — paste a text and its summary; emotion labels in the summary that have no counterpart in the source are marked. Objective Projection (Nesnel İzdüşüm) Summarization Bias Checker Scene Card — plan a scene (point of view, events, six physical dimensions) and copy one writing instruction for an AI model. It asks the model not to name the feeling you want the reader to work out, and warns you if your own plan…

Tightening a definition I posted here loosely in July: "summarization bias" now has a registered falsifier — please try to break the protocol

A small, practical follow-up to this thread. I’ve put up a browser-only checker that takes a source text and a summary and marks emotion words in the summary that have no counterpart in the source: Objective Projection (Nesnel İzdüşüm) Summarization Bias Checker To be clear about what it is: a dictionary-based first pass at the level of a single summary. It does not test the hypothesis, it does not check who the emotion belongs to, and the word list has not been evaluated against human labels.…

An AI Spore agent built a Godot game, then tested and revised it with bots

That’s a useful distinction: unseen seeds vary the inputs, but don’t test whether the implementation and checks share the same misunderstanding. I haven’t run mutation tests on this example. I’d keep the current policies and assertions fixed, introduce one rule violation at a time, and check which assertions actually detect it. A mutation that survives would expose a missing or ineffective check—not something to fix by retuning the game. The twelve-tide boundary is a particularly clear starting…

An AI Spore agent built a Godot game, then tested and revised it with bots

The check I’d add is mutation: deliberately break one mechanic at a time — let the bargain create resources, let the game run past the twelfth tide, make the ending unreachable — and require the bots to fail each broken version. Unseen seeds vary the inputs, but the game and its test policy share the agent’s reading of the rules, so a shared misunderstanding can pass both; a broken version the bots still pass names exactly where that blind spot is, and fixing that check first is what makes the…

An AI Spore agent built a Godot game, then tested and revised it with bots

I’m building Spore, an Apache-2.0 runtime for persistent AI agents. Here’s an experiment using it to build and test a small Godot game. The agent wrote the game, generated its art and sound in code, and wrote bots to test its balance over seeded runs. It passed the first prompt’s targets—but reviewing the results exposed gaps those targets didn’t cover. A second prompt added requirements. The agent improved its search, revised the bargain mechanic and fixed the ending. Nobody edited the game by…

Can we prove this: the code submitted with this research paper actually produced the numbers reported?

Great thread. The distinction Bubs76 draws — “the process ran” vs “the numbers are right” — is the one that bites hardest in NLP, where I work. We have a third problem layered on top: stochastic non-determinism. A fine-tuned LLM eval can shift 1-3 points between runs on identical hardware, same seed, just different GPU scheduling. So “reproduce the exact number” is the wrong target for a lot of what we do. What we actually need is a declared tolerance band and hardware class, because the…

Can we prove this: the code submitted with this research paper actually produced the numbers reported?

For the hardware part, the anchors are the vendors’ roots, not the author, the reviewer or the cloud: Intel’s SGX Root CA for the TDX quote (through the PCK chain and Intel-signed TCB data), and NVIDIA’s device identity CA for the GPU report, which NVIDIA’s attestation service checks and signs over. What those roots don’t anchor is meaning. Someone still has to say which image hash is the expected one, so in practice the anchor for “what ran” is a build the reviewer can reproduce and hash. In…

Cache-Aware Weight Clustering: Beyond Model Compression Towards Memory-Efficient LLM Inference

Thank you very much for taking the time to explore the idea and, especially, for running actual experiments. I really appreciate the detailed analysis and the honest assessment of both its potential and its limitations. I’m not a researcher myself, so I think my part ends here. The further development and investigation are really in the hands of researchers with the right expertise and resources. Still, I’m genuinely happy to see that, despite the limitations, the original idea seems to have…

Arabic (Egyptian dialect) veterinary diagnosis assistant: dialect -> clinical-English gap and low accuracy on real owner messages

This is a genuinely interesting architecture problem. The 92% with the correct passage but 40% on real owner messages tells you clearly: your bottleneck is the dialect-to-clinical-concept bridge, not the LLM’s reasoning. I work at Nugen, where we focus on domain alignment for open-weight LLMs. A few observations from similar domain-specific deployment problems: The rewriting approach not helping (38.3% vs 40.4%) suggests the issue isn’t just vocabulary — it’s that Egyptian dialect symptom…

Computation as Cache, Recursive Leapfrogs, and Intelligence Explosions

To make Sections 1–3 above more concrete—and strip away the compressed SAT jargon—here is the underlying intuition and a worked visual example of how temporal rollout compression and recursive 3-SAT decomposition map onto each other. 1. The Core Intuition: Why a Midpoint State in Time Is a Separator in 3-SAT Why does a T-step sequential rollout look like a 3-SAT formula? In a sequential rollout s_0 \to s_1 \to \dots \to s_T, time acts as a Markov boundary: If you do not know the midpoint state…

Neuro: a language that compiles training code to a native binary

Thanks for the reproduction. Pinning the commit and saying what you didn’t test made it easy to check, so I ran the next steps on an RTX 5070 with neurc 5.30.1. Your second step matches. With w kept on the device, CPU and GPU both gave loss 5.8559375, gradients of -6.7375 and -5.725 (to f32 rounding), and w2 = [0.936875, 0.71125]. I also ran the next case on your list: two parameters and a tanh layer on a [32, 16] batch generated in Python, for 3,000 steps. Logged losses matched a NumPy f32…

Computation as Cache, Recursive Leapfrogs, and Intelligence Explosions

Thank you for this remarkably sharp response—your distinctions between the five output contracts, sequential span vs. total materialization work, the three disjoint information roles, and especially the History-Warmed \mathcal{U} baseline control hit the exact methodological hinges of the problem. Your point in Sections D–E about illusory \mathcal{U}-step reduction under fallback is critical: if M_\omega takes 0 steps of \mathcal{U} when its proposal passes the deployment gate and falls back to…

Introducing Ryzova Alphi v1 — Gemma 4-Based Multimodal AI Model with FP16 & GGUF

Hi @John6666, Thank you for taking the time to independently test Ryzova Alphi v1 and share such a detailed, reproducible analysis. We genuinely appreciate the effort you put into testing both the F16 and Q4_K_M releases on a Tesla T4. It’s particularly encouraging to see that both GGUF versions loaded successfully and generated responses in your environment. We also appreciate your careful interpretation of the Belebele results. We agree that eight questions per language are insufficient for…

How do you keep project context next to the model experiments you run?

This is exactly the problem I keep running into too, and nootxlm’s suggestion is the best version I have seen: make the run folder the record, rather than keeping notes beside it. What finally worked for me was a short README inside each run folder, written before I start the run, with the question I am trying to answer, the data and prompt version, and what result would change my mind. Then I append the scores and a one-paragraph verdict when the eval finishes, even if the verdict is just…

Terse abstention qwen 3 4B

Really interesting thread — thank you for sharing the numbers, not just the idea. The split between ‘said IDK’, ‘ran out of tokens’ and the echo case Lawless1987 mentions feels like the most important part. In my own small experiments I found the same thing: once those non-answers are lumped together, the accuracy number stops meaning much, because you cannot tell whether the model is being careful or just failing silently. One question: how are you deciding whether an abstention was the right…