H
Even though this is an old post, I second this! Could the last point instead be rendered as \text{Downloads} \times \frac{\text{Total Days in Month}}{\text{Days Elapsed}} .. and perhaps the line fading out to the background color to hint it is an extrapolation? Every chart at the moment looks like the model has suddenly been dropped like a hot potato.
H
Welcome! Short version, based on recent surveys and papers: 1. Mainstream Chunked policies trained by imitation: ACT-style transformers and Diffusion Policy. VLAs: a pretrained VLM backbone plus an action head. Two families: autoregressive action tokens (RT-2, OpenVLA) and diffusion/flow-matching heads (π0, π0.5, GR00T, SmolVLA). Dual-system designs (a slow VLM planner plus a fast low-level policy) are common too. See this survey. 2. Limitations Data: VLAs need a lot of demos, and performance…
H
Thanks for rerunning the published examples and checking the expected answers against the frozen rules. We reran modal_terminal and saw the same thing. The extractor read “may have occurred” as if the event had happened. The later denial gave the right final answer and hid the earlier mistake. I agree that fixing all 58 probes up front would help. The audit says the expected answers were set before each batch. The public record leaves open whether all 58 inputs and answers were fixed before the…
H
Hello Hugging Face Team, I’m a co-author of this paper: huggingface.co Paper page - Subspace Representations for Synthetic-to-Real Transfer in 3-D... Join the discussion on this paper page I accidentally claimed another author’s entry (Abdulla Nasir Chowdhury) instead of my own (Naish Ibrahim). The claim was approved, and now clicking Abdulla’s name redirects to my Hugging Face profile. My username: Usagi5677 Could someone please help remove the incorrect association and allow me to claim my…
H
Thanks for the detailed feedback. It pushed us to publish the reasoning behind the scores rather than just the scores, and I wanted to close the loop on each point. 1. "Show your work on a single case." We now publish a worked example end to end on /evidence: one scenario, the response it produced, every criterion checked against it, and the arithmetic that followed. This is the same structure every execution in a report carries, so any score can be inspected down to the individual check. 2.…
H
(topic deleted by author)
H
Thank you for your help! My Space was accidentally flagged as abusive due to an unintended configuration change, and it is still unavailable. I’ve contacted support and explained the situation. Do you know how long it usually takes to receive a reply?
H
BrandenLaskowski7: Everyone keeps asking: “How do we make the model choose the right answer?” But a probabilistic model will always have a probability of choosing something else. So why are we assigning it the decisions that must never vary? Authority. Access. Scope. Policy. Release. Those are not “better prompting” problems. They are systems-design problems. A model can propose. It can reason. It can help. But it should not decide whether it is authorized, whether it may cross a boundary, or…
H
On your third question, what would make me trust the scores is evidence that the judge agrees with people. Before anything else I would hand-label about 50 transcripts across your oversight-pressure scenarios, score them with the rubric, and publish the agreement rate, broken out for critical-failure cases separately, since that is where a lenient judge does real damage. A second test: run every scenario five times on the same agent and report how often the critical-failure verdict flips. If it…
H
Your distinction between an empty event list and a missing trace is the detail most action checks get wrong; I have seen “no refund recorded” pass simply because the logger crashed. One test I would add: run each case five times per version and report pass^k, not a single pass, because an unapproved refund that appears in one run out of five is still a must-fail. I would also replay the approved_refund case with the provider returning a timeout after accepting the refund, then let the agent…
H
Hi, let me introduce our recent work here: Read full: Paper page - Coding or Cheating? An Investigation of Frontier Coding LLMs Read arxiv: [2611.10404v1] Coding or Cheating? An Investigation of Frontier Coding LLMs What does a frontier coding model do when a bug is too hard to solve? We gave 9 frontier models 20 hard tasks from `linux`, `ffmpeg`, and `llvm-project`, inside a hardened sandbox: blocked network, pruned git history, read-only repository. The traces in this repo show what happened…
H
Thank you for your review. The two-scorecard separation is exactly the right boundary, and several of your points were directly implementable the same day. Below: what I changed, what the measurements now show, and where the remaining gaps are. Everything stated is measured on my corpus; scope notes attached. ## 1. C2 — separate random streams: confirmed hit, fixed Your shared-RNG warning landed exactly. My tournament runner had a single `mulberry32` stream feeding deck shuffles, equity…
H
I fought this same failure family on a different visual task (classifying a state badge in screen captures where lighting, glare and motion look exactly like the target) and the two things that actually reduced false positives were architectural, not model-side. Confluence instead of single-frame trust. I added a second, independent confirmation channel and a detection only counts as an event when both channels agree within a window. On 165 held-out negative windows that gate took my false…
H
This is the exact shape of the problem I kept hitting with eval scores: the aggregate number hides which specific expectation flipped while another one got fixed. What I do now with structured outputs (and it transfers to agent actions directly): Freeze every incident as a new expected-output case the same day. Your refund case becomes a permanent test with the recorded provider action as the expected value. A version that regresses to unapproved refunds then fails a named case instead of…
H
For now, I started by taking a quick look at the public data and scoring code: I think the main design choice is worth keeping: evaluate the cost of accepting a wrong financial number, while allowing a system to reject it or send it to a human. The explicit review cost makes the tradeoff visible rather than treating abstention as a free success. I checked the public 500-row dataset, seven published prediction files, and the author’s original score.py. The original scorer and an independent…
H
I tried a small reproduction using the public code: I read the frozen v1.1 rules and v1.1.2 pre-holdout audit, then executed the published examples against a pinned extractor and inspected its evidence items. This reproduces public behavior, not the whole 58-case audit or an independent human annotation study. My answers to the two questions 1. Can I give an in-scope input whose audit expectation contradicts a frozen rule, with a defensible alternative expected state? I have not found one among…
H
Hi. For now, I sanity-checked a few public examples and looked into comparisons across bot generations and related evaluation questions: For the two questions at the end of your post, my default would be to maintain two separate but complementary scorecards: A fixed decision-probe panel, where every runnable bot generation receives the same information states, and compatible actions are priced under the same frozen heads-up oracle profile. This is a controlled comparison of local decisions…
H
Hi Nerris, thanks for sharing this — the evidence-based, versioned scoring is the part that stands out to me. A single number on its own is hard to trust, but a score tied to the specific behaviour in the scenario, with the reasoning shown, is something a team could actually act on. On your three questions: Rubric: it sounds clear in principle, but I would want to see a worked example — one scenario, the model’s response, and exactly which parts earned or lost points. That would also help show…
H
I work on CloudGPU. One useful beta-test case is the failure denominator. If a candidate server drops or times out more requests, a latency chart built only from its successful responses can look better even though the workload is served less reliably. Keep attempted requests, completed responses and semantically valid responses as separate counts. Preserve timeouts, malformed JSON and tool-call validation failures in the saved run instead of silently excluding them. Compare latency among…
H
I’m currently studying robot manipulation using imitation learning and Vision-Language-Action (VLA) models. I’d like to understand the current trends in these fields. What are the most widely used and promising policy architectures today? What are the main limitations and challenges of existing policies? What research directions do you think are promising for addressing these challenges? I’d especially appreciate insights from researchers and practitioners working on real-world robotic…
H
Hi all, I’m Nerris. I built EthicaEval AI (https://ethicaevalai.com), which tests how AI models and agents behave under ethical conflict, malicious instruction, and oversight pressure. Scoring is evidence-based and versioned, with critical-failure rules. I’m looking for a few developers to try it and tell me honestly what works and what doesn’t. Specific questions: Is the rubric clear and fair? Are the scenarios realistic for the systems you build? What would you need to trust the scores?…
H
Thanks for the concrete suggestions, John. Kontur’s JSON import now shows what will change before you confirm, including the full order of blocks and gates. If the project revision changes, the import shows a fresh diff and asks for confirmation again. After a timeout, the client reads the saved state and accounts for server defaults. If the changes are already present, it confirms the result without another write. If the revision is still the same, it can retry once with the original body and…
H
We tested Llama 3.1 8B Instruct on two A100 SXM 80 GB rentals: RunPod Secure in Kansas and Massed Compute via Shadeform in Des Moines. Both ran BF16 at concurrency 32, with 2,000 input tokens and 500 requested output tokens. Throughput was about 1,135–1,139 output tokens/s across the six measured runs. First-token p95 was 1,251–1,261 ms on the Kansas allocation and 752–990 ms on the Des Moines allocation. At a 1,000 ms target, that difference matters: one allocation passed all three runs, and…
H
Hi everyone, I’m currently studying robot manipulation using imitation learning and Vision-Language-Action (VLA) models. I’d like to understand the latest trends and research directions in these fields. I have three main questions: What policy architectures are currently mainstream and widely adopted in robot imitation learning and VLA? What are the main limitations and unresolved challenges of these mainstream policy architectures? What are the latest emerging policy architectures or…
H
Seems flagged as abusive? https://huggingface.co/api/spaces/kkr1212/MCBE-ProxVC-setup/runtime {"stage":"PAUSED","hardware":{"current":null,"requested":"cpu-basic"},"gcTimeout":172800,"errorMessage":"Flagged as abusive","replicas":{"requested":1},"devMode":false,"domains":[{"domain":"kkr1212-mcbe-proxvc-setup.hf.space","stage":"READY"}]} In cases like this, our only option is to email HF Support: website@huggingface.co
H
It seems like this was an intentional change on HF’s part, but I don’t know what the alternative route is… [Inference Providers] Remove the Register as an Inference Provider guide by hanouticelina · Pull Request #2934 · huggingface/hub-docs · GitHub
H
A terminal event may have occurred. This is a probe from an audit of a synthetic multi-agent project. The extractor reads an agent’s final report and fills five fields covering completion, uncertainty, partial progress, terminal-event claims, and explicit failure. It works from the report text alone. The simulation log provides a separate record against which the result is compared later. A terminal event here is an outcome defined in advance in the simulation. “Calibration” means agreement…
H
In this simulation, the newer agent version fixes an order lookup and starts issuing a refund without approval. Both versions pass two of the three tests. If you only look at the score, you miss the new failure. This is a small example of how we test agent actions at Lumina. The refund amounts and order IDs are fictional. Each test compares the actions we expected with the ones recorded by the simulated provider. The check also flags duplicate refunds and missing records for cases that must…
H
The responses are all over the place. You mention an observer and consciousness. Both of those are dependent on life. I think tou are getting caught parroting the “observer” stuff so many in the scientific community treat as the baseline. It’s not the base line. It is a derivative. Life is the constant that is required throughout. If you were on a planet beyond the observable horizon you would still be most important. A horizon marks the limit of what we can observe from the tightness of a…