Train the models
18,161 promptsFit the starting model and candidate models.
AI-safety research demonstration / Aegis 2.0
A better safety model. A clearer next move.
A model that screens AI prompts faces a tradeoff: catch unsafe requests without raising too many false alarms. We tested changes to a starting model on a public benchmark. Referee helped distinguish a supported improvement from an unanswered question about what made it work.
The payoff: evidence for the model, direction for the research.
The augmented model detected 64 additional unsafe prompts, with 11 additional false alerts. Its predictive improvement was supported; the value of one added component remained unresolved. That distinction helps a team decide what to advance and where more data could be worth the investment.
01 / From a promising result to a harder question
The first improvement gave us a reason to keep looking.
We began with a default baseline—a starting model—and nine candidate variants. Referee supported several predictive improvements. Separately, one preselected candidate detected 56 additional unsafe prompts, with 14 additional false alerts at its frozen alert threshold.
The follow-up tested an augmented model under an alternative representation: another way of presenting information to the model. We asked two questions. Did the complete model improve on the default baseline? And did the added representation contribute beyond a matched control using the same additional inputs?
Those questions need different comparisons. To keep them distinct, we fixed the candidates and the rules before examining the test outcomes.
Set the alert thresholds
4,541 promptsChoose when each model raises an alert, before examining test outcomes.
Declare the comparison
1,189 promptsSpecify the augmented candidate and matched controls without using test outcomes.
Evaluate held-out predictions
1,915 promptsSubmit fixed predictions to Referee on examples kept out of fitting and calibration.
One common test
Every candidate faced the same held-out examples, labels, baseline predictions, and evaluation contract.
No retuning after the result
The models and alert thresholds stayed fixed after test outcomes were examined.
The same benchmark, revisited
The follow-up reused the earlier experiment’s held-out examples. It checks the finding on the same data; it is not independent replication on fresh observations.
02 / What changed
The model improved. The extra detections came with extra false alerts.
Referee assessed predictive quality. The augmented model improved log loss—a measure of how well predicted probabilities match observed outcomes—by 0.03959, with a 95% uncertainty interval of [0.0269, 0.0526]. Referee supported that improvement over the default baseline within the declared evaluation.
The demonstration separately counted detections and false alerts. At the frozen thresholds, the augmented model caught 617 unsafe prompts rather than 553, while incorrectly flagging 57 safe prompts rather than 46. The table shows both sides of that tradeoff.
| Held-out measure | Default baseline | Augmented model | Difference |
|---|---|---|---|
| Unsafe prompts detected | 553 / 1,030 53.69% | 617 / 1,030 59.90% | +64 +6.21 points |
| Safe prompts flagged | 46 / 885 5.20% | 57 / 885 6.44% | +11 +1.24 points |
| Log loss (lower is better) | 0.47009 | 0.43050 | 0.03959 gain |
The two models’ alert thresholds were calibrated separately before held-out evaluation. These counts show their frozen operating points, not performance at a matched false-alert rate. Referee’s predictive judgment was based on log loss, not these counts.
03 / What the improvement did not settle
A better model does not explain every part of its success.
The comparison with the starting model answered the first question. The matched control addressed the second: was there evidence to credit the added representation itself?
The earlier strong model already achieved nearly the same improvement. A noise-control variant also retained the gain.
Referee did not establish an additional predictive benefit beyond that control. The model’s overall improvement remained supported, but this component’s incremental value remained unresolved. An improved total score was not enough to answer the narrower question.
That is useful direction, not a reason to throw the feature away. Researchers can refine the proposal or collect fresh labeled data to test its contribution. A team deciding where to spend its next week now has a specific uncertainty to investigate.
A separate check: was the fitting problem solved adequately?
Numerical certificates verified that the specified fitting problems were solved close to their global optima. That bounds the remaining optimization error for each fixed representation, training dataset, and objective. It separates fitting quality from the question of whether a representation improves predictions; it does not certify overall AI safety.
04 / The research decision
Keep the improvement. Focus the next investment.
The next step is to validate the supported model on fresh data and decide whether its detection–false-alert tradeoff suits the intended use. Alongside that work, the unresolved component offers a focused question for model refinement and further data collection.
This model is an example, not a ceiling: other models or representations could perform better or worse. The evidence is specific to this learner, baseline, dataset, metric, and evaluation design. It does not establish deployment readiness or safety certification, and it does not cover undisclosed prior adaptive searches.
That is Referee’s role: help researchers separate what they have evidence to pursue from what they still need to learn. The evaluation details below document the comparisons, checks, and limitations behind those decisions.
05 / Evaluation details
The evidence behind the story.
The complete public record continues here: aggregate comparisons, data preparation, reproducibility checks, and interpretation limits. Raw prompts, dataset files, predictions, model coefficients, and internal artifacts are not published.
Two comparisons, two different conclusions
| Comparison | Held-out log-loss gain | 95% uncertainty interval | Referee result |
|---|---|---|---|
| Augmented model versus default baseline | +0.03959 | [0.0269, 0.0526] | Predictive improvement supported |
| Added representation versus matched control | -0.00030 | [-0.0033, 0.0026] | No incremental evidence established |
Control context: Against the same default baseline, the earlier strong model achieved a log-loss gain of 0.03850; a noise-control variant achieved 0.04003. These are observed gains on the same held-out examples, not evidence that the small differences between these models are statistically established.
The first comparison supports the augmented model as a whole. The second asks whether the added representation contributed beyond a model with the same additional inputs. Its uncertainty interval includes both benefit and harm, so the individual contribution remains unresolved. “Not established yet” does not mean “useless.”
Data preparation and evaluation setup
The four partitions described above served distinct purposes: 18,161 prompts for fitting, 4,541 for threshold calibration, 1,189 for development and declaring the comparison, and 1,915 for the held-out test. Referee evaluated the fixed augmented-model and control predictions under one declared evaluation contract.
We used human prompt labels from the main dataset splits. We excluded empty, redacted, reconstruction-required, and refusal-augmentation records; normalized prompts for exact deduplication; removed cross-split overlaps; trained lightweight local classifiers; and calculated aggregate metrics. The default baseline is a simple local classifier, not a current commercial or NVIDIA safety model.
Exact-normalized prompt label conflicts in the source were 167 in training, 1 in test, and 0 in validation. The first occurrence was retained according to the pre-run rule rather than chosen for its label. These disagreements are a limitation. Near duplicates and shared source, user, or topic dependencies may remain.
Integrity and reproducibility checks
- Frozen baseline and candidate inputs were integrity-checked before evaluation.
- Intentional controls behaved consistently without changing the substantive candidate conclusions.
- Two complete fit-and-evaluate runs produced identical prediction artifacts, complete Referee results, and analytical summaries.
- A packaging-only cleanup was checked against the frozen test predictions and returned the same complete test audit; no model was refitted or selected again.
These repeatability checks concern the same experiment. They are not independent replication on fresh observations.
Limitations and interpretation
- The benchmark’s class mix is not an estimate of deployment prevalence.
- One real dataset cannot establish realized error-control performance because the truth of the candidate hypotheses is unknown.
- Annotation disagreements, near duplicates, and shared source, user, or topic dependencies may remain.
- Model-fitting and threshold-calibration uncertainty are not included in the reported conditional test intervals.
- Adaptive prior searches, future distribution shifts, adversarial adaptation, and undisclosed selection decisions are not covered.
- The augmented model is an example, not a performance ceiling. Other models or representations could perform better or worse.
Scope: a feature-quality opinion within the declared evaluation, not a safety certification, deployment recommendation, claim of the best model, or claim of superiority over other evaluation tools.
06 / Dataset and attribution
The public benchmark behind the experiment.
Source: NVIDIA, Nemotron Content Safety Dataset V2 / Aegis 2.0, used under CC BY 4.0.
Source revision: d86bb8bedff51d25ac834ab7838f1cc61acb7a2c. The license permits sharing and adaptation with attribution and change notices.
Attribution: Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien, AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails, NAACL 2025.
No ToxicChat data, vocabulary, predictions, or model coefficients were used.
Independent research demonstration
We filtered and deduplicated prompts, trained lightweight local classifiers, and publish only aggregate evaluation results. No raw prompts or dataset files are reproduced here. NVIDIA and the dataset authors did not sponsor or endorse this work.
Turn promising proposals into evidence you can build on.
Referee helps researchers determine which candidate improvements the evidence supports now—and which unresolved features deserve the next experiment.
Request beta access