Free, invite-only beta

Bring the prediction problem you want to improve.

Try independent evaluation for feature selection, model repair, or a better-informed data-collection plan. Tell us what your team predicts and what decision the evidence would change. Beta approval is manual; requesting access does not grant credits.

Request beta accessHow access and credits work →

01 / Why it is worth trying

Make your next evaluation count.

Bring one real prediction problem, the current baseline, historical evidence, and the complete candidate family. Referee returns a bounded feature-quality opinion and a focused next-research agenda.

  • Start from the objectiveJudge every candidate by whether it adds evidence for the outcome the model is meant to predict.
  • Set your evidence barChoose how demanding the statistical standard should be before evaluation begins.
  • Replace blind ablationEvaluate up to 1,000 candidates in one run instead of coordinating one separate run per feature.
  • Choose the next investmentUse the evidence to select inputs, evaluate a repair, or focus more data collection on a useful uncertainty.

02 / Beta flow

From connection to your first evaluation.

Connect and sign in

When the ML Referee connection is available to your account, connect through ChatGPT and complete Auth0 sign-in. Authentication identifies your account; it does not create credits. You can also request access with the form below before connecting.

Request beta access

Ask the connected ML Referee service to submit a bounded beta request, or use the website form below. A request cannot choose its own credit allocation.

Wait for human approval

Approval is manual and not immediate. The connected service can check the request status. ML Referee does not currently promise an approval email.

Check credits

After approval, ask ML Referee to report the available balance and readiness. Signup, connection, and a pending request do not grant credits.

Prepare and submit

Use only synthetic, non-personal, or genuinely anonymized data. Do not submit regulated data, identifiable personal data, or pseudonymized person-level records. Run the free preflight, review the quote, and submit only after the objective, baseline, candidate family, evidence bar, cost, and retention window are correct.

Current beta deadline: unused beta credits expire January 1, 2027 at 12:00 a.m. Pacific. An evaluation that reserved credits before the deadline may finish; a credit released after the deadline is revoked rather than returned to the expired balance.

03 / Evaluation costs

The cost is visible before the run begins.

Preflight and quoting do not consume credits. Submitting an accepted quote reserves its disclosed credits. Completed evaluations consume them; failed or stale runs release the reservation. Deleting a completed result does not restore a consumed credit.

  • Standard evaluationOne credit.
  • Small repair evaluationOne credit when the repair includes no more than 32 diagnostic features.
  • Larger campaigns and repairsThe greater of one credit per 32 candidates or diagnostic features, rounded up, and one credit per estimated worker minute, rounded up.
  • Measured before commitmentThe service times a pilot, adds a safety margin, and returns the required credits and estimated runtime before submission.

04 / Evidence in practice

See what a completed evaluation can tell you.

Public AI-safety case study

One supported improvement. One productive uncertainty.

Referee supported an augmented model’s predictive improvement over the experiment’s default baseline. Separately measured at independently calibrated, frozen thresholds, it detected 64 additional unsafe prompts with 11 additional false alerts. It also showed that an added feature’s incremental value remained unresolved—focusing the next experiment instead of forcing a premature conclusion.

Read the measured case study → See a forecasting repair example →
1,915held-out prompts
0.03959held-out log-loss gain
+64unsafe prompts detected
+11additional false alerts

05 / Request access

Bring one real prediction problem.

All fields are required. This beta is for adults 18 and older using the service for business or professional research. Do not include datasets, credentials, or sensitive information.

Do not submit datasets, credentials, proprietary feature values, or other sensitive information. This form is only for requesting beta access.

By submitting, you agree that ML Referee may use this information to review your request and contact you about beta access. Submission does not grant access or credits. Read the privacy notice.

06 / Boundaries

A useful agent can move quickly without becoming its own approver.

Operator controls access

Only the human operator can approve a request and set an allocation. An agent cannot approve itself, select its own credit amount, or bypass the balance.

User controls submission

The quote exposes the cost before submission. The evaluation proceeds only within the authenticated tenant’s available credits and declared contract.

Results guide research

The requester reviews supported and unresolved candidates, limitations, and next tests before constructing, tuning, or repairing a model.

Deployment remains separate

A Referee result is evidence within the submitted evaluation setup, not a guarantee of future performance or deployment readiness.

The current beta is free and card-free. Paid purchases, subscriptions, refunds, and agent-native payment are not available through the ML Referee ChatGPT connection.