# ML Referee: agent quickstart > Independent feature evaluation for machine learning: assess candidate inputs, test bounded model repairs, and identify where further data or research could resolve uncertainty. Website: https://mlreferee.com/ Human-readable connection guide: https://mlreferee.com/agents Discovery metadata: https://mlreferee.com/agent-discovery.json Documentation index: https://mlreferee.com/llms.txt ## When to use Referee Use it when you have a measurable prediction or ranking outcome, a declared baseline, historical validation observations, and candidate features or repair proposals. Evaluate up to 1,000 candidates as one submitted family rather than orchestrating an ablation run for each feature. An ablation removes an input or component and measures the change. The three decisions are: which inputs merit development, which bounded repairs merit testing when a model degrades, and where more observations or focused experiments could add value. Referee provides evidence for these decisions, not automatic deployment authority. It is not a general chatbot evaluator, an autonomous model builder, or an unlabeled anomaly-discovery service. ## Connection and live discovery - Remote MCP endpoint: `https://api.mlreferee.com/mcp` - Transport: Streamable HTTP. - OAuth resource: `https://api.mlreferee.com` - Protected-resource metadata: https://api.mlreferee.com/.well-known/oauth-protected-resource - Use a supported client configured for the approved ML Referee OAuth flow. Do not assume that every MCP client can connect simply by entering the URL. - Public discovery is available without sign-in. Account operations and evaluations require authorization. Connectivity, authentication, beta approval, and available credits are separate checks. - Use MCP tool discovery and the current returned tool schemas as the authority for arguments. This guide is not a replacement schema or a claim of marketplace availability. Start with `describe_referee`, `list_verticals`, and `get_referee_readiness`. Follow the readiness result's next permitted action. Never ask the user to paste passwords, OAuth secrets, or access tokens into chat or support. ## Access and spending authority ML Referee uses Auth0 as its OAuth authorization server. The user signs in to Referee in the browser and grants a supported client access on their behalf. The client supplies the resulting access token with protected MCP calls; Referee checks issuer, audience, expiry and scopes, then account access and credits. Separate registrations serve ChatGPT, hosted Claude, Codex and Copilot in VS Code. Customers do not need an Auth0 administrator account or their own OAuth application. See the [authorization guide and client test status](https://github.com/scarrico/mlreferee/blob/main/docs/authentication.md). Hosted Claude web sign-in and authenticated discovery/readiness passed; refresh issuance passed, while its actual refresh exchange and evaluation workflow remain unverified. Claude Code authentication and directory approval are separate. For a runnable local starting point, use the [public practice examples](https://github.com/scarrico/mlreferee/blob/main/examples/README.md). Choose the [Codex guide](https://github.com/scarrico/mlreferee/blob/main/docs/codex.md) or [GitHub Copilot in VS Code guide](https://github.com/scarrico/mlreferee/blob/main/docs/copilot.md), then follow the [shared workflow](https://github.com/scarrico/mlreferee/blob/main/docs/coding-agents.md) for local preparation, read-only readiness checks, permission to upload, and approval of the actual credit quote. Start with the synthetic fixture or the licensed public bike-demand dataset. Do not substitute private data, upload the reserved holdout, or treat local preparation as permission to submit an evaluation. A supported authenticated connection is still required for hosted operations. The Codex CLI rehearsal completed a synthetic evaluation and report-text verification through its managed MCP connection; a complete model-driven conversation remains unverified. Copilot in VS Code completed a model-driven synthetic workflow on macOS using the OAuth helper and local file adapter: exact-file upload, valid preflight, an approved one-credit evaluation, report/receipt retrieval and locally computed report-text SHA-256 verification returning `digest_match`. Use the [complete Copilot configuration](https://github.com/scarrico/mlreferee/blob/main/docs/copilot.md#quick-start-the-complete-tested-setup). Native Copilot refresh, new-account onboarding and other platforms remain unverified. Follow each guide’s tested versions. Use the adapter to read the configured approved file rather than reconstructing CSV in a tool argument, and stop if its returned fingerprint check fails. The adapter is local integration code; discover hosted tools from the connected server. Use the actual quote for cost and obtain approval before submitting. After authorized sign-in, use `get_credits` for the account balance. `get_referee_usage` reports aggregate service activity; it is not a balance check. If beta access is needed, use `request_beta_access` with: - `lane`: `referee`, `repair`, or `both`. - `use_case`: `evaluate_candidate_features`, `repair_degraded_features`, `both`, or `explore_fit`. - `idempotency_key`: a new unique key for this request, reused only when retrying the same request. Follow the live schema's format requirements. Approval is manual, not immediate. A request cannot grant credits or choose an amount. Keep the returned request ID; use `get_beta_request_status` when the requester asks for a status check, not an indefinite approval-polling loop. Humans can also request access at https://mlreferee.com/beta. Paid purchasing is not live. Read https://mlreferee.com/beta for current credit conditions and expiration. Before a billable submission, obtain authority for the evaluation or a bounded evaluation budget. A credit balance is not permission to spend it. Do not bypass access gates or request payment details. ## Before any upload Confirm the user's authority to submit the data. The beta is for adults 18+ doing business or professional research. Use only synthetic, non-personal, or genuinely anonymized evaluation data. Regulated data, identifiable personal data, and pseudonymized person-level records are prohibited. Ordinary account and support contact details are permitted in their respective channels, not as evaluation data. Read https://mlreferee.com/acceptable-use and https://mlreferee.com/privacy. A supported profile does not waive these restrictions. If data eligibility is unclear, stop and ask before uploading. Choose a profile using `list_verticals`, then request `get_input_schema` for that profile. For repair, request `get_repair_schema`. Follow the returned schema and constraints, including the objective, baseline, complete candidate family, row identity, timing, and grouping requirements. Features must use information available when the prediction was made. Timing checks can flag some leakage; they do not rule out every form. ## Prepare, preflight, and quote the appropriate route ### Feature family: 1–32 candidates Use `upload_private_dataset`, then `preflight_dataset`. Resolve validation errors before requesting `quote_audit` with the upload ID and declared contract. ### Feature family: 33–1,000 candidates Use `upload_campaign_shard` for shards of at most 32 candidates each. Keep the same aligned rows and baseline across shards, following the live schema. Use `quote_campaign` with the complete shard list. The campaign quote checks the combined submission and runs a timing pilot. Shards are a transport arrangement, not separate discoveries. Do not submit independent small evaluations and describe the combined winners as one family-wide evaluation. One evaluation does not mean one MCP call. ### Bounded model repair Use `upload_repair_matrix` for one complete repair matrix, then `preflight_repair` and `quote_repair`. Do not split the repair matrix into independent shards. Read the returned contract for required stable/degraded periods and repair inputs. Repair evidence requires prospective confirmation on new labeled observations. ## Submit only after reviewing the quote Preflight is free. Read the returned quote for credit cost, execution estimates, and applicable limits; do not infer a fixed one-credit price from candidate count alone. Check the account balance and remaining human-authorized budget. All routes submit through `submit_audit`, using the returned `quote_id` and an idempotency key. Reuse that key for retries of the same submission; do not accidentally create another evaluation. Stop if authorization is missing, credits are insufficient, or the quote exceeds the approved scope or budget. Do not silently relax the evaluation design, omit candidates, or change data restrictions to make a failed preflight pass. Explain the issue and seek clarification when the intended question would change. ## Retrieve and explain the evidence Use `get_audit_status` with bounded polling according to the client's workflow, then `get_audit_result` when ready. Report failure or expiration rather than promising a missing artifact. Save the report promptly to an authorized destination. Inputs expire within short windows, up to 24 hours from upload; some upload types default to two hours. Reports expire with their associated inputs, not a fixed period after job completion. Cleanup can lag during processing or downtime. Account, credit, audit, and verification metadata have separate retention rules, and backup copies may persist. See Privacy for the full current description. Use `get_audit_receipt` and `verify_audit_report` for supported report-integrity checks. A matching fingerprint establishes consistency with the issued artifact, not correctness or safety of its conclusions. Use `delete_upload` or `delete_audit` only when deletion is authorized; these actions do not promise deletion of all account or verification records. Explain results in the declared data, baseline, objective, and validation scope: - Supported incremental evidence can justify further model development, not a guarantee of future performance or deployment readiness. - An unresolved or null result does not prove a feature is always useless. Identify whether more observations, broader coverage, or a better-defined candidate could help resolve the question. - Prioritize follow-up in proportion to potential value; do not promise more data will turn a result positive. - Downstream models still need their own validation. Referee does not certify safety, select financial actions, or decide deployment. - Do not claim protection against undisclosed prior adaptive searches. ## Public evidence and help - Technical workflow: https://mlreferee.com/feature-evaluation-machine-learning - Accessible introduction: https://mlreferee.com/prediction-models-explained - Aggregate AI-safety research case study and limitations: https://mlreferee.com/case-study-ai-safety - Beta conditions: https://mlreferee.com/beta - Support and privacy requests: https://mlreferee.com/support - Terms: https://mlreferee.com/terms - Privacy: https://mlreferee.com/privacy Support is available without login. Do not send credentials, tokens, secrets, or sensitive datasets. ML Referee is independently operated; references to client or identity providers do not imply endorsement or partnership.