# Every Eval Ever Schema Review

> Portable context for reviewing the aggregate and instance-level JSON schemas.

Generated: 2026-07-17T01:50:41+00:00

## Aggregate evaluation

- Schema id: `aggregate`
- Version: `0.2.2`
- Source: https://raw.githubusercontent.com/evaleval/every_eval_ever/main/every_eval_ever/schemas/eval.schema.json
- Current-version comments: 79
- Prior-version comments retained: 0

### Schema outline

- `$` (object): Schema for storing and validating LLMs evaluation data, including model configuration, prompts, instances, Output, and evaluation metrics
- `schema_version` (string, required): Version of the schema used for this evaluation data
- `evaluation_id` (string, required): Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format
- `evaluation_timestamp` (string): Timestamp for when the evaluation was run
- `retrieved_timestamp` (string, required): Timestamp for when this record was created - using Unix Epoch time format
- `source_metadata` (object, required): Metadata about the source of the leaderboard data
- `source_metadata.source_name` (string): Name of the source (e.g. title of the source leaderboard or name of the platform used for the evaluation).
- `source_metadata.source_type` (string, required): Whether the data comes from a direct evaluation run or from documentation
- `source_metadata.source_organization_name` (string, required): Name of the organization that provides the data
- `source_metadata.source_organization_url` (string): URL for the organization that provides the data
- `source_metadata.source_organization_logo_url` (string): URL for the Logo for the organization that provides the data
- `source_metadata.evaluator_relationship` (string, required): Relationship between the evaluator and the model
- `source_metadata.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `eval_library` (object, required): Evaluation library/framework used to run the evaluation
- `eval_library.name` (string, required): Name of the evaluation library (e.g. inspect_ai, lm_eval, helm)
- `eval_library.version` (string, required): Version of the evaluation library. Use 'unknown' if the version is not available.
- `eval_library.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `model_info` (reference, required)
- `model_info.name` (string, required): Model name provided by evaluation source
- `model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models)
- `model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI')
- `model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI)
- `model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama).
- `model_info.inference_engine.name` (string): Name of the inference engine
- `model_info.inference_engine.version` (string): Version of the inference engine
- `model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results` (array, required): Array of evaluation results
- `evaluation_results[]` (object)
- `evaluation_results[].evaluation_result_id` (string): Stable identifier for this metric result inside an evaluation run. Recommended deterministic join key for instance-level records.
- `evaluation_results[].evaluation_name` (string, required): Name of the evaluation
- `evaluation_results[].source_data` (oneOf, required): Source of dataset for this evaluation: URL, HuggingFace dataset, or private/custom dataset.
- `evaluation_results[].source_data.oneOf[0]` (reference)
- `evaluation_results[].source_data.oneOf[0].dataset_name` (string, required): Name of the source dataset
- `evaluation_results[].source_data.oneOf[0].source_type` (constant, required)
- `evaluation_results[].source_data.oneOf[0].url` (array, required): URL(s) for the source of the evaluation data
- `evaluation_results[].source_data.oneOf[0].url[]` (string)
- `evaluation_results[].source_data.oneOf[0].additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].source_data.oneOf[1]` (reference)
- `evaluation_results[].source_data.oneOf[1].dataset_name` (string, required): Name of the source dataset
- `evaluation_results[].source_data.oneOf[1].source_type` (constant, required)
- `evaluation_results[].source_data.oneOf[1].hf_repo` (string): HuggingFace repository identifier
- `evaluation_results[].source_data.oneOf[1].hf_split` (string): One of train, val or test.
- `evaluation_results[].source_data.oneOf[1].samples_number` (integer): Number of samples in the dataset
- `evaluation_results[].source_data.oneOf[1].sample_ids` (array): Array of sample ids used for evaluation
- `evaluation_results[].source_data.oneOf[1].sample_ids[]` (string)
- `evaluation_results[].source_data.oneOf[1].additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].source_data.oneOf[2]` (reference)
- `evaluation_results[].source_data.oneOf[2].dataset_name` (string, required): Name of the source dataset
- `evaluation_results[].source_data.oneOf[2].source_type` (constant, required)
- `evaluation_results[].source_data.oneOf[2].additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].evaluation_timestamp` (string): Timestamp for when the evaluations were run
- `evaluation_results[].metric_config` (object, required): Details about the metric
- `evaluation_results[].metric_config.evaluation_description` (string): Description of the evaluation
- `evaluation_results[].metric_config.metric_id` (string): Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).
- `evaluation_results[].metric_config.metric_name` (string): Display name for the metric (e.g. Accuracy, F1-macro, pass@1).
- `evaluation_results[].metric_config.metric_kind` (string): Normalized metric family/type used for safe aggregation (e.g. accuracy, f1, auroc, rmse, mae, pass_rate, elo).
- `evaluation_results[].metric_config.metric_unit` (string): Unit of the metric values (e.g. proportion, percent, points, ms, tokens).
- `evaluation_results[].metric_config.metric_parameters` (object): Metric-specific parameters (e.g. {"k": 1} for pass@k).
- `evaluation_results[].metric_config.lower_is_better` (boolean, required): Whether a lower score is better
- `evaluation_results[].metric_config.score_type` (string): Type of score
- `evaluation_results[].metric_config.level_names` (array): Names of the score levels
- `evaluation_results[].metric_config.level_names[]` (string)
- `evaluation_results[].metric_config.level_metadata` (array): Additional Description for each Score Level
- `evaluation_results[].metric_config.level_metadata[]` (string)
- `evaluation_results[].metric_config.has_unknown_level` (boolean): Indicates whether there is an Unknown Level - if True, then a score of -1 will be treated as Unknown
- `evaluation_results[].metric_config.min_score` (number): Minimum possible score for continuous metric
- `evaluation_results[].metric_config.max_score` (number): Maximum possible score for continuous metric
- `evaluation_results[].metric_config.llm_scoring` (object): Configuration when LLM is used as scorer/judge
- `evaluation_results[].metric_config.llm_scoring.judges` (array, required): LLM judge(s) - single item for judge, multiple for jury
- `evaluation_results[].metric_config.llm_scoring.judges[]` (reference)
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info` (reference, required)
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.name` (string, required): Model name provided by evaluation source
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models)
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI')
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI)
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama).
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_engine.name` (string): Name of the inference engine
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_engine.version` (string): Version of the inference engine
- `evaluation_results[].metric_config.llm_scoring.judges[].model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].metric_config.llm_scoring.judges[].temperature` (number)
- `evaluation_results[].metric_config.llm_scoring.judges[].weight` (number): Weight of this judge's score in aggregation (used in jury)
- `evaluation_results[].metric_config.llm_scoring.judges[].additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].metric_config.llm_scoring.input_prompt` (string, required): Prompt template used for judging
- `evaluation_results[].metric_config.llm_scoring.aggregation_method` (string): How to aggregate scores when multiple judges
- `evaluation_results[].metric_config.llm_scoring.expert_baseline` (number): Expert/human baseline score for comparison
- `evaluation_results[].metric_config.llm_scoring.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].metric_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].score_details` (object, required): The score for the evaluation and related details
- `evaluation_results[].score_details.score` (number, required): The score for the evaluation
- `evaluation_results[].score_details.details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].score_details.uncertainty` (object): Quantification of uncertainty around the reported score
- `evaluation_results[].score_details.uncertainty.standard_error` (object): Standard error of the score estimate (SE_mean = standard_deviation / sqrt(num_samples))
- `evaluation_results[].score_details.uncertainty.standard_error.value` (number, required): The standard error value
- `evaluation_results[].score_details.uncertainty.standard_error.method` (string): How the standard error was computed (e.g. 'analytic', 'bootstrap', 'jackknife')
- `evaluation_results[].score_details.uncertainty.confidence_interval` (object): Lower and upper bounds for the metric at a given confidence level.
- `evaluation_results[].score_details.uncertainty.confidence_interval.lower` (number, required): Lower bound of the confidence interval
- `evaluation_results[].score_details.uncertainty.confidence_interval.upper` (number, required): Upper bound of the confidence interval
- `evaluation_results[].score_details.uncertainty.confidence_interval.confidence_level` (number): Confidence level (e.g. 0.95 for a 95% confidence interval)
- `evaluation_results[].score_details.uncertainty.confidence_interval.method` (string): How the confidence interval was computed
- `evaluation_results[].score_details.uncertainty.standard_deviation` (number): Standard deviation of the per-sample scores
- `evaluation_results[].score_details.uncertainty.num_samples` (integer): Number of samples used to compute the uncertainty estimates
- `evaluation_results[].score_details.uncertainty.num_bootstrap_samples` (integer): Number of bootstrap resamples used, if bootstrap method was applied
- `evaluation_results[].generation_config` (object)
- `evaluation_results[].generation_config.generation_args` (object): Parameters used to generate results - properties may vary by model type
- `evaluation_results[].generation_config.generation_args.temperature` (null | number): Sampling temperature
- `evaluation_results[].generation_config.generation_args.top_p` (null | number): Nucleus sampling parameter
- `evaluation_results[].generation_config.generation_args.top_k` (null | number): Top-k sampling parameter
- `evaluation_results[].generation_config.generation_args.max_tokens` (integer): Maximum number of tokens to generate
- `evaluation_results[].generation_config.generation_args.execution_command` (string): Command used to run the model to generate results
- `evaluation_results[].generation_config.generation_args.reasoning` (boolean): Whether reasoning orchain-of-thought was used to generate results
- `evaluation_results[].generation_config.generation_args.prompt_template` (string): Input prompt template for task (should contain agentic info if needed).
- `evaluation_results[].generation_config.generation_args.agentic_eval_config` (object): General configuration for agentic evaluations.
- `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools` (array): List of all available tools with their configurations
- `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[]` (object)
- `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[].name` (string): e.g. bash, calculator, ...
- `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[].description` (string)
- `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[].parameters` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].generation_config.generation_args.agentic_eval_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].generation_config.generation_args.eval_plan` (object): Plan (solvers) used in evaluation. Solvers are crucial parts of Inspect evaluations which can serve a wide variety of purposes like providing system prompts, prompt engineering, model generation or multi-turn dialog.
- `evaluation_results[].generation_config.generation_args.eval_plan.name` (string)
- `evaluation_results[].generation_config.generation_args.eval_plan.steps` (array): Array of evaluation plan steps (each step is a JSON-serialized string)
- `evaluation_results[].generation_config.generation_args.eval_plan.steps[]` (string)
- `evaluation_results[].generation_config.generation_args.eval_plan.config` (object): Additional parameters (key-value pairs, all values must be strings)
- `evaluation_results[].generation_config.generation_args.eval_limits` (object): Listed evaluation limits like time limit, message limit, token limit.
- `evaluation_results[].generation_config.generation_args.eval_limits.time_limit` (integer): Time limit for evaluation.
- `evaluation_results[].generation_config.generation_args.eval_limits.message_limit` (integer): Message limit for evaluation.
- `evaluation_results[].generation_config.generation_args.eval_limits.token_limit` (integer): Token limit for evaluation.
- `evaluation_results[].generation_config.generation_args.sandbox` (object)
- `evaluation_results[].generation_config.generation_args.sandbox.type` (string): Type of sandbox e.g. docker
- `evaluation_results[].generation_config.generation_args.sandbox.config` (string): Config file name/path e.g. compose.yaml. TODO or full config? Not sure based on the Inspect docs
- `evaluation_results[].generation_config.generation_args.max_attempts` (integer): Maximum number of submission attempts (default 1).
- `evaluation_results[].generation_config.generation_args.incorrect_attempt_feedback` (string): Feedback from the model after incorrect attempt.
- `evaluation_results[].generation_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `detailed_evaluation_results` (object): Reference to the evaluation results for all individual samples in the evaluation
- `detailed_evaluation_results.format` (string): Format of the detailed evaluation results
- `detailed_evaluation_results.file_path` (string): Path to the detailed evaluation results file
- `detailed_evaluation_results.hash_algorithm` (string): Hash algorithm used for checksum and sample_hash in instance-level data
- `detailed_evaluation_results.checksum` (string): Checksum value of the file
- `detailed_evaluation_results.total_rows` (integer): Total number of rows in the detailed evaluation results file
- `detailed_evaluation_results.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `$defs.judge_config` (object): Configuration for a single LLM judge/juror
- `$defs.judge_config.model_info` (reference, required)
- `$defs.judge_config.model_info.name` (string, required): Model name provided by evaluation source
- `$defs.judge_config.model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models)
- `$defs.judge_config.model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI')
- `$defs.judge_config.model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI)
- `$defs.judge_config.model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama).
- `$defs.judge_config.model_info.inference_engine.name` (string): Name of the inference engine
- `$defs.judge_config.model_info.inference_engine.version` (string): Version of the inference engine
- `$defs.judge_config.model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `$defs.judge_config.temperature` (number)
- `$defs.judge_config.weight` (number): Weight of this judge's score in aggregation (used in jury)
- `$defs.judge_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `$defs.model_info` (object): Complete model specification including basic information, technical configuration and inference settings
- `$defs.model_info.name` (string, required): Model name provided by evaluation source
- `$defs.model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models)
- `$defs.model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI')
- `$defs.model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI)
- `$defs.model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama).
- `$defs.model_info.inference_engine.name` (string): Name of the inference engine
- `$defs.model_info.inference_engine.version` (string): Version of the inference engine
- `$defs.model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `$defs.source_data_url` (object): URL source for the evaluation data
- `$defs.source_data_url.dataset_name` (string, required): Name of the source dataset
- `$defs.source_data_url.source_type` (constant, required)
- `$defs.source_data_url.url` (array, required): URL(s) for the source of the evaluation data
- `$defs.source_data_url.url[]` (string)
- `$defs.source_data_url.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `$defs.source_data_hf` (object): Details about HuggingFace dataset used for evaluation
- `$defs.source_data_hf.dataset_name` (string, required): Name of the source dataset
- `$defs.source_data_hf.source_type` (constant, required)
- `$defs.source_data_hf.hf_repo` (string): HuggingFace repository identifier
- `$defs.source_data_hf.hf_split` (string): One of train, val or test.
- `$defs.source_data_hf.samples_number` (integer): Number of samples in the dataset
- `$defs.source_data_hf.sample_ids` (array): Array of sample ids used for evaluation
- `$defs.source_data_hf.sample_ids[]` (string)
- `$defs.source_data_hf.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)
- `$defs.source_data_private` (object): Generic source data when neither URL array nor HuggingFace dataset applies
- `$defs.source_data_private.dataset_name` (string, required): Name of the source dataset
- `$defs.source_data_private.source_type` (constant, required)
- `$defs.source_data_private.additional_details` (object): Additional parameters (key-value pairs, all values must be strings)

### Feedback summary

# Review summary: Aggregate evaluation 0.2.2 / Entire schema

79 open comments and 0 resolved comments.

## Open issues

- **$** — I think having a bit of a convention/guidance about file naming would be helpful too - I personally don't like having spaces and dots in file paths - but thats a minor aesthetic preference _(Google Doc)_
- **$** — should be regex enforced if we do it _(Google Doc)_
- **$** — and you don't like it because it is always enforced to avoid escaping _(Google Doc)_
- **$** — We already enforce regex (well in the validation bot)? Why would it be an issue? (besides fixing existing files) _(Google Doc)_
- **$** — It isn't a problem. I meant
I agree, and we need to remember to also enforce it. _(Google Doc)_
- **$** — Are we happy with the one giving the data not being part of the submission?
otherwise would it be nice to have an email url and name and additional details to contact about the data? Like a data owner? _(Google Doc)_
- **$** — This will be part of the mandatory base-schema. _(Google Doc)_
- **$** — issue #144: in case we want to require adapters for contributors coming from a specific source, we need to add a required field like "ingestion_method" that will help us to distinguish that. _(Google Doc)_
- **schema_version** — We need to add type_of_deployment here and then indicate if its api or local model with subcategories - this has to be mandatory and in case its not closed weight then we need a mandatory link to weights location. If something special was used i.e. specific prompt template or other information they should be recorded in additional_implementation details which would be part of this field. It will ideally live at the root. _(Google Doc)_
- **schema_version** — This should be less flexible. Maybe strict typing? _(Google Doc)_
- **schema_version** — The difference between evaluation_id and metric_id - didn't quite make sense since they go together (i.e. exactly one evaluation id per metric id).


Also there was a discussion of using a 'dot schema' that never went anywhere? _(Google Doc)_
- **evaluation_id** — Needs a better docstring _(Google Doc)_
- **evaluation_id** — We have a specific format (I guess its just the file name) already - what else would we want? _(Google Doc)_
- **evaluation_id** — What about evals with the same model but different hyperparameters? Should we have the same evaluation_id for them? It will work that way in the current setup because we only use model_id in the evaluation_id. _(Google Doc)_
- **evaluation_id** — No, they will be unique for every new instance of an evaluation i.e. if its the exact same run repeated in another setup it will have new evaluation_id -- if its insufficient then we extend to include benchmark _(Google Doc)_
- **evaluation_id** — If it includes the timestamp - then it would be unique. We don't have a good human-semantic way to identify "slightly different params" unfortunately _(Google Doc)_
- **evaluation_id** — I think the evaluation_id is meant to be human consumed at the moment. Btw, I think its important to note that retrieved timestamp would be different if its the exact same record submitted twice or for example edited into a new field of schema. Should retrieved timestamp always remain the same - and should we ensure that the docstring reflect this? Otherwise we will have redundant replicas just because it changed. _(Google Doc)_
- **evaluation_timestamp** — There are two instances of this, not sure if this was intentional. _(Google Doc)_
- **evaluation_timestamp** — Is this typically available on 'scraped' evaluations? _(Google Doc)_
- **evaluation_timestamp** — It isn't often available - I think we want the one further down since it should be applied per specific metric (since they could be calculated at different times)
1 Reaktion insgesamt
Damian Stachura hat reagiert mit 👍 (2026-05-25 2:47 PM UTC) _(Google Doc)_
- **retrieved_timestamp** — why isn't it "int" then? (or number as json doesn't differentiate in and non int) _(Google Doc)_
- **source_metadata** — This was an unituitive name for people right?
the description as well, is it about the "source" of the "leaderboard"?
Maybe we need a data metadata, eval metadata and model metadata?
There are more source things below (source HF, data source etc.), what's the difference? why are they not organized together by some meaningful order (or top category) it is supposed to be something like
data:
source_type
source_org
...
eval:
source_type
source_HF bla bla (that is probably also data or model right? not eval)
... _(Google Doc)_
- **source_metadata** — I've noticed somebody get tripped up by it before - I think a consistent schema for the three parts can make sense.
Is there an edge case where the data and the eval-executor are different - do we care to cover it? (idk we technically do right now) _(Google Doc)_
- **source_metadata** — I think we need to refactor for simplicity. my original document had proposals for this. _(Google Doc)_
- **source_metadata** — I think its mostly the case that dataset creator and eval executor are different. Unless you meant the person creating the dataset and the eval itself. That is also possible, gsm8k is a subset created by the original generator and then subsequently another group cleaned it up and made a high quality subset. These are distinctions. Provenance says we have all this information. _(Google Doc)_
- **source_metadata** — Agreed _(Google Doc)_
- **source_metadata** — I think most leaderboards are created and operated by the same teams. So when you have an eval I would switch it to: dataset creator, eval creator, model creator (e.g. model releases) and third party. My only question is what to do when it is both. (e.g. openAI model, benchmark and evaluation). It can be replaced by name of evaluator, name of dataset creator name of model creator and name of contributor. This makes the other question a simple computation, give us the information of who contributes a lot which is nice.
The downside, it is a string, we know how hard those are. They would not be consistent... _(Google Doc)_
- **source_metadata.source_organization_name** — "data" for us is reports the evaluation? we should separate all those occurrences from "the data" on which evaluation is ran. I would call to use dataset and report evaluation and metadata. So we never use the word data? _(Google Doc)_
- **source_metadata.evaluator_relationship** — Why specifically to the model? evaluation to the dataset creators not interesting? _(Google Doc)_
- **eval_library** — Maybe eval_harness would be a better name than eval_library. _(Google Doc)_
- **eval_library** — I don't think either way makes a big difference. If it helps clarity then maybe sure. _(Google Doc)_
- **eval_library.name** — For evaluations run without a known/standard library, a lot of different things were submitted for this field -- just something to flag if we do want a standard here _(Google Doc)_
- **eval_library.name** — How about adding in the description. "For non standard code choose custom" ? _(Google Doc)_
- **eval_library.name** — Good idea _(Google Doc)_
- **eval_library.name** — I think unknown is better than custom. But this just needs to be flagged better in documentation and the docstring. _(Google Doc)_
- **model_info** — Is there a plan to support multi-model workflows? I think it's sensible to assume single model for most current cases, but thinking ahead what happens if someone wants to submit results for evals that include routers or delegation? _(Google Doc)_
- **model_info** — There is a plan for multi-model workflows and that would happen when we make the schema more 'modular' via assembly instead of right now. _(Google Doc)_
- **evaluation_results** — Theres no flag for suite level aggregates, entries like "Mean win rate" in helm_classic are structurally identical to the real benchmark entries below. An is_aggregate field would help, or even better would be capturing the hierarchy type. As it stands my pipeline has to guess this with heuristics. _(Google Doc)_
- **evaluation_results** — But isn't it always an aggregate? It is only not aggregate when instance level. _(Google Doc)_
- **evaluation_results** — I think the comment is as follows:
Take Global MMLU Lite, there are scores reported per language and then there are scores reported across all languages or as an average of all languages. Currently, there is no way to distinguish both except very hacky ways where. both are considered the same type of scores. They should not be, and there should be some way to reconcile both. I think its solved by some form of further nesting rather than a flag but that is the issue. _(Google Doc)_

## Hotspots

- `evaluation_results[].metric_config.metric_id`: 9 open comments
- `$`: 8 open comments
- `evaluation_id`: 6 open comments
- `source_metadata`: 6 open comments
- `evaluation_results`: 5 open comments
- `evaluation_results[].evaluation_timestamp`: 5 open comments
- `eval_library.name`: 4 open comments
- `schema_version`: 3 open comments
- `evaluation_timestamp`: 3 open comments
- `evaluation_results[].metric_config.score_type`: 3 open comments

## Instance-level evaluation

- Schema id: `instance`
- Version: `instance_level_eval_0.2.2`
- Source: https://raw.githubusercontent.com/evaleval/every_eval_ever/main/every_eval_ever/schemas/instance_level_eval.schema.json
- Current-version comments: 0
- Prior-version comments retained: 0

### Schema outline

- `$` (allOf): Schema for storing instance-level evaluation data for LLM evaluations, supporting single-turn, multi-turn, and agentic interactions
- `schema_version` (string, required): Version of the schema used for this instance data
- `evaluation_id` (string, required): Foreign key linking to the aggregate evaluation JSON. Must match the evaluation_id in the aggregate file.
- `model_id` (string, required): Identifier of the model in HuggingFace format (e.g. meta-llama/Llama-3.2-1B-Instruct)
- `evaluation_name` (string, required): The specific eval name, ideally unique (e.g. GSM8K, mmlu_physics). Primarily for display/filtering when evaluation_result_id is unavailable. If an evaluation run reports multiple aggregate results/metrics for the same sample, emit multiple instance records (one per aggregate result) rather than trying to attach a single instance record to many aggregate results.
- `evaluation_result_id` (string): Preferred deterministic foreign key to aggregate evaluation_results[].evaluation_result_id. This is intended to point to exactly one aggregate evaluation result; if a single underlying interaction/sample contributes to multiple aggregate results, emit multiple instance records with different evaluation_result_id values.
- `sample_id` (string, required): Question/sample identifier from the original dataset (e.g. gsm8k_0001)
- `sample_hash` (string | null): Hash of (input.raw + input.reference) to ensure comparison is between the same sample across models, in case sample_id is not consistent
- `interaction_type` (string, required): Type of interaction: single_turn for simple Q&A, multi_turn for conversations, agentic for tool-using agents
- `input` (object, required): Input data for the evaluation sample
- `input.raw` (string, required): The raw input as defined in the eval
- `input.formatted` (string | null): Includes chat template, CoT and all relevant modifications - basically the un-tokenized version of what the model sees
- `input.reference` (array, required): Ground truths or reference answers for comparison/scoring
- `input.reference[]` (string)
- `input.choices` (array | null): Optional list of choices for multiple-choice questions
- `input.choices[]` (string)
- `output` (object | null): Output data - only used for single_turn interactions, null for multi_turn/agentic
- `output.raw` (array, required): Complete model responses
- `output.raw[]` (string)
- `output.reasoning_trace` (array | null): Reasoning traces of the model if applicable (e.g. chain-of-thought tokens)
- `output.reasoning_trace[]` (string)
- `messages` (array | null): Full message transcript - used for multi_turn and agentic, null for single_turn. Contains all system, user, assistant, and tool messages in order.
- `messages[]` (object)
- `messages[].turn_idx` (integer, required): Index starting from 0 indicating the position in the conversation
- `messages[].role` (string, required): Role of the speaker (e.g. user, assistant, system, tool)
- `messages[].content` (string | null): The actual raw text for that particular turn (can be null if empty)
- `messages[].reasoning_trace` (string | null): Reasoning trace for that particular turn if applicable
- `messages[].tool_calls` (array | null): List of tool invocations for this turn, if applicable
- `messages[].tool_calls[]` (object)
- `messages[].tool_calls[].id` (string, required): Unique identifier for the tool call
- `messages[].tool_calls[].name` (string, required): Name of tool/function
- `messages[].tool_calls[].arguments` (object | null): Arguments used to call the tool (all values must be strings)
- `messages[].tool_call_id` (array | null): Reference to the tool call ID(s) this message has the content payload for.
- `messages[].tool_call_id[]` (string)
- `answer_attribution` (array, required): Information about how the answer was extracted from the model output
- `answer_attribution[]` (object)
- `answer_attribution[].turn_idx` (integer, required): Turn index in messages. 0 for single_turn
- `answer_attribution[].source` (string, required): Source of the extracted value (e.g. 'output.raw' or 'messages[turn_idx].content')
- `answer_attribution[].extracted_value` (string, required): Value that was extracted
- `answer_attribution[].extraction_method` (string, required): Method used to extract the value (e.g. regex, exact_match, llm_judge, custom)
- `answer_attribution[].is_terminal` (boolean, required): Whether this is the final answer (false if intermediate outputs are used to build up to a final answer)
- `evaluation` (object, required): Evaluation results and scoring data
- `evaluation.score` (number, required): Instance-level score
- `evaluation.is_correct` (boolean, required): Whether the final answer is correct
- `evaluation.num_turns` (integer | null): Number of turns in the interaction
- `evaluation.tool_calls_count` (integer | null): Count of tool calls across all turns in messages
- `token_usage` (object | null): Token usage for the model completion
- `token_usage.input_tokens` (integer, required): Total input tokens used
- `token_usage.output_tokens` (integer, required): Total output tokens used
- `token_usage.total_tokens` (integer, required): Total tokens used
- `token_usage.input_tokens_cache_write` (integer | null): Number of tokens written to the cache
- `token_usage.input_tokens_cache_read` (integer | null): Number of tokens retrieved from the cache
- `token_usage.reasoning_tokens` (integer | null): Number of tokens used for reasoning
- `performance` (object | null): Performance and latency metrics
- `performance.latency_ms` (number | null): Total latency in milliseconds
- `performance.time_to_first_token_ms` (number | null): Time to first token in milliseconds
- `performance.generation_time_ms` (number | null): Time for generation in milliseconds
- `performance.additional_details` (object | null): Additional performance metrics (key-value pairs, all values must be strings)
- `error` (string | null): Information about any error that occurred (e.g. timeout, refusal, API error)
- `metadata` (object | null): Optional metadata about the sample (e.g. subject, difficulty, tags)
- `$.allOf[0]` (field)
- `$.allOf[1]` (field)

### Feedback summary

# Review summary: Instance-level evaluation instance_level_eval_0.2.2 / Entire schema

0 open comments and 0 resolved comments.

## Open issues

No open comments in this section.

## Additional discussion context

Source: Slack discussion pasted for the schema review on 2026-07-16

Urgent documentation work

- Fix the README, synchronize it across GitHub, the EEE datastore, and the website, and clarify the guidance.
- Consider moving the contributor guide out of the README into a standalone document.
- Keep the README content exactly aligned across all published surfaces.

Schema and contributor guidance

- Add clearer meaning and naming conventions for evaluation_name.
- Explain why eval_library.name and eval_library.version matter, including their relevance to reasoning and reproducibility.
- Show the explicit fallback example: {"name": "unknown", "version": "unknown"}.
- Decide whether "unknown" is appropriate for every string and numeric value, or whether defaults should vary by type. Avoid silently treating one fallback policy as settled.
- State explicitly whether generated UUIDs must be UUID v4.

Repository references checked on 2026-07-16

- PR #136, "Fix LLM Stats evaluator provenance", is open and addresses issue #119. It infers evaluator_relationship from the underlying evaluator or score source versus the model developer, while retaining LLM Stats as the aggregator and preserving raw provenance. This is separate from the README work.
- Issue #119, "Inconsistency in evaluator_relationship", remains open. It asks for a clearer definition, possible renaming, or simplification of evaluator_relationship. Existing Google Doc comments already discuss this ambiguity, but PR #136 supplies a concrete proposed implementation.
- PR #116, "Canonical identity and schema upgrade tooling", remains open. Its README patch documents canonical identity backfill and audit commands; it does not establish a cross-platform documentation synchronization process or a UUID v4 policy.
- Issue #60 documented generated-file collisions where arc_easy and arc_challenge produced the same UUID-based filename. It is closed, but it supports documenting identifier-generation semantics explicitly.
- Issue #152 remains open for a stale documentation URL.
- PR #194 remains open and explicitly says the README is out of date after validator and metadata changes.

Coverage relative to the imported Google Doc comments

- Already represented: evaluation_name should be consistent across models; eval_library naming; using "unknown" rather than "custom" and documenting it better; evaluator_relationship ambiguity.
- Newly added through this context: cross-platform README synchronization, extracting the contributor guide, explicit eval_library fallback examples, type-specific unknown/default policy, UUID v4 policy, and current GitHub issue/PR status.

Links

- https://github.com/evaleval/every_eval_ever/pull/136
- https://github.com/evaleval/every_eval_ever/issues/119
- https://github.com/evaleval/every_eval_ever/pull/116
- https://github.com/evaleval/every_eval_ever/issues/60
- https://github.com/evaleval/every_eval_ever/issues/152
- https://github.com/evaleval/every_eval_ever/pull/194