# Review summary: Aggregate evaluation 0.2.2 / Entire schema

79 open comments and 0 resolved comments.

## Open issues

- **$** — I think having a bit of a convention/guidance about file naming would be helpful too - I personally don't like having spaces and dots in file paths - but thats a minor aesthetic preference _(Google Doc)_
- **$** — should be regex enforced if we do it _(Google Doc)_
- **$** — and you don't like it because it is always enforced to avoid escaping _(Google Doc)_
- **$** — We already enforce regex (well in the validation bot)? Why would it be an issue? (besides fixing existing files) _(Google Doc)_
- **$** — It isn't a problem. I meant
I agree, and we need to remember to also enforce it. _(Google Doc)_
- **$** — Are we happy with the one giving the data not being part of the submission?
otherwise would it be nice to have an email url and name and additional details to contact about the data? Like a data owner? _(Google Doc)_
- **$** — This will be part of the mandatory base-schema. _(Google Doc)_
- **$** — issue #144: in case we want to require adapters for contributors coming from a specific source, we need to add a required field like "ingestion_method" that will help us to distinguish that. _(Google Doc)_
- **schema_version** — We need to add type_of_deployment here and then indicate if its api or local model with subcategories - this has to be mandatory and in case its not closed weight then we need a mandatory link to weights location. If something special was used i.e. specific prompt template or other information they should be recorded in additional_implementation details which would be part of this field. It will ideally live at the root. _(Google Doc)_
- **schema_version** — This should be less flexible. Maybe strict typing? _(Google Doc)_
- **schema_version** — The difference between evaluation_id and metric_id - didn't quite make sense since they go together (i.e. exactly one evaluation id per metric id).


Also there was a discussion of using a 'dot schema' that never went anywhere? _(Google Doc)_
- **evaluation_id** — Needs a better docstring _(Google Doc)_
- **evaluation_id** — We have a specific format (I guess its just the file name) already - what else would we want? _(Google Doc)_
- **evaluation_id** — What about evals with the same model but different hyperparameters? Should we have the same evaluation_id for them? It will work that way in the current setup because we only use model_id in the evaluation_id. _(Google Doc)_
- **evaluation_id** — No, they will be unique for every new instance of an evaluation i.e. if its the exact same run repeated in another setup it will have new evaluation_id -- if its insufficient then we extend to include benchmark _(Google Doc)_
- **evaluation_id** — If it includes the timestamp - then it would be unique. We don't have a good human-semantic way to identify "slightly different params" unfortunately _(Google Doc)_
- **evaluation_id** — I think the evaluation_id is meant to be human consumed at the moment. Btw, I think its important to note that retrieved timestamp would be different if its the exact same record submitted twice or for example edited into a new field of schema. Should retrieved timestamp always remain the same - and should we ensure that the docstring reflect this? Otherwise we will have redundant replicas just because it changed. _(Google Doc)_
- **evaluation_timestamp** — There are two instances of this, not sure if this was intentional. _(Google Doc)_
- **evaluation_timestamp** — Is this typically available on 'scraped' evaluations? _(Google Doc)_
- **evaluation_timestamp** — It isn't often available - I think we want the one further down since it should be applied per specific metric (since they could be calculated at different times)
1 Reaktion insgesamt
Damian Stachura hat reagiert mit 👍 (2026-05-25 2:47 PM UTC) _(Google Doc)_
- **retrieved_timestamp** — why isn't it "int" then? (or number as json doesn't differentiate in and non int) _(Google Doc)_
- **source_metadata** — This was an unituitive name for people right?
the description as well, is it about the "source" of the "leaderboard"?
Maybe we need a data metadata, eval metadata and model metadata?
There are more source things below (source HF, data source etc.), what's the difference? why are they not organized together by some meaningful order (or top category) it is supposed to be something like
data:
source_type
source_org
...
eval:
source_type
source_HF bla bla (that is probably also data or model right? not eval)
... _(Google Doc)_
- **source_metadata** — I've noticed somebody get tripped up by it before - I think a consistent schema for the three parts can make sense.
Is there an edge case where the data and the eval-executor are different - do we care to cover it? (idk we technically do right now) _(Google Doc)_
- **source_metadata** — I think we need to refactor for simplicity. my original document had proposals for this. _(Google Doc)_
- **source_metadata** — I think its mostly the case that dataset creator and eval executor are different. Unless you meant the person creating the dataset and the eval itself. That is also possible, gsm8k is a subset created by the original generator and then subsequently another group cleaned it up and made a high quality subset. These are distinctions. Provenance says we have all this information. _(Google Doc)_
- **source_metadata** — Agreed _(Google Doc)_
- **source_metadata** — I think most leaderboards are created and operated by the same teams. So when you have an eval I would switch it to: dataset creator, eval creator, model creator (e.g. model releases) and third party. My only question is what to do when it is both. (e.g. openAI model, benchmark and evaluation). It can be replaced by name of evaluator, name of dataset creator name of model creator and name of contributor. This makes the other question a simple computation, give us the information of who contributes a lot which is nice.
The downside, it is a string, we know how hard those are. They would not be consistent... _(Google Doc)_
- **source_metadata.source_organization_name** — "data" for us is reports the evaluation? we should separate all those occurrences from "the data" on which evaluation is ran. I would call to use dataset and report evaluation and metadata. So we never use the word data? _(Google Doc)_
- **source_metadata.evaluator_relationship** — Why specifically to the model? evaluation to the dataset creators not interesting? _(Google Doc)_
- **eval_library** — Maybe eval_harness would be a better name than eval_library. _(Google Doc)_
- **eval_library** — I don't think either way makes a big difference. If it helps clarity then maybe sure. _(Google Doc)_
- **eval_library.name** — For evaluations run without a known/standard library, a lot of different things were submitted for this field -- just something to flag if we do want a standard here _(Google Doc)_
- **eval_library.name** — How about adding in the description. "For non standard code choose custom" ? _(Google Doc)_
- **eval_library.name** — Good idea _(Google Doc)_
- **eval_library.name** — I think unknown is better than custom. But this just needs to be flagged better in documentation and the docstring. _(Google Doc)_
- **model_info** — Is there a plan to support multi-model workflows? I think it's sensible to assume single model for most current cases, but thinking ahead what happens if someone wants to submit results for evals that include routers or delegation? _(Google Doc)_
- **model_info** — There is a plan for multi-model workflows and that would happen when we make the schema more 'modular' via assembly instead of right now. _(Google Doc)_
- **evaluation_results** — Theres no flag for suite level aggregates, entries like "Mean win rate" in helm_classic are structurally identical to the real benchmark entries below. An is_aggregate field would help, or even better would be capturing the hierarchy type. As it stands my pipeline has to guess this with heuristics. _(Google Doc)_
- **evaluation_results** — But isn't it always an aggregate? It is only not aggregate when instance level. _(Google Doc)_
- **evaluation_results** — I think the comment is as follows:
Take Global MMLU Lite, there are scores reported per language and then there are scores reported across all languages or as an average of all languages. Currently, there is no way to distinguish both except very hacky ways where. both are considered the same type of scores. They should not be, and there should be some way to reconcile both. I think its solved by some form of further nesting rather than a flag but that is the issue. _(Google Doc)_

## Hotspots

- `evaluation_results[].metric_config.metric_id`: 9 open comments
- `$`: 8 open comments
- `evaluation_id`: 6 open comments
- `source_metadata`: 6 open comments
- `evaluation_results`: 5 open comments
- `evaluation_results[].evaluation_timestamp`: 5 open comments
- `eval_library.name`: 4 open comments
- `schema_version`: 3 open comments
- `evaluation_timestamp`: 3 open comments
- `evaluation_results[].metric_config.score_type`: 3 open comments

## Additional discussion context

Source: Slack discussion pasted for the schema review on 2026-07-16

Urgent documentation work

- Fix the README, synchronize it across GitHub, the EEE datastore, and the website, and clarify the guidance.
- Consider moving the contributor guide out of the README into a standalone document.
- Keep the README content exactly aligned across all published surfaces.

Schema and contributor guidance

- Add clearer meaning and naming conventions for evaluation_name.
- Explain why eval_library.name and eval_library.version matter, including their relevance to reasoning and reproducibility.
- Show the explicit fallback example: {"name": "unknown", "version": "unknown"}.
- Decide whether "unknown" is appropriate for every string and numeric value, or whether defaults should vary by type. Avoid silently treating one fallback policy as settled.
- State explicitly whether generated UUIDs must be UUID v4.

Repository references checked on 2026-07-16

- PR #136, "Fix LLM Stats evaluator provenance", is open and addresses issue #119. It infers evaluator_relationship from the underlying evaluator or score source versus the model developer, while retaining LLM Stats as the aggregator and preserving raw provenance. This is separate from the README work.
- Issue #119, "Inconsistency in evaluator_relationship", remains open. It asks for a clearer definition, possible renaming, or simplification of evaluator_relationship. Existing Google Doc comments already discuss this ambiguity, but PR #136 supplies a concrete proposed implementation.
- PR #116, "Canonical identity and schema upgrade tooling", remains open. Its README patch documents canonical identity backfill and audit commands; it does not establish a cross-platform documentation synchronization process or a UUID v4 policy.
- Issue #60 documented generated-file collisions where arc_easy and arc_challenge produced the same UUID-based filename. It is closed, but it supports documenting identifier-generation semantics explicitly.
- Issue #152 remains open for a stale documentation URL.
- PR #194 remains open and explicitly says the README is out of date after validator and metadata changes.

Coverage relative to the imported Google Doc comments

- Already represented: evaluation_name should be consistent across models; eval_library naming; using "unknown" rather than "custom" and documenting it better; evaluator_relationship ambiguity.
- Newly added through this context: cross-platform README synchronization, extracting the contributor guide, explicit eval_library fallback examples, type-specific unknown/default policy, UUID v4 policy, and current GitHub issue/PR status.

Links

- https://github.com/evaleval/every_eval_ever/pull/136
- https://github.com/evaleval/every_eval_ever/issues/119
- https://github.com/evaleval/every_eval_ever/pull/116
- https://github.com/evaleval/every_eval_ever/issues/60
- https://github.com/evaleval/every_eval_ever/issues/152
- https://github.com/evaleval/every_eval_ever/pull/194