[{"id":"gdoc-a","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"I think having a bit of a convention/guidance about file naming would be helpful too - I personally don't like having spaces and dots in file paths - but thats a minor aesthetic preference","status":"open","source":"google_doc","anchor":"a","evidence":"[a][b][c][d][e]\n{[f]\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-b","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"should be regex enforced if we do it","status":"open","source":"google_doc","anchor":"b","evidence":"[a][b][c][d][e]\n{[f]\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-c","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"and you don't like it because it is always enforced to avoid escaping","status":"open","source":"google_doc","anchor":"c","evidence":"[a][b][c][d][e]\n{[f]\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-d","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"We already enforce regex (well in the validation bot)? Why would it be an issue? (besides fixing existing files)","status":"open","source":"google_doc","anchor":"d","evidence":"[a][b][c][d][e]\n{[f]\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-e","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"It isn't a problem. I meant\nI agree, and we need to remember to also enforce it.","status":"open","source":"google_doc","anchor":"e","evidence":"[a][b][c][d][e]\n{[f]\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-f","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"Are we happy with the one giving the data not being part of the submission?\notherwise would it be nice to have an email url and name and additional details to contact about the data? Like a data owner?","status":"open","source":"google_doc","anchor":"f","evidence":"[a][b][c][d][e]\n{[f]\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n    \"version\": \"0.2.2\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-g","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"This will be part of the mandatory base-schema.","status":"open","source":"google_doc","anchor":"g","evidence":"\"version\": \"0.2.2\",\n    \"type\": \"object\",\n    \"description\": \"Schema for storing and validating LLMs evaluation data, including model configuration, prompts, instances, Output, and evaluation metrics\",[g]\n    \"required\"[h]: [\n        \"schema_version\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-h","schema_id":"aggregate","schema_version":"0.2.2","path":"$","author":"Google Doc","body":"issue #144: in case we want to require adapters for contributors coming from a specific source, we need to add a required field like \"ingestion_method\" that will help us to distinguish that.","status":"open","source":"google_doc","anchor":"h","evidence":"\"type\": \"object\",\n    \"description\": \"Schema for storing and validating LLMs evaluation data, including model configuration, prompts, instances, Output, and evaluation metrics\",[g]\n    \"required\"[h]: [\n        \"schema_version\",\n        \"evaluation_id\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-i","schema_id":"aggregate","schema_version":"0.2.2","path":"schema_version","author":"Google Doc","body":"We need to add type_of_deployment here and then indicate if its api or local model with subcategories - this has to be mandatory and in case its not closed weight then we need a mandatory link to weights location. If something special was used i.e. specific prompt template or other information they should be recorded in additional_implementation details which would be part of this field. It will ideally live at the root.","status":"open","source":"google_doc","anchor":"i","evidence":"\"eval_library\",\n        \"evaluation_results\"\n    ],[i]\n    \"additionalProperties\": false,\n    \"properties\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-j","schema_id":"aggregate","schema_version":"0.2.2","path":"schema_version","author":"Google Doc","body":"This should be less flexible. Maybe strict typing?","status":"open","source":"google_doc","anchor":"j","evidence":"\"type\": \"string\",\n            \"description\": \"Version of the schema used for this evaluation data\"\n        },[j]\n[k]\n        \"evaluation_id\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-k","schema_id":"aggregate","schema_version":"0.2.2","path":"schema_version","author":"Google Doc","body":"The difference between evaluation_id and metric_id - didn't quite make sense since they go together (i.e. exactly one evaluation id per metric id).\n\n\nAlso there was a discussion of using a 'dot schema' that never went anywhere?","status":"open","source":"google_doc","anchor":"k","evidence":"\"description\": \"Version of the schema used for this evaluation data\"\n        },[j]\n[k]\n        \"evaluation_id\": {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-l","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_id","author":"Google Doc","body":"Needs a better docstring","status":"open","source":"google_doc","anchor":"l","evidence":"\"type\": \"string\",\n            \"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-m","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_id","author":"Google Doc","body":"We have a specific format (I guess its just the file name) already - what else would we want?","status":"open","source":"google_doc","anchor":"m","evidence":"\"type\": \"string\",\n            \"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-n","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_id","author":"Google Doc","body":"What about evals with the same model but different hyperparameters? Should we have the same evaluation_id for them? It will work that way in the current setup because we only use model_id in the evaluation_id.","status":"open","source":"google_doc","anchor":"n","evidence":"\"type\": \"string\",\n            \"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-o","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_id","author":"Google Doc","body":"No, they will be unique for every new instance of an evaluation i.e. if its the exact same run repeated in another setup it will have new evaluation_id -- if its insufficient then we extend to include benchmark","status":"open","source":"google_doc","anchor":"o","evidence":"\"type\": \"string\",\n            \"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-p","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_id","author":"Google Doc","body":"If it includes the timestamp - then it would be unique. We don't have a good human-semantic way to identify \"slightly different params\" unfortunately","status":"open","source":"google_doc","anchor":"p","evidence":"\"type\": \"string\",\n            \"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-q","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_id","author":"Google Doc","body":"I think the evaluation_id is meant to be human consumed at the moment. Btw, I think its important to note that retrieved timestamp would be different if its the exact same record submitted twice or for example edited into a new field of schema. Should retrieved timestamp always remain the same - and should we ensure that the docstring reflect this? Otherwise we will have redundant replicas just because it changed.","status":"open","source":"google_doc","anchor":"q","evidence":"\"type\": \"string\",\n            \"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-r","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_timestamp","author":"Google Doc","body":"There are two instances of this, not sure if this was intentional.","status":"open","source":"google_doc","anchor":"r","evidence":"\"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",\n            \"description\": \"Timestamp for when the evaluation was run\"","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T12:35:13+00:00"},{"id":"gdoc-s","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_timestamp","author":"Google Doc","body":"Is this typically available on 'scraped' evaluations?","status":"open","source":"google_doc","anchor":"s","evidence":"\"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",\n            \"description\": \"Timestamp for when the evaluation was run\"","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-t","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_timestamp","author":"Google Doc","body":"It isn't often available - I think we want the one further down since it should be applied per specific metric (since they could be calculated at different times)\n1 Reaktion insgesamt\nDamian Stachura hat reagiert mit 👍 (2026-05-25 2:47 PM UTC)","status":"open","source":"google_doc","anchor":"t","evidence":"\"description\": \"Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format\"\n        },[l][m][n][o][p][q]\n        \"evaluation_timestamp\"[r][s][t]: {\n            \"type\": \"string\",\n            \"description\": \"Timestamp for when the evaluation was run\"","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-u","schema_id":"aggregate","schema_version":"0.2.2","path":"retrieved_timestamp","author":"Google Doc","body":"why isn't it \"int\" then? (or number as json doesn't differentiate in and non int)","status":"open","source":"google_doc","anchor":"u","evidence":"},\n        \"retrieved_timestamp\": {\n            \"type\": \"string[u]\",\n            \"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-v","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata","author":"Google Doc","body":"This was an unituitive name for people right?\nthe description as well, is it about the \"source\" of the \"leaderboard\"?\nMaybe we need a data metadata, eval metadata and model metadata?\nThere are more source things below (source HF, data source etc.), what's the difference? why are they not organized together by some meaningful order (or top category) it is supposed to be something like\ndata:\nsource_type\nsource_org\n...\neval:\nsource_type\nsource_HF bla bla (that is probably also data or model right? not eval)\n...","status":"open","source":"google_doc","anchor":"v","evidence":"\"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },\n        \"source_metadata[v][w][x][y][z][aa]\": {\n            \"type\": \"object\",\n            \"description\": \"Metadata about the source of the leaderboard data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-w","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata","author":"Google Doc","body":"I've noticed somebody get tripped up by it before - I think a consistent schema for the three parts can make sense.\nIs there an edge case where the data and the eval-executor are different - do we care to cover it? (idk we technically do right now)","status":"open","source":"google_doc","anchor":"w","evidence":"\"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },\n        \"source_metadata[v][w][x][y][z][aa]\": {\n            \"type\": \"object\",\n            \"description\": \"Metadata about the source of the leaderboard data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-x","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata","author":"Google Doc","body":"I think we need to refactor for simplicity. my original document had proposals for this.","status":"open","source":"google_doc","anchor":"x","evidence":"\"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },\n        \"source_metadata[v][w][x][y][z][aa]\": {\n            \"type\": \"object\",\n            \"description\": \"Metadata about the source of the leaderboard data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-y","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata","author":"Google Doc","body":"I think its mostly the case that dataset creator and eval executor are different. Unless you meant the person creating the dataset and the eval itself. That is also possible, gsm8k is a subset created by the original generator and then subsequently another group cleaned it up and made a high quality subset. These are distinctions. Provenance says we have all this information.","status":"open","source":"google_doc","anchor":"y","evidence":"\"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },\n        \"source_metadata[v][w][x][y][z][aa]\": {\n            \"type\": \"object\",\n            \"description\": \"Metadata about the source of the leaderboard data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-z","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata","author":"Google Doc","body":"Agreed","status":"open","source":"google_doc","anchor":"z","evidence":"\"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },\n        \"source_metadata[v][w][x][y][z][aa]\": {\n            \"type\": \"object\",\n            \"description\": \"Metadata about the source of the leaderboard data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-aa","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata","author":"Google Doc","body":"I think most leaderboards are created and operated by the same teams. So when you have an eval I would switch it to: dataset creator, eval creator, model creator (e.g. model releases) and third party. My only question is what to do when it is both. (e.g. openAI model, benchmark and evaluation). It can be replaced by name of evaluator, name of dataset creator name of model creator and name of contributor. This makes the other question a simple computation, give us the information of who contributes a lot which is nice.\nThe downside, it is a string, we know how hard those are. They would not be consistent...","status":"open","source":"google_doc","anchor":"aa","evidence":"\"description\": \"Timestamp for when this record was created - using Unix Epoch time format\"\n        },\n        \"source_metadata[v][w][x][y][z][aa]\": {\n            \"type\": \"object\",\n            \"description\": \"Metadata about the source of the leaderboard data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ab","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata.source_organization_name","author":"Google Doc","body":"\"data\" for us is reports the evaluation? we should separate all those occurrences from \"the data\" on which evaluation is ran. I would call to use dataset and report evaluation and metadata. So we never use the word data?","status":"open","source":"google_doc","anchor":"ab","evidence":"\"source_organization_name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Name of the organization that provides the data[ab]\"\n                },\n                \"source_organization_url\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ac","schema_id":"aggregate","schema_version":"0.2.2","path":"source_metadata.evaluator_relationship","author":"Google Doc","body":"Why specifically to the model? evaluation to the dataset creators not interesting?","status":"open","source":"google_doc","anchor":"ac","evidence":"\"evaluator_relationship\": {\n                    \"type\": \"string\",\n                    \"description\": \"Relationship between the evaluator and the model[ac]\",\n                    \"enum\": [\n                        \"first_party\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ad","schema_id":"aggregate","schema_version":"0.2.2","path":"eval_library","author":"Google Doc","body":"Maybe eval_harness would be a better name than eval_library.","status":"open","source":"google_doc","anchor":"ad","evidence":"}\n        },\n        \"eval_library[ad][ae]\": {\n            \"type\": \"object\",\n            \"description\": \"Evaluation library/framework used to run the evaluation\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ae","schema_id":"aggregate","schema_version":"0.2.2","path":"eval_library","author":"Google Doc","body":"I don't think either way makes a big difference. If it helps clarity then maybe sure.","status":"open","source":"google_doc","anchor":"ae","evidence":"}\n        },\n        \"eval_library[ad][ae]\": {\n            \"type\": \"object\",\n            \"description\": \"Evaluation library/framework used to run the evaluation\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-af","schema_id":"aggregate","schema_version":"0.2.2","path":"eval_library.name","author":"Google Doc","body":"For evaluations run without a known/standard library, a lot of different things were submitted for this field -- just something to flag if we do want a standard here","status":"open","source":"google_doc","anchor":"af","evidence":"\"name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Name of the evaluation library (e.g. inspect_ai, lm_eval, helm)\"[af][ag][ah][ai]\n                },\n                \"version\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ag","schema_id":"aggregate","schema_version":"0.2.2","path":"eval_library.name","author":"Google Doc","body":"How about adding in the description. \"For non standard code choose custom\" ?","status":"open","source":"google_doc","anchor":"ag","evidence":"\"name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Name of the evaluation library (e.g. inspect_ai, lm_eval, helm)\"[af][ag][ah][ai]\n                },\n                \"version\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ah","schema_id":"aggregate","schema_version":"0.2.2","path":"eval_library.name","author":"Google Doc","body":"Good idea","status":"open","source":"google_doc","anchor":"ah","evidence":"\"name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Name of the evaluation library (e.g. inspect_ai, lm_eval, helm)\"[af][ag][ah][ai]\n                },\n                \"version\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ai","schema_id":"aggregate","schema_version":"0.2.2","path":"eval_library.name","author":"Google Doc","body":"I think unknown is better than custom. But this just needs to be flagged better in documentation and the docstring.","status":"open","source":"google_doc","anchor":"ai","evidence":"\"name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Name of the evaluation library (e.g. inspect_ai, lm_eval, helm)\"[af][ag][ah][ai]\n                },\n                \"version\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-aj","schema_id":"aggregate","schema_version":"0.2.2","path":"model_info","author":"Google Doc","body":"Is there a plan to support multi-model workflows? I think it's sensible to assume single model for most current cases, but thinking ahead what happens if someone wants to submit results for evals that include routers or delegation?","status":"open","source":"google_doc","anchor":"aj","evidence":"]\n        },\n        \"model_info\"[aj][ak]: {\n            \"$ref\": \"#/$defs/model_info\"\n        },","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ak","schema_id":"aggregate","schema_version":"0.2.2","path":"model_info","author":"Google Doc","body":"There is a plan for multi-model workflows and that would happen when we make the schema more 'modular' via assembly instead of right now.","status":"open","source":"google_doc","anchor":"ak","evidence":"]\n        },\n        \"model_info\"[aj][ak]: {\n            \"$ref\": \"#/$defs/model_info\"\n        },","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-al","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results","author":"Google Doc","body":"Theres no flag for suite level aggregates, entries like \"Mean win rate\" in helm_classic are structurally identical to the real benchmark entries below. An is_aggregate field would help, or even better would be capturing the hierarchy type. As it stands my pipeline has to guess this with heuristics.","status":"open","source":"google_doc","anchor":"al","evidence":"\"metric_config\",\n                    \"score_details\"\n                ],[al][am][an][ao][ap]\n                \"properties\": {\n                    \"evaluation_result_id\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-am","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results","author":"Google Doc","body":"But isn't it always an aggregate? It is only not aggregate when instance level.","status":"open","source":"google_doc","anchor":"am","evidence":"\"metric_config\",\n                    \"score_details\"\n                ],[al][am][an][ao][ap]\n                \"properties\": {\n                    \"evaluation_result_id\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-an","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results","author":"Google Doc","body":"I think the comment is as follows:\nTake Global MMLU Lite, there are scores reported per language and then there are scores reported across all languages or as an average of all languages. Currently, there is no way to distinguish both except very hacky ways where. both are considered the same type of scores. They should not be, and there should be some way to reconcile both. I think its solved by some form of further nesting rather than a flag but that is the issue.","status":"open","source":"google_doc","anchor":"an","evidence":"\"metric_config\",\n                    \"score_details\"\n                ],[al][am][an][ao][ap]\n                \"properties\": {\n                    \"evaluation_result_id\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ao","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results","author":"Google Doc","body":"What is the issue? As you say, why not do the right thing here, and allow nesting?","status":"open","source":"google_doc","anchor":"ao","evidence":"\"metric_config\",\n                    \"score_details\"\n                ],[al][am][an][ao][ap]\n                \"properties\": {\n                    \"evaluation_result_id\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ap","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results","author":"Google Doc","body":"we should do it, its just not done yet","status":"open","source":"google_doc","anchor":"ap","evidence":"\"metric_config\",\n                    \"score_details\"\n                ],[al][am][an][ao][ap]\n                \"properties\": {\n                    \"evaluation_result_id\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-aq","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_result_id","author":"Google Doc","body":"There was discussion about using a 'dot' schema for this before (and maybe incorporating the overall evaluation_id) - we should discuss\n1 Reaktion insgesamt\nHarsha hat reagiert mit 👍 (2026-05-20 5:28 PM UTC)","status":"open","source":"google_doc","anchor":"aq","evidence":"\"type\": \"string\",\n                        \"description\": \"Stable identifier for this metric result inside an evaluation run. Recommended deterministic join key for instance-level records.\"\n                    },[aq][ar]\n                    \"evaluation_name\": {\n                        \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ar","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_result_id","author":"Google Doc","body":"evaluation_result_id is unique currently for each instance of the result I think. Think we can combine the evaluation_id as a natural extension but should this be a hash we can match","status":"open","source":"google_doc","anchor":"ar","evidence":"\"type\": \"string\",\n                        \"description\": \"Stable identifier for this metric result inside an evaluation run. Recommended deterministic join key for instance-level records.\"\n                    },[aq][ar]\n                    \"evaluation_name\": {\n                        \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-as","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_name","author":"Google Doc","body":"Maybe this makes sense over the metric_name + metric_id -- from recent reviews I think we mostly need this (human identifier) + metric_kind (for various types of grouping).\n\n\nEvaluation Name should be consistent across models to allow for grouping?","status":"open","source":"google_doc","anchor":"as","evidence":"\"type\": \"string\",\n                        \"description\": \"Name of the evaluation\"\n                    },[as][at]\n                    \"source_data\": {\n                        \"description\": \"Source of dataset for this evaluation: URL, HuggingFace dataset, or private/custom dataset.\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-at","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_name","author":"Google Doc","body":"I am in favour of removing evaluation_name completely. Human readable is leading to insane ambiguity and options for mistakes.","status":"open","source":"google_doc","anchor":"at","evidence":"\"type\": \"string\",\n                        \"description\": \"Name of the evaluation\"\n                    },[as][at]\n                    \"source_data\": {\n                        \"description\": \"Source of dataset for this evaluation: URL, HuggingFace dataset, or private/custom dataset.\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-au","schema_id":"aggregate","schema_version":"0.2.2","path":"$ref","author":"Google Doc","body":"Need to refine this to specifically call out 'dataset'. In case this is not available, it should be called out and should reduce the transparency score of the json.","status":"open","source":"google_doc","anchor":"au","evidence":"}\n                        ]\n                    },[au]\n                    \"evaluation_timestamp\": {\n                        \"type\": \"string\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-av","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_timestamp","author":"Google Doc","body":"I'd keep this one","status":"open","source":"google_doc","anchor":"av","evidence":"\"type\": \"string\",\n                        \"description\": \"Timestamp for when the evaluations were run\"\n                    },[av][aw][ax][ay][az]\n                    \"metric_config\": {\n                        \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-aw","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_timestamp","author":"Google Doc","body":"Its a repetition of the exact same thing - think we should at least rename it.","status":"open","source":"google_doc","anchor":"aw","evidence":"\"type\": \"string\",\n                        \"description\": \"Timestamp for when the evaluations were run\"\n                    },[av][aw][ax][ay][az]\n                    \"metric_config\": {\n                        \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ax","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_timestamp","author":"Google Doc","body":"did we figure out what the other one was supposed to mean?\nI think my comment meant we should delete the other one","status":"open","source":"google_doc","anchor":"ax","evidence":"\"type\": \"string\",\n                        \"description\": \"Timestamp for when the evaluations were run\"\n                    },[av][aw][ax][ay][az]\n                    \"metric_config\": {\n                        \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ay","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_timestamp","author":"Google Doc","body":"Looks like just different placing. This is something that should be unique to this runs so I agree it should be here.","status":"open","source":"google_doc","anchor":"ay","evidence":"\"type\": \"string\",\n                        \"description\": \"Timestamp for when the evaluations were run\"\n                    },[av][aw][ax][ay][az]\n                    \"metric_config\": {\n                        \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-az","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].evaluation_timestamp","author":"Google Doc","body":"Most of the time we will not have this information. It will just be lying around and is prone to errors i.e. people filling it with random fields to make it pass. We need to tighten edge cases. I.e. if this was not from an actual evaluation run, present the date of the paper URL or something. Leaving it free will lead to random errors that will propagate.","status":"open","source":"google_doc","anchor":"az","evidence":"\"type\": \"string\",\n                        \"description\": \"Timestamp for when the evaluations were run\"\n                    },[av][aw][ax][ay][az]\n                    \"metric_config\": {\n                        \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ba","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"We could get rid of this one maybe? I've noticed a lot of confusion about how to use this - especially since metric_kind is the one that is used for grouping","status":"open","source":"google_doc","anchor":"ba","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bb","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"Think metric_id is unique and will be the same across the entire datastore. Think its valuable.","status":"open","source":"google_doc","anchor":"bb","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bc","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"+1 on keeping metric_id, this would help model things like pass@1, ...,  pass@10 all belonging into pass_rate for instance.\n\n\nagreed this can be clarified for users","status":"open","source":"google_doc","anchor":"bc","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bd","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"If we resolve this to something canonical than it does serve a distinct purpose.","status":"open","source":"google_doc","anchor":"bd","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-be","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"why isn't all metric information grouped under something like metric_metadata? (the actual score shouldn't be in that group). Then this metadata can be referred to by different instantiations (e.g. each instance level data), and they all come together when same.","status":"open","source":"google_doc","anchor":"be","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bf","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"Might also create some symmetry. X_name is always human interpretable x_y is always ...","status":"open","source":"google_doc","anchor":"bf","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bg","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"Most answers to \"why isnt\" would be that it evolved based on something that was previously implemented. Agree it should be a self-contained field.we should remove metric_name as it is an ambiguous field. We can remove a lot of ambiguity by removing a lot of subsequent fields such as metric_unit, metric_parameters, lower_is_btter, score_type, level_names, level_metadata, has_unknown_level, min_score, max_score etc., by just having a strict mapping auto-filled by the metric_id","status":"open","source":"google_doc","anchor":"bg","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bh","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"So you suggest a registry of metrics, and then each maps to a single instance of all those fields. What happens when it is the metric with some change, do we allow override? Do force a new one? (e.g. acc 0-1 vs. acc 0-100 how to deal with those?)","status":"open","source":"google_doc","anchor":"bh","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bi","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.metric_id","author":"Google Doc","body":"I wrote on slack, but yes they will be different. There is no issue in having as many combinations/variations as long its accessible easily. We can maintain this registry, its much more accurate.","status":"open","source":"google_doc","anchor":"bi","evidence":"\"metric_id\": {\n                                \"type\": \"string\",\n                                \"description\": \"Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo).\"[ba][bb][bc][bd][be][bf][bg][bh][bi]\n                            },\n                            \"metric_name\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bj","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.lower_is_better","author":"Google Doc","body":"I've always preferred \"objective\", which could be something like \"minimize\", \"maximize\", or something else in exotic cases. Probably not a generalization this will ever need though, but it feels nicer to me conceptually.","status":"open","source":"google_doc","anchor":"bj","evidence":"}\n                            },\n                            \"lower_is_better[bj]\": {\n                                \"type\": \"boolean\",\n                                \"description\": \"Whether a lower score is better\"","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bk","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.score_type","author":"Google Doc","body":"Score vs. metric? Why?","status":"open","source":"google_doc","anchor":"bk","evidence":"\"description\": \"Whether a lower score is better\"\n                            },\n                            \"score[bk][bl]_type\": {\n                                \"type\": \"string\",\n                                \"description\": \"Type of score\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bl","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.score_type","author":"Google Doc","body":"This is not a unique trait of every number but I guess of the metric?","status":"open","source":"google_doc","anchor":"bl","evidence":"\"description\": \"Whether a lower score is better\"\n                            },\n                            \"score[bk][bl]_type\": {\n                                \"type\": \"string\",\n                                \"description\": \"Type of score\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bm","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.level_names","author":"Google Doc","body":"what is a score level?","status":"open","source":"google_doc","anchor":"bm","evidence":"\"level_names\": {\n                                \"type\": \"array\",\n                                \"description\": \"Names of the score levels[bm]\",\n                                \"items\": {\n                                    \"type\": \"string\"","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bn","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.llm_scoring.judges","author":"Google Doc","body":"Isn;t all that part of metric and not score (I see the distinction as metric is the metadata score is the number used in this specific submission (the value the metric gets)","status":"open","source":"google_doc","anchor":"bn","evidence":"],\n                                \"properties\": {\n                                    \"judges[bn][bo]\": {\n                                        \"type\": \"array\",\n                                        \"description\": \"LLM judge(s) - single item for judge, multiple for jury\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bo","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.llm_scoring.judges","author":"Google Doc","body":"Why anon? It is me. Bad Google","status":"open","source":"google_doc","anchor":"bo","evidence":"],\n                                \"properties\": {\n                                    \"judges[bn][bo]\": {\n                                        \"type\": \"array\",\n                                        \"description\": \"LLM judge(s) - single item for judge, multiple for jury\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bp","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].metric_config.score_type","author":"Google Doc","body":"issue #144: here we would put the condition on the \"ingestion_method\" to distinguish contributions that require adapters, from those who don't","status":"open","source":"google_doc","anchor":"bp","evidence":"},\n                        \"else\": {\n                            \"if\": {[bp]\n                                \"properties\": {\n                                    \"score_type\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bq","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].generation_config.generation_args.reasoning","author":"Google Doc","body":"What about reasoning_effort, verbosity, or max_reasoning_tokens?\n\n\nPresumably they can be in generation_config.additional_details but I think users will need nudging and examples to fill those in","status":"open","source":"google_doc","anchor":"bq","evidence":"},\n                                    \"reasoning\": {\n                                        \"type\": \"boolean\",[bq][br]\n                                        \"description\": \"Whether reasoning orchain-of-thought was used to generate results\"\n                                    },","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-br","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].generation_config.generation_args.reasoning","author":"Google Doc","body":"I think we should add some comments to force this. Presumably the LLM that will write the converter should take a look. I think the solution lies in a more modular schema.","status":"open","source":"google_doc","anchor":"br","evidence":"},\n                                    \"reasoning\": {\n                                        \"type\": \"boolean\",[bq][br]\n                                        \"description\": \"Whether reasoning orchain-of-thought was used to generate results\"\n                                    },","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bs","schema_id":"aggregate","schema_version":"0.2.2","path":"evaluation_results[].generation_config.generation_args.sandbox.config","author":"Google Doc","body":"These sections are overfit to inspect.","status":"open","source":"google_doc","anchor":"bs","evidence":"\"config\": {\n                                                \"type\": \"string\",\n                                                \"description\": \"Config file name/path e.g. compose.yaml. TODO or full config? Not sure based on the Inspect docs\"[bs]\n                                            }\n                                        }","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bt","schema_id":"aggregate","schema_version":"0.2.2","path":"detailed_evaluation_results.additional_details","author":"Google Doc","body":"I think we should move this into the evaluation_results - and have a way to target a particular evaluation result within the jsonl instead of having a broad connection to 'some file'","status":"open","source":"google_doc","anchor":"bt","evidence":"}\n        }\n    },[bt]\n    \"$defs\": {\n        \"judge_config\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bu","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_url","author":"Google Doc","body":"Why call id source_data and not dataset? source vs. target?\nIt is not source vs. metadata that's just confusing dataset vs. metadata ok, dataset vs. not even saying metadata but just the specifics (evaluator, model etc.) is even less confusing.","status":"open","source":"google_doc","anchor":"bu","evidence":"}\n        },\n        \"source_data_url\": {[bu][bv]\n            \"type\": \"object\",\n            \"description\": \"URL source for the evaluation data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bv","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_url","author":"Google Doc","body":"In this sense, a refactor I long suggested\nReporting\nBenchmark\nEvaluation\nMetric\\score\nModel\n\n\nThose are the high level, then \"source\" is about who is reporting, Benchmark and model are more static, and evaluation then is separated. I think I have something of this sort on the schema feedback as well, which I think will make it all less confusing, easier to extend and more organized.\nHarsha  [5:26 PM]\nYes, makes sense","status":"open","source":"google_doc","anchor":"bv","evidence":"}\n        },\n        \"source_data_url\": {[bu][bv]\n            \"type\": \"object\",\n            \"description\": \"URL source for the evaluation data\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bw","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_url.dataset_name","author":"Google Doc","body":"is there any other dataset involved? WDYM \"source\" evaluated dataset?","status":"open","source":"google_doc","anchor":"bw","evidence":"\"dataset_name\": {\n                    \"type\": \"string\",\n                    \"description\": \"Name of the source[bw] dataset\"\n                },\n                \"source_type\": {","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bx","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_hf","author":"Google Doc","body":"So this we make into something unique that prefers this, and if doesn't have this gives a unique link to identify the most canonical source of the dataset in a way that you expect others to pick the same link if asked. If the data is not publicly available, the paper describing it, if there is none (then what? just write a name?","status":"open","source":"google_doc","anchor":"bx","evidence":"}\n        },\n        \"source_data_hf[bx][by]\": {\n            \"type\": \"object\",\n            \"description\": \"Details about HuggingFace dataset used for evaluation\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-by","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_hf","author":"Google Doc","body":"Or the canonical is by default the source_data_hf if not it is the source_data_private or something like that? So it is an automatically generated canonical form?","status":"open","source":"google_doc","anchor":"by","evidence":"}\n        },\n        \"source_data_hf[bx][by]\": {\n            \"type\": \"object\",\n            \"description\": \"Details about HuggingFace dataset used for evaluation\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-bz","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_private.source_type","author":"Google Doc","body":"This may need guidance","status":"open","source":"google_doc","anchor":"bz","evidence":"\"source_type\": {\n                    \"const\": \"other\"\n                },[bz][ca]\n                \"additional_details\": {\n                    \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"},{"id":"gdoc-ca","schema_id":"aggregate","schema_version":"0.2.2","path":"$defs.source_data_private.source_type","author":"Google Doc","body":"it does, what does it expect?","status":"open","source":"google_doc","anchor":"ca","evidence":"\"source_type\": {\n                    \"const\": \"other\"\n                },[bz][ca]\n                \"additional_details\": {\n                    \"type\": \"object\",","decision":"","created_at":"2026-07-16T09:34:13+00:00","updated_at":"2026-07-16T09:34:13+00:00"}]