l$es ddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZeeZeGdd ZeGd d ZGd d eZGddeZGddeZdeeeefde eeeffdZdZdedeedeeeeffdZdS)N) defaultdict) dataclass)AnyDictListOptionalTupleUnion) yaml_dump) get_loggerceZdZUdZeed<eed<eed<eed<eed<dZeeed<dZ eeed <dZ eeed <dZ eeed <dZ ee eefed <dZeeed <dZeeed<dZee eefed<dZeeed<dZeeed<edefdZdddefdZdS) EvalResultu Flattened representation of individual evaluation results found in model-index of Model Cards. For more information on the model-index spec, see https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1. Args: task_type (`str`): The task identifier. Example: "image-classification". dataset_type (`str`): The dataset identifier. Example: "common_voice". Use dataset id from https://hf.co/datasets. dataset_name (`str`): A pretty name for the dataset. Example: "Common Voice (French)". metric_type (`str`): The metric identifier. Example: "wer". Use metric id from https://hf.co/metrics. metric_value (`Any`): The metric value. Example: 0.9 or "20.0 ± 1.2". task_name (`str`, *optional*): A pretty name for the task. Example: "Speech Recognition". dataset_config (`str`, *optional*): The name of the dataset configuration used in `load_dataset()`. Example: fr in `load_dataset("common_voice", "fr")`. See the `datasets` docs for more info: https://hf.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset.name dataset_split (`str`, *optional*): The split used in `load_dataset()`. Example: "test". dataset_revision (`str`, *optional*): The revision (AKA Git Sha) of the dataset used in `load_dataset()`. Example: 5503434ddd753f426f4b38109466949a1217c2bb dataset_args (`Dict[str, Any]`, *optional*): The arguments passed during `Metric.compute()`. Example for `bleu`: `{"max_order": 4}` metric_name (`str`, *optional*): A pretty name for the metric. Example: "Test WER". metric_config (`str`, *optional*): The name of the metric configuration used in `load_metric()`. Example: bleurt-large-512 in `load_metric("bleurt", "bleurt-large-512")`. See the `datasets` docs for more info: https://huggingface.co/docs/datasets/v2.1.0/en/loading#load-configurations metric_args (`Dict[str, Any]`, *optional*): The arguments passed during `Metric.compute()`. Example for `bleu`: max_order: 4 verified (`bool`, *optional*): Indicates whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. Automatically computed by Hugging Face, do not set. verify_token (`str`, *optional*): A JSON Web Token that is used to verify whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. task_type dataset_type dataset_name metric_type metric_valueN task_namedataset_config dataset_splitdataset_revision dataset_args metric_name metric_config metric_argsverified verify_tokenreturncB|j|j|j|j|jfS)z9Returns a tuple that uniquely identifies this evaluation.)rrrrrselfs =lib/python3.11/site-packages/huggingface_hub/repocard_data.pyunique_identifierzEvalResult.unique_identifier|s+ N        !   otherc|jD]7\}}|dkr |dkr%t||t||krdS8dS)zx Return True if `self` and `other` describe exactly the same metric but with a different value. rrFT)__dict__itemsgetattr)r"r&key_s r#is_equal_except_valuez EvalResult.is_equal_except_valuesk m))++  FCn$$n$$s););wuc?R?R)R)Ruutr%)__name__ __module__ __qualname____doc__str__annotations__rrrrrrrrrrrrboolrpropertytupler$r-r%r#rrs))^NNN $Ix}### %)NHSM((($(M8C=''''+hsm***.2L(4S>*111"&K#%%% $(M8C='''-1K$sCx.)000 $Hhtn####'L(3-&&&  5   X  < D      r%rceZdZdZddefdZdeeeffdZ dZ ddefd Z d Z dd ed edefd Z dd ed edefdZd edefdZd ededdfdZd edefdZdS)CardDataaStructure containing metadata from a RepoCard. [`CardData`] is the parent class of [`ModelCardData`] and [`DatasetCardData`]. Metadata can be exported as a dictionary or YAML. Export can be customized to alter the representation of the data (example: flatten evaluation results). `CardData` behaves as a dictionary (can get, pop, set values) but do not inherit from `dict` to allow this export step. Fignore_metadata_errorsc :|j|dSN)r(update)r"r:kwargss r#__init__zCardData.__init__s V$$$$$r%rc|tj|j}||t |S)zConverts CardData to a dict. Returns: `dict`: CardData represented as a dictionary ready to be dumped to a YAML block for inclusion in a README.md file. )copydeepcopyr(_to_dict _remove_noner" data_dicts r#to_dictzCardData.to_dicts5M$-00  i   I&&&r%cdS)zUse this method in child classes to alter the dict representation of the data. Alter the dict in-place. Args: data_dict (`dict`): The raw dict representation of the card data. Nr7rEs r#rCzCardData._to_dicts r%Ncnt|d|S)a Dumps CardData to a YAML block for inclusion in a README.md file. Args: line_break (str, *optional*): The line break to use when dumping to yaml. Returns: `str`: CardData represented as a YAML block. F) sort_keys line_break)r rGstrip)r"rKs r#to_yamlzCardData.to_yamls-5ZPPPVVXXXr%c*|Sr<)rMr!s r#__repr__zCardData.__repr__s||~~r%r+defaultc8|j||Sz#Get value for a given metadata key.)r(getr"r+rPs r#rSz CardData.get}  g...r%c8|j||S)z#Pop value for a given metadata key.)r(poprTs r#rWz CardData.poprUr%c|j|SrRr(r"r+s r# __getitem__zCardData.__getitem__s}S!!r%valuec||j|<dS)z#Set value for a given metadata key.NrY)r"r+r\s r# __setitem__zCardData.__setitem__s" cr%c||jvS)z%Check if a given metadata key is set.rYrZs r# __contains__zCardData.__contains__sdm##r%)Fr<)r.r/r0r1r4r?rr2rrGrCrMrOrSrWr[r^r`r7r%r#r9r9sn%%t%%%% 'c3h ' ' ' '    Y Y# Y Y Y Y//s/S/C//////s/S/C////"s"s""""#s#3#4####$$$$$$$$r%r9ceZdZdZdddddddddd deeeeefdeedeedeeed eeed eeed eeed eed e ffdZ dZ xZ S) ModelCardDataa Model Card Metadata that is used by Hugging Face Hub when included at the top of your README.md Args: language (`Union[str, List[str]]`, *optional*): Language of model's training data or metadata. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". Defaults to `None`. license (`str`, *optional*): License of this model. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. Defaults to None. library_name (`str`, *optional*): Name of library used by this model. Example: keras or any library from https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Libraries.ts. Defaults to None. tags (`List[str]`, *optional*): List of tags to add to your model that can be used when filtering on the Hugging Face Hub. Defaults to None. datasets (`List[str]`, *optional*): List of datasets that were used to train this model. Should be a dataset ID found on https://hf.co/datasets. Defaults to None. metrics (`List[str]`, *optional*): List of metrics used to evaluate this model. Should be a metric name that can be found at https://hf.co/metrics. Example: 'accuracy'. Defaults to None. eval_results (`Union[List[EvalResult], EvalResult]`, *optional*): List of `huggingface_hub.EvalResult` that define evaluation results of the model. If provided, `model_name` is used to as a name on PapersWithCode's leaderboards. Defaults to `None`. model_name (`str`, *optional*): A name for this model. It is used along with `eval_results` to construct the `model-index` within the card's metadata. The name you supply here is what will be used on PapersWithCode's leaderboards. If None is provided then the repo name is used as a default. Defaults to None. ignore_metadata_errors (`str`): If True, errors while parsing the metadata section will be ignored. Some information might be lost during the process. Use it at your own risk. kwargs (`dict`, *optional*): Additional metadata that will be added to the model card. Defaults to None. Example: ```python >>> from huggingface_hub import ModelCardData >>> card_data = ModelCardData( ... language="en", ... license="mit", ... library_name="timm", ... tags=['image-classification', 'resnet'], ... ) >>> card_data.to_dict() {'language': 'en', 'license': 'mit', 'library_name': 'timm', 'tags': ['image-classification', 'resnet']} ``` NF) languagelicense library_nametagsdatasetsmetrics eval_results model_namer:rcrdrerfrgrhrirjr:c  >||_||_||_||_||_||_||_||_| dd} | ri t| \}}||_||_nG#t$r:} | rt dntd| dYd} ~ nd} ~ wwxYwtjdi| |jr@t!|jt"kr |jg|_|jtddSdS)N model-indexz model_indexerror __class__s r#r?zModelCardData.__init__sy!  (    ($jj55   +F{+S+S( L",$0!!   )NN#abbbb$SX]SSScbbbb  ""6"""   \D%&&*44%)%6$7!& !Z[[[  \ \'&s A44 B8>0B33B8c^|j%t|j|j|d<|d=|d=dSdS)z[Format the internal data dict. In this case, we convert eval results to a valid model indexNrlrirj)rieval_results_to_model_indexrjrEs r#rCzModelCardData._to_dict?sB   ('B4?TXTe'f'fIm $.)9\+B+B+B ) (r%) r.r/r0r1rr r2rrr4r?rC __classcell__rvs@r#rbrbs411l59!%&*$((,'+37$(',-\-\-\5d3i01-\# -\ sm -\ tCy! -\49%-\$s)$-\tJ/0-\SM-\!%-\-\-\-\-\-\^CCCCCCCr%rbc&eZdZdZddddddddddddddddeeeeefdeeeeefdeeeeefdeeeeefd eeeeefd eeeeefd eeed eeeeefd eeeeefdeedeedeedeeeeefde ffdZ dZ xZ S)DatasetCardDataa Dataset Card Metadata that is used by Hugging Face Hub when included at the top of your README.md Args: language (`List[str]`, *optional*): Language of dataset's data or metadata. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". license (`Union[str, List[str]]`, *optional*): License(s) of this dataset. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. annotations_creators (`Union[str, List[str]]`, *optional*): How the annotations for the dataset were created. Options are: 'found', 'crowdsourced', 'expert-generated', 'machine-generated', 'no-annotation', 'other'. language_creators (`Union[str, List[str]]`, *optional*): How the text-based data in the dataset was created. Options are: 'found', 'crowdsourced', 'expert-generated', 'machine-generated', 'other' multilinguality (`Union[str, List[str]]`, *optional*): Whether the dataset is multilingual. Options are: 'monolingual', 'multilingual', 'translation', 'other'. size_categories (`Union[str, List[str]]`, *optional*): The number of examples in the dataset. Options are: 'n<1K', '1K1T', and 'other'. source_datasets (`List[str]]`, *optional*): Indicates whether the dataset is an original dataset or extended from another existing dataset. Options are: 'original' and 'extended'. task_categories (`Union[str, List[str]]`, *optional*): What categories of task does the dataset support? task_ids (`Union[str, List[str]]`, *optional*): What specific tasks does the dataset support? paperswithcode_id (`str`, *optional*): ID of the dataset on PapersWithCode. pretty_name (`str`, *optional*): A more human-readable name for the dataset. (ex. "Cats vs. Dogs") train_eval_index (`Dict`, *optional*): A dictionary that describes the necessary spec for doing evaluation on the Hub. If not provided, it will be gathered from the 'train-eval-index' key of the kwargs. config_names (`Union[str, List[str]]`, *optional*): A list of the available dataset configs for the dataset. NF)rcrdannotations_creatorslanguage_creatorsmultilingualitysize_categoriessource_datasetstask_categoriestask_idspaperswithcode_id pretty_nametrain_eval_index config_namesr:rcrdr}r~rrrrrrrrrr:c ||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | p| dd|_ tjdi|dS)Ntrain-eval-indexr7)r}r~rcrdrrrrrrrrrWrrrr?)r"rcrdr}r~rrrrrrrrrr:r>rvs r#r?zDatasetCardData.__init__ns&%9!!2   ....  !2&(!1 XFJJ?QSW4X4X""6"""""r%c6|d|d<dS)Nrr)rWrEs r#rCzDatasetCardData._to_dicts (1 6H(I(I $%%%r%) r.r/r0r1rr r2rrr4r?rCryrzs@r#r|r|Fs%%T5937@D=A;?;?/3;?48+/%)+/8<',!"#"#"#5d3i01"#%T#Y/0 "# 'uS$s)^'<= "# $E#tCy.$9: "#"%T#Y"78"#"%T#Y"78"#"$s),"#"%T#Y"78"#5d3i01"#$C="#c]"##4."#uS$s)^45"# !%!"#"#"#"#"#"#HJJJJJJJr%r|ceZdZdZddddddddddddd deedeedeedeed eed eed eed eed eeedeeedeeedeffdZ xZ S) SpaceCardDataa Space Card Metadata that is used by Hugging Face Hub when included at the top of your README.md To get an exhaustive reference of Spaces configuration, please visit https://huggingface.co/docs/hub/spaces-config-reference#spaces-configuration-reference. Args: title (`str`, *optional*) Title of the Space. sdk (`str`, *optional*) SDK of the Space (one of `gradio`, `streamlit`, `docker`, or `static`). sdk_version (`str`, *optional*) Version of the used SDK (if Gradio/Streamlit sdk). python_version (`str`, *optional*) Python version used in the Space (if Gradio/Streamlit sdk). app_file (`str`, *optional*) Path to your main application file (which contains either gradio or streamlit Python code, or static html code). Path is relative to the root of the repository. app_port (`str`, *optional*) Port on which your application is running. Used only if sdk is `docker`. license (`str`, *optional*) License of this model. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. duplicated_from (`str`, *optional*) ID of the original Space if this is a duplicated Space. models (List[`str`], *optional*) List of models related to this Space. Should be a dataset ID found on https://hf.co/models. datasets (`List[str]`, *optional*) List of datasets related to this Space. Should be a dataset ID found on https://hf.co/datasets. tags (`List[str]`, *optional*) List of tags to add to your Space that can be used when filtering on the Hub. ignore_metadata_errors (`str`): If True, errors while parsing the metadata section will be ignored. Some information might be lost during the process. Use it at your own risk. kwargs (`dict`, *optional*): Additional metadata that will be added to the space card. Example: ```python >>> from huggingface_hub import SpaceCardData >>> card_data = SpaceCardData( ... title="Dreambooth Training", ... license="mit", ... sdk="gradio", ... duplicated_from="multimodalart/dreambooth-training" ... ) >>> card_data.to_dict() {'title': 'Dreambooth Training', 'sdk': 'gradio', 'license': 'mit', 'duplicated_from': 'multimodalart/dreambooth-training'} ``` NF) titlesdk sdk_versionpython_versionapp_fileapp_portrdduplicated_frommodelsrgrfr:rrrrrrrdrrrgrfr:c  ||_||_||_||_||_||_||_||_| |_| |_ | |_ tj di| dS)Nr7) rrrrrrrdrrrgrfrrr?)r"rrrrrrrdrrrgrfr:r>rvs r#r?zSpaceCardData.__init__sx" &,     .    ""6"""""r%) r.r/r0r1rr2intrr4r?ryrzs@r#rrs+//h $!%)(,"&"&!%)-&*(,$(',###}#c] # c] # ! #3-#3-###"##c##49%#tCy!#!%##########r%rrtrcg}|D]}|d}|d}|D]k}|dd}|dd}|dd}|dd} |dd} |dd} |dd} |dd } |d D]}|d}|d }|d}|d }|d}|d }|d }t||| |||| | | | |||||}||m||fS)aTakes in a model index and returns the model name and a list of `huggingface_hub.EvalResult` objects. A detailed spec of the model index can be found here: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1 Args: model_index (`List[Dict[str, Any]]`): A model index data structure, likely coming from a README.md file on the Hugging Face Hub. Returns: model_name (`str`): The name of the model as found in the model index. This is used as the identifier for the model on leaderboards like PapersWithCode. eval_results (`List[EvalResult]`): A list of `huggingface_hub.EvalResult` objects containing the metrics reported in the provided model_index. Example: ```python >>> from huggingface_hub.repocard_data import model_index_to_eval_results >>> # Define a minimal model index >>> model_index = [ ... { ... "name": "my-cool-model", ... "results": [ ... { ... "task": { ... "type": "image-classification" ... }, ... "dataset": { ... "type": "beans", ... "name": "Beans" ... }, ... "metrics": [ ... { ... "type": "accuracy", ... "value": 0.9 ... } ... ] ... } ... ] ... } ... ] >>> model_name, eval_results = model_index_to_eval_results(model_index) >>> model_name 'my-cool-model' >>> eval_results[0].task_type 'image-classification' >>> eval_results[0].metric_type 'accuracy' ``` nameresultstaskrsdatasetconfigsplitrevisionargsrhr\r verifyToken)rrrrrrrrrrrrrrr)rSrappend)rtrielemrrresultrrrrrrrrmetricrrrrrrr eval_results r#rmrmspL'1'1F|y/$ 1$ 1Fvv.Iv**622I!),V4L!),V4L#I.228<z_remove_none..Ps(GGQaGGr%c3`K|])\}}|| t|t|fV*dSr<r)rkvs r#rz_remove_none..RsCww1WXWdijiv,q//<??;ivivivivwwr%) isinstancelistr6setrsdictr))objs r#rDrDKs#eS)**tCyyGG#GGGGGG C  tCyyww wwwwww r%rjric tt}|D]"}||j|#g}|D]a}|d}|j|jd|j|j|j |j |j |j dd|Dd}||b||dg}t|S)aTakes in given model name and list of `huggingface_hub.EvalResult` and returns a valid model-index that will be compatible with the format expected by the Hugging Face Hub. Args: model_name (`str`): Name of the model (ex. "my-cool-model"). This is used as the identifier for the model on leaderboards like PapersWithCode. eval_results (`List[EvalResult]`): List of `huggingface_hub.EvalResult` objects containing the metrics to be reported in the model-index. Returns: model_index (`List[Dict[str, Any]]`): The eval_results converted to a model-index. Example: ```python >>> from huggingface_hub.repocard_data import eval_results_to_model_index, EvalResult >>> # Define minimal eval_results >>> eval_results = [ ... EvalResult( ... task_type="image-classification", # Required ... dataset_type="beans", # Required ... dataset_name="Beans", # Required ... metric_type="accuracy", # Required ... metric_value=0.9, # Required ... ) ... ] >>> eval_results_to_model_index("my-cool-model", eval_results) [{'name': 'my-cool-model', 'results': [{'task': {'type': 'image-classification'}, 'dataset': {'name': 'Beans', 'type': 'beans'}, 'metrics': [{'type': 'accuracy', 'value': 0.9}]}]}] ``` r)rsr)rrsrrrrc hg|]/}|j|j|j|j|j|j|jd0S))rsr\rrrrr)rrrrrrr)rrs r# z/eval_results_to_model_index..sZ   #.#0".$2". &#)#6   r%)rrrh)rr)rrr$rvaluesrrrrrrrrrD) rjritask_and_ds_types_maprmodel_index_datar sample_resultdatarts r#rxrxWs#J(--#QQ k;<CCKPPPP(//11&& &/%/ &2%2'6&4):%2   &     4 %%%% '  K  $ $$r%)rA collectionsr dataclassesrtypingrrrrr r huggingface_hub.utilsr utils.loggingr r.rorr9rbr|rr2rmrDrxr7r%r#rsd ######!!!!!!::::::::::::::::++++++%%%%%% H   DDDDDDD DN B$B$B$B$B$B$B$ B$JgCgCgCgCgCHgCgCgCTMJMJMJMJMJhMJMJMJ`N#N#N#N#N#HN#N#N#baT$sCx.-AaeCQUV`QaLaFbaaaaH   R%CR%tJ?OR%TXY]^acf^fYgThR%R%R%R%R%R%r%