+gd{c:dZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$m%Z%e e&Z'eGddZ(eGddZ)Gdde*Z+Gdde*Z,eGddZ-eGddZ.Gdde e/e.fZ0eGddZ1dS) a DatasetInfo and MetricInfo record information we know about a dataset and a metric. This includes things that we know about the dataset statically, i.e.: - description - canonical location - does it have validation and tests splits - size - etc. This also includes the things that can and should be computed once we've processed the dataset as well: - number of examples (in each split) - etc. N) dataclass)Path)ClassVarDictListOptionalUnion)config)FeaturesValue)is_remote_filesystem) SplitDict) TaskTemplatetask_template_from_dict)Version) get_logger)DatasetMetadata)asdict unique_valuesc,eZdZUdZeed<dZeed<dS)SupervisedKeysDatainputoutputN)__name__ __module__ __qualname__rstr__annotations__r-lib/python3.11/site-packages/datasets/info.pyrr9s/E3OOOFCr"rc,eZdZUdZeed<dZeed<dS)DownloadChecksumsEntryDatarkeyvalueN)rrrr&rr r'r!r"r#r%r%?s*CMMME3OOOOOr"r%ceZdZdZdS)MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nrrr__doc__r!r"r#r)r)EsEEEEr"r)ceZdZdZdS)NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr*r!r"r#r-r-Is9999r"r-cjeZdZUdZeeed<dZeeed<dZ e deddfdZ dS)PostProcessedInfoNfeaturesresources_checksumsc|j:t|jts"tj|j|_dSdSdSN)r0 isinstancer from_dictselfs r# __post_init__zPostProcessedInfo.__post_init__RsA = $Z x-P-P $$.t}==DMMM % $ $ $r"post_processed_info_dictreturncdtj|D|difd|DS)Nch|] }|j Sr!name.0fs r# z.PostProcessedInfo.from_dict..Y???!qv???r"c$i|] \}}|v || Sr!r!r@kv field_namess r# z/PostProcessedInfo.from_dict..Zs*\\\tq!1P[K[K[aK[K[K[r"r! dataclassesfieldsitems)clsr9rHs @r#r5zPostProcessedInfo.from_dictWsY??{'9#'>'>??? s]]\\\\'?'E'E'G'G\\\]]]r") rrrr0rr r r1dictr8 classmethodr5r!r"r#r/r/Ms#'Hhx '''*.$...>>> ^^:M^^^[^^^r"r/cFeZdZUdZejeZeed<ejeZ eed<ejeZ eed<ejeZ eed<dZ e eed<dZe eed <dZe eed <dZe eeed <dZe eed <dZe eed <dZe eeefed<dZe eed<dZe eed<dZe eed<dZ e eed<dZ!e eed<dZ"e eed<gdZ#e$eeed<dZ% d,de efdZ&d-dZ'dZ(e)dedfdZ*e) d.d ede ed!dfd"Z+e)d#ed!dfd$Z,d/d0d'Z-d1d(Z.d!efd)Z/e)d*ed!dfd+Z0dS)2 DatasetInfoa Information about a dataset. `DatasetInfo` documents datasets, including its name, version, and features. See the constructor arguments and properties for a full list. Not all fields are known on construction and may be updated later. Attributes: description (`str`): A description of the dataset. citation (`str`): A BibTeX citation of the dataset. homepage (`str`): A URL to the official homepage for the dataset. license (`str`): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. features ([`Features`], *optional*): The features used to specify the dataset's column types. post_processed (`PostProcessedInfo`, *optional*): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index. supervised_keys (`SupervisedKeysData`, *optional*): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). builder_name (`str`, *optional*): The name of the `GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name. config_name (`str`, *optional*): The name of the configuration derived from [`BuilderConfig`]. version (`str` or [`Version`], *optional*): The version of the dataset. splits (`dict`, *optional*): The mapping between split name and metadata. download_checksums (`dict`, *optional*): The mapping between the URL to download the dataset's checksums and corresponding metadata. download_size (`int`, *optional*): The size of the files to download to generate the dataset, in bytes. post_processing_size (`int`, *optional*): Size of the dataset in bytes after post-processing, if any. dataset_size (`int`, *optional*): The combined size in bytes of the Arrow tables for all splits. size_in_bytes (`int`, *optional*): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files). task_templates (`List[TaskTemplate]`, *optional*): The task templates to prepare the dataset for during training and evaluation. Each template casts the dataset's [`Features`] to standardized column names and types as detailed in `datasets.tasks`. **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`]. default_factory descriptioncitationhomepagelicenseNr0post_processedsupervised_keystask_templates builder_name config_nameversionsplitsdownload_checksums download_sizepost_processing_size dataset_size size_in_bytes)r]rarcr0r__INCLUDED_INFO_IN_YAMLcj8tjtstjj_j>tjt s$t j_jltjtsRtjtrtj_ntjj_j 8tj tstj j _ j etj tsKtj ttfrtj _ ntdij _ jtjttfr#djD}d|D_nHtjt"rjg_n t%j}||gng_j:tj_jfdjD_dSdSdS)NcZg|](}t|tr|nt|)Sr!)r4rrr@templates r# z-DatasetInfo.__post_init__..sE !+8\ B BiHHH_`hHiHir"cg|]}||Sr3r!rhs r#rjz-DatasetInfo.__post_init__..s&b&b&bHXMaxMaMaMar"cDg|]}|jSr!)align_with_featuresr0)r@rir7s r#rjz-DatasetInfo.__post_init__..s6'''DLH00??'''r"r!)r0r4r r5rYr/r^rrr_rfrom_split_dictrZrtuplelistr[rr)r7 templatesris` r#r8zDatasetInfo.__post_init__s\ = $Z x-P-P $$.t}==DM   *:d>QSd3e3e *"3"="=d>Q"R"RD  < #Jt|W,M,M #$,,, ?&t|44 &0>> ; ":dk9+M+M "#3DK@@DK   +Jt?SUg4h4h +$. >> R'94;O'P$$'9'Q'QD> Q'+':&;##243FGG4<4Hxjjb#   *"&t':";";D }(''''QUQd'''### + *((r"F deprecatedstorage_optionscb|dkr!tjdt|j}t j||}|d}t | }|rtjj n tj }| ||tj d5}|||dddn #1swxYwY|jrZ| ||tjd5}||ddddS#1swxYwYdSdS)ahWrite `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`. Args: dataset_info_dir (`str`): Destination directory. pretty_print (`bool`, defaults to `False`): If `True`, the JSON will be pretty-printed with the indent level of 4. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("rotten_tomatoes", split="validation") >>> ds.info.write_to_directory("/path/to/directory/") ``` rr'fs' was deprecated in favor of 'storage_options' in version 2.9.0 and will be removed in 3.0.0. You can remove this warning by passing 'storage_options=fs.storage_options' instead.rsrwb) pretty_printN)warningswarn FutureWarningrsfsspecget_fs_token_pathsrospathjoin posixpathopenr DATASET_INFO_FILENAME _dump_inforXLICENSE_FILENAME _dump_license) r7dataset_info_dirrxfsrsfs_token_pathsis_local path_joinrAs r#write_to_directoryzDatasetInfo.write_to_directorysB    Mg    !0O23CUdeee(6q(9+B///$,@BGLL). WWYY/1MNNPT U U :YZ OOALO 9 9 9 : : : : : : : : : : : : : : : < &#3V5LMMtTT &XY""1%%% & & & & & & & & & & & & & & & & & & & &s$CC C?D""D&)D&c|tjt||rdndddS)zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsrencode)r7filerxs r#rzDatasetInfo._dump_info sG 4:fTll 3N11$OOOVVW^__`````r"c`||jddS)zTDump license in `file` file-like object open in bytes mode (to support remote files)rN)rrXr)r7rs r#rzDatasetInfo._dump_license s* 4<&&w//00000r" dataset_infosc Ld|D}dtd|D}dtd|D}dtd|D}dtd|D}d}d}d}d|D} t| dkr3t t | d j| dd}n1t| r"t t | d }|r|nd}|||||||| S) Nc:g|]}||Sr3)copy)r@ dset_infos r#rjz*DatasetInfo.from_merge..s'bbbiILa))LaLaLar"z c3$K|] }|jV dSr3)rUr@infos r# z)DatasetInfo.from_merge..s%/[/[T0@/[/[/[/[/[/[r"c3$K|] }|jV dSr3)rVrs r#rz)DatasetInfo.from_merge..$,U,UtT],U,U,U,U,U,Ur"c3$K|] }|jV dSr3)rWrs r#rz)DatasetInfo.from_merge..rr"c3$K|] }|jV dSr3)rXrs r#rz)DatasetInfo.from_merge..s$+S+STDL+S+S+S+S+S+Sr"c*g|]}|j |jSr3)r[rs r#rjz*DatasetInfo.from_merge..s#ooodtObOnd1OnOnOnr"r r)rUrVrWrXr0rZr[)rrstriplenrpset intersection) rNrrUrVrWrXr0rZr[all_task_templatess r# from_mergezDatasetInfo.from_mergesbb=bbb kk-/[/[]/[/[/["["[\\bbdd ;;},U,U},U,U,UUUVV\\^^;;},U,U},U,U,UUUVV\\^^++m+S+S]+S+S+SSSTTZZ\\pomooo ! " "Q & &!"I#&8&;"<"<"IK]^_^`^`Ka"bccNN # $ $ >!#&8&;"<"<==N+9Cts#+)    r"rr:c |dkr!tjdt|j}t j||}|d}t d||stdt| }|rtj j n tj }|||tjdd 5}t#j|}d d d n #1swxYwY||S) aCreate [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_dir (`str`): The directory containing the metadata file. This should be the root directory of a specific dataset version. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import DatasetInfo >>> ds_info = DatasetInfo.from_directory("/path/to/directory/") ``` rrrurvrzLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rrencodingN)ryrzr{rsr|r}loggerr ValueErrorrr~rrrrr rrloadr5) rNrrrsrrrrAdataset_info_dicts r#from_directoryzDatasetInfo.from_directory/sNJ    Mg    !0O23CUdeee(6q(9 C1ACCDDD fdee e+B///$,@BGLL). WWYY/1MNNPS^eW f f -jk $ !  - - - - - - - - - - - - - - -}}.///sC00C47C4rcdtj|D|difd|DS)Nch|] }|j Sr!r=r?s r#rBz(DatasetInfo.from_dict..lrCr"c$i|] \}}|v || Sr!r!rEs r#rIz)DatasetInfo.from_dict..ms)UUUtq!ADTDTaDTDTDTr"r!rJ)rNrrHs @r#r5zDatasetInfo.from_dictjsY??{'9#'>'>??? sVVUUUU'8'>'>'@'@UUUVVVr"Tother_dataset_infocv|j}|jdifd|jDdS)NcHi|]\}}| |tj|Sr3rdeepcopy)r@rFrG ignore_nones r#rIz&DatasetInfo.update..rs:AqMM4=##!MMr"r!)__dict__updaterM)r7rr self_dicts ` r#rzDatasetInfo.updateosjM    .7==??     r"c^|jdid|jDS)Nc>i|]\}}|tj|Sr!r)r@rFrGs r#rIz$DatasetInfo.copy..zs( W W WADM!$4$4 W W Wr"r!) __class__rrMr6s r#rzDatasetInfo.copyys7t~XX W WATATAVAV W W WXXXr"ci}t|}|D]p}||jvret||}t|dr|||<Ct|dr|||<k|||<q|S)N _to_yaml_list_to_yaml_string)rregetattrhasattrrr)r7 yaml_dictrr&r's r# _to_yaml_dictzDatasetInfo._to_yaml_dict|s "4LL$ + +Cd111c**5/22+%*%8%8%:%:IcNNU$566+%*%:%:%<%.rCr"c$i|] \}}|v || Sr!r!rEs r#rIz/DatasetInfo._from_yaml_dict..s)MMMtq!A'>??? sNNMMMMy'8'8MMMNNNr")FrrNF)rrN)T)rrR)r:rR)1rrrr+rKfieldrrUr rVrWrXr0rr rYr/rZrr[rrr\r]r^r rr_rOr`raintrbrcrdrerr8rrrrPrrr5rrrrr!r"r#rRrR]sy,,^){(===K===%K%c:::Hc:::%K%c:::Hc:::$;$S999GS999#'Hhx '''26NH./66648OX0188837NHT,/0777#'L(3-&&&!%K#%%%-1GXeCL) *111!FHTN!!!)----#'M8C='''*.(3-..."&L(3-&&&#'M8C='''333HT#Y/'''Thl3&3&V^_cVd3&3&3&3&jaaaa111 tM':   [ :W[8080"80FNtn80 808080[80tW$W=WWW[W     YYYY t    OOOOO[OOOr"rRc`eZdZd d dZed dZededdfdZdeddfdZdS) DatasetInfosDictFr:Nci}tj|tj}tj|d}|s||}||tj|rct|dd5}d| D}tj |||rdnddddn #1swxYwYtj|r"tj t|} nt} |r9|| | t|dSdS)N README.mdwrrc4i|]\}}|t|Sr!)rr@r]rs r#rIz7DatasetInfosDict.write_to_directory..s3&&&7M{IK !2!2&&&r"rr)r~rrr DATASETDICT_INFOS_FILENAMErrexistsrrMrdumpr from_readmer to_metadata to_readme) r7dataset_infos_dir overwriterxtotal_dataset_infosdataset_infos_pathdataset_readme_pathrAdataset_infos_dictdataset_metadatas r#rz#DatasetInfosDict.write_to_directorys W\\*;V=^__ gll+>, - - U(#@@@ UA&&QdQjQjQlQl&&&" ,a\8StTTTT  U U U U U U U U U U U U U U U 7>>- . . 1.:4@S;T;TUU  .00   B  + +,< = = =  & &t,?'@'@ A A A A A B Bs*:C00C47C4ctd|tjtj|dr=t jt|dz }d|vr| |Stjtj|tj rttj|tj d5}|dtj|DcdddS#1swxYwYdS|S)NzLoading Dataset Infos from r dataset_inforrcJi|] \}}|t|!Sr!)rRr5)r@r]rs r#rIz3DatasetInfosDict.from_directory..s>:K):$[%:%:;L%M%Mr")rrr~rrrrrr from_metadatar rrrrrM)rNrrrAs r#rzDatasetInfosDict.from_directorys E2CEEFFF 7>>"',,'8+FF G G ;.:4@Q;R;RU`;`aa !111(()9::: 7>>"',,'8&:[\\ ] ] bgll#4f6WXXcjkkk ops>Bill>P>P>R>R                  355Ls9EE"%E"rct|dttfrt|dtr|d|dDSt|d}|ddd|_||j|iS|S)Nrcli|]1}|ddt|2S)r]default)rrRr)r@dataset_info_yaml_dicts r#rIz2DatasetInfosDict.from_metadata..sQ3/22=)LLkNiNi2OOr"r]r)r4rrprOrRrr])rNrrs r#rzDatasetInfosDict.from_metadatas &**>::T4L I I *>:DAA Es7G~6V  +::;KN;[\\ +;N+K+O+OP]_h+i+i (sL4lCDDD355Lr"c<|rd|D}|D] \}}||d< t|dkrptt||d<|dddkr|ddddSdSg|d<|D]>\}}|ddd|i|}|d|=dSdS)Nc>i|]\}}||Sr!)rrs r#rIz0DatasetInfosDict.to_metadata..s,"u"u"uNdk[d; 0G0G0I0I"u"u"ur"r]r rr)rMrnextitervaluesrpopappend)r7rrr]dset_info_yaml_dictrs r#rzDatasetInfosDict.to_metadatasm  T"u"uhlhrhrhtht"u"u"u 4G4M4M4O4O A A0 05@#M22&''1,,37=P=W=W=Y=Y8Z8Z3[3[ 0#N377 FF)SS$^488MMMMMTS46 0;N;T;T;V;VTT7K!7*..}dCCC.;[-cLb-c*$^4;;>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.info.write_to_directory("/path/to/directory/") ``` rrrrNr) rr~rrr METRIC_INFO_FILENAMErrrrXrr)r7metric_info_dirrxrAs r#rzMetricInfo.write_to_directory sk"',,0KLLc\c d d d Khi IfTllA<.IaaT J J J J K K K K K K K K K K K K K K K < &bgll?F4KLLc\cddd &hi %%% & & & & & & & & & & & & & & & & & & & &s#)A00A47A4<C$$C(+C(r:cRtd||stdttj|tjd5}tj |}dddn #1swxYwY| |S)aCreate MetricInfo from the JSON file in `metric_info_dir`. Args: metric_info_dir: `str` The directory containing the metadata file. This should be the root directory of a specific dataset version. Example: ```py >>> from datasets import MetricInfo >>> metric_info = MetricInfo.from_directory("/path/to/directory/") ``` zLoading Metric info from zCCalling MetricInfo.from_directory() with undefined metric_info_dir.rrN) rrrrr~rrr rrrr5)rNrrAmetric_info_dicts r#rzMetricInfo.from_directory!s  AAABBB dbcc c "',,0KLLW^ _ _ _ ,cd#y||  , , , , , , , , , , , , , , ,}}-...s(B  B B rcdtj|D|difd|DS)Nch|] }|j Sr!r=r?s r#rBz'MetricInfo.from_dict..:rCr"c$i|] \}}|v || Sr!r!rEs r#rIz(MetricInfo.from_dict..;s)TTTtq!1 CSCSaCSCSCSr"r!rJ)rNrrHs @r#r5zMetricInfo.from_dict8sY??{'9#'>'>??? sUUTTTT'7'='='?'?TTTUUUr"r)r:r)rrrr+rr r rKrrrWrXrprrrrboolrrrr]rr8rrPrrOr5r!r"r#rrsMMM/k/DDDDDD%K%c:::Hc:::$;$S999GS9990{0FFFM49FFF 1 1$ G G GNDIGGGJ FHSM   "&K#%%%!%K#%%%#'M8C='''&&&&(///[/,VV,VVV[VVVr"r)2r+rrKrr~rryrpathlibrtypingrrrrr r|rr r0r r filesystemsrr_rtasksrrutilsr utils.loggingrutils.metadatarutils.py_utilsrrrrrr% Exceptionr)r-r/rRrrrr!r"r#rsE    !!!!!!88888888888888 %%%%%%%%------88888888%%%%%%++++++11111111 H        FFFFFIFFF:::::):::  ^ ^ ^ ^ ^ ^ ^  ^ tOtOtOtOtOtOtO tOn QTQTQTQTQTtC,-QTQTQTh QVQVQVQVQVQVQV QVQVQVr"