+gdTRdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlmZddlmZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZdd lm Z ddl!m"Z"e e#Z$Gdde%Z&d'dZ'edd'dZ(d(de)de)de efdZ*edd(de)de)de efdZ+ d)de)de e eee)fde ede e ee)fde e e)e"fde e e,e)ff d Z- d)de)de e e)e"fde ede e ee)fd!e e)de e eee)ff d"Z. d*de)d#e e)de e e)e e)e e)e e)e e)fffde ede e ee)fde e e)e"fde e e,e)fd$efd%Z/ d*de)d#e e)de e e)e e)e e)e e)e e)fffde ede e ee)fde e e)e"fde e e,e)ffd&Z0dS)+z List and inspect datasets.N)PurePath)DictListMappingOptionalSequenceUnion)DownloadConfig) DownloadMode)StreamingDownloadManager) DatasetInfo)dataset_module_factoryimport_main_classload_dataset_buildermetric_module_factory) deprecated)relative_to_absolute_path) get_logger)VersionceZdZdS)SplitsNotFoundErrorN)__name__ __module__ __qualname__0lib/python3.11/site-packages/datasets/inspect.pyrr(sDrrTFchtj|}|s d|D}|s d|D}|S)a]List all the datasets scripts available on the Hugging Face Hub. Args: with_community_datasets (`bool`, *optional*, defaults to `True`): Include the community provided datasets. with_details (`bool`, *optional*, defaults to `False`): Return the full details on the datasets instead of only the short name. Example: ```py >>> from datasets import list_datasets >>> list_datasets() ['acronym_identification', 'ade_corpus_v2', 'adversarial_qa', 'aeslc', 'afrikaans_ner_corpus', 'ag_news', ... ] ``` )fullc$g|] }d|jv |S/id.0datasets r z!list_datasets..Fs$MMMs'*7L7LG7L7L7Lrcg|] }|j Srr$r&s rr)z!list_datasets..Hs7777GJ777r)huggingface_hub list_datasets)with_community_datasets with_detailsdatasetss rr,r,,sS0,,???H "NMM8MMM 877h777 OruxUse 'evaluate.list_evaluation_modules' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluatecdtj}|s d|D}|s d|D}|S)uList all the metrics script available on the Hugging Face Hub. Use `evaluate.list_evaluation_modules` instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluate Args: with_community_metrics (:obj:`bool`, optional, default ``True``): Include the community provided metrics. with_details (:obj:`bool`, optional, default ``False``): Return the full details on the metrics instead of only the short name. Example: ```py >>> from datasets import list_metrics >>> list_metrics() ['accuracy', 'bertscore', 'bleu', 'bleurt', 'cer', 'chrf', ... ] ``` c$g|] }d|jv |Sr"r$r'metrics rr)z list_metrics..ms$HHHf3fi3G3G63G3G3Grcg|] }|j Srr$r2s rr)z list_metrics..os33369333r)r+ list_metrics)with_community_metricsr.metricss rr5r5LsN>*,,G !IHHHHH 4337333 Nrpath local_pathdownload_configc t|fd|i|}t|jd}tj|}t j|}t j|D]\}} } t j |t j ||} t j | dd| D| dd<| D]S} tj t j || t j | | Ttj|| t|}t!d|d|d |d t#|d dS) a Allow inspection/modification of a dataset script by copying on local drive at local_path. Args: path (`str`): Path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'`. - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`list_datasets`]) e.g. `'squad'`, `'glue'` or `'openai/webtext'`. local_path (`str`): Path to the local folder to copy the dataset script to. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. **download_kwargs (additional keyword arguments): Optional arguments for [`DownloadConfig`] which will override the attributes of `download_config` if supplied. r:Tr(exist_okc<g|]}|d|S).__ startswithr'dirnames rr)z#inspect_dataset..,^^^7g>P>PQ\>]>]^w^^^rNz"The processing script for dataset  can be inspected at . The main class is in zP. You can modify this processing script and use it with `datasets.load_dataset("")`.)rr module_pathinspect getsourcefileosr8rFwalkjoinrelpathmakedirsshutilcopy2copystatrprintras_posix) r8r9r:download_kwargsdataset_module builder_clsmodule_source_pathmodule_source_dirpathdirpathdirnames filenames dst_dirpathfilenames rinspect_datasetrbss(,Dee/eUdeeN#N$>MMMK .{;;GOO,>??(*0E(F(F..$9gll:rwwH]/^/^__  K$////_^h^^^ ! _ _H Lgx88"',,{T\:]:] ^ ^ ^ ^----*:66J  T      5  YablYmYmYvYvYxYx   ruzUse 'evaluate.inspect_evaluation_module' instead, from the new library 🤗 Evaluate: https://huggingface.co/docs/evaluatec t|fd|i|}t|jd}tj|}t j|}t j|D]\}} } t j |t j ||} t j | dd| D| dd<| D]S} tj t j || t j | | Ttj|| t|}t!d|d |d |d t#|d dS) u Allow inspection/modification of a metric script by copying it on local drive at local_path. Use `evaluate.inspect_evaluation_module` instead, from the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate Args: path (``str``): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'`` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``) e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'`` local_path (``str``): path to the local folder to copy the datset script to. download_config (Optional ``datasets.DownloadConfig``): specific download configuration parameters. **download_kwargs (additional keyword arguments): optional attributes for DownloadConfig() which will override the attributes in download_config if supplied. r:Fr<Tr=c<g|]}|d|Sr@rCrEs rr)z"inspect_metric..rGrNz"The processing scripts for metric rHrIzP. You can modify this processing scripts and use it with `datasets.load_metric("rJ)rrrKrLrMrNr8rFrOrPrQrRrSrTrUrrVrrW) r8r9r:rX metric_modulerZr[r\r]r^r_r`ras rinspect_metricrfs0*$cccSbccM#M$=uMMMK .{;;GOO,>??(*0E(F(F..$9gll:rwwH]/^/^__  K$////^^h^^^ ! _ _H Lgx88"',,{T\:]:] ^ ^ ^ ^----*:66J  T      5  YablYmYmYvYvYxYx   r data_files download_moderevisionuse_auth_tokenc bt}fd|DS)aGet the meta information about a dataset, returned as a dict mapping config name to DatasetInfoDict. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`datasets.list_datasets`]) e.g. `'squad'`, `'glue'` or``'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. use_auth_token (`str` or `bool`, *optional*): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If `True`, or not specified, will get token from `"~/.huggingface"`. **config_kwargs (additional keyword arguments): Optional attributes for builder class which will override the attributes if supplied. Example: ```py >>> from datasets import get_dataset_infos >>> get_dataset_infos('rotten_tomatoes') {'default': DatasetInfo(description="Movie Review Dataset. This is a dataset of containing 5,331 positive and 5,331 negative processed sentences from Rotten Tomatoes movie reviews...), ...} ``` )r8rir:rhrgrjc:i|]}|td|dS))r8 config_namergr:rhrirjr)get_dataset_config_info) r'rm config_kwargsrgr:rhr8rirjs r z%get_dataset_infos..sa     ,  #!+')         r)get_dataset_config_names)r8rgr:rhrirjro config_namess``````` rget_dataset_infosrss}X, '#% L          (    rdynamic_modules_pathc t|f|||||d|}t|j}t|jp|jddgS)aGet the list of available config names for a particular dataset. Args: path (`str`): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. `'./dataset/squad'` or `'./dataset/squad/squad.py'` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with [`datasets.list_datasets`]) e.g. `'squad'`, `'glue'` or `'openai/webtext'` revision (`Union[str, datasets.Version]`, *optional*): If specified, the dataset module will be loaded from the datasets repository at this version. By default: - it is set to the local version of the lib. - it will also try to load it from the main branch if it's not available at the local version of the lib. Specifying a version that is different from your local version of the lib might cause compatibility issues. download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, defaults to `REUSE_DATASET_IF_EXISTS`): Download/generate mode. dynamic_modules_path (`str`, defaults to `~/.cache/huggingface/modules/datasets_modules`): Optional path to the directory in which the dynamic modules are saved. It must have been initialized with `init_dynamic_modules`. By default the datasets and metrics are stored inside the `datasets_modules` module. data_files (`Union[Dict, List, str]`, *optional*): Defining the data_files of the dataset configuration. **download_kwargs (additional keyword arguments): Optional attributes for [`DownloadConfig`] which will override the attributes in `download_config` if supplied, for example `use_auth_token`. Example: ```py >>> from datasets import get_dataset_config_names >>> get_dataset_config_names("glue") ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax'] ``` )rir:rhrtrgrmdefault)rrrKlistbuilder_configskeysbuilder_kwargsget) r8rir:rhrtrgrXrYrZs rrqrq sp, '#1  N$N$>??K  +0022 3 3 t8U8Y8YZgir8s8s7ttrrmreturnc tf||||||d|}|j} | j|r|n t }|||_|t|j| fd| t|j|D| _n"#t$r} td| d} ~ wwxYw| S)a Get the meta information (DatasetInfo) about a dataset for a particular config Args: path (``str``): path to the dataset processing script with the dataset builder. Can be either: - a local path to processing script or the directory containing the script (if the script has the same name as the directory), e.g. ``'./dataset/squad'`` or ``'./dataset/squad/squad.py'`` - a dataset identifier on the Hugging Face Hub (list all available datasets and ids with ``datasets.list_datasets()``) e.g. ``'squad'``, ``'glue'`` or ``'openai/webtext'`` config_name (:obj:`str`, optional): Defining the name of the dataset configuration. data_files (:obj:`str` or :obj:`Sequence` or :obj:`Mapping`, optional): Path(s) to source data file(s). download_config (:class:`~download.DownloadConfig`, optional): Specific download configuration parameters. download_mode (:class:`DownloadMode` or :obj:`str`, default ``REUSE_DATASET_IF_EXISTS``): Download/generate mode. revision (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load. As datasets have their own git repository on the Datasets Hub, the default version "main" corresponds to their "main" branch. You can specify a different version than the default "main" by using a commit SHA or a git tag of the dataset repository. use_auth_token (``str`` or :obj:`bool`, optional): Optional string or boolean to use as Bearer token for remote files on the Datasets Hub. If True, or not specified, will get token from `"~/.huggingface"`. **config_kwargs (additional keyword arguments): optional attributes for builder class which will override the attributes if supplied. )namergr:rhrirjN) base_pathr:c0i|]}|j|jdS))r~ dataset_name)r~)r'split_generatorr8s rrpz+get_dataset_config_info..s:# $/CUY&Z&Zrz#   '#%     G >> from datasets import get_dataset_split_names >>> get_dataset_split_names('rotten_tomatoes') ['train', 'validation', 'test'] ``` )rmrgr:rhrirj)rnrwrry) r8rmrgr:rhrirjrors rget_dataset_split_namesrs[Z #  '#%     D    "" # ##r)TF)N)NNNNN)NNNNNN)1__doc__rLrNrSpathlibrtypingrrrrrr r+download.download_configr download.download_managerr #download.streaming_download_managerr rrloadrrrrutils.deprecation_utilsrutils.file_utilsr utils.loggingr utils.versionrrlogger ValueErrorrr,r5strrbrfboolrsrqrnrrrrrsJ "! AAAAAAAAAAAAAAAA444444333333IIIIIIhhhhhhhhhhhh//////777777%%%%%%"""""" H       *   @ ~!!!!H&&#&3&.AY&&&&R A&&&#&@X&&&&V48048<.215 @@ @tT3/0@n-@E,"345 @ uS'\*+ @ U49-. @@@@J/3048<*.37 BuBu BuuS'\*+Bun-BuE,"345 Bu #3- Bu tT3/0 BuBuBuBuN"&_c048<.215:: :#:sHSM73c8TW=FX@Y;Y3ZZ[\:n- : E,"345 : uS'\*+ :U49-.:::::~"&_c048<.2157$7$ 7$#7$sHSM73c8TW=FX@Y;Y3ZZ[\7$n- 7$ E,"345 7$ uS'\*+ 7$U49-.7$7$7$7$7$7$r