+gd~ ^ddlZddlmZddlmZmZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZdd lm Z e!ej"Z#ej$e%Z&Gdde!Z'Gdde(Z)dZ*ej"ddgej+gdej,gdiZ-dZ.ddgZ/ddgZ0ej"ej+ej,gZ1de1DZ2de1DZ3ej"dgiZ4e*gZ5e3e2e4gZ6gdZ7d Z8gd!Z9d"e!d#e:fd$Z;d%e ee e!fd#ee!e e e!d&fffd'Zd+ee!ge efd#ee!e e!ffd,Z?d+ee!ge efd#ee!e e!ffd-Z@ dBd.e!d"e!d/e e e!d#e efd0ZA dBd.e!d%e e!d/e e e!d#e e ee'ffd1ZBd.e!d#ee!e e!ffd2ZCd.e!d#e e!fd3ZD dCd4ejEjFd"e!d.e e!d/e eGd#e ef d5ZH dCd4ejEjFd%e e!d.e e!d/e eGd#e e'f d6ZId4ejEjFd.e!d#ee!e e!ffd7ZJd4ejEjFd.e!d#e e!fd8ZK dBd9e ee'fd:e e e:e!fd#e e!fd;ZL dDd=e e ee'fd:e e e:e!fd#e e!fd>ZMGd?d&e e ee'fZNGd@dAee!eNfZOdS)EN)partial)PathPurePath)CallableDictListOptionalSetTupleUnion)LocalFileSystem) thread_map) HfFileSystem)Split)logging)is_relative_path is_remote_url request_etag) hf_hub_urlstring_to_dictceZdZdS)UrlN__name__ __module__ __qualname__3lib/python3.11/site-packages/datasets/data_files.pyrrDr rceZdZdS)EmptyDatasetErrorNrrr r!r$r$r"r r$zFdata/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*traintraining) validationvaliddevval)testtestingeval evaluationz-._ 0-9z**[{sep}/]{keyword}[{sep}]*z{keyword}[{sep}]*z{keyword}[{sep}/]**z**[{sep}/]{keyword}[{sep}/]**c>i|]}|dt|DS)cZg|](}tD]}||t)S)keywordsep)"KEYWORDS_IN_FILENAME_BASE_PATTERNSformatNON_WORDS_CHARS.0r2patterns r! z.-O    9    wO<<    r SPLIT_KEYWORDSr8splits r! r@,K&&&     %e,   &&&r c>i|]}|dt|DS)cZg|](}tD]}||t)Sr1)"KEYWORDS_IN_DIR_NAME_BASE_PATTERNSr5r6r7s r!r:z.5r;r r<r>s r!r@r@4rAr z**)z metadata.csvz**/metadata.csvzmetadata.jsonlz**/metadata.jsonlz*[])z README.mdz config.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.jsonr9returncDtfdtDS)Nc3 K|]}|vV dSNr)r8wilcard_characterr9s r! z%contains_wildcards..Rs*YY0A G+YYYYYYr )anyWILDCARD_CHARACTERS)r9s`r!contains_wildcardsrMQs' YYYYEXYYY Y YYr patterns DataFilesListct|trd|DSt|tr t|giSt|t r t|iStt |iS)a/ Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list_of _atterns cdi|]-\}}t|t|tr|n|g.Sr)str isinstancelist)r8keyvalues r!r@z%sanitize_patterns.._s:kkkJCQVC:eT#:#:G%%kkkr )rSdictitemsrRSANITIZED_DEFAULT_SPLITrT)rNs r!sanitize_patternsrZUs(D!!9kkZbZhZhZjZjkkkk Hc " "9'(44 Hd # #9'22'h88r matched_rel_pathcdt|jjD}dt|jjD}t|t|kS)u When a path matches a pattern, we additionnally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False c<g|]}|d|S__ startswithr8parts r!r:z6_is_inside_unrequested_special_dir..s-"u"u"uD_c_n_nos_t_t"u4"u"u"ur c<g|]}|d|Sr^r`rbs r!r:z6_is_inside_unrequested_special_dir..s-%o%o%otY]YhYhimYnYn%od%o%o%or )rparentpartslen)r[r9data_dirs_to_ignore_in_pathdata_dirs_to_ignore_in_patterns r!"_is_inside_unrequested_special_dirrjhsg8#v"uH=M4N4N4U4["u"u"u%o%ox7H7H7O7U%o%o%o" * + +s3Q/R/R RRr cdt|jD}dt|jD}t|t|kS)u: When a path matches a pattern, we additionnally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False cdg|]-}|drt|dhk+|.S.rasetrbs r!r:zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..sJ"""T__S=Q=Q"Z]^bZcZchkglZlZlZlZlZlr cdg|]-}|drt|dhk+|.Srmrorbs r!r:zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..sJ%%%DOOC4H4H%QTUYQZQZ_b^cQcQcQcQcQcr )rrfrg)r[r9hidden_directories_in_pathhidden_directories_in_patterns r!?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirrtswj""!"2339"""%%!'**0%%%! ) * *c2O.P.P PPr pattern_resolvercx tD]dd}||}t|dkrbd|D}fd|D fdtDt t tz z}fd|DcSt D]{g}D]P\}} |D]7}||}t|dkr||n8A#t$rYMwxYw|rfd|DcS|td |d |) a+ Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rc6g|]}|Sr)as_posix)r8ps r!r:z,_get_data_files_patterns..s ;;;1!**,,;;;r c<h|]}t|dSr?r)r8rz split_patterns r! z+_get_data_files_patterns..s(^^^aq- @ @ I^^^r c6g|]}|vt|Sr)rR)r8r?splitss r!r:z,_get_data_files_patterns..s%WWWEuPVSZZr c@i|]}||gS)r|)r5)r8r?r}s r!r@z,_get_data_files_patterns..s/ZZZ5EM00u0==>ZZZr c"i|] }|| Srr)r8r? patterns_dicts r!r@z,_get_data_files_patterns..s NNNEE=/NNNr Couldn't resolve pattern  with resolver ) ALL_SPLIT_PATTERNSreplacergDEFAULT_SPLITSsortedrpALL_DEFAULT_PATTERNSrXappendFileNotFoundError) rur9 data_files sorted_splitsnon_empty_splitsr?rNrr}rs @@@r!_get_data_files_patternsrs, [ [ '' 377%%g.. z??Q  ;; ;;;J^^^^S]^^^FWWWW^WWWZ`^,,,[[M[ZZZMZZZ Z Z Z . O O ,2244  OE8 'G!1!1'!:!:J:**(//666+%      ONNNN=MNNN N N N O bbbP`bb c ccs :D DDcg}tD]F} ||}t|dkr||7#t$rYCwxYw|r|Std|d|)zM Get the supported metadata patterns from a directory or repository. rrr)METADATA_PATTERNSrgrr)runon_empty_patternsr9metadata_filess r!_get_metadata_files_patternsrs$ --g66N>""Q&&"))'222     D "!! bbbP`bb c ccs3A A A base_pathallowed_extensionscftr!tjn2tjdtjzt fdD}fd|D}fd|D}t|t|krntt|t|z }t ddtd|n|}|s\tsMd dt}|d tz }t!|t#|S) z Return the absolute paths to all the files that match the given patterns. It also supports absolute paths in patterns. If an URL is passed, it is returned as is. rcXg|]&}|t|'Srisfilerr8filepathfss r!r:z3_resolve_single_pattern_locally..s3\\\ RZH[H[\(##\\\r c g|]}|jtvstj|jk-ttj|tjyttj|tjttj |Sr) nameFILES_TO_IGNORErrjospathrelpathrtrabspathr8rrr9s r!r:z3_resolve_single_pattern_locally.. s     M 0 0HW4E4E4Jhm4[4[2 GOOHi 0 0"'//'92U2U  5\P GOOHi 0 0"'//'92U2U  5\ RW__X & &''4[4[4[r NcTg|]$}tfd|jD"|%S)c30K|]}|ddvVdSrNrr8suffixrs r!rJz=_resolve_single_pattern_locally...1TT6!"":!33TTTTTTr rKsuffixesr8rrs r!r:z3_resolve_single_pattern_locally..O   TTTT(BSTTTTT    r z Some files matched the pattern '' at , but don't have valid data file extensions: zUnable to find ' with any supported extension )rrrjoin splitdriver3r globrgrTrploggerinforresolverMrr) rr9r glob_iter matched_pathsoutinvalid_matched_files error_msgrs ``` @r!_resolve_single_pattern_locallyrs  <',,y'22G&&w//2RV;   B\\\\BGGG4D4D\\\I     !   M%    )    s88c-(( ( ($(]););c#hh)F$G$G ! KK`7``iAXAXAZAZ``I^``    +)'22+PwPPT)__5L5L5N5NPP  ) T$?Q:R:RTT TI *** #;;r chg}|D]]}t|r#|t|4t|||D]}||^|sMd|dt |}||dt |z }t||S)a Resolve the paths and URLs of the data files from the patterns passed by the user. URLs are just returned as is. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. Here are some behaviors specific to fsspec glob that are different from glob.glob, Path.glob, Path.match or fnmatch: - '*' matches only first level items - '**' matches all items - '**/*' matches all at least second level items More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicilty mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_patterns_locally_or_by_urls >>> base_path = "." >>> resolve_patterns_locally_or_by_urls(base_path, ["src/**/*.yaml"]) [PosixPath('/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/resources/readme_structure.yaml')] Args: base_path (str): Base path to use when resolving relative paths. patterns (List[str]): Unix patterns or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=["csv", "json", "txt", "parquet"] Returns: List[Union[Path, Url]]: List of paths or URLs to the local or remote files that match the patterns. z.Unable to resolve any data file that matches 'rNr)rrrrrrrTr)rrNrrr9rrs r!#resolve_patterns_locally_or_by_urlsr)s^J((  ! ! (   c'll + + + +7 7L^__ ( (!!$'''' ( +oXooTXYbTcTcTkTkTmTmoo  ) T$?Q:R:RTT TI *** r ctt|} t|S#t$rt d|ddwxYw)u Get the default pattern from a directory testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. Some examples of supported patterns: Input: my_dataset_repository/ ├── README.md └── dataset.csv Output: {"train": ["**"]} Input: my_dataset_repository/ ├── README.md ├── train.csv └── test.csv my_dataset_repository/ ├── README.md └── data/ ├── train.csv └── test.csv my_dataset_repository/ ├── README.md ├── train_0.csv ├── train_1.csv ├── train_2.csv ├── train_3.csv ├── test_0.csv └── test_1.csv Output: {"train": [**train*], "test": ["**test*"]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train/ │ ├── shard_0.csv │ ├── shard_1.csv │ ├── shard_2.csv │ └── shard_3.csv └── test/ ├── shard_0.csv └── shard_1.csv Output: {"train": ["**train*/**"], "test": ["**test*/**"]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train-00000-of-00003.csv ├── train-00001-of-00003.csv ├── train-00002-of-00003.csv ├── test-00000-of-00001.csv ├── random-00000-of-00003.csv ├── random-00001-of-00003.csv └── random-00002-of-00003.csv Output: { "train": ["data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"], "test": ["data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"], "random": ["data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"], } In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. The directory at z doesn't contain any data filesN)rrrrr$rresolvers r!get_data_patterns_locallyrhseh6 BBHj'111 jjj ^I ^ ^ ^__eiij &Actt|} t|S#t$rtd|ddwxYw)zE Get the supported metadata patterns from a local directory. rz" doesn't contain any metadata fileN)rrrrrs r!get_metadata_patterns_locallyrsd6 BBHm+H555 mmm aI a a abbhllmr dataset_infoc t| rdnd fd tD}fd|D}fd|D}t |t |krTt t |t |z }tdd|j d|n|}|sBts3d d|j }|d t z }t|t|S) N) repo_info/cXg|]&}|t|'Srrrs r!r:zA_resolve_single_pattern_in_dataset_repository..s6qqq]_]f]fgo]p]pq(##qqqr c g|]}|jtvstj|jk-ttj|tjyttj|tj|Sr)rrrrjrrrrtrs r!r:zA_resolve_single_pattern_in_dataset_repository..s     M 0 0HW4E4E4Jhm4[4[2 GOOHi 0 0"'//'92U2U  5\P GOOHi 0 0"'//'92U2U  5\ 4[4[4[r cTg|]$}tfd|jD"|%S)c30K|]}|ddvVdSrrrs r!rJzK_resolve_single_pattern_in_dataset_repository...rr rrs r!r:zA_resolve_single_pattern_in_dataset_repository..rr zSome files matched the pattern  in dataset repository rzUnable to find r) rrrryrgrTrprridrMrr) rr9rrrrrrrrs ``` @r!-_resolve_single_pattern_in_dataset_repositoryrs  - - -B**** qqqqBGGHW