+gd [UddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z mZmZmZmZmZmZddlZddlZddlZddlmZddlmZddlmZmZm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'm(Z(e rdd l)m*Z*e%e+Z,d a-da.ede/d<da0eej1e/d<GddZ2dZ3dZ4dZ5dZ6e#dde7fdZ8de7fdZ9de:fdZ;dZ<GddZ=ee>$) $ $ % M$) $ $ $ $ $ % %r/cd|jr|dSdSr)r%detachr$r6s r-cleanupz!_TempDirWithCustomCleanup.cleanupDs3 ? ! ! # #  MMOOOOO  r/r)__name__ __module__ __qualname____doc__r.r$r9r/r-rr1sP 8888%%% r/ctdSttjat d|jDrt|dSdS)a This function registers the datasets that have cache files in _TEMP_DIR_FOR_TEMP_CACHE_FILES in order to properly delete them before deleting the temporary directory. The temporary directory _TEMP_DIR_FOR_TEMP_CACHE_FILES is used when caching is disabled. Nc3|K|]7}ttjt|djvV8dS)filenameN)rrr!parents).0 cache_files r- z?maybe_register_dataset_for_temp_dir_deletion..UsW    + 011T*Z:P5Q5Q5YY      r/)rrr"WeakSetany cache_filesadd)datasets r-,maybe_register_dataset_for_temp_dir_deletionrKIs| &-(/+2?+<+<(   !-   6 ),,W55555 66r/c<tttngSr)rlistr>r/r-(get_datasets_with_cache_file_in_temp_dirrN\s5U5a40 1 1 1giir/c dadS) When applying transforms on a dataset, the data are stored in cache files. The caching mechanism allows to reload an existing cache file if it's already been computed. Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated after each transform. If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets. More precisely, if the caching is disabled: - cache files are always recreated - cache files are written to a temporary directory that is deleted when session closes - cache files are named using a random hash instead of the dataset fingerprint - use [`~datasets.Dataset.save_to_disk`] to save a transformed dataset or it will be deleted when session closes - caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use the `download_mode` parameter in [`~datasets.load_dataset`]. TN_CACHING_ENABLEDr>r/r-enable_cachingrS`s$r/c dadS)rPFNrQr>r/r-disable_cachingrUus$r/zUse datasets.enable_caching() or datasets.disable_caching() instead. This function will be removed in a future version of datasets.booleanc$t|adS)a When applying transforms on a dataset, the data are stored in cache files. The caching mechanism allows to reload an existing cache file if it's already been computed. Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated after each transform. If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets. More precisely, if the caching is disabled: - cache files are always recreated - cache files are written to a temporary directory that is deleted when session closes - cache files are named using a random hash instead of the dataset fingerprint - use :func:`datasets.Dataset.save_to_disk` to save a transformed dataset or it will be deleted when session closes - caching doesn't affect :func:`datasets.load_dataset`. If you want to regenerate a dataset from scratch you should use the ``download_mode`` parameter in :func:`datasets.load_dataset`. NboolrR)rVs r-set_caching_enabledrZs*G}}r/returnc*ttS)a When applying transforms on a dataset, the data are stored in cache files. The caching mechanism allows to reload an existing cache file if it's already been computed. Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated after each transform. If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets. More precisely, if the caching is disabled: - cache files are always recreated - cache files are written to a temporary directory that is deleted when session closes - cache files are named using a random hash instead of the dataset fingerprint - use [`~datasets.Dataset.save_to_disk`]] to save a transformed dataset or it will be deleted when session closes - caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use the `download_mode` parameter in [`~datasets.load_dataset`]. rXr>r/r-is_caching_enabledr]s$  ! !!r/cNtd}t|atjS)z7Return a directory that is deleted when session closes.NcPtD]}|dSr)rN__del__)dsets r-r*z9get_temporary_cache_files_directory..cleanup_funcs0@BB     r/r*)rrr!rbs r-#get_temporary_cache_files_directoryrcs7&-   *CP\)])])]& ) ..r/cfd}|S)Nc0D]}|tj|<|Sr)Hasherdispatch)functtypess r-proxyzhashregister..proxys& & &A!%FOA   r/r>)rjrks` r- hashregisterrls# Lr/ceZdZUdZiZeed<dZede e e e fde fdZ edede fdZedede fdZdedd fd Zde fd Zd S) rfz-Hasher that accepts python objects as inputs.rgc6tj|_dSr)xxhashxxh64mr6s r-r.zHasher.__init__sr/valuer[ct|tr|gn|}tj}|D]}|||Sr) isinstancebytesrorpupdate hexdigest)clsrrrqxs r- hash_byteszHasher.hash_bytessU%eU33> LNN  A HHQKKKK{{}}r/cF|t|Sr)rzrrxrrs r- hash_defaultzHasher.hash_defaults~~eEll+++r/ct||jvr$|jt|||S||Sr)typergr}r|s r-hashz Hasher.hashsH ;;#, & &,3<U ,S%88 8##E** *r/Nc dt|d}||}|j|d|j|ddS)Nz==utf8utf-8)rrrqrvencode)r)rrheader_for_updatevalue_for_updates r-rvz Hasher.updatesw0e00099U++  '..v66777  &--g6677777r/c4|jSr)rqrwr6s r-rwzHasher.hexdigestsv!!!r/)r:r;r<r=rgr__annotations__r. classmethodr rur strrzrr}rrvrwr>r/r-rfrfs77Hd   uUDK%78S[,,,,,[,+++++[+ 8C8D8888 "3""""""r/rfcfddfdtjDdS)Nct|tjr$d|jDS|dS)Nc3dK|]+}|dV,dS)rN) to_stringr)rCcs r-rEz9_hash_pa_table.._hash_pa_array..s8$Y$YqQ[[]]%9%9'%B%B$Y$Y$Y$Y$Y$Yr/r)rtpa ChunkedArrayrzchunksrr)rrhashers r-_hash_pa_arrayz&_hash_pa_table.._hash_pa_arraysi eR_ - - H$$$Y$YEL$Y$Y$YYY Y$$U__%6%6%=%=g%F%FGG Gr/-c3FK|]}|dz|zVdS)rNr>)rCcolrrrs r-rEz!_hash_pa_table..s:bbS3Yc !;!;;bbbbbbr/r)joinsorted column_namesrzr)rrrrs``@r-_hash_pa_tablersyHHHHH HHbbbbbveN`GaGabbb b bE   U\\'22 3 33r/c|tjt|ddS)NT) sort_keysr)rzjsonrrr)rrrs r-_hash_dataset_infor s9   TZu FFFMMgVV W WWr/fingerprint_warningscj|j}t}t|D]9}|dkr |||||:|jD]:}|t j|d;|S)N _fingerprintrA) __dict__rfrrvrHr1r2getmtimerw)rJstaterkeyrDs r-generate_fingerprintrs  E XXFe}}"" .  c eCj!!!!)@@  bg&&z*'=>>????     r/@c<tj|d|dzdS)N0ry)random getrandbits)nbitss r-generate_random_fingerprintr$s( '' 6%( 6 6 6 66r/c t}|| ||n#trctdds)t d|ddtd<n=t d|dnt d|dtcYSxYwt|D]}|| |||4#tr{tdds5t d|d ||d |ddtd<nUt d|d ||d |dn*t d|d ||d |dtcYcSxYw| S) N(update_fingerprint_transform_hash_failedFz Transform a couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.Tz= couldn't be hashed properly, a random hash was used instead.zn couldn't be hashed properly, a random hash was used instead. This doesn't affect caching since it's disabled.z Parameter 'z'=z of the transform ) rfrvrRrgetloggerwarninginforrrw) fingerprint transformtransform_argsrrs r-update_fingerprintr(s XXF MM+- i    -  '++,VX]^^ seeee TX$%OPP qqqqrrrr KKWYWWW   +,,,,,n%%11 c 1 MM.- . . . . 1 +//0Z\abb NNicii^C-@iiT]iii X\()STTKK]c]]^C-@]]T]]]] J#JJ)<JJPYJJJ/00 0 0 0 0 0     s;BC?DB) VAR_KEYWORDr!rCps r- z1format_kwargs_for_fingerprint..s)dddQQRVWVcQcQc!&QcQcQcr/r Nc$i|] \}}|v || Sr>r>)rCkvrs r- z1format_kwargs_for_fingerprint..s+!e!e!e41aUVZdUdUd!QUdUdUdr/c$i|] \}}|v || Sr>r>)rCrrrs r-rz1format_kwargs_for_fingerprint..s+!l!l!l41aUV^kUkUk!QUkUkUkr/seed generatoriprcRi|]$}|jtjk|j|j%Sr>)defaultinspect_emptyr!rs r-rz1format_kwargs_for_fingerprint..s7RSR[_f_mRmRm RmRmRmr/)copyr signature parametersvaluesrvzipnextiteritemsrnpr get_state default_rngpop)rhrrrrrkwargs_for_fingerprintparams_rposdefault_valuesdefault_varname default_values `` r-format_kwargs_for_fingerprintrxs$$[[]]  dd'"3D"9"9"D"K"K"M"MdddABBx%%c&$&7&78888 " g'--899 : : f!e!e!e!e3I3O3O3Q3Q!e!e!em!l!l!l!l3I3O3O3Q3Q!l!l!lN ! % %f - - 5:P:T:TU`:a:a:i!y2244 AtS1 #c 499tAwD24)2G2G2M2M "; /#*#4T#:#:#E#L#L#N#NN+9*>*>*@*@88& 4 4 49OP_9`dq9q9q " & & 7 7 7 !!r/inplacefingerprint_namesc24ttstdt4ttstdtrrtdndgfd}|S)a Wrapper for dataset transforms to update the dataset fingerprint using ``update_fingerprint`` Args: inplace (:obj:`bool`): If inplace is True, the fingerprint of the dataset is updated inplace. Otherwise, a parameter "new_fingerprint" is passed to the wrapped method that should take care of setting the fingerprint of the returned Dataset. use_kwargs (:obj:`List[str]`, optional): optional white list of argument names to take into account to update the fingerprint to the wrapped method that should take care of setting the fingerprint of the returned Dataset. By default all the arguments are used. ignore_kwargs (:obj:`List[str]`, optional): optional black list of argument names to take into account to update the fingerprint. Note that ignore_kwargs prevails on use_kwargs. fingerprint_names (:obj:`List[str]`, optional, defaults to ["new_fingerprint"]): If the dataset transforms is not inplace and returns a DatasetDict, then it can require several fingerprints (one per dataset in the DatasetDict). By specifying fingerprint_names, one fingerprint named after each element of fingerprint_names is going to be passed. randomized_function (:obj:`bool`, defaults to False): If the dataset transform is random and has optional parameters "seed" and "generator", then you can set randomized_function to True. This way, even if users set "seed" and "generator" to None, then the fingerprint is going to be randomly generated depending on numpy's current state. In this case, the generator is set to np.random.default_rng(np.random.get_state()[1][0]). version (:obj:`str`, optional): version of the transform. The version is taken into account when computing the fingerprint. If a datase transform changes (or at least if the output data that are cached changes), then one should increase the version. If the version stays the same, then old cached data could be reused that are not compatible with the new transform. It should be in the format "MAJOR.MINOR.PATCH". Nz)use_kwargs is supposed to be a list, not z,ignore_kwargs is supposed to be a list, not z5fingerprint_names are only used when inplace is Falsenew_fingerprintc\s*tfdDstdrBdjjvrtdddjjvrtddt t fd }d |_|S) Nc34K|]}|jjvVdSr)__code__ co_varnames)rCr!rhs r-rEz>fingerprint_transform.._fingerprint..s-"c"c44=+D#D"c"c"c"c"c"cr/zFfunction {func} is missing parameters {fingerprint_names} in signaturerz'seed' must be in z 's signaturerz'generator' must be in )rc t|| }|r|d}|dd}nF|ttt jj} rt|j |}nND]K}| |||d<t|j |||<6t||L|g|Ri|} r||_|S)N)rrrrr fingerprint_name) rrrrrrrrrrr)rrrrJrroutrrhrrrrrs r-wrapperz._fingerprint..wrappersC%B%+$7 &&& " ^#'7ABBx#)::d48I$8O8O8Z3[3[.\.\#]#] G"4W5I9Vl"m"m(9GG$zz"233;EU./AB3E#0)=S44/00-V4D-EFFFF$w000000C 7'6$Jr/r)allrrrrr_decorator_name_) rhrrrrrrrrs ` @r-rz+fingerprint_transform.._fingerprints  gs"c"c"c"cQb"c"c"ccc geff f  OT]666 !Hd!H!H!HIII$-";;; !M4!M!M!MNNN4T7KKK t& & & & & & & & & &  & P$1 r/)rtrMrr)rrrrrrrs`````` r-fingerprint_transformrsFjT&B&BWT*EUEUWWXXX M4)H)H ZZHXHXZZ[[[R$RPQQQ->-J))QbPc6666666666p r/)rr)NNF)NNNFN)Hrrr1rr4rr" functoolsrpathlibrtypingrrrrr r r r numpyrpyarrowrrorrnamingrtablerrrrutils.deprecation_utilsr utils.loggingrutils.py_utilsrr arrow_datasetrr:rrRrrrrFrrKrNrSrUrYrZr]rrcrlrfrrrrrrrrrrr>r/r-r s SSSSSSSSSSSSSSSSSSSS 666666NNNNNNNNNNNN//////%%%%%%))))))))'&&&&&& H   HL)D ELLL>B (7?";BBB0666&jjj** J%%%%%*"D""""* /S / / / /&""""""""""""""""Tbh}.?ASTT44UT4kXXX)+d39o*** S    77S7777///d  c    (8hsmWZ'+)- % *"*" *" *" cN*"c# *" DI& *"  *" #s(^*"*"*"*"^'+)--1 %! ff fc#fDI&f S * f  f c] ffffffr/