+gd^dZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZddlmZddlmZmZmZddlmZddlmZdd l mZmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%ddl&Z&ddl'Z'ddl(Z'ddl)Z*dd l+m,Z,dd l-m.Z.ddl/m0Z0ddl/m1Z1 ddl2Z3ddl2m4Z4m5Z5n#e6$r dxZ3xZ5Z4YnwxYwe1j7e8Z9ej:Z;dZfde=fdZ?de>de>dee>e>ffdZ@dZAe dZBe dUde=fdZCdZDd ZEd!ZFd"ZGGd#d$eHZIGd%d&eJZKd'ZL dVd)eegefd*ed+eMd,eMd-eMd.eMd/e e=d0e=d1e eNd2eMd3e e>defd4ZOGd5d6ZPdWd8ZQd9e>de!e>e e>ffd:ZRd;e>de!e>e>e>e>ffd<ZSGd=d>e&jTZTd?ZUe d@ZVdAZWdBZXe0jYe,jZdCkreXedDZ[n?e0jYj\ddEe,jZdCj\kreXedFZ]e0jYe,jZdGkreXedHZ^ne0jYj\ddEe,jZdGj\kreXedIZ^n?e0jYj\ddEe,jZdCj\kreXedJZ^dKZ_e"dLZ`dMejadNedOee`fdPeHde=fdQZbdRe#ejcje'jcjfdNedOee`fdSeeHdee`fdTZddS)Xz)Some python utils function and classes. N)contextmanager)fields is_dataclass)BytesIO)ManagerPoolRLock)Empty) disk_usage)CodeType FunctionType) AnyCallableDictIterableListOptionalTupleTypeVarUnion)urlparse)version)tqdm)config)logging)FinalLiteralc|sdSgd}t|}|D]\}}||z }|dkr |dd|cSt|dS)a6Returns a human readable size string. If size_in_bytes is None, then returns "Unknown size". For example `size_str(1.5 * datasets.units.GiB) == "1.50 GiB"`. Args: size_in_bytes: `int` or `None`, the size, in bytes, that we want to format as a human-readable size string. z Unknown size))PiB )TiB)GiB@)MiB)KiBg?z.2f z bytes)floatint) size_in_bytes _NAME_LISTname size_bytesvalues 7lib/python3.11/site-packages/datasets/utils/py_utils.pysize_strr4Cs ~aaaJ-((M&))j * C<<((($(( ( ( ( -  ( ( ((sizereturnct|tr|S|drt|dddzS|drt|dddzS|drt|dddzS|d rt|ddd zS|d rt|ddd zS|d r6t|dddz}|dr|dzn|S|dr6t|dddz}|dr|dzn|S|dr6t|dddz}|dr|dzn|S|dr6t|dddz}|dr|dzn|S|dr6t|dddz}|dr|dzn|St d|d)a) Converts a size expressed as a string with digits an unit (like `"50MB"`) to an integer (in bytes). Args: size (`int` or `str`): The size to convert. Will be directly returned if an `int`. Example: ```py >>> convert_file_size_to_int("1MiB") 1048576 ``` PIBNr"TIBr$GIBr&MIBr(KIBr*PBlI5bTBlJ)GBiʚ;MBi@BKBiz`size=zM` is not in a valid format. Use an integer followed by the unit, e.g., '5GB'.) isinstancer-upperendswith ValueError)r6int_sizes r3convert_file_size_to_intrL[s$  zz||U##(49~~'' zz||U##(49~~'' zz||U##(49~~'' zz||U##(49~~'' zz||U##(49~~'' zz||T""AtCRCy>>V, $ c 2 2@x1}}@ zz||T""AtCRCy>>V, $ c 2 2@x1}}@ zz||T""AtCRCy>>U+ $ c 2 2@x1}}@ zz||T""AtCRCy>>U+ $ c 2 2@x1}}@ zz||T""AtCRCy>>U+ $ c 2 2@x1}}@ qdqqq r rrr5stringpatternc0tjdd|}tj||}|td|d|t |}tjd|}tt||}|S)aUn-format a string using a python f-string pattern. From https://stackoverflow.com/a/36838374 Example:: >>> p = 'hello, my name is {name} and I am a {age} year old {what}' >>> s = p.format(name='cody', age=18, what='quarterback') >>> s 'hello, my name is cody and I am a 18 year old quarterback' >>> string_to_dict(s, p) {'age': '18', 'name': 'cody', 'what': 'quarterback'} Args: string (str): input string pattern (str): pattern formatted like a python f-string Returns: Dict[str, str]: dictionary of variable -> value, retrieved from the input using the pattern Raises: ValueError: if the string doesn't match the pattern z{(.+?)}z (?P<_\1>.+)NzString z doesn't match the pattern ) resubsearchrJlistgroupsfindalldictzip)rMrNregexresultvalueskeys_dicts r3string_to_dictr]s. F:~w 7 7E Yuf % %F ~O6OOgOOPPP &--// " "F :j' * *D T6"" # #E Lr5cdfdt|ts|st|d|S)zbConvert an object to its dictionary representation recursively. cLt|ot|t SN)rrGtypeobjs r3_is_dataclass_instancez&asdict.._is_dataclass_instances"C  >C)>)>%>>r5c|rki}t|D]W}t||j}|jr&||jks|jddr |||j<X|St|tr.t|drt|fd|DSt|ttfr$t|fd|DSt|tr fd| DStj|S)N$include_in_asdict_even_if_is_defaultF_fieldsc&g|] }|Sri.0v _asdict_inners r3 z1asdict.._asdict_inner..s#===A}}Q//===r5c3.K|]}|VdSr`rirjs r3 z0asdict.._asdict_inner..s-;;!]]1--;;;;;;r5c@i|]\}}||Sriri)rkkrlrms r3 z1asdict.._asdict_inner..s3OOO41aMM!$$mmA&6&6OOOr5)rgetattrr0initdefaultmetadatagetrGtuplehasattrrarSrVitemscopydeepcopy)rcrYfr2rmrds r3rmzasdict.._asdict_inners[ ! !# & & &FC[[ + +% gc16&:&:;;v+!)!3!3qz~~Flns7t7t!3%*F16NM U # # &Y(?(? &499=======> > dE] + + &499;;;;s;;;;; ; T " " &OOOO399;;OOO O=%% %r5z is not a dict or a dataclass)rGrV TypeError)rcrmrds @@r3asdictrs???&&&&&&* c4 ?)?)?)D)D?3===>>> =  r5c#Kt||d}t||| dVt|||dS#t|||wxYw)z%Temporarily assign obj.attr to value.N)rtsetattr)rcattrr2originals r3temporary_assignmentrsisD$''H Cu% T8$$$$$T8$$$$s =AFseedc#Ktj}tj||rtjrddl}|j}|j||j r3|j }|j ||rtj rddl}ddlm}|j} |jj|} |j| |st-d|} | j} t1| d} | r| j}| | dVtj||rYtjrM|j||j r|j ||rJtj r@|j| | | _| r || _dSt=| ddSdSdS#tj||rYtjrM|j||j r|j ||rHtj r=|j| | | _| r|| _wt=| dwwwxYw)zUTemporarily set the random seed. This works for python numpy, pytorch and tensorflow.rN)contextzBSetting random seed for TensorFlow is only available in eager mode_rng)nprandom get_staterrTORCH_AVAILABLEtorch get_rng_state manual_seedcuda is_availableget_rng_state_allmanual_seed_all TF_AVAILABLE tensorflowtensorflow.pythonrget_global_generator Generator from_seedset_global_generatorexecuting_eagerlyrJ_seedrzr_set_global_seed set_state set_rng_stateset_rng_state_alldelattr)r set_pytorchset_tensorflownp_stater torch_statetorch_cuda_statestf tfpycontexttf_statetemp_gen tf_contexttf_seedtf_rng_initializedtf_rngs r3 temp_seedrsOy""$$HINN4-v-- l0022    &&& : " " $ $ - % < < > >  J & &t , , ,*&-*<<<<<<911339&0066 &&x000##%% cabb b ((** "$Z88  %_F##D))),  H%%%  @61 @ L & &{ 3 3 3z&&(( @ ,,->???  ,f1 , I * *8 4 4 4&J ! ,"(  F+++++ , , , , H%%%  @61 @ L & &{ 3 3 3z&&(( @ ,,->???  ,f1 , I * *8 4 4 4&J ! ,"(  F++++ , ,s (I66CL=c#jKt}|D]}||vr|||V dS)z=Iterate over iterable and return only unique values in order.N)setadd)rZseenr2s r3 unique_valuesrsL 55D    HHUOOOKKKr5cfd}|S)z4If the value is None, return None, else call `func`.c"| |ndSr`ri)r2funcs r3wrapperz'no_op_if_value_is_null..wrappers#/ttE{{{T9r5ri)rrs` r3no_op_if_value_is_nullrs#::::: Nr5c@t|D] \}}|||fcSdS)zwReturn the index and the value of the first non-null value in the iterable. If all values are None, return -1 as index.N)N) enumerate)iterableir2s r3first_non_null_valuer"s:h''5  e8OOO  8r5c'Kttj|D]!tfd|DfV"dS)z9Iterate over items of dictionaries grouped by their keys.c3(K|] }|V dSr`ri)rkdkeys r3rpzzip_dict...s'//A3//////r5N)r itertoolschainry)dictsrs @r3zip_dictr*s`Y_e455005//////////////00r5c6eZdZdZfdZfdZfdZxZS)NonMutableDictzDict where keys can only be added but not modified. Will raise an error if the user try to overwrite one key. The error message can be customized during construction. It will be formatted using {key} for the overwritten key. c|dd|_|rtdtj|i|dS)N error_msgz$Try to overwrite existing key: {key}z1NonMutableDict cannot be initialized with kwargs.)pop _error_msgrJsuper__init__)selfargskwargs __class__s r3rzNonMutableDict.__init__9sZ **  2    RPQQ Q$)&)))))r5c||vr(t|j|t||S)Nr)rJrformatr __setitem__)rrr2rs r3rzNonMutableDict.__setitem__BsG $;;T_333<<== =ww""3...r5ctfd|DrEtjt t |zt |S)Nc3 K|]}|vV dSr`ri)rkrrrs r3rpz(NonMutableDict.update..Hs'((QqDy((((((r5r)anyrJrrrrupdate)rotherrs` r3rzNonMutableDict.updateGsr ((((%((( ( ( QT_33D CJJ8N3OOPP Pww~~e$$$r5)__name__ __module__ __qualname____doc__rrr __classcell__)rs@r3rr1st*****///// %%%%%%%%%r5rceZdZdZddZdS) classpropertyz5Descriptor to be used as decorator for @classmethods.NcH|jd|Sr`)fget__get__)rrcobjtypes r3rzclassproperty.__get__Ps"/ty  w//111r5r`)rrrrrrir5r3rrMs.??222222r5rc |\ } }}}t|tst| s  |S|4tjtjkrtj|7|s5t dtjDrtdddt|tr| n|}|||dznddzt|zn|}tj|||d| 5}t|tr fd |DcdddS fd |D}t|tr|cdddSt|trt|cdddStj|cdddS#1swxYwYdS) zEApply a function recursively to each element of a nested data struct.Nc3(K|] }d|jvVdS)notebookNr)rktqdm_clss r3rpz%_single_map_nested..as+4r4rYaZ8CT5T4r4r4r4r4r4rr5r+T)endflush#rc)disablepositionunitdescc Bi|]\}}|t|dddfSNT_single_map_nested)rkrrrlfunctiontypess r3rsz&_single_map_nested..is8cccVZVWYZA)8QtT4*PQQcccr5c :g|]}t|dddfSrr)rkrlrrs r3rnz&_single_map_nested..ks0___UV((AudD$)OPP___r5)rGrVr get_verbosityWARNINGset_verbosity_warningrr__mro__printr{strrSryrarray) r data_structrank disable_tqdmr pbar_iterable pbar_descpbarmappedrrs @@r3rrTs=A:Hk5$ d k4 ( (%K1O1O%x $$$ G133goEE%'''  4r4reieq4r4r4r1r1r cr&&&&,6k4+H+HYK%%'''kMNRN^t/R3>TJJdhI m\Du[d e e e (im k4 ( ( (ccccc^bccc ( ( ( ( ( ( ( (`____Z^___F+t,, ( ( ( ( ( ( ( ( (K// (V}} ( ( ( ( ( ( ( (x'' ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (s$$G%G2#G"GG GTrr dict_onlymap_list map_tuple map_numpynum_procparallel_min_lengthrrrc h]g|sJ|rjt|rjt|rjtjtt |t st |s |S| ptj } t |t r!t| n|} |d}|dkst| |kr&fdtj | | | D} n|t| kr|nt| }g} t|D]o}t| |z}t| |z}||zt||z}||z||krdndz}| | |||| | fpt| td| Dkr9tdt| dtd | Dt d |d t| d d | Dd\}}| st%ftj}}t)|||5}|t,| } dddn #1swxYwYt d|dd| D} t dt| dt |t r/t t/|| St |tr| St |trt| Stj| S)aApply a function recursively to each element of a nested data struct. Use multiprocessing if num_proc > 1 and the length of data_struct is greater than or equal to `parallel_min_length`. Before version 2.5.0, multiprocessing was not used if `num_proc` was greater than or equal to ``len(iterable)``. Now, if `num_proc` is greater than or equal to ``len(iterable)``, `num_proc` is set to ``len(iterable)`` and multiprocessing is used. Args: function (`Callable`): Function to be applied to `data_struct`. data_struct (`Any`): Data structure to apply `function` to. dict_only (`bool`, default `False`): Whether only apply `function` recursively to `dict` values in `data_struct`. map_list (`bool`, default `True`): Whether also apply `function` recursively to `list` elements (besides `dict` values). map_tuple (`bool`, default `False`): Whether also apply `function` recursively to `tuple` elements (besides `dict` values). map_numpy (`bool, default `False`): Whether also apply `function` recursively to `numpy.array` elements (besides `dict` values). num_proc (`int`, *optional*): Number of processes. parallel_min_length (`int`, default `2`): Minimum length of `data_struct` required for parallel processing. types (`tuple`, *optional*): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`, default `True`): Whether to disable the tqdm progressbar. desc (`str`, *optional*): Prefix for the tqdm progressbar. Returns: `Any` Nrc :g|]}t|dddfSrr)rkrcrrs r3rnzmap_nested..s>    #udD$G H H   r5)rrrc3@K|]}t|dVdSrNlenrkrs r3rpzmap_nested..s,>>aAaD >>>>>>r5zHError dividing inputs iterable among processes. Total number of objects z , length: c3@K|]}t|dVdSrrrs r3rpzmap_nested..s,==Qs1Q4yy======r5z Spawning z processes for z objects in slices of c8g|]}t|dS)rrrs r3rnzmap_nested..s)VuVuVuefWZ[\]^[_W`W`VuVuVur5)NN)initargs initializerz Finished z processescg|] }|D]}| Sriri)rkproc_resrcs r3rnzmap_nested..s%AAA(AA#AAAAr5z Unpacked z objects)appendrSryrndarrayrGrVris_progress_bar_enabledrZrrrangeminsumrJloggerinfor set_lockrmaprrWr[r)rrr r r rrrrrrrr  split_kwdsindexdivmodstartrrrpools` ` r3 map_nestedr.tsAd } ) # T""" $ U### ) RZ(((e  k4 ( (%K1O1O%x $$$Hw'F'H'H#HL-7 T-J-J[tK&&(()))P[H1}}H (;;;     |HlNNN    (3x==8888c(mm 8__ a aEh--8+Ch--(*C%K#eS//1E#+eckkq9C   x%))>:>>>>> > >@+.x==@@==*=====@@   w w wX w wVuVujtVuVuVu w w   !++ >%*WWJ kH (X; G G G >4XX0*==F > > > > > > > > > > > > > > > 4444555AAfAAA 5F 555666+t$$$C ((**F33444 k4 ( ( $M  U + + $== 8F## #sKKKceZdZddZddZdS)NestedDataStructureNc||ng|_dSr`)datarr2s r3rzNestedDataStructure.__init__s ,DD" r5c||nj}t|tr4t |St|tt frfd|DS|gS)NcDg|]}|D]}|Sri)flatten)rkitem flattenedrs r3rnz/NestedDataStructure.flatten..s2SSS$ T@R@RSS9ISSSSr5)r2rGrVr6rSrZryr3s` r3r6zNestedDataStructure.flattens~'ttTY dD ! ! <<T[[]] 3 344 4 tUm , , SSSS$SSS S6Mr5r`)rrrrr6rir5r3r0r0s<5555r5r0.c ttj|j}n#t $rYdSwxYw||kSr)r ospathabspathfreeOSError) needed_bytes directory free_bytess r3has_sufficient_disk_spacerCsT : :;;@ tt * $$s14 AAurl_pathct|}d}|jdvr|jdkrd|vr?|dst d|d|dd}nZ|jd d}d |vr|d n|d f\}}|d \}}d |d |d|d}|d|}||fS)zMConvert a link to a file on a github repo in a link to the raw github object.N)httphttpss3z github.comblobz.pyzExternal import from github at z) should point to a file ending with '.py'rawrz/tree/master/zhttps://github.com/z /archive/z.zip-)rschemenetlocrIrJreplacer<split)rDparsed sub_directory github_path repo_infobranch repo_owner repo_names r3_convert_github_urlrYs h  FM }///FM\4Q4Q X  $$U++ x !v8!v!v!vwww''66HH!+abb/K?G;?V?V 1 1( ; ; ;]hjr\s Iv$-OOC$8$8 !J ZZZZ)ZZfZZZH(33633M ] ""r5 file_pathc g}t|d5}||dddn #1swxYwYtd|dg}d}|D]}t jd|}t|dkr| }|r1t jd |t j  $t jd |t j  x drt fd |Dr d rU d }t|\}}| d d||f dr?| d d ddfi d rC d }| d d|df| d d ddf|S)aFind whether we should import or clone additional files for a given processing script. And list the import. We allow: - library dependencies, - local dependencies and - external dependencies whose url is specified with a comment starting from "# From:' followed by the raw url to a file, an archive or a github repository. external dependencies will be downloaded (and extracted if needed in the dataset folder). We also add an `__init__.py` to each sub-folder of a downloaded folder so the user can import from them in the script. Note that only direct import in the dataset processing script will be handled We don't recursively explore the additional import to download further files. Example:: import tensorflow import .c4_utils import .clicr.dataset-code.build_json_dataset # From: https://raw.githubusercontent.com/clips/clicr/master/dataset-code/build_json_dataset zutf-8)encodingNz Checking z for additional imports.Fz[\s\S]*?"""[\s\S]*?rz=^import\s+(\.?)([^\s\.]+)[^#\r\n]*(?:#\s+From:\s+)?([^\r\n]*))flagszQ^from\s+(\.?)([^\s\.]+)(?:[^\s]*)\s+import\s+[^#\r\n]*(?:#\s+From:\s+)?([^\r\n]*)c3VK|]#}|ddkV$dS)rrN)group)rkimpmatchs r3rpzget_imports..Cs6??3q6U[[^^+??????r5externalrinternallibrary)openextend readlinesr$debugrPrUrra MULTILINEr_rrYr) rZlinesr~importsis_in_docstringlinedocstr_start_matchrDrSras @r3 get_importsrps( E i' * * *$a Q[[]]###$$$$$$$$$$$$$$$ LL@Y@@@AAA9;GO)R)RZ(>EE ! " "a ' '#21O   Y[_gigsttt =HdlE } ;;q>> R????w????? {{1~~ S ;;q>>*=h*G*G'- EKKNNHmTUUUUQ S EKKNNEKKNNDQRRR{{1~~ R ;;q>> 5;;q>>8TJKKKK 5;;q>>5;;q>>4PQQQQ Ns(A  A A ceZdZdZejejj ZddZ dZ dS)PicklerzHSame Pickler as the one from dill, but improved for notebooks and shellsTc t|}|tjvrtjt jdkrdn7tjjddt jdjkrd|j|j fdkr1 ddl t| fd}n#t$rYnwxYw|j|j fdkr1 ddl t| fd }n#t$rYnwxYw|j|j fd kr1 ddl t| fd }ny#t$rYnmwxYw|jd rOt!d |jDr1 ddl t| fd}n#t$rYnwxYwt&j|||dS)N0.3.6cNtjj|dSr`)dill_dilllogr%picklermsgs r3dill_logzPickler.save..dill_logds JN'',,,,,r5rbcPtjj||dSr`)rvrwr$tracerys r3r|zPickler.save..dill_logis#J%++GS99999r5)_regexPatternrc|d||j|jf}|j|||ddS)NzRe: rbz# Re)rNr] save_reducecompile)rzrcrr|rXs r3 _save_regexz!Pickler.save.._save_regexpsb ,,,777KI  ++EM4S+III &111r5)rTensorcfd}|d||f}|||||ddS)Nc.|Sr`) from_numpy)np_arrayrs r3_create_tensorz:Pickler.save.._save_tensor.._create_tensors#(#3#3H#=#==r5zTo: rbz# To)detachcpunumpyr)rzrcrrr|rs r3 _save_tensorz"Pickler.save.._save_tensors>>>>>!,,,777 # 0 0 2 2 8 8 : :<++NDc+JJJ &111r5)z tiktoken.coreEncodingc|d||j|j|j|jf}|j|||ddS)NzEnc: rbz# Enc)r0_pat_str_mergeable_ranks_special_tokensrr)rzrcrr|tiktokens r3_save_encodingz$Pickler.save.._save_encodingsh -#--888 ##,8LcNab++H,=t+MMM '222r5z spacy.langc3:K|]}|j|jfdkVdS))zspacy.languageLanguageN)rr)rkclss r3rpzPickler.save..sJFFUX.2PPFFFFFFr5cfd}|d||j|f}|||||ddS)Ncj|dd}||}||S)Nnlplang)utilget_lang_class from_config from_bytes)r bytes_datalang_clsrspacys r3 _create_langz6Pickler.save.._save_lang.._create_langsF',z'@'@vAV'W'WH"*"6"6v">">C#&>>*#=#==r5zSp: rbz# Sp)rto_bytesr)rzrcrrr|rs r3 _save_langz Pickler.save.._save_langs|>>>>> !,,,777 # CLLNN;++L$C+HHH &111r5)save_persistent_id)rarrdispatchr DILL_VERSIONrparsereleaserrrX pklregister ImportErrorrr startswithrrrrvsave) rrcrobj_typerrrrr|rXrrrs @@@@@r3rz Pickler.save^s99 7+ + +"W]7%;%;;;----$,RaR0GM'4J4J4RRR:::#X%67;PPP LLL **+*#D%x'89=PPP LLL **     +*  #D%x'89=ZZZ #OOO **+*#D$// == #FF\d\lFFFCC  LLL **     +*  #D $8JKKKKKsHB44 CCC77 DDD:: EEF## F0/F0czt|tkr"tj||dSdSr`)rarrvrrmemoize)rrcs r3rzPickler.memoizes9 99   L s + + + + +  r5N)T) rrrrrvrwMetaCatchingDictrrrr|rrrir5r3rrrrYslRRz**4<+@+E+E+G+GHHHULULULULn,,,,,r5rrcNt|d|dS)zpickle an object to a fileT)recurseN)rrdumprcfiles r3rrs( D$$$S))) Fr5c#2K ddt|jDvrZt|drJt|jt r0t |di5dVddddS#1swxYwYdSdVdS#t$rdVYdSwxYw)NPreTrainedTokenizerBasecg|] }|j Srir)rk base_classs r3rnz$_no_cache_fields..s)b)b)b***=)b)b)br5cache)rarrzrGrrVrrrbs r3_no_cache_fieldsrs  %)b)bPTUXPYPYPa)b)b)b b bW%% c39d++ c&c7B77                     EEEEE  s<ABA1$ B1A55B8A59B>BBBct}t|5t||dddn #1swxYwY|S)zpickle an object to a stringN)StringIOrrgetvaluers r3dumpsrs ::D #   S$ ==??s ;??cfd}|S)Nc&|tj<|Sr`)rrr)rts r3proxyzpklregister..proxys" r5ri)rrs` r3rrs# Lr5rtctjjd||jdst |jtj j dkrB|jtj j dds |j dkrdn#tj |j}d}tjj rt|drY|j|j|j|j|j|j|j|j|j|j||j ||j|j|jf}n|j|j|j|j|j|j|j|j|j||j ||j|j|jf}nL|j|j|j|j|j|j|j|j||j ||j|j|jf}|t:|| tjjd d S) z From dill._dill.save_code This is a modified version that removes the origin (filename + line no.) of functions created in notebooks or shells for example. zCo: rco_posonlyargcountrb# CoN)rvrwrxr% co_filenamerrrQr;r<sepco_namebasenamePY3rz co_argcountrco_kwonlyargcount co_nlocals co_stacksizeco_flagsco_code co_constsco_names co_varnames co_lnotab co_freevars co_cellvarsrr rzrcrco_firstlinenors r3 _save_coders L3LL)))&))#.. 3CO))"'+6677!;;O))"'+66r:EElSS<{j(( B!!#/22  :>6 s011$ O*)N$LKMLOK"MOO!(O)N$LKMLOK"MOO&       D Hd444 F###r5rbctjj|d||jdst |jtj j dkrB|jtj j dds |j dkrdn#tj |j}d}t|drx|j|j|j|j|j|j|j|j|j|j|j||j |j||j|j|j|j|j|jf}nt|d rl|j|j|j|j|j|j|j|j|j|j|j||j |j||j|j|j|jf}n*t|d r_|j|j|j|j|j|j|j|j|j|j|j||j ||j|j|jf}nt|d rY|j|j|j|j|j|j|j|j|j|j||j ||j|j|jf}nR|j|j|j|j|j|j|j|j|j||j ||j|j|jf}| tjj!|| tjj|d dS)NzCo: %srrr@rrrco_endlinetableco_exceptiontable co_linetablerrbr)"rvrwr$r~rrrrQr;r<rrrrzrrrrrrrrrrr co_qualnamerrco_columntablerrrr _create_coders r3 save_coderEsG 3777,))#.. 3CO))"'+6677!;;O))"'+66r:EElSS<{j(( B!!#/22  3) * *f  &%       #"%+DD.S- . .N  &%       %'DD*S. ) )8  &%       #DD&S. / /$ &%       !DD(%       D$ DJ3TsCCC 000r5z0.3.5c Jtj|stjjd|t |ddr_tjj}||dd}t|tjj vrtjj r|j n|j }ntjj r|j n|j }tt|}t |dd}t |dd}t|tjj vo|du}t!tjj |ftjj t|<tjj rdt |jd d vo|du}|rd|_|rd|_t |d d}|tjj|j||j|j|j|j|f| ndt |jd d vo|duot |dd}|rd|_|rd|_|tjj|j||j|j|j|jf| |r||_|r||_tjjr"|s |s |s|s|s|s|r|tjjd ntjjd|t |dt |dd} tjj !||| tjjddS)z From dill._dill.save_function This is a modified version that make globs deterministic since the order of the keys in the output dictionary of globalvars can change. zF1: _recurseFTrbuiltin_byrefNrrri__kwdefaults__rb# F1zF2: rrr0# F2)"rvrw_locate_functionrxr%rtdetect globalvarsidstackr __globals__ func_globalsrVsortedr{r__code__rrr_create_functionr __defaults__ __closure____dict__ func_code func_name func_defaults func_closureOLDER clear_memo StockPickler save_global) rzrcrglobsrr_memo_super fkwdefaultsr0s r3 save_functionr sz**3//? ( JN   s - - -w E22 P![3 " 3dCCCc77dj.../3z~SCOO3CSE+/:>Os?O  ..//EWh55Fw D99HWW 00Lxt7KE(+DJ,<(=(=s(BDJ RWW %z~ !WS\:r%J%JJdQW_cQc*%)GN-',G$%c+;TBB ##J/\5#,8H#/[^[gitu$ z2 F FF<t+<U;; *%)GN-',G$##J/]E3=#:KSM]_b_kl$  (!' ,#+    % % %%+ %05 %?E %NS %X` % ""$$$ JN   ' ' ' ' JN   s - - -3Z0N0NOOD J # / /4 / H H H JN   ' ' 'r5c H tj||stjjd|zt |dd}t |dd}t |dd}t |dtjj}g}|rddlm}||dd }d |j i} ntjj r|j n|j }|r"||j urt |d |j }|} nK|@|j 9t tj|j dd d|ur|} n d |j i} | |u} tt!| } | r| }n0|.tt!|}|| |urtjj rd |D} nd|D} |D]6} | | vr0|| tjj| |ffn)7|tjj| |fftjj r|j} i}dD]}t ||d}||||<|j|jkr |j|d<d | vs|j | d kr |j |d<|j }t3|tur||d <d}|r||f}tj|tjj|j| |j|j| f|f||n|j} |j$|t@|d|jffd | vs|j | d kr$|t@|d|j ff|j r$|t@|d |j fftj|tjj|j!| |j"|j#| ff|||rtItK|d}| r|r| D]}t@|d|ff} |&|n#tN$rY1wxYw|j(|tjj r$|)tUddt|)dtjjdntjjd|zt |dt |d d}tjj+,|||tjjddS)NF1: %sr _postproc_main_modified_original_mainrrTrr_mainrc,h|]}t|Srirrkgs r3 z save_function..`sCCC!1CCCr5c,h|]}t|Srirrs r3rz save_function..bsGGG!1GGGr5rr__annotations__rrrc postproc_listr cell_contents0UTF-8rF2: %srr)-rvrwrrxr%rt __builtin__ dill.detectrrrrrr_import_modulerVrr{rZ itervaluesr _setitemsrrrra_save_with_postprocrrrrrrrrrnextiterremoverJrwritebytesrr)rzrcrrrrrr globs_copyrglobs_is_globs_copyglob_ids stack_elementclosure state_dict fattrnamefattrstatetopmost_postproccellpossible_postprocr0s r3r r %sz**388H ( JN  3 / / /w D99Hd;;I$W.>EEN$W.> @VWWNM 9222222'ZT4HHH $S^404 TS__CDT " 9jN4K&K&K!('>!J!J!SJ&EE*2 9 9#.$ O OQ[]abbfppp&EE'8E#(:"5  ..//E" >" '!&)9)9););"<"<== %%z*A*A:>HCCz/@/@/B/BCCCHHGGz/D/D/F/FGGGH%.VVM$00!-0779MPUWaOb8cddd1"(($**> @S)TUUUz~+ / !Q66I#CD99E(05 9-#s|33141AJ~.U**cnj@Q.Q.Q/2~J|, ;;d**-2Jz* E.!:-E .. 3uclC??:>/#MM%W*=*=>>>>#MM#.... JN   ' ' ' ' JN  3 / / /3Z0N0NOOD J # / /4 / H H H JN   ' ' 's0R RRc  tj||st|jt ur t |dd}|tjjj}tj |d}d} tj ||j \}}t |dd|urd}n#t$rYnwxYw|rltjj |d||t |df|tjj |ddStjj |d |t |d d}t |d d}t |d d} t |d tjj} g} |rddlm} | |dd} d|ji}nv|j} | r"| | jurt |d| j} | }nK| @|j9t tj |jddd| ur| }n d|ji}|| u}t+t-|}|r|} n0| .t+t-| } | || urd| D}|D]6}||vr0||tjj|| ffn)7| tjj|| ff|j}i}dD]}t ||d}||||<|j |jkr |j |d<d|vs|j|dkr |j|d<|j}t|t*ur||d<d}|r||f}tj|tjj|j||j|j|f|f|| |rt?tA|d}|rg|re|D]b}tB|d|ff} |"|n#tF$rY1wxYw|j||$tKddctjj |dntjj |d|t |dt |dd}tjj&'|||tjj |ddS)NrT)safeF__func__zF3: %srbz# F3r rrrrrrrrrrc,h|]}t|Srirrs r3rz save_function..s???aBqEE???r5rrrrrrrr rr)(rvrwrrarr rtr!rr# _getattributerAttributeErrorr$r~rr"rrrrrVrr{rZrr%rr&rrr'r(rr)rJr*r+rr)rzrc module_namemodule _pypy_builtinfound_rrrrrrr,rr-r.r/r0r1r2r3r4r5r6r7r0s r3r r sz**388E 5CL!!11&c<>> &"&*"8"AK22;T2JJ % #z77@PQQHE1uj$773>>(, %D!J%++GXsCCC''%1D#'NNNJ%++GV<<<F J  # #GXs ; ; ;w D99Hd;;I$W.>EEN$W.> @VWWNM 9222222'ZT4HHH $S^4 _ " 9jN4K&K&K!('>!J!J!SJ&EE*2 9 9#.$ O OQ[]abbfppp&EE'8E#(:"5  ..//E" >" '!&)9)9););"<"<== %%z*A*A@?:+<+<+>+>???%.VVM$00!-0779MPUWaOb8cddd1"(($**> @S)TUUUoGJM 2 2 Y55$,1Jy)3<//-0-= >*&&#.E* <(LEE{{$&&). :& *z) J * *,s|UCLRURbdk.lnst+ +    ;#'Y-=-=-?-?(@(@$#G#G  ;/ ; ' ; ;-4t_c6R,S)%,334EFFFF)%%%$H%,+->?? eC&9&9:::: J  # #GV 4 4 4 4 J  # #GXs ; ; ;3Z0N0NOOD J # / /4 / H H H J  # #GV 4 4 4s$ =C CCQ22 Q?>Q?ctj|j|j|j|j|j}|j|_|Sr`)rr rrrrrr)rrYs r3copyfuncrD?s9   t/?PTPacgcs t tF /F Mr5Yqueuer.rc\t|di|D]\}}|j||S)Nri)rput)rFrrrrYs r3_write_generator_to_queuerIHsCtt~~f~~.. 6 & Hr5r-kwargs_iterablec#Kttjjrtn t j}|5}|fd|D} jdVn:#t$r-td|Drj rYnYnwxYwP d|Dn#d|DwxYw ddddS#1swxYwYdS)NcLg|] }t|f!Sri) apply_asyncrI)rkrrr-rFs r3rnz&iflatmap_unordered..Ws>   SYD  6f8M N N   r5T皙?timeoutc3>K|]}|VdSr`)readyrk async_results r3rpz%iflatmap_unordered.._s.RRL<--//RRRRRRr5c:g|]}|dS)rNrO)rxrSs r3rnz&iflatmap_unordered..cs) N N N \  d  + + N N Nr5) rGmultiprocessingr-rr multiprocessQueuerxr allempty)r-rrJ manager_clsmanager async_resultsrFs`` @r3iflatmap_unorderedr^Ns (o.B.GHHb''lNbK O'       ]l    O #%)D1111111RRMRRRRRWbW\WbWdWd   O N N N N N N N N N N N N N NOOOOOOOOOOOOOOOOOOsN%C&(C*A>=C>2B50C4B55C: C&CC&&C*-C*)FF) FTFFNrNTN)r9)err| functoolsrmultiprocessing.poolrVr;rFrPr contextlibr dataclassesrriorrrrr r shutilr r r typingrrrrrrrrr urllib.parserrvrWmultiprocess.poolrr packagingr tqdm.autorrrrtyping_extensions_typing_extensionsrrr get_loggerrr$ lru_cacherr4r-rrLr]rrrrrrrrVrpropertyrrboolryr.r0rCrYrprrrrrrrrrrrr rDrErXrIr-r^rir5r3rps   %%%%%%,,,,,,,,""""""0000000000((((((((WWWWWWWWWWWWWWWWWWWWWW!!!!!! 02222000000000000+///5550  H % %  )))0)s5c?)ss)s)s)s)sX3c3hB$$$N%%%2,2,C2,2,2,2,j000%%%%%T%%%822222H222(((F" !p$p$ucz"p$p$p$ p$  p$  p$smp$p$ E?p$p$ 3-p$ p$p$p$p$f        %%%%###%Xc]0B*C####&F3F5c3);#<FFFFR_,_,_,_,_,dl_,_,_,D        w////[\\\\|  !$  g(>(>(FFF[OOOd w////[FFFFP  !$  g(>(>(FFF[JJJJX  !$  g(>(>(FFF[GGGT  GCLL U[ hqkAQ8R \` eh    O $)<+<+AA BO 3 # $Od^ O a[ OOOOOOs B'' B54B5