+gd,[LdZddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl m Z mZddlmZddlmZmZeGd d ZeGd d ZGd dejZGddeZGddeZeZGddeZGddeZGddeZGddeZGddZ ej!ddd gZ"Gd!d"Z#Gd#d$e$Z%eGd%d&Z&dS)'zSplits related API.N) dataclass)DictListOptionalUnion)FileInstructionsmake_file_instructions) _split_re)NonMutableDictasdictceZdZUdZeed<dZeed<dZeed<dZ e e eed<e j ddd i Ze eed <ed ZdS) SplitInfonamer num_bytes num_examplesN shard_lengths$include_in_asdict_even_if_is_defaultT)defaultmetadata dataset_namecdt|j|gt|j}|jS)/Returns the list of dict(filename, take, skip).r split_infos instruction)r rstrrfile_instructions)self instructionss /lib/python3.11/site-packages/datasets/splits.pyrzSplitInfo.file_instructions/s9."DI   --)__name__ __module__ __qualname__rr__annotations__rintrrrr dataclassesfieldrpropertyrr#r"rr sD#NNNIsL#)-M8DI&--- #4+"3 FM###L(3-..X...r#rcJeZdZUdZeed<edZedZdS) SubSplitInfozWrapper around a sub split info. This class expose info on the subsplit: ``` ds, info = datasets.load_dataset(..., split='train[75%:]', with_info=True) info.splits['train[75%:]'].num_examples ``` r!c|jjS)z.Returns the number of example in the subsplit.)r!rr s r"rzSubSplitInfo.num_examplesGs --r#c|jjS)r)r!rr0s r"rzSubSplitInfo.file_instructionsLs 22r#N) r$r%r&__doc__r r'r+rrr,r#r"r.r.;sb#""" ..X.33X333r#r.cLeZdZdZejdZdZdZdZ ddZ dS) SplitBaseaAbstract base class for Split compositionality. See the [guide on splits](../loading#slice-splits) for more information. There are three parts to the composition: 1) The splits are composed (defined, merged, split,...) together before calling the `.as_dataset()` function. This is done with the `__add__`, `__getitem__`, which return a tree of `SplitBase` (whose leaf are the `NamedSplit` objects) ``` split = datasets.Split.TRAIN + datasets.Split.TEST.subsplit(datasets.percent[:50]) ``` 2) The `SplitBase` is forwarded to the `.as_dataset()` function to be resolved into actual read instruction. This is done by the `.get_read_instruction()` method which takes the real dataset splits (name, number of shards,...) and parse the tree to return a `SplitReadInstruction()` object ``` read_instruction = split.get_read_instruction(self.info.splits) ``` 3) The `SplitReadInstruction` is then used in the `tf.data.Dataset` pipeline to define which files to read and how to skip examples within file. c td)zParse the descriptor tree and compile all read instructions together. Args: split_dict: `dict`, The `dict[split_name, SplitInfo]` of the dataset Returns: split_read_instruction: `SplitReadInstruction` zAbstract method)NotImplementedErrorr split_dicts r"get_read_instructionzSplitBase.get_read_instructionts""3444r#c\t|ttfrdStd)*Equality: datasets.Split.TRAIN == 'train'.Fz6Equality is not implemented between merged/sub splits.) isinstance NamedSplitrr6r others r"__eq__zSplitBase.__eq__s- ej#. / / 5!"Z[[[r#c.|| S)z+InEquality: datasets.Split.TRAIN != 'test'.)r@r>s r"__ne__zSplitBase.__ne__s;;u%%%%r#c"t||S)z4Merging: datasets.Split.TRAIN + datasets.Split.TEST.) _SplitMergedr>s r"__add__zSplitBase.__add__sD%(((r#Nc td||||fDdkrtdt|tr|}n/t|tr|}nt|t r|}|s|s|std|dd}|rd|cxkrdksntd |d|z fd t |D}t |d jd|d <||tfd |DS|rt|S|rt| fd |D}d}d}g}|D],} || z }| t |||}-t |d jd|d <||tfd|DStd)a6Divides this split into subsplits. There are 3 ways to define subsplits, which correspond to the 3 arguments `k` (get `k` even subsplits), `percent` (get a slice of the dataset with `datasets.percent`), and `weighted` (get subsplits with proportions specified by `weighted`). Example:: ``` # 50% train, 50% test train, test = split.subsplit(k=2) # 50% train, 25% test, 25% validation train, test, validation = split.subsplit(weighted=[2, 1, 1]) # Extract last 20% subsplit = split.subsplit(datasets.percent[-20:]) ``` Warning: k and weighted will be converted into percent which mean that values below the percent will be rounded up or down. The final split may be bigger to deal with remainders. For instance: ``` train, test, valid = split.subsplit(k=3) # 33%, 33%, 34% s1, s2, s3, s4 = split.subsplit(weighted=[2, 2, 1, 1]) # 33%, 33%, 16%, 18% ``` Args: arg: If no kwargs are given, `arg` will be interpreted as one of `k`, `percent`, or `weighted` depending on the type. For example: ``` split.subsplit(10) # Equivalent to split.subsplit(k=10) split.subsplit(datasets.percent[:-20]) # percent=datasets.percent[:-20] split.subsplit([1, 1, 2]) # weighted=[1, 1, 2] ``` k: `int` If set, subdivide the split into `k` equal parts. percent: `datasets.percent slice`, return a single subsplit corresponding to a slice of the original split. For example: `split.subsplit(datasets.percent[-20:]) # Last 20% of the dataset`. weighted: `list[int]`, return a list of subsplits whose proportions match the normalized sum of the list. For example: `split.subsplit(weighted=[1, 1, 2]) # 25%, 25%, 50%`. Returns: A subsplit or list of subsplits extracted from this split object. c34K|]}t|VdSN)bool).0xs r" z%SplitBase.subsplit..s(<<1tAww<<<<<.assert_slices_coverage..s9EEUAIIcNN344EEEEEEr#rO)sumrPrQ)slicess r"assert_slices_coveragez2SplitBase.subsplit..assert_slices_coveragesDEEfEEErJJdSXY\S]S]N^N^^^^^^^r#rrOz,Subsplit k should be between 0 and 100, got cBg|]}t|z|dzzS)r)slice)rJishifts r" z&SplitBase.subsplit..s/JJJAeAIA77JJJr#c38K|]}t|VdSrH _SubSplitrJrTr s r"rLz%SplitBase.subsplit..-<<4++<<<<<.s";;;Qa5(;;;r#c38K|]}t|VdSrHr_ras r"rLz%SplitBase.subsplit..rbr#zCould not determine the split) rU ValueErrorr<r(rYrPrQstarttupler`append) r argkpercentweightedrWrVrgstopvr[rds ` @@r"subsplitzSplitBase.subsplits}d <<a( ;<<< < < A AKLL L c3   AA U # # GG T " " H W  T#TTT   _ _ _  >q<<<T7++ +  >MME;;;;(;;;HEDF    eE400111vbz/55F2J " "6 * * *<<<f>f>f>f>f>r#r4) metaclassceZdZdZdS)PercentSliceMetacTt|tstd||S)Nz7datasets.percent should only be called with slice, not )r<rYrf)cls slice_values r" __getitem__zPercentSliceMeta.__getitem__s3+u-- fdWbddee er#N)r$r%r&ryr,r#r"rurus#r#ruceZdZdZdS) PercentSlicezSyntactic sugar for defining slice subsplits: `datasets.percent[75:-5]`. See the [guide on splits](../loading#slice-splits) for more information. N)r$r%r&r2r,r#r"r{r{s Dr#r{c$eZdZdZdZdZdZdS)rDz0Represent two split descriptors merged together.c"||_||_dSrH)_split1_split2)r split1split2s r"__init__z_SplitMerged.__init__s  r#ct|j|}|j|}||zSrH)r~r9r)r r8read_instruction1read_instruction2s r"r9z!_SplitMerged.get_read_instructions: L==jII L==jII #444r#c\dt|jdt|jdS)N(z + ))reprr~rr0s r"__repr__z_SplitMerged.__repr__s/?4 %%??$t|*<*<????r#Nr$r%r&r2rr9rr,r#r"rDrDsL::555 @@@@@r#rDc$eZdZdZdZdZdZdS)r`z,Represent a sub split of a split descriptor.c"||_||_dSrH)_split _slice_value)r splitrxs r"rz_SubSplit.__init__&s 'r#cL|j||jSrH)rr9rr7s r"r9z_SubSplit.get_read_instruction*s {// ;;D.fs-'q'q'qL](9(?(?(D(DQ(G'q'q'qr#+zSplit name should match 'z ' but got 'z'.)_namerrematchr rf)r rsplit_names_from_instruction split_names r"rzNamedSplit.__init__ds 'q'qaeakakloapap'q'q'q$6 c cJ8Iz22 c !aY!a!aS]!a!a!abbb c c cr#c|jSrHrr0s r"__str__zNamedSplit.__str__ks zr#cd|jdS)Nz NamedSplit(rrr0s r"rzNamedSplit.__repr__ns,TZ,,,,r#ct|tr|j|jkSt|trdSt|tr |j|kSt d|d|)r;Fz%Equality not supported between split z and )r<r=rr4rrfr>s r"r@zNamedSplit.__eq__qs| eZ ( ( Y:, , y ) ) Y5 s # # Y:& &WTWWPUWWXX Xr#c"|j|jkSrHrr>s r"__lt__zNamedSplit.__lt__|szEK''r#c*t|jSrH)hashrr0s r"__hash__zNamedSplit.__hash__sDJr#c6t||jSrH)SplitReadInstructionrr7s r"r9zNamedSplit.get_read_instructions#Jtz$:;;;r#N) r$r%r&r2rrrr@rrr9r,r#r"r=r=9s((Tccc--- Y Y Y(((   <<<<.s!RRR1!44RRRr#)valuesrUr)r r8read_instructionss r"r9z"NamedSplitAll.get_read_instructions:RRj>O>O>Q>QRRR$&:&<&<===r#)r$r%r&r2rrr9 __classcell__rs@r"rrs\II     !!!>>>>>>>r#rcneZdZdZedZedZedZeZ dZ dS)Splita"`Enum` for dataset splits. Datasets are typically split into different subsets to be used at various stages of training and evaluation. - `TRAIN`: the training data. - `VALIDATION`: the validation data. If present, this is typically used as evaluation data while iterating on a model (e.g. changing hyperparameters, model architecture, etc.). - `TEST`: the testing data. This is the data to report metrics on. Typically you do not want to use this during model iteration as you may overfit to it. - `ALL`: the union of all defined dataset splits. All splits, including compositions inherit from `datasets.SplitBase`. See the [guide](../load_hub#splits) on splits for more information. Example: ```py >>> datasets.SplitGenerator( ... name=datasets.Split.TRAIN, ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and extract(url)}, ... ), ... datasets.SplitGenerator( ... name=datasets.Split.VALIDATION, ... gen_kwargs={"split_key": "validation", "files": dl_manager.download_and extract(url)}, ... ), ... datasets.SplitGenerator( ... name=datasets.Split.TEST, ... gen_kwargs={"split_key": "test", "files": dl_manager.download_and extract(url)}, ... ) ``` traintest validationcH|dkrtnt|S)z9Create a custom split with datasets.Split('custom_name').r)rr=)rwrs r"__new__z Split.__new__s "&%--}Z5E5EEr#N) r$r%r&r2r=TRAINTEST VALIDATIONrALLrr,r#r"rrsm!!F Jw  E :f  DL))J -//CFFFFFr#rSlicedSplitInfo split_inforxc2eZdZdZddZdZdZdZdZdS) raObject containing the reading instruction for the dataset. Similarly to `SplitDescriptor` nodes, this object can be composed with itself, but the resolution happens instantaneously, instead of keeping track of the tree, such as all instructions are compiled and flattened in a single SplitReadInstruction object containing the list of files and slice to use. Once resolved, the instructions can be accessed with: ``` read_instructions.get_list_sliced_split_info() # List of splits to use ``` Nctd|_|r&|t|ddSdS)Nz?Overlap between splits. Split {key} has been added with itself.) error_msg)rrx)r _splitsaddr)r rs r"rzSplitReadInstruction.__init__sO%0tuuu  O HH_ MMM N N N N N O Or#c.||j|jj<dS)z,Add a SlicedSplitInfo the read instructions.N)rrr)r sliced_splits r"rzSplitReadInstruction.adds 6B \,1222r#ct}|j|j|j|j|S)zMerging split together.)rrupdate)r r?rs r"rEzSplitReadInstruction.__add__sH 122!((666!((777  r#ct}|jD]^}|jt d|jjd|}||d<|tdi|_|S)z Sub-splits.NzTrying to slice Split z which has already been slicedrxr,) rrrrxrfrr_asdictrr)r rxrros r"ryz SplitReadInstruction.__getitem__s122$$&& 8 8A}( !k!,:K!k!k!klll A*Am   ! !/"6"6A"6"6 7 7 7 7  r#cNt|jSrH)rPrrr0s r"get_list_sliced_split_infoz/SplitReadInstruction.get_list_sliced_split_infosDL''))***r#rH) r$r%r&r2rrrEryrr,r#r"rrsu  OOOO BBB!!! ! ! !+++++r#rceZdZdZddfd Zdeeefffd Zdeeefde ffd Z d e ffd Z e d Z edd eeefd eefdZdZdZdefdZededdfdZxZS) SplitDictzSplit info object.NrcHtj|i|||_dSrH)rrr)r rargskwargsrs r"rzSplitDict.__init__s-$)&)))(r#keyct||vr.tt|St|j||}t |S)Nr)rrryr rrr.)r rr!rs r"ryzSplitDict.__getitem__ sl s88t  77&&s3xx00 02& KKMML  -- -r#valuec||jkrtd|d|jd||vrtd|dt||dS)Nz!Cannot add elem. (key mismatch: 'z' != 'z')Split  already present)rrfr __setitem__)r rrrs r"rzSplitDict.__setitem__sw %*  ZZZEJZZZ[[ [ $;;;c;;;<< < C'''''r#rc|j|vrtd|jd|j|_t|j|dS)zAdd the split info.rrN)rrfrrr)r rrs r"rz SplitDict.add sY ?d " "GjoGGGHH H"&"3  JOZ88888r#cXtd|DS)z$Return the total number of examples.c3$K|] }|jV dSrH)rrSs r"rLz/SplitDict.total_num_examples..*s$99a1>999999r#)rUrr0s r"total_num_exampleszSplitDict.total_num_examples's)994;;==999999r#rrcBt|tr!t|}||r|ddnd}||}|D]8}t|tr t di|}||9|S)zIReturns a new SplitDict initialized from a Dict or List of `split_infos`.Nrrrr,)r<dictrPrgetrr)rwrrr8rs r"from_split_dictzSplitDict.from_split_dict,s k4 ( ( 5{113344K  ALV;q>--n===RVLSl333 % ' 'J*d++ 5&4444 NN: & & & &r#cg}|D]5\}}tj|}||_||6|S)z0Returns a list of SplitInfo protos that we have.)itemscopydeepcopyrri)r outrrs r" to_split_dictzSplitDict.to_split_dict>sS&*jjll # # "J z22J(JO JJz " " " " r#cft||jSrH)rrrrr0s r"rzSplitDict.copyGs'((););)=)=t?PQQQr#returncd|D}|D]}|dd|D]}|dd|S)Nc,g|]}t|Sr,)r rSs r"r\z+SplitDict._to_yaml_list..Ks777Qvayy777r#rr)rpop)r rsplit_info_dicts r" _to_yaml_listzSplitDict._to_yaml_listJsv77$"4"4"6"6777" 7 7O    6 6 6 6" 6 6O    5 5 5 5 r# yaml_datac,||SrH)r)rwrs r"_from_yaml_listzSplitDict._from_yaml_listTs""9---r#rH)r$r%r&r2rrr4rryrrrr+r classmethodrrrrrrrPrrrrs@r"rrs+/))))))) .uY^4 . . . . . .(uY^4(Y((((((9i999999::X:%d *;8TW=["RRRt.....[.....r#rc|eZdZUdZeed<ejeZ e ed<ejdZ e ed<dZ d S) SplitGeneratoraDefines the split information for the generator. This should be used as returned value of `GeneratorBasedBuilder._split_generators`. See `GeneratorBasedBuilder._split_generators` for more info and example of usage. Args: name (`str`): Name of the `Split` for which the generator will create the examples. **gen_kwargs (additional keyword arguments): Keyword arguments to forward to the `DatasetBuilder._generate_examples` method of the builder. Example: ```py >>> datasets.SplitGenerator( ... name=datasets.Split.TRAIN, ... gen_kwargs={"split_key": "train", "files": dl_manager.download_and_extract(url)}, ... ) ``` r)default_factory gen_kwargsF)initrct|j|_t|jt|j|_dS)N)r)rrr=rrr0s r" __post_init__zSplitGenerator.__post_init__xs9 NN 49#333r#N)r$r%r&r2rr'r)r*rrrrrrr,r#r"rrYsz2 III({(>>>J>>>-K-5999J 99944444r#r)'r2rq collectionsrr)rrtypingrrrr arrow_readerr r namingr utils.py_utilsr r rr.ABCMetar4typerur{rlrDr`r=rr namedtuplerrrrrr,r#r"r sn    !!!!!!............BBBBBBBB22222222 ....... .4 3333333 3,b>b>b>b>b>#+b>b>b>b>Vt      -       @ @ @ @ @9 @ @ @ EEEEE EEE,J<J<J<J<J<J<J<J > > > >J > > >,F,F,F,F,F,F,F,F`)+(4+4+4+4+4+4+4+4+nQ.Q.Q.Q.Q.Q.Q.Q.h !4!4!4!4!4!4!4 !4!4!4r#