+gdFddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlZddlmZer*ddlmZ ddlmZn #e$rYnwxYw ddlZn #e$rYnwxYwejdduZejd duZejeZGd d eZ Gd d e Z!Gdde Z"Gdde Z#Gdde Z$GddZ%Gdde%Z&Gdde%Z'GddZ(dS)N)PurePath) TYPE_CHECKINGDictList NamedTupleOptionalUnion)logging)Dataset Elasticsearch elasticsearchfaissceZdZdS) MissingIndexN)__name__ __module__ __qualname__/lib/python3.11/site-packages/datasets/search.pyrr"sDrrc<eZdZUeeed<eeed<dS) SearchResultsscoresindicesNrrrrfloat__annotations__intrrrrr&s2 K #YrrcTeZdZUeeeed<eeeed<dS)BatchedSearchResults total_scores total_indicesNrrrrr"r"+s;tE{####S ?"""""rr"c0eZdZUeeed<eed<dS)NearestExamplesResultsrexamplesNrrrrrrdictrrrr&r&0s) KNNNNNrr&cHeZdZUeeeed<eeed<dS)BatchedNearestExamplesResultsr#total_examplesNr(rrrr+r+5s7tE{####Jrr+ceZdZdZd dedefdZd dedefdZde e e ffdZ e de e e fddfd Zd S) BaseIndexzBase class for indexing kreturnc t)z To implement. This method has to return the scores and the indices of the retrieved examples given a certain query. NotImplementedError)selfqueryr0kwargss rsearchzBaseIndex.search=s "!rc gg}}|D]E}|||\}}||||Ft||S)a Find the nearest examples indices to the query. Args: queries (`Union[List[str], np.ndarray]`): The queries as a list of strings if `column` is a text index or as a numpy array if `column` is a vector index. k (`int`): The number of examples to retrieve per query. Ouput: total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. total_indices (`List[List[int]]`): The indices of the retrieved examples per query. )r8appendr") r5queriesr0r7r#r$r6rrs r search_batchzBaseIndex.search_batchDsn')"m  * *E"kk%33OFG    ' ' '   ) ) ) )#L-@@@rfilect)zSerialize the index on diskr3)r5r=s rsavezBaseIndex.saveVs!!rct)zDeserialize the index from diskr3)clsr=s rloadzBaseIndex.loadZs "!rNr/)rrr__doc__r rr8r"r<r strrr? classmethodrBrrrr.r.:s!!""s"m""""AAsA>RAAAA$"sH}-"""""c8m,""""["""rr.c eZdZdZ ddeedeededdeedeef d Zdd e e ed fd eefd Z ddede fdZ ddedefdZdS)ElasticSearchIndexa/ Sparse index using Elasticsearch. It is used to index text and run queries based on BM25 similarity. An Elasticsearch server needs to be accessible, and a python client is declared with ``` es_client = Elasticsearch([{'host': 'localhost', 'port': '9200'}]) ``` for example. Nhostport es_clientr es_index_namees_index_configctstd|||td|pd}|pd}ddl}ddlm}||n||t |dg|_||n7dtj tj j z|_||nd d d d d diiddddd ddiid|_dS)Nz}You must install ElasticSearch to use ElasticSearchIndex. To do so you can run `pip install elasticsearch==7.7.1 for example`zBPlease specify either `es_client` or `(host, port)`, but not both. localhosti#rr )rIrJhuggingface_datasets_r analyzer stop_standardstandard _english_)typez stopwords)number_of_shardsanalysis propertiestextBM25)rUrQ similarity)settingsmappings)_has_elasticsearch ImportError ValueErrorelasticsearch.helpersrrrErKospathbasenametempfileNamedTemporaryFilenamerLrM)r5rIrJrKrLrMrrs r__init__zElasticSearchIndex.__init__jsH" P   d&6$:Jabb b"{|t$$$$//////&/&;Y]gjkogpgpPqPqOrAsAs( M(27+;+;H.passage_generators!"+I"6"6>>JAw#*6?1======>>#,I"6"666JAw#*155555566rN)clientrmactionsr z>Some documents failed to be added to ElasticSearch. Failures: /zIndexed dz documents)rLrMrKrcreatelenr tqdmis_progress_bar_enabledrhelpersstreaming_bulkupdateloggerwarninginfo) r5rirj index_name index_confignumber_of_docsprogress successesrxesokactions `` r add_documentsz ElasticSearchIndex.add_documentssl ' +  %%J\%JJJYwOnOpOpKpqqq  6 6 6 6 6 6 #"""*33>%%''4    JB OOA    OII I & & NN|QTU^Q_Q_`iQi||lopylzlz||     6y666677777rr/r6r1c |jjd |jd|dgddi|dd|}|dd}td|Dd |DS) amFind the nearest examples indices to the query. Args: query (`str`): The query as a string. k (`int`): The number of examples to retrieve. Ouput: scores (`List[List[float]`): The retrieval scores of the retrieved examples. indices (`List[List[int]]`): The indices of the retrieved examples. multi_matchrY cross_fields)r6fieldsrU)r6sizerlhitscg|] }|d S)_scorer.0hits r z-ElasticSearchIndex.search..s<<<c(m<<.s#>_>_>_SVs3u:>_>_>_rr)rKr8rLr)r5r6r0r7responsers rr8zElasticSearchIndex.searchs)4>( $)UvhXf+g+ghrstt     '<_>_Z^>_>_>_```rr0c  ddl}dgt|zdgt|z}}|j|5 fdt |D}|j|D]2} || } | } | j|| <| j|| <3 dddn #1swxYwYt||S)Nr) max_workerscBi|]\}}jj|fi|Sr)submitr8)rrvr6executorr0r7r5s r z3ElasticSearchIndex.search_batch..s=vvvW_WXZ_xt{E1OOOOQRvvvr)r$r#) concurrent.futuresr~futuresThreadPoolExecutorru as_completedresultrrr") r5r;r0rr7 concurrentr#r$future_to_indexfuturermresultsrs ` ` ` @rr<zElasticSearchIndex.search_batchs9!!!!'+fs7||&;dVc'll=Rm   2 2{ 2 K K 7xvvvvvvvclmtcucuvvvO$,99/JJ 7 7'/)/&-n U#'. e$$  7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7$-l[[[[s A,CC C)NNNNNNrC)r/r/)rrrrDrrEr r)rhr rrrr8r"r<rrrrHrH`s(#"/3'+*. $ $ sm$ sm$ O, $  } $ "$ $ $ $ $ L"8"8uT#Y -A'B"8HUXM"8"8"8"8HaaCaMaaaa& \ \s \Nb \ \ \ \ \ \rrHc eZdZdZ ddeeeeefdeedeededfdZ dd ee j d fd eed edeedee f dZ eddddeeeeefddfdZd de j defdZd de j defdZddeeefdeefdZe d!deeefdeeeeefdeeddfdZdS)" FaissIndexa Dense index using Faiss. It is used to index vectors. Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. You can find more information about Faiss here: - For index types and the string factory: https://github.com/facebookresearch/faiss/wiki/The-index-factory - For GPU settings: https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU Ndevicestring_factory metric_type custom_index faiss.Indexc||td||td||_||_||_||_t st ddS)a$ Create a Dense index using Faiss. You can specify `device` if you want to run it on GPU (`device` must be the GPU index). You can find more information about Faiss here: - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory NzFPlease specify either `string_factory` or `custom_index` but not both.zsCannot pass both 'custom_index' and 'device'. Pass 'custom_index' already transferred to the target device instead.a{You must install Faiss to use FaissIndex. To do so you can run `conda install -c pytorch faiss-cpu` or `conda install -c pytorch faiss-gpu`. A community supported package is also available on pypi: `pip install faiss-cpu` or `pip install faiss-gpu`. Note that pip may not have the latest version of FAISS, and thus, some of the latest features and bug fixes may not be available.)r`rrr faiss_index _has_faissr_)r5rrrrs rrhzFaissIndex.__init__s  %,*Beff f  ,":X  ,&' T   rvectorsr rj batch_size train_size faiss_verbosecddl}|j|t|dnt|d|}|j;|j|j||j}nK|j||j|j}n.|j|j|}n|j||j}|||j|_t dt|j|||j_ t|jdr|jj||jj_ t|jdr|jj||jj_ t|jdr|jj||jj_ |b| |d|n|d||} t dt| d|j| nt d t d t|d t%jt)dt||t%j D]>} | || | |zn|| | |z|} |j| ?dS) z Add vectors to the index. If the arrays are inside a certain column, you can specify it using the `column` argument. rNzCreated faiss index of type rm quantizerclustering_indexz"Training the index with the first z vectorszEIgnored the training step of the faiss index as `train_size` is None.zAdding z vectors to the faiss index)rr)rrr~rr index_factory IndexFlat_faiss_index_to_devicerrrrUverbosehasattrrmrrtrainr rrangeradd) r5rrjrrrrrrm train_vecsrvvecss r add_vectorszFaissIndex.add_vectorss     #&,n3wqz???#gaj>P:Q:QD".#+/E/d6IJJEE/E/d6I4K[\\EE#++EOD11EE+EOD$2BCCE#::5$+NND  KKOtD &.t'55 C$:J:T:`5B *2t');<< JAQAbAnL  fc " " {{6E688 ..y&%HH43E:: u . . 0E0T&\\JJJEEFd6llFFFno   rr/r6c t|jdkr8t|jdks|jddkrtd|dd}|jjst j|d}|jj ||fi|\}}t|d|d tS)awFind the nearest examples indices to the query. Args: query (`np.array`): The query as a numpy array. k (`int`): The number of examples to retrieve. Ouput: scores (`List[List[float]`): The retrieval scores of the retrieved examples. indices (`List[List[int]]`): The indices of the retrieved examples. r rzHShape of query is incorrect, it has to be either a 1D array or 2D (1, N)rCorder) r~shaper`reshapeflags c_contiguousnpasarrayrr8rastyper )r5r6r0r7r;rrs rr8zFaissIndex.searchVs u{  q c%+&6&6!&;&;u{1~QR?R?Rghh h--2&&}) 5j444G1$*1'1GGGGVAY (9(9#(>(>???rr;c t|jdkrtd|jjst j|d}|jj||fi|\}}t|| tS)aFind the nearest examples indices to the queries. Args: queries (`np.array`): The queries as a numpy array. k (`int`): The number of examples to retrieve. Ouput: total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. total_indices (`List[List[int]]`): The indices of the retrieved examples per query. rzShape of query must be 2Drr) r~rr`rrrrrr8r"rr )r5r;r0r7rrs rr<zFaissIndex.search_batchjs w}   " "899 9}) 5j444G1$*1'1GGGG#FGNN3,?,?@@@rr=storage_optionsc ~ddl}|j=t|jttt fr|j|j}n|j}tj t|dfi|pi5}|j ||j |j |jddddS#1swxYwYdS)z Serialize the FaissIndex on diskrNwb)rrrr rrindex_gpu_to_cpurfsspecopenrE write_indexBufferedIOWriterPyCallbackIOWriterwrite)r5r=rrrmfs rr?zFaissIndex.save|s ; "z$+T5?Q'R'R "*E*4+;< train_size (Optional `int`): If the index needs a training step, specifies how many vectors will be used to train the index. faiss_verbose (`bool`, defaults to False): Enable the verbosity of the Faiss index. Nrrrrrjrrrrrr) r5rjrrrrrrrrrs radd_faiss_indexzIndexableMixin.add_faiss_indexsp@$.#9ZZv  .k`l     J:]j    %0 j!!!rexternal_arraysc tt||||} | |d||| | |j|<dS)a8Add a dense index using Faiss for fast retrieval. The index is created using the vectors of `external_arrays`. You can specify `device` if you want to run it on GPU (`device` must be the GPU index). You can find more information about Faiss here: - For `string factory`: https://github.com/facebookresearch/faiss/wiki/The-index-factory Args: external_arrays (`np.array`): If you want to use arrays from outside the lib for the index, you can set `external_arrays`. It will use `external_arrays` to create the Faiss index instead of the arrays in the given `column`. index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs. If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU. string_factory (Optional `str`): This is passed to the index factory of Faiss to create the index. Default index class is IndexFlatIP. metric_type (Optional `int`): Type of metric. Ex: `faiss.METRIC_INNER_PRODUCT` or `faiss.METRIC_L2`. custom_index (Optional `faiss.Index`): Custom Faiss index that you already have instantiated and configured for your needs. batch_size (Optional `int`): Size of the batch to use while adding vectors to the FaissIndex. Default value is 1000. train_size (Optional `int`): If the index needs a training step, specifies how many vectors will be used to train the index. faiss_verbose (`bool`, defaults to False): Enable the verbosity of the Faiss index. rNrr) r5rrrrrrrrrrs r$add_faiss_index_from_external_arraysz3IndexableMixin.add_faiss_index_from_external_arrayssb@!.k`l     DZJfs    %0 j!!!rr=rc||}t|ts#td|dt |d|||t d|d|dS)aSave a FaissIndex on disk. Args: index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`). storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. zIndex 'z' is not a FaissIndex but a '')rzSaved FaissIndex z at N)rrrr`rUr?rr)r5rr=rrms rsave_faiss_indexzIndexableMixin.save_faiss_indexsz**%,, `^z^^PTUZP[P[^^^__ _ 4 999 > >>>>?????rc 4t|||}|jjt |kr3t d|d|d|jjdt |d ||j|<td|d|d S) aLoad a FaissIndex from disk. If you want to do additional configurations, you can have access to the faiss index object by doing `.get_index(index_name).faiss_index` to make it fit your needs. Args: index_name (`str`): The index_name/identifier of the index. This is the index_name that is used to call `.get_nearest` or `.search`. file (`str`): The path to the serialized faiss index on disk or remote URI (e.g. `"s3://my-bucket/index.faiss"`). device (Optional `Union[int, List[int]]`): If positive integer, this is the index of the GPU to use. If negative integer, use all GPUs. If a list of positive integers is passed in, run only on those GPUs. By default it uses the CPU. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. )rrz1Index size should match Dataset size, but Index 'z' at z has z elements while the dataset has z examples.zLoaded FaissIndex z from N) rrBrntotalr~r`rrr)r5rr=rrrms rload_faiss_indexzIndexableMixin.load_faiss_index"s0V_UU   #s4yy 0 0pJppUYpp`e`q`xpp[^_c[d[dppp %* j! AAA4AABBBBBrrIrJrKrrLrMc|||n|}t|||||}|||||j|<dS)aAdd a text index using ElasticSearch for fast retrieval. Args: column (`str`): The column of the documents to add to the index. index_name (Optional `str`): The index_name/identifier of the index. This is the index name that is used to call `.get_nearest` or `.search`. By default it corresponds to `column`. host (Optional `str`, defaults to localhost): host of where ElasticSearch is running port (Optional `str`, defaults to 9200): port of where ElasticSearch is running es_client (Optional `elasticsearch.Elasticsearch`): The elasticsearch client used to create the index if host and port are None. es_index_name (Optional `str`): The elasticsearch index name used to create the index. es_index_config (Optional `dict`): The configuration of the elasticsearch index. Default config is: Config:: { "settings": { "number_of_shards": 1, "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}}, }, "mappings": { "properties": { "text": { "type": "text", "analyzer": "standard", "similarity": "BM25" }, } }, } NrIrJrKrLrM)rj)rHrr) r5rjrrIrJrKrLrMes_indexs radd_elasticsearch_indexz&IndexableMixin.add_elasticsearch_indexBs`Z$.#9ZZv %DI]ds    tF333$, j!!!rc>t||||||j|<dS)aALoad an existing text index using ElasticSearch for fast retrieval. Args: index_name (`str`): The `index_name`/identifier of the index. This is the index name that is used to call `get_nearest` or `search`. es_index_name (`str`): The name of elasticsearch index to load. host (`str`, *optional*, defaults to `localhost`): Host of where ElasticSearch is running. port (`str`, *optional*, defaults to `9200`): Port of where ElasticSearch is running. es_client (`elasticsearch.Elasticsearch`, *optional*): The elasticsearch client used to create the index if host and port are `None`. es_index_config (`dict`, *optional*): The configuration of the elasticsearch index. Default config is: ``` { "settings": { "number_of_shards": 1, "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}}, }, "mappings": { "properties": { "text": { "type": "text", "analyzer": "standard", "similarity": "BM25" }, } }, } ``` rN)rHr)r5rrLrIrJrKrMs rload_elasticsearch_indexz'IndexableMixin.load_elasticsearch_indexvs2V%7DI]ds% % %  j!!!rc|j|=dS)zDrop the index with the specified column. Args: index_name (`str`): The `index_name`/identifier of the index. Nrrs r drop_indexzIndexableMixin.drop_indexs M* % % %rr/r6r0c `|||j|j||fi|S)a]Find the nearest examples indices in the dataset to the query. Args: index_name (`str`): The name/identifier of the index. query (`Union[str, np.ndarray]`): The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (`int`): The number of examples to retrieve. Returns: - scores (`List[List[float]`): The retrieval scores of the retrieved examples. - indices (`List[List[int]]`): The indices of the retrieved examples. )r rr8)r5rr6r0r7s rr8zIndexableMixin.searchs< ((444/t}Z(/qCCFCCCrr;c `|||j|j||fi|S)aFind the nearest examples indices in the dataset to the query. Args: index_name (`str`): The `index_name`/identifier of the index. queries (`Union[List[str], np.ndarray]`): The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (`int`): The number of examples to retrieve per query. Returns: - total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. - total_indices (`List[List[int]]`): The indices of the retrieved examples per query. )r rr<)r5rr;r0r7s rr<zIndexableMixin.search_batchs<" ((4445t}Z(5gqKKFKKKrc |||j|||fi|\}}d|D}t|dt|||S)a=Find the nearest examples in the dataset to the query. Args: index_name (`str`): The index_name/identifier of the index. query (`Union[str, np.ndarray]`): The query as a string if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (`int`): The number of examples to retrieve. Returns: - scores (`List[float]`): The retrieval scores of the retrieved examples. - examples (`dict`): The retrieved examples. cg|] }|dk| Srrrrvs rrz7IndexableMixin.get_nearest_examples..s444QQ!VVqVVVrN)r r8r&r~)r5rr6r0r7rr top_indicess rget_nearest_examplesz#IndexableMixin.get_nearest_examplesst" ((444%$+j%EEfEE44'444 %f-?s;/?/?-?&@${BSTTTrc |j|||fi|\}}dt||D}fd|D}t||S)aFind the nearest examples in the dataset to the query. Args: index_name (`str`): The `index_name`/identifier of the index. queries (`Union[List[str], np.ndarray]`): The queries as a list of strings if `index_name` is a text index or as a numpy array if `index_name` is a vector index. k (`int`): The number of examples to retrieve per query. Returns: - total_scores (`List[List[float]`): The retrieval scores of the retrieved examples per query. - total_examples (`List[dict]`): The retrieved examples per query. cVg|]&\}}|dtd|D'S)Ncg|] }|dk| Sr)rr*s rrzHIndexableMixin.get_nearest_examples_batch...s;;;!AFFAFFFr)r~)rscores_i indices_is rrz=IndexableMixin.get_nearest_examples_batch..sN   #) ..s<<.s-[[['<<'<<<=[[[r)r r<zipr+)r5rr;r0r7r#r$ total_sampless` rget_nearest_examples_batchz)IndexableMixin.get_nearest_examples_batchs" ((444&7d&7 GQ&Y&YRX&Y&Y# m  '*<'G'G   \[[[][[[ ,\=IIIr)NNNNNrNF)NNNNrNFrr)NNNNNNrrC)&rrrrDrhrrrErrr rr r.rrr r rrrrrrrrr)r r"r$rr8r"r<r&r,r+r6rrrrrsG55111""""""+s+t++++c #d3i#### )C )I ) ) ) )%)26(,%)04$(#'0'0'0SM'0sDI~./ '0 ! '0 c] '0}-'0'0SM'0'0'0'0'0Z37(,%)04$(#&0&0&0&0sDI~./ &0 ! &0 c] &0}-&0&0SM&0&0&0&0&0P@@3@eCM6J@]efj]k@@@@,37*. CCCCM"CsDI~./ C "$ CCCCF%)""/3'+*.2-2-2-SM2-sm 2- sm 2- O, 2- }2-"$2-2-2-2-p#"/3*.- - - - sm - sm - O, - "$- - - - ^&S&&&&DDDU3=-ADcD]jDDDD&NPLLL(-d3i.A(BLGJL LLLL*FHUUU&+CM&:U?BU UUUU.NPJJJ(-d3i.A(BJGJJ &JJJJJJrr))importlib.util importlibrbrepathlibrtypingrrrrrr rnumpyrutilsr arrow_datasetr rrr_rutil find_specr^r get_loggerrr Exceptionrrr"r&r+r.rHrrrrrrBs  IIIIIIIIIIIIIIII   &&&&&& ///////             ^--o>>dJ ^ % %g . .d :   H % %     9   J #####:### Z J #"#"#"#"#"#"#"#"Lr\r\r\r\r\r\r\r\jBBBBBBBBJhJhJhJhJhJhJhJhJhJhJs#AA A  AAA