c_gdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z mZmZmZmZmZmZmZmZmZddl ZddlmZddlmZddlmZmZdd l m!Z!dd l"m#Z#ej$e%Z&e'e(ej)fZ*e'e(ej)ejfZ+d Z,Gd d ej-Z.e.Z/e.Z0e.Z1GddZ2e2Z3dZ4dZ5 ddZ6dZ7dZ8dZ9ddddej:e ddfdZ;dZdde fdZ?dS) uThis module implements word vectors, and more generally sets of vectors keyed by lookup tokens/ints, and various similarity look-ups. Since trained word vectors are independent from the way they were trained (:class:`~gensim.models.word2vec.Word2Vec`, :class:`~gensim.models.fasttext.FastText` etc), they can be represented by a standalone structure, as implemented in this module. The structure is called "KeyedVectors" and is essentially a mapping between *keys* and *vectors*. Each vector is identified by its lookup key, most often a short string token, so this is usually a mapping between {str => 1D numpy array}. The key is, in the original motivating case, a word (so the mapping maps words to 1D vectors), but for some models, the key can also correspond to a document, a graph node etc. (Because some applications may maintain their own integral identifiers, compact and contiguous starting at zero, this class also supports use of plain ints as keys – in that case using them as literal pointers to the position of the desired vector in the underlying array, and saving the overhead of a lookup map entry.) Why use KeyedVectors instead of a full model? ============================================= +---------------------------+--------------+------------+-------------------------------------------------------------+ | capability | KeyedVectors | full model | note | +---------------------------+--------------+------------+-------------------------------------------------------------+ | continue training vectors | ❌ | ✅ | You need the full model to train or update vectors. | +---------------------------+--------------+------------+-------------------------------------------------------------+ | smaller objects | ✅ | ❌ | KeyedVectors are smaller and need less RAM, because they | | | | | don't need to store the model state that enables training. | +---------------------------+--------------+------------+-------------------------------------------------------------+ | save/load from native | | | Vectors exported by the Facebook and Google tools | | fasttext/word2vec format | ✅ | ❌ | do not support further training, but you can still load | | | | | them into KeyedVectors. | +---------------------------+--------------+------------+-------------------------------------------------------------+ | append new vectors | ✅ | ✅ | Add new-vector entries to the mapping dynamically. | +---------------------------+--------------+------------+-------------------------------------------------------------+ | concurrency | ✅ | ✅ | Thread-safe, allows concurrent vector queries. | +---------------------------+--------------+------------+-------------------------------------------------------------+ | shared RAM | ✅ | ✅ | Multiple processes can re-use the same data, keeping only | | | | | a single copy in RAM using | | | | | `mmap `_. | +---------------------------+--------------+------------+-------------------------------------------------------------+ | fast load | ✅ | ✅ | Supports `mmap `_ | | | | | to load data from disk instantaneously. | +---------------------------+--------------+------------+-------------------------------------------------------------+ TL;DR: the main difference is that KeyedVectors do not support further training. On the other hand, by shedding the internal data structures necessary for training, KeyedVectors offer a smaller RAM footprint and a simpler interface. How to obtain word vectors? =========================== Train a full model, then access its `model.wv` property, which holds the standalone keyed vectors. For example, using the Word2Vec algorithm to train the vectors .. sourcecode:: pycon >>> from gensim.test.utils import lee_corpus_list >>> from gensim.models import Word2Vec >>> >>> model = Word2Vec(lee_corpus_list, vector_size=24, epochs=100) >>> word_vectors = model.wv Persist the word vectors to disk with .. sourcecode:: pycon >>> from gensim.models import KeyedVectors >>> >>> word_vectors.save('vectors.kv') >>> reloaded_word_vectors = KeyedVectors.load('vectors.kv') The vectors can also be instantiated from an existing file on disk in the original Google's word2vec C format as a KeyedVectors instance .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> wv_from_text = KeyedVectors.load_word2vec_format(datapath('word2vec_pre_kv_c'), binary=False) # C text format >>> wv_from_bin = KeyedVectors.load_word2vec_format(datapath("euclidean_vectors.bin"), binary=True) # C bin format What can I do with word vectors? ================================ You can perform various syntactic/semantic NLP word tasks with the trained vectors. Some of them are already built-in .. sourcecode:: pycon >>> import gensim.downloader as api >>> >>> word_vectors = api.load("glove-wiki-gigaword-100") # load pre-trained word-vectors from gensim-data >>> >>> # Check the "most similar words", using the default "cosine similarity" measure. >>> result = word_vectors.most_similar(positive=['woman', 'king'], negative=['man']) >>> most_similar_key, similarity = result[0] # look at the first match >>> print(f"{most_similar_key}: {similarity:.4f}") queen: 0.7699 >>> >>> # Use a different similarity measure: "cosmul". >>> result = word_vectors.most_similar_cosmul(positive=['woman', 'king'], negative=['man']) >>> most_similar_key, similarity = result[0] # look at the first match >>> print(f"{most_similar_key}: {similarity:.4f}") queen: 0.8965 >>> >>> print(word_vectors.doesnt_match("breakfast cereal dinner lunch".split())) cereal >>> >>> similarity = word_vectors.similarity('woman', 'man') >>> similarity > 0.8 True >>> >>> result = word_vectors.similar_by_word("cat") >>> most_similar_key, similarity = result[0] # look at the first match >>> print(f"{most_similar_key}: {similarity:.4f}") dog: 0.8798 >>> >>> sentence_obama = 'Obama speaks to the media in Illinois'.lower().split() >>> sentence_president = 'The president greets the press in Chicago'.lower().split() >>> >>> similarity = word_vectors.wmdistance(sentence_obama, sentence_president) >>> print(f"{similarity:.4f}") 3.4893 >>> >>> distance = word_vectors.distance("media", "media") >>> print(f"{distance:.1f}") 0.0 >>> >>> similarity = word_vectors.n_similarity(['sushi', 'shop'], ['japanese', 'restaurant']) >>> print(f"{similarity:.4f}") 0.7067 >>> >>> vector = word_vectors['computer'] # numpy vector of a word >>> vector.shape (100,) >>> >>> vector = word_vectors.wv.get_vector('office', norm=True) >>> vector.shape (100,) Correlation with human opinion on word similarity .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> similarities = model.wv.evaluate_word_pairs(datapath('wordsim353.tsv')) And on word analogies .. sourcecode:: pycon >>> analogy_scores = model.wv.evaluate_word_analogies(datapath('questions-words.txt')) and so on. N)Integral)Iterable) dotfloat32doublezerosvstackndarraysumprodargmaxdtypeascontiguousarray frombuffer)stats)cdist)utilsmatutils) Dictionary) deprecatedc|gSt|ts-t|trt|jdkr|gSt|tr't|jdkrt |S|S)zEnsure that the specified value is wrapped in a list, for those supported cases where we also accept a single key or vector.N) isinstance _KEY_TYPESr lenshapelist)values :lib/python3.11/site-packages/gensim/models/keyedvectors.py _ensure_listr!s  %$$E7)C)CEKHXHX\]H]w%!!c%+&6&6!&;E{{ Lc deZdZdejdfdZdZfdZdZdTdZ dZ d Z dUd Z d Z d ZdVd ZdWdZeddZdXdZdZdYdZdZdZdZdZdZeddZdZedZejdZd Z dWd!Z!ed"Z"e"jd#Z"ed$Z#e#jd%Z#ed&Z$e$jd'Z$d(Z%fd)Z& dZd+Z'd[d,Z(d[d-Z)d[d.Z*d\d/Z+ d]d0Z,d\d1Z-d2Z.e/d3Z0d^d5Z1d6Z2d7Z3d8Z4e/d9Z5 d_d<Z6e/d=Z7e/d>Z8 d`dAZ9edBdWdCZ:dDZ;dadEZ< dbdHZ=e>ddd@dIde?dfdJZ@dcdLZA dddMeBdNeCdOeCdPdfdQZDdRZEdSZFxZGS)e KeyedVectorsrNc||_dg|z|_d|_i|_t ||f||_d|_i|_||_dS)aMapping between keys (such as words) and vectors for :class:`~gensim.models.Word2Vec` and related models. Used to perform operations on the vectors such as vector lookup, distance, similarity etc. To support the needs of specific models and other downstream uses, you can also set additional attributes via the :meth:`~gensim.models.keyedvectors.KeyedVectors.set_vecattr` and :meth:`~gensim.models.keyedvectors.KeyedVectors.get_vecattr` methods. Note that all such attributes under the same `attr` name must have compatible `numpy` types, as the type and storage array for such attributes is established by the 1st time such `attr` is set. Parameters ---------- vector_size : int Intended number of dimensions for all contained vectors. count : int, optional If provided, vectors wil be pre-allocated for at least this many vectors. (Otherwise they can be added later.) dtype : type, optional Vector dimensions will default to `np.float32` (AKA `REAL` in some Gensim code) unless another type is provided here. mapfile_path : string, optional Currently unused. Nrr) vector_size index_to_key next_index key_to_indexrvectorsnormsexpandos mapfile_path)selfr'countrr.s r __init__zKeyedVectors.__init__s`4'!FUNe[1???   (r"cP|jjd|jdt|dS)Nz ) __class____name__r'rr/s r __str__zKeyedVectors.__str__s0.)]]8H]]CPTII]]]]r"ctt|j|i|t|dr|t|ds9|jd|jdd|_t|ds7|jdd|_|jj d|_ t|d sd|_ t|d si|_ d |jvr| t|d st||_dSdS) zXHandle special requirements of `.load()` protocol, usually up-converting older versions.doctagsr( index2word index2entityNr+syn0rr,r-r*r))superr$_load_specialshasattr_upconvert_old_d2vkv__dict__popr(r+rr'r,r-_upconvert_old_vocabrr)r/argskwargsr4s r r>zKeyedVectors._load_specialssG0lD!!0$A&AAA 4 # # (  % % ' ' 't^,, i $ 1 1, @Q@QR`bf@g@g h hD tY'' 5=,,VT::DL#|1!4D tW%% DJtZ(( DM  . (  % % ' ' 't\** (!$iiDOOO ( (r"c|jdd}i|_|D]\}||}|j|j|<|jD])}||j||j|*]d|jvr4|jdtj |jd<dSdS)z\Convert a loaded, pre-gensim-4.0.0 version instance that had a 'vocab' dict of data objects.vocabN sample_int) rArBr*keysindex set_vecattrr-astypenpuint32)r/ old_vocabkold_vattrs r rCz!KeyedVectors._upconvert_old_vocabsM%%gt44 !! J JAaLE#(;D a ++-- J J  dEN44HIIII J 4= ( X*.- *E*L*LRY*W*WDM, ' ' ' X Xr"c |4tj}fd|D}tj}t ||D]\}}|t ur tj}|turt}|jvrtj ||j|<Sj|}tj ||j std|d|d|j t||krt|}tj ||j j|<|dt||fj|dt||f<dS)aAEnsure arrays for given per-vector extra-attribute names & types exist, at right size. The length of the index_to_key list is canonical 'intended size' of KeyedVectors, even if other properties (vectors array) hasn't yet been allocated or expanded. So this allocation targets that size. Nc4g|]}j|jS)r-r).0rSr/s r z2KeyedVectors.allocate_vecattrs..6s#AAA4T]4(.AAAr"r&zCan't allocate type z for attribute z#, conflicts with its existing type )rr-rJrr(zipintrNint64strobjectr issubdtyper TypeErrormin)r/attrstypes target_sizerSt prev_expando prev_counts` r allocate_vecattrszKeyedVectors.allocate_vecattrs+s  B++--..EAAAA5AAAE$+,, 5%(( s sGD!Cx HCx 4=( &(h{!&D&D&D d#=.L=L$677 M1MMTMM8D8JMM<  K/ \**J"$(;l>P"Q"Q"QDM$ DPQoSVWacnSoSoQoQpDrDM$  >#j+">"> > ? A A) s sr"c||gt|g||}||j||<dS)aSet attribute associated with the given key to value. Parameters ---------- key : str Store the attribute for this vector key. attr : str Name of the additional attribute to store for the given key. val : object Value of the additional attribute to store for the given key. Returns ------- None )rarbN)rgtype get_indexr-)r/keyrSvalrKs r rLzKeyedVectors.set_vecattrNsO& dVDII;???s##%( dE"""r"cR||}|j||S)aGet attribute value associated with given key. Parameters ---------- key : str Vector key for which to fetch the attribute value. attr : str Name of the additional attribute to fetch for the given key. Returns ------- object Value of the additional attribute fetched for the given key. )rjr-)r/rkrSrKs r get_vecattrzKeyedVectors.get_vecattres'$s##}T"5))r"ct|j|jf}t||j||_|d|_dS)zPMake underlying vectors match index_to_key size; random-initialize any new rows.) prior_vectorsseedN)rr(r' prep_vectorsr+rgr,)r/rq target_shapes r resize_vectorszKeyedVectors.resize_vectorszsPD-..0@A #L SWXXX      r"c*t|jSN)rr(r6s r __len__zKeyedVectors.__len__s4$%%%r"ct|tr|Stfd|DS)abGet vector representation of `key_or_keys`. Parameters ---------- key_or_keys : {str, list of str, int, list of int} Requested key or list-of-keys. Returns ------- numpy.ndarray Vector representation for `key_or_keys` (1D if `key_or_keys` is single key, otherwise - 2D). c:g|]}|SrV get_vectorrWrkr/s r rXz,KeyedVectors.__getitem__..s%CCCts++CCCr")rrr{r )r/ key_or_keyss` r __getitem__zKeyedVectors.__getitem__sL k: . . 0??;// /CCCC{CCCDDDr"c|j|d}|dkr|St|ttjfr$d|cxkrt |jkrnn|S||Std|d)zReturn the integer index (slot/position) where the given key's vector is stored in the backing vectors array. rNKey 'z ' not present) r*getrrZrNintegerrr(KeyError)r/rkdefaultrls r rjzKeyedVectors.get_indexs ##C,, !8 7J c2:. / / 7A 7 7 7 7s4CT?U?U 7 7 7 7 7J  7N5355566 6r"Fc||}|r0||j||j|z }n |j|}|d|S)aGet the key's vector, as a 1D numpy array. Parameters ---------- key : str Key for vector to return. norm : bool, optional If True, the resulting vector will be L2-normalized (unit Euclidean length). Returns ------- numpy.ndarray Vector for the specified key. Raises ------ KeyError If the given key doesn't exist. F)write)rj fill_normsr+r,setflags)r/rknormrKresults r r{zKeyedVectors.get_vectorsl0s##  ) OO   \%(4:e+<|| |} |||| zz }|t||z }|st#d| d|dkr||z }|r,t%j|t*}|S)aGet the mean vector for a given list of keys. Parameters ---------- keys : list of (str or int or ndarray) Keys specified by string or int ids or numpy array. weights : list of float or numpy.ndarray, optional 1D array of same size of `keys` specifying the weight for each key. pre_normalize : bool, optional Flag indicating whether to normalize each keyvector before taking mean. If False, individual keyvector will not be normalized. post_normalize: bool, optional Flag indicating whether to normalize the final mean vector. If True, normalized mean vector will be return. ignore_missing : bool, optional If False, will raise error if a key doesn't exist in vocabulary. Returns ------- numpy.ndarray Mean vector for the list of keys. Raises ------ ValueError If the size of the list of `keys` and `weights` doesn't match. KeyError If any of the key doesn't exist in vocabulary and `ignore_missing` is false. rz!cannot compute mean with no inputNz8keys and weights array must have same number of elementsrrz' not present in vocabulary)r ValueErrorrrrNarrayonesrrr'r+r enumerater abs __contains__r{rrunitvecrMREAL) r/rJweights pre_normalizepost_normalizeignore_missingmean total_weightidxrkvecs r get_mean_vectorzKeyedVectors.get_mean_vectorsD t99> B@AA A gt $ $ (hw''G  )gc$ii((G t99 a( ( J x($,*<== !$ I IHC#w'' I s**GCL 1 11 ""3'' Iooc o>> s**GCL 1 11 # IGsGGGHHH I !  ',&D  7#D))0066D r"cp|j}|t|ks |j|`t|}tjdt ||g|g||dz|_n.||j|<||j|<||j |<|xjdz c_|S)aAdd one new vector at the given key, into existing slot if available. Warning: using this repeatedly is inefficient, requiring a full reallocation & copy, if this instance hasn't been preallocated to be ready for such incremental additions. Parameters ---------- key: str Key identifier of the added vector. vector: numpy.ndarray 1D numpy array with the vector values. Returns ------- int Index of the newly added vector, so that ``self.vectors[result] == vector`` and ``self.index_to_key[result] == key``. NzAdding single vectors to a KeyedVectors which grows by one each time can be costly. Consider adding in batches or preallocating to the required size.r) r)rr(warningswarn UserWarning add_vectorsrgr*r+)r/rkvector target_indexs r add_vectorzKeyedVectors.add_vectors* 3t99 $ !(9,(G !t99L MT      cUVH - - -  " " $ $ $*Q.DOO/2D l +%1D c ")/DL & OOq OOr"cbttr,gtj|dd}n)t|t rtj|}ifdDtjtt}tD]\}}|j vrd||<tj |dD]@}|}tjj |<j|At!j||jjf_D]5\}} tjj|| |fj|<6|rUfdtj |dD} ||j| <D]\}} | |j|| <dSdS) a^Append keys and their vectors in a manual way. If some key is already in the vocabulary, the old vector is kept unless `replace` flag is True. Parameters ---------- keys : list of (str or int) Keys specified by string or int ids. weights: list of numpy.ndarray or numpy.ndarray List of 1D np.array vectors or a 2D np.array of vectors. replace: bool, optional Flag indicating whether to replace vectors for keys which already exist in the map; if True - replace vectors, otherwise - keep old vectors. rrNc*g|]}|jSrVr&)rWrQextrass r rXz,KeyedVectors.add_vectors..Os.V.V.V1vay.V.V.Vr"r&TrcFg|]}|SrVrj)rWrrJr/s r rXz,KeyedVectors.add_vectors..cs)___3T^^DI66___r")rrrNrreshaperrgrJrrboolrr*nonzeror(appendr r+rMrr-) r/rJrrreplace in_vocab_maskrrkrSextra in_vocab_idxss `` ` r rzKeyedVectors.add_vectors6sX dJ ' ' (6Dhw''//266GG  & & (hw''G  F v{{}}.V.V.V.V .V.V.VWWWT$777 !$ * *HCd'' *%) c":}n--a0 * *Cs)C%():%;%;D c "   $ $S ) ) ) )t|Wm^-D-K-KDLL^-_-_`aa ! Z ZKD%"$)T]4-@%BW,X"Y"YDM$    J_____"*]B[B[\]B^___M*1-*@DL '% J J e5:=5I d#M22  J J J Jr"ct|ts|g}|dd}|||ddS)aAdd keys and theirs vectors in a manual way. If some key is already in the vocabulary, old vector is replaced with the new one. This method is an alias for :meth:`~gensim.models.keyedvectors.KeyedVectors.add_vectors` with `replace=True`. Parameters ---------- keys : {str, int, list of (str or int)} keys specified by their string or int ids. weights: list of numpy.ndarray or numpy.ndarray List of 1D np.array vectors or 2D np.array of vectors. rrT)rN)rrrr)r/rJrs r __setitem__zKeyedVectors.__setitem__hsQ$%% -6Dooa,,G w55555r"c6||ddkS)a9Can this model return a single index for this key? Subclasses that synthesize vectors for out-of-vocabulary words (like :class:`~gensim.models.fasttext.FastText`) may respond True for a simple `word in wv` (`__contains__()`) check but False for this more-specific check. rrrr/rks r has_index_forzKeyedVectors.has_index_for}s~~c2&&!++r"c,||Srv)rrs r rzKeyedVectors.__contains__s!!#&&&r"cJ|tfd|DS)z6Get the `key` from `keys_list` most similar to `key1`.c<g|]}|SrV similarity)rWrkkey1r/s r rXz6KeyedVectors.most_similar_to_given..s' Q Q Qs!;!; Q Q Qr")r )r/r keys_lists`` r most_similar_to_givenz"KeyedVectors.most_similar_to_givens/ Q Q Q Q Qy Q Q QRRSSr"c|}||}tj|||kd}fd|DS)z@Get all keys that are closer to `key1` than `key2` is to `key1`.rc6g|]}|kj|SrVr()rWrKe1_indexr/s r rXz,KeyedVectors.closer_than..s,___UUV^M^_!%(___r") distancesrjrNwhere)r/rkey2 all_distancese2_indexcloser_node_indicesrs` @r closer_thanzKeyedVectors.closer_thansut,, >>$''>>$'' h}}X7N'NOOPQR_____6I____r"zUse closer_than insteadc.|||Srv)r)r/word1word2s r words_closer_thanzKeyedVectors.words_closer_thansu---r"cNt|||dzS)z]Rank of the distance of `key2` from `key1`, in relation to distances of all keys from `key1`.r)rr)r/rrs r rankzKeyedVectors.ranks%4##D$//00144r"c td)NzThe `.vectors_norm` attribute is computed dynamically since Gensim 4.0.0. Use `.get_normed_vectors()` instead. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4AttributeErrorr6s r vectors_normzKeyedVectors.vectors_norms b   r"cdSrvrV)r/_s r rzKeyedVectors.vectors_norms r"cl||j|jdtjfz S)aGet all embedding vectors normalized to unit L2 length (euclidean), as a 2D numpy array. To see which key corresponds to which vector = which array row, refer to the :attr:`~gensim.models.keyedvectors.KeyedVectors.index_to_key` attribute. Returns ------- numpy.ndarray: 2D numpy array of shape ``(number_of_keys, embedding dimensionality)``, L2-normalized along the rows (key vectors). .)rr+r,rNnewaxisr6s r get_normed_vectorszKeyedVectors.get_normed_vectorss. |djbj999r"cr|j|r-tj|jd|_dSdS)z Ensure per-vector norms are available. Any code which modifies vectors should ensure the accompanying norms are either recalculated or 'None', to trigger a full recalculation later on-request. Nraxis)r,rNlinalgrr+)r/forces r rzKeyedVectors.fill_normss> : > > 1==DJJJ > >r"c td)NzThe index2entity attribute has been replaced by index_to_key since Gensim 4.0.0. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4rr6s r r;zKeyedVectors.index2entity b   r"c||_dSrvrr/rs r r;zKeyedVectors.index2entity!r"c td)NzThe index2word attribute has been replaced by index_to_key since Gensim 4.0.0. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4rr6s r r:zKeyedVectors.index2wordrr"c||_dSrvrrs r r:zKeyedVectors.index2wordrr"c td)Na!The vocab attribute was removed from KeyedVector in Gensim 4.0.0. Use KeyedVector's .key_to_index dict, .index_to_key list, and methods .get_vecattr(key, attr) and .set_vecattr(key, attr, new_val) instead. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4rr6s r rHzKeyedVectors.vocabs b   r"c.|dSrv)rHrs r rHzKeyedVectors.vocabs r"ctsdStjjdddd}fd|D_jD]}j||j|<tjr,tdj|_dtjD_ dS)zGSort the vocabulary so the most frequent words have the lowest indexes.Nr0rc*g|]}j|SrVr)rWrr/s r rXz=KeyedVectors.sort_by_descending_frequency..s!TTTT.s3TTTr"zDsorting after vectors have been allocated is expensive & error-proneci|]\}}|| SrVrV)rWiwords r z=KeyedVectors.sort_by_descending_frequency..sQQQDT1QQQr") rrNargsortr-r(rgr+loggerwarningrr*)r/count_sorted_indexesrQs` r sort_by_descending_frequencyz)KeyedVectors.sort_by_descending_frequencys4yy  F!z$-*@AA$$B$GTTTT?STTT     F FA#}Q/0DEDM!   t|   > NNa b b b<(<=DLQQId>O4P4PQQQr"cHtt|j|i|dS)aSave KeyedVectors to a file. Parameters ---------- fname : str Path to the output file. See Also -------- :meth:`~gensim.models.keyedvectors.KeyedVectors.load` Load a previously saved model. N)r=r$saverDs r rzKeyedVectors.saves- 'lD!!&777777r" c t|tr|dkrgSt|}t|}|pt j}|rd|}g}t jt jt |dt jt |zf} t||zD]V\} } t| tr| | 0| | d| d| | <W || ddd} fd|D|+t|tr|| |Stj|| j|z |sSt#j|t zd } fd | D}|d|S) aFind the top-N most similar keys. Positive keys contribute positively towards the similarity, negative keys negatively. This method computes cosine similarity between a simple mean of the projection weight vectors of the given keys and the vectors for each key in the model. The method corresponds to the `word-analogy` and `distance` scripts in the original word2vec implementation. Parameters ---------- positive : list of (str or int or ndarray) or list of ((str,float) or (int,float) or (ndarray,float)), optional List of keys that contribute positively. If tuple, second element specifies the weight (default `1.0`) negative : list of (str or int or ndarray) or list of ((str,float) or (int,float) or (ndarray,float)), optional List of keys that contribute negatively. If tuple, second element specifies the weight (default `-1.0`) topn : int or None, optional Number of top-N similar keys to return, when `topn` is int. When `topn` is None, then similarities for all keys are returned. clip_start : int Start clipping index. clip_end : int End clipping index. restrict_vocab : int, optional Optional integer which limits the range of vectors which are searched for most-similar values. For example, restrict_vocab=10000 would only check the first 10000 key vectors in the vocabulary order. (This may be meaningful if you've sorted the vocabulary by descending frequency.) If specified, overrides any values of ``clip_start`` or ``clip_end``. Returns ------- list of (str, float) or numpy.array When `topn` is int, a sequence of (key, similarity) is returned. When `topn` is None, then similarities for all keys are returned as a one-dimensional numpy array with the size of the vocabulary. rrgTF)rrrcg|]A}t|t|,|BSrV)rrrrjr|s r rXz-KeyedVectors.most_similar..Js[   $':c:3N3N SWSeSefiSjSj NN3     r"Ntopnreversechg|].}|zv j|zt|f/SrVr(float)rWsimall_keys clip_startdistsr/s r rXz-KeyedVectors.most_similar..VsV   j 0A  sZ/ 0%c 2C2C D   r")rrr!rrr+rN concatenaterr_EXTENDED_KEY_TYPESrrrZ most_similarrr,rr)r/positivenegativerrclip_endrestrict_vocabindexerrJweightritemrbestrrrs` ` @@r rzKeyedVectors.most_similarsHP dH % % $( I )))) 0s4<00  &J%HX!7!7H @V@V9V WXX"8h#677 & &IC$ 344 & D!!!! DG$$$"1gs ##D&UYjo#pp    +/     4:dC#8#8 4''d33 3DLH!45t<>>r"c4||g||S)aFind the top-N most similar keys. Parameters ---------- key : str Key topn : int or None, optional Number of top-N similar keys to return. If topn is None, similar_by_key returns the vector of similarity scores. restrict_vocab : int, optional Optional integer which limits the range of vectors which are searched for most-similar values. For example, restrict_vocab=10000 would only check the first 10000 key vectors in the vocabulary order. (This may be meaningful if you've sorted the vocabulary by descending frequency.) Returns ------- list of (str, float) or numpy.array When `topn` is int, a sequence of (key, similarity) is returned. When `topn` is None, then similarities for all keys are returned as a one-dimensional numpy array with the size of the vocabulary. r rr r)r/rkrr s r rzKeyedVectors.similar_by_key`s!0  3%d> ZZZr"c4||g||S)aFind the top-N most similar keys by vector. Parameters ---------- vector : numpy.array Vector from which similarities are to be computed. topn : int or None, optional Number of top-N similar keys to return, when `topn` is int. When `topn` is None, then similarities for all keys are returned. restrict_vocab : int, optional Optional integer which limits the range of vectors which are searched for most-similar values. For example, restrict_vocab=10000 would only check the first 10000 key vectors in the vocabulary order. (This may be meaningful if you've sorted the vocabulary by descending frequency.) Returns ------- list of (str, float) or numpy.array When `topn` is int, a sequence of (key, similarity) is returned. When `topn` is None, then similarities for all keys are returned as a one-dimensional numpy array with the size of the vocabulary. rr)r/rrr s r similar_by_vectorzKeyedVectors.similar_by_vectorzs!0  6(n ]]]r"cvddlm}t|}t|}fd|D}fd|D}|t|z }|t|z }|dks|dkrtd|||r|s)tdt dSt||gtd krd Stt|} tt|} tj fd | D} tj fd | D}  | }  | }tft }t| | |tj| |<t#t%|dkr)tdt dSfd}||}||}||||S)uCompute the Word Mover's Distance between two documents. When using this code, please consider citing the following papers: * `Rémi Flamary et al. "POT: Python Optimal Transport" `_ * `Matt Kusner et al. "From Word Embeddings To Document Distances" `_. Parameters ---------- document1 : list of str Input document. document2 : list of str Input document. norm : boolean Normalize all word vectors to unit length before computing the distance? Defaults to True. Returns ------- float Word Mover's distance between `document1` and `document2`. Warnings -------- This method only works if `POT `_ is installed. If one of the documents have no words that exist in the vocab, `float('inf')` (i.e. infinity) will be returned. Raises ------ ImportError If `POT `_ isn't installed. r)emd2cg|]}|v| SrVrVrWtokenr/s r rXz+KeyedVectors.wmdistance.."CCCuUd]CUCCCr"cg|]}|v| SrVrVrs r rXz+KeyedVectors.wmdistance..rr"zARemoved %d and %d OOV words from document 1 and 2 (respectively).zGAt least one of the documents had no words that were in the vocabulary.inf) documentsrc>g|]}|SrrzrWrrr/s r rXz+KeyedVectors.wmdistance..)OOOUtu488OOOr"c>g|]}|Sr$rzr%s r rXz+KeyedVectors.wmdistance..r&r"r&g:0yE>z;The distance matrix is all zeros. Aborting (returning inf).ctt}|}t|}|D]\}}|t |z ||<|S)Nr&)rrdoc2bowrr)documentdnbowdoc_lenrfreq dictionary vocab_lens r r,z%KeyedVectors.wmdistance..nbowsaiv...A%%h//D(mmG! / / Tg.#Hr")otrrrinforrrrsetrNrdoc2idxrrrix_rnp_sum)r/ document1 document2rr len_pre_oov1 len_pre_oov2diff1diff2doclist1doclist2v1v2 doc1_indices doc2_indicesdistance_matrixr,d1d2r/r0s` ` @@r wmdistancezKeyedVectors.wmdistancessN 9~~ 9~~ CCCC CCC CCCC CCC s9~~-s9~~- 19 k  k KK[]bdi j j j  NNd e e e<< 9i*@AAA  OO > 3I''I'' XOOOOOhOOO P P XOOOOOhOOO P P!))(33 !))(33  I 6fEEE>CBmm|\::; vo&& ' '$ . KKU V V V<<       T)__ T)__tBO,,,r"c t|tr|dkrgSt|}t|}t|tr|g}t|tr|g}fd||zD fd|D}fd|D}|st dfd|D}fd|D}t |d t |d d zz |s Stj |t zd } fd |D}|d|S)aFind the top-N most similar words, using the multiplicative combination objective, proposed by `Omer Levy and Yoav Goldberg "Linguistic Regularities in Sparse and Explicit Word Representations" `_. Positive words still contribute positively towards the similarity, negative words negatively, but with less susceptibility to one large distance dominating the calculation. In the common analogy-solving case, of two positive and one negative examples, this method is equivalent to the "3CosMul" objective (equation (4)) of Levy and Goldberg. Additional positive or negative examples contribute to the numerator or denominator, respectively - a potentially sensible but untested extension of the method. With a single positive example, rankings will be the same as in the default :meth:`~gensim.models.keyedvectors.KeyedVectors.most_similar`. Allows calls like most_similar_cosmul('dog', 'cat'), as a shorthand for most_similar_cosmul(['dog'], ['cat']) where 'dog' is positive and 'cat' negative Parameters ---------- positive : list of str, optional List of words that contribute positively. negative : list of str, optional List of words that contribute negatively. topn : int or None, optional Number of top-N similar words to return, when `topn` is int. When `topn` is None, then similarities for all words are returned. restrict_vocab : int or None, optional Optional integer which limits the range of vectors which are searched for most-similar values. For example, restrict_vocab=10000 would only check the first 10000 node vectors in the vocabulary order. This may be meaningful if vocabulary is sorted by descending frequency. Returns ------- list of (str, float) or numpy.array When `topn` is int, a sequence of (word, similarity) is returned. When `topn` is None, then similarities for all words are returned as a one-dimensional numpy array with the size of the vocabulary. rcvh|]5}t|t|jv |6SrV)rr r*rjrWrr/s r z3KeyedVectors.most_similar_cosmul..+sU   %)dG,, 159J1J NN4    r"clg|]0}t|tr|dn|1STrrr\r{rIs r rXz4KeyedVectors.most_similar_cosmul..0M   1;40E0E ODOODtO , , ,4   r"clg|]0}t|tr|dn|1SrLrMrIs r rXz4KeyedVectors.most_similar_cosmul..4rNr"z'cannot compute similarity with no inputcVg|]%}dtj|jz zdz &Srrrr+r,rWtermr/s r rXz4KeyedVectors.most_similar_cosmul..>6[[[$q3t|T22TZ??1D[[[r"cVg|]%}dtj|jz zdz &SrQrRrSs r rXz4KeyedVectors.most_similar_cosmul..?rUr"rrgư>Trc\g|](}|vj|t|f)SrVr)rWr all_wordsrr/s r rXz4KeyedVectors.most_similar_cosmul..Fs>fff#QT\eQef4$S)5s+<+<=fffr"N) rrr! init_simsr\rr rrr) r/r r rr  pos_dists neg_distsrrrXrs ` @@r most_similar_cosmulz KeyedVectors.most_similar_cosmulsT dH % % $( I ))))  h $ $ " zH h $ $ " zH    -5-@                     HFGG G\[[[RZ[[[ [[[[RZ[[[ YQ'''4 +B+B+BX+MN LD3y>>,A4PPPfffffffffete}r"cfd|D}t|t|kr:t|t|z }td||st dt fd|Dt} |d}t||}tt||dS)aRank the given words by similarity to the centroid of all the words. Parameters ---------- words : list of str List of keys. use_norm : bool, optional Whether to calculate centroid using unit-normed vectors; default True. Returns ------- list of (float, str) Ranked list of (similarity, key), most-similar to the centroid first. cg|]}|v| SrVrVrIs r rXz3KeyedVectors.rank_by_centrality..[s"===t =d===r"zGvectors for words %s are not present in the model, ignoring these wordsz'cannot select a word from an empty listc>g|]}|Sr$rz)rWrr/use_norms r rXz3KeyedVectors.rank_by_centrality..as)VVV4$//$X/>>VVVr"T)r)r) rrr3rrrr rMrrrsortedrY)r/wordsr` used_words ignored_wordsr+rrs` ` r rank_by_centralityzKeyedVectors.rank_by_centralityIs ====u=== z??c%jj ( uJJZ8M NNdfs t t t HFGG GVVVVV:VVVWW^^_cdd##GD#AAGT""c%,,d;;;;r"cD||ddS)a Which key from the given list doesn't go with the others? Parameters ---------- words : list of str List of keys. Returns ------- str The key further away from the mean of all keys. rr)re)r/rbs r doesnt_matchzKeyedVectors.doesnt_matchfs"&&u--b1!44r"ctj|}tj|d}t||}|||zz }|S)a*Compute cosine similarities between one vector and a set of other vectors. Parameters ---------- vector_1 : numpy.ndarray Vector from which similarities are to be computed, expected shape (dim,). vectors_all : numpy.ndarray For each row in vectors_all, distance from vector_1 is computed, expected shape (num_vectors, dim). Returns ------- numpy.ndarray Contains cosine distance between `vector_1` and each row in `vectors_all`, shape (num_vectors,). rr)rNrrr)vector_1 vectors_allr all_norms dot_products similaritiess r cosine_similaritiesz KeyedVectors.cosine_similaritiesvsR"y~~h''INN;QN77 ;11 #ti'78 r"rVct|tr|}n|}|sj}nfd|D}j|}d||z S)ahCompute cosine distances from given word or vector to all words in `other_words`. If `other_words` is empty, return distance between `word_or_vector` and all words in vocab. Parameters ---------- word_or_vector : {str, numpy.ndarray} Word or vector from which distances are to be computed. other_words : iterable of str For each word in `other_words` distance from `word_or_vector` is computed. If None or empty, distance of `word_or_vector` from all words in vocab is computed (including itself). Returns ------- numpy.array Array containing distances to all words in `other_words` from input `word_or_vector`. Raises ----- KeyError If either `word_or_vector` or any word in `other_words` is absent from vocab. c:g|]}|SrVrrIs r rXz*KeyedVectors.distances..s%JJJdT^^D11JJJr"r)rrr{r+rn)r/word_or_vector other_words input_vector other_vectors other_indicess` r rzKeyedVectors.distancess. nj 1 1 *??>::LL)L 8 LMMJJJJkJJJM L7M4++L-HHHHr"c4d|||z S)aXCompute cosine distance between two keys. Calculate 1 - :meth:`~gensim.models.keyedvectors.KeyedVectors.similarity`. Parameters ---------- w1 : str Input key. w2 : str Input key. Returns ------- float Distance between `w1` and `w2`. rrr/w1w2s r distancezKeyedVectors.distances"4??2r****r"cttj||tj||S)aCompute cosine similarity between two keys. Parameters ---------- w1 : str Input key. w2 : str Input key. Returns ------- float Cosine similarity between `w1` and `w2`. )rrrrws r rzKeyedVectors.similaritys2 8#DH--x/?R/I/IJJJr"c t|rt|std||d}||d}tt j|t j|S)a<Compute cosine similarity between two sets of keys. Parameters ---------- ws1 : list of str Sequence of keys. ws2: list of str Sequence of keys. Returns ------- numpy.ndarray Similarities between `ws1` and `ws2`. z)At least one of the passed list is empty.F)r)rZeroDivisionErrorrrrr)r/ws1ws2mean1mean2s r n_similarityzKeyedVectors.n_similaritys C QSXX Q#$OPP P$$S$>>$$S$>>8#E**H,`_. Parameters ---------- analogies : str Path to file, where lines are 4-tuples of words, split into sections by ": SECTION NAME" lines. See `gensim/test/test_data/questions-words.txt` as example. restrict_vocab : int, optional Ignore all 4-tuples containing a word not in the first `restrict_vocab` words. This may be meaningful if you've sorted the model vocabulary by descending frequency (which is standard in modern word embedding models). case_insensitive : bool, optional If True - convert all words to their uppercase form before evaluating the performance. Useful to handle case-mismatch between training tokens and words in the test set. In case of multiple case variants of a single word, the vector for the first occurrence (also the most frequent if vocabulary is sorted) is taken. dummy4unknown : bool, optional If True - produce zero accuracies for 4-tuples with out-of-vocabulary words. Otherwise, these tuples are skipped entirely and not used in the evaluation. similarity_function : str, optional Function name used for similarity calculation. Returns ------- score : float The overall evaluation score on the entire evaluation set sections : list of dict of {str : str or list of tuple of (str, str, str, str)} Results broken down by each section of the evaluation set. Each dict contains the name of the section under the key 'section', and lists of correctly and incorrectly predicted 4-tuples of words under the keys 'correct' and 'incorrect'. Nc`i|]*}||+SrVupperrjrWrQr/s r rz8KeyedVectors.evaluate_word_analogies..0/PPP 4>>!#4#4PPPr"c<i|]}||SrVrrs r rz8KeyedVectors.evaluate_word_analogies..2'HHH4>>!,,HHHr"rz=Evaluating word analogies for top %i words in the model on %srbz: rrrz,Missing section header before line #%i in %sc6g|]}|SrVrrWrs r rXz8KeyedVectors.evaluate_word_analogies..Es 0W0W0W$0W0W0Wr"cg|]}|SrVrVrs r rXz8KeyedVectors.evaluate_word_analogies..Gs0O0O0O$0O0O0Or"zSkipping invalid line #%i in %srz-Zero accuracy for line #%d with OOV words: %srz$Skipping line #%i with OOV words: %s)r r rr z%s: expected %s, predicted %srzTotal accuracyc3&K|] }|dV dS)rNrVrWss r z7KeyedVectors.evaluate_word_analogies..ns&9Y9Y1!I,9Y9Y9Y9Y9Y9Yr"c3&K|] }|dV dS)rNrVrs r rz7KeyedVectors.evaluate_word_analogies..os&;];]qAkN;];];];];];]r"dz0Quadruplets with out-of-vocabulary words: %.1f%%zrNB: analogies containing OOV words were skipped from evaluation! To change this behavior, use "dummy4unknown=True")r(reversedrr2ropenr to_unicode startswithrrlstripstriprsplitdebugr*rrr itertoolschain from_iterabler)r/ analogiesr case_insensitive dummy4unknownsimilarity_functionok_keysok_vocaboovsectionsrquadruplets_nofinline_nolineabcexpectedoriginal_key_to_indexignore predictedsimselementtotal oov_ratioanalogies_scores` r evaluate_word_analogiesz$KeyedVectors.evaluate_word_analogiesscR#O^O4  IPPPPhw>O>OPPPHHHHHHhw6G6GHHHH SUcenooo' Z 4 ( (/ IC!*3. I. I '--??4((,IC 00099'BBB*.++d*;*;*A*A*C*CPRacddGG"p()W[bdmZn)nooo!+P0W0W$**,,0W0W0W-Aq!XX0O0O$**,,0O0O0O-Aq!X%!!! $EwPYZZZ !#a'N(!AX,=!(AR!V^fnVn!q(h"LL)XZacgcmcmcocoppp#K077Aq(8KLLLL"LL)OQXZ^ZdZdZfZfggg ,0,=)(0D%AYF $I ,,q!fsQRcq,rrD(=D%#'"":J$ZGAJ$4$4$6$6$6PWXYPZ $0"Yf5L"(H4q & -Ldjjll\dfo p p p!E H,I *111aH2EFFFF ,33Q1h4GHHHH]. I/ I/ I/ I/ I/ I/ I/ I/ I/ I/ I/ I/ I/ I/ I/ I`  7 OOG $ $ $  - -g 6 6 6(IO999Y9YPX9Y9Y9YYYZZio;;;];]T\;];];]]]^^   #JJ/#5  F RRR  KKD   ;;EBB((s9B-L71A E;:L7;&F%!L7$F%%FL77L;>L;ct|dt|d}}||zdkr2td|dd|z||zz |||zdSdS)Nrrrrrrrrs r log_accuracyzKeyedVectors.log_accuracy~s !344c'+:N6O6O Y  "  KK$ "EGOw7J$KWV]`iVi       r"ctd||dtd||dtd|dS)Nz0Pearson correlation coefficient against %s: %.4frz.rr"c<i|]}||SrVrrs r rz4KeyedVectors.evaluate_word_pairs..rr"rencoding#c6g|]}|SrVrrs r rXz4KeyedVectors.evaluate_word_pairs..s (X(X(X$(X(X(Xr"cg|]}|SrVrVrs r rXz4KeyedVectors.evaluate_word_pairs..s(P(P(P$(P(P(Pr"zSkipping invalid line #%d in %srz/Zero similarity for line #%d with OOV words: %sr"z$Skipping line #%d with OOV words: %sz(No valid similarity judgements found in z8: either invalid format or all are out-of-vocabulary in rz>Pearson correlation coefficient against %s: %f with p-value %fzJSpearman rank-order correlation coefficient against %s: %f with p-value %fzPairs with unknown words: %d)r(rr*rrrrrrrr_rr2rrrrrr spearmanrpearsonrr)r/r delimiterrr rrrrsimilarity_goldsimilarity_modelrrrrrrrrrrrs` r evaluate_word_pairsz KeyedVectors.evaluate_word_pairss>V#O^O4  IPPPPhw>O>OPPPHHHHHHhw6G6GHHHH373Dh0t0 6EH555 C%.s^^CCMGT!4??3#7#7! !+Q(X(X$**YBWBW(X(X(XIAq##(P(P$**Y:O:O(P(P(PIAq##Cjj& 2!!! $EwPUVVV !(!AX,=!q(g"LL)Z\ceieoeoeqeqrrr,33C888+2237777"KK(NPWY]YcYcYeYefff #**3///$++DOOAq,A,ABBBB/C C C C C C C C C C C C C C C C4!6D   5D  5 5 5 5?##s+;'<'<<<<< 75770477 ??4DEE.2BCC  Hc S%9%99C?IIc c/&:&:S&@ACGI UW\^efg^hjqrsjtuuu X 8A;      3S999 $$Wh 5III)++sV H)6.H%AC>=H>-D.+H-D..CH H)HH)HH)) H2zmUse fill_norms() instead. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4c||r0td|dSdS)a(Precompute data helpful for bulk similarity calculations. :meth:`~gensim.models.keyedvectors.KeyedVectors.fill_norms` now preferred for this purpose. Parameters ---------- replace : bool, optional If True - forget the original vectors and only keep the normalized ones. Warnings -------- You **cannot sensibly continue training** after doing a replace on a model's internal KeyedVectors, and a replace is no longer necessary to save RAM. Do not use this method. zXdestructive init_sims(replace=True) deprecated & no longer required for space-efficiencyN)rrrunit_normalize_all)r/rs r rYzKeyedVectors.init_simssQ,   & NNu v v v  # # % % % % % & &r"c||xj|jdtjfzc_tjt |jf|_dS)z{Destructively scale all vectors to unit-length. You cannot sensibly continue training after such a step. .N)rr+r,rNrrrr6s r rzKeyedVectors.unit_normalize_allsS   3 ?33 Wc$,//122 r"c|||}|stdt|||t d|Dz }|S)aCompute the relative cosine similarity between two words given top-n similar words, by `Artuur Leeuwenberga, Mihaela Velab , Jon Dehdaribc, Josef van Genabithbc "A Minimally Supervised Approach for Synonym Extraction with Word Embeddings" `_. To calculate relative cosine similarity between two words, equation (1) of the paper is used. For WordNet synonyms, if rcs(topn=10) is greater than 0.10 then wa and wb are more similar than any arbitrary word pairs. Parameters ---------- wa: str Word for which we have to look top-n similar word. wb: str Word for which we evaluating relative cosine similarity with wa. topn: int, optional Number of top-n similar words to look with respect to wa. Returns ------- numpy.float64 Relative cosine similarity between wa and wb. zFCannot calculate relative cosine similarity without any similar words.c3 K|] \}}|V dSrvrV)rWrrs r rz:KeyedVectors.relative_cosine_similarity..6s&3K3KFAsC3K3K3K3K3K3Kr")rrrrr )r/wawbrrrcss r relative_cosine_similarityz'KeyedVectors.relative_cosine_similaritysm0##B-- geff fDOOB++,,3K3Kd3K3K3K0K0KL r"r0c |tj}|sdnd} jvr-tjfd} n8|t ddtdj} |t d |tj || 5} | D]F} | || d | d d G dddn #1swxYwYt d |j|tjjfjjksJd} t%jD]\}}||krn| dz } t'jt+d| | }tj || 5}|r3| |d jd d |D]}|}|r[| ||d d |t.zg| ||d d d|Dd d  ddddS#1swxYwYdS)aStore the input-hidden weight matrix in the same format used by the original C word2vec-tool, for compatibility. Parameters ---------- fname : str File path to save the vectors to. fvocab : str, optional File path to save additional vocabulary information to. `None` to not store the vocabulary. binary : bool, optional If True, the data wil be saved in binary word2vec format, else it will be saved in plain text. total_vec : int, optional Explicitly specify total number of vectors (in case word vectors are appended with document vectors afterwards). write_header : bool, optional If False, don't write the 1st line declaring the count of vectors and dimensions. This is the format used by e.g. gloVe vectors. prefix : str, optional String to prepend in front of each stored word. Default = no prefix. append : bool, optional If set, open `fname` in `ab` mode instead of the default `wb` mode. sort_attr : str, optional Sort the output vectors in descending order of this attribute. Default: most frequent keys first. Nrabc2| Srv)rn)rQr/ sort_attrs r z3KeyedVectors.save_word2vec_format..\sUYUeUefgirUsUsTsr")rkzCannot store vocabulary with 'z'' because that attribute does not existzIattribute %s not present in %s; will store in internal index_to_key orderzstoring vocabulary in %s  rz(storing %sx%s projection weights into %srrc34K|]}t|VdSrv)repr)rWrls r rz4KeyedVectors.save_word2vec_format..s(8Y8Ysc8Y8Y8Y8Y8Y8Yr")rr(r-rar*rJrrrr2rrrrnencoder'r+rrrrrangerMrtobytesjoin)r/fnamefvocabbinary total_vec write_headerprefixrrmodestore_order_vocab_keysvoutrindex_id_countrrl keys_to_writefoutrk key_vectors` ` r save_word2vec_formatz!KeyedVectors.save_word2vec_format:s:  /D-..I!+ttt  % 7%+D,=,B,B,D,DJsJsJsJsJs%t%t%t " "  v !t)!t!t!tuuu NN[4   &*%6 "  g KK2F ; ; ;FD)) gT2ggDJJ&V$VV1A1A$ 1R1RVVV]]^deeffffg g g g g g g g g g g g g g g g  > 4K[]bcccD%&&(89T\=OOOOO  122  FAsCx  a NN!a(@(@BXYY Zt $ $ n O i>>$*:>>>EEfMMNNN$ n n!#Y nJJ&0#00077??*BSBSTXBYBYBaBaBcBccddddJJ&]#]]8Y8Yj8Y8Y8Y0Y0Y]]]ddekllmmmm  n n n n n n n n n n n n n n n n n n ns&?A DDDC-K  KKstrictc 2t||||||||| S)aLoad KeyedVectors from a file produced by the original C word2vec-tool format. Warnings -------- The information stored in the file is incomplete (the binary tree is missing), so while you can query for word similarity etc., you cannot continue training with a model loaded this way. Parameters ---------- fname : str The file path to the saved word2vec-format file. fvocab : str, optional File path to the vocabulary.Word counts are read from `fvocab` filename, if set (this is the file generated by `-save-vocab` flag of the original C tool). binary : bool, optional If True, indicates whether the data is in binary word2vec format. encoding : str, optional If you trained the C model using non-utf8 encoding for words, specify that encoding in `encoding`. unicode_errors : str, optional default 'strict', is a string suitable to be passed as the `errors` argument to the unicode() (Python 2.x) or str() (Python 3.x) function. If your source file may include word tokens truncated in the middle of a multibyte unicode character (as is common from the original word2vec.c tool), 'ignore' or 'replace' may help. limit : int, optional Sets a maximum number of word-vectors to read from the file. The default, None, means read all. datatype : type, optional (Experimental) Can coerce dimensions to a non-default float type (such as `np.float16`) to save memory. Such types may result in much slower bulk operations or incompatibility with optimized routines.) no_header : bool, optional Default False means a usual word2vec-format file, with a 1st line declaring the count of following vectors & number of dimensions. If True, the file is assumed to lack a declaratory (vocab_size, vector_size) header and instead start with the 1st vector, and an extra reading-pass will be used to discover the number of vectors. Works only with `binary=False`. Returns ------- :class:`~gensim.models.keyedvectors.KeyedVectors` Loaded model. )rrrunicode_errorslimitdatatype no_header)_load_word2vec_format) clsrrrrrrrrs r load_word2vec_formatz!KeyedVectors.load_word2vec_formats1^% vfxXf(i    r"r"c d}td|tj|d5}tj||}d|D\} } | |jkstd| |fz|r ttj | z} t| D]} g} | d}|d krn|d kr| |8tjd | || } t!j| | t }| |jvr?|dz }||j|| <||j|| <nt-|D]\}}tj||| d}t1|| dzkrtd|z|dd|ddD}} | |jvr?|dz }||j|| <||j|| <dddn #1swxYwY|dd|d|jjd|dS)a Merge in an input-hidden weight matrix loaded from the original C word2vec-tool format, where it intersects with the current vocabulary. No words are added to the existing vocabulary, but intersecting words adopt the file's weights, and non-intersecting words are left alone. Parameters ---------- fname : str The file path to load the vectors from. lockf : float, optional Lock-factor value to be set for any imported word-vectors; the default value of 0.0 prevents further updating of the vector during subsequent training. Use 1.0 to allow further training updates of merged vectors. binary : bool, optional If True, `fname` is in the binary word2vec C format. encoding : str, optional Encoding of `text` for `unicode` function (python2 only). unicode_errors : str, optional Error handling behaviour, used as parameter for `unicode` function (python2 only). r"loading projection weights from %srrc34K|]}t|VdSrvrZrWxs r rz9KeyedVectors.intersect_word2vec_format..s(&F&F!s1vv&F&F&F&F&F&Fr"z&incompatible vector size %d in file %sTr  r"rerrorsr&rz;invalid vector on line %s (is this really the text format?)c,g|]}t|SrV)rrs r rXz:KeyedVectors.intersect_word2vec_format..s.J.J.J1tAww.J.J.Jr"Nintersect_word2vec_formatzmerged z vectors into z matrix from )msg)rr2rrrreadlinerr'rrritemsizerreadrrrN fromstringr*r+rj vectors_lockfrrstripradd_lifecycle_eventr)r/rlockfrrr overlap_countrheader vocab_sizer' binary_lenrrchrrrpartss r rz&KeyedVectors.intersect_word2vec_formatsG.  8%@@@ Zt $ $ I%cllnnxHHHF&F&Fv||~~&F&F&F #J $"22 b !I[Z_L`!`aaa I"4[[1K? z**IIAD, XXa[[:"!;, KKOOO , !+CHHTNNXVdeeeD mCHHZ,@,@MMMGt00I%* =D T^^D%9%9:CH*4>>$+?+?@I &/s^^IIMGT!,T[[]]XVdeeekkloppE5zz[1_4r()fip)pqqq$)!H.J.Jabb .J.J.J'Dt00I%* =D T^^D%9%9:CH*4>>$+?+?@A I I I I I I I I I I I I I I IB   '_-__t|7I__X]__ !     sIJJJrJallow_inference copy_vecattrsreturnc gt}}|D]=}||vr7||||r|n|jvr||>t |jt ||jj}|D]r}||}t|d||t ||rF|j D]>} | || | || /#t$rY;wxYws|S)aProduce vectors for all given keys as a new :class:`KeyedVectors` object. Notes ----- The keys will always be deduplicated. For optimal performance, you should not pass entire corpora to the method. Instead, you should construct a dictionary of unique words in your corpus: >>> from collections import Counter >>> import itertools >>> >>> from gensim.models import FastText >>> from gensim.test.utils import datapath, common_texts >>> >>> model_corpus_file = datapath('lee_background.cor') # train word vectors on some corpus >>> model = FastText(corpus_file=model_corpus_file, vector_size=20, min_count=1) >>> corpus = common_texts # infer word vectors for words from another corpus >>> word_counts = Counter(itertools.chain.from_iterable(corpus)) # count words in your corpus >>> words_by_freq = (k for k, v in word_counts.most_common()) >>> word_vectors = model.wv.vectors_for_all(words_by_freq) # create word-vectors for words in your corpus Parameters ---------- keys : iterable The keys that will be vectorized. allow_inference : bool, optional In subclasses such as :class:`~gensim.models.fasttext.FastTextKeyedVectors`, vectors for out-of-vocabulary keys (words) may be inferred. Default is True. copy_vecattrs : bool, optional Additional attributes set via the :meth:`KeyedVectors.set_vecattr` method will be preserved in the produced :class:`KeyedVectors` object. Default is False. To ensure that *all* the produced vectors will have vector attributes assigned, you should set `allow_inference=False`. Returns ------- keyedvectors : :class:`~gensim.models.keyedvectors.KeyedVectors` Vectors for all the given keys. r&N)r3addr*rr$r'rr+r_add_word_to_kvr-rLrnr) r/rJr&r'rHseenrkkvrrSs r vectors_for_allzKeyedVectors.vectors_for_alls+X#%%t & &C$ & ?I448IJ&LL%%% $*CJJdl>P Q Q Q  C3iG Bc7CJJ ? ? ?  MDsD$2B2B32M2MNNNN# s7+C## C0/C0c|j|_||jD]/}||d}||jzdz}||j|<0|jd=|jdkr3ttd|jdz|j z|_ n |j |_ |j |_ |`|` |`|`|` dS)zXConvert a deserialized older Doc2VecKeyedVectors instance to latest generic KeyedVectorsoffsetrrrN)r9rHrCr*rJrn max_rawintr-rr offset2doctagr( vectors_docsr+r0)r/rQ old_offset true_indexs r r@z!KeyedVectors._upconvert_old_d2vkv;s\  !!###"'')) . .A))!X66J#do59J#-D a M( # ?R  3 $U1do.A%B%B C CdFX XD   $ 2D ( L   J O    r"c td)Nz7Call similarity_unseen_docs on a Doc2Vec model instead.)NotImplementedErrorrs r similarity_unseen_docsz#KeyedVectors.similarity_unseen_docsOs!"[\\\r")NN)rrv)F)NTFT)NF)NNrrNNN)rN)T)NNrN)rV)rTFr)rrrTF)r)NFNTrFr0)r"Frr)TF)Hr5 __module__ __qualname__rNrr1r7r>rCrgrLrnrtrwr~rjr{rrrrrrrrrrrrpropertyrsetterrrr;r:rHrrrrrrrFr\rerg staticmethodrnrrzrrrrrrrrYrrr classmethodrr rrrr.r@r8 __classcell__)r4s@r r$r$s*+2:D+)+)+)+)Z^^^(((((2 X X X!s!s!s!sF))).****&&&EEE& 7 7 7 7    DZ())00*)0????B&&&P0J0J0J0Jd666* , , ,'''TTT```Z)**..+*.555  X    ::: > > > >  X """  X """  X  \\ R R R88888"QU)-RRRRh????[[[[4^^^^4[-[-[-[-|IMVVVVp<<<<:555 \, I I I ID+++&KKK$EEE,\6FJ5Cw)w)w)w)r\CC\C 39HMg,g,g,g,RZ ^&&& &.333@RV/6LnLnLnLn\#EFS[1 1 1 [1 f= = = = ~GK.3>>H>t>'+>8F>>>>@(]]]]]]]r"r$c eZdZdZdZdZdS) CompatVocabc Hd|_|j|dS)zA single vocabulary item, used internally for collecting per-word frequency/sampling info, and for constructing binary trees (incl. both word leaves and inner nodes). Retained for now to ease the loading of older models. rN)r0rAupdate)r/rFs r r1zCompatVocab.__init__[s'   V$$$$$r"c"|j|jkSrv)r0)r/others r __lt__zCompatVocab.__lt__dszEK''r"cfdtjD}jjdd|dS)Nc^g|])}|d|dj|*S)r:)rrAr|s r rXz'CompatVocab.__str__..hsAppp\_\j\jkn\o\op333 c 2 23pppr")rarAr4r5r)r/valss` r r7zCompatVocab.__str__gsJppppvdm?T?Tppp>222DIIdOOOODDr"N)r5r9r:r1rFr7rVr"r rArAYsF%%%(((EEEEEr"rAc(||rtd|dS|||}|||z }n*||vr ||}ntd|d}||d|dS)Nz>- -    *j(a_```aar"ct|D]S}|} | dkrtdt| |||\} } t ||| | |TdS)Nr"r_)rrr`_word2vec_line_to_vectorr+) rr-rNr"r'rrrrrrrs r _word2vec_read_textrgs}$$??||~~ 3; ecdd d0xQYZZ gFD':>>>> ??r"ctj|||d}|dfd|ddD}}||fS)Nrrrc&g|] }|SrVrV)rWrrs r rXz,_word2vec_line_to_vector..s!>>>qxx{{>>>r"r)rrrr)rrrrr%rrs ` r rfrfse  T[[]]Xn U U U [ [\_ ` `E!H>>>>E!""I>>>'D =r"cd}tjD]K}|}|dks||krn(|r't||||\}} t | }L||fS)Nr")rr0rrfr) rrrrrr'r"rrrs r _word2vec_detect_sizes_textrksKo''## ||~~ 3; *-  E   0xQYZZ g'll { ""r"Frric vd} |td|i} tj|d5} | D]Q} tj| |\} }t|| | <R dddn #1swxYwYtd|tj|d5} |rQ|rtdt| ||||\}}| tj|d} nItj| |}d|D\}}|rt||}|||| }|rt| || ||||| | nt| || |||||dddn #1swxYwY|jjd t#|krgtd |jjd t#|t%|jdt#||_t#||f|jjksJ|d d |jjd|jjd||||S)a"Load the input-hidden weight matrix from the original C word2vec-tool format. Note that the information stored in the file is incomplete (the binary tree is missing), so while you can query for word similarity etc., you cannot continue training with a model loaded this way. Parameters ---------- fname : str The file path to the saved word2vec-format file. fvocab : str, optional File path to the vocabulary. Word counts are read from `fvocab` filename, if set (this is the file generated by `-save-vocab` flag of the original C tool). binary : bool, optional If True, indicates whether the data is in binary word2vec format. encoding : str, optional If you trained the C model using non-utf8 encoding for words, specify that encoding in `encoding`. unicode_errors : str, optional default 'strict', is a string suitable to be passed as the `errors` argument to the unicode() (Python 2.x) or str() (Python 3.x) function. If your source file may include word tokens truncated in the middle of a multibyte unicode character (as is common from the original word2vec.c tool), 'ignore' or 'replace' may help. limit : int, optional Sets a maximum number of word-vectors to read from the file. The default, None, means read all. datatype : type, optional (Experimental) Can coerce dimensions to a non-default float type (such as `np.float16`) to save memory. Such types may result in much slower bulk operations or incompatibility with optimized routines.) binary_chunk_size : int, optional Read input file in chunks of this many bytes for performance reasons. Returns ------- object Returns the loaded model as an instance of :class:`cls`. Nzloading word counts from %srrRr z.no_header only available for text-format filesrc,g|]}t|SrVrrs r rXz)_load_word2vec_format.. s&F&F&F!s1vv&F&F&Fr"r&rz=duplicate words detected, shrinking matrix size from %i to %ir zloaded z matrix of type z from )rrr)rr2rrrrrrZr7rkcloserr`rdrgr+rrrrr)r rrrrrrrrrarNrrrr0r"r'r!r-s r rrspRF * 16::: Z % % * * *#.tNKKKQQSSYY[[ e"5zzt  * * * * * * * * * * * * * * * *  KK4e<<< E4 nC  G v)*Z[[[*Ec5RZ\jlt*u*u' K IIKKK*UD))CC%cllnnxHHHF&F&Fv||~~&F&F&F #J  0Z//J Sj 9 9 9  n !R[(NTego     R[(Tbdl m m m+nnnnnnnnnnnnnnn, zc"gg%> K J Q R   'rz)CGG)'<== GG[ !RZ%5 5555 Wbj& W W 8H W WPU W W Is%ABB BC-G  GGc$tj|i|S)zPAlias for :meth:`~gensim.models.keyedvectors.KeyedVectors.load_word2vec_format`.)r$r )rErFs r r r &s  ,d =f = ==r"c|rItjtj||dz}n tj}||tdz |z S)zGet a random vector, derived deterministically from `seed_string` if supplied. Useful for initializing KeyedVectors that will be the starting projection/input layers of _2Vec models. lg?)rNrandom GeneratorSFC64r default_prngrMr)size seed_stringhashfxnonces r pseudorandom_weak_vectorry+sr "y""29??77;3G3G*3T#U#UVV! KK   $ $T * *S 0D 88r"c$|tjd}|j|kr|S|\}}tj|}|||}|dz}|dz}||z}||d|jdd|jdf<|S) zReturn a numpy array of the given shape. Reuse prior_vectors object or values to extent possible. Initialize new values randomly if requested. N)rr)rqr&g@g?rr)rNrrrq default_rng)rsrprqr target_countr'rng new_vectorss r rrrr8s )(( l* ,L+ )  T  * *C**\*77K3K3K;KFSK-%a((!M,?,B*BBC r")r])@__doc__loggingsysrrnumbersrtypingrnumpyrrrrrr r r r6r r rrrrNscipyrscipy.spatial.distancergensimrrgensim.corpora.dictionaryr gensim.utilsr getLoggerr5rr\rZrrrr!SaveLoadr$Word2VecKeyedVectorsDoc2VecKeyedVectorsEuclideanKeyedVectorsrAVocabr+r\rdrgrfrkmaxsizerr hashryrrrVr"r rs^^@ ((((((""""""""000000######  8 $ $3 # CRZ8   }]}]}]}]}]5>}]}]}]B4$"$EEEEEEEE(  ...(***4aaaa&??? # # # xkDEZVVVVr>>> 04T 9 9 9 9.2r"