c\dZddlZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z mZmZmZmZddl ZddlmZmZdd lmZdd lmZmZdd lmZmZeje Z! dd l"m#Z#m$Z$m%Z%n#e&$rej'wxYw dd l(m)Z)m*Z*m+Z+m,Z,n"#e&$rdZ, d#dZ) d$dZ* d$dZ+YnwxYwGddeddZ-e GddZ.GddeZ/Gddej0Z1Gddej0Z2Gdd Z3Gd!d"Z4dS)%a Introduction ============ Learn paragraph and document embeddings via the distributed memory and distributed bag of words models from `Quoc Le and Tomas Mikolov: "Distributed Representations of Sentences and Documents" `_. The algorithms use either hierarchical softmax or negative sampling; see `Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean: "Efficient Estimation of Word Representations in Vector Space, in Proceedings of Workshop at ICLR, 2013" `_ and `Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean: "Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013" `_. For a usage example, see the `Doc2vec tutorial `_. **Make sure you have a C compiler before installing Gensim, to use the optimized doc2vec routines** (70x speedup compared to plain NumPy implementation, https://rare-technologies.com/parallelizing-word2vec-in-python/). Usage examples ============== Initialize & train a model: .. sourcecode:: pycon >>> from gensim.test.utils import common_texts >>> from gensim.models.doc2vec import Doc2Vec, TaggedDocument >>> >>> documents = [TaggedDocument(doc, [i]) for i, doc in enumerate(common_texts)] >>> model = Doc2Vec(documents, vector_size=5, window=2, min_count=1, workers=4) Persist a model to disk: .. sourcecode:: pycon >>> from gensim.test.utils import get_tmpfile >>> >>> fname = get_tmpfile("my_doc2vec_model") >>> >>> model.save(fname) >>> model = Doc2Vec.load(fname) # you can continue training with the loaded model! Infer vector for a new document: .. sourcecode:: pycon >>> vector = model.infer_vector(["system", "response"]) N) namedtuple defaultdict)Iterable) default_timer) dataclass)zerosfloat32vstackintegerdtype)utilsmatutils) deprecated)Word2Vec FAST_VERSION) KeyedVectorspseudorandom_weak_vector)train_document_dbowtrain_document_dmtrain_document_dm_concat)d2v_train_epoch_dbowd2v_train_epoch_dm_concatd2v_train_epoch_dmCORPUSFILE_VERSIONFTc tdNz4Training with corpus_file argument is not supported.NotImplementedError)model corpus_fileoffset start_doctag _cython_vocab _cur_epoch_expected_examples_expected_wordswork_neu1 docvecs_count word_vectors word_locks train_words learn_doctags learn_words learn_hiddendoctag_vectors doctag_lockss 5lib/python3.11/site-packages/gensim/models/doc2vec.pyrr_""XYYYc tdrrr r!r"r#r$r%r&r'r(r)r*r+r,r.r/r0r1r2s r3rrer4r5c tdrrr7s r3rrkr4r5ceZdZdZdZdS)TaggedDocumentaRepresents a document along with a tag, input document format for :class:`~gensim.models.doc2vec.Doc2Vec`. A single document, made up of `words` (a list of unicode string tokens) and `tags` (a list of tokens). Tags may be one or more unicode string tokens, but typical practice (which will also be the most memory-efficient) is for the tags list to include a unique integer id as the only tag. Replaces "sentence as a list of words" from :class:`gensim.models.word2vec.Word2Vec`. c@|jjd|jd|jdS)zHuman readable representation of the object's state, used for debugging. Returns ------- str Human readable representation of the object's state (words and tags). ) __class____name__wordstagsselfs r3__str__zTaggedDocument.__str__|s( $~666 DIIINNr5N)r? __module__ __qualname____doc__rDr5r3r:r:rs2 O O O O Or5r:z words tagscleZdZUdZdZeed<eed<eed<edZej dZdS) DoctagaMA dataclass shape-compatible with keyedvectors.SimpleVocab, extended to record details of string document tags discovered during the initial vocabulary scan. Will not be used if all presented document tags are ints. No longer used in a completed model: just used during initial scan, and for backward compatibility. ) doc_countindex word_countrKrLrMc|jSNrKrBs r3countz Doctag.counts ~r5c||_dSrOrP)rCnew_vals r3rQz Doctag.counts  r5N) r?rErFrG __slots__int__annotations__propertyrQsetterrHr5r3rJrJsx 5INNN JJJOOO X \!!\!!!r5rJceZdZ d+fd Zed Zed Zeed d Zej ed dZdZ fdZ dZ d,dZ dZ d-fd ZedZdZdZd.dZdZdZd/dZed d0d!Zefd"Zd1fd# Z d2d%Zd3d&Zd'Zd4d)Zd.d*ZxZ S)5Doc2VecNdrrH Tc |}|||_t||_t||_t||_|r2|r0|d|zz|z|_t d|j||_| pt|j| |_ tj dt|j _tt |jd|||jd|zdz|j| |||d |dS) u/Class for training, using and evaluating neural networks described in `Distributed Representations of Sentences and Documents `_. Parameters ---------- documents : iterable of list of :class:`~gensim.models.doc2vec.TaggedDocument`, optional Input corpus, can be simply a list of elements, but for larger corpora,consider an iterable that streams the documents directly from disk/network. If you don't supply `documents` (or `corpus_file`), the model is left uninitialized -- use if you plan to initialize it in some other way. corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `documents` to get performance boost. Only one of `documents` or `corpus_file` arguments need to be passed (or none of them, in that case, the model is left uninitialized). Documents' tags are assigned automatically and are equal to line number, as in :class:`~gensim.models.doc2vec.TaggedLineDocument`. dm : {1,0}, optional Defines the training algorithm. If `dm=1`, 'distributed memory' (PV-DM) is used. Otherwise, `distributed bag of words` (PV-DBOW) is employed. vector_size : int, optional Dimensionality of the feature vectors. window : int, optional The maximum distance between the current and predicted word within a sentence. alpha : float, optional The initial learning rate. min_alpha : float, optional Learning rate will linearly drop to `min_alpha` as training progresses. seed : int, optional Seed for the random number generator. Initial vectors for each word are seeded with a hash of the concatenation of word + `str(seed)`. Note that for a fully deterministically-reproducible run, you must also limit the model to a single worker thread (`workers=1`), to eliminate ordering jitter from OS thread scheduling. In Python 3, reproducibility between interpreter launches also requires use of the `PYTHONHASHSEED` environment variable to control hash randomization. min_count : int, optional Ignores all words with total frequency lower than this. max_vocab_size : int, optional Limits the RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM. Set to `None` for no limit. sample : float, optional The threshold for configuring which higher-frequency words are randomly downsampled, useful range is (0, 1e-5). workers : int, optional Use these many worker threads to train the model (=faster training with multicore machines). epochs : int, optional Number of iterations (epochs) over the corpus. Defaults to 10 for Doc2Vec. hs : {1,0}, optional If 1, hierarchical softmax will be used for model training. If set to 0, and `negative` is non-zero, negative sampling will be used. negative : int, optional If > 0, negative sampling will be used, the int for negative specifies how many "noise words" should be drawn (usually between 5-20). If set to 0, no negative sampling is used. ns_exponent : float, optional The exponent used to shape the negative sampling distribution. A value of 1.0 samples exactly in proportion to the frequencies, 0.0 samples all words equally, while a negative value samples low-frequency words more than high-frequency words. The popular default value of 0.75 was chosen by the original Word2Vec paper. More recently, in https://arxiv.org/abs/1804.04212, Caselles-Dupré, Lesaint, & Royo-Letelier suggest that other values may perform better for recommendation applications. dm_mean : {1,0}, optional If 0, use the sum of the context word vectors. If 1, use the mean. Only applies when `dm` is used in non-concatenative mode. dm_concat : {1,0}, optional If 1, use concatenation of context vectors rather than sum/average; Note concatenation results in a much-larger model, as the input is no longer the size of one (sampled or arithmetically combined) word vector, but the size of the tag(s) and all words in the context strung together. dm_tag_count : int, optional Expected constant number of document tags per document, when using dm_concat mode. dbow_words : {1,0}, optional If set to 1 trains word-vectors (in skip-gram fashion) simultaneous with DBOW doc-vector training; If 0, only trains doc-vectors (faster). trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during current method call and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. callbacks : :obj: `list` of :obj: `~gensim.models.callbacks.CallbackAny2Vec`, optional List of callbacks that need to be executed/run at specific stages during training. shrink_windows : bool, optional New in 4.1. Experimental. If True, the effective window size is uniformly sampled from [1, `window`] for each target word during training, to match the original word2vec algorithm's approximate weighting of context words by distance. Otherwise, the effective window size is always fixed to `window` words to either side. Some important internal attributes are the following: Attributes ---------- wv : :class:`~gensim.models.keyedvectors.KeyedVectors` This object essentially contains the mapping between words and embeddings. After training, it can be used directly to query those embeddings in various ways. See the module level docstring for examples. dv : :class:`~gensim.models.keyedvectors.KeyedVectors` This object contains the paragraph vectors learned from the training data. There will be one such vector for each unique document tag supplied during training. They may be individually accessed using the tag as an indexed-access key. For example, if one of the training documents used a tag of 'doc003': .. sourcecode:: pycon >>> model.dv['doc003'] Nz)using concatenative %d-dimensional layer1) mapfile_pathr\r ) sentencesr! vector_sizesg null_word callbackswindowepochsshrink_windowsrH) cbow_meanrU dbow_words dm_concat dm_tag_count layer1_sizeloggerinfordrdvnponesREAL vectors_lockfsuperrZ__init__)rC documentsr!rddm_meandmrlrmrnrr dv_mapfilecomment trim_rulergrhrirjkwargscorpus_iterabler>s r3rxzDoc2Vec.__init__s)n$  %$DNj//Y --  W) W ,F ;{JD  KKCTEU V V V&O T%5JOOO!# 6 6 6%gt% %#(B!|n)  r5c|j S)zIndicates whether 'distributed memory' (PV-DM) will be used, else 'distributed bag of words' (PV-DBOW) is used. rerBs r3r{z Doc2Vec.dm5s 7{r5c|jS)zIndicates whether 'distributed bag of words' (PV-DBOW) will be used, else 'distributed memory' (PV-DM) is used. rrBs r3dbowz Doc2Vec.dbow=s wr5z-The `docvecs` property has been renamed `dv`.c|jSrOrrrBs r3docvecszDoc2Vec.docvecsEs wr5c||_dSrOr)rCvalues r3rzDoc2Vec.docvecsJsr5c6d|j_d|j_dS)z!Resets the current word vectors. N)wvnormsrrrBs r3_clear_post_trainzDoc2Vec._clear_post_trainOs  r5ctt||j|jdzdS)Ni)seed)rwrZ init_weightsrrresize_vectorsr)rCr>s r3rzDoc2Vec.init_weightsTsE gt))+++ DI$455555r5cf|jj|j_|jj|j_|jj|j_|j|_|j|_|jj|j_|jj|j_|jj|j_|dS)aCopy shareable data structures from another (possibly pre-trained) model. This specifically causes some structures to be shared, so is limited to structures (like those rleated to the known word/tag vocabularies) that won't change during training or thereafter. Beware vocabulary edits/updates to either model afterwards: the partial sharing and out-of-band modification may leave the other model in a broken state. Parameters ---------- other_model : :class:`~gensim.models.doc2vec.Doc2Vec` Other model whose internal data structures will be copied over to the current object. N)r key_to_index index_to_keyexpandos cum_table corpus_countrrr)rC other_models r3 reset_fromzDoc2Vec.reset_fromYs +~:*~:&>2$.'4*~:*~:&>2 r5c  |\} } |jj}|jj}| |}| |}|jr9t ||||||||| | t |j|||j\}}}nl|jr3t||||||||| | t |j|| \}}}n2t||||||||| | t |j|| \}}}|||fS)N)r1 doctags_lockfr-r1r) rrvectorsrvrerlenrlrmrr)rCr! thread_idr" cython_vocabthread_private_mem cur_epochtotal_examples total_wordsoffsets start_doctagsrr(neu1r1rr#examplestally raw_tallys r3_do_train_epochzDoc2Vec._do_train_epochrs0( d- #$Y/ 7 L)=k6<y T4TW-]X\Xg*i*i*i &HeYY^ L)Bk6<y T4TW-]*L*L*L &HeYY *<k6<y T4TW-]*L*L*L &HeY  ))r5c |\}}d}|D]}fd|jD}jj} jj} jr%|t |j|||j| | z }Yjr |t|j||||| | z }|t|j||||| | z }| |fS)a?Train model using `job` data. Parameters ---------- job : iterable of list of :class:`~gensim.models.doc2vec.TaggedDocument` The corpus chunk to be used for training this batch. alpha : float Learning rate to be used for training this batch. inits : (np.ndarray, np.ndarray) Each worker threads private work memory. Returns ------- (int, int) 2-tuple (effective word count after ignoring unknown words and sentence length trimming, total word count). rcVg|]%}|jv j|&SrH)rr get_index).0tagrCs r3 z)Doc2Vec._do_train_job..s6[[[CSWSZN[dg//44[[[r5)r-r1rr) rArrrrvrerr@rlrmrr_raw_word_count) rCjobalphainitsr(rrdocdoctag_indexesr1rs ` r3 _do_train_jobzDoc2Vec._do_train_jobs$ d  C[[[[[[[N!W_N G1Mw ,#)^UDdo#1 1#)^UD$#1 *#)^UD$#1d**3////r5r`?c  ||td||td|1tj|std|z|'t |t std|z|(|||j\} }| | d<|| d<tt|j d ||||||||| | | d | dS) a Update the model's neural weights. To support linear learning-rate decay from (initial) `alpha` to `min_alpha`, and accurate progress-percentage logging, either `total_examples` (count of documents) or `total_words` (count of raw words in documents) **MUST** be provided. If `documents` is the same corpus that was provided to :meth:`~gensim.models.word2vec.Word2Vec.build_vocab` earlier, you can simply use `total_examples=self.corpus_count`. To avoid common mistakes around the model's ability to do multiple training passes itself, an explicit `epochs` argument **MUST** be provided. In the common and recommended case where :meth:`~gensim.models.word2vec.Word2Vec.train` is only called once, you can set `epochs=self.iter`. Parameters ---------- corpus_iterable : iterable of list of :class:`~gensim.models.doc2vec.TaggedDocument`, optional Can be simply a list of elements, but for larger corpora,consider an iterable that streams the documents directly from disk/network. If you don't supply `documents` (or `corpus_file`), the model is left uninitialized -- use if you plan to initialize it in some other way. corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `documents` to get performance boost. Only one of `documents` or `corpus_file` arguments need to be passed (not both of them). Documents' tags are assigned automatically and are equal to line number, as in :class:`~gensim.models.doc2vec.TaggedLineDocument`. total_examples : int, optional Count of documents. total_words : int, optional Count of raw words in documents. epochs : int, optional Number of iterations (epochs) over the corpus. start_alpha : float, optional Initial learning rate. If supplied, replaces the starting `alpha` from the constructor, for this one call to `train`. Use only if making multiple calls to `train`, when you want to manage the alpha learning-rate yourself (not recommended). end_alpha : float, optional Final learning rate. Drops linearly from `start_alpha`. If supplied, this replaces the final `min_alpha` from the constructor, for this one call to :meth:`~gensim.models.doc2vec.Doc2Vec.train`. Use only if making multiple calls to :meth:`~gensim.models.doc2vec.Doc2Vec.train`, when you want to manage the alpha learning-rate yourself (not recommended). word_count : int, optional Count of words already trained. Set this to 0 for the usual case of training on all words in documents. queue_factor : int, optional Multiplier for size of queue (number of workers * queue_factor). report_delay : float, optional Seconds to wait before reporting progress. callbacks : :obj: `list` of :obj: `~gensim.models.callbacks.CallbackAny2Vec`, optional List of callbacks that need to be executed/run at specific stages during training. NzCEither one of corpus_file or corpus_iterable value must be providedzJBoth corpus_file and corpus_iterable must not be provided at the same timezDParameter corpus_file must be a valid path to a file, got %r insteadzEcorpus_iterable must be an iterable of TaggedDocument, got %r insteadrr) rr!rrri start_alpha end_alpharM queue_factor report_delayrgrH) TypeErrorospathisfile isinstancer-_get_offsets_and_start_doctags_for_corpusfileworkersrwrZtrain)rCrr!rrrirrrMrrrgrrrr>s r3rz Doc2Vec.trainsIt  c? cabb b  j jhii i  r27>>++F+F rbeppqq q  wz/8/T/T wcfuuvv v  4%)%W%WXceieq%r%r "G] 'F9 &3F? #"gt" a+){{iT^%LI  a aZ`  a a a a ar5cF tj| fdtD}g}g}t j|d5}d}d}t |D]\} } |t|krn|t| z} |t|krZ||| krN|||| |dz }|t|kr ||| kN| }dddn #1swxYwY||fS)aGet offset and initial document tag in a corpus_file for each worker. Firstly, approximate offsets are calculated based on number of workers and corpus_file size. Secondly, for each approximate offset we find the maximum offset which points to the beginning of line and less than approximate offset. Parameters ---------- corpus_file : str Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. workers : int Number of workers. Returns ------- list of int, list of int Lists with offsets and document tags with length = number of workers. c:g|]}tz|zSrH)rU)ricorpus_file_sizers r3rzIDoc2Vec._get_offsets_and_start_doctags_for_corpusfile..s,WWW1#.'9A=>>WWWr5rb)moderr\N) rrgetsizeranger open enumeraterappend) clsr!rapprox_offsetsrrfincurr_offset_idx prev_fileposline_noline curr_fileposrs ` @r3rz5Doc2Vec._get_offsets_and_start_doctags_for_corpusfile s(7??;77WWWWWgWWW Z $ / / / ,3OL!*3 , , "c.&9&99E+c$ii7 %^)<)<<)P_A`coAo)NN<000!((111#q(O &^)<)<<)P_A`coAo) ,  , , , , , , , , , , , , , , ," %%sB0DDDc4td|DS)a%Get the number of words in a given job. Parameters ---------- job : iterable of list of :class:`~gensim.models.doc2vec.TaggedDocument` Corpus chunk. Returns ------- int Number of raw words in the corpus chunk. c3>K|]}t|jVdSrO)rr@)rsentences r3 z*Doc2Vec._raw_word_count..Ds,;;83x~&&;;;;;;r5)sum)rCrs r3rzDoc2Vec._raw_word_count6s!;;s;;;;;;r5c`dt|jzdt|jzzS)zGet estimated memory for tag lookup, 0 if using pure int tags. Returns ------- int The estimated RAM required to look up a tag in bytes. <)rrrrBs r3estimated_lookup_memoryzDoc2Vec.estimated_lookup_memoryFs)CLL 3TW#555r5c t|trtd|p|j}|p|j}|p|j}t |jjd |}| d|jj}tj dt}dg}t|jt}|js t#j|jt} ||z t'|dz dz } t)|D]a} |jrt+|||||dd|| n:|jrt/|||||| dd|| nt1|||||| dd|| || z}b|dS) aInfer a vector for given post-bulk training document. Notes ----- Subsequent calls to this function may infer different representations for the same document. For a more stable representation, increase the number of epochs to assert a stricter convergence. Parameters ---------- doc_words : list of str A document for which the vector representation will be inferred. alpha : float, optional The initial learning rate. If unspecified, value from model initialization will be reused. min_alpha : float, optional Learning rate will linearly drop to `min_alpha` over all inference epochs. If unspecified, value from model initialization will be reused. epochs : int, optional Number of times to train the new document. Larger values take more time, but may improve quality and run-to-run stability of inferred vectors. If unspecified, the `epochs` value from model initialization will be reused. Returns ------- np.ndarray The inferred paragraph vector for the new document. zVParameter doc_words of infer_vector() must be a list of strings (not a single string). ) seed_stringr\rbrF)r/r0r1r)rstrrr min_alpharirrrrdjoinreshapersrtrurrorer zeros_alignedmaxrrrmrr) rC doc_wordsrrrir1rrr(r alpha_deltars r3 infer_vectorzDoc2Vec.infer_vectorQs8 i % % vtuu u#/ &4;1$'2ESVS[S[\eSfSfggg'//473FGG... T%T222w H)$*:$GGGDy(C A,>,>> v ! !Aw #)^UD %E.hu ()^UD$ %E.hu ")^UD$ %E.hu [ EEa  r5ct|tttfr#|jvr j|Sj|St fd|DS)auGet the vector representation of (possibly multi-term) tag. Parameters ---------- tag : {str, int, list of str, list of int} The tag (or tags) to be looked up in the model. Returns ------- np.ndarray The vector representations of each tag as a matrix (will be 1D if `tag` was a single tag) c g|] }| SrHrH)rrrCs r3rz'Doc2Vec.__getitem__..s,,,1tAw,,,r5)rrrUr rrrr )rCrs` r3 __getitem__zDoc2Vec.__getitem__sh cCg/ 0 0 $'! $ws|#73< ,,,,,,,---r5cg}|jr|d|jz|jr3|jr|dne|dnO|jr|dn2|jr|dn|d|d|jjz|jr|d|jz|j r|d |jr|jr$|jr|d |j z|j d kr|d |j z|j d kr|d|j z|j d kr|d|j z|jjdd|dS)zAbbreviated name reflecting major configuration parameters. Returns ------- str Human readable representation of the models internal state. z"%s"zdbow+wrzdm/czdm/mzdm/szd%dzn%dhszw%dr\zmc%drzs%gzt%dr<,r=)r}rrerlrmrkrrrdnegativerrh min_countsamplerr>r?r)rCsegmentss r3rDzDoc2Vec.__str__s < 3 OOFT\1 2 2 2 7 , ())))''''~ ,''''>,OOF++++OOF+++ 33444 = 3 OOEDM1 2 2 2 7 " OOD ! ! !w 147 1t 1 OOEDK/ 0 0 0 >A  5 OOFT^3 4 4 4 ;? 1 OOEDK/ 0 0 0 222CHHX4F4F4F4FGGr5F*dt_c d}|rH|r)t|jt|jz}|j|||||r-d}d} |rd}d} |j|||||| ddSdS)a&Store the input-hidden weight matrix in the same format used by the original C word2vec-tool. Parameters ---------- fname : str The file path used to save the vectors in. doctag_vec : bool, optional Indicates whether to store document vectors. word_vec : bool, optional Indicates whether to store word vectors. prefix : str, optional Uniquely identifies doctags from word vocab, and avoids collision in case of repeated string in doctag and word vocab. fvocab : str, optional Optional file path used to save the vocabulary. binary : bool, optional If True, the data will be saved in binary word2vec format, otherwise - will be saved in plain text. NTFrK)prefixfvocabbinary write_headerr sort_attr)rrrrsave_word2vec_format) rCfname doctag_vecword_vecrrr total_vecrrs r3rzDoc2Vec.save_word2vec_formats(  K 8LL3tw<<7 G ( ( J J J  'LF $  G ( (fVF)&% ) ' ' ' ' ' ' 'r5zGensim 4.0.0 implemented internal optimizations that make calls to init_sims() unnecessary. init_sims() is now obsoleted and will be completely removed in future versions. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4c<|j|dS)a Precompute L2-normalized vectors. Obsoleted. If you need a single unit-normalized vector for some key, call :meth:`~gensim.models.keyedvectors.KeyedVectors.get_vector` instead: ``doc2vec_model.dv.get_vector(key, norm=True)``. To refresh norms after you performed some atypical out-of-band vector tampering, call `:meth:`~gensim.models.keyedvectors.KeyedVectors.fill_norms()` instead. Parameters ---------- replace : bool If True, forget the original trained vectors and only keep the normalized ones. You lose information if you do this. )replaceN)rr init_sims)rCrs r3rzDoc2Vec.init_simss#. '*****r5c tt|j|ddi|S#t$r!}td|d}~wwxYw)aLoad a previously saved :class:`~gensim.models.doc2vec.Doc2Vec` model. Parameters ---------- fname : str Path to the saved file. *args : object Additional arguments, see `~gensim.models.word2vec.Word2Vec.load`. **kwargs : object Additional arguments, see `~gensim.models.word2vec.Word2Vec.load`. See Also -------- :meth:`~gensim.models.doc2vec.Doc2Vec.save` Save :class:`~gensim.models.doc2vec.Doc2Vec` model. Returns ------- :class:`~gensim.models.doc2vec.Doc2Vec` Loaded model. rethrowTzModel load error. Was model saved using code from an older Gensim version? Try loading older model using gensim-3.8.3, then re-saving, to restore compatibility with current code.N)rwrZloadAttributeErrorrperror)rargsraer>s r3r z Doc2Vec.loadsp0 +5#&&+TJ4J6JJ J    LL3 4 4 4H  s!% AA  Ac|pi}||d<t|j|jzt t jz|d<tt| ||S)aEstimate required memory for a model using current settings. Parameters ---------- vocab_size : int, optional Number of raw words in the vocabulary. report : dict of (str, int), optional A dictionary from string representations of the **specific** model's memory consuming members to their size in bytes. Returns ------- dict of (str, int), optional A dictionary from string representations of the model's memory consuming members to their size in bytes. Includes members from the base classes as well as weights and tag lookup memory estimation specific to the class. doctag_lookup doctag_syn0)report) rrrrrdr ruitemsizerwrZestimate_memory)rC vocab_sizerr>s r3rzDoc2Vec.estimate_memory1so&2"&">">"@"@ #DG t/? ?%++BV V}Wd##33Jv3NNNr5'c |||||\}} | |_||_|jd|||d|} || d| d<||dS) ao Build vocabulary from a sequence of documents (can be a once-only generator stream). Parameters ---------- documents : iterable of list of :class:`~gensim.models.doc2vec.TaggedDocument`, optional Can be simply a list of :class:`~gensim.models.doc2vec.TaggedDocument` elements, but for larger corpora, consider an iterable that streams the documents directly from disk/network. See :class:`~gensim.models.doc2vec.TaggedBrownCorpus` or :class:`~gensim.models.doc2vec.TaggedLineDocument` corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `documents` to get performance boost. Only one of `documents` or `corpus_file` arguments need to be passed (not both of them). Documents' tags are assigned automatically and are equal to a line number, as in :class:`~gensim.models.doc2vec.TaggedLineDocument`. update : bool If true, the new words in `documents` will be added to model's vocab. progress_per : int Indicates how many words to process before showing/updating the progress. keep_raw_vocab : bool If not true, delete the raw vocabulary after the scaling is done and free up RAM. trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during current method call and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. **kwargs Additional key word arguments passed to the internal vocabulary construction. )rr! progress_perr~)updatekeep_raw_vocabr~num_retained_wordsrmemoryrNrH) scan_vocabrcorpus_total_words prepare_vocabrprepare_weights) rCrr!rrrr~rrr report_valuess r3 build_vocabzDoc2Vec.build_vocabIsR%)OO+%%4% % ! \)"-**w&clwwpvww "&"6"6-PdBe"6"f"f h F+++++r5ctd|}tdt|t||pd|_||_||||}||d|d<| |d S) aBuild vocabulary from a dictionary of word frequencies. Build model vocabulary from a passed dictionary that contains a (word -> word count) mapping. Words must be of type unicode strings. Parameters ---------- word_freq : dict of (str, int) Word <-> count mapping. keep_raw_vocab : bool, optional If not true, delete the raw vocabulary after the scaling is done and free up RAM. corpus_count : int, optional Even if no corpus is provided, this argument can set corpus_count explicitly. trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during :meth:`~gensim.models.doc2vec.Doc2Vec.build_vocab` and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. update : bool, optional If true, the new provided words in `word_freq` dict will be added to model's vocab. z$processing provided word frequencieszziEach 'words' should be a list of words (usually unicode strings). First 'words' here is instead plain %s.zQPROGRESS: at example #%i, processed %i words (%i words/s), %i word types, %i tags)rLrMrK)r~z*More unique tags (%i) than documents (%i).zHighest int doctag (%i) larger than count of documents (%i). This means at least %i excess, unused slots (%i bytes) will be allocated for vectors.rMrK)#rrrUrrr@rrpwarningtyperqrrAr rrKrMrJrmax_vocab_sizer prune_vocabrdr rurrLlistrrrritemsr set_vecattrr')rCrrr~ document_nor min_reduceinterval_startinterval_countchecked_string_typesvocab max_rawintdoctags_lookup doctags_listdocument interval_ratedocument_lengthrwordrkeytdts r3 _scan_vocabzDoc2Vec._scan_vocabs   &72 C    %.%?%?% % !K' *hnc22NNBX^,, %)$\)Q. -!,~!=-//TbBb c  gmSZZ\IZIZ"/!,!(.11O} 1 1cC?331!$Z!5!5JJn,1&s+55:55&s+66/I666.43|;L;LYhtu.v.v.vs+$++C0000  ! !d q 3x~.. .K" s5zzD4G'G !%yIIIIa "Q |  | + j NNG\IZIZ\h i i i  $  NN]L*|*Cl*d.>>tAUU     ? I# W W,:3,?,E ,RUV,Vs#))aa 8 899LHL+#))++ > >EAr&(hDG  # G  < ? ? ? G  ; = = = =L((r5順ctd|t|}||||\}}tdt |jt |j||||fS)aOCreate the model's vocabulary: a mapping from unique words in the corpus to their frequency count. Parameters ---------- documents : iterable of :class:`~gensim.models.doc2vec.TaggedDocument`, optional The tagged documents used to create the vocabulary. Their tags can be either str tokens or ints (faster). corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `documents` to get performance boost. Only one of `documents` or `corpus_file` arguments need to be passed (not both of them). progress_per : int Progress will be logged every `progress_per` documents. trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during :meth:`~gensim.models.doc2vec.Doc2Vec.build_vocab` and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. Returns ------- (int, int) Tuple of `(total words in the corpus, number of documents)`. z%collecting all words and their countsNzTcollected %i word types and %i unique tags from a corpus of %i examples and %i words)rprqTaggedLineDocumentrBrr'rr)rCrr!rr~rrs r3rzDoc2Vec.scan_vocabsB  ;<<<  >0==O$($4$4_lT]$^$^! \ b   TW|[   L((r5c|||||}|||||}tjtj|tj|S)aCompute cosine similarity between two post-bulk out of training documents. Parameters ---------- model : :class:`~gensim.models.doc2vec.Doc2Vec` An instance of a trained `Doc2Vec` model. doc_words1 : list of str Input document. doc_words2 : list of str Input document. alpha : float, optional The initial learning rate. min_alpha : float, optional Learning rate will linearly drop to `min_alpha` as training progresses. epochs : int, optional Number of epoch to train the new document. Returns ------- float The cosine similarity between `doc_words1` and `doc_words2`. )rrrri)rrsdotrunitvec)rC doc_words1 doc_words2rrrid1d2s r3similarity_unseen_docszDoc2Vec.similarity_unseen_docs'sf0  5I^d  e e   5I^d  e evh&r**H,s@r3rZrZsqr^`04T T T T T T lXXZ?@@A@X ^Z?@@A@^ 66666 6LP****:'0'0'0TX\15BDPaPaPaPaPaPad)&)&[)&V<<< 6 6 6@!@!@!@!D...('H'H'HR%'%'%'%'NZ ^ +++  +([@OOOOOO2V[,02,2,2,2,h0,0,0,0,dH)H)H)T,),),),)\BBBBBBBBr5rZceZdZdZdS) Doc2VecVocabCObsolete class retained for now as load-compatibility state captureNr?rErFrGrHr5r3rQrQDMMMMr5rQceZdZdZdS)Doc2VecTrainablesrRNrSrHr5r3rVrVHrTr5rVceZdZdZdZdS)TaggedBrownCorpusc||_dS)zReader for the `Brown corpus (part of NLTK data) `_. Parameters ---------- dirname : str Path to folder with Brown corpus. N)dirname)rCrZs r3rxzTaggedBrownCorpus.__init__Ms r5c #Ktj|jD]}tj|j|}tj|sGt j|d5}t|D]^\}}t j |}d| D}d|D}|sFt||d|gV_ dddn #1swxYwYdS)zIterate through the corpus. Yields ------ :class:`~gensim.models.doc2vec.TaggedDocument` Document from `source`. rcg|]=}t|ddk(|d>S)/r`)rsplit)rr@s r3rz.TaggedBrownCorpus.__iter__..js@!_!_!_1AGGTWLLHYHY]^H^!_!''#,,!_!_!_r5cg|]B\}}|dd!|d|ddCS)Nr`r])isalphalower)rtokenrs r3rz.TaggedBrownCorpus.__iter__..lsUrrrJE3`cdfefdf`g`o`o`q`qr s2A2ww?rrrr5_SENT_N) rlistdirrZrrrr rr to_unicoder^r:)rCrritem_nor token_tagsr@s r3__iter__zTaggedBrownCorpus.__iter__XskZ -- S SEGLLu55E7>>%(( E4(( SC%.s^^ S SMGT +D11D"`!_ !_!_!_JrrR\rrrE ! (1P0QRRRRRR S S S S S S S S S S S S S S S S  S Ss8A/C44C8 ;C8 Nr?rErFrxrhrHr5r3rXrXLs7   SSSSSr5rXceZdZdZdZdS)rEc||_dS)aBIterate over a file that contains documents: one line = :class:`~gensim.models.doc2vec.TaggedDocument` object. Words are expected to be already preprocessed and separated by whitespace. Document tags are constructed automatically from the document line number (each document gets a unique integer tag). Parameters ---------- source : string or a file-like object Path to the file on disk, or an already-open file object (must support `seek(0)`). Examples -------- .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.models.doc2vec import TaggedLineDocument >>> >>> for document in TaggedLineDocument(datapath("head500.noblanks.cor")): ... pass N)source)rCrls r3rxzTaggedLineDocument.__init__ss. r5c#K |jdt|jD]<\}}tt j||gV=dS#t$rt j|jd5}t|D]<\}}tt j||gV= dddYdS#1swxYwYYdSwxYw)zIterate through the lines in the source. Yields ------ :class:`~gensim.models.doc2vec.TaggedDocument` Document from `source` specified in the constructor. rrN) rlseekrr:r rer^r r)rCrfrrs r3rhzTaggedLineDocument.__iter__s T K  Q   !*4;!7!7 P P $U%5d%;%;%A%A%C%CgYOOOOOO P P T T TDK.. T#%.s^^TTMGT()9$)?)?)E)E)G)G'SSSSSST T T T T T T T T T T T T T T T T T T T T Ts7A+A11$DA C1# D1C5 5D8C5 9D?DNrirHr5r3rErErs72TTTTTr5rE)NNFTTTNN)NNTTTNN)5rGloggingr collectionsrrcollections.abcrtimeitr dataclassesrnumpyrr rur r r rsgensimr r gensim.utilsr gensim.modelsrrgensim.models.keyedvectorsrr getLoggerr?rpgensim.models.doc2vec_innerrrr ImportError NO_CYTHON gensim.models.doc2vec_corpusfilerrrrr:rJrZSaveLoadrQrVrXrErHr5r3rs44l ////////$$$$$$ !!!!!!@@@@@@@@@@@@@@""""""""######00000000MMMMMMMM  8 $ $lllllllllll /Z ZZZimcg?CZZZZqufjDHZZZZgkcg(,ZZZZZZ!Z.OOOOOZZ 0,??OOO, !!!!!!! !*cBcBcBcBcBhcBcBcBLNNNNN5>NNNNNNNNNNN#S#S#S#S#S#S#S#SL-T-T-T-T-T-T-T-T-T-Ts& A11 A> BB.-B.