c dZddlmZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z ddlmZddlZddlZddlZddlmZmZdd lmZddlZdd lmZmZmZdd lmZm Z dd l!m"Z"m#Z#dd lm$Z$ddl%m&Z&ej'e(Z) ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0n#e1$re"j2wxYw ddl3m4Z4m5Z5m6Z6n#e1$r dZ6dZ4dZ5YnwxYwGdde"j7Z8GddZ9GddZ:GddZ;GddZ<Gdde"j7Z=Gd d!e"j7Z>Gd"d#e d#d$Z?d%Z@d&ZAe(d'kriddlBZBejCd(ejD)e)Ed*d+FejGejHIejGdZJeKejGd,kr4eLeMd-eNzejOd.dd/lPm8Z8ejQd01eBjRZSeSTd2d3d45eSTd6d78eSTd9d:eUd;<eSTd=d>eUd?<eSTd@dAeVdB<eSTdCdDeUddd.gEeSTdFdGeUd;<eSTdHdIeUdJ<eSTdKdLeUd;<eSTdMdNeUd;<eSTdOdPeUd.dd.gEeSTdQdReUddd.gEeSTdSdT8eSWZXeXjYdkrd.ZZndZZe;eXj[Z\e8e\eXj]eXj^eXj_eXj`eXjaeZeXjbeXjcd.eXjdU ZeeXjfr)eXjfZgeejhiegeXjjVneXj[ZgeekegdWzeXjjd.kr eejhiegdXzd4VneejhiegdYzdZVeXjlreeleXjle)Ed[eJdSdS)\aO Introduction ============ This module implements the word2vec family of algorithms, using highly optimized C routines, data streaming and Pythonic interfaces. The word2vec algorithms include skip-gram and CBOW models, using either hierarchical softmax or negative sampling: `Tomas Mikolov et al: Efficient Estimation of Word Representations in Vector Space `_, `Tomas Mikolov et al: Distributed Representations of Words and Phrases and their Compositionality `_. Other embeddings ================ There are more ways to train word vectors in Gensim than just Word2Vec. See also :class:`~gensim.models.doc2vec.Doc2Vec`, :class:`~gensim.models.fasttext.FastText`. The training algorithms were originally ported from the C package https://code.google.com/p/word2vec/ and extended with additional functionality and `optimizations `_ over the years. For a tutorial on Gensim word2vec, with an interactive web app trained on GoogleNews, visit https://rare-technologies.com/word2vec-tutorial/. Usage examples ============== Initialize a model with e.g.: .. sourcecode:: pycon >>> from gensim.test.utils import common_texts >>> from gensim.models import Word2Vec >>> >>> model = Word2Vec(sentences=common_texts, vector_size=100, window=5, min_count=1, workers=4) >>> model.save("word2vec.model") **The training is streamed, so ``sentences`` can be an iterable**, reading input data from the disk or network on-the-fly, without loading your entire corpus into RAM. Note the ``sentences`` iterable must be *restartable* (not just a generator), to allow the algorithm to stream over your dataset multiple times. For some examples of streamed iterables, see :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus` or :class:`~gensim.models.word2vec.LineSentence`. If you save the model you can continue training it later: .. sourcecode:: pycon >>> model = Word2Vec.load("word2vec.model") >>> model.train([["hello", "world"]], total_examples=1, epochs=1) (0, 2) The trained word vectors are stored in a :class:`~gensim.models.keyedvectors.KeyedVectors` instance, as `model.wv`: .. sourcecode:: pycon >>> vector = model.wv['computer'] # get numpy vector of a word >>> sims = model.wv.most_similar('computer', topn=10) # get other similar words The reason for separating the trained vectors into `KeyedVectors` is that if you don't need the full model state any more (don't need to continue training), its state can be discarded, keeping just the vectors and their keys proper. This results in a much smaller and faster object that can be mmapped for lightning fast loading and sharing the vectors in RAM between processes: .. sourcecode:: pycon >>> from gensim.models import KeyedVectors >>> >>> # Store just the words + their trained embeddings. >>> word_vectors = model.wv >>> word_vectors.save("word2vec.wordvectors") >>> >>> # Load back with memory-mapping = read-only, shared across processes. >>> wv = KeyedVectors.load("word2vec.wordvectors", mmap='r') >>> >>> vector = wv['computer'] # Get numpy vector of a word Gensim can also load word vectors in the "word2vec C format", as a :class:`~gensim.models.keyedvectors.KeyedVectors` instance: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> # Load a word2vec model stored in the C *text* format. >>> wv_from_text = KeyedVectors.load_word2vec_format(datapath('word2vec_pre_kv_c'), binary=False) >>> # Load a word2vec model stored in the C *binary* format. >>> wv_from_bin = KeyedVectors.load_word2vec_format(datapath("euclidean_vectors.bin"), binary=True) It is impossible to continue training the vectors loaded from the C format because the hidden weights, vocabulary frequencies and the binary tree are missing. To continue training, you'll need the full :class:`~gensim.models.word2vec.Word2Vec` object state, as stored by :meth:`~gensim.models.word2vec.Word2Vec.save`, not just the :class:`~gensim.models.keyedvectors.KeyedVectors`. You can perform various NLP tasks with a trained model. Some of the operations are already built-in - see :mod:`gensim.models.keyedvectors`. If you're finished training a model (i.e. no more updates, only querying), you can switch to the :class:`~gensim.models.keyedvectors.KeyedVectors` instance: .. sourcecode:: pycon >>> word_vectors = model.wv >>> del model to trim unneeded model state = use much less RAM and allow fast loading and memory sharing (mmap). Embeddings with multiword ngrams ================================ There is a :mod:`gensim.models.phrases` module which lets you automatically detect phrases longer than one word, using collocation statistics. Using phrases, you can learn a word2vec model where "words" are actually multiword expressions, such as `new_york_times` or `financial_crisis`: .. sourcecode:: pycon >>> from gensim.models import Phrases >>> >>> # Train a bigram detector. >>> bigram_transformer = Phrases(common_texts) >>> >>> # Apply the trained MWE detector to a corpus, using the result to train a Word2vec model. >>> model = Word2Vec(bigram_transformer[common_texts], min_count=1) Pretrained models ================= Gensim comes with several already pre-trained models, in the `Gensim-data repository `_: .. sourcecode:: pycon >>> import gensim.downloader >>> # Show all available models in gensim-data >>> print(list(gensim.downloader.info()['models'].keys())) ['fasttext-wiki-news-subwords-300', 'conceptnet-numberbatch-17-06-300', 'word2vec-ruscorpora-300', 'word2vec-google-news-300', 'glove-wiki-gigaword-50', 'glove-wiki-gigaword-100', 'glove-wiki-gigaword-200', 'glove-wiki-gigaword-300', 'glove-twitter-25', 'glove-twitter-50', 'glove-twitter-100', 'glove-twitter-200', '__testing_word2vec-matrix-synopsis'] >>> >>> # Download the "glove-twitter-25" embeddings >>> glove_vectors = gensim.downloader.load('glove-twitter-25') >>> >>> # Use the downloaded vectors as usual: >>> glove_vectors.most_similar('twitter') [('facebook', 0.948005199432373), ('tweet', 0.9403423070907593), ('fb', 0.9342358708381653), ('instagram', 0.9104824066162109), ('chat', 0.8964964747428894), ('hashtag', 0.8885937333106995), ('tweets', 0.8878158330917358), ('tl', 0.8778461217880249), ('link', 0.8778210878372192), ('internet', 0.8753897547721863)] )divisionN) default_timer) defaultdict namedtuple)Iterable) GeneratorType)QueueEmpty)float32)keep_vocab_itemcall_on_class_only deprecated) KeyedVectorspseudorandom_weak_vector)utilsmatutils)Vocab)get_supported_extensions)train_batch_sgtrain_batch_cbowscore_sentence_sgscore_sentence_cbowMAX_WORDS_IN_BATCH FAST_VERSION)train_epoch_sgtrain_epoch_cbowCORPUSFILE_VERSIONc  tdNz3Training with corpus_file argument is not supported RuntimeError model corpus_fileoffset _cython_vocab _cur_epoch_expected_examples_expected_words_work_neu1 compute_losss 6lib/python3.11/site-packages/gensim/models/word2vec.pyrrPQQQc  tdr r!r#s r.rrr/r0ceZdZdddddddddddd d dd dedd dded d ddd fdZ dAdZ dBdZdZdCdZ dDdZ dEdZ dZ dZ dFdZdGdZeddZdZdZeddGd Z dEd!Zd"Zd#Z dHd&Z dId'Zd(ZdId)Z dJd*Z dKd+Z dLd,Zd-Zd.Z d/Z!dMd0Z"dId1Z#d2Z$d3Z%d4Z&e'd5dd$dfd6Z(dNd8Z)d9Z*d:Z+fd;Z,fd<Z-e.d d=fd> Z/fd?Z0d@Z1xZ2S)OWord2VecNdg?MbP?g-C6?r?FTc |}t||_t| |_||_d|_d|_||_t| |_t||_ t| |_ t||_ t||_ tj| |_t| |_t||_||_t||_t||_d|_t||_d|_d|_||_||_||_||_||_||_d|_ d|_!tE|dstG||_$tj%dtL|j$_'||_(| |_)tE|ds||_*||_+tX|_-||p|.|||dz|/||||0|||j|j|j|j |j |j| n8|tb2d |rtb2d |3d ti| dS)uc Train, use and evaluate neural networks described in https://code.google.com/p/word2vec/. Once you're finished training a model (=no more updates, only querying) store and use only the :class:`~gensim.models.keyedvectors.KeyedVectors` instance in ``self.wv`` to reduce memory. The full model can be stored/loaded via its :meth:`~gensim.models.word2vec.Word2Vec.save` and :meth:`~gensim.models.word2vec.Word2Vec.load` methods. The trained word vectors can also be stored/loaded from a format compatible with the original word2vec implementation via `self.wv.save_word2vec_format` and :meth:`gensim.models.keyedvectors.KeyedVectors.load_word2vec_format`. Parameters ---------- sentences : iterable of iterables, optional The `sentences` iterable can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network. See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus` or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples. See also the `tutorial on data streaming in Python `_. If you don't supply `sentences`, the model is left uninitialized -- use if you plan to initialize it in some other way. corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `sentences` to get performance boost. Only one of `sentences` or `corpus_file` arguments need to be passed (or none of them, in that case, the model is left uninitialized). vector_size : int, optional Dimensionality of the word vectors. window : int, optional Maximum distance between the current and predicted word within a sentence. min_count : int, optional Ignores all words with total frequency lower than this. workers : int, optional Use these many worker threads to train the model (=faster training with multicore machines). sg : {0, 1}, optional Training algorithm: 1 for skip-gram; otherwise CBOW. hs : {0, 1}, optional If 1, hierarchical softmax will be used for model training. If 0, and `negative` is non-zero, negative sampling will be used. negative : int, optional If > 0, negative sampling will be used, the int for negative specifies how many "noise words" should be drawn (usually between 5-20). If set to 0, no negative sampling is used. ns_exponent : float, optional The exponent used to shape the negative sampling distribution. A value of 1.0 samples exactly in proportion to the frequencies, 0.0 samples all words equally, while a negative value samples low-frequency words more than high-frequency words. The popular default value of 0.75 was chosen by the original Word2Vec paper. More recently, in https://arxiv.org/abs/1804.04212, Caselles-Dupré, Lesaint, & Royo-Letelier suggest that other values may perform better for recommendation applications. cbow_mean : {0, 1}, optional If 0, use the sum of the context word vectors. If 1, use the mean, only applies when cbow is used. alpha : float, optional The initial learning rate. min_alpha : float, optional Learning rate will linearly drop to `min_alpha` as training progresses. seed : int, optional Seed for the random number generator. Initial vectors for each word are seeded with a hash of the concatenation of word + `str(seed)`. Note that for a fully deterministically-reproducible run, you must also limit the model to a single worker thread (`workers=1`), to eliminate ordering jitter from OS thread scheduling. (In Python 3, reproducibility between interpreter launches also requires use of the `PYTHONHASHSEED` environment variable to control hash randomization). max_vocab_size : int, optional Limits the RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM. Set to `None` for no limit. max_final_vocab : int, optional Limits the vocab to a target vocab size by automatically picking a matching min_count. If the specified min_count is more than the calculated min_count, the specified min_count will be used. Set to `None` if not required. sample : float, optional The threshold for configuring which higher-frequency words are randomly downsampled, useful range is (0, 1e-5). hashfxn : function, optional Hash function to use to randomly initialize weights, for increased training reproducibility. epochs : int, optional Number of iterations (epochs) over the corpus. (Formerly: `iter`) trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. sorted_vocab : {0, 1}, optional If 1, sort the vocabulary by descending frequency before assigning word indexes. See :meth:`~gensim.models.keyedvectors.KeyedVectors.sort_by_descending_frequency()`. batch_words : int, optional Target size (in words) for batches of examples passed to worker threads (and thus cython routines).(Larger batches will be passed if individual texts are longer than 10000 words, but the standard cython code truncates to that maximum.) compute_loss: bool, optional If True, computes and stores loss value which can be retrieved using :meth:`~gensim.models.word2vec.Word2Vec.get_latest_training_loss`. callbacks : iterable of :class:`~gensim.models.callbacks.CallbackAny2Vec`, optional Sequence of callbacks to be executed at specific stages during training. shrink_windows : bool, optional New in 4.1. Experimental. If True, the effective window size is uniformly sampled from [1, `window`] for each target word during training, to match the original word2vec algorithm's approximate weighting of context words by distance. Otherwise, the effective window size is always fixed to `window` words to either side. Examples -------- Initialize and train a :class:`~gensim.models.word2vec.Word2Vec` model .. sourcecode:: pycon >>> from gensim.models import Word2Vec >>> sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]] >>> model = Word2Vec(sentences, min_count=1) Attributes ---------- wv : :class:`~gensim.models.keyedvectors.KeyedVectors` This object essentially contains the mapping between words and embeddings. After training, it can be used directly to query those embeddings in various ways. See the module level docstring for examples. rNwvr7dtype layer1_sizecorpus_iterabler%passes)rAr% trim_rule) rAr%total_examples total_wordsepochs start_alpha end_alphar- callbackszThe rule, if given, is only used to prune vocabulary during build_vocab() and is not stored as part of the model. Model initialized without sentences. trim_rule provided, if any, will be ignored.zCallbacks are no longer retained by the model, so must be provided whenever training is triggered, as in initialization with a corpus or calling `train()`. The callbacks provided in this initialization without triggering train will be ignored.created)params)5int vector_sizeworkersrF train_counttotal_train_time batch_wordssgfloatalpha min_alphawindowboolshrink_windowsnprandom RandomStatehsnegative ns_exponent cbow_meanr-running_training_lossmin_alpha_yet_reached corpus_countcorpus_total_wordsmax_final_vocabmax_vocab_size min_countsample sorted_vocab null_word cum_table raw_vocabhasattrrr<onesREAL vectors_lockfhashfxnseedr?commentr load_check_corpus_sanity build_vocabtrainloggerwarningadd_lifecycle_eventstr)self sentencesr%rMrTrVrfrergrqrNrUrRr\r]r^r_rprFrirCrhrQr-rIrrrdrXrAs r.__init__zWord2Vec.__init__sL${++7||   !&b''5\\ y))&kk ">22i++D11 b''H  &Y ..%&"%*5\\""#.," ("tT"" 0";//DG!# 6 6 6  t]++ +*D  &  #+ #  % %oS^hnqrhr % t t t   _+aj  k k k JJ /[Y]Yj 3DKUYU_.t7HT]  _ _ _ _  DCDDD #"###   3t99 =====r0'c `|||d|||||\}} | |_||_|jd |||d|} || d| d<|||d |t| d S) aBuild vocabulary from a sequence of sentences (can be a once-only generator stream). Parameters ---------- corpus_iterable : iterable of list of str Can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network. See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus` or :class:`~gensim.models.word2vec.LineSentence` module for such examples. corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `sentences` to get performance boost. Only one of `sentences` or `corpus_file` arguments need to be passed (not both of them). update : bool If true, the new words in `sentences` will be added to model's vocab. progress_per : int, optional Indicates how many words to process before showing/updating the progress. keep_raw_vocab : bool, optional If False, the raw vocabulary will be deleted after the scaling is done to free up RAM. trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during current method call and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. **kwargs : object Keyword arguments propagated to `self.prepare_vocab`. r7r@)rAr% progress_perrC)updatekeep_raw_vocabrCnum_retained_words vocab_sizememoryrru)rrCNr:) rt scan_vocabrbrc prepare_vocabestimate_memoryprepare_weightsryrz) r{rAr%rrrrCkwargsrErb report_valuess r.ruzWord2Vec.build_vocabsR !!/{cd!eee$(OO+S_kt%4%v%v! \("-**w&clwwpvww "&"6"6-PdBe"6"f"f h F+++   vY XXXXXr0ctd|}tdt|t||pd|_||_||||}||d|d<| |d S) aBuild vocabulary from a dictionary of word frequencies. Parameters ---------- word_freq : dict of (str, int) A mapping from a word in the vocabulary to its frequency count. keep_raw_vocab : bool, optional If False, delete the raw vocabulary after the scaling is done to free up RAM. corpus_count : int, optional Even if no corpus is provided, this argument can set corpus_count explicitly. trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during current method call and is not stored as part of the model. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. update : bool, optional If true, the new provided words in `word_freq` dict will be added to model's vocab. z$Processing provided word frequenciesz:collected %i unique word types, with total frequency of %ir)rrCrrrrrN) rwinfolensumvaluesrbrkrrr)r{ word_freqrrbrCrrkrs r.build_vocab_from_freqzWord2Vec.build_vocab_from_freqs>  :;;;  H  NNC 0 0 2 233   )-A"**.T]fl*mm "&"6"6-PdBe"6"f"f h F+++++r0c 6d}d}d}tt}d}t|D]\}} |sBt| tr(t dt| |dz }||zdkr*t d||t|| D]} || xxdz cc<|t| z }|j r4t||j krtj ||||dz }|dz} ||_ || fS)Nrrr7zoEach 'sentences' item should be a list of words (usually unicode strings). First item here is instead plain %s.zDPROGRESS: at sentence #%i, processed %i words, keeping %i word typesrC)rrL enumerate isinstancerzrwrxtyperrrer prune_vocabrk) r{r|rrC sentence_norE min_reducevocabchecked_string_typessentencewordrbs r. _scan_vocabzWord2Vec._scan_vocab%sY   C   %.y%9%9  !K' *h,,NN?X %)$\)Q.  Zc%jj! ! !d q 3x== (K" s5zzD4G'G !%yIIIIa "Q L((r0ctd|rt|}||||\}}tdt |j||||fS)Nz%collecting all words and their countszFcollected %i word types from a corpus of %i raw words and %i sentences)rwr LineSentencerrrk)r{rAr%rrNrCrErbs r.rzWord2Vec.scan_vocabEs~ ;<<<  8*;77O$($4$4_lT]$^$^! \ T   l   L((r0c |pj}|pj}dx}}|_jt jfdd} d} jt| krj| jdz} t| |_ ddjd |d | d j |st d dg} } |s&gj _ |_|_ij _jD]\} }t!| |j|rb| | | |z } |sEtj j j j| <j j | |dz }||z }|s6j j D])} j | dj| *t| |z}t| dzt|dz } ddjdt| d|dd|d|d | |z}| dzt|dz } ddjd| d|dd|d|d nBt ddx}}g}g}jD]\} }t!| |j|rj | r|| ||z }|s U|| ||z }|sEtj j j j| <j j | |dz }||z }|sj dgt+dgj j D]S} j | dj | dj| dzTt|t|z|z}t|dzt|dz }t|dzt|dz } ddt|d|dd|dt|d|dd|d ||z} ||z} |s| }n6|d kr|| z}n*t1|d!t3jd"zzd#z }d$\}}| D]}}j|}t3j||z dz||z z}|d kr|dz }|||zz }nd }||z }|s1j |d%t3j|d&z~|sH|sFt d'tjt9t0_t d(|| dd)|d|d*zt| dz d+d,| d || |t1|t| d-}jrjr|sj j!r"j#r$|S).a$Apply vocabulary settings for `min_count` (discarding less-frequent words) and `sample` (controlling the downsampling of more-frequent words). Calling with `dry_run=True` will only simulate the provided settings and report the size of the retained vocabulary, effective corpus length, and estimated memory requirements. Results are both printed via logging and returned as a dict. Delete the raw vocabulary after the scaling is done to free up RAM, unless `keep_raw_vocab` is set. rNcj|SN)rk)rr{s r.z(Word2Vec.prepare_vocab..ms$.Y]J^r0T)keyreverser7rzmax_final_vocab=z and min_count=z resulted in calc_min_count=z, effective_min_count=msgzCreating a fresh vocabularyrcountr4zeffective_min_count=z retains z unique words (z.2fz% of original z, drops )z leaves z word corpus (z"Updating model with new vocabulary)attrstypeszadded z new unique words (z) and increased the count of z pre-existing words (?r8r5rr sample_intlz.deleting the raw counts dictionary of %i itemsz*sample=%g downsamples %i most-common wordszdownsampling leaves estimated Y@.1fz %% of prior ) drop_unique retain_totaldownsample_uniquedownsample_totalr)%rfrgeffective_min_countrdsortedrkkeysrmaxryrwrr< index_to_key key_to_indexitemsr append set_vecattr has_index_forallocate_vecattrsr get_vecattrgetrLrYsqrtuint32rri add_null_wordrhsort_by_descending_frequencyr\create_binary_treer]make_cum_table)r{rrrCrfrgdry_run drop_totalrrhcalc_min_countr retain_wordsrvoriginal_unique_totalretain_unique_pctoriginal_total retain_pct new_totalpre_exist_total new_wordspre_exist_wordspre_exist_unique_pctnew_unique_pctthreshold_countrrwword_probabilityrs` r.rzWord2Vec.prepare_vocabSs / &4;#$$ [$-    !$."5"5"7"7=^=^=^=^hlmmmLN#c,&7&77 X!% T=Q0R!SVW!W'*>9'E'ED $  $ $jt';jjIjj)7jjOSOgjj %   R 7 KK5 6 6 6)*B,L *')$!*$ ')$>//11 $ $a"4D,DPYZZZ$ ''--- A%L":589M5N5N,T2,33D9991$K!OJJ M G0MMDG''gt~d7KLLLL$' $5$5 $C ! #L 1 1C 7#>SUV:W:W W   $ $q4+CqqcR^N_N_qq/pqqCXqqbmqqq %   *J6N%+c.!.D.DDJ  $ $\4+C\\\\\"[\\6D\\NX\\\ %     KK< = = =*+ +II O>//11 $ $a"4D,DPYZZZ$w,,T22 >'..t444'1,&! !((...!Q &>9>$IK$W !#&#7#7##=DY[\@]@]#]  ^^c1C8Mq4Q4QQN  $ $lS^^llbll,llKNK_K_ll+?kllShlll %   %6L$6L A*OO c\ A$|3OO"&A N";a"?@@O.2++ ` `Aq!A "O(; <  !      3V 3 G 0 0 2 2 2 7 &  # # % % % = "    ! ! !r0c|pt|j}|pi}||jrdndz|d<||jzt jt jz|d<|jr,||jzt jt jz|d<|j r,||jzt jt jz|d<t| |d<t d||j|d|S) a7Estimate required memory for a model using current settings and provided vocabulary size. Parameters ---------- vocab_size : int, optional Number of unique tokens in the vocabulary report : dict of (str, int), optional A dictionary from string representations of the model's memory consuming members to their size in bytes. Returns ------- dict of (str, int) A dictionary from string representations of the model's memory consuming members to their size in bytes. iirvectorssyn1syn1negtotalzBestimated required memory for %i words and %i dimensions: %i bytes)rr<r\rMrYr>rnitemsizer?r]rrrwr)r{rreports r.rzWord2Vec.estimate_memory s  /3tw<< 2$tw(?C@w&)99BHTNN`_ using stored vocabulary word counts. Frequent words will have shorter binary codes. Called internally from :meth:`~gensim.models.word2vec.Word2VecVocab.build_vocab`. N)_assign_binary_codesr<r{s r.rzWord2Vec.create_binary_tree1s TW%%%%%r0c:t|jj}tj|tj|_d}t|D]7}|j|d}||t|j zz }8d}t|D]T}|j|d}||t|j zz }t||z |z|j|<Ut|jdkr|jd|ksJdSdS)aCreate a cumulative-distribution table using stored vocabulary word counts for drawing random words in the negative-sampling training routines. To draw a word index, choose a random integer up to the maximum value in the table (cum_table[-1]), then finding that integer's sorted insertion point (as if by `bisect_left` or `ndarray.searchsorted()`). That insertion point is the drawn index, coming up in proportion equal to the increment at that slot. r=rrrN) rr<rrYzerosrrjrangerrSr^round)r{domainrtrain_words_pow word_indexr cumulatives r.rzWord2Vec.make_cum_table9s,-.. *BI>>> ++ > >JG'' G<"%/ / / / 0 0 / /r0c^|s|dS|dS)zBBuild tables and model weights based on final vocabulary settings.N) init_weightsupdate_weights)r{rs r.rzWord2Vec.prepare_weightsQs< "            ! ! ! ! !r0zBUse gensim.models.keyedvectors.pseudorandom_weak_vector() directlyc0t|||jS)N) seed_stringrp)rrp)r{rrMs r. seeded_vectorzWord2Vec.seeded_vectorYs' VZVbccccr0c~td|j|j|jr9t jt|j|j ft|_ |j r;t jt|j|j ft|_ dSdS)z_Reset all projection weights to an initial (untrained) state, but keep the existing vocabulary.zresetting layer weightsrqr=N)rwrr<resize_vectorsrqr\rYrrr?rnrr]rrs r.rzWord2Vec.init_weights]s -... DI... 7 O#dg,,0@!ANNNDI = R8S\\43C$DDQQQDLLL R Rr0cXtdt|jjst dt|jj}|j|jt|jj|z }|jr@tj |j tj ||j ftg|_ |jrDtj ||j ft}tj |j|g|_dSdS)zTCopy all the existing weights, and reset the weights for the newly added vocabulary.zupdating layer weightszYou cannot do an online vocabulary-update of a model which has no prior vocabulary. First build the vocabulary of your model with a corpus before doing an online update.rr=N)rwrrr<rr"rrqr\rYvstackrrr?rnr]r)r{preresize_count gained_vocabpads r.rzWord2Vec.update_weightsgs  ,---47?## h dgo.. DI...47?++o= 7 g 49bh dFV7W_c.d.d.d"effDI = :(L$*:;4HHHC9dlC%899DLLL : :r0zGensim 4.0.0 implemented internal optimizations that make calls to init_sims() unnecessary. init_sims() is now obsoleted and will be completely removed in future versions. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4c<|j|dS)a Precompute L2-normalized vectors. Obsoleted. If you need a single unit-normalized vector for some key, call :meth:`~gensim.models.keyedvectors.KeyedVectors.get_vector` instead: ``word2vec_model.wv.get_vector(key, norm=True)``. To refresh norms after you performed some atypical out-of-band vector tampering, call `:meth:`~gensim.models.keyedvectors.KeyedVectors.fill_norms()` instead. Parameters ---------- replace : bool If True, forget the original trained vectors and only keep the normalized ones. You lose information if you do this. )replaceN)r< init_sims)r{rs r.rzWord2Vec.init_simszs#. '*****r0c |\} } |jr"t|||||||| | |j \} } }n!t|||||||| | |j \} } }| | |fSr)rRrr-r)r{r% thread_idr& cython_vocabthread_private_mem cur_epochrDrErworkneu1examplestally raw_tallys r._do_train_epochzWord2Vec._do_train_epochs( d 7 )7k6< T49J** &HeYY *:k6< T49J** &HeY  ))r0c |\}}d}|jr|t|||||jz }n|t||||||jz }|||fS)aTrain the model on a single batch of sentences. Parameters ---------- sentences : iterable of list of str Corpus chunk to be used in this training batch. alpha : float The learning rate used in this batch. inits : (np.ndarray, np.ndarray) Each worker threads private work memory. Returns ------- (int, int) 2-tuple (effective word count after ignoring unknown words and sentence length trimming, total word count). r)rRrr-r_raw_word_count)r{r|rTinitsr r rs r. _do_train_jobzWord2Vec._do_train_jobsu$ d 7 ] ^D)UD$BSTT TEE %dIudD$J[\\ \Ed**95555r0cd|j_dS)z;Clear any cached values that training may have invalidated.N)r<normsrs r._clear_post_trainzWord2Vec._clear_post_trains r0rrc  |p|j|_|p|j|_||_|||||||||dd|jdt|jd|j d|j d|j d |j d |j d |jd |j | |_d|_| D]}||d}d}t'dz }d}t)|jD]x}| D]}||||j|f|||| | | d| \}}}n|j|f|||| d| \}}}||z }||z }||z }| D]}||yt'|z }||||||xjdz c_|| D]}||||fS)aUpdate the model's neural weights from a sequence of sentences. Notes ----- To support linear learning-rate decay from (initial) `alpha` to `min_alpha`, and accurate progress-percentage logging, either `total_examples` (count of sentences) or `total_words` (count of raw words in sentences) **MUST** be provided. If `sentences` is the same corpus that was provided to :meth:`~gensim.models.word2vec.Word2Vec.build_vocab` earlier, you can simply use `total_examples=self.corpus_count`. Warnings -------- To avoid common mistakes around the model's ability to do multiple training passes itself, an explicit `epochs` argument **MUST** be provided. In the common and recommended case where :meth:`~gensim.models.word2vec.Word2Vec.train` is only called once, you can set `epochs=self.epochs`. Parameters ---------- corpus_iterable : iterable of list of str The ``corpus_iterable`` can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network, to limit RAM usage. See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus` or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples. See also the `tutorial on data streaming in Python `_. corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `sentences` to get performance boost. Only one of `sentences` or `corpus_file` arguments need to be passed (not both of them). total_examples : int Count of sentences. total_words : int Count of raw words in sentences. epochs : int Number of iterations (epochs) over the corpus. start_alpha : float, optional Initial learning rate. If supplied, replaces the starting `alpha` from the constructor, for this one call to`train()`. Use only if making multiple calls to `train()`, when you want to manage the alpha learning-rate yourself (not recommended). end_alpha : float, optional Final learning rate. Drops linearly from `start_alpha`. If supplied, this replaces the final `min_alpha` from the constructor, for this one call to `train()`. Use only if making multiple calls to `train()`, when you want to manage the alpha learning-rate yourself (not recommended). word_count : int, optional Count of words already trained. Set this to 0 for the usual case of training on all words in sentences. queue_factor : int, optional Multiplier for size of queue (number of workers * queue_factor). report_delay : float, optional Seconds to wait before reporting progress. compute_loss: bool, optional If True, computes and stores loss value which can be retrieved using :meth:`~gensim.models.word2vec.Word2Vec.get_latest_training_loss`. callbacks : iterable of :class:`~gensim.models.callbacks.CallbackAny2Vec`, optional Sequence of callbacks to be executed at specific stages during training. Examples -------- .. sourcecode:: pycon >>> from gensim.models import Word2Vec >>> sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]] >>> >>> model = Word2Vec(min_count=1) >>> model.build_vocab(sentences) # prepare the model vocabulary >>> model.train(sentences, total_examples=model.corpus_count, epochs=model.epochs) # train word vectors (1, 30) )rFrDrEr@rvztraining model with z workers on z vocabulary and z features, using sg=z hs=z sample=z negative=z window=z shrink_windows=rrrh㈵>N)r rDrE queue_factor report_delayrI)r rDrErIr7)rTrUrF_check_training_sanityrtryrNrr<r?rRr\rgr]rVrXr-r`on_train_beginrron_epoch_begin _train_epoch_train_epoch_corpusfile on_epoch_end_log_train_endrOr on_train_end)r{rAr%rDrErFrGrH word_countrrr-rIrcallbacktrained_word_countraw_word_countstart job_tallyr trained_word_count_epochraw_word_count_epochjob_tally_epoch total_elapseds r.rvzWord2Vec.trains&Z!.DJ "4dn  ##6.^i#jjj !!/{ci!jjj   ft|ffTWff#ff9=ffFJgffW[Wbff Mff37;ffPTPcff !   )%("! * *H  # #D ) ) ) )') t{++ , ,I% . .''---- 3RcRVRc#S3/8 +,Ua'S3S3,2S3S3O(*> SoRVRnS3+4^al'S3S3+1S3S3O(*> ": :  2 2N  (I% , ,%%d++++ ,&%/  N,> yYYY A    ! ( (H  ! !$ ' ' ' '!>11r0c  |} |j||||| |f||d| \} } } || | | f|ddS)aTrain the model on a `corpus_file` in LineSentence format. This function will be called in parallel by multiple workers (threads or processes) to make optimal use of multicore machines. Parameters ---------- corpus_file : str Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. thread_id : int Thread index starting from 0 to `number of workers - 1`. offset : int Offset (in bytes) in the `corpus_file` for particular worker. cython_vocab : :class:`~gensim.models.word2vec_inner.CythonVocab` Copy of the vocabulary in order to access it without GIL. progress_queue : Queue of (int, int, int) A queue of progress reports. Each report is represented as a tuple of these 3 elements: * Size of data chunk processed, for example number of sentences in the corpus chunk. * Effective word count used in training (after ignoring unknown words and trimming the sentence length). * Total word count used in training. **kwargs : object Additional key word parameters for the specific model inheriting from this class. )rDrEN)_get_thread_working_memrput)r{r%rr&rprogress_queuer rDrErr r rrs r._worker_loop_corpusfilez Word2Vec._worker_loop_corpusfileMs8"99;;%9T%9 FL:Li&N){&N&NFL&N&N"% HeY78884     r0cV|}d} |}||dnJ|\}}||||\}} |t ||| f|dz }vt d|dS)aTrain the model, lifting batches of data from the queue. This function will be called in parallel by multiple workers (threads or processes) to make optimal use of multicore machines. Parameters ---------- job_queue : Queue of (list of objects, float) A queue of jobs still to be processed. The worker will take up jobs from this queue. Each job is represented by a tuple where the first element is the corpus chunk to be processed and the second is the floating-point learning rate. progress_queue : Queue of (int, int, int) A queue of progress reports. Each report is represented as a tuple of these 3 elements: * Size of data chunk processed, for example number of sentences in the corpus chunk. * Effective word count used in training (after ignoring unknown words and trimming the sentence length). * Total word count used in training. rTNr7z!worker exiting, processed %i jobs)r/rr0rrrwdebug) r{ job_queuer1r jobs_processedjob data_iterablerTrrs r. _worker_loopzWord2Vec._worker_looprs&"99;; --//C ""4(((#& M5#11-HZ[[ E9   M 2 2E9E F F F a N   8.IIIIIr0cgd}}d\}} |d|} d} t|D]\} } || g}||z|jkr|| ||z }D| dz } ||| f|r| t |z } d| z|z }n |||z }d|z|z }|||} | g|}}|r| dz } ||| f| dkr%|jdkrt dt|j D]}|dt d| dS) awFill the jobs queue using the data found in the input stream. Each job is represented by a tuple where the first element is the corpus chunk to be processed and the second is a dictionary of parameters. Parameters ---------- data_iterator : iterable of list of objects The input dataset. This will be split in chunks and these chunks will be pushed to the queue. job_queue : Queue of (list of object, float) A queue of jobs still to be processed. The worker will take up jobs from this queue. Each job is represented by a tuple where the first element is the corpus chunk to be processed and the second is the floating-point learning rate. cur_epoch : int, optional The current training epoch, needed to compute the training parameters for each job. For example in many implementations the learning rate would be dropping with the number of epochs. total_examples : int, optional Count of objects in the `data_iterator`. In the usual case this would correspond to the number of sentences in a corpus. Used to log progress. total_words : int, optional Count of total objects in `data_iterator`. In the usual case this would correspond to the number of raw words in a corpus. Used to log progress. rrrr7rztrain() called with an empty iterator (if not intended, be sure to provide a corpus that offers restartable iteration = an iterable).Nzjob loop exiting, total %i jobs) _get_next_alpharrrQrr0rrOrwrxrrNr4)r{ data_iteratorr5r rDrE job_batch batch_size pushed_wordspushed_examples next_alphajob_nodata_idxdata data_lengthepoch_progress_s r. _job_producerzWord2Vec._job_producers2!#A: (,% o))#y99 ' 66 < *__w6{ "&,%) 5#''))F '1,' Y[rsss17 .HmY NI X %M - /  i 'N$oo-G+% 5""~y-"K1CWNNN& 4 )&) 5, //E)  }nnk )< > > > (!>9<z4Word2Vec._train_epoch_corpusfile..Rsg     3,9< size = number of workers * queue_factor. report_delay : float, optional Number of seconds between two consecutive progress report messages in the logger. Returns ------- (int, int, int) The training report for this epoch consisting of three elements: * Size of data chunk processed, for example number of sentences in the corpus chunk. * Effective word count used in training (after ignoring unknown words and trimming the sentence length). * Total word count used in training. maxsizer7cJg|]}tjjf S))rZr[)r\r]r9)r^rGr5r1r{s r.raz)Word2Vec._train_epoch..sK     (1 3 3 3   r0)r rDrErYTF)r rDrErrL) r rNrrr\r]rHrir(rS)r{r8r rDrErrrIrNrjr&r'r)r5r1s` @@r.rzWord2Vec._train_epochfs1B,"=>>>  q(8DL'HIII      4<((     y'%+!*n]hiikkk l l l   F FM LLNNNN8<8P8P I>#,TY9Q9 9 5NI ">9<%d&6dCCC%d&6dCCCTzr0c4td|DS)aGet the number of words in a given job. Parameters ---------- job: iterable of list of str The corpus chunk processed in a single batch. Returns ------- int Number of raw words in the corpus chunk. c34K|]}t|VdSr)r)r^rs r. z+Word2Vec._raw_word_count..s(55X3x==555555r0)r)r{r7s r.rzWord2Vec._raw_word_counts!55555555r0c ||td||td|1tj|std|z|'t |t std|z|.t |t r|dkrtd|d|Wtj|\}}|tvrtd |d dSdS) z0Checks whether the corpus parameters make sense.NzCEither one of corpus_file or corpus_iterable value must be providedzJBoth corpus_file and corpus_iterable must not be provided at the same timezDParameter corpus_file must be a valid path to a file, got %r insteadzKThe corpus_iterable must be an iterable of lists of strings, got %r insteadr7z3Using a generator as corpus_iterable can't support z$ passes. Try a re-iterable sequence.zcTraining from compressed files is not supported with the `corpus_path` argument. Please decompress z" or use `corpus_iterable` instead.) TypeErrorrerfisfilerrrsplitextlowerr)r{rAr%rBrG corpus_exts r.rtzWord2Vec._check_corpus_sanitys  c? cabb b  j jhii i  r27>>++F+F rbeppqq q  qz/8/T/T q]`ooqq q  t:o}+U+U tZ`cdZd trfrrrtt t  G,,[99MAz!!%=%?%?? Y)4YYY    r0c 4|j|jkrtd|jjst dt|jjst d||td||dkrtddS)a6Checks whether the training parameters make sense. Parameters ---------- epochs : int Number of training epochs. A positive integer. total_examples : int, optional Number of documents in the corpus. Either `total_examples` or `total_words` **must** be supplied. total_words : int, optional Number of words in the corpus. Either `total_examples` or `total_words` **must** be supplied. **kwargs : object Unused. Present to preserve signature among base and inherited implementations. Raises ------ RuntimeError If one of the required training pre/post processing steps have not been performed. ValueError If the combination of input parameters is inconsistent. z6Effective 'alpha' higher than previous training cyclesz9you must first build vocabulary before training the modelz5you must initialize vectors before training the modelNzYou must specify either total_examples or total_words, for proper learning-rate and progress calculations. If you've just built the vocabulary using the same corpus, using the count cached in the model is sufficient: total_examples=model.corpus_count.rzRYou must specify an explicit epochs count. The usual value is epochs=model.epochs.) rTrarwrxr<rr"rrrb)r{rFrDrErs r.rzWord2Vec._check_training_sanitys, :2 2 U NNS T T Tw# \Z[[ [47?## XVWW W  > Q   sVq[ sqrr r s sr0c N|rRtd|d|z|z || z |dntj|tj|dStd|d|z|z || z |dntj|tj|dS)aCallback used to log progress for long running jobs. Parameters ---------- job_queue : Queue of (list of object, float) The queue of jobs still to be performed by workers. Each job is represented as a tuple containing the batch of data to be processed and the floating-point learning rate. progress_queue : Queue of (int, int, int) A queue of progress reports. Each report is represented as a tuple of these 3 elements: * size of data chunk processed, for example number of sentences in the corpus chunk. * Effective word count used in training (after ignoring unknown words and trimming the sentence length). * Total word count used in training. cur_epoch : int The current training iteration through the corpus. example_count : int Number of examples (could be sentences for example) processed until now. total_examples : int Number of all examples present in the input corpus. raw_word_count : int Number of words used in training until now. total_words : int Number of all words in the input corpus. trained_word_count : int Number of effective words used in training until now (after ignoring unknown words and trimming the sentence length). elapsed : int Elapsed time since the beginning of training in seconds. Notes ----- If you train the model via `corpus_file` argument, there is no job_queue, so reported job_queue size will always be equal to -1. zPEPOCH %i - PROGRESS: at %.2f%% examples, %.0f words/s, in_qsize %i, out_qsize %irNrzMEPOCH %i - PROGRESS: at %.2f%% words, %.0f words/s, in_qsize %i, out_qsize %i)rwrrqsize) r{r5r1r rMrDr'rEr&rRs r.rJzWord2Vec._log_progresssL   KKb5=0>ACUX_C_CU[-C-CU[Q_E`E`      KK_5>1K?ASV]A]CU[-C-CU[Q_E`E`     r0c td||||||z |rdS|r#||krtd||||r%||kr!td|||dSdSdS)aCallback used to log the end of a training epoch. Parameters ---------- cur_epoch : int The current training iteration through the corpus. example_count : int Number of examples (could be sentences for example) processed until now. total_examples : int Number of all examples present in the input corpus. raw_word_count : int Number of words used in training until now. total_words : int Number of all words in the input corpus. trained_word_count : int Number of effective words used in training until now (after ignoring unknown words and trimming the sentence length). elapsed : int Elapsed time since the beginning of training in seconds. is_corpus_file_mode : bool Whether training is file-based (corpus_file argument) or not. Warnings -------- In case the corpus is changed while the epoch was running. zZEPOCH %i: training on %i raw words (%i effective words) took %.1fs, %.0f effective words/sNzGEPOCH %i: supplied example count (%i) did not equal expected count (%i)zHEPOCH %i: supplied raw word count (%i) did not equal expected count (%i))rwrrx) r{r rMrDr'rEr&rRrLs r.rKzWord2Vec._log_epoch_endJs>  h ~'97DVY`D`     F  n =  NNY[d~     ;.8  NNZ\e          r0c X|dd|d|d|dd||z dd d S) aCallback to log the end of training. Parameters ---------- raw_word_count : int Number of words used in the whole training. trained_word_count : int Number of effective words used in training (after ignoring unknown words and trimming the sentence length). total_elapsed : int Total time spent during training in seconds. job_tally : int Total number of jobs processed during training. rvz training on z raw words (z effective words) took rzs, z.0fz effective words/srN)ry)r{r'r&r-r)s r.r"zWord2Vec._log_train_end~s    e> e e7I e e! d e e*<}*L d e e e !     r0g.Ac "tdjtjjjjjj jj stdjstdfdtd}}t|jzt|dzjzfdtjD}|D]} d | _| d } t#jt& d } d } t)t+jt)||} t/| \}}|dz |zkr.td |dz}t3td||n_#t2$rRtd|| z dztjD]}dd } YnwxYw | |dzks| sh| }| |z } | dz } t|z }||kr'tdd| z| |z ||z}| |dzkf| hn#t:$rYnwxYwdt|z }dj_td| || |z d| S)aScore the log probability for a sequence of sentences. This does not change the fitted model in any way (see :meth:`~gensim.models.word2vec.Word2Vec.train` for that). Gensim has currently only implemented score for the hierarchical softmax scheme, so you need to have run word2vec with `hs=1` and `negative=0` for this to work. Note that you should specify `total_sentences`; you'll run into problems if you ask to score more than this number of sentences but it is inefficient to set the value too high. See the `article by Matt Taddy: "Document Classification by Inversion of Distributed Language Representations" `_ and the `gensim demo `_ for examples of how to use such scores in document classification. Parameters ---------- sentences : iterable of list of str The `sentences` iterable can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network. See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus` or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples. total_sentences : int, optional Count of sentences. chunksize : int, optional Chunksize of jobs queue_factor : int, optional Multiplier for size of queue (number of workers * queue_factor). report_delay : float, optional Seconds to wait before reporting progress. zoscoring sentences with %i workers on %i vocabulary and %i features, using sg=%s hs=%s sample=%s and negative=%sz7you must first build vocabulary before scoring new datazWe have currently only implemented score for the hierarchical softmax scheme, so you need to have run word2vec with hs=1 and negative=0 for this to work.cftjdt}tj jt} }|dSd}|D]B\}}| krn6 jrt ||}nt |||}| |<|dz }C |u)zZCompute log probability for each sentence, lifting lists of sentences from the jobs queue.r7r=TNr) rYrrnrrrr?rrRrrr0) r r r7ns sentence_idrscorer5r1r{sentence_scorestotal_sentencess r. worker_loopz#Word2Vec.score..worker_loops8AT***D)$*:$GGGD 'mmooE-0)K"o5wP 1$$ G G 3D(D$ O O38OK0!GBB""2&&& 'r0rrlr7c:g|]}tjS))rZ)r\r])r^rGrs r.raz"Word2Vec.score..s'UUUA9#;777UUUr0Trr=FzMterminating after %i sentences (set higher total_sentences if you want more).zputting job #%i in the queuez;reached end of input; waiting to finish %i outstanding jobsNz/PROGRESS: at %.2f%% sentences, %.0f sentences/srz1scoring %i sentences took %.1fs, %.0f sentences/s)rwrrNrr<r?rRr\rgr]rr"rr rrir(rrrrnrrgroupernextrx StopIterationr4r0rr r)r{r|r chunksizerrr(rNrNrjsentence_count push_done done_jobs jobs_sourcerBrrGrrRr5r1rrs` ` @@@@r.rzWord2Vec.scores@   : L#dg,,(8$'47 K    w# ZXYY Yw ^   ' ' ' ' ' ' ' ' '(+__c{,"=>>>  q(8DL'HIIIUUUUt|ATATUUU  F FM LLNNNN"0MMM   i .B.BI N NOO !  ! $[ 1 1 QJ)+o=*NNg'aKF'//) ;VDDD e$$$$  ! ! ! Y[adm[mpq[qrrrt|,,((AMM$''''  !  6A:. i '++I66B"b(NNI+oo5G+-= M!N2NW4L'. &<  6A:. i     A! F //E)  ? G^g%=   //s''A 0 for this to work.rrz>Parameters required for predicting the output words not found.cVg|]%}|jv j|&Sr:)r< get_index)r^rr{s r.raz0Word2Vec.predict_output_word..1s6ZZZ!QRVRY\Z**1--ZZZr0zHAll the input context words are out-of-vocabulary for the current model.Nr)axisT)topnrcDg|]}jj||fSr:)r<r)r^index1 prob_valuesr{s r.raz0Word2Vec.predict_output_word..@s-^^^%f-{6/BC^^^r0)r]r"rlr<rwrxrYrrr_rexpdotrTrargsort)r{context_words_listr word2_indicesl1 top_indicesrs` @r.predict_output_wordzWord2Vec.predict_output_wordsD,} W  tw ** a'$ 2J2J a_`` `ZZZZ7IZZZ   NNe f f f4 VDGOM2 ; ; ;  %T^ % #m$$ $BfRVB 7788 rvk*** &{tLLL ^^^^^R]^^^^r0ct|j|_|jj|j_|jj|j_|jj|j_|j|_|j|_|dS)a=Borrow shareable pre-built structures from `other_model` and reset hidden layer weights. Structures copied are: * Vocabulary * Index to word mapping * Cumulative frequency table (used for negative sampling) * Cached corpus length Useful when testing multiple models on the same corpus in parallel. However, as the models then share all vocabulary-related structures other than vectors, neither should then expand their vocabulary (which could leave the other in an inconsistent, broken state). And, any changes to any per-word 'vecattr' will affect both models. Parameters ---------- other_model : :class:`~gensim.models.word2vec.Word2Vec` Another model to copy the internal structures from. N) rrMr<rrexpandosrjrbr)r{ other_models r. reset_fromzWord2Vec.reset_fromBsp*t/00*~:*~:&>2$.'4 r0c~|jjdt|jjd|jjd|jdS)zHuman readable representation of the model's state. Returns ------- str Human readable representation of the model's state, including the vocabulary size, vector size and learning rate. z) __class____name__rr<rrMrTrs r.__str__zWord2Vec.__str___sJ N # # #S)=%>%>%>%>@S@S@SUYU_U_U_  r0cHtt|j|i|dS)a%Save the model. This saved model can be loaded again using :func:`~gensim.models.word2vec.Word2Vec.load`, which supports online training and getting vectors for vocabulary words. Parameters ---------- fname : str Path to the file. N)superr3save)r{r[rrs r.rz Word2Vec.savems- #h"D3F33333r0c t|dg}tt||||||||S)zFArrange any special handling for the `gensim.utils.SaveLoad` protocol.rj)setunionrr3_save_specials) r{fname separately sep_limitignorepickle_protocolcompresssubnamers r.rzWord2Vec._save_specialszsTV""K?33Xt$$33 :y&/8WVV Vr0)rethrowc: tt|j|i|}t|ts1d}t dt |dt ||S#t$r%}|r|td|d}~wwxYw)aLoad a previously saved :class:`~gensim.models.word2vec.Word2Vec` model. See Also -------- :meth:`~gensim.models.word2vec.Word2Vec.save` Save model. Parameters ---------- fname : str Path to the saved file. Returns ------- :class:`~gensim.models.word2vec.Word2Vec` Loaded model. TzModel of type z can't be loaded by zModel load error. Was model saved using code from an older Gensim Version? Try loading older model using gensim-3.8.3, then re-saving, to restore compatibility with current code.N) rr3rsrAttributeErrorrrzrwerror)clsrr[rr$aers r.rsz Word2Vec.loads( -E(C((-t>v>>EeX.. i$nQUV[Q\Q\Q\Q\^abe^f^f^f%ghhhL     LL3 4 4 4H sA'A++ B5 BBc 2tt|j|i|t|dsd|_|jr)t|jdr|t|dsd|_t|dsd|_ t|jds:t|jdr%tj d t |j_ t|d s*tj|j |_t|d sd|_d|_t|ds|j|_|`t|dsd|_t|dr+dD]&}t-||t/|j|'|`t|dr@dD];}t|j|r$t-||t/|j|<|`t|ds d|_dSdS)zXHandle special requirements of `.load()` protocol, usually up-converting older versions.r^r9rrbNrcrorr7r=rZrrOrrFrd vocabulary)rerfrgrhrirk trainables)rpr?rqrrrXT)rr3_load_specialsrlr^r]r<rrbrcrYrmrnrorZ default_rngrqrOrPiterrFrdsetattrgetattrrrrX)r{r[rars r.rzWord2Vec._load_specialss=,h,d=f===t]++ $#D  = "WTWn== "    ! ! !t^,, % $D t122 +&*D #tw00 ;WTWi5P5P ;$&GAT$:$:$:DG !tX&& @)//TY/??DKt]++ & D $%D !tX&& )DK t.// (#'D 4 & & h > >a!!M>M>M>`V[,01Y1Y1Y1Yh^c/,/,/,/,b)))@ ) ) ) )AE16wwwwr>... &&&00000""""ZTUUddVUdRRR:::&Z ^ +++  +,.2****&6664 \`FGLNG2G2G2G2T[\-1#!#!#!#!J J J JDE@E@E@E@PTXDHH=H=H=H=V^`?=?=?=?=DPT8:9=9=9=9=v0   666 .&s&s&s&sP333j222h   (03s3xx3UVef0000B-_-_-_-_^:     4 4 4 4 4VVVVV!&      [ D#'#'#'#'#'J * * * * * * *r0r3ceZdZdZdZdS) BrownCorpusc||_dS)zIterate over sentences from the `Brown corpus `_ (part of `NLTK data `_). N)dirname)r{rs r.r}zBrownCorpus.__init__s  r0c#Ktj|jD]}tj|j|}tj|sGt j|d5}|D]G}t j|}d| D}d|D}|sC|VH dddn #1swxYwYdS)Nrbcg|]=}t|ddk(|d>S)/r)rsplit)r^ts r.raz(BrownCorpus.__iter__..s@!_!_!_1AGGTWLLHYHY]^H^!_!''#,,!_!_!_r0cg|]B\}}|dd!|d|ddCS)Nrr)isalpharz)r^tokentags r.raz(BrownCorpus.__iter__..sUrrrJE3`cdfefdf`g`o`o`q`qr s2A2ww?rrrr0) relistdirrrfjoinrxropen to_unicoder)r{rfinline token_tagswordss r.__iter__zBrownCorpus.__iter__s,Z --  EGLLu55E7>>%(( E4(( C   D +D11D"`!_ !_!_!_JrrR\rrrE ! KKKK     s8A CC C N)rrrr}rr:r0r.rrs2     r0rceZdZefdZdZdS) Text8Corpusc"||_||_dS)zbIterate over sentences from the "text8" corpus, unzipped from http://mattmahoney.net/dc/text8.zip.N)rmax_sentence_length)r{rrs r.r}zText8Corpus.__init__s #6   r0c#Kgd}}tj|jd5} ||dz}||krBtj|}|||r|Vn|d}|dkrJtj|d|||dfng|f\}}||t||j kr8|d|j V||j d}t||j k83 ddddS#1swxYwYdS)Nr0rTi  r) rrrreadrrextendrfindstriprr)r{rrestrtextr last_tokens r.rzText8Corpus.__iter__sS$ Z D ) ) CS Cchhtnn,4<!,T2288::EOOE***'&!ZZ-- =G1_]u/[j[0ABBHHJJ#JKK06688::SUW[R\t&&&(mmt'??C"#>> from gensim.test.utils import datapath >>> sentences = LineSentence(datapath('lee_background.cor')) >>> for sentence in sentences: ... pass N)sourcerlimitr{rrrs r.r}zLineSentence.__init__ s* #6  r0c#K |jdtj|j|jD]n}t j|}d}|t|kr1||||j zV||j z }|t|k1odS#t$rt j |jd5}tj||jD]n}t j|}d}|t|kr1||||j zV||j z }|t|k1o dddYdS#1swxYwYYdSwxYw)z(Iterate through the lines in the source.rrN) rseek itertoolsislicerrrrrrrr)r{rirs r.rzLineSentence.__iter__%s 6 K  Q   !(djAA 2 2'--3355#d))m2q!d&>">>????11A#d))m2 2 2  6 6 6DK.. 6#%,S$*==66D +D117799DAc$ii-6"1a$*B&B#BCCCCT55c$ii-66 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6s7B(B..$E:B E+ E:+E/ /E:2E/ 3E:9E:rr:r0r.rr s<3ET266666r0rc eZdZedfdZdZdS)PathLineSentencesNc|_|_|_tjjrBt dt djg_ntj jrtj jd_t djtj j_fdjD_j ntdt dd jd S) aMLike :class:`~gensim.models.word2vec.LineSentence`, but process all files in a directory in alphabetical order by filename. The directory must only contain files that can be read by :class:`gensim.models.word2vec.LineSentence`: .bz2, .gz, and text files. Any file not ending with .bz2 or .gz is assumed to be a text file. The format of files (either text, or compressed text files) in the path is one sentence = one line, with words already preprocessed and separated by whitespace. Warnings -------- Does **not recurse** into subdirectories. Parameters ---------- source : str Path to the directory. limit : int or None Read only the first `limit` lines from each file. Read all if limit is None (the default). z=single file given as source, rather than a directory of filesz=consider using models.word2vec.LineSentence for a single filezreading directory %sc$g|] }j|z Sr:)r)r^filenamer{s r.raz.PathLineSentences.__init__.._s XXX8 h 6XXXr0z"input is neither a file nor a pathz$files read into PathLineSentences:%s N)rrrrerfrxrwr4 input_filesisdirrrrsortrbrs` r.r}zPathLineSentences.__init__=s6, #6  7>>$+ & & C LLX Y Y Y LLX Y Y Y $ }D   W]]4; ' ' C',,t{B77DK KK. < < <!z$+66D XXXXtGWXXXD    ! ! # # # #ABB B :DIIdFV.s?ii733QdCCiiiiiir0r7)rindexleftright) listrrrheapqheapifyheappopheappushrr)r<heaprmin1min2s` r. _build_heapr&s iiiiUSVWYWfSgSgMhMhiii i iD M$ 3r77Q;    ]4((%-*=*=d  (dj!8CGG RV^bccc     Kr0ctdt|t|}|stddSd}|dggfg}|rf|\}}}|dt|krT|d}||d|||d|t t||}ntjt||j t|z gztj }| |j tjt|dgztj|f| |jtjt|dgztj|f|ftd |dS) a Appends a binary code to each vocab term. Parameters ---------- wv : KeyedVectors A collection of word-vectors. Sets the .code and .point attributes of each node. Each code is a numpy.array containing 0s and 1s. Each point is an integer. z)constructing a huffman tree from %i wordsz,built huffman tree with maximum node depth 0Nrr7codepointr=z-built huffman tree with maximum node depth %i)rwrrr&poprrrYarrayrrrrruint8r)r<r# max_depthstacknodecodespointsks r.rrs KK;SWWEEE r??D   BCCCI1gr2  E  \#iikkeV 7SWW  \QA NN1fe , , , NN1gv . . .CJJ 22IIXd6lldj3r77.B-CC29UUUF LL$)RXd5kkQC.?rx%P%P%PRXY Z Z Z LL$*bhtE{{aS/@&Q&Q&QSYZ [ [ [  \ KK?KKKKKr0__main__z:%(asctime)s : %(threadName)s : %(levelname)s : %(message)s)formatlevelz running %s rrr7)r3raise)allz-trainz0Use text data from file TRAIN to train the modelT)helprequiredz-outputz2Use file OUTPUT to save the resulting word vectors)r9z-windowz6Set max skip length WINDOW between words; default is 5r5)r9rdefaultz-sizez(Set size of word vectors; default is 100r4z-samplezSet threshold for occurrence of words. Those that appear with higher frequency in the training data will be randomly down-sampled; default is 1e-3, useful range is (0, 1e-5)r6z-hsz1Use Hierarchical Softmax; default is 0 (not used))r9rr;choicesz -negativezRNumber of negative examples; default is 5, common values are 3 - 10 (0 = not used)z-threadsz Use THREADS threads (default 12) z-iterz(Run more training iterations (default 5)z -min_countzKThis will discard words that appear less than MIN_COUNT times; default is 5z-cbowzOUse the continuous bag of words model; default is 1 (use 0 for skip-gram model)z-binaryz=Save the resulting vectors in binary mode; default is 0 (off)z -accuracyz6Use questions from file ACCURACY to evaluate the model) rMrfrNrVrgrRr\r]r_rF)binaryz.modelz .model.binz .model.txtFzfinished running %s)mr __future__rloggingsysrertimeitr collectionsrrcollections.abcrrrr\rrhqueuer r numpyr rnrY gensim.utilsr r rgensim.models.keyedvectorsrrgensimrrrsmart_open.compressionr getLoggerrrwgensim.models.word2vec_innerrrrrrr ImportError NO_CYTHONrcrrrSaveLoadr3rrrrr rrr&rargparse basicConfigINFOrrargvrfbasenameprogramrprintglobalslocalsexitgensim.models.word2vecseterrArgumentParserparser add_argumentrLrS parse_argsr[cbowskipgramrvcorpussizerfthreadsrVrgr\r]rr$outputoutfiler<save_word2vec_formatr>raccuracyr:r0r.risIkkZ  ////////$$$$$$ !!!!!!HHHHHHHHHHMMMMMMMM""""""""-,,,,,;;;;;;  8 $ $  /RfffffffffffRRRRRR RRRRRR"a*a*a*a*a*u~a*a*a*H7        4CCCCCCCC8.6.6.6.6.6.6.6.6b3636363636363636l     EN           (((((zz*&ABB((( *L*L*L^ zS0OOOGKl KK chhsx00111gsx{++G s38}}q ggii "VVXX-... ////// BI' $X $ & &F 'Ydhiii  (\]]]  (`gjtuvvv &PWZdghhh ;D   G !aV n !  )KRU_abbb &PWZdefff h ! g !aV W !aV  *bccc     D yA~ \$* % %F HDI{4;8!DI   E  {'+ %%gdk%BBBB* 7X%&&& {aL %%g &