c0dZddlZddlZddlmZmZmZddlZ ddl m Z ddl m Z mZddl mZddlmZ ddlmZmZmZmZmZdd lmZmZn#e$rejwxYwejeZGd d e Z Gd d ej!Z"Gddej!Z#dZ$d#dZ%d#dZ&d$dZ'dZ(d%dZ)Gdde Z*dZ+d&dZ,dZ-dZ.d Z/d!Z0dd"l1m2Z2e*e2_*dS)'a" Introduction ------------ Learn word representations via fastText: `Enriching Word Vectors with Subword Information `_. This module allows training word embeddings from a training corpus with the additional ability to obtain word vectors for out-of-vocabulary words. This module contains a fast native C implementation of fastText with Python interfaces. It is **not** only a wrapper around Facebook's implementation. This module supports loading models trained with Facebook's fastText implementation. It also supports continuing training from such models. For a tutorial see :ref:`sphx_glr_auto_examples_tutorials_run_fasttext.py`. Usage examples -------------- Initialize and train a model: .. sourcecode:: pycon >>> from gensim.models import FastText >>> from gensim.test.utils import common_texts # some example sentences >>> >>> print(common_texts[0]) ['human', 'interface', 'computer'] >>> print(len(common_texts)) 9 >>> model = FastText(vector_size=4, window=3, min_count=1) # instantiate >>> model.build_vocab(corpus_iterable=common_texts) >>> model.train(corpus_iterable=common_texts, total_examples=len(common_texts), epochs=10) # train Once you have a model, you can access its keyed vectors via the `model.wv` attributes. The keyed vectors instance is quite powerful: it can perform a wide range of NLP tasks. For a full list of examples, see :class:`~gensim.models.keyedvectors.KeyedVectors`. You can also pass all the above parameters to the constructor to do everything in a single line: .. sourcecode:: pycon >>> model2 = FastText(vector_size=4, window=3, min_count=1, sentences=common_texts, epochs=10) The two models above are instantiated differently, but behave identically. For example, we can compare the embeddings they've calculated for the word "computer": .. sourcecode:: pycon >>> import numpy as np >>> >>> np.allclose(model.wv['computer'], model2.wv['computer']) True In the above examples, we trained the model from sentences (lists of words) loaded into memory. This is OK for smaller datasets, but for larger datasets, we recommend streaming the file, for example from disk or the network. In Gensim, we refer to such datasets as "corpora" (singular "corpus"), and keep them in the format described in :class:`~gensim.models.word2vec.LineSentence`. Passing a corpus is simple: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> corpus_file = datapath('lee_background.cor') # absolute path to corpus >>> model3 = FastText(vector_size=4, window=3, min_count=1) >>> model3.build_vocab(corpus_file=corpus_file) # scan over corpus to build the vocabulary >>> >>> total_words = model3.corpus_total_words # number of words in the corpus >>> model3.train(corpus_file=corpus_file, total_words=total_words, epochs=5) The model needs the `total_words` parameter in order to manage the training rate (alpha) correctly, and to give accurate progress estimates. The above example relies on an implementation detail: the :meth:`~gensim.models.fasttext.FastText.build_vocab` method sets the `corpus_total_words` (and also `corpus_count`) model attributes. You may calculate them by scanning over the corpus yourself, too. If you have a corpus in a different format, then you can use it by wrapping it in an `iterator `_. Your iterator should yield a list of strings each time, where each string should be a separate word. Gensim will take care of the rest: .. sourcecode:: pycon >>> from gensim.utils import tokenize >>> from gensim import utils >>> >>> >>> class MyIter: ... def __iter__(self): ... path = datapath('crime-and-punishment.txt') ... with utils.open(path, 'r', encoding='utf-8') as fin: ... for line in fin: ... yield list(tokenize(line)) >>> >>> >>> model4 = FastText(vector_size=4, window=3, min_count=1) >>> model4.build_vocab(corpus_iterable=MyIter()) >>> total_examples = model4.corpus_count >>> model4.train(corpus_iterable=MyIter(), total_examples=total_examples, epochs=5) Persist a model to disk with: .. sourcecode:: pycon >>> from gensim.test.utils import get_tmpfile >>> >>> fname = get_tmpfile("fasttext.model") >>> >>> model.save(fname) >>> model = FastText.load(fname) Once loaded, such models behave identically to those created from scratch. For example, you can continue training the loaded model: .. sourcecode:: pycon >>> import numpy as np >>> >>> 'computation' in model.wv.key_to_index # New word, currently out of vocab False >>> old_vector = np.copy(model.wv['computation']) # Grab the existing vector >>> new_sentences = [ ... ['computer', 'aided', 'design'], ... ['computer', 'science'], ... ['computational', 'complexity'], ... ['military', 'supercomputer'], ... ['central', 'processing', 'unit'], ... ['onboard', 'car', 'computer'], ... ] >>> >>> model.build_vocab(new_sentences, update=True) # Update the vocabulary >>> model.train(new_sentences, total_examples=len(new_sentences), epochs=model.epochs) >>> >>> new_vector = model.wv['computation'] >>> np.allclose(old_vector, new_vector, atol=1e-4) # Vector has changed, model has learnt something False >>> 'computation' in model.wv.key_to_index # Word is still out of vocab False .. Important:: Be sure to call the :meth:`~gensim.models.fasttext.FastText.build_vocab` method with `update=True` before the :meth:`~gensim.models.fasttext.FastText.train` method when continuing training. Without this call, previously unseen terms will not be added to the vocabulary. You can also load models trained with Facebook's fastText implementation: .. sourcecode:: pycon >>> cap_path = datapath("crime-and-punishment.bin") >>> fb_model = load_facebook_model(cap_path) Once loaded, such models behave identically to those trained from scratch. You may continue training them on new data: .. sourcecode:: pycon >>> 'computer' in fb_model.wv.key_to_index # New word, currently out of vocab False >>> old_computer = np.copy(fb_model.wv['computer']) # Calculate current vectors >>> fb_model.build_vocab(new_sentences, update=True) >>> fb_model.train(new_sentences, total_examples=len(new_sentences), epochs=model.epochs) >>> new_computer = fb_model.wv['computer'] >>> np.allclose(old_computer, new_computer, atol=1e-4) # Vector has changed, model has learnt something False >>> 'computer' in fb_model.wv.key_to_index # New word is now in the vocabulary True If you do not intend to continue training the model, consider using the :func:`gensim.models.fasttext.load_facebook_vectors` function instead. That function only loads the word embeddings (keyed vectors), consuming much less CPU and RAM: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> cap_path = datapath("crime-and-punishment.bin") >>> wv = load_facebook_vectors(cap_path) >>> >>> 'landlord' in wv.key_to_index # Word is out of vocabulary False >>> oov_vector = wv['landlord'] # Even OOV words have vectors in FastText >>> >>> 'landlady' in wv.key_to_index # Word is in the vocabulary True >>> iv_vector = wv['landlady'] Retrieve the word-vector for vocab and out-of-vocab word: .. sourcecode:: pycon >>> existent_word = "computer" >>> existent_word in model.wv.key_to_index True >>> computer_vec = model.wv[existent_word] # numpy vector of a word >>> >>> oov_word = "graph-out-of-vocab" >>> oov_word in model.wv.key_to_index False >>> oov_vec = model.wv[oov_word] # numpy vector for OOV word You can perform various NLP word tasks with the model, some of them are already built-in: .. sourcecode:: pycon >>> similarities = model.wv.most_similar(positive=['computer', 'human'], negative=['interface']) >>> most_similar = similarities[0] >>> >>> similarities = model.wv.most_similar_cosmul(positive=['computer', 'human'], negative=['interface']) >>> most_similar = similarities[0] >>> >>> not_matching = model.wv.doesnt_match("human computer interface tree".split()) >>> >>> sim_score = model.wv.similarity('computer', 'human') Correlation with human opinion on word similarity: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> similarities = model.wv.evaluate_word_pairs(datapath('wordsim353.tsv')) And on word analogies: .. sourcecode:: pycon >>> analogies_result = model.wv.evaluate_word_analogies(datapath('questions-words.txt')) N)onesvstackfloat32)Word2Vec) KeyedVectors prep_vectors)utils) deprecated)train_batch_anyMAX_WORDS_IN_BATCHcompute_ngramscompute_ngrams_bytes ft_hash_bytes)train_epoch_sgtrain_epoch_cbowcReZdZdddddddddddddd dd dedddd dd ded ddffd ZdZfdZddZ ddZ dZ e dd dZ e ej dd!dZej dd!dZfdZe fdZfdZxZS)"FastTextNrdg?gMbP?g-C6?g?iTctj|_tj|_||_| dkrt d| |_||krd}t|||||_tdt|j_ tdt|j_ tt|||| ||||||||| ||| ||||| |||||dS)uf#Train, use and evaluate word representations learned using the method described in `Enriching Word Vectors with Subword Information `_, aka FastText. The model can be stored/loaded via its :meth:`~gensim.models.fasttext.FastText.save` and :meth:`~gensim.models.fasttext.FastText.load` methods, or loaded from a format compatible with the original Fasttext implementation via :func:`~gensim.models.fasttext.load_facebook_model`. Parameters ---------- sentences : iterable of list of str, optional Can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network. See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus' or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples. If you don't supply `sentences`, the model is left uninitialized -- use if you plan to initialize it in some other way. corpus_file : str, optional Path to a corpus file in :class:`~gensim.models.word2vec.LineSentence` format. You may use this argument instead of `sentences` to get performance boost. Only one of `sentences` or `corpus_file` arguments need to be passed (or none of them, in that case, the model is left uninitialized). min_count : int, optional The model ignores all words with total frequency lower than this. vector_size : int, optional Dimensionality of the word vectors. window : int, optional The maximum distance between the current and predicted word within a sentence. workers : int, optional Use these many worker threads to train the model (=faster training with multicore machines). alpha : float, optional The initial learning rate. min_alpha : float, optional Learning rate will linearly drop to `min_alpha` as training progresses. sg : {1, 0}, optional Training algorithm: skip-gram if `sg=1`, otherwise CBOW. hs : {1,0}, optional If 1, hierarchical softmax will be used for model training. If set to 0, and `negative` is non-zero, negative sampling will be used. seed : int, optional Seed for the random number generator. Initial vectors for each word are seeded with a hash of the concatenation of word + `str(seed)`. Note that for a fully deterministically-reproducible run, you must also limit the model to a single worker thread (`workers=1`), to eliminate ordering jitter from OS thread scheduling. (In Python 3, reproducibility between interpreter launches also requires use of the `PYTHONHASHSEED` environment variable to control hash randomization). max_vocab_size : int, optional Limits the RAM during vocabulary building; if there are more unique words than this, then prune the infrequent ones. Every 10 million word types need about 1GB of RAM. Set to `None` for no limit. sample : float, optional The threshold for configuring which higher-frequency words are randomly downsampled, useful range is (0, 1e-5). negative : int, optional If > 0, negative sampling will be used, the int for negative specifies how many "noise words" should be drawn (usually between 5-20). If set to 0, no negative sampling is used. ns_exponent : float, optional The exponent used to shape the negative sampling distribution. A value of 1.0 samples exactly in proportion to the frequencies, 0.0 samples all words equally, while a negative value samples low-frequency words more than high-frequency words. The popular default value of 0.75 was chosen by the original Word2Vec paper. More recently, in https://arxiv.org/abs/1804.04212, Caselles-Dupré, Lesaint, & Royo-Letelier suggest that other values may perform better for recommendation applications. cbow_mean : {1,0}, optional If 0, use the sum of the context word vectors. If 1, use the mean, only applies when cbow is used. hashfxn : function, optional Hash function to use to randomly initialize weights, for increased training reproducibility. iter : int, optional Number of iterations (epochs) over the corpus. trim_rule : function, optional Vocabulary trimming rule, specifies whether certain words should remain in the vocabulary, be trimmed away, or handled using the default (discard if word count < min_count). Can be None (min_count will be used, look to :func:`~gensim.utils.keep_vocab_item`), or a callable that accepts parameters (word, count, min_count) and returns either :attr:`gensim.utils.RULE_DISCARD`, :attr:`gensim.utils.RULE_KEEP` or :attr:`gensim.utils.RULE_DEFAULT`. The rule, if given, is only used to prune vocabulary during :meth:`~gensim.models.fasttext.FastText.build_vocab` and is not stored as part of themodel. The input parameters are of the following types: * `word` (str) - the word we are examining * `count` (int) - the word's frequency count in the corpus * `min_count` (int) - the minimum count threshold. sorted_vocab : {1,0}, optional If 1, sort the vocabulary by descending frequency before assigning word indices. batch_words : int, optional Target size (in words) for batches of examples passed to worker threads (and thus cython routines).(Larger batches will be passed if individual texts are longer than 10000 words, but the standard cython code truncates to that maximum.) min_n : int, optional Minimum length of char n-grams to be used for training word representations. max_n : int, optional Max length of char ngrams to be used for training word representations. Set `max_n` to be lesser than `min_n` to avoid char ngrams being used. word_ngrams : int, optional In Facebook's FastText, "max length of word ngram" - but gensim only supports the default of 1 (regular unigram word handling). bucket : int, optional Character ngrams are hashed into a fixed number of buckets, in order to limit the memory usage of the model. This option specifies the number of buckets used by the model. The default value of 2000000 consumes as much memory as having 2000000 more in-vocabulary words in your model. callbacks : :obj: `list` of :obj: `~gensim.models.callbacks.CallbackAny2Vec`, optional List of callbacks that need to be executed/run at specific stages during training. max_final_vocab : int, optional Limits the vocab to a target vocab size by automatically selecting ``min_count```. If the specified ``min_count`` is more than the automatically calculated ``min_count``, the former will be used. Set to ``None`` if not required. shrink_windows : bool, optional New in 4.1. Experimental. If True, the effective window size is uniformly sampled from [1, `window`] for each target word during training, to match the original word2vec algorithm's approximate weighting of context words by distance. Otherwise, the effective window size is always fixed to `window` words to either side. Examples -------- Initialize and train a `FastText` model: .. sourcecode:: pycon >>> from gensim.models import FastText >>> sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]] >>> >>> model = FastText(sentences, min_count=1) >>> say_vector = model.wv['say'] # get vector for word >>> of_vector = model.wv['of'] # get vector for out-of-vocab word Attributes ---------- wv : :class:`~gensim.models.fasttext.FastTextKeyedVectors` This object essentially contains the mapping between words and embeddings. These are similar to the embedding computed in the :class:`~gensim.models.word2vec.Word2Vec`, however here we also include vectors for n-grams. This allows the model to compute embeddings even for **unseen** words (that do not exist in the vocabulary), as the aggregate of the n-grams included in the word. After training the model, this attribute can be used directly to query those embeddings in various ways. Check the module level docstring for some examples. rzGGensim's FastText implementation does not yet support word_ngrams != 1.rdtype) sentences corpus_fileworkers vector_sizeepochs callbacks batch_words trim_rulesgalphawindowmax_vocab_sizemax_final_vocab min_countsample sorted_vocab null_word ns_exponenthashfxnseedhsnegative cbow_mean min_alphashrink_windowsN)r call_on_class_onlyloadload_fasttext_formatr"NotImplementedError word_ngramsFastTextKeyedVectorswvrREALvectors_vocab_lockfvectors_ngrams_lockfsuperr__init__)selfrrr%r1r r&r'r*r(r:r+r0rr4r2r.r3r/r!r-min_nmax_nr,bucketr$r#r"r)r5 __class__s 6lib/python3.11/site-packages/gensim/models/fasttext.pyrAzFastText.__init__sb, $)$<!" !  q%&opp p& 5= F&{E5&II'+1D&9&9&9#'+AT':':':$ h&&['Wbkq[IRT\ajp)?\['"x9 ' @ @ @ @ @c~t|jj}t|j}|jj}|dks Jd|dks Jdt dt |j_t dt |j_|jr||_ |j r||_ ||_ dS)Nrz.expected num_vectors to be initialized alreadyz-expected vocab_size to be initialized alreadyrr) lenr<vectorsr rr=r?r>r1syn1r2syn1neg layer1_size)rB hidden_output num_vectors vocab_sizer s rG_init_post_loadzFastText._init_post_loads$'/** \\ g) QPP PPPPA~NNNNNN(,AT':':':$&*1D&9&9&9# 7 &%DI = )(DL&rHctt||jdS)z;Clear any cached values that training may have invalidated.N)r@r_clear_post_trainr<adjust_vectors)rBrFs rGrTzFastText._clear_post_trains9 h//111      rHc|pt|j}|jtjtjjz}|jtjtjjz}|pi}t|j|jrdndz|d<t|j|z|d<|jj }|jrt|j|z|d<|j rt|j|z|d<|jj r|jj |z|d<d}|jj D]D}t||jj |jj|jj }|t|z }Ed d t|jzzd |zz|d <t||d <t"dt|j||j|d |S)zUEstimate memory that will be needed to train a model, and print the estimates to log.iivocab syn0_vocabrLrM syn0_ngramsr@r buckets_wordtotalzNestimated required memory for %i words, %i buckets and %i dimensions: %i bytes)rJr<r nprritemsizerNr1rEr2 key_to_indexft_ngram_hashesrCrDsumvaluesloggerinfo) rBrQreportvec_sizel1_size num_buckets num_ngramswordhashess rGestimate_memoryzFastText.estimate_memorys/3tw<< #bhrz&:&:&CC"RXbj%9%9%BB2dg,,*A##cBw"47||h6|gn 7 4 \\G3F6N = 7 #DG w 6F9  7> R$(GNX$=F= !J, * *(tw}dgmTW^\\c&kk) &(3TW+=%>!j.%QF> "fmmoo..w \ LL+t'7    rHc |\} } |jrt|||||||| | \} } }nt|||||||| | \} } }| | |fSN)r%rr)rBr thread_idoffset cython_vocabthread_private_mem cur_epochtotal_examples total_wordskwargsworkneu1examplestally raw_tallys rG_do_train_epochzFastText._do_train_epochs( d 7 )7k6<NT_aegk** &HeYY*:k6<NT_aegk** &HeY ))rHc`|\}}t|||||}|||fS)aTrain a single batch of sentences. Return 2-tuple `(effective word count after ignoring unknown words and sentence length trimming, total word count)`. Parameters ---------- sentences : iterable of list of str Can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from disk/network. See :class:`~gensim.models.word2vec.BrownCorpus`, :class:`~gensim.models.word2vec.Text8Corpus` or :class:`~gensim.models.word2vec.LineSentence` in :mod:`~gensim.models.word2vec` module for such examples. alpha : float The current learning rate. inits : tuple of (:class:`numpy.ndarray`, :class:`numpy.ndarray`) Each worker's private work memory. Returns ------- (int, int) Tuple of (effective word count after ignoring unknown words and sentence length trimming, total word count) )r _raw_word_count)rBrr&initsrxryr{s rG _do_train_jobzFastText._do_train_jobs;, didCCd**95555rHzGensim 4.0.0 implemented internal optimizations that make calls to init_sims() unnecessary. init_sims() is now obsoleted and will be completely removed in future versions. See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4Fc<|j|dS)a Precompute L2-normalized vectors. Obsoleted. If you need a single unit-normalized vector for some key, call :meth:`~gensim.models.keyedvectors.KeyedVectors.get_vector` instead: ``fasttext_model.wv.get_vector(key, norm=True)``. To refresh norms after you performed some atypical out-of-band vector tampering, call `:meth:`~gensim.models.keyedvectors.KeyedVectors.fill_norms()` instead. Parameters ---------- replace : bool If True, forget the original trained vectors and only keep the normalized ones. You lose information if you do this. )replaceN)r< init_sims)rBrs rGrzFastText.init_simss#. '*****rHzuse load_facebook_vectors (to use pretrained embeddings) or load_facebook_model (to continue training with the loaded full model, more RAM) insteadutf8c$t||S)zDeprecated. Use :func:`gensim.models.fasttext.load_facebook_model` or :func:`gensim.models.fasttext.load_facebook_vectors` instead. encoding)load_facebook_model)cls model_filers rGr8zFastText.load_fasttext_format8s#:AAAArHct|j|}|jD]\}}t |||dS)zLoad data from a binary file created by Facebook's native FastText. Parameters ---------- encoding : str, optional Specifies the encoding. rN)_load_fasttext_format file_name__dict__itemssetattr)rBrmattrvals rGload_binary_datazFastText.load_binary_dataFsX "$.8 D D D))++ % %ID# D$ $ $ $ $ % %rHcHtt|j|i|dS)aSave the Fasttext model. This saved model can be loaded again using :meth:`~gensim.models.fasttext.FastText.load`, which supports incremental training and getting vectors for out-of-vocabulary words. Parameters ---------- fname : str Store the model to this file. See Also -------- :meth:`~gensim.models.fasttext.FastText.load` Load :class:`~gensim.models.fasttext.FastText` model. N)r@rsaverBargsrwrFs rGrz FastText.saveWs- #h"D3F33333rHcHtt|j|ddi|S)aLoad a previously saved `FastText` model. Parameters ---------- fname : str Path to the saved file. Returns ------- :class:`~gensim.models.fasttext.FastText` Loaded model. See Also -------- :meth:`~gensim.models.fasttext.FastText.save` Save :class:`~gensim.models.fasttext.FastText` model. rethrowT)r@rr7)rrrwrFs rGr7z FastText.loadis,()uXs##($GGGGGrHctt|j|i|t|dr|j|j_|`dSdS)XHandle special requirements of `.load()` protocol, usually up-converting older versions.rEN)r@r_load_specialshasattrrEr<rs rGrzFastText._load_specialssU,h,d=f=== 4 " " ![DGN   rH)NNF)r)__name__ __module__ __qualname__hashr rArRrTrmr}rr r classmethodr r8rrr7r __classcell__rFs@rGrrs!%4A!QT\aQ $!DqRS_eDQR^_ghpq4M_km!%d h@h@h@h@h@h@T'''(!!!!! @.2****"6666Z ^ +++  +(U NBBB [ BU N % % %  %44444$HHHH[H*rHrceZdZdZdS) FastTextVocabzmThis is a redundant class. It exists only to maintain backwards compatibility with older gensim versions.Nrrr__doc__rrHrGrrs####rHrceZdZdZdS)FastTextTrainablesz7Obsolete class retained for backward-compatible load()sNrrrHrGrrsAAAArHrct||kr!td|t|fztj|t}||dt|<|S)z3Pad array with additional entries filled with ones.z5the new number of rows %i must be greater than old %irN)rJ ValueErrorr^rr=)rnew_lennew_arrs rG _pad_onesrsg 1vvfPT[]`ab]c]cSddeeeggT***GGGSVVG NrHutf-8c&t||dS)aLoad the model from Facebook's native fasttext `.bin` output file. Notes ------ Facebook provides both `.vec` and `.bin` files with their modules. The former contains human-readable vectors. The latter contains machine-readable vectors along with other model parameters. This function requires you to **provide the full path to the .bin file**. It effectively ignores the `.vec` output file, since it is redundant. This function uses the smart_open library to open the path. The path may be on a remote host (e.g. HTTP, S3, etc). It may also be gzip or bz2 compressed (i.e. end in `.bin.gz` or `.bin.bz2`). For details, see ``__. Parameters ---------- model_file : str Path to the FastText output files. FastText outputs two model files - `/path/to/model.vec` and `/path/to/model.bin` Expected value for this example: `/path/to/model` or `/path/to/model.bin`, as Gensim requires only `.bin` file to the load entire fastText model. encoding : str, optional Specifies the file encoding. Examples -------- Load, infer, continue training: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> cap_path = datapath("crime-and-punishment.bin") >>> fb_model = load_facebook_model(cap_path) >>> >>> 'landlord' in fb_model.wv.key_to_index # Word is out of vocabulary False >>> oov_term = fb_model.wv['landlord'] >>> >>> 'landlady' in fb_model.wv.key_to_index # Word is in the vocabulary True >>> iv_term = fb_model.wv['landlady'] >>> >>> new_sent = [['lord', 'of', 'the', 'rings'], ['lord', 'of', 'the', 'flies']] >>> fb_model.build_vocab(new_sent, update=True) >>> fb_model.train(sentences=new_sent, total_examples=len(new_sent), epochs=5) Returns ------- gensim.models.fasttext.FastText The loaded model. See Also -------- :func:`~gensim.models.fasttext.load_facebook_vectors` loads the word embeddings only. Its faster, but does not enable you to continue training. Tr full_model)r)pathrs rGrrs| !T J J JJrHc4t||d}|jS)arLoad word embeddings from a model saved in Facebook's native fasttext `.bin` format. Notes ------ Facebook provides both `.vec` and `.bin` files with their modules. The former contains human-readable vectors. The latter contains machine-readable vectors along with other model parameters. This function requires you to **provide the full path to the .bin file**. It effectively ignores the `.vec` output file, since it is redundant. This function uses the smart_open library to open the path. The path may be on a remote host (e.g. HTTP, S3, etc). It may also be gzip or bz2 compressed. For details, see ``__. Parameters ---------- path : str The location of the model file. encoding : str, optional Specifies the file encoding. Returns ------- gensim.models.fasttext.FastTextKeyedVectors The word embeddings. Examples -------- Load and infer: >>> from gensim.test.utils import datapath >>> >>> cap_path = datapath("crime-and-punishment.bin") >>> fbkv = load_facebook_vectors(cap_path) >>> >>> 'landlord' in fbkv.key_to_index # Word is out of vocabulary False >>> oov_vector = fbkv['landlord'] >>> >>> 'landlady' in fbkv.key_to_index # Word is in the vocabulary True >>> iv_vector = fbkv['landlady'] See Also -------- :func:`~gensim.models.fasttext.load_facebook_model` loads the full model, not just word embeddings, and enables you to continue model training. Fr)rr<)rrrs rGload_facebook_vectorsrs!j'th5QQQJ =rHTc 8tj|d5}tjj|||}dddn #1swxYwYt |j|j|j |j t|j dkt|j dk|j|j|j|j|j }|j|_|j|_|j|_|j|_|dd|jjd |_|j|j||jtA||!d d |jjd |j" |S)aLoad the input-hidden weight matrix from Facebook's native fasttext `.bin` output files. Parameters ---------- model_file : str Full path to the FastText model file. encoding : str, optional Specifies the file encoding. full_model : boolean, optional If False, skips loading the hidden output matrix. This saves a fair bit of CPU time and RAM, but prevents training continuation. Returns ------- :class: `~gensim.models.fasttext.FastText` The loaded model. rbrNr) r r'r!r2r1r%rEr*r+rCrDT)updater*rr8zloaded z' weight matrix for fastText model from )msg)#r opengensimmodels _fasttext_binr7rdimwsepochnegintlossmodelrEr*tminnmaxnntokenscorpus_total_words raw_vocabnwordsrQ prepare_vocabvectors_ngramsshapenum_original_vectorsr<init_post_loadrRrO _check_modeladd_lifecycle_eventname)rrrfinrrs rGrrs& J % %\ M ' , ,S8PZ , [ [\\\\\\\\\\\\\\\ Etw qv{   qw!|  x+sff   E !yEkEO8EL|E tq111!"!1!7!:E HA,--- !/***  _a&, _ _UXU] _ _ Ls(A  AAc|jj|jjjdkr*t d|jjd|jjdt |drQ|jJ|jj|jjdkr*t d|jjd|jjdt|j|jkr-t dt|jd |jd t|j|j kr5t d t|j|j dSdS) zJModel sanity checks. Run after everything has been completely initialized.rz.mismatch between vector size in model params (z) and model vectors ()rMNz) and trainables (z#mismatch between final vocab size (z' words), and expected number of words (z words)zPmismatch between final vocab size (%s words), and expected vocab size (%s words)) r<r rrrrrMrJrrQrdwarning)rs rGrrXs_t14.4Q77 j   !$"5"5"5    q) 4 qyq1 1 *D$$$ad&9&9&9   14yyAH jAD 1888     14yyAL   ^ IIq|       rHrrc^||d}tjj||||dS)a\Saves word embeddings to the Facebook's native fasttext `.bin` format. Notes ------ Facebook provides both `.vec` and `.bin` files with their modules. The former contains human-readable vectors. The latter contains machine-readable vectors along with other model parameters. **This function saves only the .bin file**. Parameters ---------- model : gensim.models.fasttext.FastText FastText model to be saved. path : str Output path and filename (including `.bin` extension) encoding : str, optional Specifies the file encoding. Defaults to utf-8. lr_update_rate : int This parameter is used by Facebook fasttext tool, unused by Gensim. It defaults to Facebook fasttext default value `100`. In very rare circumstances you might wish to fiddle with it. word_ngrams : int This parameter is used by Facebook fasttext tool, unused by Gensim. It defaults to Facebook fasttext default value `1`. In very rare circumstances you might wish to fiddle with it. Returns ------- None )lr_update_rater:N)rrrr)rrrrr:fb_fasttext_parameterss rGsave_facebook_modelrxs9B1?{[[ M$$UD2H(SSSSSrHceZdZdeffd ZefdZfdZdZfdZ fdZ dfd Z fd Z dd Z d Zd ZdZxZS)r;rctt||||||_||_||_d|_tj||f||_ d|_ d|_ dS)aVectors and vocab for :class:`~gensim.models.fasttext.FastText`. Implements significant parts of the FastText algorithm. For example, the :func:`word_vec` calculates vectors for out-of-vocabulary (OOV) entities. FastText achieves this by keeping vectors for ngrams: adding the vectors for the ngrams of an entity yields the vector for the entity. Similar to a hashmap, this class keeps a fixed number of buckets, and maps all ngrams to buckets using a hash function. Parameters ---------- vector_size : int The dimensionality of all vectors. min_n : int The minimum number of characters in an ngram max_n : int The maximum number of characters in an ngram bucket : int The number of buckets. count : int, optional If provided, vectors will be pre-allocated for at least this many vectors. (Otherwise they can be added later.) dtype : type, optional Vector dimensions will default to `np.float32` (AKA `REAL` in some Gensim code) unless another type is provided here. Attributes ---------- vectors_vocab : np.array Each row corresponds to a vector for an entity in the vocabulary. Columns correspond to vector dimensions. When embedded in a full FastText model, these are the full-word-token vectors updated by training, whereas the inherited vectors are the actual per-word vectors synthesized from the full-word-token and all subword (ngram) vectors. vectors_ngrams : np.array A vector for each ngram across all entities in the vocabulary. Each row is a vector that corresponds to a bucket. Columns correspond to vector dimensions. buckets_word : list of np.array For each key (by its index), report bucket slots their subwords map to. )r countrNrT) r@r;rArCrDrEr\r^zeros vectors_vocabrcompatible_hash)rBr rCrDrErrrFs rGrAzFastTextKeyedVectors.__init__s}\ "D))22{RW_d2eee    Xuk&:%HHH"#rHc Ftt|j|fi|S)aLoad a previously saved `FastTextKeyedVectors` model. Parameters ---------- fname : str Path to the saved file. Returns ------- :class:`~gensim.models.fasttext.FastTextKeyedVectors` Loaded model. See Also -------- :meth:`~gensim.models.fasttext.FastTextKeyedVectors.save` Save :class:`~gensim.models.fasttext.FastTextKeyedVectors` model. )r@r;r7)rfname_or_handlerwrFs rGr7zFastTextKeyedVectors.loads+(5u)3//4_OOOOOrHc|tt|j|i|t|tst dt |zt |dr |jdurt dt |ds+t |drtdt|_ t |d s+t |d rtdt|_ t|j j dkrtdt|_ t|j j dkrtdt|_ t |d r|js|t |d r|j|d Sd S)rz;Loaded object of type %s, not expected FastTextKeyedVectorsrFzPre-gensim-3.8.x fastText models with nonstandard hashing are no longer compatible. Loading your old model into gensim-3.8.3 & re-saving may create a model compatible with gensim 4.x.r>rrrr?rr\rKN)r@r;r isinstance TypeErrortyperrrr=r>r?rJrr\recalc_char_ngram_bucketsrKrUrs rGrz#FastTextKeyedVectors._load_specialss8"D))8$I&III$ 455 hY\`ae\f\ffgg gt.// 43G53P v t233 ;o8V8V ;'+AT':':':D $t344 <GW9X9X <(,Qd(;(;(;D % t'- . . 2 ;'+AT':':':D $ t(. / /! 3 <(,Qd(;(;(;D %t^,, -D4E -  * * , , ,tY'' "4< "    ! ! ! ! ! " "rHc.|jdkr ||jvSdS)aCheck if `word` or any character ngrams in `word` are present in the vocabulary. A vector for the word is guaranteed to exist if current method returns True. Parameters ---------- word : str Input word. Returns ------- bool True if `word` or any character ngrams in `word` are present in the vocabulary, False otherwise. Note ---- This method **always** returns True with char ngrams, because of the way FastText works. If you want to check if a word is an in-vocabulary term, use this instead: .. pycon: >>> from gensim.test.utils import datapath >>> from gensim.models import FastText >>> cap_path = datapath("crime-and-punishment.bin") >>> model = FastText.load_fasttext_format(cap_path, full_model=False) >>> 'steamtrain' in model.wv.key_to_index # If False, is an OOV term False rT)rEr`)rBrks rG __contains__z!FastTextKeyedVectors.__contains__s&< ;!  4,, ,4rHcHtt|j|i|dS)zSave object. Parameters ---------- fname : str Path to the output file. See Also -------- :meth:`~gensim.models.fasttext.FastTextKeyedVectors.load` Load object. N)r@r;rrs rGrzFastTextKeyedVectors.save&s. /"D)).??????rHc t|ddg}tt||||||||S)zCArrange any special handling for the gensim.utils.SaveLoad protocolr\rK)setunionr@r;_save_specials) rBfname separately sep_limitignorepickle_protocolcompresssubnamerFs rGrz#FastTextKeyedVectors._save_specials6sXV""NI#@AA)400?? :y&/8WVV VrHFc:||jvr*tt|||S|jdkrt dt j|jj dt j }|j}t||j |j |j}t|dkrtd||S|D] }|||z }|r"|t j|z S|t|z S)aGet `word` representations in vector space, as a 1D numpy array. Parameters ---------- word : str Input word. norm : bool, optional If True, resulting vector will be L2-normalized (unit Euclidean length). Returns ------- numpy.ndarray Vector representation of `word`. Raises ------ KeyError If word and all its ngrams not in vocabulary. )normrz3cannot calculate vector for OOV word without ngramsrrz=could not extract any ngrams from %r, returning origin vector)r`r@r; get_vectorrEKeyErrorr^rrrrrarCrDrJrdrlinalgr)rBrkrword_vec ngram_weights ngram_hashesnhrFs rGrzFastTextKeyedVectors.get_vector=s* 4$ $ 4-t44??4?PP P [A  4PQQ Qx 3 9! !>!> ?#dFWBXBX XD  F!FS):%;%;; !233  GAt#%8dj$*dkJJi$ $ $ D a   rHr)r)rrrr=rArr7rrrrrrrrrUrrrs@rGr;r;sP@A5$5$5$5$5$5$nPPPP[P*"""""0!!!F@@@@@ VVVVV-4-4-4-4-4-4^KKKKK"    8666&       rHr;c|j\}}d|z d|z }}|||||ft}t ||gS)zz_unpack..s@ G G GVaQ G G G GY G G G G GAq G G GrHc(i|]\}}|k ||Srrr)s rGr-z_unpack..s(III&1a!y.IAIIIrH)rr^randomr0r& concatenaterr) rnum_rows hash2indexr0fill more_dimsrand_objswapr+rr,s @rG_unpackr7s2LGI H i  A9 d 8i/ : : NAvI)=>? @ @ H G G Gz//11 G G GDKKIIIIJ$4$4$6$6IIIJJJ 1Av q!fI1a& HrHc(|tztkSro)_MB_MASK _MB_START)bs rG_is_utf8_continuer>*s x<9 $$rHcFt|||}fd|D}|S)azCalculate the ngrams of the word and hash them. Parameters ---------- word : str The word to calculate ngram hashes for. minn : int Minimum ngram length maxn : int Maximum ngram length num_buckets : int The number of buckets Returns ------- A list of hashes (integers), one per each detected ngram. c4g|]}t|zSr)r)r*nris rG z#ft_ngram_hashes..Bs& E E EmA, E E ErH)r)rkrrriencoded_ngramsrls ` rGrara.s4&*$d;;N E E E En E E EF MrH) keyedvectors)r)rT)rrr)rN)3rloggingnumpyr^rrrr=gensim.models._fasttext_binrgensim.models.word2vecrgensim.models.keyedvectorsrrr gensim.utilsr gensim.models.fasttext_innerr r r rr!gensim.models.fasttext_corpusfilerr ImportError NO_CYTHON getLoggerrrdrSaveLoadrrrrrrrrr;r&r7r;r<r>ra gensim.modelsrDrrHrGrRsll\//////////""""++++++AAAAAAAA###### SRRRRRRRR /  8 $ $uuuuuxuuup #####EN### BBBBBBBB>K>K>K>KB6666rAAAAH   @"T"T"T"TJm m m m m <m m m ` J J J J b   %%%2'&&&&&$8 !!!s A A