c6dZddlZddlZddlmZddlZddlmZddl Z ddl m Z m Z ej eZedZedZdZd Zd ZGd d e jZGd deZGddeZeZdS)a Automatically detect common phrases -- aka multi-word expressions, word n-gram collocations -- from a stream of sentences. Inspired by: * `Mikolov, et. al: "Distributed Representations of Words and Phrases and their Compositionality" `_ * `"Normalized (Pointwise) Mutual Information in Collocation Extraction" by Gerlof Bouma `_ Examples -------- .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.models.word2vec import Text8Corpus >>> from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS >>> >>> # Create training corpus. Must be a sequence of sentences (e.g. an iterable or a generator). >>> sentences = Text8Corpus(datapath('testcorpus.txt')) >>> # Each sentence must be a list of string tokens: >>> first_sentence = next(iter(sentences)) >>> print(first_sentence[:10]) ['computer', 'human', 'interface', 'computer', 'response', 'survey', 'system', 'time', 'user', 'interface'] >>> >>> # Train a toy phrase model on our training corpus. >>> phrase_model = Phrases(sentences, min_count=1, threshold=1, connector_words=ENGLISH_CONNECTOR_WORDS) >>> >>> # Apply the trained phrases model to a new, unseen sentence. >>> new_sentence = ['trees', 'graph', 'minors'] >>> phrase_model[new_sentence] ['trees_graph', 'minors'] >>> # The toy model considered "trees graph" a single phrase => joined the two >>> # tokens into a single "phrase" token, using our selected `_` delimiter. >>> >>> # Apply the trained model to each sentence of a corpus, using the same [] syntax: >>> for sent in phrase_model[sentences]: ... pass >>> >>> # Update the model with two new sentences on the fly. >>> phrase_model.add_vocab([["hello", "world"], ["meow"]]) >>> >>> # Export the trained model = use less RAM, faster processing. Model updates no longer possible. >>> frozen_model = phrase_model.freeze() >>> # Apply the frozen model; same results as before: >>> frozen_model[new_sentence] ['trees_graph', 'minors'] >>> >>> # Save / load models. >>> frozen_model.save("/tmp/my_phrase_model.pkl") >>> model_reloaded = Phrases.load("/tmp/my_phrase_model.pkl") >>> model_reloaded[['trees', 'graph', 'minors']] # apply the reloaded model to a sentence ['trees_graph', 'minors'] N)log)getfullargspec)utils interfacesz-infz; a an the for of with without at from to in on by and or cV||z}|dkrtS||z t|z |zS)aKBigram scoring function, based on the original `Mikolov, et. al: "Distributed Representations of Words and Phrases and their Compositionality" `_. Parameters ---------- worda_count : int Number of occurrences for first word. wordb_count : int Number of occurrences for second word. bigram_count : int Number of co-occurrences for phrase "worda_wordb". len_vocab : int Size of vocabulary. min_count: int Minimum collocation count threshold. corpus_word_count : int Not used in this particular scoring technique. Returns ------- float Score for given phrase. Can be negative. Notes ----- Formula: :math:`\frac{(bigram\_count - min\_count) * len\_vocab }{ (worda\_count * wordb\_count)}`. r)NEGATIVE_INFINITYfloat) worda_count wordb_count bigram_count len_vocab min_countcorpus_word_countdenoms 5lib/python3.11/site-packages/gensim/models/phrases.pyoriginal_scorerrYs;: + %E z!  9 $e 4y @@c||kr\t|}||z }||z }||z } t|||zz t| z S#t$r tcYSwxYwtS)aCalculation NPMI score based on `"Normalized (Pointwise) Mutual Information in Colocation Extraction" by Gerlof Bouma `_. Parameters ---------- worda_count : int Number of occurrences for first word. wordb_count : int Number of occurrences for second word. bigram_count : int Number of co-occurrences for phrase "worda_wordb". len_vocab : int Not used. min_count: int Ignore all bigrams with total collected count lower than this value. corpus_word_count : int Total number of words in the corpus. Returns ------- float If bigram_count >= min_count, return the collocation score, in the range -1 to 1. Otherwise return -inf. Notes ----- Formula: :math:`\frac{ln(prop(word_a, word_b) / (prop(word_a)*prop(word_b)))}{ -ln(prop(word_a, word_b)}`, where :math:`prob(word) = \frac{word\_count}{corpus\_word\_count}` )r r ValueErrorr) r r r r rrpapbpabs r npmi_scorerr|s>y !!"344 , , , ,.. %sb2g''3s88)3 3 % % %$ $ $ $ % ! s%A A A ct|}|} t|}tj|g|}n#t$rd|fcYSwxYwt |t rd|fS||urd|fSd|fS)aCheck whether `obj` is a single document or an entire corpus. Parameters ---------- obj : object Return ------ (bool, object) 2-tuple ``(is_single_document, new_obj)`` tuple, where `new_obj` yields the same sequence as the original `obj`. Notes ----- `obj` is a single document if it is an iterable of strings. It is a corpus if it is an iterable of documents. TF)iternext itertoolschain StopIteration isinstancestr)objobj_iter temp_iterpeeks r _is_singler&s$CyyHIH~~?D6844 Sy$X~Ch #:s%9A  A cPeZdZdZdZdZdZdZdZe fdZ xZ S)_PhrasesTransformationz Abstract base class for :class:`~gensim.models.phrases.Phrases` and :class:`~gensim.models.phrases.FrozenPhrases`. c.t||_dS)N) frozensetconnector_words)selfr+s r__init__z_PhrasesTransformation.__init__s(99rc td)zScore a single phrase candidate. Returns ------- (str, float) 2-tuple of ``(delimiter-joined phrase, phrase score)`` for a phrase, or ``(None, None)`` if not a phrase. z*ABC: override this method in child classes)NotImplementedError)r,word_aword_b in_betweens rscore_candidatez&_PhrasesTransformation.score_candidates""NOOOrc#Kdg}}|D]m}||jvrD|r=||||\}}| ||fVdg}}4|dfV|D]}|dfV |g}}J|g}}O|r||g|dfVn|r|dfV|D] }|dfV dSdS)aAnalyze a sentence, concatenating any detected phrases into a single token. Parameters ---------- sentence : iterable of str Token sequence representing the sentence to be analyzed. Yields ------ (str, {float, None}) Iterate through the input sentence tokens and yield 2-tuples of: - ``(concatenated_phrase_tokens, score)`` for token sequences that form a phrase. - ``(word, None)`` if the token is not a part of a phrase. N)r+r3append)r,sentence start_tokenr2wordphrasescorews ranalyze_sentencez'_PhrasesTransformation.analyze_sentences@ #'Z  % %D4// %7$($8$8dJ$W$WMFE ;$em+++26Z *4////!+**A"#T'MMMM26Z /3BKK%%%d++++*$$$$  t# # # #  g     rct|\}}|s||Sd||DS)aConvert the input sequence of tokens ``sentence`` into a sequence of tokens where adjacent tokens are replaced by a single token if they form a bigram collocation. If `sentence` is an entire corpus (iterable of sentences rather than a single sentence), return an iterable that converts each of the corpus' sentences into phrases on the fly, one after another. Parameters ---------- sentence : {list of str, iterable of list of str} Input sentence or a stream of sentences. Return ------ {list of str, iterable of list of str} Sentence with phrase tokens joined by ``self.delimiter``, if input was a single sentence. A generator of such sentences if input was a corpus. s cg|]\}}|Sr?).0token_s r z6_PhrasesTransformation.__getitem__...sFFF(%FFFr)r&_applyr<)r,r6 is_singles r __getitem__z"_PhrasesTransformation.__getitem__sT()22 8 );;x(( (FFd&;&;H&E&EFFFFrcXi}|D]$}||D] \}}||||< %|S)aGet all unique phrases (multi-word expressions) that appear in ``sentences``, and their scores. Parameters ---------- sentences : iterable of list of str Text corpus. Returns ------- dict(str, float) Unique phrases found in ``sentences``, mapped to their scores. Example ------- .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.models.word2vec import Text8Corpus >>> from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS >>> >>> sentences = Text8Corpus(datapath('testcorpus.txt')) >>> phrases = Phrases(sentences, min_count=1, threshold=0.1, connector_words=ENGLISH_CONNECTOR_WORDS) >>> >>> for phrase, score in phrases.find_phrases(sentences).items(): ... print(phrase, score) )r<)r, sentencesresultr6r9r:s r find_phrasesz#_PhrasesTransformation.find_phrases0sW6! + +H!%!6!6x!@!@ + + +%*F6N + rc tt|j|i| t di}t t |\}}t|tr& fd|D _ n:t|tr% fd|D _ n#t$rYnwxYwt dsFt d|jt dt _t drt jt"rĉ jdkrGt d|jt d t _nr jd krGt d |jt d t$ _n t'd |jd jdt dsRt dr j _ `n3t d|jt- _t dsAt d|jt dd _t ddrt t j}t|t"std|jt5 jtdi} jD]\}} | |t#|d<| _t jt"st# jd _ S)aLoad a previously saved :class:`~gensim.models.phrases.Phrases` / :class:`~gensim.models.phrases.FrozenPhrases` model. Handles backwards compatibility from older versions which did not support pluggable scoring functions. Parameters ---------- args : object See :class:`~gensim.utils.SaveLoad.load`. kwargs : object See :class:`~gensim.utils.SaveLoad.load`. phrasegramscvi|]5\}}tj|d|d6S)utf8encodingr! delimiterjoinr@keyvalmodels r z/_PhrasesTransformation.load..isP%%% S,,S11FCCCSV%%%rcji|]/\}}tj|d|0S)rNrOrRrUs rrYz/_PhrasesTransformation.load..nsL%%% S,,S11FCCCS%%%rscoringz3older version of %s loaded without scoring functionzEsetting pluggable scoring method to original_scorer for compatibilitydefaultz;older version of %s loaded with "default" scoring parameterz;setting scoring method to original_scorer for compatibilitynpmiz8older version of %s loaded with "npmi" scoring parameterz7setting scoring method to npmi_scorer for compatibilityzfailed to load z model, unknown scoring ""r+ common_termszCloaded older version of %s, setting connector_words to an empty setrz4older version of %s loaded without corpus_word_countzFsetting corpus_word_count to 0, do not use it in your scoring functionrvocabNz6old version of %s loaded, upgrading %i words in memoryz> Iu%'' %%%%$/$5$5$7$7%%%!!Iu-- %%%%$/$5$5$7$7%%%!    D  ui(( , NNPRUR^ _ _ _ NNb c c c+EM 5) $ $ p%--- p=I- pNN#`bebnoooNN#`aaa$3EMM]f,pNN#]_b_klllNN#\]]]$/EMM$%ns|%n%n^c^k%n%n%nooou/00 4un-- 4(-(:%&&dfifrsss(1 %u122 ( NNQSVS_ ` ` ` NNc d d d&'E # 5'4 ( ( $U[))**DdC(( $ TVYVbdghmhsdtdtuuu Z[[["'+"3"3"5"5==JC7>> from gensim.test.utils import datapath >>> from gensim.models.word2vec import Text8Corpus >>> from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS >>> >>> # Load corpus and train a model. >>> sentences = Text8Corpus(datapath('testcorpus.txt')) >>> phrases = Phrases(sentences, min_count=1, threshold=1, connector_words=ENGLISH_CONNECTOR_WORDS) >>> >>> # Use the model to detect phrases in a new sentence. >>> sent = [u'trees', u'graph', u'minors'] >>> print(phrases[sent]) [u'trees_graph', u'minors'] >>> >>> # Or transform multiple sentences at once. >>> sents = [[u'trees', u'graph', u'minors'], [u'graph', u'minors']] >>> for phrase in phrases[sents]: ... print(phrase) [u'trees_graph', u'minors'] [u'graph_minors'] >>> >>> # Export a FrozenPhrases object that is more efficient but doesn't allow any more training. >>> frozen_phrases = phrases.freeze() >>> print(frozen_phrases[sent]) [u'trees_graph', u'minors'] Notes ----- The ``scoring="npmi"`` is more robust when dealing with common words that form part of common bigrams, and ranges from -1 to 1, but is slower to calculate than the default ``scoring="default"``. The default is the PMI-like scoring as described in `Mikolov, et. al: "Distributed Representations of Words and Phrases and their Compositionality" `_. To use your own custom ``scoring`` function, pass in a function with the following signature: * ``worda_count`` - number of corpus occurrences in `sentences` of the first token in the bigram being scored * ``wordb_count`` - number of corpus occurrences in `sentences` of the second token in the bigram being scored * ``bigram_count`` - number of occurrences in `sentences` of the whole bigram * ``len_vocab`` - the number of unique tokens in `sentences` * ``min_count`` - the `min_count` setting of the Phrases class * ``corpus_word_count`` - the total number of tokens (non-unique) in `sentences` The scoring function must accept all these parameters, even if it doesn't use them in its scoring. The scoring function **must be pickleable**. )r+rzmin_count should be at least 1r\z0threshold should be positive for default scoringr]rQz5threshold should be between -1 and 1 for npmi scoringzunknown scoring method string z specifiedr r r r rrcBg|]}|tdv|S)r) getargspec)r@paramr[s rrCz$Phrases.__init__.."s2```ET[I\I\]^I_<_`u```rz-scoring function missing expected parameters zCustom scoring function in z must be pickle-ableNcreatedzbuilt  in .2fsmsg)rar-rr r!rrcallabler[r thresholdmax_vocab_sizer` min_reducerS progress_perrpickleloadsdumps PickleErrorrqritime add_vocabadd_lifecycle_event) r,rHrrrrSrr[r+scoring_paramsmissingstartrqs ` rr-zPhrases.__init__suB 999 > ?=>> > > Qg2 QOPP P f  V)b. VIM VTUU U gs # # W)# W)F" W% !U'!U!U!UVVV    G   \````.```G \&  !ZQX!Z!Z[[["", "(!" r Ldl33 4 4 4 4! r r r$%p4>CZ%p%p%pqq q r  cIKKE NN9 % % %  $ $Y4aT4a4aty{{UZGZ4a4a4a4a $ b b b b b c cs 0+E2Fcld|jjt|j|j|j|jfzS)Nz;%s<%i vocab, min_count=%s, threshold=%s, max_vocab_size=%s>)rqrirkr`rrrr,s r__str__zPhrases.__str__<s5L N #S__dn ND/P   rc d\}}}i}tdt|D]\}} ||zdkr*td||t|dg} } | D]} | |vrl|| ddz|| <| It j| g| | g} || }||ddz||<| g} } n| | | |dz }t||krtj |||dz }tdt|||dz|||fS)z@Collect unigram and bigram counts from the `sentences` iterable.)r|rrQz%collecting all words and their countsrz?PROGRESS: at sentence #%i, processed %i words and %i word typesNrQzWcollected %i token types (unigram + bigrams) from a corpus of %i words and %i sentences) rgrj enumeraterkgetrrrTr5r prune_vocab)rHrrSr+r sentence_no total_wordsrr`r6r7r2r8 phrase_tokensjoined_phrase_tokens r _learn_vocabzPhrases._learn_vocabBs08, [* ;<<<%.y%9%9  !K\)Q.  Uc%jj'+BK ! !.,"'))D!"4"4q"8E$K"[(1 zTXSY(Z(Z .7nn].K.K+5:YY?RTU5V5VYZ5Z12.2BKK ,%%d+++q 5zzN* !%444a  e JJ [1_   5+--rc||||j|j|j|j\}}}|xj|z c_|jrtdt||t|j ||_ | D]+\}}|j |d|z|j|<,t|j|jkr/tj|j|j |xj dz c_ n||_td|dS)a~Update model parameters with new `sentences`. Parameters ---------- sentences : iterable of list of str Text corpus to update this model's parameters from. Example ------- .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.models.word2vec import Text8Corpus >>> from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS >>> >>> # Train a phrase detector from a text corpus. >>> sentences = Text8Corpus(datapath('testcorpus.txt')) >>> phrases = Phrases(sentences, connector_words=ENGLISH_CONNECTOR_WORDS) # train model >>> assert len(phrases.vocab) == 37 >>> >>> more_sentences = [ ... [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there'], ... [u'machine', u'learning', u'can', u'be', u'new', u'york', u'sometimes'], ... ] >>> >>> phrases.add_vocab(more_sentences) # add new sentences to model >>> assert len(phrases.vocab) == 60 )rrSrr+zmerging %i counts into %srrQz merged %sN)rrrSrr+rr`rgrjrkmaxrrdrrr)r,rHrr`rr8counts rrzPhrases.add_vocabes1F*.):): d&9T^*D>DO${{}} C C e#':>>$#:#:U#B 4  4:!44 %!$*do>>>1$DJ K&&&&&rc|j|d}|dkrdS|j|d}|dkrdS|j|g|z|gz}|j|d}|dkrdS||||t |j|j|j}||jkrdS||fS)NrNNr}) r`rrSrTr[rkrrr) r,r0r1r2 word_a_cnt word_b_cntr9 phrase_cntr:s rr3zPhrases.score_candidatesZ^^FA.. ? :Z^^FA.. ? :$$fX %:fX%EFFZ^^FA.. ? : " $*ooSWSi   DN " :u}rc t|S)a Return an object that contains the bare minimum of information while still allowing phrase detection. See :class:`~gensim.models.phrases.FrozenPhrases`. Use this "frozen model" to dramatically reduce RAM footprint if you don't plan to make any further changes to your `Phrases` model. Returns ------- :class:`~gensim.models.phrases.FrozenPhrases` Exported object that's smaller, faster, but doesn't support model updates. ) FrozenPhrasesrs rfreezezPhrases.freezesT"""rci|j}}|D]e}||j}t|dkr0||d|d|dd\}}||||<f|S)zExtract all found phrases. Returns ------ dict(str, float) Mapping between phrases and their scores. rr|rQ)r`splitrSrkr3)r,rI source_vocabrAunigramsr9r:s rexport_phraseszPhrases.export_phrasess "4: ! ' 'E{{4>22H8}}q   00!hrlHUVWYUYN[[MFE '!&v r)rirrrsrtr*r-r staticmethodrrr3rrrvrws@rryrys55!A#syy{{QcQcQcQcQcQcf    . .\ .D5'5'5'n2### rryc$eZdZdZdZdZdZdS)razMinimal state & functionality exported from a trained :class:`~gensim.models.phrases.Phrases` model. The goal of this class is to cut down memory consumption of `Phrases`, by discarding model state not strictly needed for the phrase detection task. Use this instead of `Phrases` if you do not need to update the bigram statistics with new documents any more. c x|j|_|j|_|j|_|j|_|j|_t d|tj}||_ | dd|d|dtj|z ddd S) a Parameters ---------- phrases_model : :class:`~gensim.models.phrases.Phrases` Trained phrases instance, to extract all phrases from. Notes ----- After the one-time initialization, a :class:`~gensim.models.phrases.FrozenPhrases` will be much smaller and faster than using the full :class:`~gensim.models.phrases.Phrases` model. Examples ---------- .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.models.word2vec import Text8Corpus >>> from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS >>> >>> # Load corpus and train a model. >>> sentences = Text8Corpus(datapath('testcorpus.txt')) >>> phrases = Phrases(sentences, min_count=1, threshold=1, connector_words=ENGLISH_CONNECTOR_WORDS) >>> >>> # Export a FrozenPhrases object that is more efficient but doesn't allow further training. >>> frozen_phrases = phrases.freeze() >>> print(frozen_phrases[sent]) [u'trees_graph', u'minors'] zexporting phrases from %srz exported z from rrrrN) rrrSr[r+rgrjrrrLr)r, phrases_modelrs rr-zFrozenPhrases.__init__s>'0&0&0$, ,< /??? (7799   0uD0u0u 0u0u[_[d[f[fin[n0u0u0u0u vvvvvrc`d|jjt|j|j|jfzS)Nz*%s<%i phrases, min_count=%s, threshold=%s>)rqrirkrLrrrs rrzFrozenPhrases.__str__ s1; N #S)9%:%:DNDN?   rc|j|g|z|gz}|j|t}||jkr||fSdS)Nr)rSrTrLrrr)r,r0r1r2r9r:s rr3zFrozenPhrases.score_candidates\$$fX %:fX%EFF $$V->?? 4> ! !5= zrN)rirrrsrtr-rr3r?rrrrsO'w'w'wR   rr)rtloggingrmathrrinspectrrrgensimrr getLoggerrirgr rr*rENGLISH_CONNECTOR_WORDSrrr&TransformationABCr(ryrPhraserr?rrrs 88t 000000 $$$$$$$$  8 $ $E&MM $) UWW  A A AF+!+!+!\!!!HUUUUUZ9UUUpppppp$pppf =====*===@ r