Qdp|dZddlZddlZddlmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZedgdZGddZGddZGddZ GddZ!Gdde!Z"dZ#e$dkr e#gdZ%dS)a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation tokenwrapConcordanceLine)leftqueryrightoffset left_print right_printlinecTeZdZdZedZdddfdZdZdZd d Z d d Z dS) ContextIndexa A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. c|dkr||dz nd}|t|dz kr||dznd}||fS)z;One left token and one right token, normalized to lowercaser*START**END*)lowerlen)tokensirrs )lib/python3.11/site-packages/nltk/text.py_default_contextzContextIndex._default_context.sf)*Qva!e}""$$$I)*c&kkAo)=)=q1u ##%%%7e}Nc|SNxs r$zContextIndex.5sQr&c,|__|r|_n j_rfdDt fdt D_t fdt D_dS)Nc*g|]}| |Sr)r)).0tfilters r$ z)ContextIndex.__init__..=s&555A66!995a555r&c3tK|]2\}}||fV3dSr()_key _context_funcr/r#wselfr"s r$ z(ContextIndex.__init__..>sW% % >BaTYYq\\4--fa88 9% % % % % % r&c3tK|]2\}}||fV3dSr()r5r4r6s r$r9z(ContextIndex.__init__..AsW% % >BaT   * *DIIaLL 9% % % % % % r&)r4_tokensr5r%CFD enumerate_word_to_contexts_context_to_words)r8r" context_funcr1keys`` ` r$__init__zContextIndex.__init__5s   7!-D  !%!6D   65555555F!$% % % % % FOPVFWFW% % % " " "%% % % % % FOPVFWFW% % % " " r&c|jS)zw :rtype: list(str) :return: The document that this context index was created from. r;r8s r$r"zContextIndex.tokensE |r&c||}t|j|}i}|jD]%\}}t |t|||<&|S)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r4setr>itemsr )r8word word_contextsscoresr7 w_contextss r$word_similarity_dictz!ContextIndex.word_similarity_dictMsq yyD24899 !399;; B BMAz!-ZAAF1II r&cDtt}|j||D]M}|j|D]=}||kr5||xx|j|||j||zz cc<>Nt ||jdd|S)NT)rAreverse)rintr>r4r?sortedget)r8rJnrLcr7s r$ similar_wordszContextIndex.similar_words\sS!!' $8  A+A.  991III.q1$7$:PQR:STU:VVIII  f&*d;;;BQB??r&FcrfdDfdDfdttD}ttj|r%|r#t ddstStfdD}|S)a Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. c:g|]}|Sr))r4r/r7r8s r$r2z0ContextIndex.common_contexts..qs#---!1---r&cDg|]}tj|Sr))rHr>rZs r$r2z0ContextIndex.common_contexts..rs)BBBqC.q122BBBr&c0g|]}| |Sr)r))r/r#contextswordss r$r2z0ContextIndex.common_contexts..ss&HHHaHQKHqHHHr&z%The following word(s) were not found: c3DK|]}j|D] }|v|V dSr()r>)r/r7rVcommonr8s r$r9z/ContextIndex.common_contexts..{sI$*@*C%&qF{{{{{{{r&)ranger!rrH intersection ValueErrorjoinr)r8r^fail_on_unknownemptyfdrar]s`` @@r$common_contextszContextIndex.common_contextsfs.---u---BBBBEBBBHHHHH5U#4#4HHH((33  _ DchhuooVV V ::  BIr&rO)F) __name__ __module__ __qualname____doc__ staticmethodr%rBr"rNrWrir)r&r$rr&s\ -1;;       @@@@r&rc@eZdZdZdfdZdZdZdZd dZd d Z d S)ConcordanceIndexzs An index that can be used to look up the offset locations at which a given word occurs in a document. c|Sr(r)r*s r$r,zConcordanceIndex.sQr&c||_ ||_ tt|_ t |D]:\}}||}|j||;dS)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r;r4rlist_offsetsr=append)r8r"rAindexrJs r$rBzConcordanceIndex.__init__s   D#D)) L$V,, . .KE499T??D M$  & &u - - - - . .r&c|jS)z{ :rtype: list(str) :return: The document that this concordance index was created from. rDrEs r$r"zConcordanceIndex.tokensrFr&cF||}|j|S)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r4rur8rJs r$offsetszConcordanceIndex.offsetss yy}T""r&cXdt|jt|jfzS)Nz+)r!r;rurEs r$__repr__zConcordanceIndex.__repr__s/<      @   r&Pc t|tr|}n|g}|td|z dz dz}|dz}g}||d}t |ddD]H\}fd||D}t ||}I|r|D]d|jt|z} |jtd|z } |jt|z|z} d| | d} d| d|} d| | | g}t| | | | | |}| ||S)z Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. r_rrNc h|] }|z dz  S)rr))r/rr#s r$ z4ConcordanceIndex.find_concordance..s!LLLvFQJNLLLr&) isinstancertr!rer{r=rSrcr;maxrrv)r8rJwidthphrase half_widthcontextconcordance_listr{ word_offsets query_word left_context right_contextrr line_printconcordance_liner#s @r$find_concordancez!ConcordanceIndex.find_concordances dD ! ! FFVFc#((6"2"2333a7A= 1*,,vay)) ,, A AGAtLLLLd9K9KLLLL\66w??@@GG  : : : XXdl1q3v;;3F&GHH #|C1w;,?,?!,CD $ QV_q7{-J K  XXl33ZKLLA !hh}55kzkB  XXz:{&KLL #2 !$$ !''(89999r&c<|||}|stddSt|t|}td|dt|dt |d|D]\}}t|jdS)a Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )rz no matchesz Displaying z of z matches:N)rprintminr!r=r)r8rJrlinesrr#rs r$print_concordancez"ConcordanceIndex.print_concordances 00U0CC - ,     s#34455E KKK3/?+@+@KKK L L L'01A&5&1I'J'J - -##&+,,,, - -r&N)r~)r~r) rkrlrmrnrBr"r{r}rrr)r&r$rqrqs $/;....4###   * * * * X------r&rqceZdZdZdZdZdS) TokenSearchera A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cNdd|D|_dS)Nc3&K|] }d|zdzV dS)<>Nr))r/r7s r$r9z)TokenSearcher.__init__..s*::aC!GcM::::::r&)re_raw)r8r"s r$rBzTokenSearcher.__init__s(GG::6::::: r&c~tjdd|}tjdd|}tjdd|}tjdd|}tj||j}|D];}|ds$|drt d >> from nltk.text import TokenSearcher >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srrz(?:<(?:rz)>)z (?]z$Bad regexp for TokenSearcher.findallcHg|]}|ddd S)rz><splitr/hs r$r2z)TokenSearcher.findall..0s,222!B$ d##222r&)resubfindallr startswithendswithrd)r8regexphitsrs r$rzTokenSearcher.findall s0r6**i00eV,, ff55z&$),, I IA<<$$ IC I !GHHH32T222 r&N)rkrlrmrnrBrr)r&r$rrs<;;;'''''r&rceZdZdZdZd!dZdZdZd"d Zd"d Z d#d Z d#dZ dZ dZ dZd$dZd$dZdZd%dZd&dZdZdZdZejdZdZdZd ZdS)'Texta A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNcb|jrt|}||_|r ||_dSd|ddvrK|ddd}dd|d|D|_dSdd|ddDd z|_dS) zv Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]NrOr_c34K|]}t|VdSr(strr/toks r$r9z Text.__init__..]s( C CcS C C C C C Cr&rc34K|]}t|VdSr(rrs r$r9z Text.__init__.._s( @ @cS @ @ @ @ @ @r&z...) _COPY_TOKENSrtr"namerwre)r8r"rends r$rBz Text.__init__Ns   "&\\F  IDIII F3B3K  "+##C((C C CVAcE] C C CCCDIII @ @VBQBZ @ @ @@@5HDIIIr&c|j|Sr()r")r8r#s r$ __getitem__zText.__getitem__es{1~r&c*t|jSr()r!r"rEs r$__len__z Text.__len__hs4;r&Orcd|jvrt|jd|_|j|||S)a Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexc*|Sr(r ss r$r,z"Text.concordance..17799r&rA)__dict__rqr"rrr8rJrrs r$ concordancezText.concordanceosP t} 4 4&6 !4!4'''D #&88ueLLLr&cd|jvrt|jd|_|j||d|S)a Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rc*|Sr(rrs r$r,z'Text.concordance_list..rr&rN)rrqr"rrrs r$rzText.concordance_listsW t} 4 4&6 !4!4'''D #&77eDDVeVLLr&rOrcd|jvr|j|kr |j|ks||_||_ddlm}|dt j|j|}| d| fdt}t| |j||_|jS)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocationsr) stopwordsenglishrcVt|dkp|vS)N)r!r )r7 ignored_wordss r$r,z'Text.collocation_list..s#s1vvz/WQWWYY-=Wr&)r_num _window_size nltk.corpusrr^r from_wordsr"apply_freq_filterapply_word_filterr rtnbestlikelihood_ratior)r8num window_sizerfinderbigram_measuresrs @r$collocation_listzText.collocation_lists t} , , S  ![00DI +D  . - - - - -%OOI66M,7 [QQF  $ $Q ' ' '  $ $%W%W%W%W X X X133O!% _=sCC""D !!r&cd|||D}tt|ddS)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE United States; fellow citizens; years ago; four years; Federal Government; General Government; American people; Vice President; God bless; Chief Justice; one another; fellow Americans; Old World; Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian tribes; public debt; foreign nations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int c$g|] \}}|dz|zSr_r)r/w1w2s r$r2z%Text.collocations..s1   $b"BHrM   r&; ) separatorN)rrr)r8rrcollocation_stringss r$ collocationszText.collocationssU&  (,(=(=c;(O(O    i+t<<<=====r&c6|j|S)zJ Count the number of times this word appears in the text. )r"countrzs r$rz Text.count{  &&&r&c6|j|S)zQ Find the index of the first occurrence of the word in the text. )r"rwrzs r$rwz Text.indexrr&ctr()NotImplementedError)r8methods r$ readabilityzText.readabilitys!!r&cd|jvrt|jdd|_|jjvrttfdD}d| |D}tt|dStddS) a~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexc*|Sr()isalphar*s r$r,zText.similar..saiikkr&c*|Sr(rrs r$r,zText.similar..sr&)r1rAc3FK|]}|D]}|v|k |VdSr(r))r/r7rVr]wcirJs r$r9zText.similar..sWQ==d*3 r&cg|]\}}|Sr)r)r/r7_s r$r2z Text.similar..s77741aQ777r&z No matchesN) rrr"rr r> conditionsrHr most_commonrr)r8rJrrhr^r]rs ` @@r$similarz Text.similars ! 5 5'3 $9$9?R?R(((D $ zz||&8 3>>## # #3t9~~H))B 872>>##6#6777E )E"" # # # # # ,     r&cd|jvrt|jd|_ |j|d}|st ddSd||D}t td|DdS#t$r}t |Yd}~dSd}~wwxYw) aY Find contexts where the specified words appear; list most frequent common contexts first. :param words: The words used to seed the similarity search :type words: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.common_contexts() rc*|Sr(rrs r$r,z&Text.common_contexts..rr&rTzNo common contexts were foundcg|]\}}|Sr)r)rs r$r2z(Text.common_contexts..s"E"E"EA1"E"E"Er&c3,K|]\}}|dz|zVdS)rNr)rs r$r9z'Text.common_contexts..s.LL&"bS2 LLLLLLr&N) rrr"rrirrrrd)r8r^rrhranked_contextses r$rizText.common_contextss ! 5 5'3 !4!4(((D $ )99%FFB N566666"E"E1D1D"E"E"EiLLOLLLLLMMMMM    !HHHHHHHHH s,BAB C&B;;Cc*ddlm}|||dS)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN) nltk.drawr)r8r^rs r$rzText.dispersion_plot!s. .-----e$$$$$r&rcxt||\}}t|}||||S)N)order)r r fit)r8tokenized_sentsrU train_data padded_sentsmodels r$_train_default_ngram_lmzText._train_default_ngram_lm.s<#.Ds/! ! ! $DJJsOO! ! ! r&r__trigram_modelzBuilding ngram index...)filer)rUrz!The `length` must be more than 0.) text_seed random_seedzzrrN)rrer"_tokenized_sentshasattrrsysstderrrrr!r=generatervr) r8lengthrrgenerated_tokensidxtokenprefix output_strs r$rz Text.generate4s ! ! (5chht{6K6K(L(L! ! ! t-..  +#* = = = ="&">">%#?##D zzz>zzz"##f,,'#,,i[- / / U E>>F??E ''.... 1 K"##f,,/8?)$$s**Ri(8&(ABBB  jr&c:|j|S)zc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r8argss r$r(z Text.plotbs !tzz|| $''r&cJd|jvrt||_|jS)z. :seealso: nltk.prob.FreqDist _vocab)rrr+rEs r$r'z Text.vocabis& 4= ( ("4..DK{r&cd|jvrt||_|j|}d|D}t t |ddS)a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searcherc8g|]}d|Sr)rers r$r2z Text.findall..s"*** ***r&rN)rrr-rrr)r8rrs r$rz Text.findallrsh. DM 1 1#0#6#6D #++F33**T*** id##$$$$$r&z \w+|[\.\!\?]c|dz }|dkrK|j||s+|dz}|dkr |j||+|dkr||nd}|dz}|t|krX|j||s8|dz }|t|kr |j||8|t|kr||nd}||fS)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrrr) _CONTEXT_REmatchr!)r8r"r#jrrs r$_contextz Text._contexts E1ffT-33F1I>>f FA1ffT-33F1I>>fFFvayy  E#f++ood&6&<&rrEs r$__str__z Text.__str__di''r&cd|jzSr5r6rEs r$r}z Text.__repr__r8r&r()rr)rOrrj)r)rNr)rkrlrmrnrrBrrrrrrrrwrrrirrrr(r'rrcompiler0r3r7r}r)r&r$rr4s.LIIII.   MMMM*MMMM(!"!"!"!"F>>>>0''' ''' """     D8 % % % ,,,,\(((%%%D"*_--K0((((((((r&rc*eZdZdZdZdZdZdZdS)TextCollectiona;A collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> from nltk.book import text1, text2, text3 >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. ctdr fdD|_t|t i|_dS)Nr^c:g|]}|Sr))r^)r/fsources r$r2z+TextCollection.__init__..s#@@@!fll1oo@@@r&)rfileids_textsrrBr _idf_cache)r8r@s `r$rBzTextCollection.__init__sh 67 # # A@@@@v~~/?/?@@@F  d-f55666r&cL||t|z S)z"The frequency of the term in text.)rr!r8termtexts r$tfzTextCollection.tfszz$#d))++r&c2|j}|ytfd|jD}t|jdkrt d|r$t t|j|z nd}||j<|S)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Ncg|]}|vd S)Tr))r/rGrFs r$r2z&TextCollection.idf..sHHHD44<<4<<resA 8888888888555555;;;;;;77777777777777%%%%%%''''''22222222*MMM XXXXXXXXvx-x-x-x-x-x-x-x-v55555555p~(~(~(~(~(~(~(~(D +4+4+4+4+4T+4+4+4\999< zDFFF   r&