U Lep @s2dZddlZddlZddlmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZeddddddddgZGdddZGdddZGdddZ GdddZ!Gddde!Z"d d!Z#e$d"kr e#dddddgZ%dS)#a  This module brings together a variety of NLTK functionality for text analysis, and provides simple, interactive interfaces. Functionality includes: concordancing, collocation discovery, regular expression search over tokenized strings, and distributional similarity. N)Counter defaultdict namedtuple)reduce)log)BigramCollocationFinder)MLE)padded_everygram_pipeline)BigramAssocMeasures f_measure)ConditionalFreqDist)FreqDist) sent_tokenize)LazyConcatenation tokenwrapConcordanceLineleftqueryrightoffset left_print right_printlinec@sTeZdZdZeddZddddfddZd d Zd d ZdddZ dddZ dS) ContextIndexa A bidirectional index between words and their 'contexts' in a text. The context of a word is usually defined to be the words that occur in a fixed window around the word; but other definitions may also be used by providing a custom context function. cCsH|dkr||dnd}|t|dkr<||dnd}||fS)z;One left token and one right token, normalized to lowercaser*START**END*)lowerlen)tokensirrr!h/mounts/lovelace/software/anaconda3/envs/qiime2-amplicon-2024.2/lib/python3.8/site-packages/nltk/text.py_default_context.s$zContextIndex._default_contextNcCs|SNr!xr!r!r"5zContextIndex.csv|__|r|_nj_r6fddDtfddtD_tfddtD_dS)Ncsg|]}|r|qSr!r!).0t)filterr!r" =sz)ContextIndex.__init__..c3s(|] \}}||fVqdSr$)_key _context_funcr)r wselfrr!r" >sz(ContextIndex.__init__..c3s(|] \}}||fVqdSr$)r.r-r/r1r!r"r3As)r-_tokensr.r#CFD enumerate_word_to_contexts_context_to_words)r2rZ context_funcr+keyr!)r+r2rr"__init__5s zContextIndex.__init__cCs|jS)zw :rtype: list(str) :return: The document that this context index was created from. r4r2r!r!r"rEszContextIndex.tokenscCsF||}t|j|}i}|jD]\}}t|t|||<q&|S)z Return a dictionary mapping from words to 'similarity scores,' indicating how often these two words occur in the same context. )r-setr7itemsr )r2wordZ word_contextsscoresr0Z w_contextsr!r!r"word_similarity_dictMs  z!ContextIndex.word_similarity_dictcCsvtt}|j||D]D}|j|D]4}||kr&|||j|||j||7<q&qt||jddd|S)NT)r9reverse)rintr7r-r8sortedget)r2r?nr@cr0r!r!r" similar_words\s zContextIndex.similar_wordsFcsfddDfddDfddttD}ttj|rf|rftddn&sptStfddD}|Sd S) a Find contexts where the specified words can all appear; and return a frequency distribution mapping each context to the number of times that context was used. :param words: The words used to seed the similarity search :type words: str :param fail_on_unknown: If true, then raise a value error if any of the given words do not occur at all in the index. csg|]}|qSr!)r-r)r0r<r!r"r,qsz0ContextIndex.common_contexts..csg|]}tj|qSr!)r=r7rJr<r!r"r,rscsg|]}|s|qSr!r!)r)r )contextswordsr!r"r,ssz%The following word(s) were not found: c3s*|]"}j|D]}|kr|VqqdSr$)r7r)r0rH)commonr2r!r"r3{s  z/ContextIndex.common_contexts..N)rangerrr= intersection ValueErrorjoinr )r2rLZfail_on_unknownemptyfdr!)rOrKr2rLr"common_contextsfs  zContextIndex.common_contexts)rB)F) __name__ __module__ __qualname____doc__ staticmethodr#r:rrArIrVr!r!r!r"r&s  rc@sLeZdZdZddfddZddZdd Zd d Zdd dZdddZ dS)ConcordanceIndexzs An index that can be used to look up the offset locations at which a given word occurs in a document. cCs|Sr$r!r%r!r!r"r'r(zConcordanceIndex.cCsF||_||_tt|_t|D]"\}}||}|j||qdS)a Construct a new concordance index. :param tokens: The document (list of tokens) that this concordance index was created from. This list can be used to access the context of a given word occurrence. :param key: A function that maps each token to a normalized version that will be used as a key in the index. E.g., if you use ``key=lambda s:s.lower()``, then the index will be case-insensitive. N)r4r-rlist_offsetsr6append)r2rr9indexr?r!r!r"r:s   zConcordanceIndex.__init__cCs|jS)z{ :rtype: list(str) :return: The document that this concordance index was created from. r;r<r!r!r"rszConcordanceIndex.tokenscCs||}|j|S)z :rtype: list(int) :return: A list of the offset positions at which the given word occurs. If a key function was specified for the index, then given word's key will be looked up. )r-r^r2r?r!r!r"offsetss zConcordanceIndex.offsetscCsdt|jt|jfS)Nz+)rr4r^r<r!r!r"__repr__szConcordanceIndex.__repr__Pc s@t|tr|}n|g}|td|dd}|d}g}||d}t|ddD].\}fdd||D}t||}qZ|r<|D]d|jt|} |jt d|} |jt||} d| | d} d| d|} d| | | g}t | | | | | |}| |q|S) z Find all concordance lines given the query word. Provided with a list of words, these will be found as a phrase. rMrrNcsh|]}|dqS)rr!)r)rr r!r" sz4ConcordanceIndex.find_concordance..) isinstancer]rrSrbr6rErQr4maxrr_)r2r?widthphraseZ half_widthcontextconcordance_listrbZ word_offsetsZ query_wordZ left_contextZ right_contextrrZ line_printconcordance_liner!rgr"find_concordances:  z!ConcordanceIndex.find_concordancecCsl|j||d}|stdnLt|t|}td|dt|dt|d|D]\}}t|jqTdS)a Print concordance lines given the query word. :param word: The target word or phrase (a list of strings) :type word: str or list :param lines: The number of lines to display (default=25) :type lines: int :param width: The width of each line, in characters (default=80) :type width: int :param save: The option to save the concordance. :type save: bool )rkz no matchesz Displaying z of z matches:N)rpprintminrr6r)r2r?rklinesrnr ror!r!r"print_concordances  z"ConcordanceIndex.print_concordanceN)rd)rdrq) rWrXrYrZr:rrbrcrprur!r!r!r"r\s  ,r\c@s eZdZdZddZddZdS) TokenSearchera A class that makes it easier to use regular expressions to search over tokenized strings. The tokenized string is converted to a string where tokens are marked with angle brackets -- e.g., ``''``. The regular expression passed to the ``findall()`` method is modified to treat angle brackets as non-capturing parentheses, in addition to matching the token boundaries; and to have ``'.'`` not match the angle brackets. cCsddd|D|_dS)Ncss|]}d|dVqdS)<>Nr!rJr!r!r"r3sz)TokenSearcher.__init__..)rS_rawr1r!r!r"r:szTokenSearcher.__init__cCs~tdd|}tdd|}tdd|}tdd|}t||j}|D] }|dsJ|drJtd qJd d |D}|S) a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.text import TokenSearcher >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str z\srwrxz(?:<(?:ryz)>)z (?]z$Bad regexp for TokenSearcher.findallcSsg|]}|dddqS)rz><splitr)hr!r!r"r,0sz)TokenSearcher.findall..)resubfindallrz startswithendswithrR)r2regexphitsrr!r!r"r s zTokenSearcher.findallN)rWrXrYrZr:rr!r!r!r"rvs rvc@seZdZdZdZd6ddZddZdd Zd7d d Zd8ddZ d9ddZ d:ddZ ddZ ddZ ddZd;ddZdd'd(Zd)d*Zd+d,Zd-d.Zed/Zd0d1Zd2d3Zd4d5ZdS)?Texta A wrapper around a sequence of simple (string) tokens, which is intended to support initial exploration of texts (via the interactive console). Its methods perform a variety of analyses on the text's contexts (e.g., counting, concordancing, collocation discovery), and display the results. If you wish to write a program which makes use of these analyses, then you should bypass the ``Text`` class, and use the appropriate analysis function or class directly instead. A ``Text`` is typically initialized from a given document or corpus. E.g.: >>> import nltk.corpus >>> from nltk.text import Text >>> moby = Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt')) TNcCs|jrt|}||_|r ||_ndd|ddkrb|ddd}ddd|d|D|_n"ddd|dd Dd |_dS) zv Create a Text object. :param tokens: The source text. :type tokens: sequence of str ]NrBrMcss|]}t|VqdSr$strr)tokr!r!r"r3]sz Text.__init__..rcss|]}t|VqdSr$rrr!r!r"r3_sz...) _COPY_TOKENSr]rnamer`rS)r2rrendr!r!r"r:Ns z Text.__init__cCs |j|Sr$)r)r2r r!r!r" __getitem__eszText.__getitem__cCs t|jSr$)rrr<r!r!r"__len__hsz Text.__len__OrqcCs.d|jkrt|jddd|_|j|||S)a Prints a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` _concordance_indexcSs|Sr$rsr!r!r"r'r(z"Text.concordance..r9)__dict__r\rrrur2r?rkrtr!r!r" concordanceos  zText.concordancecCs4d|jkrt|jddd|_|j||d|S)a Generate a concordance for ``word`` with the specified context window. Word matching is not case-sensitive. :param word: The target word or phrase (a list of strings) :type word: str or list :param width: The width of each line, in characters (default=80) :type width: int :param lines: The number of lines to display (default=25) :type lines: int :seealso: ``ConcordanceIndex`` rcSs|Sr$rrr!r!r"r'r(z'Text.concordance_list..rN)rr\rrrprr!r!r"rns  zText.concordance_listrBrecsd|jkr|j|kr|j|ks||_||_ddlm}|dt|j|}| d| fddt }t | |j||_|jS)a Return collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocation_list()[:2] [('United', 'States'), ('fellow', 'citizens')] :param num: The maximum number of collocations to return. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int :rtype: list(tuple(str, str)) _collocationsr) stopwordsenglishrecst|dkp|kS)N)rr)r0Z ignored_wordsr!r"r'r(z'Text.collocation_list..)rZ_numZ _window_size nltk.corpusrrLrZ from_wordsrZapply_freq_filterZapply_word_filterr r]ZnbestZlikelihood_ratior)r2num window_sizerfinderZbigram_measuresr!rr"collocation_lists$    zText.collocation_listcCs*dd|||D}tt|dddS)a Print collocations derived from the text, ignoring stopwords. >>> from nltk.book import text4 >>> text4.collocations() # doctest: +NORMALIZE_WHITESPACE United States; fellow citizens; years ago; four years; Federal Government; General Government; American people; Vice President; God bless; Chief Justice; one another; fellow Americans; Old World; Almighty God; Fellow citizens; Chief Magistrate; every citizen; Indian tribes; public debt; foreign nations :param num: The maximum number of collocations to print. :type num: int :param window_size: The number of tokens spanned by a collocation (default=2) :type window_size: int cSsg|]\}}|d|qSrMr!r)Zw1Zw2r!r!r"r,sz%Text.collocations..; ) separatorN)rrrr)r2rrZcollocation_stringsr!r!r" collocationss zText.collocationscCs |j|S)zJ Count the number of times this word appears in the text. )rcountrar!r!r"rsz Text.countcCs |j|S)zQ Find the index of the first occurrence of the word in the text. )rr`rar!r!r"r`sz Text.indexcCstdSr$)NotImplementedError)r2methodr!r!r" readabilityszText.readabilitycsd|jkr$t|jddddd|_|jjkrttfddD}dd | |D}t t |nt d d S) a~ Distributional similarity: find other words which appear in the same contexts as the specified word; list most similar words first. :param word: The word used to seed the similarity search :type word: str :param num: The number of words to generate (default=20) :type num: int :seealso: ContextIndex.similar_words() _word_context_indexcSs|Sr$)isalphar%r!r!r"r'r(zText.similar..cSs|Sr$rrr!r!r"r'r()r+r9c3s0|](}|D]}|kr|ks|VqqdSr$r!rNrKZwcir?r!r"r3s  zText.similar..cSsg|] \}}|qSr!r!r)r0_r!r!r"r,sz Text.similar..z No matchesN) rrrrrr7Z conditionsr=r most_commonrrr)r2r?rrUrLr!rr"similars   z Text.similarc Csd|jkrt|jddd|_zJ|j|d}|s.rTzNo common contexts were foundcSsg|] \}}|qSr!r!rr!r!r"r,sz(Text.common_contexts..css|]\}}|d|VqdS)rNr!rr!r!r"r3sz'Text.common_contexts..N) rrrrrVrrrrrR)r2rLrrUZranked_contextser!r!r"rVs  zText.common_contextscCsddlm}|||dS)z Produce a plot showing the distribution of the words through the text. Requires pylab to be installed. :param words: The words to be plotted :type words: list(str) :seealso: nltk.draw.dispersion_plot() r)dispersion_plotN)Z nltk.drawr)r2rLrr!r!r"r!s zText.dispersion_plotrcCs(t||\}}t|d}||||S)N)order)r rfit)r2Ztokenized_sentsrGZ train_dataZ padded_sentsmodelr!r!r"_train_default_ngram_lm.s  zText._train_default_ngram_lmd*c Csddtd|jD|_t|dsFtdtjd|j|jdd|_ g}|d ksZt d t ||krt |j j |||d D](\}}|d krq||d krq||q||d7}qZ|rd|dnd}|t|d|}t||S)a Print random text, generated using a trigram language model. See also `help(nltk.lm)`. :param length: The length of text to generate (default=100) :type length: int :param text_seed: Generation can be conditioned on preceding context. :type text_seed: list(str) :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. (default=42) :type random_seed: int cSsg|]}|dqSrr|)r)sentr!r!r"r,Dsz!Text.generate..rM_trigram_modelzBuilding ngram index...)filer)rGrz!The `length` must be more than 0.) text_seed random_seedzzrrwN)rrSrZ_tokenized_sentshasattrrrsysstderrrrAssertionErrorrr6generater_r) r2lengthrrZgenerated_tokensidxtokenprefixZ output_strr!r!r"r4s:     z Text.generatecGs|j|S)zc See documentation for FreqDist.plot() :seealso: nltk.prob.FreqDist.plot() )vocabplot)r2argsr!r!r"rbsz Text.plotcCsd|jkrt||_|jS)z. :seealso: nltk.prob.FreqDist _vocab)rr rr<r!r!r"ris  z Text.vocabcCs@d|jkrt||_|j|}dd|D}tt|ddS)a Find instances of the regular expression in the text. The text is a list of tokens, and a regexp pattern to match a single token must be surrounded by angle brackets. E.g. >>> from nltk.book import text1, text5, text9 >>> text5.findall("<.*><.*>") you rule bro; telling you bro; u twizted bro >>> text1.findall("(<.*>)") monied; nervous; dangerous; white; white; white; pious; queer; good; mature; white; Cape; great; wise; wise; butterless; white; fiendish; pale; furious; better; certain; complete; dismasted; younger; brave; brave; brave; brave >>> text9.findall("{3,}") thread through those; the thought that; that the thing; the thing that; that that thing; through these than through; them that the; through the thick; them that they; thought that the :param regexp: A regular expression :type regexp: str _token_searchercSsg|]}d|qSr)rSr~r!r!r"r,sz Text.findall..rN)rrvrrrrr)r2rrr!r!r"rrs    z Text.findallz \w+|[\.\!\?]cCs|d}|dkr*|j||s*|d8}q|dkr:||nd}|d}|t|krl|j||sl|d7}qF|t|kr||nd}||fS)z One left & one right token, both case-normalized. Skip over non-sentence-final punctuation. Used by the ``ContextIndex`` that is created for ``similar()`` and ``common_contexts()``. rrrr) _CONTEXT_REmatchr)r2rr jrrr!r!r"_contexts  z Text._contextcCs d|jSNz rr<r!r!r"__str__sz Text.__str__cCs d|jSrrr<r!r!r"rcsz Text.__repr__)N)rrq)rrq)rBre)rBre)rB)rB)r)rNr)rWrXrYrZrr:rrrrnrrrr`rrrVrrrrrrrcompilerrrrcr!r!r!r"r4s0    #  "   . " rc@s0eZdZdZddZddZddZdd Zd S) TextCollectiona;A collection of texts, which can be loaded with list of texts, or with a corpus consisting of one or more texts, and which supports counting, concordancing, collocation discovery, etc. Initialize a TextCollection as follows: >>> import nltk.corpus >>> from nltk.text import TextCollection >>> from nltk.book import text1, text2, text3 >>> gutenberg = TextCollection(nltk.corpus.gutenberg) >>> mytexts = TextCollection([text1, text2, text3]) Iterating over a TextCollection produces all the tokens of all the texts in order. cs@tdr fddD|_t|ti|_dS)NrLcsg|]}|qSr!)rL)r)fsourcer!r"r,sz+TextCollection.__init__..)rZfileids_textsrr:r _idf_cache)r2rr!rr"r:s  zTextCollection.__init__cCs||t|S)z"The frequency of the term in text.)rrr2termtextr!r!r"tfszTextCollection.tfcsj|j}|dkrftfdd|jD}t|jdkrBtd|rXtt|j|nd}||j<|S)zThe number of texts in the corpus divided by the number of texts that the term appears in. If a term does not appear in the corpus, 0.0 is returned.Ncsg|]}|krdqS)Tr!)r)rrr!r"r,sz&TextCollection.idf..rz+IDF undefined for empty document collectiong)rrFrrrRr)r2ridfmatchesr!rr"rs  zTextCollection.idfcCs|||||Sr$)rrrr!r!r"tf_idfszTextCollection.tf_idfN)rWrXrYrZr:rrrr!r!r!r"rs rcCsddlm}t|jdd}t|ttd|dttd|dttd|ttd|dd d d gttd | d ttdtd|dtd|ddtd| ddS)Nr)brownnews) categoriesz Concordance:zDistributionally similar words:z Collocations:zDispersion plot:reportZsaidZ announcedzVocabulary plot:2z Indexing:ztext[3]:rz text[3:5]:ztext.vocab()['news']:) rrrrLrrrrrrrr)rrr!r!r"demos.    r__main__)&rZrr collectionsrrr functoolsrmathrZnltk.collocationsrZnltk.lmrZnltk.lm.preprocessingr Z nltk.metricsr r Znltk.probabilityr r5r Z nltk.tokenizerZ nltk.utilrrrrr\rvrrrrW__all__r!r!r!r" sD        [{8.