c8FdZddlZddlZddlZddlmZddlmZegdZ ej dej ej zej Zej dej Zej dej Zej d ej Zej d ej Zej d ej Zej d ej Zd#dZd#dZdZdZd$dZd$dZdZdZdZdZdZeZ d%dZ!dZ"deeeeeeegZ#e#fdZ$d Z%d!Z&d"Z'dS)&aThis module contains methods for parsing and preprocessing strings. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import remove_stopwords, preprocess_string >>> remove_stopwords("Better late than never, but better never late.") u'Better late never, better late.' >>> >>> preprocess_string("Hel 9lo Wo9 rld! Th3 weather_is really g00d today, isn't it?") [u'hel', u'rld', u'weather', u'todai', u'isn'] N)utils) PorterStemmer(Qallsixjustlessbeingindeedovermoveanywayfournotownthroughusingfiftywheremillonlyfindbeforeonewhosesystemhow somewheremuchthickshowhadenoughshouldtomustwhomseeming yourselvesunderourstwohasmight thereafterlatterlydothemhisaroundthangetverydenonecannoteveryuntheyfrontduringthusnowhimnorname regardingseveral hereafterdidalwayswhodidnwhitherthissomeoneeithereachbecome thereuponsometimesidetowardsthereintwelvebecauseoftentenourdoingkmegsomebackusedupgonamelycomputerarefurtherbeyond ourselvesyetoutevenwillwhatstillforbottomminesincepleasefortyperits everythingbehinddoesvariousabovebetweenitneitherseemedeveracrossshesomehowbewefullneversixtyhoweverhere otherwisewere whereuponnowherealthoughfoundalonerealongquitefifteenbybothaboutlastwouldanythingviamanycouldthenceputagainstkeepetcamountbecameltdhenceontoorconamongalreadyco afterwardsformerlywithinseemsintootherswhilewhateverexceptdownherseveryonedoneleastanotherwhoevermoreovercouldnt throughoutanyhowyourselfthreefromherfewtogethertopthereduebeennextanyoneelevencrycall thereforeinterestthenthru themselveshundredreallysincereemptymorehimself elsewheremostlyonfireambecomingherebyamongstelsepart everywheretookgherselfformerthosehememyselfmadetwentythesewasbillcantusuntilbesides neverthelessbelowanywhereninecanwhetherofyourtowardmysay somethingand whereafterwhenevergivealmostwhereverisdescribe beforehandhereindoesnanasitselfathaveinseemwhenceieanyfillagainhasntinctherebythinnoperhapslatter meanwhilewhendetailsamewhereinbesidealsothatothertakewhichbecomesyouifnobodyunlesswhereasseethoughmayafteruponmosthereuponeightbutseriousnothingsuchwhyoffadonwherebythirdiwholenoone sometimeswellamoungstyourstheirratherwithoutsofivethefirstwithmakeoncez([%s])+z <([^>]+)>z[0-9]+z\Wz([a-z]+)([0-9]+))flagsz([0-9]+)([a-z]+)z(\s)+ctj|}dt||S)aFRemove :const:`~gensim.parsing.preprocessing.STOPWORDS` from `s`. Parameters ---------- s : str stopwords : iterable of str, optional Sequence of stopwords If None - using :const:`~gensim.parsing.preprocessing.STOPWORDS` Returns ------- str Unicode string without `stopwords`. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import remove_stopwords >>> remove_stopwords("Better late than never, but better never late.") u'Better late never, better late.'  )r to_unicodejoinremove_stopword_tokenssplit)s stopwordss z*remove_stopword_tokens..vs# @ @ @ei)? @E @ @ @ra) STOPWORDS)tokensr^s `r_r[r[cs," @ @ @ @v @ @ @@rac`tj|}td|S)a<Replace ASCII punctuation characters with spaces in `s` using :const:`~gensim.parsing.preprocessing.RE_PUNCT`. Parameters ---------- s : str Returns ------- str Unicode string without punctuation characters. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import strip_punctuation >>> strip_punctuation("A semicolon is a stronger break than a comma, but not as much as a full stop!") u'A semicolon is a stronger break than a comma but not as much as a full stop ' rX)rrYRE_PUNCTsubr]s r_strip_punctuationrnys(* A <<Q  rac`tj|}td|S)aRemove tags from `s` using :const:`~gensim.parsing.preprocessing.RE_TAGS`. Parameters ---------- s : str Returns ------- str Unicode string without tags. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import strip_tags >>> strip_tags("Hello World!") u'Hello World!' )rrYRE_TAGSrlrms r_ strip_tagsrrs(* A ;;r1  ractj|}dt||S)aRemove words with length lesser than `minsize` from `s`. Parameters ---------- s : str minsize : int, optional Returns ------- str Unicode string without short words. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import strip_short >>> strip_short("salut les amis du 59") u'salut les amis' >>> >>> strip_short("one two three four five six seven eight nine ten", minsize=5) u'three seven eight' rX)rrYrZremove_short_tokensr\)r]minsizes r_ strip_shortrws92 A 88' 7;; < <.s) ? ? ?eUw)> ?E ? ? ?rard)rirvs `r_rurus @ ? ? ?v ? ? ??rac`tj|}td|S)aRemove digits from `s` using :const:`~gensim.parsing.preprocessing.RE_NUMERIC`. Parameters ---------- s : str Returns ------- str Unicode string without digits. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import strip_numeric >>> strip_numeric("0text24gensim365test") u'textgensimtest' rp)rrY RE_NUMERICrlrms r_ strip_numericr}s(* A >>"a  rac`tj|}td|S)a6Remove non-alphabetic characters from `s` using :const:`~gensim.parsing.preprocessing.RE_NONALPHA`. Parameters ---------- s : str Returns ------- str Unicode string with alphabetic characters only. Notes ----- Word characters - alphanumeric & underscore. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import strip_non_alphanum >>> strip_non_alphanum("if-you#can%read$this&then@this#method^works") u'if you can read this then this method works' rX)rrY RE_NONALPHArlrms r_strip_non_alphanumrs(2 A ??3 " ""rac`tj|}td|S)aPRemove repeating whitespace characters (spaces, tabs, line breaks) from `s` and turns tabs & line breaks into spaces using :const:`~gensim.parsing.preprocessing.RE_WHITESPACE`. Parameters ---------- s : str Returns ------- str Unicode string without repeating in a row whitespace characters. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import strip_multiple_whitespaces >>> strip_multiple_whitespaces("salut" + '\r' + " les" + '\n' + " loulous!") u'salut les loulous!' rX)rrY RE_WHITESPACErlrms r_strip_multiple_whitespacesrs*, A   S! $ $$ractj|}td|}td|S)aAdd spaces between digits & letters in `s` using :const:`~gensim.parsing.preprocessing.RE_AL_NUM`. Parameters ---------- s : str Returns ------- str Unicode string with spaces between digits & letters. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import split_alphanum >>> split_alphanum("24.0hours7 days365 a1b2c3") u'24.0 hours 7 days 365 a 1 b 2 c 3' z\1 \2)rrY RE_AL_NUMrl RE_NUM_ALrms r_split_alphanumr,s;* A h""A ==1 % %%ractj|}tdfd|DS)aTransform `s` into lowercase and stem it. Parameters ---------- text : str Returns ------- str Unicode lowercased and porter-stemmed version of string `text`. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import stem_text >>> stem_text("While it is quite useful to be able to search a large collection of documents almost instantly.") u'while it is quit us to be abl to search a larg collect of document almost instantly.' rXc3BK|]}|VdSN)stem)rewordps r_ zstem_text..]s-::TAFF4LL::::::ra)rrYrrZr\)textrs @r_ stem_textrFsM*  D ! !DA 88::::TZZ\\::: : ::rautf8strictcRtj|||S)aLowercase `text` and convert to unicode, using :func:`gensim.utils.any2unicode`. Parameters ---------- text : str Input text. encoding : str, optional Encoding that will be used for conversion. errors : str, optional Error handling behaviour, used as parameter for `unicode` function (python2 only). Returns ------- str Unicode version of `text`. See Also -------- :func:`gensim.utils.any2unicode` Convert any string to unicode-string. )rrYlower)rencodingerrorss r_lower_to_unicodercs!.  DJJLL(F ; ;;racdtj|dDS)zSplit line by spaces, used in :class:`gensim.corpora.lowcorpus.LowCorpus`. Parameters ---------- s : str Some line. Returns ------- list of str List of tokens from `s`. cg|]}||Srdrd)rers r_rgz"split_on_space..s L L LTt LD L L LrarX)rrYstripr\rms r_split_on_spacer}s> M LU-a006688>>sCC L L LLrac*|Sr)r)xs r_rsaggiiracrtj|}|D] }||}|S)aApply list of chosen filters to `s`. Default list of filters: * :func:`~gensim.parsing.preprocessing.strip_tags`, * :func:`~gensim.parsing.preprocessing.strip_punctuation`, * :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces`, * :func:`~gensim.parsing.preprocessing.strip_numeric`, * :func:`~gensim.parsing.preprocessing.remove_stopwords`, * :func:`~gensim.parsing.preprocessing.strip_short`, * :func:`~gensim.parsing.preprocessing.stem_text`. Parameters ---------- s : str filters: list of functions, optional Returns ------- list of str Processed strings (cleaned). Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import preprocess_string >>> preprocess_string("Hel 9lo Wo9 rld! Th3 weather_is really g00d today, isn't it?") [u'hel', u'rld', u'weather', u'todai', u'isn'] >>> >>> s = "Hel 9lo Wo9 rld! Th3 weather_is really g00d today, isn't it?" >>> CUSTOM_FILTERS = [lambda x: x.lower(), strip_tags, strip_punctuation] >>> preprocess_string(s, CUSTOM_FILTERS) [u'hel', u'9lo', u'wo9', u'rld', u'th3', u'weather', u'is', u'really', u'g00d', u'today', u'isn', u't', u'it'] )rrYr\)r]filtersfs r_preprocess_stringrsAJ A  AaDD 7799racd|DS)a'Apply :const:`~gensim.parsing.preprocessing.DEFAULT_FILTERS` to the documents strings. Parameters ---------- docs : list of str Returns ------- list of list of str Processed documents split by whitespace. Examples -------- .. sourcecode:: pycon >>> from gensim.parsing.preprocessing import preprocess_documents >>> preprocess_documents(["Hel 9lo Wo9 rld!", "Th3 weather_is really g00d today, isn't it?"]) [[u'hel', u'rld'], [u'weather', u'todai', u'isn']] c,g|]}t|Srd)r)reds r_rgz(preprocess_documents..s! / / /Q a / / /rard)docss r_preprocess_documentsrs* 0 /$ / / //ractj|d5}|cdddS#1swxYwYdS)Nrb)ropenread)pathfins r_ read_filers D$  3xxzzs 7;;c>dtj|DS)Nc,g|]}t|Srd)r)refnames r_rgzread_files..s = = =Ie   = = =ra)glob)patterns r_ read_filesrs = =$)G*<*< = = ==rar)rs)rr)(__doc__rstringrgensimrgensim.parsing.porterr frozensetrhcompileescape punctuationUNICODErkrqr|rrrrr`r[rnrrrwrur}rrrrrrrDEFAULT_FILTERSrrrrrdrar_rs   ////// I   @ 2:j929V-?#@#@@"* M M "*\2: . . RZ 2: . . bj ++ BJ*"* = = = BJ*"* = = =  8RZ00 BBBB8AAAA,   42====:@@@@&!!!2###:%%%4&&&4;;;4<<<<4MMM$%6 k9"1((((V0000 >>>>>ra