ctdZddlZddlZddlmZmZmZejeZ Gddej Z dS)aThis module allows simple Bag of Words (BoW) represented corpus to be transformed into log entropy space. It implements Log Entropy Model that produces entropy-weighted logarithmic term frequency representation. Empirical study by Lee et al. 2015 [1]_ suggests log entropy-weighted model yields better results among other forms of representation. References ---------- .. [1] Lee et al. 2005. An Empirical Evaluation of Models of Text Document Similarity. https://escholarship.org/uc/item/48g155nq N) interfacesmatutilsutilsc,eZdZdZddZdZdZdZdS) LogEntropyModelaObjects of this class realize the transformation between word-document co-occurrence matrix (int) into a locally/globally weighted matrix (positive floats). This is done by a log entropy normalization, optionally normalizing the resulting documents to unit length. The following formulas explain how o compute the log entropy weight for term :math:`i` in document :math:`j`: .. math:: local\_weight_{i,j} = log(frequency_{i,j} + 1) P_{i,j} = \frac{frequency_{i,j}}{\sum_j frequency_{i,j}} global\_weight_i = 1 + \frac{\sum_j P_{i,j} * log(P_{i,j})}{log(number\_of\_documents + 1)} final\_weight_{i,j} = local\_weight_{i,j} * global\_weight_i Examples -------- .. sourcecode:: pycon >>> from gensim.models import LogEntropyModel >>> from gensim.test.utils import common_texts >>> from gensim.corpora import Dictionary >>> >>> dct = Dictionary(common_texts) # fit dictionary >>> corpus = [dct.doc2bow(row) for row in common_texts] # convert to BoW format >>> model = LogEntropyModel(corpus) # fit model >>> vector = model[corpus[1]] # apply model to document Tcp||_d|_d|_i|_|||dSdS)a4 Parameters ---------- corpus : iterable of iterable of (int, int) Input corpus in BoW format. normalize : bool, optional If True, the resulted log entropy weighted vector will be normalized to length of 1, If False - do nothing. rN) normalizen_docsn_wordsentr initialize)selfcorpusr s >lib/python3.11/site-packages/gensim/models/logentropy_model.py__init__zLogEntropyModel.__init__;sJ#    $ OOF # # # # # $ $c@|jjd|jd|jdS)Nz) __class____name__r r )rs r__str__zLogEntropyModel.__str__Ns*.2n.E.E.Et{{{TXT`T`T`aarctdi}d\}}t|D]_\}}|dzdkrtd||t|z }|D]!\}}||d|z||<"`|dz|_||_td|jt||jtdd}t|D]v\}}|D]n\} } t| || z tj t| || z z} |j | d | z|j | <ow||krtd td |j D]7} d|j | tj |jdzz z|j | <8d S) a#Calculates the global weighting for all terms in a given corpus and transforms the simple count representation into the log entropy normalized space. Parameters ---------- corpus : iterable of iterable of (int, int) Corpus is BoW format zcalculating counts)ri'rz!PROGRESS: processing document #%iz]calculating global log entropy weights for %i documents and %i features (%i matrix non-zeros)ziterating over corpusgz;LogEntropyModel doesn't support generators as training dataziterating over keysN) loggerinfo enumeratelengetr r debugfloatmathlogr ValueError) rr glob_freqglob_num_wordsdoc_nobowterm_id term_countdoc_no2keyfreqps rr zLogEntropyModel.initializeQs  ())) !&$V,, L LKFC~" I ?HHH c#hh &N'* L L#%.]]7A%>%>%K '"" Lqj %   k KY     ,---%f-- = =LGS  = = T4[[9S>1TXeDkkIVYN>Z5[5[[!%sC!8!81!< # = f  \Z[[ [ *+++9 L LC3$(4;?2K2K!KKDIcNN L Lrctj|\}}|r|Sfd|D}jrt j|}|S)a&Get log entropy representation of the input vector and/or corpus. Parameters ---------- bow : list of (int, int) Document in BoW format. Returns ------- list of (int, float) Log-entropy vector for passed `bow`. cg|]B\}}|jv|tj|dzj|zfCS)r)r r"r#r).0r)tfrs r z/LogEntropyModel.__getitem__..s`   $)# dhrAv&&w)?)?? @   r)r is_corpus_applyr runitvec)rr(r4vectors` r __getitem__zLogEntropyModel.__getitem__}s{-- 3  $;;s## #    "    > .%f--F rN)T)r __module__ __qualname____doc__rrr r8rrrrsf>$$$$&bbb*L*L*LXrr) r;loggingr"gensimrrr getLoggerrrTransformationABCrr<rrrAs   ..........  8 $ $}}}}}j2}}}}}r