cdZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl Z ddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZejeZGddeZdZdZ GddeZ!dS)aAuthor-topic model. This module trains the author-topic model on documents and corresponding author-document dictionaries. The training is online and is constant in memory w.r.t. the number of documents. The model is *not* constant in memory w.r.t. the number of authors. The model can be updated with additional documents after training has been completed. It is also possible to continue training on the existing data. The model is closely related to :class:`~gensim.models.ldamodel.LdaModel`. The :class:`~gensim.models.atmodel.AuthorTopicModel` class inherits :class:`~gensim.models.ldamodel.LdaModel`, and its usage is thus similar. The model was introduced by `Rosen-Zvi and co-authors: "The Author-Topic Model for Authors and Documents" `_. The model correlates the authorship information with the topics to give a better insight on the subject knowledge of an author. .. _'Online Learning for LDA' by Hoffman et al.: online-lda_ .. _online-lda: https://papers.neurips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0-Paper.pdf Example ------- .. sourcecode:: pycon >>> from gensim.models import AuthorTopicModel >>> from gensim.corpora import mmcorpus >>> from gensim.test.utils import common_dictionary, datapath, temporary_file >>> author2doc = { ... 'john': [0, 1, 2, 3, 4, 5, 6], ... 'jane': [2, 3, 4, 5, 6, 7, 8], ... 'jack': [0, 2, 4, 6, 8] ... } >>> >>> corpus = mmcorpus.MmCorpus(datapath('testcorpus.mm')) >>> >>> with temporary_file("serialized") as s_path: ... model = AuthorTopicModel( ... corpus, author2doc=author2doc, id2word=common_dictionary, num_topics=4, ... serialized=True, serialization_path=s_path ... ) ... ... model.update(corpus, author2doc) # update the author-topic model with additional documents >>> >>> # construct vectors for authors >>> author_vecs = [model.get_author_topics(author) for author in model.id2author.values()] N)chain)deepcopy)copyfile)isfile)remove)gammaln)utils)LdaModel)LdaState)dirichlet_expectationmean_absolute_difference)MmCorpusceZdZdZdZdS)AuthorTopicStatez\Encapsulate information for computation of :class:`~gensim.models.atmodel.AuthorTopicModel`.c||_tj||_tj||_d|_tj|_dS)a  Parameters ---------- eta: numpy.ndarray Dirichlet topic parameter for sparsity. lambda_shape: (int, int) Initialize topic parameters. gamma_shape: int Initialize topic parameters. rN)etanpzerossstatsgammanumdocsfloat64dtype)selfr lambda_shape gamma_shapes 5lib/python3.11/site-packages/gensim/models/atmodel.py__init__zAuthorTopicState.__init__UsAh|,, Xk**  Z N)__name__ __module__ __qualname____doc__rrrrrRs)ff     rrci}t|D]?\}}g}|D]\}}||vr|||||<@|S)aQCreate a mapping from document IDs to author IDs. Parameters ---------- corpus: iterable of list of (int, float) Corpus in BoW format. author2doc: dict of (str, list of int) Mapping of authors to documents. Returns ------- dict of (int, list of str) Document to Author mapping. ) enumerateitemsappend)corpus author2doc doc2authord_ author_idsa a_doc_idss rconstruct_doc2authorr1isz J&!!##1 &,,.. % %LAyI~ %!!!$$$" 1 rct}|D]\}}|D]}|| i}|D]@}g||<|D]$\}}||vr|||%A|S)aMake a mapping from author IDs to document IDs. Parameters ---------- doc2author: dict of (int, list of str) Mapping of document id to authors. Returns ------- dict of (str, list of int) Mapping of authors to document ids. )setr'addr()r+ authors_idsr,r0r/r*a_idss rconstruct_author2docr7s%%K"((** 9  A OOA     J (( 1 "((** ( (HAuEz (1 $$Q''' ( rceZdZdZ ddZdZdZdZdZddZ ddZ ddZ ddZ d dZ d!dZd!dZd!dZd!dZdS)"AuthorTopicModelzWThe constructor estimates the author-topic model parameters based on a training corpus.Nd2?? symmetric MbP?F{Gz?crtj|_d}d|_d|_||_||jt d|jMtdtj ||_t|j|_ nNt|jdkr/dt|jz|_ nd|_ |j dkrt dtd|j i|_i|_||_||_d|_||_| |_| |_||_d|_d|_||_| |_||_i|_i|_||_ |r|st d |r|rtC|r Jd ||_"|#|$| d \|_%|_&|j%j'|jfks+Jd tQ|j%j'|jfz|$| d \|_)|_*|j)j'|j fksS|j)j'|j|j fks7JdtQ|j)j'|j |j|j fztj+||_,||_-||_.t_|j)|j|j f|j|jf|_0|j,1dd|j|j f|j0_2tj3ti|j0j2|_5|(||&|jdu}|6||||dSdSdS)a Parameters ---------- corpus : iterable of list of (int, float), optional Corpus in BoW format num_topics : int, optional Number of topics to be extracted from the training corpus. id2word : :class:`~gensim.corpora.dictionary.Dictionary`, optional A mapping from word ids (integers) to words (strings). author2doc : dict of (str, list of int), optional A dictionary where keys are the names of authors and values are lists of document IDs that the author contributes to. doc2author : dict of (int, list of str), optional A dictionary where the keys are document IDs and the values are lists of author names. chunksize : int, optional Controls the size of the mini-batches. passes : int, optional Number of times the model makes a pass over the entire training data. iterations : int, optional Maximum number of times the model loops over each document. decay : float, optional A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten when each new document is examined. Corresponds to :math:`\kappa` from `'Online Learning for LDA' by Hoffman et al.`_ offset : float, optional Hyper-parameter that controls how much we will slow down the first steps the first few iterations. Corresponds to :math:`\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_ alpha : {float, numpy.ndarray of float, list of float, str}, optional A-priori belief on document-topic distribution, this can be: * scalar for a symmetric prior over document-topic distribution, * 1D array of length equal to num_topics to denote an asymmetric user defined prior for each topic. Alternatively default prior selecting strategies can be employed by supplying a string: * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`, * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`, * 'auto': Learns an asymmetric prior from the corpus (not available if `distributed==True`). eta : {float, numpy.ndarray of float, list of float, str}, optional A-priori belief on topic-word distribution, this can be: * scalar for a symmetric prior over topic-word distribution, * 1D array of length equal to num_words to denote an asymmetric user defined prior for each word, * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination. Alternatively default prior selecting strategies can be employed by supplying a string: * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`, * 'auto': Learns an asymmetric prior from the corpus. update_every : int, optional Make updates in topic probability for latest mini-batch. eval_every : int, optional Calculate and estimate log perplexity for latest mini-batch. gamma_threshold : float, optional Threshold value of gamma(topic difference between consecutive two topics) until which the iterations continue. serialized : bool, optional Indicates whether the input corpora to the model are simple lists or saved to the hard-drive. serialization_path : str, optional Must be set to a filepath, if `serialized = True` is used. minimum_probability : float, optional Controls filtering the topics returned for a document (bow). random_state : {int, numpy.random.RandomState}, optional Set the state of the random number generator inside the author-topic model. FNr<zYat least one of corpus/id2word must be specified, to establish input space dimensionalityzHno word id mapping provided; initializing from corpus, assuming identityrzIcannot compute the author-topic model over an empty collection (no terms)z Vocabulary consists of %d words.z{If serialized corpora are used, a the path to a folder where the corpus should be saved must be provided (serialized_path).zpA file already exists at the serialization_path path; choose a different serialization_path, or delete the file.alphaz6Invalid alpha shape. Got shape %s, but expected (%d, )rzAInvalid eta shape. Got shape %s, but expected (%d, 1) or (%d, %d)Y@rC)chunks_as_numpy)7rrr dispatcher numworkersid2word ValueErrorloggerwarningr dict_from_corpuslen num_termsmaxkeysinfor*r+ distributed num_topics num_authors chunksizedecayoffsetminimum_probability num_updates total_docspasses update_every eval_every author2id id2author serializedrserialization_pathinit_empty_corpusinit_dir_priorrEoptimize_alphashapestrr optimize_etaget_random_state random_state iterationsgamma_thresholdrstaterrexpr expElogbetaupdate)rr)rUrJr*r+rWr]rlrXrYrErr^r_rmrbrcrZrkrT use_numpys rrzAuthorTopicModel.__init__sNZ    dl k  <  NNe f f f 1&99DL ..DNN    " T\%6%6%8%8!9!99DNNDN >Q  jhii i 6GGG&$"  #6  ($$  0 W   M, M011 M MM M M M#5    *.*=*=eW*M*M' D'zDO#55 p p DDJL\H]H]_c_nGo o p p p'+&9&9#u&E&E#$#4>"33 tx~$/[_[iIj7j  O  $.$/4> R S   "2<@@%.&dh$.0QTXTdfjfuSvww  -33D)doW[WeEfgg 6"7 8I"J"JKK  S: SZ St3I KK J K R R R R R S S S Src p|jjd|jd|jd|jd|jd|jd S)zGet a string representation of object. Returns ------- str String representation of current instance. z ) __class__r rPrUrVrXrWrs r__str__zAuthorTopicModel.__str__EsQ^ $ $ $dnnndoootGWGWGWY]YcYcYceiesesesu urc|jr5tj|jgt|j|_dSg|_dS)zInitialize an empty corpus. If the corpora are to be treated as lists, simply initialize an empty list. If serialization is used, initialize an empty corpus using :class:`~gensim.corpora.mmcorpus.MmCorpus`. N)rbr serializercr)rvs rrdz"AuthorTopicModel.init_empty_corpusQsI ?   t6 ; ; ;"4#:;;DKKKDKKKrc|jrt|tr|jj|jks Jdt |j|}t |j|jdz|jdz|j_tj|j|t|j|_t|jdzdSt|ts Jd|j |dS)aAdd new documents from `corpus` to `self.corpus`. If serialization is used, then the entire corpus (`self.corpus`) is re-serialized and the new documents are added in the process. If serialization is not used, the corpus, as a list of documents, is simply extended. Parameters ---------- corpus : iterable of list of (int, float) Corpus in BoW format Raises ------ AssertionError If serialized == False and corpus isn't list. zUInput corpus cannot have the same file path as the model corpus (serialization_path).z.tmpz8If serialized == False, all input corpora must be lists.N) rb isinstancerr)inputrrrcryrlistextend)rr) corpus_chains r extend_corpuszAuthorTopicModel.extend_corpus`s" ? '&(++ l{(FL8llklll f55L T,d.E.N O O O $ 7& @DK   t6 E E E"4#:;;DK 4*V3 4 4 4 4 4fd++ g g-g g g g K  v & & & & &rcb|d}||dz}|S)aEfficiently computes the normalizing factor in phi. Parameters ---------- expElogthetad: numpy.ndarray Value of variational distribution :math:`q(\theta|\gamma)`. expElogbetad: numpy.ndarray Value of variational distribution :math:`q(\beta|\lambda)`. Returns ------- float Value of normalizing factor. raxisg0.++)sumdot)r expElogthetad expElogbetadexpElogtheta_sumphinorms rcompute_phinormz AuthorTopicModel.compute_phinorms8 ),,!,44"&&|44v=rc  t|n#t$rt|}YnwxYwt|dkr(tdt||rt jj}nd}d}t jdj f} t|D]\} } | || } n| } | r:t| ddtt j fs d| D} n d| D} t j| t} t jd| Dtt|  }t jfd j| Dt}jj|ddf}|}t)|}t j|}jdd| f}||}t/jD]}|}t j||z |j}t|D]I\}}jtjj|||ddfz|zz||ddf<Jd|z |z||zz}t)|}t j|}||}t=||}|j k}|r|dz }n|jj|ddf<t j!| |g} |rE|"d }|dd| fxxt j#|j||z z cc<t|dkr/td |t|j|r |jz}| |fS) a Give a `chunk` of sparse document vectors, update gamma for each author corresponding to the `chuck`. Warnings -------- The whole input chunk of document is assumed to fit in RAM, chunking of a large corpus must be done earlier in the pipeline. Avoids computing the `phi` variational parameter directly using the optimization presented in `Lee, Seung: "Algorithms for non-negative matrix factorization", NIPS 2001 `_. Parameters ---------- chunk : iterable of list of (int, float) Corpus in BoW format. author2doc : dict of (str, list of int), optional A dictionary where keys are the names of authors and values are lists of document IDs that the author contributes to. doc2author : dict of (int, list of str), optional A dictionary where the keys are document IDs and the values are lists of author names. rhot : float Value of rho for conducting inference on documents. collect_sstats : boolean, optional If True - collect sufficient statistics needed to update the model's topic-word distributions, and return `(gamma_chunk, sstats)`. Otherwise, return `(gamma_chunk, None)`. `gamma_chunk` is of shape `len(chunk_authors) x self.num_topics`,where `chunk_authors` is the number of authors in the documents in the current chunk. chunk_doc_idx : numpy.ndarray, optional Assigns the value for document index. Returns ------- (numpy.ndarray, numpy.ndarray) gamma_chunk and sstats (if `collect_sstats == True`, otherwise - None) r<z/performing inference on a chunk of %i documentsNrc2g|]\}}t|Sr$)int.0idxr-s r z.AuthorTopicModel.inference..s"222FCs3xx222rcg|]\}}|Sr$r$rs rrz.AuthorTopicModel.inference..s---vsAs---rrc3 K|] \}}|V dSNr$rr-cnts r z-AuthorTopicModel.inference..&55vq#s555555rrcountc32K|]}j|VdSrr`rr/rs rrz-AuthorTopicModel.inference..*$X$X1T^A%6$X$X$X$X$X$Xrrz.%i/%i documents converged within %i iterations)$rO TypeErrorr}rLdebugr zeros_likerprrUr&r{rintegerarrayfromiterr+rnrcopyr rorrangerlrTrEr*rar ravelrmvstackrouter)rchunkr*r+rhotcollect_sstats chunk_doc_idxr converged gamma_chunkr,docdoc_noidscts authors_dgammad tilde_gamma Elogthetadrrrr- lastgammarair/meanchange_gammagamma_conditionexpElogtheta_sum_as` r inferencezAuthorTopicModel.inferencesXJ JJJJ   KKEEE  u::> X LLJCPUJJ W W W  ]4#344FFF h4?344  &&I PI PFAs &q) .:c!fQi#rz1CDD .22c222-----(3c***C+55555SCQQQC $X$X$X$XPV@W$X$X$X`cdddIZ%il3F ++--K/{;;JF:..M+AAAsF3L**=,GGG4?++  ',,.. fS7]LN;;&y11EB dodnQ.?@AAMRTVWVWVWRWDXX[^^_ AAA&& !4x61D;4FF 3;?? "z 2 2 ..}lKK$??K P&3%6%6A%6%>%>"qqq#v"(+=+?w"O"OO u::>  LL@3u::t     ' d& &FF""s //c||j}|||||d|\}}|xj|z c_|xjt |z c_|S)aPerforms inference (E-step) on a chunk of documents, and accumulate the collected sufficient statistics. Parameters ---------- chunk : iterable of list of (int, float) Corpus in BoW format. author2doc : dict of (str, list of int), optional A dictionary where keys are the names of authors and values are lists of document IDs that the author contributes to. doc2author : dict of (int, list of str), optional A dictionary where the keys are document IDs and the values are lists of author names. rhot : float Value of rho for conducting inference on documents. state : int, optional Initializes the state for a new E iteration. chunk_doc_idx : numpy.ndarray, optional Assigns the value for document index. Returns ------- float Value of gamma for training of model. NTrr)rnrrrrO) rrr*r+rrnrrrs rdo_estepzAuthorTopicModel.do_estep+sj4  JE :z4}'  v    U#  rc<|t|}td|D}d|zt|z }||||||zz }td|t j| t|||S)aCalculate per-word likelihood bound, using the `chunk` of documents as evaluation corpus. Parameters ---------- chunk : iterable of list of (int, float) Corpus in BoW format. chunk_doc_idx : numpy.ndarray, optional Assigns the value for document index. total_docs : int, optional Initializes the value for total number of documents. Returns ------- float Value of per-word likelihood bound. Nc3*K|]}|D] \}}|V dSrr$)rdocumentr-rs rrz2AuthorTopicModel.log_perplexity..ds3LL88LLC3LLLLLLLrr?)subsample_ratioz]%.3f per-word bound, %.1f perplexity estimate based on a corpus of %i documents with %i words)rOrboundrLrSrexp2)rrrr\ corpus_wordsr perwordbounds rlog_perplexityzAuthorTopicModel.log_perplexityOs&  $UJLLuLLLLL  *SZZ7zz%zXX | +-  k "'<-00#e**l   rc (%&jj|j}|j}| j} | j} | j} t|}t|}|Ijdks JddtjD} tj }n||td|t||}n|t|}t|} t|%nC#t$r6t dt%d|D%YnwxYw%dkrt ddSxj%z c_|g}t)|D]1}j |s||2t|}t1|D])\}}|jzj|<|j|jz<*xj|z c_jd d |jf}t?j j!j|gj!_|"D]\}}%fd |D}|"D]J\}}j |r!j |#|@|j |<K|"D]\}}|j$|<tK} j &D]}| '|t)| } t| }tQ|j)j!xj*|z c_*|rd }tQ||j+zz}nd }|}tQ|| pdj+zz}tYd||z }t -d|j|||||| | ||zdkrt d&fd}t|D]&j.r@t -dj+j./j!n%taj1j!j2j3d}d}d} t1tij5| | D] \}!}"fd|"D}#| t|#z } | r2| |ks|!dz| j+zzdkr6|#|"|j.rKt -d&|!zt|#z|j.7|#nt -d&|!zt|#z|8|#j j$|||"}$j9r:|$|d}~#|r|!dz|j+zzdkrƉj.r3t -dj.;}<||&dk~j.r:t -dj./j!n%taj1j!j2j3d}d}| |krt{d|r^j.r3t -dj.;}<||&dk~dS)aTrain the model with new documents, by EM-iterating over `corpus` until the topics converge (or until the maximum number of allowed iterations is reached). Notes ----- This update also supports updating an already trained model (`self`) with new documents from `corpus`; the two models are then merged in proportion to the number of old vs. new documents. This feature is still experimental for non-stationary input streams. For stationary input (no topic drift in new documents), on the other hand, this equals the online update of `'Online Learning for LDA' by Hoffman et al.`_ and is guaranteed to converge for any `decay` in (0.5, 1]. Additionally, for smaller corpus sizes, an increasing `offset` may be beneficial (see Table 1 in the same paper). If update is called with authors that already exist in the model, it will resume training on not only new documents for that author, but also the previously seen documents. This is necessary for those authors' topic distributions to converge. Every time `update(corpus, author2doc)` is called, the new documents are to appended to all the previously seen documents, and author2doc is combined with the previously seen authors. To resume training on all the data seen by the model, simply call :meth:`~gensim.models.atmodel.AuthorTopicModel.update`. It is not possible to add new authors to existing documents, as all documents in `corpus` are assumed to be new documents. Parameters ---------- corpus : iterable of list of (int, float) The corpus in BoW format. author2doc : dict of (str, list of int), optional A dictionary where keys are the names of authors and values are lists of document IDs that the author contributes to. doc2author : dict of (int, list of str), optional A dictionary where the keys are document IDs and the values are lists of author names. chunksize : int, optional Controls the size of the mini-batches. decay : float, optional A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten when each new document is examined. Corresponds to :math:`\kappa` from `'Online Learning for LDA' by Hoffman et al.`_ offset : float, optional Hyper-parameter that controls how much we will slow down the first steps the first few iterations. Corresponds to :math:`\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_ passes : int, optional Number of times the model makes a pass over the entire training data. update_every : int, optional Make updates in topic probability for latest mini-batch. eval_every : int, optional Calculate and estimate log perplexity for latest mini-batch. iterations : int, optional Maximum number of times the model loops over each document gamma_threshold : float, optional Threshold value of gamma(topic difference between consecutive two topics) until which the iterations continue. chunks_as_numpy : bool, optional Whether each chunk passed to :meth:`~gensim.models.atmodel.AuthorTopicModel.inference` should be a numpy array of not. Numpy can in some settings turn the term IDs into floats, these will be converted back into integers in inference, which incurs a performance hit. For distributed computing (not supported now) it may be desirable to keep the chunks as numpy arrays. Nrz2update() was called with no documents to train on.cg|]}|Sr$r$)rr,s rrz+AuthorTopicModel.update..sBBBaBBBrz`at least one of author2doc/doc2author must be specified, to establish input space dimensionality4input corpus stream has no len(); counting documentsc3K|]}dVdSr<Nr$rr-s rrz*AuthorTopicModel.update..s"&9&9Qq&9&9&9&9&9&9rz5AuthorTopicModel.update() called with an empty corpusrFrCc*g|]}|jzz Sr$r\)rr,len_input_corpusrs rrz+AuthorTopicModel.update..s&SSSa1t.1AASSSronlinebatchr<zrunning %s author-topic training, %s topics, %s authors, %i passes over the supplied corpus of %i documents, updating model once every %i documents, evaluating perplexity every %i documents, iterating %ix with a convergence threshold of %frAzxtoo few updates, training might not converge; consider increasing the number of passes or iterations to improve accuracycBtzjz z Sr)powr[)rWrXrYpass_rsrrhoz$AuthorTopicModel.update..rho@s&v~)9I)EFOO Orzinitializing %s workers)rrF)as_numpyc*g|]}j|Sr$)r))rr,rs rrz+AuthorTopicModel.update..Os???AQ???rrz5PROGRESS: pass %i, dispatching documents up to #%i/%iz%PROGRESS: pass %i, at document #%i/%iTzFreached the end of input; now waiting for all remaining jobs to finishzinitializing workerszIinput corpus size changed during training (don't use generators as input))>rXrYr]r^r_rlrmrr\rrOr*rKr1r7rrLrMrrsortedrRgetr(r&rVr`rarkrrUrrrnr'r~r+r3valuesrqminrWrrIrQrSrHresetrrrrgr grouperrputjobrrf update_alphagetstatedo_mstep RuntimeError)'rr)r*r+rWrXrYr]r^r_rlrmrGtrain_corpus_idxnum_input_authors new_authorsr/num_new_authorsa_ida_name gamma_newdoc_idsr,a_list lencorpus updatetype updateafter evalafterupdates_per_passrotherdirtyreallenchunk_norrgammatrrs'` ``` @@rrqzAuthorTopicModel.updatensF  JE  ![F  ![F  -,L  )J  )J  3"2Oj)) j))  Q 8?Q& \ \(\ \ \ \BB5+A+ABBB  #DO 4 4   j  v  >1&*EE  >1*== !$J  :#&v;;   : : :UVVV#&&9&9&&9&9&9#9#9    : 1$ VWWW OO/ /OO   v & & &KJOO--.. * ***1--*&&q)))!+..O!*+ 6 6 A A f)-0@)@v&:@td&6677    /  )//i/SWSbAcddI!y$**:I)FGGDJ )..00 T T 7SSSSS7SSS)..00 1 1 7?&&q))1OA&--g6666*1DOA&&(--// , , 6%+"" #uu ?1133 1 1 ''0000 &&677 ())  7It~66I i'  $!Ji)G))STTKK J#K JO!t#F#RSS q)k"9::  ? ):FI{ z?     f $r )  NN]    P P P P P P P P P6]]E E E T 5tGGG%%dj1111)4:3D3JFSSEG+4M"2IXXX,Z,Z/ "/ "'-???????3u::%TGy$8Tx!|PZ]a]lPl>mqr>rT''}'SSS?9KKOx)3c%jj@) O**51111KK?x)3c%jj@) "]]5$/4?TWTWTYTY[`boppF*9))&##%%888 "X\lT_6T$UYZ$Z "; $lmmm $ 8 8 : :MM##%% :::\ $:;;;--dj9999 04:;L;RTZ [ [!E)# p"#nooo ?7KK hiii O4466E cceeUEAI666KE E s6D=EEc j}t|}tj|}jj} |"| j}j}|stdne|T|R| D]+} j | std,|rtdntdt| } tj| } d} d}t|D]]\}}|r ||}n|}tj fdj|Dt}tj d |Dtt| }tj d |Dtt| }|jzd krt d || |ddf|dd|f}| tjdt|z t)|z|tj|zz } _| |z} | D]} j| } |tjj| | ddfz | | ddfzz }|tjt1| | ddft1jz z }|t1tjjt1tj| | ddfz z }|jt|z z}d}|tjj|z |zz }|tjt1|t1jz z }tjj}|tjt1|t1tj|dz z }| |z|z}|S)aEstimate the variational bound of documents from `corpus`. :math:`\mathbb{E_{q}}[\log p(corpus)] - \mathbb{E_{q}}[\log q(corpus)]` Notes ----- There are basically two use cases of this method: #. `chunk` is a subset of the training corpus, and `chunk_doc_idx` is provided, indicating the indexes of the documents in the training corpus. #. `chunk` is a test set (held-out data), and `author2doc` and `doc2author` corresponding to this test set are provided. There must not be any new authors passed to this method, `chunk_doc_idx` is not needed in this case. Parameters ---------- chunk : iterable of list of (int, float) Corpus in BoW format. chunk_doc_idx : numpy.ndarray, optional Assigns the value for document index. subsample_ratio : float, optional Used for calculation of word score for estimation of variational bound. author2doc : dict of (str, list of int), optional A dictionary where keys are the names of authors and values are lists of documents that the author contributes to. doc2author : dict of (int, list of str), optional A dictionary where the keys are document IDs and the values are lists of author names. Returns ------- float Value of variational bound score. NzdEither author dictionaries or chunk_doc_idx must be provided. Consult documentation of bound method.z=bound cannot be called with authors not seen during training.znEither author dictionaries or chunk_doc_idx must be provided, not both. Consult documentation of bound method.zlEither both author2doc and doc2author should be provided, or neither. Consult documentation of bound method.gc32K|]}j|VdSrrrs rrz)AuthorTopicModel.bound..rrrc3 K|] \}}|V dSrr$)ridr-s rrz)AuthorTopicModel.bound..s&33eb!r333333rrc3 K|] \}}|V dSrr$rs rrz)AuthorTopicModel.bound..rrrzbound: at document #%i in chunkr?r<)rn get_lambdar rrorr*r+rKrRrr&rrrOrWrLrrlogrrr`rErrVr)rrrrr*r+_lambdaElogbetarprr/ Elogtheta expElogtheta word_score theta_scorer,rrrrrr beta_scoresum_eta total_scores` rrzAuthorTopicModel.boundsZT*''))(11fX&&     * JJ  !=     __&& f f**1--f$%deeef  =  9  *%00 vi((   && ] ]FAs &q) $X$X$X$XPV@W$X$X$X`cdddI+33s3333c#hhOOOC+55555SCQQQC4>!Q& C >BBB**< 111 +E{STSTSTVYSYGZ[[G "&s9~~!566SACGGBFSZOOD\D\\ \JJ o% "" V VAq!A 264:ad #;yAAA"NOO OK 26'%111+"6"69L9L"LMM MK 726$*#5#566aQRQRQRd ATAT9U9UU UKK t'#j//99  bfdh0H<=== bfWW--0A0AABBB &""bfWW--w8J8J0K0KKLLL  ;.; rc td)aOverride :meth:`~gensim.models.ldamodel.LdaModel.get_document_topics` and simply raises an exception. Warnings -------- This method invalid for model, use :meth:`~gensim.models.atmodel.AuthorTopicModel.get_author_topics` or :meth:`~gensim.models.atmodel.AuthorTopicModel.get_new_author_topics` instead. Raises ------ NotImplementedError Always. ziMethod "get_document_topics" is not valid for the author-topic model. Use the "get_author_topics" method.)NotImplementedError)rword_idrZs rget_document_topicsz$AuthorTopicModel.get_document_topics s" 2   rcH fd} fd} t|}nC#t$r6tdt d|D}YnwxYw|dkrt ddt tjj|z d}j }j vrt d |j <j |< j < D] }gj |<jd d |jf} t#jjj| gj_ |j j |d \} } |} |n#|wxYw| S)a5Infers topics for new author. Infers a topic distribution for a new author over the passed corpus of docs, assuming that all documents are from this single new author. Parameters ---------- corpus : iterable of list of (int, float) Corpus in BoW format. minimum_probability : float, optional Ignore topics with probability below this value, if None - 1e-8 is used. Returns ------- list of (int, float) Topic distribution for the given `corpus`. cFtjdzdzj S)Nr<)rrYrXrvsrrz3AuthorTopicModel.get_new_author_topics..rho/s"t{Q*TZK88 8rcjjddj_j=j}j|=j=D] }j|= dS)Nr)rnrr*r`rar+)r new_doc_idcorpus_doc_idxnew_author_namers rrollback_new_author_chageszJAuthorTopicModel.get_new_author_topics..rollback_new_author_chages2sk#z/"5DJ 0>/2Dt$/, 0 0 OJ// 0 0rrc3K|]}dVdSrr$rs rrz9AuthorTopicModel.get_new_author_topics..As""5"51"5"5"5"5"5"5rrzDAuthorTopicModel.get_new_author_topics() called with an empty corpusplaceholder_namer<z4self.author2id already has 'placeholder_name' authorrFrCFr)rOrrLrMrrKr}rr\rVr`rar*r+rkrrUrrrnrget_author_topics)rr)rZrrrr author_idrrrr-new_author_topicsrrs` @@rget_new_author_topicsz&AuthorTopicModel.get_new_author_topicss& 9 9 9 9 9 0 0 0 0 0 0 0 6"6{{   6 6 6 NNQ R R R""5"5f"5"5"555    6 q  ecdd d,eDOT_GW5WXXYY$ dn , USTT T*3'$3y!,:(( < >> from gensim.models import AuthorTopicModel >>> from gensim.corpora import mmcorpus >>> from gensim.test.utils import common_dictionary, datapath, temporary_file >>> author2doc = { ... 'john': [0, 1, 2, 3, 4, 5, 6], ... 'jane': [2, 3, 4, 5, 6, 7, 8], ... 'jack': [0, 2, 4, 6, 8] ... } >>> >>> corpus = mmcorpus.MmCorpus(datapath('testcorpus.mm')) >>> >>> with temporary_file("serialized") as s_path: ... model = AuthorTopicModel( ... corpus, author2doc=author2doc, id2word=common_dictionary, num_topics=4, ... serialized=True, serialization_path=s_path ... ) ... ... model.update(corpus, author2doc) # update the author-topic model with additional documents >>> >>> # construct vectors for authors >>> author_vecs = [model.get_author_topics(author) for author in model.id2author.values()] Ng:0yE>c*g|]\}}|k ||fSr$r$)rtopicid topicvaluerZs rrz6AuthorTopicModel.get_author_topics..s>   &9gz00 j !   r)r`rZrQrnrrr&)r author_namerZr topic_dist author_topicss ` rrz"AuthorTopicModel.get_author_topicsdsXN;/  ;"&": !"5t<<Z%il3c$*:J9VWVWVW<:X6Y6YY     =Fz=R=R   rct|tr2g}|D],}||||-n|||}|S)aPGet topic distribution for input `author_names`. Parameters ---------- author_names : {str, list of str} Name(s) of the author for which the topic distribution needs to be estimated. eps : float, optional The minimum probability value for showing the topics of a given author, topics with probability < `eps` will be ignored. Returns ------- list of (int, float) **or** list of list of (int, float) Topic distribution for the author(s), type depends on type of `author_names`. )rZ)r{r}r(r)r author_namesepsr'r/s r __getitem__zAuthorTopicModel.__getitem__sz" lD ) ) RE! Q Q T33A33OOPPPP Q**r?r@r@r<rArBFNrCN)FN)NN) NNNNNNNNNNNF)Nr?NNr)r r!r"r#rrwrdrrrrrrqrrrrr'r$rrr9r9s]aa^bLOPRMQ8< _S_S_S_SB u u u   #'#'#'J*O#O#O#O#b""""H>hlKO5:ZZZZx}}}}~    &F!F!F!F!P9999vrr9)"r#logging itertoolsrrrshutilros.pathrosrnumpyr scipy.specialrgensimr gensim.modelsr gensim.models.ldamodelr gensim.matutilsr r gensim.corporar getLoggerr rLrr1r7r9r$rrr5s//l!!!!!!""""""++++++KKKKKKKK######  8 $ $     x   .4