cdZddlZddlZddlmZmZddlmZddl m Z m Z ddl m Z ejeZGdde jZGd d e jZGd d e jZd ZdZdS)a&Lda Sequence model, inspired by `David M. Blei, John D. Lafferty: "Dynamic Topic Models" `_. The original C/C++ implementation can be found on `blei-lab/dtm `_. TODO: The next steps to take this forward would be: #. Include DIM mode. Most of the infrastructure for this is in place. #. See if LdaPost can be replaced by LdaModel completely without breaking anything. #. Heavy lifting going on in the Sslm class - efforts can be made to cythonise mathematical methods, in particular, update_obs and the optimization takes a lot time. #. Try and make it distributed, especially around the E and M step. #. Remove all C/C++ coding style/syntax. Examples -------- Set up a model using have 30 documents, with 5 in the first time-slice, 10 in the second, and 15 in the third .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus >>> from gensim.models import LdaSeqModel >>> >>> ldaseq = LdaSeqModel(corpus=common_corpus, time_slice=[2, 4, 3], num_topics=2, chunksize=1) Persist a model to disk and reload it later .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> temp_file = datapath("model") >>> ldaseq.save(temp_file) >>> >>> # Load a potentially pre-trained model from disk. >>> ldaseq = LdaSeqModel.load(temp_file) Access the document embeddings generated from the DTM .. sourcecode:: pycon >>> doc = common_corpus[1] >>> >>> embedding = ldaseq[doc] N)digammagammaln)optimize)utilsmatutils)ldamodelcteZdZdZ dd Zd ZdZdZdZdZ dZ ddZ ddZ ddZ dZdZdZdZdS) LdaSeqModelzCEstimate Dynamic Topic Model parameters based on a training corpus.N{Gz? gensim?{Gzt?dc v||_||jtd|jMtdt j||_t |j|_n(|jrt |j|_nd|_|^ t ||_nH#t$r;tdtd|D|_YnwxYw||_ |j t ||_ ||_ t ||_ tj|||_g|_t%|D]?}t'|j |j|j | | }|j|@d|_d|_d|_d|_||t3d|D|_|d krWt7j||j|j | |j| tj }tj|jj |_ |d kr#tj|jj |_ |d kr||_ |!| | |j|j |"|| |||dSdSdS) a Parameters ---------- corpus : {iterable of list of (int, float), scipy.sparse.csc}, optional Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`). If not given, the model is left untrained (presumably because you want to call :meth:`~gensim.models.ldamodel.LdaSeqModel.update` manually). time_slice : list of int, optional Number of documents in each time-slice. Each time slice could for example represent a year's published papers, in case the corpus comes from a journal publishing over multiple years. It is assumed that `sum(time_slice) == num_documents`. id2word : dict of (int, str), optional Mapping from word IDs to words. It is used to determine the vocabulary size, as well as for debugging and topic printing. alphas : float, optional The prior probability for the model. num_topics : int, optional The number of requested latent topics to be extracted from the training corpus. initialize : {'gensim', 'own', 'ldamodel'}, optional Controls the initialization of the DTM model. Supports three different modes: * 'gensim': Uses gensim's LDA initialization. * 'own': Uses your own initialization matrix of an LDA model that has been previously trained. * 'lda_model': Use a previously used LDA model, passing it through the `lda_model` argument. sstats : numpy.ndarray , optional Sufficient statistics used for initializing the model if `initialize == 'own'`. Corresponds to matrix beta in the linked paper for time slice 0, expected shape (`self.vocab_len`, `num_topics`). lda_model : :class:`~gensim.models.ldamodel.LdaModel` Model whose sufficient statistics will be used to initialize the current object if `initialize == 'gensim'`. obs_variance : float, optional Observed variance used to approximate the true and forward variance as shown in `David M. Blei, John D. Lafferty: "Dynamic Topic Models" `_. chain_variance : float, optional Gaussian parameter defined in the beta distribution to dictate how the beta values evolve over time. passes : int, optional Number of passes over the corpus for the initial :class:`~gensim.models.ldamodel.LdaModel` random_state : {numpy.random.RandomState, int}, optional Can be a np.random.RandomState object, or the seed to generate one. Used for reproducibility of results. lda_inference_max_iter : int, optional Maximum number of iterations in the inference step of the LDA training. em_min_iter : int, optional Minimum number of iterations until converge of the Expectation-Maximization algorithm em_max_iter : int, optional Maximum number of iterations until converge of the Expectation-Maximization algorithm. chunksize : int, optional Number of documents in the corpus do be processed in in a chunk. NzYat least one of corpus/id2word must be specified, to establish input space dimensionalityzHno word id mapping provided; initializing from corpus, assuming identityrz4input corpus stream has no len(); counting documentsc3K|]}dVdS)N).0_s 9lib/python3.11/site-packages/gensim/models/ldaseqmodel.py z'LdaSeqModel.__init__..s"%8%8Aa%8%8%8%8%8%8)num_time_slices vocab_len num_topicschain_variance obs_variancec34K|]}t|VdSN)len)rlines rrz'LdaSeqModel.__init__..s("@"@3t99"@"@"@"@"@"@rr )id2wordrpassesalpha random_statedtyperown)#r& ValueErrorloggerwarningrdict_from_corpusr$r corpus_len TypeErrorsum time_slicerrnpfullalphas topic_chainsrangesslmappend top_doc_phis influencerenormalized_influenceinfluence_sum_lglmax max_doc_lenrLdaModelfloat64 transposestatesstatsinit_ldaseq_ss fit_lda_seq)selfcorpusr3r&r6r initializerE lda_modelr!r r'r)lda_inference_max_iter em_min_iter em_max_iter chunksizetopicsslm_s r__init__zLdaSeqModel.__init__Gsl  dl k  <  NNe f f f 1&99DL ..DNN \  ..DNNDN  9 9"%f++ 9 9 9UVVV"%%8%8%8%8%8"8"8 9% ? 3#&z??D $":gj&11 :&& , ,E $ 4[_[j-LE   $ $U + + + +!&*#!%  b* b""@"@"@"@"@@@D X% C$-DLT_!<* !l9?+ABB Z' C l9?+ABB U" %$     dk4; W W W   V%;[+W` a a a a a' b b b bsB44AC98C9c||_t|jD].\}}|dd|f}t||||/dS)aInitialize State Space Language Model, topic-wise. Parameters ---------- topic_chain_variance : float Gaussian parameter defined in the beta distribution to dictate how the beta values evolve. topic_obs_variance : float Observed variance used to approximate the true and forward variance as shown in `David M. Blei, John D. Lafferty: "Dynamic Topic Models" `_. alpha : float The prior probability for the model. init_suffstats : numpy.ndarray Sufficient statistics used for initializing the model, expected shape (`self.vocab_len`, `num_topics`). N)r6 enumerater7r9sslm_counts_init)rHtopic_chain_variancetopic_obs_variancer(init_suffstatskchainrEs rrFzLdaSeqModel.init_ldaseq_sssi" !$"344 [ [HAu#AAAqD)F  ! !%);=QSY Z Z Z Z [ [rc d}d}d}d} |j} |j} |j} |j} d}|dz}d}||ks||kr||krtd|td|}g}t | D]+}|tj | | f,tj | | f}tj | | dzf}| |||||||\}}||_ td | |}||z }||z dkr&||kr||z}td |tj ||z |z }||kr| }td |d }td ||||dz }||k||kr||k|S)aFit a LDA Sequence model (DTM). This method will iteratively setup LDA models and perform EM steps until the sufficient statistics convergence, or until the maximum number of iterations is reached. Because the true posterior is intractable, an appropriately tight lower bound must be used instead. This function will optimize this bound, by minimizing its true Kullback-Liebler Divergence with the true posterior. Parameters ---------- corpus : {iterable of list of (int, float), scipy.sparse.csc} Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`). lda_inference_max_iter : int Maximum number of iterations for the inference step of LDA. em_min_iter : int Minimum number of time slices to be inspected. em_max_iter : int Maximum number of time slices to be inspected. chunksize : int Number of documents to be processed in each chunk. Returns ------- float The highest lower bound for the true posterior produced after all iterations. g-C6?r irrz EM iter %izE StepzM Stepz,Bound went down, increasing iterations to %iz)Starting final iterations, max iter is %i?:;;F""6?FFESiktuu E6 DK KK ! ! !11/BBK [ E !Q& d)J6<*m;* JLbccc'59#4 "ABBK00 ")1& GI_```! KKVX]_dfq r r r QJEYk!, {5H'H, eWbNb, \ rc |j}|j} d} tj||j|jt j} t j| |f| _ t|j || } d} | dkr"| ||||| | || || \} }n<| dkr6| ||||||| | || || \} }| |fS)aInference (or E-step) for the lower bound EM optimization. This is used to set up the gensim :class:`~gensim.models.ldamodel.LdaModel` to be used for each time-slice. It also allows for Document Influence Model code to be written in. Parameters ---------- corpus : {iterable of list of (int, float), scipy.sparse.csc} Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`). topic_suffstats : numpy.ndarray Sufficient statistics for time slice 0, used for initializing the model if `initialize == 'own'`, expected shape (`self.vocab_len`, `num_topics`). gammas : numpy.ndarray Topic weight variational parameters for each document. If not supplied, it will be inferred from the model. lhoods : list of float The total log probability lower bound for each topic. Corresponds to the phi variational parameters in the linked paper. iter_ : int Current iteration. lda_inference_max_iter : int Maximum number of iterations for the inference step of LDA. chunksize : int Number of documents to be processed in each chunk. Returns ------- (float, list of float) The first value is the highest lower bound for the true posterior. The second value is the list of optimized dirichlet variational parameters for the approximation of the posterior. rr(r&r*)r@rldaDTMDIM)rrrrAr6r&r4rBr`topicsLdaPostr@ inferDTMseqInfluenceTotalFixed inferDIMseq)rHrIrnrbrorlrLrOrrrjrtldapostmodels rrazLdaSeqModel.lda_seq_infer8s D_ N :T[RVR^fhfpqqqXy*566 d&6:SVWWW E>  ,,'=yME66e^   $ $V , , , ,,'=yME6 f}rc d} d} d} ||| }tjtj|j}t t j|| D]\}}|D]}| || kr| dz } ||| }d} || }|| }||_||_ ||_ |dkr t || | d| }nt || | || }|t || ||}|j|| <||z }| dz } | dz } Ό||fS)aCompute the likelihood of a sequential corpus under an LDA seq model, and reports the likelihood bound. Parameters ---------- corpus : {iterable of list of (int, float), scipy.sparse.csc} Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`). topic_suffstats : numpy.ndarray Sufficient statistics of the current model, expected shape (`self.vocab_len`, `num_topics`). gammas : numpy.ndarray Topic weight variational parameters for each document. If not supplied, it will be inferred from the model. lhoods : list of float of length `self.num_topics` The total log probability bound for each topic. Corresponds to phi from the linked paper. lda : :class:`~gensim.models.ldamodel.LdaModel` The trained LDA model of the previous iteration. ldapost : :class:`~gensim.models.ldaseqmodel.LdaPost` Posterior probability variables for the given LDA model. This will be used as the true (but intractable) posterior. iter_ : int The current iteration. bound : float The LDA bound produced after all iterations. lda_inference_max_iter : int Maximum number of iterations for the inference step of LDA. chunksize : int Number of documents to be processed in each chunk. Returns ------- (float, list of float) The first value is the highest lower bound for the true posterior. The second value is the list of optimized dirichlet variational parameters for the approximation of the posterior. rrN)rL)make_lda_seq_slicer4cumsumarrayr3rTrgroupergammalhooddocrx fit_lda_postupdate_lda_seq_ss)rHrIrnrbrortr|rlrjrLrO doc_indextimedoc_numr3chunk_nochunkrgamr doc_lhoods rryzLdaSeqModel.inferDTMseqqsH %%c400Yrx8899 (vy)I)IJJ  OHe  z$// AID11#t<qqq$wGCJqqq!t  GDK((  rcd}t|jD]F\}}td|t|||}||z }G|S)aWFit the sequential model topic-wise. Parameters ---------- topic_suffstats : numpy.ndarray Sufficient statistics of the current model, expected shape (`self.vocab_len`, `num_topics`). Returns ------- float The sum of the optimized lower bounds for all topics. rzFitting topic number %i)rTr7r-r_r9fit_sslm)rHrnrrYrZ lhood_terms rrczLdaSeqModel.fit_lda_seq_topicsse!$"344  HAu KK11 5 5 5uoa.@AAJ Z EE rcg}t|jD],}|||||-|S)aGet the most relevant words for a topic, for each timeslice. This can be used to inspect the evolution of a topic through time. Parameters ---------- topic : int The index of the topic. top_terms : int, optional Number of most relevant words associated with the topic to be returned. Returns ------- list of list of str Top `top_terms` relevant terms for the topic for each time slice. )r8rr: print_topic)rHrP top_termsrwrs rprint_topic_timeszLdaSeqModel.print_topic_timessQ"$.// D DD MM$**5$ BB C C C C rrcLfdtjDS)agGet the most relevant words for every topic. Parameters ---------- time : int, optional The time slice in which we are interested in (since topics evolve over time, it is expected that the most relevant words will also gradually change). top_terms : int, optional Number of most relevant words to be returned for each topic. Returns ------- list of list of (str, float) Representation of all topics. Each of them is represented by a list of pairs of words and their assigned probability. c>g|]}|Sr)r)rrPrHrrs r z,LdaSeqModel.print_topics..s+]]]U  i88]]]r)r8r)rHrrs```r print_topicszLdaSeqModel.print_topicss2$^]]]]]eDOF\F\]]]]rcjjtjtj|z t j|d}fd|D}|S)aGet the list of words most relevant to the given topic. Parameters ---------- topic : int The index of the topic to be inspected. time : int, optional The time slice in which we are interested in (since topics evolve over time, it is expected that the most relevant words will also gradually change). top_terms : int, optional Number of words associated with the topic to be returned. Returns ------- list of (str, float) The representation of this topic. Each element in the list includes the word itself, along with the probability assigned to it by the topic. T)reversec:g|]}j||fSrr&)rid_rHrPs rrz+LdaSeqModel.print_topic..2s)DDDsDL%uSz2DDDr)r7rr4rCexpr2rargsort)rHrPrrbestnbeststrs`` rrzLdaSeqModel.print_topics(!%(3 U##uT{## #  4@@@DDDDDeDDDrc|j|jdddtjfz }||S)a+Get the topic mixture for a document. Uses the priors for the dirichlet distribution that approximates the true posterior with the optimal lower bound, and therefore requires the model to be already trained. Parameters ---------- doc_number : int Index of the document for which the mixture is returned. Returns ------- list of length `self.num_topics` Probability for each topic in the mixture (essentially a point in the `self.num_topics - 1` simplex. raxisN)rbr2r4newaxis)rH doc_number doc_topics r doc_topicszLdaSeqModel.doc_topics5s:$K$+//q/"9"9!!!RZ-"HH $$rc jjdddtjfz }d fdt jD}g}tjj}t |D]?\}}|t||D]\} } || xx| z cc<@fdttj D} |tj |||| fS)aGet the information needed to visualize the corpus model at a given time slice, using the pyLDAvis format. Parameters ---------- time : int The time slice we are interested in. corpus : {iterable of list of (int, float), scipy.sparse.csc}, optional The corpus we want to visualize at the given time slice. Returns ------- doc_topics : list of length `self.num_topics` Probability for each topic in the mixture (essentially a point in the `self.num_topics - 1` simplex. topic_term : numpy.ndarray The representation of each topic as a multinomial over words in the vocabulary, expected shape (`num_topics`, vocabulary length). doc_lengths : list of int The number of words in each document. These could be fixed, or drawn from a Poisson distribution. term_frequency : numpy.ndarray The term frequency matrix (denoted as beta in the original Blei paper). This could also be the TF-IDF representation of the corpus, expected shape (number of documents, length of vocabulary). vocab : list of str The set of unique terms existing in the cropuse's vocabulary. rrNc0||z Sr#)r2)xs r normalizez&LdaSeqModel.dtm_vis..normalizefsquuww; rcpg|]2\}}tj|jj3Sr)r4rrT)rrYrZrrs rrz'LdaSeqModel.dtm_vis..isL   5 IbfU-/566 7 7   rc*g|]}j|Srr)rirHs rrz'LdaSeqModel.dtm_vis..vsCCCQaCCCr) rbr2r4rrTr7r`rr:r$r8r&r) rHrrIr topic_term doc_lengthsterm_frequencydoc_nortermfreqvocabrs `` @rdtm_viszLdaSeqModel.dtm_visJsF4K$+//q/"9"9!!!RZ-"HH         %d&788    $.11$V,, - -KFC   s3xx ( ( (! - - dt$$$,$$$$ -DCCC%DL0A0A*B*BCCC"(:.. ^URRrcg}||D]6}g}|D]\}}||||7|S)awGet the coherence for each topic. Can be used to measure the quality of the model, or to inspect the convergence through training via a callback. Parameters ---------- time : int The time slice. Returns ------- list of list of str The word representation for each topic, for each time slice. This can be used to check the time coherence of topics as time evolves: If the most relevant words remain the same then the topic has somehow converged or is relatively static, if they change rapidly the topic is evolving. )rr:)rHrcoherence_topicsrwcoherence_topicworddists r dtm_coherencezLdaSeqModel.dtm_coherencezss$''-- 5 5F O$ - - d&&t,,,,  # #O 4 4 4 4rctj|j|j|jt j}t j|j|jf|_ t|jt|||}g}t|j D]J}|||}t|d||}||K|j|jz }|S)aGet the topic mixture for the given document, using the inferred approximation of the true posterior. Parameters ---------- doc : list of (int, float) The doc in BOW format. Can be an unseen document. Returns ------- list of float Probabilities for each topic in the mixture. This is essentially a point in the `num_topics - 1` simplex. rs)rr@rtrr)rrArr6r&r4rBr`rrwrxr$r8rrrr:rr2)rHrrKr| time_lhoodsrrrs r __getitem__zLdaSeqModel.__getitem__s%dk4w_phi_l w_phi_sum w_phi_l_sqm_update_coeff_g)rHrrrr!r s rrRz sslm.__init__s".(,$8Y899(I#?@@Hi1)<=>> )_q-@!ABB Hi11D%EFF)_q-@!ABB H_-- # !%  $rc t|jD]]\}}tjtj|jdd|dzf|jdd|dzfdz z|j|<^|jS)auUpdate the Zeta variational parameter. Zeta is described in the appendix and is equal to sum (exp(mean[word] + Variance[word] / 2)), over every time-slice. It is the value of variational parameter zeta which maximizes the lower bound. Returns ------- list of float The updated zeta values for each time slice. Nrr\)rTrr4r2rrr)rHjvals r update_zetazsslm.update_zetas  ** ] ]FAs6"&111a!e8)}|jr|j||dz |z|jzz }nd}|||dz |zz||<?||||<t|dz ddD][}||dkr(t j|||||zz d}nd}|||dz|z zd|z ||zz||<\||fS)aGet the variance, based on the `Variational Kalman Filtering approach for Approximate Inference (section 3.1) `_. This function accepts the word to compute variance for, along with the associated sslm class object, and returns the `variance` and the posterior approximation `fwd_variance`. Notes ----- This function essentially computes Var[\beta_{t,w}] for t = 1:T .. :math:: fwd\_variance[t] \equiv E((beta_{t,w}-mean_{t,w})^2 |beta_{t}\ for\ 1:t) = (obs\_variance / fwd\_variance[t - 1] + chain\_variance + obs\_variance ) * (fwd\_variance[t - 1] + obs\_variance) .. :math:: variance[t] \equiv E((beta_{t,w}-mean\_cap_{t,w})^2 |beta\_cap_{t}\ for\ 1:t) = fwd\_variance[t - 1] + (fwd\_variance[t - 1] / fwd\_variance[t - 1] + obs\_variance)^2 * (variance[t - 1] - (fwd\_variance[t-1] + obs\_variance)) Parameters ---------- word: int The word's ID. chain_variance : float Gaussian parameter defined in the beta distribution to dictate how the beta values evolve over time. Returns ------- (numpy.ndarray, numpy.ndarray) The first returned value is the variance of each word in each time slice, the second value is the inferred posterior variance for the same pairs. rrrrr\)rrrr8r!r4power) rHrr INIT_VARIANCE_CONSTrrrtcs rcompute_post_variancezsslm.compute_post_variancesUL#  =&(. (+>> Qq!a% I IA  %a!e)<~)MPTPa)ab<A#6#GHLOO#1o q1ub"%% a aAA$ Hl1oa>1QRUVWWQ. @Aq1uP\]^P_F_`HQKK%%rc|j}|j|}|j|}|j|}|j|}d|d<t d|dzD]C}|j||dz |z|jzz } | ||dz zd| z ||dz zz||<D||||<t |dz ddD]9}|dkrd} n||||zz } | ||zd| z ||dzzz||<:||fS)uGet the mean, based on the `Variational Kalman Filtering approach for Approximate Inference (section 3.1) `_. Notes ----- This function essentially computes E[eta_{t,w}] for t = 1:T. .. :math:: Fwd_Mean(t) ≡ E(beta_{t,w} | beta_ˆ 1:t ) = (obs_variance / fwd_variance[t - 1] + chain_variance + obs_variance ) * fwd_mean[t - 1] + (1 - (obs_variance / fwd_variance[t - 1] + chain_variance + obs_variance)) * beta .. :math:: Mean(t) ≡ E(beta_{t,w} | beta_ˆ 1:T ) = fwd_mean[t - 1] + (obs_variance / fwd_variance[t - 1] + obs_variance) + (1 - obs_variance / fwd_variance[t - 1] + obs_variance)) * mean[t] Parameters ---------- word: int The word's ID. chain_variance : float Gaussian parameter defined in the beta distribution to dictate how the beta values evolve over time. Returns ------- (numpy.ndarray, numpy.ndarray) The first returned value is the mean of each word in each time slice, the second value is the inferred posterior mean for the same pairs. rrrrr)rrrrrr8r!) rHrr rrrrrrrs rcompute_post_meanzsslm.compute_post_mean*s>D  htn(. y=& q!a% E EA!\!a%%8>%IDL]%]^Ahq1uo-Q#a!e*0DDHQKK1+Qq1ub"%% > >A$ H"l1o&FG(1+oQ$q1u+(==DGGX~rctj|jD]L\\}}}|j||dztj|j|z |j||<M|jS)aCompute the expected log probability given values of m. The appendix describes the Expectation of log-probabilities in equation 5 of the DTM paper; The below implementation is the result of solving the equation and is implemented as in the original Blei DTM code. Returns ------- numpy.ndarray of float The expected value for the log probabilities for each word and time slice. r)r4 ndenumeraterrlogr)rHwrrs rcompute_expected_log_probzsslm.compute_expected_log_probbsi>$/:: O OKFQC$(IaLQ$7"&1:N:N$NDOA q ! !rc|j}|j}tj|}|t |z}|d|z z }|t |z}tj|}tj||d|||_||_ ||_ t|D]^}| ||j \|j |<|j|<|||j \|j|<|j|<_||_||_dS)aInitialize the State Space Language Model with LDA sufficient statistics. Called for each topic-chain and initializes initial mean, variance and Topic-Word probabilities for the first time-slice. Parameters ---------- obs_variance : float, optional Observed variance used to approximate the true and forward variance. chain_variance : float Gaussian parameter defined in the beta distribution to dictate how the beta values evolve over time. sstats : numpy.ndarray Sufficient statistics of the LDA model. Corresponds to matrix beta in the linked paper for time slice 0, expected shape (`self.vocab_len`, `num_topics`). r]rrN)rrr4rr2rrepeatreshaperr!r r8rrrrrrrrrr)rHr!r rEWrlog_norm_countsrs rrUzsslm.sslm_counts_initss9" N  '&//3///37"3///&11Ioqq999BB1aHH(,q \ \A595O5OPQSWSf5g5g 2DM! d/2-1-C-CAtGZ-[-[ *DIaL$-**$$&& 88::rcj}d}d}d}d}|dz}dtfdt|DD\__|d}d} d} | dkr||}| d kr||}t d |||kr| |kr| dz } |} ||\_ _ | dkr||}| d kr||}tj||z |z }t d | ||||kr| |k_|S) a Fits variational distribution. This is essentially the m-step. Maximizes the approximation of the true posterior for a particular topic using the provided sufficient statistics. Updates the values using :meth:`~gensim.models.ldaseqmodel.sslm.update_obs` and :meth:`~gensim.models.ldaseqmodel.sslm.compute_expected_log_prob`. Parameters ---------- sstats : numpy.ndarray Sufficient statistics for a particular topic. Corresponds to matrix beta in the linked paper for the current time slice, expected shape (`self.vocab_len`, `num_topics`). Returns ------- float The lower bound for the true posterior achieved using the fitted approximate distribution. rgư>r\rc3>K|]}tj|VdSr#r4rrrs rrz sslm.fit_sslm..s* o oQRXa[[ o o o o o orc3NK|]}|jV dSr#)rr rrrHs rrz sslm.fit_sslm..s7'm'm_`(B(B1dFY(Z(Z'm'm'm'm'm'mrrrurvzinitial sslm bound is %fr^)rzipr8rrr2 compute_boundcompute_bound_fixedr-r_ update_obsrrr4rdrr) rHrErrjrmsslm_fit_threshold sslm_max_iter convergedtotalsrlr}s ` rrz sslm.fit_sslms( N ! &*  p o#'m'm'm'mdijkdldl'm'm'm"n o o o ) t(## E> 7&&vv66E E> =,,VV<K|]}tj|VdSr#rrs rrz%sslm.compute_bound..s* k kQRXa[[ k k k k k krc3NK|]}|jV dSr#)rr rs rrz%sslm.compute_bound..s7'i'i[\(>(>q$BU(V(V'i'i'i'i'i'irc3fK|]+}j|dj|z V,dS)rN)r)rrrHrs rrz%sslm.compute_bound..s@NN$-"1% a(8(;;NNNNNNrr\zComputing bound, all timesrrr)rrr rr8rrrrr2r-r_rr4rr)rHrErrterm_1term_2term_3rentr mprev_mvrs` @rrzsslm.compute_bounds & N  , l k#'i'i'i'i`efg`h`h'i'i'i"j k k k ! 4=$$&& NNNNNU1XXNNNNNQRRUcc 0111q!a% 2 2AFFC1XX % %IaLO1a!e,M!$Q'Xa&j!,,N0BCNHZ[^`^des^t^ttu&)AE*Q..rvayy1}$QUm^bfTYq1u-=&>&>>F 6F?S(61 1CC rc jd}d}d}|j}|j}d}tj||dzf} d} t |D]K} || } d} t t | D]}| | || |zz } tj| } | |kr$| "|j| }tj|} ~| |kr!tjt | } t |D]"}| | || || |<#tj|}|| || | |f}|j| }d}|dkr%tj tt||||d}|d kr |dz }| |kr|} ||j| <M||_|j|jfS) a Optimize the bound with respect to the observed variables. TODO: This is by far the slowest function in the whole algorithm. Replacing or improving the performance of this would greatly speed things up. Parameters ---------- sstats : numpy.ndarray Sufficient statistics for a particular topic. Corresponds to matrix beta in the linked paper for the first time slice, expected shape (`self.vocab_len`, `num_topics`). totals : list of int of length `len(self.time_slice)` The totals for each time slice. Returns ------- (numpy.ndarray of float, numpy.ndarray of float) The updated optimized values for obs and the zeta variational parameter. r\r gMbP?rrNru)ffprimex0gtolargsepsilondisprv)rrr4r`r8r$sqrtrrcompute_mean_derivrfmin_cgf_obsdf_obsrr)rHrErOBS_NORM_CUTOFF STEP_SIZETOLrrrunsmean_deriv_mtxnorm_cutoff_obsrw_counts counts_normrrrderivrr}s rrzsslm.update_obss,  N  1a!e*--q% "% "AayHK3x==)) 9 9x{Xa[88 '+..K_, " "hqk"$'#,,07!xH 66HqYYA(,(?(?1nUVFW(X(XN1%% Xv~q%GhqkE>"*3StU^efCE> 0*&)O! $$&& x""rc|j}|j|}d|d<td|dzD]W}|jdkr$|j||dz |jz|jzz }nd}|||dz z}||dz kr|d|z z }|||<Xt|dz ddD]H}|jdkrd}n|j|||jzz }|||zd|z ||dzzz||<I|S)aHelper functions for optimizing a function. Compute the derivative of: .. :math:: E[eta_{t,w}]/d obs_{s,w} for t = 1:T. Parameters ---------- word : int The word's ID. time : int The time slice. deriv : list of float Derivative for each time slice. Returns ------- list of float Mean derivative for each time slice. rrrrr)rrr8r!r ) rHrrrrrrrrs rrzsslm.compute_mean_derivds42  }T* aq!a%  A 3& %a!e)"  4#667a:a=!8YI\=\ ]]u}u,u4E!HH r)NNNrr)rrrrrRrrrrrUrrrrr/rrrr9r9s  %%%%4 =&=&=&~666p"&;&;&;P999v@@@DJ#J#J#X000dFFFFFrr9cDeZdZdZd dZdZdZdZdZ d d Z d Z dS)rxaPosterior values associated with each set of documents. TODO: use **Hoffman, Blei, Bach: Online Learning for Latent Dirichlet Allocation, NIPS 2010.** to update phi, gamma. End game would be to somehow replace LdaPost entirely with LdaModel. NcT||_||_||_||_|jt j||_|jt j|dz|_|8|6t j||f|_t j||f|_d|_d|_ dS)a=Initialize the posterior value structure for the given LDA model. Parameters ---------- doc : list of (int, int) A BOW representation of the document. Each element in the list is a pair of a word's ID and its number of occurences in the document. lda : :class:`~gensim.models.ldamodel.LdaModel`, optional The underlying LDA model. max_doc_len : int, optional The maximum number of words in a document. num_topics : int, optional Number of topics discovered by the LDA model. gamma : numpy.ndarray, optional Topic weight variational parameters for each document. If not supplied, it will be inferred from the model. lhood : float, optional The log likelihood lower bound. Nr) rrtrrr4r`philog_phi doc_weightrenormalized_doc_weight)rHrrtr@rrrs rrRzLdaPost.__init__s(  : .*--DJ : 2*q.11DJ  ?z ?xj 9::DH8[*$=>>DL'+$$$rcz|jj}tj|}t |D]}t |j|||< d}|jD]\}}t |D]1}|||jj||z|j ||<2|j |} |j |} | d} t dt| D]} tj | | | } | | z } tj | } | |j |<| |j |<|dz }|j |j fS)uUpdate variational multinomial parameters, based on a document and a time-slice. This is done based on the original Blei-LDA paper, where: log_phi := beta * exp(Ψ(gamma)), over every topic for every word. TODO: incorporate lee-sueng trick used in **Lee, Seung: Algorithms for non-negative matrix factorization, NIPS 2001**. Parameters ---------- doc_number : int Document number. Unused. time : int Time slice. Unused. Returns ------- (list of float, list of float) Multinomial parameters, and their logarithm, for each word in the document. rr)rtrr4r`r8rrrrwr3r2r$ logaddexpr) rHrrrdigrYnword_idcount log_phi_rowphi_rowr rs r update_phizLdaPost.update_phi sM,X( hz""z"" , ,ATZ]++CFF "h  NGU:&& J J%(Vdhog.Fq.I%I Q"",q/KhqkGAA1c+..// 4 4LKN33&/Kf[))G)DLO!DHQK FAAx%%rc tj|jj|_d}|jD]Q\}}|j|}t|jjD] }|j|xx|||zz cc<!|dz }R|jS)a:Update variational dirichlet parameters. This operations is described in the original Blei LDA paper: gamma = alpha + sum(phi), over every topic for every word. Returns ------- list of float The updated gamma parameters for each word in the document. rr) r4rrtr(rrr2r8r)rHr9r:r;r=rYs r update_gammazLdaPost.update_gamma@sWTX^,, "h  NGUhqkG48.// 4 4 1 e!33 FAAzrc.td|jD}|j|jjdt ||jjz zd|jjz |jdt|jddf<dS)z"Initialize variational posterior. c3 K|] \}}|V dSr#rrr:r;s rrz(LdaPost.init_lda_post..X&99nguE999999rrr]N) r2rrfillrtr(floatrr2r$)rHtotals r init_lda_postzLdaPost.init_lda_postVs9999999 q)E%LL48;N,NNOOO&)DH,?&?#dh--"###rc |jj}tj|j}t tj|jjt |z }||j|<t|}d}t|D]}t|j||z }|jj||j|z |zt |j|zt |jj|z }d} |j D]j\} } |j | |dkrI|| |j | |z||jj | |z|j | |z zz }| dz } k||j|<||z }|S)zCompute the log likelihood bound. Returns ------- float The optimal lower bound for the true posterior using the approximate distribution. rurr)rtrr4r2rrr(rrr8rr2rwr3) rHr gamma_sumrdigsumr}rY e_log_theta_krr9r:r;s rcompute_lda_lhoodzLdaPost.compute_lda_lhood^sX( F4:&& tx~..//')2D2DD!& :##z""  A$DJqM22V;M"TZ]2mC 1 &&')01B)C)CD A"&(  8A;q>A%t A.-$(/RYBZ[\B]2]`d`lmn`opq`r2rstJQ&DJqM Z EE r:0yE>rc |td|jD} d} | dkr |} d} d}d}|dz }| } ||_d} | dkst $|||\|_|_ n9| dkr3t ,| ||t ||| \|_|_ |} tj | | z | | zz }||kr||kr|dz }| } ||_d} | dkst $|||\|_|_ n9| dkr3t ,| ||t ||| \|_|_ |} tj | | z | | zz }||kr||k| S)aPosterior inference for lda. Parameters ---------- doc_number : int The documents number. time : int Time slice. ldaseq : object Unused. LDA_INFERENCE_CONVERGED : float Epsilon value used to check whether the inference step has sufficiently converged. lda_inference_max_iter : int Maximum number of iterations in the inference step. g : object Unused. Will be useful when the DIM model is implemented. g3_matrix: object Unused. Will be useful when the DIM model is implemented. g4_matrix: object Unused. Will be useful when the DIM model is implemented. g5_matrix: object Unused. Will be useful when the DIM model is implemented. Returns ------- float The optimal lower bound for the true posterior using the approximate distribution. c3 K|] \}}|V dSr#rrCs rrz'LdaPost.fit_lda_post..rDrrurvrr) rHr2rrMr@rr9r>r2r3update_phi_fixedr4rd)rHrrldaseqLDA_INFERENCE_CONVERGEDrLg g3_matrix g4_matrix g5_matrixrGr}r lhood_oldrrls rrzLdaPost.fit_lda_posts:> 9999999 E>  &&((     &&((  E> tT t%)__Z%F%F "DHdll e^ t t%)%:%::tTS\^gir%s%s "DHdl&&((GY.9u3DEFF 11 Ke?U6U K QJEI**,,DJE~ x x)-T)J)J&$,,% xD x)-)>)>z4QUW`bkmv)w)w&$,**,,EU!2y57H IJJI11 Ke?U6U K rc|jj}t|D]M}||}d}|jD]4\}} |||xx| |j||zz cc<|dz }5|||<N|S)aUpdate lda sequence sufficient statistics from an lda posterior. This is very similar to the :meth:`~gensim.models.ldaseqmodel.LdaPost.update_gamma` method and uses the same formula. Parameters ---------- time : int The time slice. doc : list of (int, float) Unused but kept here for backwards compatibility. The document set in the constructor (`self.doc`) is used instead. topic_suffstats : list of float Sufficient statistics for each topic. Returns ------- list of float The updated sufficient statistics for each topic. rr)rtrr8rr2) rHrrrnrrYtopic_ssr9r:r;s rrzLdaPost.update_lda_seq_sss,X( z"" * *A&q)HA"&(  !$'''548A;q>+AA'''Q!)OA  r)NNNNNN)rNrNNNN) rrrrrRr>r@rHrMrrrrrrxrxs$,$,$,$,L1&1&1&f,@@@...`NRaeKKKKZ     rrxc |\}}}}}}d}t|} d} d} d} d} d}||j|<|||j\|j|<|j|<|j|}|j|}td| dzD]z}||}||dz }||z } | | | zz } | ||dz |z||dz tj |||dz zz|j |dz z z z } d}|dkr {|jdkr1| d|jzz } | |d|dzd|z|jzz z } nd} | | z| z|z }|S)aFunction which we are optimising for minimizing obs. Parameters ---------- x : list of float The obs values for this word. sslm : :class:`~gensim.models.ldaseqmodel.sslm` The State Space Language Model for DTM. word_counts : list of int Total word counts for each time slice. totals : list of int of length `len(self.time_slice)` The totals for each time slice. mean_deriv_mtx : list of float Mean derivative for each time slice. word : int The word's ID. deriv : list of float Mean derivative for each time slice. Returns ------- list of float The value of the objective function evaluated at point `x`. rrrr\rurvrr) r$rrr rrrr8r4rr)rrr9r#rrrrr$rrr'r(r)r*rrrr mean_t_prevr}finals rrrs4>B:D+v~tUI AA C E E E EDHTN+/+A+A$H[+\+\(DIdOT]4( 9T?D}T"H 1a!e__  a1q5k {" s QU#f,va!e}rvfxXY{]^F^?_?_/_bfbklmpqlqbr/rrr E>   S A 3345Q$q')Q]T=P-PQQeme#e+ ,E Lrc|\}}}}}}||j|<|||j\|j|<|j|<d}|dkr||||||}nR|dkrL|tjtj tj tj tj |}tj|S)aDerivative of the objective function which optimises obs. Parameters ---------- x : list of float The obs values for this word. sslm : :class:`~gensim.models.ldaseqmodel.sslm` The State Space Language Model for DTM. word_counts : list of int Total word counts for each time slice. totals : list of int of length `len(self.time_slice)` The totals for each time slice. mean_deriv_mtx : list of float Mean derivative for each time slice. word : int The word's ID. deriv : list of float Mean derivative for each time slice. Returns ------- list of float The derivative of the objective function evaluated at point `x`. rurv)rrr rrr/compute_obs_deriv_fixedprr#rr9rr4negative) rrr9r#rrrrr}s rrrIs4>B:D+v~tUDHTN+/+A+A$H[+\+\(DIdOT]4( E ~N&&t[&.RWXX %N,, FAM18QVQ5EuNN ;u  r)rloggingnumpyr4 scipy.specialrrscipyrr rr gensim.modelsr getLoggerrr-SaveLoadr r9rxrrrrrrisZ//b********""""""""""""""  8 $ $k k k k k %.k k k \iiiii5>iiiX\\\\\en\\\@ GGGT&&&&&r