cdZddlZddlZddlmZmZmZddlZddlm Z m Z m Z ddl Z ddlmZddlmZddlmZddlmZmZmZdd lmZejeZd Ze je jj Z!eGd d Z"eGd dZ#dZ$dZ%dZ&dZ'dZ(dZ)d"dZ*d"dZ+dZ,dZ-d"dZ.dZ/dZ0dZ1dZ2GddeZ3Gd d!Z4dS)#a Ensemble Latent Dirichlet Allocation (eLDA), an algorithm for extracting reliable topics. The aim of topic modelling is to find a set of topics that represent the global structure of a corpus of documents. One issue that occurs with topics extracted from an NMF or LDA model is reproducibility. That is, if the topic model is trained repeatedly allowing only the random seed to change, would the same (or similar) topic representation be reliably learned. Unreliable topics are undesireable because they are not a good representation of the corpus. Ensemble LDA addresses the issue by training an ensemble of topic models and throwing out topics that do not reoccur across the ensemble. In this regard, the topics extracted are more reliable and there is the added benefit over many topic models that the user does not need to know the exact number of topics ahead of time. For more information, see the :ref:`citation section ` below, watch our `Machine Learning Prague 2019 talk `_, or view our `Machine Learning Summer School poster `_. Usage examples -------------- Train an ensemble of LdaModels using a Gensim corpus: .. sourcecode:: pycon >>> from gensim.test.utils import common_texts >>> from gensim.corpora.dictionary import Dictionary >>> from gensim.models import EnsembleLda >>> >>> # Create a corpus from a list of texts >>> common_dictionary = Dictionary(common_texts) >>> common_corpus = [common_dictionary.doc2bow(text) for text in common_texts] >>> >>> # Train the model on the corpus. corpus has to be provided as a >>> # keyword argument, as they are passed through to the children. >>> elda = EnsembleLda(corpus=common_corpus, id2word=common_dictionary, num_topics=10, num_models=4) Save a model to disk, or reload a pre-trained model: .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> >>> # Save model to disk. >>> temp_file = datapath("model") >>> elda.save(temp_file) >>> >>> # Load a potentially pretrained model from disk. >>> elda = EnsembleLda.load(temp_file) Query, the model using new, unseen documents: .. sourcecode:: pycon >>> # Create a new corpus, made of previously unseen documents. >>> other_texts = [ ... ['computer', 'time', 'graph'], ... ['survey', 'response', 'eps'], ... ['human', 'system', 'computer'] ... ] >>> other_corpus = [common_dictionary.doc2bow(text) for text in other_texts] >>> >>> unseen_doc = other_corpus[0] >>> vector = elda[unseen_doc] # get topic probability distribution for a document Increase the ensemble size by adding a new model. Make sure it uses the same dictionary: .. sourcecode:: pycon >>> from gensim.models import LdaModel >>> elda.add_model(LdaModel(common_corpus, id2word=common_dictionary, num_topics=10)) >>> elda.recluster() >>> vector = elda[unseen_doc] To optimize the ensemble for your specific case, the children can be clustered again using different hyperparameters: .. sourcecode:: pycon >>> elda.recluster(eps=0.2) .. _Citation: Citation -------- BRIGL, Tobias, 2019, Extracting Reliable Topics using Ensemble Latent Dirichlet Allocation [Bachelor Thesis]. Technische Hochschule Ingolstadt. Munich: Data Reply GmbH. Supervised by Alex Loosley. Available from: https://www.sezanzeb.de/machine_learning/ensemble_LDA/ N)ProcessPipe ProcessError)SetOptionalList)cosine) dataclass)utils)ldamodel ldamulticore basemodel)SaveLoadg?c|eZdZUeed<eeed<eeed<eeed<eed<eeed<dS)Topicis_coreneighboring_labelsneighboring_topic_indiceslabelnum_neighboring_labelsvalid_neighboring_labelsN)__name__ __module__ __qualname__bool__annotations__rintr9lib/python3.11/site-packages/gensim/models/ensemblelda.pyrr{sg MMMC   "3x''' C=!#h&&&&&rrcPeZdZUeed<eeeed<eed<eed<dS)Clustermax_num_neighboring_labelsrr num_coresN)rrrrrrrrrr r"r"sB ####SX&&& JJJNNNNNrr"c2|jo|j|jhkS)zCheck if the topic is a valid core, i.e. no neighboring valid cluster is overlapping with it. Parameters ---------- topic : :class:`Topic` topic to validate )rrr)topics r _is_valid_corer's = Ne< MNrcV|D]%}|jD]}||vr||&dS)zTRemove a label from every set in "neighboring_labels" for each core in ``clusters``.N)rremove)rclustersclusterneighboring_labels_sets r _remove_from_all_setsr-sV55&-&@ 5 5 ".. 5&--e444 555rcLtfd|jD|kS)zYCheck if the cluster has at least ``min_cores`` of cores that belong to no other cluster.cg|] }|hk Srr).0rrs r z,_contains_isolated_cores..s"ccc2D"ug-cccr)sumr)rr+ min_coress` r _contains_isolated_coresr4s/ ccccHbccc d dhq qqrc g}|D]\}}d}g}|D]1}t|j|}||j2d|D}|t |||t d|Dtdt ||S)aAggregate the labeled topics to a list of clusters. Parameters ---------- grouped_by_labels : dict of (int, list of :class:`Topic`) The return value of _group_by_labels. A mapping of the label to a list of each topic which belongs to the label. Returns ------- list of :class:`Cluster` It is sorted by max_num_neighboring_labels in descending order. There is one single element for each cluster. rc8g|]}t|dk|S)r)len)r0xs r r1z%_aggregate_topics..s'JJJAs1vvzJaJJJrc g|] }|j | Sr)rr0r&s r r1z%_aggregate_topics..sFFFU F5FFFr)r#rrr$zfound %s clusters) itemsmaxrappendrr"r7loggerinfo)grouped_by_labelsr*rtopicsr#rr&s r _aggregate_topicsrBsH*0022  v%&" @ @E),U-IKe)f)f &  % %e&> ? ? ? ?JJ);JJJ'A1FFfFFFGG         KK#S]]333 Orci}|D]M}|jrDt|j|_|j}||vrg||<|||N|S)aQGroup all the learned cores by their label, which was assigned in the cluster_model. Parameters ---------- cbdbscan_topics : list of :class:`Topic` A list of topic data resulting from fitting a :class:`~CBDBSCAN` object. After calling .fit on a CBDBSCAN model, the results can be retrieved from it by accessing the .results member, which can be used as the argument to this function. It is a list of infos gathered during the clustering step and each element in the list corresponds to a single topic. Returns ------- dict of (int, list of :class:`Topic`) A mapping of the label to a list of topics that belong to that particular label. Also adds a new member to each topic called num_neighboring_labels, which is the number of neighboring_labels of that topic. )rr7rrrr=)cbdbscan_topicsr@r&rs r _group_by_labelsrEsv& 33 = 3+.u/G+H+HE (KE-- .+-!%( e $ + +E 2 2 2 rc|D]-\}}||.|D]+}|r|~,dS)a4Close pipes and terminate processes. Parameters ---------- pipes : {list of :class:`multiprocessing.Pipe`} list of pipes that the processes use to communicate with the parent processes : {list of :class:`multiprocessing.Process`} list of worker processes N)closeis_alive terminate)pipes processesi parent_conn child_connprocesss r _teardownrPs$) Z            Grc|d}tj|ddd}||k}||d}||kS)z3Original masking method. Returns a new binary mask.Ngffffff?)npsortcumsum)a thresholdsorted_a largest_masssmallest_valids r mass_maskingr[sU wqzz$$B$H??$$y0Ll+B/N  rc|d}|tj|dddtt||zkS)z1Faster masking method. Returns a new binary mask.Ng)\(?rR)rSrTrr7)rVrWs r rank_maskingr] sE rwqzz$$B$CFFY$6 7 78 88rc6d}t||d}|D]0}d|_|j|krd|_t|j|1d|DD]9}|j}t |||rd|_"d|_t||:d|DS)z`Check which clusters from the cbdbscan step are significant enough. is_valid is set accordingly.c*|j|j|jfSN)r#r$r)r+s r _cluster_sort_keyz-_validate_clusters.._cluster_sort_keys173DgmSSrF)keyreverseNc g|] }|j | Sr`is_validr0r+s r r1z&_validate_clusters..'s WWWg>NWGWWWrTc g|] }|j | Srrergs r r1z&_validate_clusters../s G G Gg6F GG G G Gr)sortedrfr$r-rr4)r*r3rasorted_clustersr+rs r _validate_clustersrks TTTX+ > :#G  $G  !% 9 9 9 9 G G? G G GGrcfdt|D}t||}g}g}|}t|D]}t\} } d} ||dz kr|} nt|||z z } || dd| } | | | f} t t | }|||| | f||| z}#t$r/t d|t||wxYw|D]\} }| }| js/xj|z c_t!jd|D}n|}t!jj|g_|D]}|dS)aGenerate the topic models to form the ensemble in a multiprocessed way. Depending on the used topic model this can result in a speedup. Parameters ---------- ensemble: EnsembleLda the ensemble num_models : int how many models to train in the ensemble ensemble_workers : int into how many processes to split the models will be set to max(workers, num_models), to avoid workers that are supposed to train 0 models. to get maximum performance, set to the number of your cores, if non-parallelized models are being used in the ensemble (LdaModel). For LdaMulticore, the performance gain is small and gets larger for a significantly smaller corpus. In that case, ensemble_workers=2 can be used. cNg|]!}jt"Sr random_staterandint_MAX_RANDOM_STATEr0_ensembles r r1z4_generate_topic_models_multiproc..Ks+aaa!X*223DEEaaarrNtargetargscould not start process c6g|]}|Sr get_topics)r0ms r r1z4_generate_topic_models_multiproc..ys "B"B"Ba1<<>>"B"B"Br)rangeminrrr_generate_topic_models_workerr=startrr>errorrPrecvrGmemory_friendly_ttdatmsrS concatenatettdarI)rt num_modelsensemble_workers random_statesworkersrKrJnum_models_unhandledrLrMrNnum_subprocess_modelsrandom_states_for_workerrxrOrsanswerrs` r _generate_topic_models_multiprocr2sE2baaauU_O`O`aaaM"J//G I E% 7^^"&&& Z ! !  N%9 ! !$'(<! (L$M$M !$12F1F1G1G#HI_J_I_#` /1I:V %BNNNG   W % % % LL+z2 3 3 3 MMOOO $9 9     LL7A77 8 8 8 eY ' ' '   > > Q!!##,  LLF "LL>"B"B6"B"B"BCCDDD t'<== s AC..9D'c|fdt|D}t||ksJj}d}t|D]q}|||d<di|}t jj|g_j sxj |gz c_ r|j j _|j_dS)afTrain the topic models that form the ensemble. Parameters ---------- ensemble: EnsembleLda the ensemble num_models : int number of models to be generated random_states : list list of numbers or np.random.RandomState objects. Will be autogenerated based on the ensembles RandomState if None (default). NcNg|]!}jt"Srrnrrs r r1z*_generate_topic_models..s+eeea.667HIIeeerror)r~r7gensim_kw_argscopyget_topic_model_classrSrrr|rrstatesstatsr2 sstats_sumeta)rtrrkwargstmrLs` r _generate_topic_modelsrsfeeeeSXYcSdSdeee }   ++++  $ ) ) + +F B:   ! !!.q!1~ -X + + - - 7 7 7 7 r}}'GHH , ! LLRD LL(/--//H6HLLLrctd|dt||||jr||jn||j|dS)zzWrapper for _generate_topic_models to write the results into a pipe. This is intended to be used inside a subprocess.zspawned worker to generate z topic models)rtrrN)r>r?rrsendrrrG)rtrrpipes r rrs KKGjGGGHHHHS`aaaa$  (-     (,JJLLLLLrctjt|t|f}|jddkr|jddkrd}t |D]\}}|||} || } || z }t |D]]\} } ||z| kr d||| <| | } | t krd}nt| | }|||| <^td|z|jdz |jdz d}t d|d|d|S)aCalculate an (asymmetric) distance from each topic in ``ttda1`` to each topic in ``ttda2``. Parameters ---------- ttda1 and ttda2: 2D arrays of floats Two ttda matrices that are going to be used for distance calculation. Each row in ttda corresponds to one topic. Each cell in the resulting matrix corresponds to the distance between a topic pair. start_index : int this function might be used in multiprocessing, so start_index has to be set as ttda1 is a chunk of the complete ttda in that case. start_index would be 0 if ``ttda1 == self.ttda``. When self.ttda is split into two pieces, each 100 ttdas long, then start_index should be be 100. default is 0 masking_method: function masking_threshold: float Returns ------- 2D numpy.ndarray of floats Asymmetric distance matrix of size ``len(ttda1)`` by ``len(ttda2)``. rrudzthe given threshold of z covered on average z % of tokens) rSndarrayr7shape enumerater2&_COSINE_DISTANCE_CALCULATION_THRESHOLDr roundr>r?)ttda1ttda2 start_indexmasking_methodmasking_threshold distances avg_mask_sizettd1_idxttd1mask ttd1_maskedttd2_idxttd2 ttd2_maskeddistancepercents r +_calculate_asymmetric_distance_matrix_chunkrs: CJJE 344I {1~"kek!nq0"k (.. 9 9NHd!>$(9::Dt*K TXXZZ 'M#,E"2"2 9 9$k)X545Ih'1#4j ??$$(NN@ HH%k;??H08 (#H--! 9$m+ek!nr?rrrG) worker_id entire_ttda ttdas_sentn_ttdasrrrrdistance_chunks r "_asymmetric_distance_matrix_workerr s KKj)jj'jjjkkk  :#77 8E@%+ N IIy.)***JJLLLLLrc g}g}d}t|D]} t\}} d} ||dz krt||z } n%tt||z ||z z } |||| ||| f} t t | } || z }|| ||| f| #t$r/t d|t||wxYwg} |D]E\}}| \}}| | |F|D]} | tj| S)Nrrurvry)r~rr7rrrr=rrr>rrPrrGrIrSr)rrrrrKrJrrLrMrNrrxrOrrsrrs r /_calculate_assymetric_distance_matrix_multiprocr!s I EJ 7^^ &*ff #KGGaK Ok**Z7s;//*<1MNN{JIZ\fgD%GdSSSG ' !J   W % % % LL+z2 3 3 3 MMOOOO    LL7A77 8 8 8 eY ' ' '   I )) Q$/$4$4$6$6! >(((( >) $ $$s B8C9D c eZdZdZdddddddedddf dZd Zfd Zejje_d Z d Z dd Z dZ ddZ ddZddZdZdZdZdZdZdZedZxZS) EnsembleLdaaAEnsemble Latent Dirichlet Allocation (eLDA), a method of training a topic model ensemble. Extracts stable topics that are consistently learned across multiple LDA models. eLDA has the added benefit that the user does not need to know the exact number of topics the topic model should extract ahead of time. r N皙?ruTc  d| vrd| d<d| vrd| d<| d?| d7tdtj| d| d<| d| dt dt |t kr"t |tjr||_ n9tjtj d} || vrt d| ||_ ||_ | |_ ||_| |_| |_||_d|_tj| |_d|_d|_g|_t1jdt5| df|_d |_|dkrdS| ddSd | vr| d dkrdSd | vr| d dkrdStd |d |d|dkrt?|||ntA|||!|"|||#||$dS)a-Create and train a new EnsembleLda model. Will start training immediatelly, except if iterations, passes or num_models is 0 or if the corpus is missing. Parameters ---------- topic_model_class : str, topic model, optional Examples: * 'ldamulticore' (default, recommended) * 'lda' * ldamodel.LdaModel * ldamulticore.LdaMulticore ensemble_workers : int, optional Spawns that many processes and distributes the models from the ensemble to those as evenly as possible. num_models should be a multiple of ensemble_workers. Setting it to 0 or 1 will both use the non-multiprocessing version. Default: 1 num_models : int, optional How many LDA models to train in this ensemble. Default: 3 min_cores : int, optional Minimum cores a cluster of topics has to contain so that it is recognized as stable topic. epsilon : float, optional Defaults to 0.1. Epsilon for the CBDBSCAN clustering that generates the stable topics. ensemble_workers : int, optional Spawns that many processes and distributes the models from the ensemble to those as evenly as possible. num_models should be a multiple of ensemble_workers. Setting it to 0 or 1 will both use the nonmultiprocessing version. Default: 1 memory_friendly_ttda : boolean, optional If True, the models in the ensemble are deleted after training and only a concatenation of each model's topic term distribution (called ttda) is kept to save memory. Defaults to True. When False, trained models are stored in a list in self.tms, and no models that are not of a gensim model type can be added to this ensemble using the add_model function. If False, any topic term matrix can be suplied to add_model. min_samples : int, optional Required int of nearby topics for a topic to be considered as 'core' in the CBDBSCAN clustering. masking_method : function, optional Choose one of :meth:`~gensim.models.ensemblelda.mass_masking` (default) or :meth:`~gensim.models.ensemblelda.rank_masking` (percentile, faster). For clustering, distances between topic-term distributions are asymmetric. In particular, the distance (technically a divergence) from distribution A to B is more of a measure of if A is contained in B. At a high level, this involves using distribution A to mask distribution B and then calculating the cosine distance between the two. The masking can be done in two ways: 1. mass: forms mask by taking the top ranked terms until their cumulative mass reaches the 'masking_threshold' 2. rank: forms mask by taking the top ranked terms (by mass) until the 'masking_threshold' is reached. For example, a ranking threshold of 0.11 means the top 0.11 terms by weight are used to form a mask. masking_threshold : float, optional Default: None, which uses ``0.95`` for "mass", and ``0.11`` for masking_method "rank". In general, too small a mask threshold leads to inaccurate calculations (no signal) and too big a mask leads to noisy distance calculations. Defaults are often a good sweet spot for this hyperparameter. distance_workers : int, optional When ``distance_workers`` is ``None``, it defaults to ``os.cpu_count()`` for maximum performance. Default is 1, which is not multiprocessed. Set to ``> 1`` to enable multiprocessing. **gensim_kw_args Parameters for each gensim model (e.g. :py:class:`gensim.models.LdaModel`) in the ensemble. id2wordNcorpuszHno word id mapping provided; initializing from corpus, assuming identityzat least one of corpus/id2word must be specified, to establish input space dimensionality. Corpus should be provided using the `corpus` keyword argument.)ldar z\topic_model_class should be one of 'lda', 'ldamulticode' or a model inheriting from LdaModelrT iterationspassesz generating z topic models using z workersru)%r>warningr dict_from_corpus ValueErrortype issubclassr LdaModeltopic_model_classr LdaMulticorerrrdistance_workersrrclassic_model_representationget_random_staterorrrrSemptyr7r#asymmetric_distance_matrix_outdatedgetr?rr$_generate_asymmetric_distance_matrix_generate_topic_clusters_generate_stable_topicsgenerate_gensim_representation)selfrrr3epsilonrr min_samplesrrrrorkindss r __init__zEnsembleLda.__init__[sP N * -(,N9 % > ) ,'+N8 $ ) $ Y^H5M Y NNe f f f(-(>~h?W(X(XN9 % ) $ 1I -  ! " "d * >z:KXM^/_/_ >%6D " " ( , 9E!-  /&++<%=D "$,$8! 0!2,-1)"2<@@Ha^I%>!?!?@AA 370 ?  F   h ' '  F > ) n\.Ja.O  F ~ % .*Ba*G  F \*\\BR\\\]]] a  5 ,T:?O P P P P "4 4 4 4 11333 %%g{;;; $$Y/// ++-----rc ~|jd} tj|j}t ||j|_|`|`nu#t $r0td|jd|jd|YnrAttributeError)r instructionmodules r rz!EnsembleLda.get_topic_model_classs6  ! %  "01OPP)09V)W)W&211&    oT-Joo5ooaloo"    %T-J%%6%%"%%  %%s7A7B5<6B54B5c&|"|jj|_|jj|_t |ddd|d<tt|j |i|dS)Nignorer)r) rrrrrr frozensetrunionsuperrsave)rrxr __class__s r rzEnsembleLda.saves  % % ' ' L-1-C-ND *,0,B,KD )$VZZ"%=%=>>DDE\]]x%k4  %t6v66666rc"g|_d|_dS)zRemove the stored gensim models and only keep their ttdas. This frees up memory, but you won't have access to the individual models anymore if you intended to use them outside of the ensemble. TN)rrrs r convert_to_memory_friendlyz&EnsembleLda.convert_to_memory_friendlys $(!!!rctd|j}|dkr=d|jvr4|jd'|jdD]}|D] }||dz }||_|}t |}|dkr#tdd|_dS|j}|j |d<||d<d|d <| d i|}|j }|dkr%|j j }||_d} t|tt fr|t |dzg|z} n{t |jdkr| gg|z} t |jdkr2t%j| d dddf} t%j||z gg|z| z} || z} | |z} | t$j|j _ |||_|S) aCreate a gensim model from the stable topics. The returned representation is an Gensim LdaModel (:py:class:`gensim.models.LdaModel`) that has been instantiated with an A-priori belief on word probability, eta, that represents the topic-term distributions of any stable topics the were found by clustering over the ensemble of topic distributions. When no stable topics have been detected, None is returned. Returns ------- :py:class:`gensim.models.LdaModel` A Gensim LDA Model classic_model_representation for which: ``classic_model_representation.get_topics() == self.get_topics()`` zQgenerating classic gensim model representation based on results from the ensemblerrNruz\the model did not detect any stable topic. You can try to adjust epsilon: recluster(eps=...)r num_topicsraxisr)r>r?rrr|r7rrrrrrrr2 isinstancerfloatrrSarrayastypefloat32 sync_state) rrdocumenttoken stable_topicsnum_stable_topicsparamsrreta_sumnormalization_factorrs r rz*EnsembleLda.generate_gensim_representations  ghhh_  ? )x4+>> )tGZ[cGd ) /9 + +%++E%(*JJ+(DO))  ..  !  LL%   15D - F$))++u 0|x(Dt'A'A'C'C'M'Mf'M'M$+. ? )5;BFFHHJ(DO  cC< ( ( =Sq!122236GGGG39~~" <GGII;-*;;39~~! =(3777??111d7#;<< "x*7H*H)I(JM^(^__bii!55# 4:MM"*4M4M$*1$//111,H)++rct|tjtfstj|g}n)tj|}t |dksJ|jr9d}g}t|jtj tfr|}d}nt|dt|r:tj d|Dd}td|D}nut|dtjr0tj d|Dd}t |}n%tdt|d||xj|z c_n|xj|z c_ng}t|jtj tfrtd t|dt|r;|D]}|xj|jz c_tj d |Dd}nt|dtjrC|xj|z c_tj d |Dd}n%tdt|d|:||jzt |jkrt&d t |j|_t&d |jdt |jd|jjd|jdkr1td|jjdd|jddtj|j|d|_d|_dS)aAdd the topic term distribution array (ttda) of another model to the ensemble. This way, multiple topic models can be connected to an ensemble manually. Make sure that all the models use the exact same dictionary/idword mapping. In order to generate new stable topics afterwards, use: 2. ``self.``:meth:`~gensim.models.ensemblelda.EnsembleLda.recluster` The ttda of another ensemble can also be used, in that case set ``num_new_models`` to the ``num_models`` parameter of the ensemble, that means the number of classic models in the ensemble that generated the ttda. This is important, because that information is used to estimate "min_samples" for _generate_topic_clusters. If you trained this ensemble in the past with a certain Dictionary that you want to reuse for other models, you can get it from: ``self.id2word``. Parameters ---------- target : {see description} 1. A single EnsembleLda object 2. List of EnsembleLda objects 3. A single Gensim topic model (e.g. (:py:class:`gensim.models.LdaModel`) 4. List of Gensim topic models if memory_friendly_ttda is True, target can also be: 5. topic-term-distribution-array example: [[0.1, 0.1, 0.8], [...], ...] [topic1, topic2, ...] with topic being an array of probabilities: [token1, token2, ...] token probabilities in a single topic sum to one, therefore, all the words sum to len(ttda) num_new_models : integer, optional the model keeps track of how many models were used in this ensemble. Set higher if ttda contained topics from more than one model. Default: None, which takes care of it automatically. If target is a 2D-array of float values, it assumes 1. If the ensemble has ``memory_friendly_ttda`` set to False, then it will always use the number of models in the target parameter. rrucg|] }|j Srrr0rts r r1z)EnsembleLda.add_model..&L&L&Lx}&L&L&Lrrcg|] }|j Sr)rr s r r1z)EnsembleLda.add_model..s*V*V*V88+>*V*V*Vrc6g|]}|Srr{r0models r r1z)EnsembleLda.add_model..$&N&N&Neu'7'7'9'9&N&N&Nrz6target is of unknown type or a list of unknown types: Nzttda arrays cannot be added to ensembles, for which memory_friendly_ttda=False, you can call convert_to_memory_friendly, but it will discard the stored gensim models and only keep the relevant topic term distributions from them.cg|] }|j Srr r s r r1z)EnsembleLda.add_model..r rc6g|]}|Srr{rs r r1z)EnsembleLda.add_model..rrztnum_new_models will be ignored. num_models should match the number of stored models for a memory unfriendly ensemblezensemble contains z models and z topics nowz4target ttda dimensions do not match. Topics must be rRz but was z elements largeT)rrSrlistrr7rdtypernumberrrr2rBaseTopicModelrrrtolistr>r?rrr=r)rrwnum_new_modelsdetected_num_modelsrrts r add_modelzEnsembleLda.add_modelrsd&2:t"455 #Xvh''FFXf%%Fv;;? " " "  $A ,#$ D&,++-- 5/ABB m&'##F1ItDzz22 m~&L&LV&L&L&LSTUUU&)*V*Vv*V*V*V&W&W##F1Iy'?@@ m~&N&Nv&N&N&NUVWWW&)&kk##!!kZ^_efg_hZiZi!k!klll 2#66>1D&,++-- 5/ABB m \F1ItDzz22 m &--HHH ,HHH~&L&LV&L&L&LSTUUUF1Iy'?@@ mFMMOO+~&N&Nv&N&N&NUVWWW!!kZ^_efg_hZiZi!k!klll nt.NRUVZV^R_R_._  E"$(mmDO aaac$)nnaaabbb 9?1 A . "tyWYGZ""eieopres"""  IdiA666 48000rc|j}d|_tdt |jdt |jd|5|dkr/t |j|jd|j|j|_ dS|tj }t||j|j|j |_ dS) aCalculate the pairwise distance matrix for all the ttdas from the ensemble. Returns the asymmetric pairwise distance matrix that is used in the DBSCAN clustering. Afterwards, the model needs to be reclustered for this generated matrix to take effect. Fz generating a z x z asymmetric distance matrix...Nrurr)rrrr) rrr>r?r7rrrrasymmetric_distance_matrixos cpu_countr)rrs r rz0EnsembleLda._generate_asymmetric_distance_matrixs'490 eC NNees49~~eeefff  7a< .Yii#2"&"8 ///D + + + ),...] I#2"&"8 ///D + + +rc|3t|jdz }td|ntdt |||_|j|jdS)aRun the CBDBSCAN algorithm on all the detected topics and label them with label-indices. The final approval and generation of stable topics is done in ``_generate_stable_topics()``. Parameters ---------- eps : float dbscan distance scale min_samples : int, optional defaults to ``int(self.num_models / 2)``, dbscan min neighbours threshold required to consider a topic to be a core. Should scale with the number of models, ``self.num_models`` Nz6fitting the clustering model, using %s for min_sampleszfitting the clustering modelepsr)rrr>r?CBDBSCAN cluster_modelfitrrr#rs r rz$EnsembleLda._generate_topic_clusterss  8do122K KKPR] ^ ^ ^ ^ KK6 7 7 7%#;GGG t>?????rc |dkrd}|Rtdtdt|jdz dz}td|ntd|jj}t|}t|}t||}d|D|D]}fd |j D|_ tjt|}|j|}tjd |D|tj} t'| } tj| t'|jf} t-| D]L\} tjfd t-|D} | d | | <M||_| |_td t'| dS)a{Generate stable topics out of the clusters. The function finds clusters of topics using a variant of DBScan. If a cluster has enough core topics (c.f. parameter ``min_cores``), then this cluster represents a stable topic. The stable topic is specifically calculated as the average over all topic-term distributions of the core topics in the cluster. This function is the last step that has to be done in the ensemble. After this step is complete, Stable topics can be retrieved afterwards using the :meth:`~gensim.models.ensemblelda.EnsembleLda.get_topics` method. Parameters ---------- min_cores : int Minimum number of core topics needed to form a cluster that represents a stable topic. Using ``None`` defaults to ``min_cores = min(3, max(1, int(self.num_models /4 +1)))`` rruNrz0generating stable topics, using %s for min_coreszgenerating stable topicsch|] }|j Srrrgs r z6EnsembleLda._generate_stable_topics..WsLLL' LLLrch|]}|v| Srr)r0rvalid_cluster_labelss r r,z6EnsembleLda._generate_stable_topics..Zs5...00....rcg|] }|j Srr+r:s r r1z7EnsembleLda._generate_stable_topics..bs J J J J J Jrc2g|]\}}|k|Srr)r0tr&r topic_labelss r r1z7EnsembleLda._generate_stable_topics..ks/)t)t)tHAu[ghi[jns[s)t%)t)t)trrzfound %s stable topics)rr<rrr>r?r%resultsrErBrkrrrS vectorizer'rruniquer7rrrmeanvalid_clustersr)rr3rDr@r*r7r&valid_core_mask valid_topics unique_labelsrr label_indextopics_of_clusterrr2r.s @@@r rz#EnsembleLda._generate_stable_topics4s)( > I  4As1c$/A*=*A&B&BCCDDI KKJI V V V V KK2 3 3 3,4,_==$%677+Hi@@LL^LLL$  E....#(#;...E * * 7",~66GGy1 x J J/ J J JKKO\  ,//  .."3S5F5F!GHH #,M":": H H K ")t)t)t)t)t ,@W@W)t)t)t u u ):)?)?Q)?)G)GM+ & &,* ,c-.@.@AAAAArc|jr.td|||||||dS)a]Reapply CBDBSCAN clustering and stable topic generation. Stable topics can be retrieved using :meth:`~gensim.models.ensemblelda.EnsembleLda.get_topics`. Parameters ---------- eps : float epsilon for the CBDBSCAN algorithm, having the same meaning as in classic DBSCAN clustering. default: ``0.1`` min_samples : int The minimum number of samples in the neighborhood of a topic to be considered a core in CBDBSCAN. default: ``int(self.num_models / 2)`` min_cores : int how many cores a cluster has to have, to be treated as stable topic. That means, how many topics that look similar have to be present, so that the average topic in those is used as stable topic. default: ``min(3, max(1, int(self.num_models /4 +1)))`` z7asymmetric distance matrix is outdated due to add_modelN)rr>r?rrrr)rr#rr3s r reclusterzEnsembleLda.reclusterss|(  3 8 KKQ R R R  5 5 7 7 7 %%c;777 $$Y/// ++-----rc|jS)zReturn only the stable topics from the ensemble. Returns ------- 2D Numpy.numpy.ndarray of floats List of stable topic term distributions )rrs r r|zEnsembleLda.get_topicss !!rc|j6t|jdkrtdtddS)z[Check if stable topics and the internal gensim representation exist. Raise an error if not.Nrzno stable topic was detectedz*use generate_gensim_representation() first)rr7rrrs r _ensure_gensim_representationz)EnsembleLda._ensure_gensim_representationsQ  , O4%&&!+ O !?@@@ !MNNN  O OrcD||j|S)z/See :meth:`gensim.models.LdaModel.__getitem__`.)rAr)rrLs r __getitem__zEnsembleLda.__getitem__s" **,,,033rcN||jj|i|S)z-See :meth:`gensim.models.LdaModel.inference`.)rAr inferencerposargsrs r rEzEnsembleLda.inferences/ **,,,:t0:GNvNNNrcN||jj|i|S)z2See :meth:`gensim.models.LdaModel.log_perplexity`.)rArlog_perplexityrFs r rIzEnsembleLda.log_perplexitys/ **,,,?t0?SFSSSrcN||jj|i|S)z0See :meth:`gensim.models.LdaModel.print_topics`.)rAr print_topicsrFs r rKzEnsembleLda.print_topicss/ **,,,=t0=wQ&QQQrc|jdS)zUReturn the :py:class:`gensim.corpora.dictionary.Dictionary` object used in the model.r)rrs r rzEnsembleLda.id2words"9--rr`)rN)rNN)rrr__doc__r[rrrrrrrrrrr>r|rArCrErIrKpropertyr __classcell__)rs@r rrSs%3q!$\TT Q.Q.Q.Q.f&&&277777=(DL)))W,W,W,rF8F8F8F8P!!!F@@@@.=B=B=B=B~....H " " "OOO444 OOO TTT RRR ..X.....rrceZdZdZdZdZdS)r$a A Variation of the DBSCAN algorithm called Checkback DBSCAN (CBDBSCAN). The algorithm works based on DBSCAN-like parameters 'eps' and 'min_samples' that respectively define how far a "nearby" point is, and the minimum number of nearby points needed to label a candidate datapoint a core of a cluster. (See https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html). The algorithm works as follows: 1. (A)symmetric distance matrix provided at fit-time (called 'amatrix'). For the sake of example below, assume the there are only five topics (amatrix contains distances with dim 5x5), T_1, T_2, T_3, T_4, T_5: 2. Start by scanning a candidate topic with respect to a parent topic (e.g. T_1 with respect to parent None) 3. Check which topics are nearby the candidate topic using 'self.eps' as a threshold and call them neighbours (e.g. assume T_3, T_4, and T_5 are nearby and become neighbours) 4. If there are more neighbours than 'self.min_samples', the candidate topic becomes a core candidate for a cluster (e.g. if 'min_samples'=1, then T_1 becomes the first core of a cluster) 5. If candidate is a core, CheckBack (CB) to find the fraction of neighbours that are either the parent or the parent's neighbours. If this fraction is more than 75%, give the candidate the same label as its parent. (e.g. in the trivial case there is no parent (or neighbours of that parent), a new incremental label is given) 6. If candidate is a core, recursively scan the next nearby topic (e.g. scan T_3) labeling the previous topic as the parent and the previous neighbours as the parent_neighbours - repeat steps 2-6: 2. (e.g. Scan candidate T_3 with respect to parent T_1 that has parent_neighbours T_3, T_4, and T_5) 3. (e.g. T5 is the only neighbour) 4. (e.g. number of neighbours is 1, therefore candidate T_3 becomes a core) 5. (e.g. CheckBack finds that two of the four parent and parent neighbours are neighbours of candidate T_3. Therefore the candidate T_3 does NOT get the same label as its parent T_1) 6. (e.g. Scan candidate T_5 with respect to parent T_3 that has parent_neighbours T_5) The CB step has the effect that it enforces cluster compactness and allows the model to avoid creating clusters for unstable topics made of a composition of multiple stable topics. c"||_||_dS)aCreate a new CBDBSCAN object. Call fit in order to train it on an asymmetric distance matrix. Parameters ---------- eps : float epsilon for the CBDBSCAN algorithm, having the same meaning as in classic DBSCAN clustering. min_samples : int The minimum number of samples in the neighborhood of a topic to be considered a core in CBDBSCAN. Nr"r's r rzCBDBSCAN.__init__s&rcd_dtt|D|t jddt dD}t|d}d|Dd fd tdkr3 d}|tdk3_ d S) z5Apply the algorithm to an asymmetric distance matrix.rc g|];}tdttddt.sW $ $ $  #&55*-%%'(),     $ $ $ rrucg|] \}}||f Srrr0indexrs r r1z CBDBSCAN.fit.. s!!o!o!ox8U"3!o!o!orrc|dSNrr)rs r zCBDBSCAN.fit.. s \def\grrbcg|]\}}|Srr)r0rrWs r r1z CBDBSCAN.fit.. s!]!]!]OHe%!]!]!]rNctdt|Dd} fd|D}t|}| jkrd |_| j} xjdz c_nF|| jk}|dkr j} xjdz c_| |_|D]u} |j& | ||||gz |j | |j |vdS|d |_dS| |_dS) aExtend the cluster in one direction. Results are accumulated to ``self.results``. Parameters ---------- topic_index : int The topic that might be added to the existing cluster, or which might create a new cluster if necessary. current_label : int The label of the cluster that might be suitable for ``topic_index`` cg|] \}}||f SrrrVs r r1z4CBDBSCAN.fit..scan_topic..s1'xu%rc|dSrYr)r8s r rZz2CBDBSCAN.fit..scan_topic.. s adrr[c0g|]\}}|jk|Sr)r#)r0rrWrs r r1z4CBDBSCAN.fit..scan_topic.."s,(n(n(n?8UZbeiemZm(n(n(n(nrTNrug?rR) rirr7rr next_labelr#r6rr)raddr) topic_index current_labelparent_neighborsneighbors_sortedrnum_neighboring_topicsclose_parent_neighbors_maskneighboring_topic_index amatrix_copyordered_min_similarity scan_topicrtopic_clustering_resultss r rlz CBDBSCAN.fit..scan_topics &+4\+5N+O+O#N     )o(n(n(nFV(n(n(n %%()B%C%C "&)99' P@D(5=!-$(OMOOq(OOO3?{2KL\2]`d`h2h/27799D@-(, 1,>K(5;/Hll+/0GHNv.556MNNN" #:MKdhsgtKtuuu,-DE_ccdoppp,-DEX\\]jkkkkll!PBD,[9???BO,[9???r)NN) rar~r7rrS fill_diagonalrrripopr3) ramatrixmin_distance_per_topicmin_distance_per_topic_sortednext_topic_indexrjrkrlrms ` @@@@r r&z CBDBSCAN.fitsf $ $ S\\** $ $ $  ||~~  q)))!o!o9UaUeUeklUeUmUmKnKn!o!o!o(./EKgKg(h(h(h%!]!]?\!]!]!]A PA PA PA PA PA PA PA PA PA PH())Q. )599!<<  J' ( ( (())Q. )0 rN)rrrrMrr&rrr r$r$sB!!F ' ' '`0`0`0`0`0rr$r`)5rMloggingrmultiprocessingrrrrtypingrrrnumpyrSscipy.spatial.distancer dataclassesr gensimr gensim.modelsr r r gensim.utilsr getLoggerrr>riinfoint32r<rqrr"r'r-r4rBrErPr[r]rkrrrrrrrr$rrr rsWWp 7777777777&&&&&&&&&&))))))!!!!!!;;;;;;;;;;!!!!!!  8 $ $*.&BHRX&&* ''''''' '   O O O555rrr $$$NB(9999HHH:MMM`&&&&R0CCCL0/%/%/%dn .n .n .n .n .(n .n .n .bR0R0R0R0R0R0R0R0R0R0r