cf >dZddlZddlZddlmZddlZddlm Z m Z ddlm Z ddl m Z mZmZmZmZddlmZejeZdhZhd Zed d Zee jejejejee jejej ejee jej!ej"ejee j#ejej$ejee j#ejej$ejd Z%d dddddZ&Gdde j'Z(dS)aCalculate topic coherence for topic models. This is the implementation of the four stage topic coherence pipeline from the paper `Michael Roeder, Andreas Both and Alexander Hinneburg: "Exploring the space of topic coherence measures" `_. Typically, :class:`~gensim.models.coherencemodel.CoherenceModel` used for evaluation of topic models. The four stage pipeline is basically: * Segmentation * Probability Estimation * Confirmation Measure * Aggregation Implementation of this pipeline allows for the user to in essence "make" a coherence measure of his/her choice by choosing a method in each of the pipelines. See Also -------- :mod:`gensim.topic_coherence` Internal functions for pipelines. N) namedtuple) interfacesmatutils)utils) segmentationprobability_estimationdirect_confirmation_measureindirect_confirmation_measure aggregation)unique_ids_from_segmentsu_mass>c_vc_ucic_w2vc_npmiCoherence_Measurezseg, prob, conf, aggr)r rrrrn )rrrrr ceZdZdZ d!dZed"dZed"dZed Z d Z e d Z e j d Z e d Zej dZe dZe dZej dZdZdZdZdZdZedZdZd#dZd$dZdZdZdZdZd ZdS)%CoherenceModelaZObjects of this class allow for building and maintaining a model for topic coherence. Examples --------- One way of using this feature is through providing a trained topic model. A dictionary has to be explicitly provided if the model does not contain a dictionary already .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary >>> from gensim.models.ldamodel import LdaModel >>> from gensim.models.coherencemodel import CoherenceModel >>> >>> model = LdaModel(common_corpus, 5, common_dictionary) >>> >>> cm = CoherenceModel(model=model, corpus=common_corpus, coherence='u_mass') >>> coherence = cm.get_coherence() # get coherence value Another way of using this feature is through providing tokenized topics such as: .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary >>> from gensim.models.coherencemodel import CoherenceModel >>> topics = [ ... ['human', 'computer', 'system', 'interface'], ... ['graph', 'minors', 'trees', 'eps'] ... ] >>> >>> cm = CoherenceModel(topics=topics, corpus=common_corpus, dictionary=common_dictionary, coherence='u_mass') >>> coherence = cm.get_coherence() # get coherence value Nrc Z||td||td|_|||td|;t|jtjrtd|j_n|_|_|_jtj_|_ |_ |tvrRt j |dr|_ njj fdj D_ nJtd||d kr|n1|tvrj td |ntd || _|_d_d_|_| d kr| n#t)d t+jd z _dS) aj Parameters ---------- model : :class:`~gensim.models.basemodel.BaseTopicModel`, optional Pre-trained topic model, should be provided if topics is not provided. Currently supports :class:`~gensim.models.ldamodel.LdaModel`, :class:`~gensim.models.ldamulticore.LdaMulticore`. Use `topics` parameter to plug in an as yet unsupported model. topics : list of list of str, optional List of tokenized topics, if this is preferred over model - dictionary should be provided. texts : list of list of str, optional Tokenized texts, needed for coherence models that use sliding window based (i.e. coherence=`c_something`) probability estimator . corpus : iterable of list of (int, number), optional Corpus in BoW format. dictionary : :class:`~gensim.corpora.dictionary.Dictionary`, optional Gensim dictionary mapping of id word to create corpus. If `model.id2word` is present, this is not needed. If both are provided, passed `dictionary` will be used. window_size : int, optional Is the size of the window to be used for coherence measures using boolean sliding window as their probability estimator. For 'u_mass' this doesn't matter. If None - the default window sizes are used which are: 'c_v' - 110, 'c_uci' - 10, 'c_npmi' - 10. coherence : {'u_mass', 'c_v', 'c_uci', 'c_npmi'}, optional Coherence measure to be used. Fastest method - 'u_mass', 'c_uci' also known as `c_pmi`. For 'u_mass' corpus should be provided, if texts is provided, it will be converted to corpus using the dictionary. For 'c_v', 'c_uci' and 'c_npmi' `texts` should be provided (`corpus` isn't needed) topn : int, optional Integer corresponding to the number of top words to be extracted from each topic. processes : int, optional Number of processes to use for probability estimation phase, any value less than 1 will be interpreted as num_cpus - 1. Nz*One of model or topics has to be provided.z7dictionary has to be provided if topics are to be used.z*One of texts or corpus has to be provided.zThe associated dictionary should be provided with the corpus or 'id2word' for topic model should be set as the associated dictionary.rcDg|]}j|S) dictionarydoc2bow).0textselfs z+CoherenceModel.__init__..s)TTTt66t<<TTTzQEither 'corpus' with 'dictionary' or 'texts' should be provided for %s coherence.rz,'texts' should be provided for %s coherence.z(%s coherence is not currently supported.) ValueError keyed_vectors isinstanceid2wordrFakeDictr coherence window_sizeSLIDING_WINDOW_SIZEStextscorpusBOOLEAN_DOCUMENT_BASED is_corpusSLIDING_WINDOW_BASED_topn_model _accumulator_topicstopicsmaxmp cpu_count processes) r!modelr7r.r/rr,r'r+topnr;s ` r"__init__zCoherenceModel.__init__~s(J  XV XIJJ J  XJ XVWW W*  KU Kv KIJJ J  )%-88 0 STTT#(-(DO#&   D3DNCD   . . Tv&&q) @$  @TTTTTTT  45>@@@ ' ! Tm T  . . Tz \ !OQZ[[[ \GSS S     &/1nT#aRSAS:T:Tr$c Xfd|D}|d<|d<j|fi|S)aInitialize a CoherenceModel with estimated probabilities for all of the given models. Use :meth:`~gensim.models.coherencemodel.CoherenceModel.for_topics` method. Parameters ---------- models : list of :class:`~gensim.models.basemodel.BaseTopicModel` List of models to evaluate coherence of, each of it should implements :meth:`~gensim.models.basemodel.BaseTopicModel.get_topics` method. dictionary : :class:`~gensim.corpora.dictionary.Dictionary` Gensim dictionary mapping of id word. topn : int, optional Integer corresponding to the number of top words to be extracted from each topic. kwargs : object Sequence of arguments, see :meth:`~gensim.models.coherencemodel.CoherenceModel.for_topics`. Return ------ :class:`~gensim.models.coherencemodel.CoherenceModel` CoherenceModel with estimated probabilities for all of the given models. Example ------- .. sourcecode:: pycon >>> from gensim.test.utils import common_corpus, common_dictionary >>> from gensim.models.ldamodel import LdaModel >>> from gensim.models.coherencemodel import CoherenceModel >>> >>> m1 = LdaModel(common_corpus, 3, common_dictionary) >>> m2 = LdaModel(common_corpus, 5, common_dictionary) >>> >>> cm = CoherenceModel.for_models([m1, m2], common_dictionary, corpus=common_corpus, coherence='u_mass') c>g|]}|Sr)top_topics_as_word_lists)rr<clsrr=s r"r#z-CoherenceModel.for_models..s+\\\E#..uj$GG\\\r$rr=) for_topics)rBmodelsrr=kwargsr7s` `` r" for_modelszCoherenceModel.for_modelssVF]\\\\\U[\\\)|vs~f/////r$c js(djD_g}|D]<}t j||d}fd|D}||=|S)aGet `topn` topics as list of words. Parameters ---------- model : :class:`~gensim.models.basemodel.BaseTopicModel` Pre-trained topic model. dictionary : :class:`~gensim.corpora.dictionary.Dictionary` Gensim dictionary mapping of id word. topn : int, optional Integer corresponding to the number of top words to be extracted from each topic. Return ------ list of list of str Top topics in list-of-list-of-words format. ci|]\}}|| Srr)rkvs r" z;CoherenceModel.top_topics_as_word_lists..s"P"P"PDAq1a"P"P"Pr$Tr=reversec*g|]}j|Sr)id2token)r_idrs r"r#z;CoherenceModel.top_topics_as_word_lists..s!AAACz*3/AAAr$)rOtoken2iditems get_topicsrargsortappend)r<rr= str_topicstopicbestnbeststrs ` r"rAz'CoherenceModel.top_topics_as_word_listss&" Q"P"PJ4G4M4M4O4O"P"P"PJ  %%'' ' 'E$UtDDDEAAAA5AAAG   g & & & &r$c |stdtd|Drtdd}|D]$}|D]}t|t|} %t |d||}t j|}tj dt|t|td|gt|d|}| ||_ |S) aInitialize a CoherenceModel with estimated probabilities for all of the given topics. Parameters ---------- topics_as_topn_terms : list of list of str Each element in the top-level list should be the list of topics for a model. The topics for the model should be a list of top-N words, one per topic. Return ------ :class:`~gensim.models.coherencemodel.CoherenceModel` CoherenceModel with estimated probabilities for all of the given models. zlen(topics) must be > 0.c3<K|]}t|dkVdS)rN)len)r topic_listss r" z,CoherenceModel.for_topics..1s/MMs;1$MMMMMMr$z%found empty topic listing in `topics`rr=z.Number of relevant terms for all %d models: %d)r7r=r) r&anyr8r\minpoprflattenlogginginforestimate_probabilitiesr=)rBtopics_as_topn_termsrEr= topic_listrW super_topiccms r"rCzCoherenceModel.for_topicss* $ 9788 8 MM8LMMM M M FDEE E. - -J# - -4U,, -6::fd++T22m$899  < $ % %s;'7'7 9 9 9 RK=s;7G7G R R6 R R !!### r$c*t|jSN)strmeasurer!s r"__str__zCoherenceModel.__str__Ds4<   r$c|jS)zGet `self._model` field. Return ------ :class:`~gensim.models.basemodel.BaseTopicModel` Used model. )r4rns r"r<zCoherenceModel.modelGs {r$c|||_|2|}||||_dSdS)zSet `self._model` field. Parameters ---------- model : :class:`~gensim.models.basemodel.BaseTopicModel` Input model. N)r4 _get_topics_update_accumulatorr6)r!r< new_topicss r"r<zCoherenceModel.modelSsM  &))++J  $ $Z 0 0 0%DLLL & &r$c|jS)zGet number of top words `self._topn`. Return ------ int Integer corresponding to the number of top words. r3rns r"r=zCoherenceModel.topncs zr$ct|jd}||k}|j||_|r|j|_dSdS|rt d|z||_dS)zSet number of top words `self._topn`. Parameters ---------- topn : int Number of top words. rNz7Model unavailable and topic sizes are less than topn=%d)r\r6r<r3r4r&)r!r=current_topic_lengthrequires_expansions r"r=zCoherenceModel.topnos~ #4<?331D8 : DJ! )![  ) )" c !Z]a!abbbDJJJr$c&t|jS)zMake pipeline, according to `coherence` parameter value. Return ------ namedtuple Pipeline that contains needed functions/method for calculated coherence. )COHERENCE_MEASURESr+rns r"rmzCoherenceModel.measures"$.11r$c~tjdjkrfdjDSjS)zGet topics `self._topics`. Return ------ list of list of str Topics as list of tokens. rc.g|]}|djSrkrv)rrWr!s r"r#z)CoherenceModel.topics..s$AAA5E+4:+&AAAr$)r\r6r3rns`r"r7zCoherenceModel.topicssF t|A  $* , AAAADLAAA A< r$cp|Yg}|D],}||}||-|j td|jn>|j5|}td|jnd}||||_dS)zSet topics `self._topics`. Parameters ---------- topics : list of list of str Topics. NzJThe currently set model '%s' may be inconsistent with the newly set topicsz(Setting topics to those of the model: %s) _ensure_elements_are_idsrUr<loggerwarningrrdebugrsr6)r!r7rtrWtopic_token_idss r"r7zCoherenceModel.topicss  J 3 3"&"?"?"F"F!!/2222z `J   Z ))++J LLCTZ P P P PJ   ,,,! r$c*fd|D}fd|D}t|t|krtj|St|t|krtj|Std)NcPg|]"}|jjvjj|#Sr)rrQ)rtr!s r"r#z;CoherenceModel._ensure_elements_are_ids..s2ggg1doNfIfg4?3A6gggr$c&g|] }|jv |Sr)r)rir!s r"r#z;CoherenceModel._ensure_elements_are_ids..s%AAAaA,@AAAAr$zEunable to interpret topic as either a list of tokens or a list of ids)r\nparrayr&)r!rWids_from_tokens ids_from_idss` r"rz'CoherenceModel._ensure_elements_are_idssgggggggAAAA5AAA   #l"3"3 3 f8O,, ,   _!5!5 5 f8L)) )dee er$cv||r#tdd|_dSdS)NzEWiping cached accumulator since it does not contain all relevant ids.)_relevant_ids_will_differrrr5r!rts r"rsz"CoherenceModel._update_accumulatorsC  ) )* 5 5 % LL` a a a $D    % %r$c|j||sdSt|j|}|jj| S)NF)r5_topics_differr rmseg relevant_ids issuperset)r!rtnew_sets r"rz(CoherenceModel._relevant_ids_will_differsa   D,?,? ,K,K 5*4<+;+;J+G+GHH$1<g|]}tj|dS)TrL)rrT)rrWr=s r"r#z9CoherenceModel._get_topics_from_model..s8EJ T4@@@r$zmThis topic model is not currently supported. Supported topic models should implement the `get_topics` method.)rSAttributeErrorr&)r<r=s `r"rz%CoherenceModel._get_topics_from_modelss" >  ""  > > >=>> > >s#=c@|j|jS)zSegment topic, alias for `self.measure.seg(self.topics)`. Return ------ list of list of pair Segmented topics. )rmrr7rns r"segment_topicszCoherenceModel.segment_topicss| ,,,r$cL||}|jtvr&|j|j||_nTt|j||j |j |j }|jdkr |j |d<|jjdi||_|jS)anAccumulate word occurrences and co-occurrences from texts or corpus using the optimal method for the chosen coherence metric. Notes ----- This operation may take quite some time for the sliding window based coherence methods. Parameters ---------- segmented_topics : list of list of pair, optional Segmented topics, typically produced by :meth:`~gensim.models.coherencemodel.CoherenceModel.segment_topics`. Return ------ :class:`~gensim.topic_coherence.text_analysis.CorpusAccumulator` Corpus accumulator. N)r.segmented_topicsrr,r;rr<r) rr+r0rmprobr/r5dictr.rr,r;r')r!rrEs r"rez%CoherenceModel.estimate_probabilitiess&  5#2244  >3 3 < $ 1 1$+?O P PD  j3C?8H.***F~( 5"&"4w 1 1 ; ;F ; ;D   r$FcZ|j}|||j}|j||t ||}|jtvs |jdkrn.|jdkr|j|d<d|d<d|d <n|jd k|d <|j||jfi|S) aGet list of coherence values for each topic based on pipeline parameters. Parameters ---------- segmented_topics : list of list of (int, number) Topics. with_std : bool, optional True to also include standard deviation across topic segment sets in addition to the mean coherence for each topic. with_support : bool, optional True to also include support across topic segments. The support is defined as the number of pairwise similarity comparisons were used to compute the overall topic coherence. Return ------ list of float Sequence of similarity measure for each topic. N)with_std with_supportrrr7nlrrmr%gammar normalize) rmrr7r5rerr+r0conf)r!rrrrmrEs r"get_coherence_per_topicz&CoherenceModel.get_coherence_per_topic&s(,  8&{{4;77    :  ' '(8 9 9 9xlCCC >3 3 ?t~7P ?  ^u $ ?#{F8  %F9 F7OO#'>X#=F; w|,d.?JJ6JJJr$c6|j|S)aAggregate the individual topic coherence measures using the pipeline's aggregation function. Use `self.measure.aggr(topic_coherences)`. Parameters ---------- topic_coherences : list of float List of calculated confirmation measure on each set in the segmented topics. Returns ------- float Arithmetic mean of all the values contained in confirmation measures. )rmaggr)r!topic_coherencess r"aggregate_measuresz!CoherenceModel.aggregate_measuresLs|  !1222r$cT|}||S)zGet coherence value based on pipeline parameters. Returns ------- float Value of coherence. )rr)r!confirmed_measuress r" get_coherencezCoherenceModel.get_coherence]s+"99;;&&'9:::r$cJfd|D}|S)aQCompare topic models by coherence value. Parameters ---------- models : :class:`~gensim.models.basemodel.BaseTopicModel` Sequence of topic models. Returns ------- list of (float, float) Sequence of pairs of average topic coherence and average coherence. cFg|]}|jSr)rr=)rr<r!s r"r#z1CoherenceModel.compare_models..ws+ZZZ%33E49EEZZZr$)compare_model_topics)r!rD model_topicss` r"compare_modelszCoherenceModel.compare_modelsis4[ZZZSYZZZ ((666r$c|j}|j} ||}||_||_n#||_||_wxYw|S)aPerform the coherence evaluation for each of the models. Parameters ---------- model_topics : list of list of str list of list of words for the model trained with that number of topics. Returns ------- list of (float, float) Sequence of pairs of average topic coherence and average coherence. Notes ----- This first precomputes the probabilities once, then evaluates coherence for each model. Since we have already precomputed the probabilities, this simply involves using the accumulated stats in the :class:`~gensim.models.coherencemodel.CoherenceModel` to perform the evaluations, which should be pretty quick. )r6r=_compare_model_topicsr7)r!r orig_topics orig_topn coherencess r"rz#CoherenceModel.compare_model_topicszs]*l I  "33LAAJ%DK!DII&DK!DI ! ! ! !s 4Acg}t|jdz d}tt|j|d}t |D]\}}||_i}|D]t}||_|} tj| } tj | | tj | <| | | f||<ut| \} } tj| d} tj| } t!jd|| fz|| | f|S)a6Get average topic and model coherences. Parameters ---------- model_topics : list of list of str Topics from the model. Returns ------- list of (float, float) Sequence of pairs of average topic coherence and average coherence. r%rz Avg coherence for model %d: %.5f)r`r=listrange enumerater7rrrnanmeanisnanrzipvaluesvstackmeanrcrdrU)r!rrlast_topn_value topn_grid model_numr7coherence_at_nnrfilled_coherencesavg_coherencesavg_topic_coherencesmodel_coherences r"rz$CoherenceModel._compare_model_topicssl di!mQ//ty/2>>?? !*$>!ACL]A^A^!"(+<"="=>%5t7N7NO`7a7a$bq!!/2N4I4I4K4K/L , n#%9-=#>#>#C#CA#F#F  gn55O L;y/>ZZ [ [ [   3_E F F F Fr$) NNNNNNNrrr)rrk)NFF)__name__ __module__ __qualname____doc__r> classmethodrF staticmethodrArCropropertyr<setterr=rmr7rrsrrrrrrrerrrrrrrr$r"rr\sp  BUY[]ZUZUZUZUx%0%0%0[%0N\8""["H!!!  X  \ & &\ &  X  [[* 2 2X 2   X   ]""]"8fff%%% FFFBBB BBB>>\>4 - - -"!"!"!"!H$K$K$K$KL333" ; ; ;777"@)))))r$r))rrcmultiprocessingr9 collectionsrnumpyrgensimrrrgensim.topic_coherencerrr r r -gensim.topic_coherence.probability_estimationr getLoggerrrr0r2_make_pipeline s_one_prep_boolean_documentlog_conditional_probabilityarithmetic_mean s_one_setp_boolean_sliding_windowcosine_similarity p_word2vecword2vec_similarity s_one_onelog_ratio_measurer{r-TransformationABCrrr$r"rs:,""""""'''''''' SRRRRR  8 $ $":::/1HIIn1#?#  >7%7#    ^)%9#  ^7#5#  n7#5# 3D    g g g g g Z1g g g g g r$