cdZddlZddlmZddlZddlZddlZddlZddl m Z ddl m Z ddl mZddlmZddlmZddlmZGd d ZGd d eZGd deZGddeZdS)z8 Created on Thu Oct 27 11:04:27 2022 @author: 20200016 N)Counter)svds) dok_matrix) Clustering)CoherenceModel)Word2VeccneZdZdZ d5dZdZed Zed Zd Z ed Z ed Z edZ edZ d6dZdZdZedZdZedZedZedZe d7dZedZedZedZedZ d8dZedZedZ d9d"Z d:d%Z d;d'Z dz*FlsaModel._convert_bow..s>>>TQAq>>>r4Nr>r)token2iditems itertoolsrepeatappend)rr1id2token data_listrBdoc_listrC_s r2r<zFlsaModel._convert_bows,?>W%5%;%;%=%=>>>  ' 'CH 6 6")$A7766AOOHSV$455556   X & & & &r4c t|jD]\}}t|tst dt |dt |dkstdt |dt|D]0\}}t|t st d|d|d1t|jtr |jd krtd t|j tr |j d krtd |j d vr/|j d vr&gd}td ||jdvrtdt|jts|jdkrtd|j dvrtddS)z Check whether the input data has the right format. Correct format: list of list of str (tokens) The function raises an error if the format is incorrect. zcorpus variable at index z is not a listrz&The corpus has an empty list at index z* and should contain at least one str valuezWord z of document z is not a strr>z(Please use a positive int for num_topicsz'Please use a positive int for num_wordsflsaflsa-w)entropyidfr probidf)Invalid word weighting methodzPlease choose between:z''entropy', 'idf', 'normal' and'probidf' )r zfst-psogkz@Invalid 'cluster_method. Please choose: 'fcm', 'fst-pso' or 'gk'z)Please use a positive int for svd_factors)rVrWflsa-ez"Please select a correct "algoritm"N) enumeraterr7r? TypeErrorstrlenr;rrArrrjoinrr)r0irBjwordwarnings r2rzFlsaModel._check_variablessD  ,, N NFAsc4(( 2 ; #A 0222s88a< B <FF@BBB%S>> N N4!$,,N#$LA$L$LA$L$L$LMMMN N$/3// I4?Q3F IGHH H$.#.. H$.12D HFGG G >  0)2 0GSXXg..// /  '  T RTT T$*C00 JT5E5I JHII I >"  C ABB B  C Cr4cVtd|D}|t|fS)a( Create the vocabulary from 'corpus'. Parameters ---------- corpus : list of lists of str The input file used to initialize the model. Returns ------- set of str All the vocabulary words. c3$K|] }|D]}|V dSrrF)rGlisels r2 z/FlsaModel._create_vocabulary..s/<<<<"<<<<<<\}}t |}|D]}||||||f<?|S)a, Creates a sparse matrix showing the frequency of each words in documents. (See: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.dok_matrix.html) Axes: rows: documents (size: number of documents in corpus) columns: words (size: vocabulary length) Parameters ---------- corpus : list of lists of str The input file used to initialize the model. vocabulary_size : int Number of unique words in the corpus. word_to_index: dict {str : int} Maps each unique vocabulary word to a unique index number. Returns ------- scipy.sparse.dok_matrix sparse matrix representation of the local term weights. dtype)rrbnpfloat32r_rkeys)rvocabulary_sizerqsparse_local_term_weightsdocument_indexrvdocument_counterrfs r2!_create_sparse_local_term_weightsz+FlsaModel._create_sparse_local_term_weights-s8%/ [[  *%%%! )2&(9(9 / / $NH&x00 (--// / /).*"M$$78 /)(r4cJt|}|dvr|td|dvr"|td|td|dvr|td|d vr|td |d kr||||||} ni|d kr|||||} nJ|d kr||} n.|dkr|||||} ntd|| S)a  Apply a word_weighting method on the sparse_local_term_weights to create sparse_global_term_weights. (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- corpus : list of lists of str The input file used to initialize the model. word_weighting : str Indicates the method used for word_weighting. Choose from: - entropy - normal - idf - probidf vocabulary_size : int Number of unique words in the corpus. sparse_local_term_weights : scipy.sparse.dok_matrix A sparse matrix showing the frequency of each words in documents. word_to_index : dict {str : int} Maps each unique vocabulary word to a unique index number. index_to_word : dict {int : str} Maps each unique index number to a unique vocabulary word. sum_words : collections.Counter {str : int} The count of each word in the corpus. Returns ------- scipy.sparse.dok_matrix sparse matrix representation of the global term weights. )rXr Nz5Please feed the algorithm 'sparse_local_term_weights')rXz)Please feed the algorithm 'index_to_word'z%Please feed the algorithm 'sum_words')rXrYrZz+Please feed the algorithm 'vocabulary_size')rYrZz)Please feed the algorithm 'word_to_index'rXrYr rZr[)rbr;_calculate_entropy_calculate_idf_calculate_normal_calculate_probidfmultiplytocsc) r0rrr}r~rrrqru num_documentsglobal_term_weightss r2"_create_sparse_global_term_weightsz,FlsaModel._create_sparse_global_term_weightsVsRF 2 2 Z( Z !XYYY [ ( J N !LMMM J !HIII : : P P !NOOO / / N N !LMMM Y & >"&"9"9)y ##   u $ >"&"5"5 ##   x ' >"&"8"89R"S"S   y ( >"&"9"9 ##  <== =(112EFFLLNNNr4c|||||}|dd}tjfd|DS)a Use the entropy word weighting method. (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- num_documents : int The number of documents in the corpus. vocabulary_size : int Number of unique words in the corpus. sparse_local_term_weights : scipy.sparse.dok_matrix A sparse matrix showing the frequency of each words in documents. index_to_word : dict {int : str} Maps each unique index number to a unique vocabulary word. sum_words : collections.Counter {str : int} The count of each word in the corpus. Returns ------- numpy.array : float rcDg|]}d|tjz zSr>rzlog2)rGsummed_p_log_p_irs r2 z0FlsaModel._calculate_entropy..s/nnnK[- 0F0FFFnnnr4)_create_p_log_p_ijsumtolistrzarray)r0rr}r~rrru p_log_p_ijsummed_p_log_ps ` r2rzFlsaModel._calculate_entropyss<,,   %    $**1133A6xnnnn_mnnnooor4c||||}|dd}tjfd|DS)av Use the idf word weightingg method. (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- num_documents : int The number of documents in the corpus. vocabulary_size : int Number of unique words in the corpus. corpus : list of lists of str The input file used to initialize the model. word_to_index: dict {str : int} Maps each unique vocabulary word to a unique index number. Returns ------- numpy.array : float rc>g|]}tj|z SrFr)rG word_countrs r2rz,FlsaModel._calculate_idf..s(\\\!;<<\\\r4_create_sparse_binary_dtmrrrzr)r0rr}rrqbinary_sparse_dtm summed_wordss ` r2rzFlsaModel._calculate_idfsr6!::      ),,Q//6688; x\\\\|\\\]]]r4c||}|dd}tjd|DS)a Use the normal word weightingg method. (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- sparse_local_term_weights : scipy.sparse.dok_matrix A sparse matrix showing the frequency of each words in documents. Returns ------- numpy.array : float rc<g|]}dtj|z Sr)mathsqrt)rGrs r2rz/FlsaModel._calculate_normal.. s'TTTdi 334TTTr4)rrrrzr)r~ squared_dtmrs r2rzFlsaModel._calculate_normalsY$0889RSS "q))002215 xTT|TTTUUUr4c||||}|dd}tjfd|DS)az Use the probidf word weightingg method. (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- num_documents : int The number of documents in the corpus. vocabulary_size : int Number of unique words in the corpus. corpus : list of lists of str The input file used to initialize the model. word_to_index: dict {str : int} Maps each unique vocabulary word to a unique index number. Returns ------- numpy.array : float rcDg|]}tj|z |z SrFr)rGbinary_word_countrs r2rz0FlsaModel._calculate_probidf...sELLL.-2C"CGX!XYYLLLr4r)r0rr}rrqrsummed_binary_words_lists ` r2rzFlsaModel._calculate_probidf s6!::      $5#8#8#;#;#B#B#D#DQ#G xLLLL2JLLLMM Mr4ct||ftj}t|D]}||d}|d}||dk} t| D]@\} } ||| } | || z } | tj| z|||| f<A|S)aA Create probability of word i in document j, multiplied by its base-2 logarithm. (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- num_documents : int The number of documents in the corpus. vocabulary_size : int Number of unique words in the corpus. sparse_local_term_weights : scipy.sparse.dok_matrix A sparse matrix showing the frequency of each words in documents. index_to_word : dict {int : str} Maps each unique index number to a unique vocabulary word. sum_words : collections.Counter {str : int} The count of each word in the corpus. Returns ------- scipy.sparse.dok_matrix rxr) rrzr{rangegetrowtoarraynonzeror_r)rr}r~rrrurre row_counts word_indexnon_zero_row_countsrdcountrfprob_ijs r2rzFlsaModel._create_p_log_p_ij1s< O ,BJ }%% J JA299!<<DDFFqIJ#++--a0J",Z1_"= %&9:: J J5$Z]3)D/1/69I9I/I 1jm+,, Jr4ct||ftj}t|D]E\}}t|d}t |D]}||||||f<F|S)a Create a binary sparse document-term-matrix (used for idf and probidf). (See: https://link.springer.com/article/10.1007/s40815-017-0327-9) Parameters ---------- num_documents : int The number of documents in the corpus. vocabulary_size : int Number of unique words in the corpus. corpus : list of lists of str The input file used to initialize the model. word_to_index: dict {str : int} Maps each unique vocabulary word to a unique index number. Returns ------- scipy.sparse.dok_matrix rxr>)rrzr{r_rpfromkeysrm) rr}rrqr doc_indexrvbinary_document_counterrfs r2rz#FlsaModel._create_sparse_binary_dtm\s6' O ,BJ$-V#4#4 W W Ix&*mmHa&@&@ #H  W W9PQU9V")"/"5#677 W! r4clt||\}}}|dvr|S|dvr|jStdd)a Perform singular decomposition for dimensionality reduction. (See: https://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm) For SVD on a sparse matrix, the sparsesvd package is used (https://pypi.org/project/sparsesvd/) Parameters ---------- algorithm : str Indicator for which algorithm is being trained ('flsa' or 'flsa-w'). sparse_weighted_matrix : scipy.sparse.dok_matrix Sparse global term matrix. svd_factors : int The number of singular values to include. Returns ------- numpy.array : float rVrWzInvalid algorithm selected.z1Only "flsa" ans "flsa-w" are currently supported.)rTr;)rsparse_weighted_matrixrsvd_urSsvd_vs r2_create_projected_dataz FlsaModel._create_projected_datasb4 " q%   L  " 7N6LNN Nr4cftj||}||\}}}|S)a7 Perform clustering on the projected data. The pyFUME package is used for clustering: (https://pyfume.readthedocs.io/en/latest/Clustering.html) Parameters ---------- data: numpy.array The output from self._create_projected_data(). number_of_clusters : int The number of clusters (topics). method : str The cluster method, choose from: 'fcm', 'gk', 'fst-pso'. Returns ------- numpy.array : float )nr_clusdata)method)r Clusterercluster)rnumber_of_clustersr clustererrSpartition_matrixs r2_create_partition_matrixz"FlsaModel._create_partition_matrixsG0(& "+!2!2&!2!A!A Qr4ctjd|dD}t|dd}||z S)aK Get the probability of document j. Parameters ---------- sparse_matrix : scipy.sparse.dok_matrix A sparse matrix representation of the global term weights. Returns ------- numpy.array : float (shape: number of documents x 1) cg|] }|d S)rrF)rGrBs r2rz5FlsaModel._create_prob_document_j..s Q Q QCQ Q Q Qr4r>rrzrrr) sparse_matrix document_sum total_sum_ds r2_create_prob_document_jz!FlsaModel._create_prob_document_jss x Q Q=3D3DQ3G3G3N3N3P3P Q Q QRR -++A..5577:;; k))r4ctj|d}t|dd}||z dS)aD Get the probability of word i. Parameters ---------- sparse_matrix : scipy.sparse.dok_matrix A sparse matrix representation of the global term weights. Returns ------- numpy.array : float (shape: vocabulary_size x 1) rr)rword_sum total_sum_ws r2_create_prob_word_izFlsaModel._create_prob_word_isi8M--a007799::-++A..5577:;; ;&**r4c6tj|j|S)a Get the probability of topic k. Parameters ---------- prob_topic_given_word_transpose : numpy.array : float The output from self._create_partition_matrix(). prob_word_i : numpy.array : float The output from self._create_prob_word_i(). Returns ------- numpy.array : float (shape: 1 x number of topics) )rzmatmulr)prob_topic_given_word_transpose prob_word_is r2_create_prob_topic_kzFlsaModel._create_prob_topic_ks(y8:KHHHr4c|dvr%|tdd|tddS|dvr&|tdd|tdd dS|d vr%|tdd |td dStd )zR Check whether the algorithms are being fed the right attributes. rNzPlease feed the methodz1'prob_topic_given_document_transpose' to run flsaz9Please feed the method 'global_term_weights', to run flsarz/'prob_topic_given_word_transpose' to run flsa-wz,Please feed the method 'global_term_weights'z to run flsa-wr^z.'prob_topic_given_word_transpose' to run modelz9Please feed the method 'local_term_weights', to run modelz)Your algorithm is currently not supported)r;)r#prob_topic_given_document_transposerlocal_term_weightsrs r2_check_passed_variablesz!FlsaModel._check_passed_variabless&   J2 V !9!TVVV" ^ !\]]] ^ ^ * $ J. T !9!RTTT" 3 !O!1333 3 3 J/ S !9!QSSS! ^ !\]]] ^ ^HII Ir4c|||||||dvrY|||_|||_|dvr |||j|_nX|dvrT|||_|||_|||j|_|dvr|j|jzj}||dz }tj ||dz |_ tj |j j||_ |j}|j |fS|dvr |j|jzj} | | dz |_ |dvr5tj ||dz j|_ n8|dvr4tj ||dz j|_ |j |jzjtj|jz } tj | |j }||jzj|jz }|j |fStd ) a Method that performs matrix multiplications to obtain the output matrices. The 'algorithm' parameter is generic and the other ones depend on the selected algorithm. The other parameters passed into this method depend on the used algorithm. Parameters ---------- algorithm : str Indicator for which algorithm is being trained ('flsa' or 'flsa-w'). global_term_weights : scipy.sparse.dok_matrix The output from self._create_partition_matrix(). prob_topic_given_document_transpose : numpy.array : float The output from self._create_partition_matrix() (flsa) prob_topic_given_word_transpose : numpy.array : float (flsa-w) Returns ------- numpy.array : float The prbability of a word given a topic. numpy.array : float The prbability of a topic given a document. rUrrrr)axisr>)rWr^z"algorithm" is unknown.)rrr&rr'rr(rrrzasarrayr*rr)rr;) r0rrrrrprob_document_and_topicprob_document_given_topicprob_topic_given_documentprob_word_and_topicprob_document_given_words r2_create_probability_matricesz&FlsaModel._create_probability_matrices(sB $$  / +         !% 8 89L M MD $($@$@AT$U$UD !J& %)%>%>3%&&" !% 8 89K L LD $($@$@AS$T$TD !!%!:!:/1B""D    J'J'LtOd'd&g #(?BYB]B]cdB]BeBe(e %-/Z8KNaNeNefgNhNh8h-i-iD **,).0)++D ')L(M %.0II I  J$C#DtGX#X"[ *=@S@W@W]^@W@_@_*_D 'J& n13N)r7rAr;rshaperpargsortrO)topnprob_word_given_topicrrtop_dictionaries topic_indexnew_dicthighest_weight_indicesrs r2_create_dictlist_topnzFlsaModel._create_dictlist_topns6$$$ L LJKK K !6!%J%R%R%T%TVZUZU[U[%\ "4   6K ,7z233  # #H - - - -r4ct|tsd|cxkrdkrnntdg}t|jdD]}t }d}d}t j|dd|fddd}t j|dd|fddd} ||kr-||||| |<|||z }|dz }||k-| ||S)a Create a list with dictionaries of word probabilities per topic based on the percentile. - Keys: all the indices of words from prob_word_given_topic who's weight's are amongst the top percentage. - Values: the probability associated to a word. Parameters ---------- perc : float The top percentile words to include (needs only to be used when 'method=percentile'). prob_word_given_topic : numpy.array : float Matrix that gives the probability of a word given a topic. index_to_word : dict {int : str} Maps each unique index number to a unique vocabulary word. Returns ------- list of dicts {int : float} Keys: all the indices of words from prob_word_given_topic who's weight's are amongst the top percentage. Values: the probability associated to a word. rr>z1Please choose a number between 0 and 1 for 'perc'N) r7floatr;rrrprzsortrrO) percrrrtop_listtoprrrdweights word_indicess r2_create_dictlist_percentilez%FlsaModel._create_dictlist_percentilesN<$&& R1 R R R R R R R R RPQQ Q.4Q788 & &CvvHEAg3AAAsF;<z3'prob_word_given_topic' has more columns then rows,z) probably you need to take the transpose.)z8It seems like 'prob_word_given_topic' and 'index_to_wordz2are not aligned. The number of vocabulary words inz)'prob_word_given_topic' deviate from the z#number of words in 'index_to_word'.r\z'Please choose a boolean for "formatted"rz*"z" + )r)rr#r7rzndarrayr`rprArr;rbr|warningswarnrcboolrrraroundrO) r0 formattedrrrrrg topic_listr weight_wordssorted_highest_weight_indicesr word_lists r2 show_topicszFlsaModel.show_topicss6 ! @$($? ! q= 'I  0 /M/<< a_`` `-.. SQRR R)S)) JY!^ JHII I &q ),A,G,J J JRHJJ J::: ! &q )S1C1C1E1E-F-F F - M#((7++ , , ,)T** HFGG G  $%:%@%CDD D D ! 0Eaaan0U0]0]0_0_aj`j`k`k0lmqmqoqmq0r-"?RRJ S/DZQ\E\/]_`)a)a%b%b'+&,.;J.G&HJP&QRLL!!; SbS0A"BCCCC $%:%@%CDD - -  0Eaaan0U0]0]0_0_aj`j`k`k0lmqmqoqmq0r-"?@@J$$]:%>????!!),,,, r4r皙?c |||j}g}|dvrt|dd|dkr||||j}n||||j}|D]t}dg|jdz} |D]I} t|jdD],} | | xx|| | dz cc<-J| | utj |S)a  Create a topic embedding for each input document, to be used as input to predictive models. Parameters ---------- corpus : list of lists of str The input file used to initialize the model. prob_word_given_topic : numpy.array : float Matrix that gives the probability of a word given a topic. method : str Method to select words to be included in the embedding. (choose from 'topn', 'percentile'): - topn: for each topic the top n words with the highest probability are included. - percentile: for each topic all words with highest probabilities are assigned while the cumulative probability is lower than the percentile. topn : int The top-n words to include (needs only to be used when 'method=topn'). perc: float The benchmark percentile until which words need to be added (between 0 and 1). Returns ------- numpy.array : float Array in which each row gives the topic embedding for the associated document. N)r percentilez#is not a valid option for 'method'.z% Choose either 'topn' or 'percentile'rrr>) rr)r;rr#rrrgetrOrzr) r0rrrrrtop_distdictlistrB topic_weightsrfrds r2get_topic_embeddingzFlsaModel.get_topic_embedding#s`N   @$($? ! / / FV%JDFF F V  11+T-@HH77+T-@H + +CC"7"=a"@@M A A4:1=>>AAA!!$$$ a(@(@@$$$$A OOM * * * *x!!!r4c_vc |||j}|d}tj|fd|D}t ||||t |d|_|jS)a\ Calculate the coherence score for the generated topic. Parameters ---------- corpus : list of lists of str The input file used to initialize the model. topics : list of lists of str The words per topics, equivalent to self.show_topics(formatted=True). coherence : str The type of coherence to be calculated. Choose from: 'u_mass', 'c_v', 'c_uci', 'c_npmi'. Returns ------- float The coherence score. NTrc:g|]}|SrF)doc2bow)rGtextr1s r2rz1FlsaModel.get_coherence_score..s%???good++???r4r)topicstextsrr9 coherencer)rr r8r:rrb get_coherencer+)r0rrr corpus_bowr1s @r2get_coherence_scorezFlsaModel.get_coherence_scoreas2  6f 6[F%%%55F$V,,??????? -VAY    moo ##r4c||d}t}d}|D])}|||t|z }*t||z |_|jS)a'' Calculate the diversity score for the generated topic. Diversity = number of unique words / number of total words. See: https://tinyurl.com/2bs84zd8 Parameters ---------- topics : list of lists of str The words per topics, equivalent to self.show_topics(formatted=True). Returns ------- float The diversity score. NTrr)r rmrtrbr,)r0r unique_words total_wordsrs r2get_diversity_scorezFlsaModel.get_diversity_scores*  6%%%55Fuu   $ $C    $ $ $ 3s88 #KK"<00;>##r4c|||j}|d}|j|||||_|j|||_|j|jzS)a'' Calculate the interpretability score for the generated topics. Interpretability = coherence * diversity. (see: https://tinyurl.com/2bs84zd8) Parameters ---------- corpus : list of lists of str The input file used to initialize the model. topics : list of lists of str The words per topics, equivalent to self.show_topics(formatted=True). coherence : str The type of coherence to be calculated. Choose from: 'u_mass', 'c_v', 'c_uci', 'c_npmi'. Returns ------- float The interpretability score. NTr)rr r+rr,r!)r0rrrs r2get_interpretability_scorez$FlsaModel.get_interpretability_scores8  6f 6[F%%%55F   #'#;#;$$D   D#'#;#;F#C#CD #d&:::r4c|jS)ap Returns a set of all the words in the corpus Example: After initializing an instance of the flsamodel models as 'model' corpus = [['this', 'is', 'the', 'first', 'file'], ['and', 'this', 'is', 'second', 'file']] model.get_vocabulary() >>> {'this', 'is', 'the', 'first', 'file', 'and', 'second'} )rr0s r2get_vocabularyzFlsaModel.get_vocabularys r4c|jS)z Get the term-topic matrix. Returns ------- numpy.ndarray The probability for each word in each topic, shape (num_topics, vocabulary_size). )r.r%s r2 get_topicszFlsaModel.get_topicss yr4c|jS)aD Returns the number of words in the vocabulary Example: After initializing an instance of the flsamodel models as 'model' corpus = [['this', 'is', 'the', 'first', 'file'], ['and', 'this', 'is', 'second', 'file']] model.get_vocabulary_size() >>> 7 )r r%s r2get_vocabulary_sizezFlsaModel.get_vocabulary_sizes $$r4c|jS)z Obtain a dictionary that maps each vocabulary word to an index. Returns ------- dict of {str : int} word to int mapping. )r"r%s r2get_word_to_indexzFlsaModel.get_word_to_index ""r4c|jS)z Obtain a dictionary that maps index numbers to vocabulary words. Returns ------- dict of {int : str} int to word mapping. )r#r%s r2get_index_to_wordzFlsaModel.get_index_to_word r-r4c|jS)z Return the input file. Returns ------- list of list of str The input file 'corpus'. )rr%s r2 get_corpuszFlsaModel.get_corpuss {r4c|jS)z Return the probabilities per word. Returns ------- np.array of float The probabilities per word. )r&r%s r2get_prob_word_izFlsaModel.get_prob_word_is   r4c|jS)z Return the probabilities per document. Returns ------- np.array of float The probabilities per document. )r'r%s r2get_prob_document_jzFlsaModel.get_prob_document_j*s $$r4c|jS)z Return the probabilities per topic. Returns ------- np.array of float The probabilities per topic. )r(r%s r2get_prob_topic_kzFlsaModel.get_prob_topic_k5s !!r4cdt|tstd|drt |d}n<|drt |dzd}nt |dzd}t j|||dS)ah'' Saves the object to the drive, using the pickle library. Parameters ---------- filepath : str The directory in which the file should be stored, either with or without the file name. Returns ------- float The interpretability score. (Make sure that "filepath" has type "str".picklewb/ model.pickleN)r7rar;endswithopenpickledumpclose)r0filepath pickle_outs r2savezFlsaModel.save@s$(C(( IGHH H   Y ' ' :h--JJ   s # # :h7>>JJh2D99J D*%%%r4c<t|tstd|ds |dr|dz }n|dz }t |d}t j|j|_|dS)aj'' Loads the object from the drive, using the pickle library. Parameters ---------- filepath : str The directory in which the file should be stored, either with or without the file name. Returns ------- float The interpretability score. r9r:r<r=z /model.picklerbN) r7rar;r>r?r@load__dict__rB)r0rCinfiles r2rHzFlsaModel.load]s$(C(( IGHH H  ++ ,  %% ,N*O+h%% F++4  r4) r r r rNNNNN)NNNNN)r )NNNN)TNrN)Nrr r )NNrr).__name__ __module__ __qualname____doc__r3r staticmethodr6r<rrr!r$rrrrrrrrrrrrrrrrrr rrr!r#r&r(r*r,r/r1r3r5r7rErHrFr4r2r r s2D# &4&4&4&4P<\8\>4C4C4Cl++\+",,\,2\&&)&)\&)X!&*QOQOQOQOf&p&p&pP"^"^"^HVV\V*$M$M$ML((\(T"!"!\"!H"N"N\"NH   \ <**\*(++\+&II\I*!J!J\!JL15,0# $ ]4]4]4]4~% % \% N++\+^"& DDDDR#' <"<"<"<"@ '$'$'$'$V$$$$B ';';';';R     %%% # # # # # #    ! ! ! % % % " " ":r4r c2eZdZdZ dfd ZdZxZS) Flsaa5 Class to run the FLSA algorithm (see: https://tinyurl.com/mskjaeuu). Parameters ---------- corpus : list of lists of str The input file used to initialize the model. num_topics : int The number of topics that the model should train. num_words : int Indicates how many words per topic should be shown. word_weighting : str Indicates the method used for word_weighting. Choose from: - entropy - normal - idf - probidf svd_factors : int The number of singular values to include. cluster_method : str The cluster algorithm to be used ('fcm', 'gk', 'fst-pso'). r rr c Xtd||||||dS)NrVrrrrrrrsuperr3r0rrrrrr __class__s r2r3z Flsa.__init__sE !))#      r4c p||j|j|j}||j|j|j||j|j|j}|d||j }| ||j |j }| d||S)* Method to obtain the matrices after the model has been initialized. Returns ------- numpy.array : float The prbability of a word given a topic. numpy.array : float The prbability of a topic given a document. rrr}r~rrrqrurVrrrrrr)rrrrrr r"rrr#r%rrrrrrr0sparse_document_term_matrixsparse_global_term_weightingprojected_datars r2r-zFlsa._get_matricess'+&L&L K  !  ''# (,'N'N;. 1&A--o(O(($44#?(5  88#&9 000@ <1 r4rRr rr rKrLrMrNr3r- __classcell__rXs@r2rQrQ{sd4# &'''''''r4rQc2eZdZdZ dfd ZdZxZS) FlsaWaU Class to train the FLSA-W algorithm. See: https://ieeexplore.ieee.org/abstract/document/9660139 Parameters ---------- corpus : list of lists of str The input file used to initialize the model. num_topics : int The number of topics that the model should train. num_words : int Indicates how many words per topic should be shown. word_weighting : str Indicates the method used for word_weighting. Choose from: - entropy - normal - idf - probidf svd_factors : int The number of singular values to include. cluster_method : str The cluster algorithm to be used ('fcm', 'gk', 'fst-pso'). rRr rr c Xtd||||||dS)NrWrTrUrWs r2r3zFlsaW.__init__sE !))#      r4c p||j|j|j}||j|j|j||j|j|j}|d||j }| ||j |j }| d||S)rZr[rWr\r])rrrr^r_s r2r-zFlsaW._get_matricess'+&L&L K  !  ''# (,'N'N;. 1&A--o(O(($44#?(5  88#&9 00,< <1 r4rcrdrfs@r2rhrhsd8# ('''''''r4rhc<eZdZdZ d fd Zd Zd ZxZS) FlsaEa Class to train the FLSA-E algorithm. See: https://tinyurl.com/5n8utppk Parameters ---------- corpus : list of lists of str The input file used to initialize the model. num_topics : int The number of topics that the model should train. num_words : int Indicates how many words per topic should be shown. cluster_method : str The cluster algorithm to be used ('fcm', 'gk', 'fst-pso'). min_count : int Ignores all words with total frequency lower than this. window : int Maximum distance between the current and predicted word within a sentence. vector_size : int Dimensionality of the word vectors. workers : int Use these many worker threads to train the model ( = faster training with multicore machines). rRr r>r rc xd|_d|_td|||||||| dS)N.r^) rrrrrrrrr)modelword_embeddingrVr3) r0rrrrrrrrrXs r2r3zFlsaE.__init__?sZ ! !)#      r4cVt||||||_|jjjS)a Method to train a word embedding on the corpus. Parameters ---------- data : list of lists of str The input file used to initialize the model. min_count : int Ignores all words with total frequency lower than this. window : int Maximum distance between the current and predicted word within a sentence. vector_size : int Dimensionality of the word vectors. workers : int Use these many worker threads to train the model ( = faster training with multicore machines). ) sentencesrrrr)rrowvvectors)r0rrrrrs r2get_word_embeddingzFlsaE.get_word_embeddingZs84#  z}$$r4c<||j|j|j}||j|j|j|j|j|_ | |j |j |j }| d||S)z Method to run after the FLSA_E class has been initialized to obtain the output matrices. Returns: - Numpy array: prob_word_given_topic - Numpy array: prob_topic_given_document )rrrrrr]r^)rrr)rrr r"rurrrrrprrrr)r0r`rs r2r-zFlsaE._get_matrices~s'+&L&L K  !  ''# #55n(;L 6 88$#&9 00,<:1 r4)rRr r>rmr r)rKrLrMrNr3rur-rerfs@r2rlrl&sz8 6"%"%"%H"""""""r4rl)rNr collectionsrrr@rMnumpyrzscipy.sparse.linalgr scipy.sparserpyfumergensim.corporar8gensim.models.coherencemodelr gensim.modelsrr rQrhrlrFr4r2rs   $$$$$$###### 777777""""""bbbbbbbbJ+QQQQQ9QQQhTTTTTITTTnzzzzzIzzzzzr4