cSdZddlZddlmZddlZddlZddlmZm Z m Z ddl m Z ej eZdZdd Zd Zd Zd Ze d ddZGddejZdS)zThis module implements functionality related to the `Term Frequency - Inverse Document Frequency `_ class of bag-of-words vector space models. N)partial) interfacesmatutilsutils) deprecatedct|trttjd|r_tjd|}t d|d|dt|trt|dkrt d|z|\}}}|d vr"t d ||d vr"t d ||d vr"t d||dkrd}|dkrd}|dkrd}||z|zS)aCheck the validity of `smartirs` parameters. Parameters ---------- smartirs : str `smartirs` or SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System, a mnemonic scheme for denoting tf-idf weighting variants in the vector space model. The mnemonic for representing a combination of weights takes the form ddd, where the letters represents the term weighting of the document vector. for more information visit `SMART Information Retrieval System `_. Returns ------- str of (local_letter, global_letter, normalization_letter) local_letter : str Term frequency weighing, one of: * `b` - binary, * `t` or `n` - raw, * `a` - augmented, * `l` - logarithm, * `d` - double logarithm, * `L` - log average. global_letter : str Document frequency weighting, one of: * `x` or `n` - none, * `f` - idf, * `t` - zero-corrected idf, * `p` - probabilistic idf. normalization_letter : str Document normalization, one of: * `x` or `n` - none, * `c` - cosine, * `u` - pivoted unique, * `b` - pivoted character length. Raises ------ ValueError If `smartirs` is not a string of length 3 or one of the decomposed value doesn't fit the list of permissible values. z...\....z(?P...)\.(?P...)zThe notation {ddd}.{qqq} specifies two term-weighting schemes, one for collection documents ({ddd}) and one for queries ({qqq}). You must train two separate tf-idf models.dddqqq)r r z"Expected a string of length 3 got btnaldLz=Expected term frequency weight to be one of 'btnaldL', got {}xnftpzGExpected inverse document frequency weight to be one of 'xnftp', got {}xncubz:Expected normalization weight to be one of 'xncub', got {}tnx) isinstancestrrematch ValueErrorformatgrouplen)smartirsrw_tfw_dfw_ns 8lib/python3.11/site-packages/gensim/models/tfidfmodel.pyresolve_weightsrsX(C   RXk8%D%D  6AA 99?KK&&KK&&:@::    h $ $JH (:J=HIIIOD$ 9gX__`deefff 7qbiijnooppp 'cU\\]`aabbb s{ s{ cz $; @cz|tjt||z tj|z zS)aCompute inverse-document-frequency for a term with the given document frequency `docfreq`: :math:`idf = add + log_{log\_base} \frac{totaldocs}{docfreq}` Parameters ---------- docfreq : {int, float} Document frequency. totaldocs : int Total number of documents. log_base : float, optional Base of logarithm. add : float, optional Offset. Returns ------- float Inverse document frequency. )nplogfloat)docfreq totaldocslog_baseadds rdf2idfr+is5* i((7233bfX6F6FF FFr cHfd|DS)a`Pre-compute the inverse document frequency mapping for all terms. Parameters ---------- wglobal : function Custom function for calculating the "global" weighting function. See for example the SMART alternatives under :func:`~gensim.models.tfidfmodel.smartirs_wglobal`. dfs : dict Dictionary mapping `term_id` into how many documents did that term appear in. total_docs : int Total number of documents. Returns ------- dict of (int, float) Inverse document frequencies in the format `{term_id_1: idfs_1, term_id_2: idfs_2, ...}`. c0i|]\}}||Sr.).0termiddf total_docswglobals r z#precompute_idfs..s+ J J J FGGB ++ J J Jr )items)r3dfsr2s` `rprecompute_idfsr7s,* K J J J Jciikk J J JJr c|dkr|S|dkrdtj|zS|dkr,dtjdtj|zzS|dkrdd|z|dz zS|d kr(|d d S|d krCdtj|zdtj|dzz Sd S)a<Calculate local term weight for a term using the weighting scheme specified in `local_scheme`. Parameters ---------- tf : int Term frequency. local : {'b', 'n', 'a', 'l', 'd', 'L'} Local transformation scheme. Returns ------- float Calculated local weight. rldag?r)axisbboolintLN)r$log2maxastypemean)tf local_schemes rsmartirs_wlocalrHs s B   B272;;  B271rwr{{?++++  BcBhQ/00  Byy  ''...  BBGBKKAQ(@(@$@AABBr c|dkrdS|dkrtjd|z|z S|dkrtj|dz|z S|dkr+tdtjd|z|z |z SdS)azCalculate global document weight based on the weighting scheme specified in `global_scheme`. Parameters ---------- docfreq : int Document frequency. totaldocs : int Total number of documents. global_scheme : {'n', 'f', 't', 'p'} Global transformation scheme. Returns ------- float Calculated global weight. r?frprN)r$rBrC)r'r( global_schemes rsmartirs_wglobalrNs$Fs # FwsY0111 # Fw C72333 # F1bgsY8GCDDEEEFFr z!Function will be removed in 4.0.0Fc|dkr!|rtj||\}}||fS|S|dkrtj||SdS)aNormalize a vector using the normalization scheme specified in `norm_scheme`. Parameters ---------- x : numpy.ndarray The tf-idf vector. norm_scheme : {'n', 'c'} Document length normalization scheme. return_norm : bool, optional Return the length of `x` as well? Returns ------- numpy.ndarray Normalized array. float (only if return_norm is set) Norm of `x`. r return_normcN)runitvec)r norm_schemerQ_lengths rsmartirs_normalizerWsn(c<   ( DDDIAvf9 H  <{;;;;<>> import gensim.downloader as api >>> from gensim.models import TfidfModel >>> from gensim.corpora import Dictionary >>> >>> dataset = api.load("text8") >>> dct = Dictionary(dataset) # fit dictionary >>> corpus = [dct.doc2bow(line) for line in dataset] # convert corpus to BoW format >>> >>> model = TfidfModel(corpus) # fit model >>> vector = model[corpus[0]] # apply model to the first corpus document NTg?c |_|||c___d\___|t|nd_| _ |_ d_ |rAj\} } } tt| _tt| _|r|rtd|j|jc__|j_|j_d|D_t-jjj_|s|_n|r|n |sdSj | dvrtd dS| dvr.t1jrtd | dvr |s|std dS| d krd jzjz _ dS| dkr?d t3fd|Dzjz _ dSdS)uCompute TF-IDF by multiplying a local component (term frequency) with a global component (inverse document frequency), and normalizing the resulting documents to unit length. Formula for non-normalized weight of term :math:`i` in document :math:`j` in a corpus of :math:`D` documents .. math:: weight_{i,j} = frequency_{i,j} * log_2 \frac{D}{document\_freq_{i}} or, more generally .. math:: weight_{i,j} = wlocal(frequency_{i,j}) * wglobal(document\_freq_{i}, D) so you can plug in your own custom :math:`wlocal` and :math:`wglobal` functions. Parameters ---------- corpus : iterable of iterable of (int, int), optional Input corpus id2word : {dict, :class:`~gensim.corpora.Dictionary`}, optional Mapping token - id, that was used for converting input data to bag of words format. dictionary : :class:`~gensim.corpora.Dictionary` If `dictionary` is specified, it must be a `corpora.Dictionary` object and it will be used. to directly construct the inverse document frequency mapping (then `corpus`, if specified, is ignored). wlocals : callable, optional Function for local weighting, default for `wlocal` is :func:`~gensim.utils.identity` (other options: :func:`numpy.sqrt`, `lambda tf: 0.5 + (0.5 * tf / tf.max())`, etc.). wglobal : callable, optional Function for global weighting, default is :func:`~gensim.models.tfidfmodel.df2idf`. normalize : {bool, callable}, optional Normalize document vectors to unit euclidean length? You can also inject your own function into `normalize`. smartirs : str, optional SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System, a mnemonic scheme for denoting tf-idf weighting variants in the vector space model. The mnemonic for representing a combination of weights takes the form XYZ, for example 'ntc', 'bpn' and so on, where the letters represents the term weighting of the document vector. Term frequency weighing: * `b` - binary, * `t` or `n` - raw, * `a` - augmented, * `l` - logarithm, * `d` - double logarithm, * `L` - log average. Document frequency weighting: * `x` or `n` - none, * `f` - idf, * `t` - zero-corrected idf, * `p` - probabilistic idf. Document normalization: * `x` or `n` - none, * `c` - cosine, * `u` - pivoted unique, * `b` - pivoted character length. Default is 'nfc'. For more information visit `SMART Information Retrieval System `_. pivot : float or None, optional In information retrieval, TF-IDF is biased against long documents [1]_. Pivoted document length normalization solves this problem by changing the norm of a document to `slope * old_norm + (1.0 - slope) * pivot`. You can either set the `pivot` by hand, or you can let Gensim figure it out automatically with the following two steps: * Set either the `u` or `b` document normalization in the `smartirs` parameter. * Set either the `corpus` or `dictionary` parameter. The `pivot` will be automatically determined from the properties of the `corpus` or `dictionary`. If `pivot` is None and you don't follow steps 1 and 2, then pivoted document length normalization will be disabled. Default is None. See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/. slope : float, optional In information retrieval, TF-IDF is biased against long documents [1]_. Pivoted document length normalization solves this problem by changing the norm of a document to `slope * old_norm + (1.0 - slope) * pivot`. Setting the `slope` to 0.0 uses only the `pivot` as the norm, and setting the `slope` to 1.0 effectively disables pivoted document length normalization. Singhal [2]_ suggests setting the `slope` between 0.2 and 0.3 for best results. Default is 0.25. See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/. References ---------- .. [1] Singhal, A., Buckley, C., & Mitra, M. (1996). `Pivoted Document Length Normalization `_. *SIGIR Forum*, 51, 176–184. .. [2] Singhal, A. (2001). `Modern information retrieval: A brief overview `_. *IEEE Data Eng. Bull.*, 24(4), 35–43. )NNNN-q=)rG)rMz_constructor received both corpus and explicit inverse document frequencies; ignoring the corpusc4i|]\}}|t|Sr.)r)r/r0terms rr4z'TfidfModel.__init__..{s$WWWLFDfc$iiWWWr ubz0constructor received pivot; ignoring smartirs[2]z1constructor received smartirs; ignoring normalizezBconstructor received no corpus or dictionary; ignoring smartirs[2]urJr>c3TK|]"}j|j|dzzV#dSrJN)cfs term_lens)r/r0selfs r z&TfidfModel.__init__..sL##FL DN6$:S$@A######r )id2wordwlocalr3 normalizenum_docsnum_nnzidfsrrslopepivotepsrrHrNloggerwarningrbcopyr6r5rcr7 initializecallablesumkeys) rdcorpusrf dictionaryrgr3rhrrmrln_tfn_dfn_ns ` r__init__zTfidfModel.__init__s| 4:GY1 T\4>1A. t|TY5=S111t     I"mOD$!/EEEDK"#34HHHDL   u+5*=z?Q 'DM4<!~**,,DH!~**,,DHWWJDTDTDVDVWWWDN' dh NNDI *)   OOF # # # #   F : d{ SQRRR F $; P8DN33 P NNN O O O $; z &  NN_ ` ` ` ` ` CZ t|+dm;DJJJ CZ s####PZP_P_PaPa###   DJJJ  r cRtt|j|i|}t|dsGd|_t d|jt d|jt|dsGd|_t d|jt d|jt|d sGd|_ t d |jt d |j |S) zLoad a previously saved TfidfModel class. Handles backwards compatibility from older TfidfModel versions which did not use pivoted document normalization. rmNz,older version of %s loaded without pivot argzSetting pivot to %s.rlg?z,older version of %s loaded without slope argzSetting slope to %s.rz/older version of %s loaded without smartirs argzSetting smartirs to %s.) superrYloadhasattrrmroinfo__name__rlr)clsargskwargsmodel __class__s rr~zTfidfModel.loads ,j#&&+T B B B r c@|jjd|jd|jdS)Nz )rrrirj)rds r__str__zTfidfModel.__str__s*040G0G0GX\XdXdXdeer c ^tdi}d\}}t|D]_\}}|dzdkrtd||t|z }|D]!\}}||ddz||<"`|dz|_||_d|_||_d|_ t|j |j|j|_ | dd |jd |r$t|dzndd |jd  dS)zCompute inverse document weights, which will be used to modify term frequencies for documents. Parameters ---------- corpus : iterable of iterable of (int, int) Input corpus. zcollecting document frequencies)ri'rz!PROGRESS: processing document #%ir:Nrrzcalculated IDF weights for z documents and z features (z matrix non-zeros))msg)ror enumeratergetrirjrbr6 term_lengthsr7r3rkadd_lifecycle_eventrCru)rdrvr6numnnzdocnobowr0rUs rrrzTfidfModel.initializesr  5666 #F++ 5 5JE3u}! H ?GGG c#hh F  5 5 !ggfa0014F  5    #DL$(DMJJ    ?dm??cfLmCPSPXPXPZPZOO^_L_L_lm??"l??? !     r r[cP |_tj|\}}|r|Sgg}}|D]/\}}||||0t j|}fdt||D}j rj d} | dks | dvr*j #j tj |d\} } |} n| dkr6j tj |d\} } ntj |} n| d krtj |dd \} } n| d krtfd |D} nljdurtj _njdurtj_j |d\} } n|} j fd| D} n,djz j zj| zz fd|D} | S)aGet the tf-idf representation of an input vector and/or corpus. bow : {list of (int, int), iterable of iterable of (int, int)} Input document in the `sparse Gensim bag-of-words format `_, or a streamed corpus of such documents. eps : float Threshold value, will remove all position that have tfidf-value less than `eps`. Returns ------- vector : list of (int, float) TfIdf vector, if `bow` is a single document :class:`~gensim.interfaces.TransformedCorpus` TfIdf corpus, if `bow` is a corpus. cg|]U\}}tj|djk6||j|zfVS)r")absrkrrn)r/r0rFrds r z*TfidfModel.__getitem__..sl   SvWZA[A[=\=\_c_g=g R$)--/// 0   r rr^NTrPrRr_unique)rQnormr>c3DK|]\}}|j|dzzVdSra)rc)r/r0freqrds rrez)TfidfModel.__getitem__.. s9]]QUtt~f'='CD]]]]]]r FcNg|]!\}}t|jk||f"Sr.)rrn)r/r0weightrds rrz*TfidfModel.__getitem__..s:hhhQTU[Q\Q\_c_gQghFF+hhhr r:cg|]A\}}t|tz jk-||tz fBSr.)rr&rn)r/r0r pivoted_normrds rrz*TfidfModel.__getitem__..sa"FFvl 3 3344tx?% "5"556r )rnr is_corpus_applyappendrgr$arrayziprrmrrSrtrhidentityrl)rdrrnr termid_arraytf_arrayr0rFvectorrzrUold_norm norm_vectorrs` @r __getitem__zTfidfModel.__getitem__s$-- 3  $;;s## # "$Rh   JFB    ' ' ' OOB    ;;rx1122    !,99    = 5-"Ccz ^cTk ^dj ^:M"*"26t"L"L"LKAx$  ^:;"*"26t"L"L"LKAxx"*"26":":KK ^&.v4hWWW 88 ^]]]]Y\]]]]]~% 0!)!15( 0!&z 5"nnVnFF 88"nnV44 : hhhh+hhhKK Ndj84:;PPL&,K r )r[)r __module__ __qualname____doc__rrr+r{ classmethodr~rrrr __classcell__)rs@rrYrYs&#DT%.4$dRVQQQQf[(fff! ! ! FLLLLLLLLr rY)r!r")F)rlogging functoolsrrnumpyr$gensimrrr gensim.utilsr getLoggerrrorr+r7rHrNrWTransformationABCrYr.r rrsS  ..........######  8 $ $LLL^GGGG0KKK0BBB<FFF8  /00<<<10<:nnnnn-nnnnnr