cBdZddlmZmZddlmZddlZddlZddlm Z m Z ddl Z ej eZGdde jeZGd d eZGd d eZGd deZdS)aThis module implements functionality related to the `Okapi Best Matching `_ class of bag-of-words vector space models. Robertson and Zaragoza [1]_ describe the original algorithm and its modifications. .. [1] Robertson S., Zaragoza H. (2015). `The Probabilistic Relevance Framework: BM25 and Beyond, `_. )ABCMetaabstractmethod) defaultdictN) interfacesutilscLeZdZdZddZedZedZdZdS)BM25ABCzObjects of this abstract class realize the transformation between word-document co-occurrence matrix (int) into a BM25 matrix (positive floats). Concrete subclasses of this abstract class implement different BM25 scoring functions. Nc"d\|_|_|rx|rtdt |j}||jz |_||j |j|_dS|rtd}d}d}|D]E}|t|z }td|DD]}||xxdz cc<|dz }F||z |_||||_dSdS)aPre-compute the average length of a document and inverse term document frequencies, which will be used to weight term frequencies for the documents. Parameters ---------- corpus : iterable of iterable of (int, int) or None, optional An input corpus, which will be used to compute the average length of a document and inverse term document frequencies. If None, then `dictionary` will be used to compute the statistics. If both `corpus` and `dictionary` are None, the statistics will be left unintialized. Default is None. dictionary : :class:`~gensim.corpora.Dictionary` An input dictionary, which will be used to compute the average length of a document and inverse term document frequencies. If None, then `corpus` will be used to compute the statistics. If both `corpus` and `dictionary` are None, the statistics will be left unintialized. Default is None. Attributes ---------- avgdl : float The average length of a document. idfs : dict of (int, float) A mapping from term ids to inverse term document frequencies. NNzDconstructor received both corpus and dictionary; ignoring the corpuscdS)Nrr 7lib/python3.11/site-packages/gensim/models/bm25model.pyz"BM25ABC.__init__..Asarrc3 K|] \}}|V dSNr ).0term_id_s r z#BM25ABC.__init__..Fs&"A"Azw7"A"A"A"A"A"ArN) avgdlidfsloggerwarningsumcfsvaluesnum_docsprecompute_idfsdfsrlenset)selfcorpus dictionary num_tokensr!rbowrs r__init__zBM25ABC.__init__ s52!+ DI   gefffZ^224455J#j&99DJ,,Z^Z=PQQDIII  ii((CJH  c#hh& ""A"AS"A"A"AAA&&GLLLA%LLLLA #h.DJ,,S(;;DIII DrcdS)aPrecompute inverse term document frequencies, which will be used to weight term frequencies for the documents. Parameters ---------- dfs : dict of (int, int) A mapping from term ids to term document frequencies. num_docs : int The total number of documents in the training corpus. Returns ------- idfs : dict of (int, float) A mapping from term ids to inverse term document frequencies. Nr )r$r!rs rr zBM25ABC.precompute_idfsNs $ rcdS)aCompute vector space weights for a set of terms in a document. Parameters ---------- num_tokens : int The number of tokens in the document. term_frequencies : ndarray 1D array of term frequencies. idfs : ndarray 1D array of inverse term document frequencies. Returns ------- term_weights : ndarray 1D array of vector space weights. Nr )r$r'term_frequenciesrs rget_term_weightszBM25ABC.get_term_weightsbs & rctj|\}}|r||Std|D}ggg}}}|D]^\}}||||||j|pd_tj|tj|}}| |||} dt|| D} | S)Nc3 K|] \}}|V dSrr )rrfreqs rrz&BM25ABC.__getitem__..|s&77-'4777777rgc6g|]\}}|t|fSr )float)rrweights r z'BM25ABC.__getitem__..s7   eFmm $   r) r is_corpus_applyrappendrgetnparrayr-zip) r$r(r5r'term_idsr,rrterm_frequency term_weightsvectors r __getitem__zBM25ABC.__getitem__ws!-- 3  $;;s## #77377777 +-r2D"'* 7 7 #G^ OOG $ $ $  # #N 3 3 3 KK g..5# 6 6 6 6!#*:!;!;RXd^^$,,Z9I4PP   8\**     rr ) __name__ __module__ __qualname____doc__r)rr r-r@r rrr r sx ,,,,\  ^ &  ^ (rr ) metaclassc0eZdZdZd fd ZdZdZxZS) OkapiBM25ModelavThe original Okapi BM25 scoring function of Robertson et al. [2]_. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora import Dictionary >>> from gensim.models import OkapiBM25Model >>> from gensim.test.utils import common_texts >>> >>> dictionary = Dictionary(common_texts) # fit dictionary >>> model = OkapiBM25Model(dictionary=dictionary) # fit model >>> >>> corpus = [dictionary.doc2bow(line) for line in common_texts] # convert corpus to BoW format >>> vector = model[corpus[0]] # apply model to the first corpus document References ---------- .. [2] Robertson S. E., Walker S., Jones S., Hancock-Beaulieu M. M., Gatford M. (1995). `Okapi at TREC-3 `_. *NIST Special Publication 500-226*. N???cx|||c|_|_|_t||dS)u Pre-compute the average length of a document and inverse term document frequencies, which will be used to weight term frequencies for the documents. Parameters ---------- corpus : iterable of iterable of (int, int) or None, optional An input corpus, which will be used to compute the average length of a document and inverse term document frequencies. If None, then `dictionary` will be used to compute the statistics. If both `corpus` and `dictionary` are None, the statistics will be left unintialized. Default is None. dictionary : :class:`~gensim.corpora.Dictionary` An input dictionary, which will be used to compute the average length of a document and inverse term document frequencies. If None, then `corpus` will be used to compute the statistics. If both `corpus` and `dictionary` are None, the statistics will be left unintialized. Default is None. k1 : float A positive tuning parameter that determines the impact of the term frequency on its BM25 weight. Singhal [5]_ suggests to set `k1` between 1.0 and 2.0. Default is 1.5. b : float A tuning parameter between 0.0 and 1.0 that determines the document length normalization: 1.0 corresponds to full document normalization, while 0.0 corresponds to no length normalization. Singhal [5]_ suggests to set `b` to 0.75, which is the default. epsilon : float A positive tuning parameter that lower-bounds an inverse document frequency. Defaults to 0.25. Attributes ---------- k1 : float A positive tuning parameter that determines the impact of the term frequency on its BM25 weight. Singhal [3]_ suggests to set `k1` between 1.0 and 2.0. Default is 1.5. b : float A tuning parameter between 0.0 and 1.0 that determines the document length normalization: 1.0 corresponds to full document normalization, while 0.0 corresponds to no length normalization. Singhal [3]_ suggests to set `b` to 0.75, which is the default. epsilon : float A positive tuning parameter that lower-bounds an inverse document frequency. Defaults to 0.25. References ---------- .. [3] Singhal, A. (2001). `Modern information retrieval: A brief overview `_. *IEEE Data Eng. Bull.*, 24(4), 35–43. N)k1bepsilonsuperr))r$r%r&rLrMrN __class__s rr)zOkapiBM25Model.__init__s<\)+Aw% ,,,,,rcXd}t}g}|D]\\}}tj||z dztj|dzz }|||<||z }|dkr||]|t |z } |j| z} |D]}| ||<|S)Nr?)dictitemsmathlogr7r"rN) r$r!ridf_sumr negative_idfsrr0idf average_idfepss rr zOkapiBM25Model.precompute_idfssvv  YY[[ . .MGT(8d?S011DHTCZ4H4HHCDM sNGQw .$$W---D ) l[($  GDMM rcx|||jdzz||jd|jz |j|z|jz zzzz z}|SNrrLrMrr$r'r,rr>s rr-zOkapiBM25Model.get_term_weightse/47Q;?!1DGq46zDF@JMKMQZMX@X5Y"Y Z[ r)NNrHrIrJrArBrCrDr)r r- __classcell__rPs@rrGrGsf./-/-/-/-/-/-b$rrGc0eZdZdZdfd ZdZdZxZS) LuceneBM25ModeluThe scoring function of Apache Lucene 8 [4]_. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora import Dictionary >>> from gensim.models import LuceneBM25Model >>> from gensim.test.utils import common_texts >>> >>> dictionary = Dictionary(common_texts) # fit dictionary >>> corpus = [dictionary.doc2bow(line) for line in common_texts] # convert corpus to BoW format >>> >>> model = LuceneBM25Model(dictionary=dictionary) # fit model >>> vector = model[corpus[0]] # apply model to the first corpus document References ---------- .. [4] Kamphuis, C., de Vries, A. P., Boytsov, L., Lin, J. (2020). Which BM25 Do You Mean? `A Large-Scale Reproducibility Study of Scoring Variants `_. In: Advances in Information Retrieval. 28–34. NrHrIcj||c|_|_t||dSa Pre-compute the average length of a document and inverse term document frequencies, which will be used to weight term frequencies for the documents. Parameters ---------- corpus : iterable of iterable of (int, int) or None, optional An input corpus, which will be used to compute the average length of a document and inverse term document frequencies. If None, then `dictionary` will be used to compute the statistics. If both `corpus` and `dictionary` are None, the statistics will be left unintialized. Default is None. dictionary : :class:`~gensim.corpora.Dictionary` An input dictionary, which will be used to compute the average length of a document and inverse term document frequencies. If None, then `corpus` will be used to compute the statistics. If both `corpus` and `dictionary` are None, the statistics will be left unintialized. Default is None. k1 : float A positive tuning parameter that determines the impact of the term frequency on its BM25 weight. Singhal [5]_ suggests to set `k1` between 1.0 and 2.0. Default is 1.5. b : float A tuning parameter between 0.0 and 1.0 that determines the document length normalization: 1.0 corresponds to full document normalization, while 0.0 corresponds to no length normalization. Singhal [5]_ suggests to set `b` to 0.75, which is the default. Attributes ---------- k1 : float A positive tuning parameter that determines the impact of the term frequency on its BM25 weight. Singhal [3]_ suggests to set `k1` between 1.0 and 2.0. Default is 1.5. b : float A tuning parameter between 0.0 and 1.0 that determines the document length normalization: 1.0 corresponds to full document normalization, while 0.0 corresponds to no length normalization. Singhal [3]_ suggests to set `b` to 0.75, which is the default. NrLrMrOr)r$r%r&rLrMrPs rr)zLuceneBM25Model.__init__ 5Fa ,,,,,rct}|D]9\}}tj|dztj|dzz }|||<:|S)Ng?rRrSrTrUrVr$r!rrrr0rYs rr zLuceneBM25Model.precompute_idfs0s\vv YY[[  MGT(8c>**TXdSj-A-AACDMM rcb||||jd|jz |j|z|jz zzzz z}|Sr]r^r_s rr-z LuceneBM25Model.get_term_weights7sZ/!1DGq46zDF@JMKMQZMX@X5Y"Y Z[ rNNrHrIrarcs@rreresf0$-$-$-$-$-$-Lrrec0eZdZdZdfd ZdZdZxZS) AtireBM25ModeluThe scoring function of Trotman et al. [5]_. Examples -------- .. sourcecode:: pycon >>> from gensim.corpora import Dictionary >>> from gensim.models import AtireBM25Model >>> from gensim.test.utils import common_texts >>> >>> dictionary = Dictionary(common_texts) # fit dictionary >>> corpus = [dictionary.doc2bow(line) for line in common_texts] # convert corpus to BoW format >>> >>> model = AtireBM25Model(dictionary=dictionary) # fit model >>> vector = model[corpus[0]] # apply model to the first corpus document References ---------- .. [5] Trotman, A., Jia X., Crane M., `Towards an Efficient and Effective Search Engine `_, In: SIGIR 2012 Workshop on Open Source Information Retrieval. 40–47. NrHrIcj||c|_|_t||dSrgrhris rr)zAtireBM25Model.__init__Vrjrct}|D]3\}}tj|tj|z }|||<4|Srrlrms rr zAtireBM25Model.precompute_idfs|sRvv YY[[  MGT(8$$tx~~5CDMM rcx|||jdzz||jd|jz |j|z|jz zzzz z}|Sr]r^r_s rr-zAtireBM25Model.get_term_weightsr`rrorarcs@rrqrq>sf.$-$-$-$-$-$-Lrrq)rDabcrr collectionsrloggingrUgensimrrnumpyr9 getLoggerrArTransformationABCr rGrerqr rrr|sg('''''''###### $$$$$$$$  8 $ $rrrrrj*grrrrj_____W___DJJJJJgJJJZIIIIIWIIIIIr