cFdZddlZddlmZddlZddlmZGddZGddej Z Gd d ej Z dS) aA Produce a translation matrix to translate words from one language to another, using either a standard nearest neighbour method or a globally corrected neighbour retrieval method [1]_. This method can be used to augment the existing phrase tables with more candidate translations, or filter out errors from the translation tables and known dictionaries [2]_. What's more, it also works for any two sets of named-vectors where there are some paired-guideposts to learn the transformation. Examples -------- How to make translation between two set of word-vectors ======================================================= Initialize two word-vector models .. sourcecode:: pycon >>> from gensim.models import KeyedVectors >>> from gensim.test.utils import datapath >>> >>> model_en = KeyedVectors.load_word2vec_format(datapath("EN.1-10.cbow1_wind5_hs0_neg10_size300_smpl1e-05.txt")) >>> model_it = KeyedVectors.load_word2vec_format(datapath("IT.1-10.cbow1_wind5_hs0_neg10_size300_smpl1e-05.txt")) Define word pairs (that will be used for construction of translation matrix) .. sourcecode:: pycon >>> word_pairs = [ ... ("one", "uno"), ("two", "due"), ("three", "tre"), ("four", "quattro"), ("five", "cinque"), ... ("seven", "sette"), ("eight", "otto"), ... ("dog", "cane"), ("pig", "maiale"), ("fish", "cavallo"), ("birds", "uccelli"), ... ("apple", "mela"), ("orange", "arancione"), ("grape", "acino"), ("banana", "banana") ... ] Fit :class:`~gensim.models.translation_matrix.TranslationMatrix` .. sourcecode:: pycon >>> trans_model = TranslationMatrix(model_en, model_it, word_pairs=word_pairs) Apply model (translate words "dog" and "one") .. sourcecode:: pycon >>> trans_model.translate(["dog", "one"], topn=3) OrderedDict([('dog', [u'cane', u'gatto', u'cavallo']), ('one', [u'uno', u'due', u'tre'])]) Save / load model .. sourcecode:: pycon >>> with temporary_file("model_file") as fname: ... trans_model.save(fname) # save model to file ... loaded_trans_model = TranslationMatrix.load(fname) # load model How to make translation between two :class:`~gensim.models.doc2vec.Doc2Vec` models ================================================================================== Prepare data and models .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.test.test_translation_matrix import read_sentiment_docs >>> from gensim.models import Doc2Vec >>> >>> data = read_sentiment_docs(datapath("alldata-id-10.txt"))[:5] >>> src_model = Doc2Vec.load(datapath("small_tag_doc_5_iter50")) >>> dst_model = Doc2Vec.load(datapath("large_tag_doc_10_iter50")) Train backward translation .. sourcecode:: pycon >>> model_trans = BackMappingTranslationMatrix(data, src_model, dst_model) >>> trans_matrix = model_trans.train(data) Apply model .. sourcecode:: pycon >>> result = model_trans.infer_vector(dst_model.dv[data[3].tags]) References ---------- .. [1] Dinu, Georgiana, Angeliki Lazaridou, and Marco Baroni. "Improving zero-shot learning by mitigating the hubness problem", https://arxiv.org/abs/1412.6568 .. [2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. "Distributed Representations of Words and Phrases and their Compositionality", https://arxiv.org/abs/1310.4546 N) OrderedDict)utilsc6eZdZdZdZeddZdZdS)Spacez3An auxiliary class for storing the the words space.cx||_||_i|_t|jD]\}}||j|<dS)z Parameters ---------- matrix : iterable of numpy.ndarray Matrix that contains word-vectors. index2word : list of str Words which correspond to the `matrix`. N)mat index2word word2index enumerate)selfmatrixr idxwords @lib/python3.11/site-packages/gensim/models/translation_matrix.py__init__zSpace.__init__osQ$"4?33 ( (IC$'DOD ! ! ( (Ncng}g}|N|D]J}||||j||KnR|jD]J}||||j||Kt ||S)a Construct a space class for the lexicon, if it's provided. Parameters ---------- lang_vec : :class:`~gensim.models.keyedvectors.KeyedVectors` Model from which the vectors will be extracted. lexicon : list of str, optional Words which contains in the `lang_vec`, if `lexicon = None`, the lexicon is all the lang_vec's word. Returns ------- :class:`~gensim.models.translation_matrix.Space` Object that stored word-vectors )appendvectors get_index index_to_keyr)clslang_veclexiconwordsritems rbuildz Space.builds&  G G G T""" 8+H,>,>t,D,DEFFFF G !- G G T""" 8+H,>,>t,D,DEFFFFS%   rc|jtjtjtj|jddz |_dS)z"Normalize the word vectors matrix.T)axiskeepdimsN)rnpsqrtsumsquare)r s r normalizezSpace.normalizes<8bgbfRYtx-@-@qSW&X&X&XYYYr)N)__name__ __module__ __qualname____doc__r classmethodrr&rrrrls^==((($!!![!BZZZZZrrc>eZdZdZd dZdZfdZdZd d ZxZ S) TranslationMatrixa0Objects of this class realize the translation matrix which maps the source language to the target language. The main methods are: We map it to the other language space by computing z = Wx, then return the word whose representation is close to z. For details on use, see the tutorial notebook [3]_ Examples -------- .. sourcecode:: pycon >>> from gensim.models import KeyedVectors >>> from gensim.test.utils import datapath >>> en = datapath("EN.1-10.cbow1_wind5_hs0_neg10_size300_smpl1e-05.txt") >>> it = datapath("IT.1-10.cbow1_wind5_hs0_neg10_size300_smpl1e-05.txt") >>> model_en = KeyedVectors.load_word2vec_format(en) >>> model_it = KeyedVectors.load_word2vec_format(it) >>> >>> word_pairs = [ ... ("one", "uno"), ("two", "due"), ("three", "tre"), ("four", "quattro"), ("five", "cinque"), ... ("seven", "sette"), ("eight", "otto"), ... ("dog", "cane"), ("pig", "maiale"), ("fish", "cavallo"), ("birds", "uccelli"), ... ("apple", "mela"), ("orange", "arancione"), ("grape", "acino"), ("banana", "banana") ... ] >>> >>> trans_model = TranslationMatrix(model_en, model_it) >>> trans_model.train(word_pairs) >>> trans_model.translate(["dog", "one"], topn=3) OrderedDict([('dog', [u'cane', u'gatto', u'cavallo']), ('one', [u'uno', u'due', u'tre'])]) References ---------- .. [3] https://github.com/RaRe-Technologies/gensim/blob/3.2.0/docs/notebooks/translation_matrix.ipynb Ncd|_d|_||_||_t j||_d|_d|_d|_ |?t|ddkrtd| |dSdS)a Parameters ---------- source_lang_vec : :class:`~gensim.models.keyedvectors.KeyedVectors` Word vectors for source language. target_lang_vec : :class:`~gensim.models.keyedvectors.KeyedVectors` Word vectors for target language. word_pairs : list of (str, str), optional Pairs of words that will be used for training. random_state : {None, int, array_like}, optional Seed for random state. NrzBEach training data item must contain two different language words.) source_word target_wordsource_lang_vectarget_lang_vecrget_random_state random_statetranslation_matrix source_space target_spacelen ValueErrortrain)r r3r4 word_pairsr6s rrzTranslationMatrix.__init__s ..!2<@@"&    #:a=!!Q& g !efff JJz " " " " " # #rcht|\__tjt j_tjt j_ j j jj fdjDddf}j j fdjDddf}tj ||dd_dS)zBuild the translation matrix to map from source space to target space. Parameters ---------- word_pairs : list of (str, str), optional Pairs of words that will be used for training. c4g|]}jj|Sr,)r8r .0rr s r z+TranslationMatrix.train..$#d#d#d4D$5$@$F#d#d#drNc4g|]}jj|Sr,)r9r r@s rrBz+TranslationMatrix.train..rCrr)zipr1r2rrr3setr8r4r9r&rr"linalglstsqr7)r r=m1m2s` rr<zTranslationMatrix.trains.1*-=*$*!KK(S>STT!KK(S>STT ##%%% ##%%%   "#d#d#d#dSWSc#d#d#dfgfgfg#g h   "#d#d#d#dSWSc#d#d#dfgfgfg#g h"$)//"b""="=a"@rc~|dddg|d<tt|j|i|dS)zcSave the model to a file. Ignores (doesn't store) the `source_space` and `target_space` attributes.ignorer8r9N)getsuperr.save)r argskwargs __class__s rrPzTranslationMatrix.savesI!::h0PQQx+&&+TL  3 MJ   #2O  3 MJ   #2O  F "w/677G:s7||c,6G6G'GHHH'..tG4F4F|4T4T/U/UW_``G ;;L8I8I8O8OPST[P\P\8]8]^^LL ;; EEL{{?55        #11,??f\-/B/F/HIII   <z"*Za"@"@"@qIIIHZ:(=AFFFNNZ ;;;N&--"<00 ; ;IC%' "4[[ U U-a1H1N.NO &--l.El.STTTT$:OD ! !rNN)rXrNNN) r'r(r)r*rr<rPrWrt __classcell__)rSs@rr.r.s$$J####<AAA,===== ___ VVVVVVVVrr.c&eZdZdZddZdZdZdS)BackMappingTranslationMatrixaRealize the BackMapping translation matrix which maps the source model's document vector to the target model's document vector (old model). BackMapping translation matrix is used to learn a mapping for two document vector spaces which we specify as source document vector and target document vector. The target document vectors are trained on a superset corpus of source document vectors; we can incrementally increase the vector in the old model through the BackMapping translation matrix. For details on use, see the tutorial notebook [3]_. Examples -------- .. sourcecode:: pycon >>> from gensim.test.utils import datapath >>> from gensim.test.test_translation_matrix import read_sentiment_docs >>> from gensim.models import Doc2Vec, BackMappingTranslationMatrix >>> >>> data = read_sentiment_docs(datapath("alldata-id-10.txt"))[:5] >>> src_model = Doc2Vec.load(datapath("small_tag_doc_5_iter50")) >>> dst_model = Doc2Vec.load(datapath("large_tag_doc_10_iter50")) >>> >>> model_trans = BackMappingTranslationMatrix(src_model, dst_model) >>> trans_matrix = model_trans.train(data) >>> >>> result = model_trans.infer_vector(dst_model.dv[data[3].tags]) Nc||_||_||_tj||_d|_|||dSdS)a_ Parameters ---------- source_lang_vec : :class:`~gensim.models.doc2vec.Doc2Vec` Source Doc2Vec model. target_lang_vec : :class:`~gensim.models.doc2vec.Doc2Vec` Target Doc2Vec model. tagged_docs : list of :class:`~gensim.models.doc2vec.TaggedDocument`, optional. Documents that will be used for training, both the source language document vector and target language document vector trained on those tagged documents. random_state : {None, int, array_like}, optional Seed for random state. N) tagged_docsr3r4rr5r6r7r<)r r3r4rzr6s rrz%BackMappingTranslationMatrix.__init__sa '..!2<@@"&  $ JJ{ # # # # # $ $rcfd|D}fd|D}tj||dd_jS)a)Build the translation matrix to map from the source model's vectors to target model's vectors Parameters ---------- tagged_docs : list of :class:`~gensim.models.doc2vec.TaggedDocument`, Documents that will be used for training, both the source language document vector and target language document vector trained on those tagged documents. Returns ------- numpy.ndarray Translation matrix that maps from the source model's vectors to target model's vectors. cbg|]+}jj|j,Sr,)r3dvtagsflattenr@s rrBz6BackMappingTranslationMatrix.train..2 S S Std"%di088:: S S Srcbg|]+}jj|j,Sr,)r4r}r~rr@s rrBz6BackMappingTranslationMatrix.train..rrrEr)r"rHrIr7)r rzrJrKs` rr<z"BackMappingTranslationMatrix.trains`T S S S{ S S S S S S S{ S S S"$)//"b""="=a"@&&rc6tj||jS)a|Translate the target model's document vector to the source model's document vector Parameters ---------- target_doc_vec : numpy.ndarray Document vector from the target document, whose document are not in the source model. Returns ------- numpy.ndarray Vector `target_doc_vec` in the source model. )r"rUr7)r target_doc_vecs r infer_vectorz)BackMappingTranslationMatrix.infer_vectorsvnd&=>>>rru)r'r(r)r*rr<rr,rrrxrxpsP8$$$$4'''*?????rrx) r*r\ collectionsrnumpyr"gensimrrSaveLoadr.rxr,rrrs^^@######9Z9Z9Z9Z9Z9Z9Z9ZxEEEEEEEEPZ?Z?Z?Z?Z?5>Z?Z?Z?Z?Z?r