QdPdZddlmZddlmZddlmZddlmZdZ GddZ Gd d Z Gd d Z d S)a Common methods and classes for all IBM models. See ``IBMModel1``, ``IBMModel2``, ``IBMModel3``, ``IBMModel4``, and ``IBMModel5`` for specific implementations. The IBM models are a series of generative models that learn lexical translation probabilities, p(target language word|source language word), given a sentence-aligned parallel corpus. The models increase in sophistication from model 1 to 5. Typically, the output of lower models is used to seed the higher models. All models use the Expectation-Maximization (EM) algorithm to learn various probability tables. Words in a sentence are one-indexed. The first word of a sentence has position 1, not 0. Index 0 is reserved in the source sentence for the NULL token. The concept of position does not apply to NULL, but it is indexed at 0 by convention. Each target word is aligned to exactly one source word or the NULL token. References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. ) insort_left defaultdict)deepcopy)ceilc\d}|D]&}t|j}t||}'|S)z :param sentence_aligned_corpus: Parallel corpus under consideration :type sentence_aligned_corpus: list(AlignedSent) :return: Number of words in the longest target language sentence of ``sentence_aligned_corpus`` r)lenwordsmax)sentence_aligned_corpusmax_maligned_sentencems 8lib/python3.11/site-packages/nltk/translate/ibm_model.pylongest_target_sentence_lengthr.s> E3  & ' 'Au  LcjeZdZdZdZdZdZdZdZdZ dd Z dd Z dd Z d Z dZdZdZdZdS)IBMModelz0 Abstract base class for all IBM models g-q=cX|||dSN) init_vocabreset_probabilitiesselfr s r__init__zIBMModel.__init__Hs- /000   """""rctd_ td_ tfd_ d_dS)Nc"tdS)NctjSrrMIN_PROBrrz@IBMModel.reset_probabilities....Ns (9rrr!rrr"z.IBMModel.reset_probabilities..NsK 9 9::rc"tdS)Nc"tdS)Nc"tdS)NctjSrrr!rrr"zdIBMModel.reset_probabilities........Ws @Qrrr!rrr"zRIBMModel.reset_probabilities......WsK8Q8Q,R,Rrrr!rrr"z@IBMModel.reset_probabilities....Ws $R$RSSrrr!rrr"z.IBMModel.reset_probabilities..VsKSSrc(tfdS)NcjSr)r rsrr"z@IBMModel.reset_probabilities....`s t}rrr)srr"z.IBMModel.reset_probabilities..`s;?T?T?T?T3U3Urg?)rtranslation_tablealignment_tablefertility_tablep1r)s`rrzIBMModel.reset_probabilitiesLsw!, : :" "  +      ++U+U+U+UVV   rcdS)z Initialize probability tables to a uniform distribution Derived classes should implement this accordingly. Nr!rs rset_uniform_probabilitiesz"IBMModel.set_uniform_probabilitiesns rct}t}|D]6}||j||j7|d||_ ||_dSr)setupdater motsadd src_vocab trg_vocab)rr r5r6rs rrzIBMModel.init_vocabvsEE EE 7 4 4    -3 4 4 4   -2 3 3 3 3 d" #  rcJt}t|j}t|j}||}||}||||}td|dzD]}td|dzD]l} |||| }|||}|||} || |j |j kr|}m||fS)a Sample the most probable alignments from the entire alignment space First, determine the best alignment according to IBM Model 2. With this initial alignment, use hill climbing to determine the best alignment according to a higher IBM Model. Add this alignment and its neighbors to the sample set. Repeat this process with other initial alignments obtained by pegging an alignment point. Hill climbing may be stuck in a local maxima, hence the pegging and trying out of different alignments. :param sentence_pair: Source and target language sentence pair to generate a sample of alignments from :type sentence_pair: AlignedSent :return: A set of best alignments represented by their ``AlignmentInfo`` and the best alignment of the set for convenience :rtype: set(AlignmentInfo), AlignmentInfo r) r1r r3r best_model2_alignment hillclimbr2 neighboringrangescore) r sentence_pairsampled_alignmentslrinitial_alignmentpotential_alignmentbest_alignmentji neighborss rsamplezIBMModel.samples8.!UU  " # #  # $ $!66}EE"nn->??!!$"2"23F"G"GHHH,q!a% 9 9A1a!e__ 9 9$($>$>}aQR$S$S!&*nn5F&J&J# ,,-@!DD ")))444&,~/CCC%8N  9">11rNrcdg|jz}dg|jz}t|dz }t|dz }dg|dzz}dt|dzD} td|dzD]} | |kr|} nqd} tj} || } td|dzD]G}||}|j| ||j|| ||z}|| kr|} |} H| || <| | | tt|t|t|| S)aT Finds the best alignment according to IBM Model 2 Used as a starting point for hill climbing in Models 3 and above, because it is easier to compute than the best alignments in higher models :param sentence_pair: Source and target language sentence pair to be word-aligned :type sentence_pair: AlignedSent :param j_pegged: If specified, the alignment point of j_pegged will be fixed to i_pegged :type j_pegged: int :param i_pegged: Alignment point to j_pegged :type i_pegged: int NUNUSEDr8rcg|]}gSr!r!).0rEs r z2IBMModel.best_model2_alignment..s******r) r3r r r<rr r*r+append AlignmentInfotuple)rr>j_peggedi_pegged src_sentence trg_sentencer@r alignmentceptsrDbest_imax_alignment_probtrEsalignment_probs rr9zIBMModel.best_model2_alignments|&v 22  zM$77    !    !C1q5M **U1q5\\***q!a% $ $AH}}!%-%6" Oq!a%##A$QA.q1!4t7KA7Nq7QRS7TUV7WW#&);;;-;*!"!IaL &M  # # # # )  eL1153F3F   rc|}||} |}|||D]!}||}||kr|}|}"||krnB||_|S)a, Starting from the alignment in ``alignment_info``, look at neighboring alignments iteratively for the best one There is no guarantee that the best alignment in the alignment space will be found, because the algorithm might be stuck in a local maximum. :param j_pegged: If specified, the search will be constrained to alignments where ``j_pegged`` remains unchanged :type j_pegged: int :return: The best alignment found from hill climbing :rtype: AlignmentInfo )prob_t_a_given_sr;r=)ralignment_inforPrTmax_probability old_alignmentneighbor_alignmentneighbor_probabilitys rr:zIBMModel.hillclimbs # // :: %M&*&6&6y(&K&K ; ;"'+'<'<=O'P'P$'/99 2I&:OM)) * rcbt}t|jdz }t|jdz }|j}|j}t d|dzD]}||krt d|dzD]} t|} t|} ||} | | |<t| | || |  |tt| |j|j| } | | t d|dzD]}||krt d|dzD]}||kr||krt|} t|} ||}||} || |<| | |<| | |t| ||| |  |t| | |tt| |j|j| } | | |S)a Determine the neighbors of ``alignment_info``, obtained by moving or swapping one alignment point :param j_pegged: If specified, neighbors that have a different alignment point from j_pegged will not be considered :type j_pegged: int :return: A set neighboring alignments represented by their ``AlignmentInfo`` :rtype: set(AlignmentInfo) r8r)r1r rRrSrTrUr<listrrremoverNrOr4)rr]rPrFr@roriginal_alignmentoriginal_ceptsrDrE new_alignment new_ceptsold_inew_alignment_infoother_jother_is rr;zIBMModel.neighboringsdEE + , ,q 0 + , ,q 0+5'-q!a% 6 6AH}}q!a%66A$();$<$  # ) )      rc6t|j|S)zL Fertility of word in position ``i`` of the source sentence )r rU)rrEs rfertility_of_izAlignmentInfo.fertility_of_is4:a=!!!rcJ|j|}|j|d|kS)zo :return: Whether the word in position ``j`` of the target sentence is a head word r)rTrU)rrDrEs r is_head_wordzAlignmentInfo.is_head_words& N1 z!}Q1$$rc|dSt|j|t|j|z }tt |S)z :return: The ceiling of the average positions of the words in the tablet of cept ``i``, or 0 if ``i`` is None Nr)sumrUr intr)rrEaverage_positions rcenter_of_ceptzAlignmentInfo.center_of_ceptsL 91tz!}--DJqM0B0BB4())***rc|j|}|dkrtd|dz }|dkr=||dkr$|dz}|dkr||dk$|dkrd}|S)zm :return: The previous cept of ``j``, or None if ``j`` belongs to the first cept rzNWords aligned to NULL cannot have a previous cept because NULL has no positionr8N)rT ValueErrorr)rrDrE previous_cepts rrzAlignmentInfo.previous_cepts N1  664 A aD$7$7 $F$F!$K$K Q MaD$7$7 $F$F!$K$K A   Mrc|j|}|j||}|dkrdS|j||dz S)z :return: The position of the previous word that is in the same tablet as ``j``, or None if ``j`` is the first word of the tablet rNr8)rTrUindex)rrDrEtablet_positions rprevious_in_tabletz AlignmentInfo.previous_in_tabletsO N1 *Q---a00 a  4z!}_q011rcg}tdt|jD]4}|j|dz }|dkrd}||dz |f5|S)z :return: Zero-indexed alignment, suitable for use in external ``nltk.translate`` modules like ``nltk.translate.Alignment`` :rtype: list(tuple) r8rN)r<r rSrTrM)rzero_indexed_alignmentrDrEs rrz$AlignmentInfo.zero_indexed_alignmentsr "$q#d/0011 6 6Aq!A%A1uu " ) )1q5!* 5 5 5 5%%rc"|j|jkSr)rTrothers r__eq__zAlignmentInfo.__eq__s~00rc||k Srr!rs r__ne__zAlignmentInfo.__ne__s5=  rc*t|jSr)hashrTr)s r__hash__zAlignmentInfo.__hash__sDN###rN)rrrrrrrrrrrrrrr!rrrNrNs  ( ( ( T""" %%% + + +& 2 2 2 & & &111!!!$$$$$rrNc*eZdZdZdZdZdZdZdS)CountszK Data object to store counts of various parameters during training ctd|_td|_d|_d|_td|_td|_dS)Nc"tdS)NcdSNrr!r!rrr"z3Counts.__init__.... rrr!rrr"z!Counts.__init__.. [-E-ErcdSrr!r!rrr"z!Counts.__init__.. srrc"tdS)NcdSrr!r!rrr"z3Counts.__init__....rrrr!rrr"z!Counts.__init__..rrcdSrr!r!rrr"z!Counts.__init__..sr)rrnrpr{r-rvrwr)s rrzCounts.__init__ s`$%E%EFF(55$%E%EFF%0%=%="""rc|j|}|j|}|j|}|j||xx|z cc<|j|xx|z cc<dSr)rTrSrRrnrp)rcountr]rDrErXrYs rupdate_lexical_translationz!Counts.update_lexical_translationsq  $Q '  ' *  ' * q!% 1&rct|jdz }|d}|xj||zz c_|xj|d|zz |zz c_dS)Nr8r)r rSrr-r{)rrr]rfertility_of_nulls rupdate_null_generationzCounts.update_null_generationse + , ,q 0*99!<< $u,, A---66rctdt|jD]T}|j|}||}|j||xx|z cc<|j|xx|z cc<UdSr)r<r rRrrvrw)rrr]rErYrxs rupdate_fertilityzCounts.update_fertility sq#n9::;; 3 3A+A.A //22C N3  " " "e + " " "  &q ) ) )U 2 ) ) ) )  3 3rN)rrrrrrrrr!rrrrsZ>>>'''777 33333rrN) rbisectr collectionsrcopyrmathrrrrNrr!rrrs@######   @@@@@@@@F D$D$D$D$D$D$D$D$N3333333333r