Qd$PxdZddlZddlmZddlmZGddZGddZGd d ZdS) a& A decoder that uses stacks to implement phrase-based translation. In phrase-based translation, the source sentence is segmented into phrases of one or more words, and translations for those phrases are used to build the target sentence. Hypothesis data structures are used to keep track of the source words translated so far and the partial output. A hypothesis can be expanded by selecting an untranslated phrase, looking up its translation in a phrase table, and appending that translation to the partial output. Translation is complete when a hypothesis covers all source words. The search space is huge because the source sentence can be segmented in different ways, the source phrases can be selected in any order, and there could be multiple translations for the same source phrase in the phrase table. To make decoding tractable, stacks are used to limit the number of candidate hypotheses by doing histogram and/or threshold pruning. Hypotheses with the same number of words translated are placed in the same stack. In histogram pruning, each stack has a size limit, and the hypothesis with the lowest score is removed when the stack is full. In threshold pruning, hypotheses that score below a certain threshold of the best hypothesis in that stack are removed. Hypothesis scoring can include various factors such as phrase translation probability, language model probability, length of translation, cost of remaining words to be translated, and so on. References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. N defaultdict)logceZdZdZdZedZejdZdZdZ dZ dZ d Z d Z d Zed Zd S) StackDecodera Phrase-based stack decoder for machine translation >>> from nltk.translate import PhraseTable >>> phrase_table = PhraseTable() >>> phrase_table.add(('niemand',), ('nobody',), log(0.8)) >>> phrase_table.add(('niemand',), ('no', 'one'), log(0.2)) >>> phrase_table.add(('erwartet',), ('expects',), log(0.8)) >>> phrase_table.add(('erwartet',), ('expecting',), log(0.2)) >>> phrase_table.add(('niemand', 'erwartet'), ('one', 'does', 'not', 'expect'), log(0.1)) >>> phrase_table.add(('die', 'spanische', 'inquisition'), ('the', 'spanish', 'inquisition'), log(0.8)) >>> phrase_table.add(('!',), ('!',), log(0.8)) >>> # nltk.model should be used here once it is implemented >>> from collections import defaultdict >>> language_prob = defaultdict(lambda: -999.0) >>> language_prob[('nobody',)] = log(0.5) >>> language_prob[('expects',)] = log(0.4) >>> language_prob[('the', 'spanish', 'inquisition')] = log(0.2) >>> language_prob[('!',)] = log(0.1) >>> language_model = type('',(object,),{'probability_change': lambda self, context, phrase: language_prob[phrase], 'probability': lambda self, phrase: language_prob[phrase]})() >>> stack_decoder = StackDecoder(phrase_table, language_model) >>> stack_decoder.translate(['niemand', 'erwartet', 'die', 'spanische', 'inquisition', '!']) ['nobody', 'expects', 'the', 'spanish', 'inquisition', '!'] c||_||_d|_ d|_ d|_ d|_|dS)aG :param phrase_table: Table of translations for source language phrases and the log probabilities for those translations. :type phrase_table: PhraseTable :param language_model: Target language model. Must define a ``probability_change`` method that calculates the change in log probability of a sentence, if a given string is appended to it. This interface is experimental and will likely be replaced with nltk.model once it is implemented. :type language_model: object dg?N) phrase_tablelanguage_model word_penaltybeam_threshold stack_size _StackDecoder__distortion_factor%_StackDecoder__compute_log_distortion)selfr r s )rr$_StackDecoder__log_distortion_factorrs r__compute_log_distortionz%StackDecoder.__compute_log_distortions?  #s * *+.t99D ( ( (+.t/G+H+HD ( ( (rc jt|}t|}fdtd|dzD}t}|d||}|}|D]}|D]} t|| } | D]} || d| d} j | D]|} | | | }t|| | j | } ||||_ |}|||}Ќ||stjdgS||}|S)z :param src_sentence: Sentence to be translated :type src_sentence: list(str) :return: Translated sentence :rtype: list(str) cDg|]}tjjS)_Stackrr).0_rs r z*StackDecoder.translate..s8    4?D$7 8 8   rr) raw_scoresrc_phrase_span trg_phrasepreviouszYUnable to translate all words. The source sentence contains words not in the phrase table)tuplelenrange _Hypothesispushfind_all_src_phrasescompute_future_scoresr valid_phrasesr translations_forexpansion_scorer) future_scoretotal_translated_wordswarningswarnbesttranslation_so_far)r src_sentencesentencesentence_lengthstacksempty_hypothesis all_phrasesfuture_score_tablestack hypothesispossible_expansionsr( src_phrasetranslation_optionr'new_hypothesis total_wordsbest_hypothesiss` r translatezStackDecoder.translates-&&h--    1o122   '==q '(((//99 !77AA A AE# A A &2&@&@''#(;AAO!)/!*<q?Q*Q!RJ.2.?.P.P"//AA*%)$8$8&(:O%% *5&/,;'9'D%/ *** 7;6G6G*,>773'5&K&K&M&M {+00@@@@!AA  A0o&  M#   I 1668811333rct|}d|D}td|D]I}t|dz|dzD]0}|||}||jvr|||1J|S)a@ Finds all subsequences in src_sentence that have a phrase translation in the translation table :type src_sentence: tuple(str) :return: Subsequences that have a phrase translation, represented as a table of lists of end positions. For example, if result[2] is [5, 6, 9], then there are three phrases starting from position 2 in ``src_sentence``, ending at positions 5, 6, and 9 exclusive. The list of ending positions are in ascending order. :rtype: list(list(int)) cg|]}gSr!r!)r#r$s rr%z5StackDecoder.find_all_src_phrases..s333"333rrr&)r,r-r append)rr;r=phrase_indicesstartendpotential_phrases rr0z!StackDecoder.find_all_src_phrasessl++33l3331o.. 6 6EUQY!(;<< 6 6#/c #: #t'888"5)00555 6rc"td}tdt|dzD]}tdt||z dzD]}||z}|||}||jvrM|j|dj}||j|z }||||<t|dz|D]<}||||||z} | |||kr | |||<=|S)a Determines the approximate scores for translating every subsequence in ``src_sentence`` Future scores can be used a look-ahead to determine the difficulty of translating the remaining parts of a src_sentence. :type src_sentence: tuple(str) :return: Scores of subsequences referenced by their start and end positions. For example, result[2][5] is the score of the subsequence covering positions 2, 3, and 4. :rtype: dict(int: (dict(int): float)) c"tdS)Nc tdS)N-inf)floatr!rrzFStackDecoder.compute_future_scores....s vrrr!rrrWz4StackDecoder.compute_future_scores..s[1F1F%G%Grr&r)rr-r,r r3log_probr probability) rr;scores seq_lengthrOrPphrasescoremidcombined_scores rr1z"StackDecoder.compute_future_scoressJGGHH3|#4#4q#899 < >vFFT0<>J!M11!((%)<==== -''rN)__name__ __module__ __qualname____doc__rpropertyrsetterrrJr0r1r5r4re staticmethodr2r!rrrr1s:((((((T((X((((III747474r0"""H2GGG\rrcHeZdZdZ d dZdZdZdZd Zd Z d Z dS) r.a0 Partial solution to a translation. Records the word positions of the phrase being translated, its translation, raw score, and the cost of the untranslated parts of the sentence. When the next phrase is selected to build upon the partial solution, a new _Hypothesis object is created, with a back pointer to the previous hypothesis. To find out which words have been translated so far, look at the ``src_phrase_span`` in the hypothesis chain. Similarly, the translation output can be found by traversing up the chain. r r!NcL||_||_||_||_||_dS)a :param raw_score: Likelihood of hypothesis so far. Higher is better. Does not account for untranslated words. :type raw_score: float :param src_phrase_span: Span of word positions covered by the source phrase in this hypothesis expansion. For example, (2, 5) means that the phrase is from the second word up to, but not including the fifth word in the source sentence. :type src_phrase_span: tuple(int) :param trg_phrase: Translation of the source phrase in this hypothesis expansion :type trg_phrase: tuple(str) :param previous: Previous hypothesis before expansion to this one :type previous: _Hypothesis :param future_score: Approximate score for translating the remaining words not covered by this hypothesis. Higher means that the remaining words are easier to translate. :type future_score: float N)r'r(r)r*r5)rr'r(r)r*r5s rrz_Hypothesis.__init__as/>#.$  (rc |j|jzS)zd Overall score of hypothesis after accounting for local and global features )r'r5rs rr]z_Hypothesis.scores ~ 111rc|}|||g}d}|D]$}||kr|||f|dz}%|S)a. Starting from each untranslated word, find the longest continuous span of untranslated positions :param sentence_length: Length of source sentence being translated by the hypothesis :type sentence_length: int :rtype: list(tuple(int, int)) rr&)translated_positionssortrM)rr=r|rarOrPs rraz_Hypothesis.untranslated_spanss $88::!!#####O444'  Cs{{"))5#,777!GEE!!rcg}|}|jD|j}|t|d|d|j}|jD|S)z List of positions in the source sentence of words already translated. The list is not sorted. :rtype: list(int) Nrr&)r*r(extendr-)rr|current_hypothesistranslated_spans rr|z _Hypothesis.translated_positionssf "! )50@O ' 'oa.@/RSBT(U(U V V V!3!< !)5$#rcDt|Sr)r,r|rs rr6z"_Hypothesis.total_translated_wordss4,,..///rc6g}||||Sr)_Hypothesis__build_translation)r translations rr:z_Hypothesis.translation_so_fars#    {333rc|jdS||j|||jdSr)r*rrr))rrCoutputs r__build_translationz_Hypothesis.__build_translationsD   & F   !4f=== j+,,,,,r)r r!r!Nr ) rqrrrsrtrr]rar|r6r:rr!rrr.r.Rs    #)#)#)#)J222"""2 $ $ $000 -----rr.cBeZdZdZd dZdZdZdZdZd Z d Z e Z d S) r"z+ Collection of _Hypothesis objects r r c||_g|_|dkrtd|_dSt ||_dS)z :param beam_threshold: Hypotheses that score less than this factor of the best hypothesis are discarded from the stack. Value must be between 0.0 and 1.0. :type beam_threshold: float r rUN)max_sizeitemsrV_Stack__log_beam_thresholdr)rrrs rrz_Stack.__init__sG!  S (-f D % % %(+N(;(;D % % %rcB|j||jddt|j|jkr6|jt|j|jk6|dS)a Add ``hypothesis`` to the stack. Removes lowest scoring hypothesis if the stack is full. After insertion, hypotheses that score less than ``beam_threshold`` times the score of the best hypothesis are removed. c*|Sr)r])hs rrWz_Stack.push..saggiirT)keyreverseN)rrMr}r,rpopthreshold_prunerrCs rr/z _Stack.pushs *%%% //>>>$*oo -- JNN   $*oo -- rc|jsdS|jd|jz}t|jD]6}||kr|j4dSdSNr)rr]rreversedr)r thresholdrCs rrz_Stack.threshold_prunesz  FJqM''))D,EE "4:..  J!!I--        rc.|jr |jdSdS)ze :return: Hypothesis with the highest score in the stack :rtype: _Hypothesis rNrrs rr9z _Stack.bests : !:a= trc*t|jSr)iterrrs r__iter__z_Stack.__iter__sDJrc||jvSrrrs r __contains__z_Stack.__contains__sTZ''rc2t|jdkSr)r,rrs r__bool__z_Stack.__bool__s4:!##rN)r r ) rqrrrsrtrr/rr9rrr __nonzero__r!rrr"r"s < < < <         ((($$$KKKrr") rtr7 collectionsrmathrrr.r"r!rrrs""H######^^^^^^^^B o-o-o-o-o-o-o-o-d??????????r