Qd"dZddlZ ddlmZddlmZn#e$r dZdZdZYnwxYwej dZ Gd d Z d Z d Z e fd Ze fdZdZdZdS)z A port of the Gale-Church Aligner. Gale & Church (1993), A Program for Aligning Sentences in Bilingual Corpora. https://aclweb.org/anthology/J93-1004.pdf N)logsf)normct|}ddd|zzz }|tj| |zdz |d|d|d|d|d|d |d |d |d zzzzzzzzzzzzzzzzzzz}|d kr|Sd|z S)zComplementary error function.?gś??g5?g`yg?gƸ?gꪂIǿg#v?g9)gS?gޅ1Ogv(?gg@)absmathexp)xztrs :lib/python3.11/site-packages/nltk/translate/gale_church.pyerfccrs FF S1W   BF "' *"#$/&''1Aq:~9U4V'V'X%X#"!"       < 88H7NcVddt|tjdz zz S)u>Return the area under the normal distribution from M{-∞..x}.rr)rr sqrtr s rnorm_cdfr@s'3q49Q<</00000rc tjdt|z S#t$rt dcYSwxYw)Nr-inf)r logr ValueErrorfloatrs r norm_logsfrDsN !8A O,, , ! ! !==  !s#&AArc(eZdZdddddddZdZdZdS) LanguageIndependentgׁsF?g{Gz?gbX9ȶ?gI +?))rr)rr)rr)rr)rr)rrrg333333@N)__name__ __module__ __qualname__PRIORSAVERAGE_CHARACTERSVARIANCE_CHARACTERSrrrrNs>  Frrc g}t|t|f}|dkrtd|Dr ||\}}n&#t$r|ddz |ddz f}YNwxYwt|D]C}t|D]1}||d|z dz |d|z dz f2D|d|z |d|z f}|dkrtd|D|dddS)a Traverse the alignment cost from the tracebacks and retrieves appropriate sentence pairs. :param backlinks: A dictionary where the key is the alignment points and value is the cost (referencing the LanguageIndependent.PRIORS) :type backlinks: dict :param source_sents_lens: A list of target sentences' lengths :type source_sents_lens: list(int) :param target_sents_lens: A list of target sentences' lengths :type target_sents_lens: list(int) )rrc3"K|] }|dkV dS)rNr%).0ps r ztrace..ns&$>$>Q!V$>$>$>$>$>$>rrrN)lenall TypeErrorrangeappend) backlinkssource_sents_lenstarget_sents_lenslinkspositionsr ijs rtracer9`sZ E%&&,=(>(>?H f  $>$>X$>$>$>!>!>  X&DAqq     a!q9H H q I IA1XX I I hqkAo18A;?Q3FGHHHH IQK!OXa[1_5 f  $>$>X$>$>$>!>!>  2;s A A0/A0ctfdt|dD}tfdt|dD} |||jz zdz }||jz|z tj||jzz } n#t $rtdcYSwxYwttt| ztj |j |z S)aPReturns the log probability of the two sentences C{source_sents[i]}, C{target_sents[j]} being aligned with a specific C{alignment}. @param i: The offset of the source sentence. @param j: The offset of the target sentence. @param source_sents: The list of source sentence lengths. @param target_sents: The list of target sentence lengths. @param alignment: The alignment type, a tuple of two integers. @param params: The sentence alignment parameters. @returns: The log probability of a specific alignment between the two sentences, given the parameters. c34K|]}|z dz VdSrNr%)r(offsetr7 source_sentss rr*z!align_log_prob..0 M Mvl1v:>* M M M M M Mrrc34K|]}|z dz VdSr<r%)r(r=r8 target_sentss rr*z!align_log_prob..r?rrrr) sumr/r#r rr$ZeroDivisionErrorrLOG2rrrr") r7r8r>rA alignmentparamsl_sl_tmdeltas ```` ralign_log_probrK|s  M M M M My|9L9L M M M M MC M M M M My|9L9L M M M M MC322 2a 7v0036$) * *; ;   V}}Js5zz** *TXfmI6N-O-O O PPs$:BB;:B;c t|j}gg}i}tt |dzD] }tt |dzD]}t d}d} |D]`} d| dz } || dz } | t | ks| dkr3|| | t ||||| |z} | |kr| }| } a|t dkrd}| |||f<|d|t |dkr|d|gt|||S)aReturn the sentence alignment of two text blocks (usually paragraphs). >>> align_blocks([5,5,5], [7,7,7]) [(0, 0), (1, 1), (2, 2)] >>> align_blocks([10,5,5], [12,20]) [(0, 0), (1, 1), (2, 1)] >>> align_blocks([12,20], [10,5,5]) [(0, 0), (1, 1), (1, 2)] >>> align_blocks([10,2,10,10,2,10], [12,3,20,3,12]) [(0, 0), (1, 1), (2, 2), (3, 2), (4, 3), (5, 4)] @param source_sents_lens: The list of source sentence lengths. @param target_sents_lens: The list of target sentence lengths. @param params: the sentence alignment parameters. @return: The sentence alignments, a list of index pairs. rinfNr+rr) listr"keysr/r,rrKr0popr9)r2r3rFalignment_typesDr1r7r8min_dist min_alignaprev_iprev_jr)s r align_blocksrXs$6=--//00O AI 3())A- . .s,--122 # #AU||HI$ " "adQqTSVVG##vzzfIf%q+->6))x<< H !I5<<'' )Iq!f  bELL " " " " q66A:: EE!HHH  -/@ A AArct|t|krtdfdt||DS)aCreates the sentence alignment of two texts. Texts can consist of several blocks. Block boundaries cannot be crossed by sentence alignment links. Each block consists of a list that contains the lengths (in characters) of the sentences in this block. @param source_blocks: The list of blocks in the source text. @param target_blocks: The list of blocks in the target text. @param params: the sentence alignment parameters. @returns: A list of sentence alignment lists z>Source and target texts do not have the same number of blocks.c8g|]\}}t||Sr%)rX)r( source_block target_blockrFs r zalign_texts..s9    &L, \<88   r)r,rzip) source_blocks target_blocksrFs `r align_textsrasj =S//// L       *-m]*K*K   rc#XKfd} |V )zSplits an iterator C{it} at values of C{split_value}. Each instance of C{split_value} is swallowed. The iterator produces subiterators which need to be consumed fully before the next subiterator can be used. c3\K|}|kr |V}|kdSdSNnext)firstvit split_values r_chunk_iteratorz!split_at.._chunk_iteratorsF ;GGG A;rre)rirjrks`` rsplit_atrlsN )obggii((((()rc<fdt||DS)zParses a stream of tokens and splits it into sentences (using C{soft_delimiter} tokens) and blocks (using C{hard_delimiter} tokens) for use with the L{align_texts} function. cDg|]}dt|DS)c@g|]}td|DS)c34K|]}t|VdSrd)r,)r(tokens rr*z;parse_token_stream....s(44uE 444444r)rB)r( sentence_its rr]z1parse_token_stream...s?    44 444 4 4   rrl)r(block_itsoft_delimiters rr]z&parse_token_stream..sL       '.AA      rrs)streamruhard_delimiters ` rparse_token_streamrxs:     !88    r)__doc__r rrr scipy.stats ImportErrorrrrrDrr9rKrXrarlrxr%rrr|sV 4!((((((       1!1!1!%%%N111!!!!![1!htx{{$8QQQ8?R3B3B3B3Bl6I:)))$     s &&