Qd@dZddlZddlmZmZddlmZdZej Z dZ dZ ddl mZn#e$rd ZYnwxYwdZ d Z d Z Gd d eZGddeZGddeZGddeZGddZdS)z Provides scoring functions for a number of association measures through a generic, abstract implementation in ``NgramAssocMeasures``, and n-specific ``BigramAssocMeasures`` and ``TrigramAssocMeasures``. N)ABCMetaabstractmethodreducec*tj|SN)_mathlog2)xs 8lib/python3.11/site-packages/nltk/metrics/association.pyr s%*Q--c$td|S)Nc ||zSr)r ys r r z..s Qrr)ss r r r sV..22rg#B ;) fisher_exactctrNotImplementedError)_args_kwargss r rrs!!rc(eZdZdZdZeedZeedZe dZ edZ e dZ e dZ ed Ze d Ze d Ze d Ze d ZdS)NgramAssocMeasuresa An abstract class defining a collection of generic association measures. Each public method returns a score, taking the following arguments:: score_fn(count_of_ngram, (count_of_n-1gram_1, ..., count_of_n-1gram_j), (count_of_n-2gram_1, ..., count_of_n-2gram_k), ..., (count_of_1gram_1, ..., count_of_1gram_n), count_of_total_words) See ``BigramAssocMeasures`` and ``TrigramAssocMeasures`` Inheriting classes should define a property _n, and a method _contingency which calculates contingency values from marginals in order for all association measures defined here to be usable. rc td)z>Calculates values of a contingency table from marginal values.?The contingency table is not availablein the general ngram caser marginalss r _contingencyzNgramAssocMeasures._contingencyB" P   rc td)ACalculates values of contingency table marginals from its values.rr) contingencys r _marginalszNgramAssocMeasures._marginalsJr#rc#Kt}dtjD}ttD]/t fd|D|jdz zz V0dS)3Calculates expected values for a contingency table.cg|]}d|zS)r).0is r z7NgramAssocMeasures._expected_values..Vs...1Q...rc3~K|]6tfdtdjzDV7dS)c3@K|]}|zzk|VdSrr)r,r contr-js r z@NgramAssocMeasures._expected_values...]s9PPAa!eQ=O=OQ=O=O=O=OPPrN)sumrange_n)r,r2clsr1r-s @r r3z6NgramAssocMeasures._expected_values..\siPPPPPPq#&y)9)9PPPPPrr+N)r5r6r7len_product)r8r1n_allbitsr-s`` @r _expected_valuesz#NgramAssocMeasures._expected_valuesRsD ..cf ...s4yy!!  A!SVaZ( *      rc8|t|tz S)z Scores ngrams by their frequency)NGRAMTOTALr s r raw_freqzNgramAssocMeasures.raw_freqcs)E"222rc|tt|t|t|jdz zz z |tt zdzz S)zScores ngrams using Student's t test with independence hypothesis for unigrams, as in Manning and Schutze 5.3.1. r+g?)r?r:UNIGRAMSr@r7_SMALLr8r!s r student_tzNgramAssocMeasures.student_thsS e y*++y/?CFQJ/OP Q u  &3 ./ /rc|j|}||}tdt||DS)zZScores ngrams using Pearson's chi-square as in Manning and Schutze 5.3.3. c3BK|]\}}||z dz|tzz VdS)r4N)rDr,obsexps r r3z,NgramAssocMeasures.chi_sq..ys8UUcC#I!#sV|4UUUUUUr)r"r=r5zip)r8r!r1expss r chi_sqzNgramAssocMeasures.chi_sqrsK  s+##D))UUSt__UUUUUUrc|t|ddzt|tz S)zScores ngrams using a variant of mutual information. The keyword argument power sets an exponent (default 3) for the numerator. No logarithm of the result is calculated. power)r?getr:rC)r!kwargss r mi_likezNgramAssocMeasures.mi_like{s= 6::gq#9#99H h = =   rct|t|t|jdz zztt |t z S)z^Scores ngrams by pointwise mutual information, as in Manning and Schutze 5.4. r+)_log2r?r@r7r:rCrEs r pmizNgramAssocMeasures.pmisQ Yu% %(8SVaZ(HHIIE Yx( ) )M M   rc |j|}dtdt|||DzS)zFScores ngrams using likelihood ratios as in Manning and Schutze 5.3.4.r4c3fK|],\}}|t||tzz tzzV-dSr)_lnrDrIs r r3z6NgramAssocMeasures.likelihood_ratio..sU  S #cS6\*V344 4      r)r"r5rLr=r8r!r1s r likelihood_ratioz#NgramAssocMeasures.likelihood_ratios` s+3  c&:&:4&@&@AA       rct|t|t|jdz zz }|tt |t|z dz zS)z1Scores ngrams using the Poisson-Stirling measure.r+)r:rCr@r7r?rV)r8r!rKs r poisson_stirlingz#NgramAssocMeasures.poisson_stirlingsRy*++y/?CFQJ/OP55)9C)?#@#@1#DEErcV|j|}|dt|ddz S)z&Scores ngrams using the Jaccard index.rNr)r"r5r[s r jaccardzNgramAssocMeasures.jaccards0 s+AwT#2#Y''rN)__name__ __module__ __qualname____doc__r7 staticmethodrr"r' classmethodr=rArFrNrTrWr\r^r`rrr rr-sf$ B  ^\   ^\ [ 33\3//[/VV[V  \   [   [ FF[F (([(((rr) metaclassceZdZdZdZedZedZedZe dZ e dZ e dZ ed Z d S) BigramAssocMeasuresa A collection of bigram association measures. Each association measure is provided as a function with three arguments:: bigram_score_fn(n_ii, (n_ix, n_xi), n_xx) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_ii counts ``(w1, w2)``, i.e. the bigram being scored - n_ix counts ``(w1, *)`` - n_xi counts ``(*, w2)`` - n_xx counts ``(*, *)``, i.e. any bigram This may be shown with respect to a contingency table:: w1 ~w1 ------ ------ w2 | n_ii | n_oi | = n_xi ------ ------ ~w2 | n_io | n_oo | ------ ------ = n_ix TOTAL = n_xx r4c>|\}}||z }||z }|||||z |z |z fS)zECalculates values of a bigram contingency table from marginal values.r)n_ii n_ix_xi_tuplen_xxn_ixn_xin_oin_ios r r"z BigramAssocMeasures._contingencys<% td{d{dD$+"4t";<>> TrigramAssocMeasures._contingency(1, (1, 1, 1), (1, 73, 1), 2000) (1, 0, 0, 0, 0, 72, 0, 1927) r)n_iii n_iix_tuple n_ixx_tuplen_xxxn_iixn_ixin_xiin_ixxn_xixn_xxin_oiin_ioin_iion_ooin_oion_ioon_ooos r r"z!TrigramAssocMeasures._contingencys!,u +u    %- %- %- %-5=EMueUE5%GGrc|\}}}}}}}}|||z||z||zf||z|z|z||z|z|z||z|z|zft|fS)zCalculates values of contingency table marginals from its values. >>> TrigramAssocMeasures._marginals(1, 0, 0, 0, 0, 72, 0, 1927) (1, (1, 1, 1), (1, 73, 1), 2000) r5) r&rrrrrrrrs r r'zTrigramAssocMeasures._marginals&s BM>ueUE5%  U]EEM55= 9 %- %- %-      rNrarbrcrdr7rer"r'rrr rrs\& BHH\H$  \   rrcBeZdZdZdZedZedZdS)QuadgramAssocMeasuresaF A collection of quadgram association measures. Each association measure is provided as a function with five arguments:: trigram_score_fn(n_iiii, (n_iiix, n_iixi, n_ixii, n_xiii), (n_iixx, n_ixix, n_ixxi, n_xixi, n_xxii, n_xiix), (n_ixxx, n_xixx, n_xxix, n_xxxi), n_all) The arguments constitute the marginals of a contingency table, counting the occurrences of particular events in a corpus. The letter i in the suffix refers to the appearance of the word in question, while x indicates the appearance of any word. Thus, for example: - n_iiii counts ``(w1, w2, w3, w4)``, i.e. the quadgram being scored - n_ixxi counts ``(w1, *, *, w4)`` - n_xxxx counts ``(*, *, *, *)``, i.e. any quadgram ruc|\}}}}|\} } } } } }|\}}}}||z }||z }||z }| |z |z |z }| |z |z |z }| |z |z |z }||z |z |z |z |z |z |z }||z }||z |z |z }| |z |z |z }||z |z |z |z |z |z |z }| |z |z |z }||z |z |z |z |z |z |z }||z |z |z |z |z |z |z } ||z |z |z |z |z |z |z |z |z |z |z |z |z |z | z }!||||||||||||||| |!fS)zXCalculates values of a quadgram contingency table from marginal values. r)"n_iiii n_iiix_tuple n_iixx_tuple n_ixxx_tuplen_xxxxn_iiixn_iixin_ixiin_xiiin_iixxn_ixixn_ixxin_xixin_xxiin_xiixn_ixxxn_xixxn_xxixn_xxxin_oiiin_ioiin_iioin_ooiin_oioin_iooin_oooin_iiion_oiion_ioion_ooion_iioon_oioon_iooon_oooos" r r"z"QuadgramAssocMeasures._contingencyPs> ,8(;G8+7(&&&&6)F2&6)F2&6)F2&6)F2V;fDvMPVV&&6)F2&6)F2&6)F2V;fDvMPVV&6)F2&6)F2V;fDvMPVV&6)F2V;fDvMPVV                      (                !  rc|\}}}}}}}}} } } } } }}}|| z}||z}||z}||z}||z| z| z}||z| z| z}||z|z|z}||z|z|z}||z|z|z}||z| z| z}||z|z| z|z| z| z|z}||z|z| z|z| z| z|z}||z|z| z|z| z| z| z}||z|z|z|z|z|z|z}t|}|||||f||||||f||||f|fS)aCalculates values of contingency table marginals from its values. QuadgramAssocMeasures._marginals(1, 0, 2, 46, 552, 825, 2577, 34967, 1, 0, 2, 48, 7250, 9031, 28585, 356653) (1, (2, 553, 3, 1), (7804, 6, 3132, 1378, 49, 2), (38970, 17660, 100, 38970), 440540) r) r&rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr;s r r'z QuadgramAssocMeasures._marginalss. #                &&&&&6)F2&6)F2&6)F2&6)F2&6)F2&6)F2&6)F2V;fDvMPVV&6)F2V;fDvMPVV&6)F2V;fDvMPVV&6)F2V;fDvMPVVK    VVV , VVVVV < VVV ,    rNrrrr rr9sZ( B9 9 \9 v1 1 \1 1 1 rrc.eZdZdZdZedZdS)ContingencyMeasureszWraps NgramAssocMeasures classes such that the arguments of association measures are contingency table values rather than marginals. c d|jjz|j_t|D]d}|drt ||}|ds|||}t |||edS)zAConstructs a ContingencyMeasures given a NgramAssocMeasures class Contingency___N) __class__radir startswithgetattr_make_contingency_fnsetattr)selfmeasureskvs r __init__zContingencyMeasures.__init__s"/(2D2M"MX  A||D!! !$$A<<$$ ;--h:: D!Q       rcFfd}j|_j|_|S)zFrom an association measure function, produces a new function which accepts contingency table values as its arguments. cj|Sr)r')r&rold_fns r resz5ContingencyMeasures._make_contingency_fn..ress6.8. <= =r)rdra)rrrs`` r rz(ContingencyMeasures._make_contingency_fns;  > > > > > >n   rN)rarbrcrdrrerrrr rrsH      \   rr)rdmathr abcrr functoolsrrVlogrZr:rD scipy.statsr ImportErrorr?rCr@rrirrrrrr rs '''''''' i 2 2 "(((((((""""""""" ) 7 9t(t(t(t(t(7t(t(t(t(nV(V(V(V(V(,V(V(V(r9 9 9 9 9 -9 9 9 xE E E E E .E E E Ps ,77