CdddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z ddlmZmZddlZddlmZmZmZmZmZmZddlmZdd lmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)m*Z*dd l+m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8ddl9m:Z:ddl;mZ>ddl?m@Z@mAZAmBZBmCZCmDZDmEZEd[dZFdZGdZH d\dZI d]dZJ d^dZKeBed_dZLeBed_dZMe&ejNejOeBed`dZNdad ZPe&ejQejReBed`d!ZQdad"ZSeBed`d#ZTeBed`d$ZUeBed_d%ZVeBed_d&ZWeBedad'd(d)ZXeBedad'd(d*ZYeBed`d+ZZd,Z[eBed`d-Z\d.Z]ejLeGd/dfd0Z^ejLeGd/ddfd1Z_dbd2Z`eBed_d3ZaeBed_d4Zbd5ejLeGd/dfd6Zcd7Zdd5dd/ejLddfd8Zed5dd/ejLddfd9Zf dcd:ZgeBedcd;ZheBe dcd<Zid=Zjd>ZkeBedcd?ZleBe dcd@ZmdddAZndBZod[dCZpdedDZqdedEZr d`dFZsdGZtdHZueBed`dIZveBed`dJZweBed`dKZxeBed`dLZydMZzdNZ{dfdOZ| dgdPZ}dQZ~dRZeBedhdSZeBedhdTZdidVZdidWZeBedjdXZeBedkdYZeBedkdZZdS)l) annotationsN)Iterable)partial)productrepeat)IntegralNumber) accumulatecomposedropget partition_allpluck)config)chunk) blockwise)Array _concatenate2 asanyarray broadcast_to handle_out implementsunknown_chunk_message)arangediagonal) divide_lookupnannumel_lookup numel_lookup) array_safe asarray_safe compute_meta is_arraylikemeta_from_array validate_axis)oneszeros)tokenize) lol_tuples)HighLevelGraph)applydeepmap derived_fromfuncname getargspecis_series_likec d}tjttj|||}||||S)Nc>t|dtdS)N__array_priority__z-inf)getattrfloat)xs 5lib/python3.11/site-packages/dask/array/reductions.pyzdivide..5sGA3U6]]CC)keydtype)rdispatchtypebuiltinsmax)abr;r9fs r6dividerC4sJ C CCtHLA3$?$?$?@@AAA 1Q   r8c t|fi|SN)rr5kwargss r6numelrH:s  $ $V $ $$r8c t|fi|SrE)rrFs r6nannumelrJ>s 1 ' ' ' ''r8FTc  !tt|jttrft |j|t ddt|jvrt||}dt|jvrt||}t|r|j }tt|j}||f}| Zt| } t||j}n-#t $r t d|jd|jwxYw|||fz }t||g|Rd||pt d}t fd t#|jD|_| jt)|d rZ t+||j|jdd }n;#t0$r!t+||j|jd }Ynt $rYnwxYwd}t3|||||||| | }|r9 dkr3t fdt#|jD|_| | |_t5| |S)anGeneral version of reductions Parameters ---------- x: Array Data being reduced along one or more axes chunk: callable(x_chunk, [weights_chunk=None], axis, keepdims) First function to be executed when resolving the dask graph. This function is applied in parallel to all original chunks of x. See below for function parameters. combine: callable(x_chunk, axis, keepdims), optional Function used for intermediate recursive aggregation (see split_every below). If omitted, it defaults to aggregate. If the reduction can be performed in less than 3 steps, it will not be invoked at all. aggregate: callable(x_chunk, axis, keepdims) Last function to be executed when resolving the dask graph, producing the final output. It is always invoked, even when the reduced Array counts a single chunk along the reduced axes. axis: int or sequence of ints, optional Axis or axes to aggregate upon. If omitted, aggregate along all axes. keepdims: boolean, optional Whether the reduction function should preserve the reduced axes, leaving them at size ``output_size``, or remove them. dtype: np.dtype data type of output. This argument was previously optional, but leaving as ``None`` will now raise an exception. split_every: int >= 2 or dict(axis: int), optional Determines the depth of the recursive aggregation. If set to or more than the number of input chunks, the aggregation will be performed in two steps, one ``chunk`` function per input chunk and a single ``aggregate`` function at the end. If set to less than that, an intermediate ``combine`` function will be used, so that any one ``combine`` or ``aggregate`` function has no more than ``split_every`` inputs. The depth of the aggregation graph will be :math:`log_{split_every}(input chunks along reduced axes)`. Setting to a low value can reduce cache size and network transfers, at the cost of more CPU and a larger dask graph. Omit to let dask heuristically decide a good default. A default can also be set globally with the ``split_every`` key in :mod:`dask.config`. name: str, optional Prefix of the keys of the intermediate and output nodes. If omitted it defaults to the function names. out: Array, optional Another dask array whose contents will be replaced. Omit to create a new one. Note that, unlike in numpy, this setting gives no performance benefits whatsoever, but can still be useful if one needs to preserve the references to a previously existing Array. concatenate: bool, optional If True (the default), the outputs of the ``chunk``/``combine`` functions are concatenated into a single np.array before being passed to the ``combine``/``aggregate`` functions. If False, the input of ``combine`` and ``aggregate`` will be either a list of the raw outputs of the previous step or a single output, and the function will have to concatenate it itself. It can be useful to set this to False if the chunk and/or combine steps do not produce np.arrays. output_size: int >= 1, optional Size of the output of the ``aggregate`` function along the reduced axes. Ignored if keepdims is False. weights : array_like, optional Weights to be used in the reduction of `x`. Will be automatically broadcast to the shape of `x`, and so must have a compatible shape. For instance, if `x` has shape ``(3, 4)`` then acceptable shapes for `weights` are ``(3, 4)``, ``(4,)``, ``(3, 1)``, ``(1, 1)``, ``(1)``, and ``()``. Returns ------- dask array **Function Parameters** x_chunk: numpy.ndarray Individual input chunk. For ``chunk`` functions, it is one of the original chunks of x. For ``combine`` and ``aggregate`` functions, it's the concatenation of the outputs produced by the previous ``chunk`` or ``combine`` functions. If concatenate=False, it's a list of the raw outputs from the previous functions. weights_chunk: numpy.ndarray, optional Only applicable to the ``chunk`` function. Weights, with the same shape as `x_chunk`, to be applied during the reduction of the individual input chunk. If ``weights`` have not been provided then the function may omit this parameter. When `weights_chunk` is included then it must occur immediately after the `x_chunk` parameter, and must also have a default value for cases when ``weights`` are not provided. axis: tuple Normalized list of axes to reduce upon, e.g. ``(0, )`` Scalar, negative, and None axes have been normalized away. Note that some numpy reduction functions cannot reduce along multiple axes at once and strictly require an int in input. Such functions have to be wrapped to cope. keepdims: bool Whether the reduction function should preserve the reduced axes or remove them. NzMust specify dtyper;r:zWeights with shape z' are not broadcastable to x with shape T)axiskeepdimstokenr;c3PK|] \}}|vrft|zn|V!dSrElen.0icrM output_sizes r6 zreduction..sR8<1199Q!r8_meta)rMrNcomputing_metarMrN)name concatenate reduced_metarKc30K|]\}}|vrfn|VdSrErSs r6rXzreduction..sF  371aa4ii[NNQ      r8)tuplerangendim isinstancerr$ ValueErrorr.argsrr/valuesrrshaperr4 enumeratechunks_chunkshasattrr!r;rY TypeError _tree_reducer)r5r aggregaterMrNr; split_everycombiner\outr]rWmetaweightsindsrfwgttmpr^results ` ` r6 reductionryBsDf |U16]]##$!!w qv & &D }-...*U##(((U+++*Y'',,,IU333 a H qv  D t9D!! sAG,,CC   -ci--#$7--   d   t   !%DQV   C@I#*@U@UCK |7++| 'qwdTRVLL   'qwdTLLL    D      !   F K1$$     ;DSZ;P;P       c6 " ""s$?D*D?*G (H4 H?Hc @ptjddttrfd|DnwttrSt jtdt|pdz zd} t || ntdd} t|j D]b\} } | vrY| dkrMtt j| tjtj| | } ct!|p||d} |r1t#| t!t$t'| } t)| dz D],}t+| |d||pt-|p|d z| }-t!|||} |r1t#| t!t$t'| } t+| ||||pt-|d z| S)zPerform the tree reduction step of a reduction. Lower level, users should use ``reduction`` or ``arg_reduction`` directly. rpc>i|]}||dS)r )rTkrps r6 z _tree_reduce..s)>>>Aq+//!Q//>>>r8rKr}z#split_every must be a int or a dictTr[axesz-partial)r;r\r^z -aggregate)rNr;r\r^)rr rddictrr>r?intrRfromkeysreri numblocksmathceillogrr rsortedrbpartial_reducer-)r5rorMrNr;rprqr\r]r^ndepthrUfunc_s ` r6rnrns@"=M1!=!=K+t$$@>>>>>>> K * *@ L[Q#d)).q-ABCCQ G GmmD!,, >??? E!+&&UU1   A! 3 3 UDIdhq+a.6Q6Q,R,RSSTTE 7'idT B B BDHtW]FFFGG 519         8(7#7i88JF%    94( ; ; ;DHtW]FFFGG   )hy))\ 9!   r8c |pt|dzt||||z}fdt|jD}t t t t t|}fdt|jD} |sMfdt |j Dfd} t | |}t| | } i} t|t |D]}\} } fdt| D}tfdt| D}t|jft |j ||}||f| |f| z<~tj|| |g}|j}|a ||d }nR#t$$r: ||}n*#t&$r}d t)|vr|}Yd }~nd }~wwxYwYnt&$rYnwxYwt+|re|j t| krMt| d kr|}n%|dt| z}t1j|rt5||| |St7jt:5||}d d d n #1swxYwYt5||| |S)aPartial reduction across multiple axes. Parameters ---------- func : function x : Array split_every : dict Maximum reduction block sizes in each dimension. Examples -------- Reduce across axis 0 and 2, merging a maximum of 1 block in the 0th dimension, and 3 blocks in the 2nd dimension: >>> partial_reduce(np.min, x, {0: 1, 2: 3}) # doctest: +SKIP -c g|]C\}}tt|dt|DSrK)listrr rb)rTrUrrps r6 z"partial_reduce..PsS    Q ];??1a00%(( ; ;<<   r8c |g|]8\}}|vr-tdt||Dn|9S)c3K|]}dVdSrKNr`)rTps r6rXz,partial_reduce...Vs" : :Aa : : : : : :r8)rar)rTrUrVrps r6rz"partial_reduce..Usc Q?@;>N>N : :{1~q99 : : ::::TUr8cg|]}|v| Sr`r`rTrUrps r6rz"partial_reduce..Zs#EEE!0D0DA0D0D0Dr8c$t|SrEr~)rout_axiss r6r7z partial_reduce..[s3x++r8cVi|]%\}}t|dk|v||d&S)rKrrQ)rTrUjrps r6rz"partial_reduce..`sB   A#a&&A++!;BVBVAqtBVBVBVr8c30K|]}|dv |VdS)rNr`rs r6rXz!partial_reduce..cs1DD1! 0C0CQ0C0C0C0CDDr8 dependenciesNT)rZz&zero-size array to reduction operationrrr:rs)r-r'rirrmaprbrRrjrcrziprr(r\r)from_collectionsrYrmrestrr"sumreshapenpisscalarr contextlibsuppressAttributeErrorastype)rr5rprNr;r\r^partskeys out_chunksgetterdskrrfreedummyggraphrsers ` @r6rr:s(  $3&$;RW)X)XX      ,,   E Cs3// 0D))J .EEEEuQV}}EEE++++64  &&,,-- CD'5/**%%1    "+A,,   DDDD ! DDDDD y%--u = = !9TGaK  +D#QC H H HE 7D 4 T:::DD ( ( ( (tL)) ( ( (;s1vvEE'D (     D  D8di3z??:: z??a  88::DD<<s: 677D {49UD*E::::   0 0 & &;;u%%D & & & & & & & & & & & & & & &UD*48888sT" F00 G?; GG? G.G)$G?)G..G?3 G?>G?6KKKc |Attjd|jdt }t |tjtj|||||}|S)NrKr:r;rMrNr;rprr)r3rr&r;robjectryr)r@rMr;rNrprrrxs r6rrsl }!'2226688'6JJ        F Mr8c ||}nAttjd|jdt }t |tjtj|||||SNrr:r;r)r3rr%r;prodrryrr@rMr;rNrprrdts r6rrsp   RWT1116688'6 J J         r8c dt|ttjt|||j|| SN)rqrMrNr;rprr)ry chunk_minrminr;r@rMrNrprrs r6rr:    g     r8c|jdkrtg||j|jSt j|||S)z-Version of np.min which ignores size 0 arraysrndminr;r[)sizerrcr;rrr5rMrNs r6rr?v{{"aqvQW====vadX6666r8c dt|ttjt|||j|| Sr)ry chunk_maxrr?r;rs r6r?r?rr8c|jdkrtg||j|jSt j|||S)z-Version of np.max which ignores size 0 arraysrrr[)rrrcr;rr?rs r6rrrr8c Xt|tjtj||d||SNboolr)ryranyrs r6rr5         r8c Xt|tjtj||d||Sr)ryrallrs r6rrrr8c ||}nAttjtjd|jdt }t|tjtj|||||Sr) r3rnansumrr%r;rryrrs r6rrsn   U\"'$ag">">">??& Q Q         r8c ||}nAttjtjd|jdt }t|tjtj |||||Sr) r3rrrr%r;rrynanprodrrs r6rr sn   U\"'$ag">">">??& Q Q         r8 sequential)methodc nttjtjd|||||t j S)aiDask added an additional keyword-only argument ``method``. method : {'sequential', 'blelloch'}, optional Choose which method to use to perform the cumsum. Default is 'sequential'. * 'sequential' performs the cumsum of each prior block before the current block. * 'blelloch' is a work-efficient parallel cumsum. It exposes parallelism by first taking the sum of each block and combines the sums via a binary tree. This method may be faster or more memory efficient depending on workload, scheduler, and hardware. More benchmarking is necessary. rrrrpreop) cumreductionr nancumsumoperatoraddrrr5rMr;rrrs r6rrs<       i    r8c nttjtjd|||||t j S)atDask added an additional keyword-only argument ``method``. method : {'sequential', 'blelloch'}, optional Choose which method to use to perform the cumprod. Default is 'sequential'. * 'sequential' performs the cumprod of each prior block before the current block. * 'blelloch' is a work-efficient parallel cumprod. It exposes parallelism by first taking the product of each block and combines the products via a binary tree. This method may be faster or more memory efficient depending on workload, scheduler, and hardware. More benchmarking is necessary. rKr)rr nancumprodrmulrrrs r6rr6s=       j    r8c tj|jrtdt|jdkrtdt |t t |||j||S)N Arrays chunk sizes are unknown. rzAzero-size array to reduction operation fmin which has no identityr)risnanrrerry _nanmin_skipr;rs r6nanminrP xUS. d3ir8rrMc|dSNrr`r s r6r7zmean_combine.. $w-r8rr`)rdrr+rr) pairsrrHr;rMrZrGnsrtotalsrs r6 mean_combiners eT " "7E P'' / / /5B( bt$$$(==d=f==A // 7 7F 0M&t , , , 0 E Ed Ef E EEU # ##r8c d|std|n|}t||}tj|f||d|}|r|Std|}t||jd||d|}tjdd5t |||cdddS#1swxYwYdS) Nc|dSr r`r s r6r7zmean_agg..r r8rrMr;c|dSrr`r s r6r7zmean_agg..rr8rrCinvalidr:r`)r+rrrerrstaterC) rr;rMrZrGrrrrs r6mean_aggrs&7E P'' / / /5Bbt$$$A q3t533F33A // 7 7F 0M&t , , , 0 Rd% R R6 R RE Hh 7 7 7--eQe,,,------------------sB%%B),B)c ||}nY|jtkrt}nAttjtjd|jdt}t |tt||||t|d S)Nrrhr;r;FrMrNr;rprqrrr]) r;rr3rmeanr&ryrrrrs r6rrs   F    RWRXD@@@AA7F S S        r8cZ||}nAttjtjd|jdt }t |tttj tt|||||dtttj t S)Nrrr;rrHFrMrNr;rprrr]rq)r3rrr%r;rryrrrrrJrrrs r6nanmeanr#s   RWRW4qw???@@'6 R R    H===   %,hGGG    r8r}c ^ |rS|fi}|tj}fdi}tjdd5||z dddn #1swxYwY fdt d|dzD} tj| d} ||| d S) Nr;rrc2g|]}z |zfdiSr:r`)rTrUAr;rGrus r6rz moment_chunk..s8 P P Pq##q1ul 2 2% 26 2 2 P P Pr8r}rKrMrrM)rrint64rrbstack) r&orderrrHr;rZrGrrxsr+r's ` ` ` ` @r6 moment_chunkr0s& a6A A C ) ) )& ) )E Hh 7 7 7 AI P P P P P P P PE!UQY33D3F33cc Z 77%)7-377 A1eai VVu%%):):T^ETUI=V=V)VW USSCQ./*a-?UUdUfUU UU Hr8c t|ts|g}|d<dd<|std|n|tjddi}|r|Sttd|} ttd|| jddi} t jd d 5t| || } t| | | z dddn #1swxYwYfd td |dzD} t j | d} | || dS)Nr;TrNc|dSr r`r s r6r7z moment_combine..r r8rrMc|dSrr`r s r6r7z moment_combine..  W r8c|dSNr+r`r s r6r7z moment_combine..! DIr8rrr:c :g|]}t|Sr`)r6)rTor3rMr4rGrrs r6rz"moment_combine..)s=     r2z1c4@@   r8r}rKr(r)r*r`) rdrr+rrrrrCrbr-)rr.ddofr;rrMrZrGrrrmur/r+r3r4rs `` ` @@@r6moment_combinerB s eT " "F7OF:7E P'' / / /5B r % % %B##D#F##A 7#=#=uEED Q Q QF w55u==D I I IB FJ + +D +F + +E Hh 7 7 7:: E1E * * *FBe444r9 :::::::::::::::         q%!)$$   B "A + ++s(C;;C?C?c 2t|ts|g}||d<|}d|d<|std|n|} t | |} | jd d|i|} |r| St td||} t td||} t | jd d|i|| | } tjd d 5t | | | | z }dddn #1swxYwYt| | |||||}| jd d|i||z }t|tr|d kr tj }n&|tj j urtj ||d k<t ||| S)Nr;TrNc|dSr r`r s r6r7zmoment_agg..Dr r8rrMc|dSrr`r s r6r7zmoment_agg..Kr:r8c|dSr<r`r s r6r7zmoment_agg..Lr=r8r:rrrr`)rdrcopyr+rrrCrrr6r nanmamasked)rr.r@r;rrMrZrG keepdim_kwrrrr3rAr4r+ denominators r6 moment_aggrM1s eT " "F7OJ!Jz7E P'' / / /5B r % % %B''D'J''A 7#=#=uEED Q Q QF w55u==D I I IB   333 33Qe D D DB Hh 7 7 7::FBe444r9 ::::::::::::::: r2z5#tVDDA!%,,T,V,,t3K+v&&. ??&K BEL ( (')v K!O$ ![ . . ..s%DD D cZt|tr|dkrtd|dkr`||}|dkr"t |j|jd|jSt|j|jd|jS||} nAttj tjd|j d t} t|tt | tt"|| ||| ||d tt$|  S)a Calculate the nth centralized moment. Parameters ---------- a : Array Data over which to compute moment order : int Order of the moment that is returned, must be >= 2. axis : int, optional Axis along which the central moment is computed. The default is to compute the moment of the flattened array. dtype : data-type, optional Type to use in computing the moment. For arrays of integer type the default is float64; for arrays of float types it is the same as the array type. keepdims : bool, optional If this is set to True, the axes which are reduced are left in the result as dimensions with size one. With this option, the result will broadcast correctly against the original array. ddof : int, optional "Delta Degrees of Freedom": the divisor used in the calculation is N - ddof, where N represents the number of elements. By default ddof is zero. Returns ------- moment : Array References ---------- .. [1] Pebay, Philippe (2008), "Formulas for Robust, One-Pass Parallel Computation of Covariances and Arbitrary-Order Statistical Moments", Technical Report SAND2008-6212, Sandia National Laboratories. rzOrder must be an integer >= 0r}r)r)rjr;rsNrrr;)r.)r.r@Fr")rdrrerr%rhrjrYr&r3rvarr;rryrr0rMrB) r@r.rMr;rNr@rprrreducedrs r6momentrQas:L eX & &:%!))8999 qyy%%T%"" A:: gnDw}  M'.7=       RVBG$ag>>>??& Q Q   E*** %d333  e444    r8c ||}nAttjtjd|jdt }t |ttt|||||td|d S)Nrrr;)r@rOF)rMrNr;rprqr\rrr]) r3rrOr%r;rryr0rrMrBr@rMr;rNr@rprrrs r6rOrOs   RVBG$ag>>>??& Q Q   &&&       r8c ||}nAttjtjd|jdt }t |tttj ttttj |||||tttj |d S) Nrrr;r!)rr@)rFr)r3rrOr%r;rryrr0rrrJrMrBrSs r6nanvarrUs   RVBG$ag>>>??& Q Q   %,h???  555 BI666     r8ctj|}t|tjjr1|js*|jrtjjS|SrE) rsqrtrdrI masked_arrayrhmaskrrJ)r@r?s r6_sqrtrZsO  A!RU'((QVZZ\\u| Hr8cvt|dr|t|St|S)aA version of sqrt that properly handles scalar masked arrays. To mimic ``np.ma`` reductions, we need to convert scalar masked arrays that have an active mask to the ``np.ma.masked`` singleton. This is properly handled automatically for reduction code, but not for ufuncs. We implement a simple version here, since calling `np.ma.sqrt` everywhere is significantly more expensive. _elemwise)rlr\rZ)r@s r6 safe_sqrtr]s5q+%{{5!$$$ 88Or8c tt|||||||}|r ||jkr||}|SN)rMr;rNr@rprr)r]rOr;rr@rMr;rNr@rprrrxs r6stdrasd  #     F &&,&&u%% Mr8c tt|||||||}|r ||jkr||}|Sr_)r]rUr;rr`s r6nanstdrcsf #     F &&,&&u%% Mr8c2t|trX|dj|djksJt||djks|djdkrdn|d}n-t||jks |jdkrdn|d}|d}|d}|C||||}||}||}n|||}t jttt|j }| ||t|}||}||}|r*t j ||}t j ||}||fS)z3Merge intermediate results from ``arg_*`` functionsvalsargrKNrr[r)) rdrrcrRravelrogridrarslicerhinsert expand_dims)datarMargfuncrNrerf local_argsrus r6 _arg_combineros$ MF| DK$444444yyDL---f1Ba1G1G Da  4yyDI--attT!W $--D.d++C 9r8cht||jks |jdkrdn|d}|||d}|||d}|jdkr|y|\}} tj|d|j} t dt|| D} tj| | |dd<n||z }t|tj j rhd|j vr tj |} ntj |} tj || } tj||jd|jfd|jfg } n#t$$rt'} YnwxYw|| d<|| d<| S) NrKrTr[c3&K|] \}}||zV dSrEr`)rTr?rUs r6rXzarg_chunk..7s*CCAa!eCCCCCCr8rrerfr)rRrcr unravel_indexrgrhrarravel_multi_indexrdrIrX__name__minimum_fill_valuemaximum_fill_valuefilled empty_liker;rmr)rrmr5rM offset_infoarg_axisrerfoffset total_shapeind total_ind fill_valuerxs r6 arg_chunkr/s4yyAF**afkktttAwH 44 0 0 0D '!(T 2 2 2Cvzz  "- FK"399;;q>17;;CCC#fc2B2BCCCCCI))[AACFF ; C$*++. G$ $ $11$77JJ11$77Ju||D*--  FDJ+?%AS*T    F6NF5M Ms,F F%$F%c t|||d\}} tj||jd|jfd|jfg}n#t $rt }YnwxYw||d<||d<|S)NTrNrerfr)rorrxrhr;rmr)rmrlrMrGrfrerxs r6 arg_combinerPsT44@@@IC  FDJ+?%AS*T    F6NF5M Ms,AA A c 4t||||dS)Nrr)ro)rmrlrMrNrGs r6arg_aggr`s dGh ? ? ? BBr8c t||||\}}tjtj|rt d|S)NrzAll NaN slice encountered)rorrrre)rmrlrMrNrGrfres r6 nanarg_aggrdsJT48DDDIC vbhtnn64555 Jr8c $tt|jd}nLttr$t |jf|jdk}nt ddD]W} |j| } t| dkr5tj |  rtdXdt||||jtt!t#t|j} tt!d|jD} |r#t'| t)|j} nt-d | } tfd t/|jD} fd t'| | D}tjt3dgt5| }d}t7|r |}|j}t;j||g }t?|| ||}tA||||||}tC||S)a7Generic function for argreduction. Parameters ---------- x : Array chunk : callable Partialed ``arg_chunk``. combine : callable Partialed ``arg_combine``. agg : callable Partialed ``arg_agg``. axis : int, optional split_every : int or dict, optional NTrKz(axis must be either `None` or int, got ''zArg-reductions do not work with arrays that have unknown chunksizes. At some point in your computation this array lost chunking information. A possible solution is with x.compute_chunk_sizes()z arg-reduce-c3\K|]'}ttj|dddV(dS)Nr(r)r rr)rTbds r6rXz arg_reduction..s8UUbZ b"gqAAUUUUUUr8rc3NK|]\}}|vrdt|zn|V dS)rNrQ)rTrUrVrMs r6rXz arg_reduction..s=WW!QAII4#a&&==1WWWWWWr8c4i|]\}}f|zf|z|fSr`r`)rTroffrMrr\olds r6rz!arg_reduction..sD    Q ! ecVaZs3   r8rr)r;rs)rNr;rprq)"rarbrcrdrr$rmrjrRrrrrer'r\rrrrrrrhrriargminr r#r"r;r)rrrnr)r5rrqaggrMrNrprrrgaxrjroffsetsryrr;rsrrwrxr\rs ` ` @@r6 arg_reductionrks" |U16]]## D( # #LT16**w! J4JJJKKK  " v;;??rx//3355?,  J$5';GG I ID &C UAK001 2 2D7UUAHUUUVWWG .'6!'??33 DGW-- WWWW9QXCVCVWWW W WF       D+..   C IlA3_Q-?-?@@@ A AE DE   +D#QC H H HE tV5t < < ..Y1VVVeaqDyy%)))qcVVVr8)rrrir)rTrUrMr5s @r6rz'prefixscan_blelloch..Vsb+++  VVVVVy?U?UVVV   +++r8c,g|]}fd|DS)c&g|] }jf|zSr`)r\)rTindexbatchess r6rz2prefixscan_blelloch...^s">>>W\Oe+>>>r8r`)rTrers r6rz'prefixscan_blelloch..^s.SSS4>>>>>>>SSSr8r}rKr)"flattenrechunk npartitionsr3rr%r;rrdrr$rcrtr' map_blocksrbrrRrappendr>r?rrlog2rr\r rr)rrrjr)rrrr5rMr;rrr\base_keyindices last_index full_indices prefix_valsrn_valslevelstridestride2rUnew_valsrleft_val right_valr9indexesrevaldepsrrxrs `` @r6prefixscan_blellochr,sh: | IIKK  q}  5 5 }RWT999::GVLL dH % %%%% qv & &Dm L LhtT5%EJJ L LDwHll5td%lHHG+++++ q{4()) +++Wj<TSSS7SSSK C   F E {{7Q;88 * *25AJ AJ 7Q33)).E8Y#U*eQZ7C %x;CHOOC((((!) AF qLG QJE",q!ty6Q;1G1G'H'H"HIIAqjj7V+a/AA * *25AJ AJ 7Q33)).E8Y#U*eQZ7C %x;CHOOC((((!) AG qLF QJEqjj!!_  E! E! %C5 ! !!a!6!6 DD  MGT!'400   s'VI%)Hu$%%  <1s7|  +D#D I I IE 5$'- 8 8F c6 " ""r8c |dkr'|tdt|||||S|dkrtd|/jd|8t |tjd j d t}ttsJtj|| } |jd t#||||} j} t'ddd} | fzt'ddfz| fjz dz zz} t)t+fdt-jD}t/}|D]i}t1fdt-|D}t2tjj|| j fd|if|| df|z<| jf|z|| f|z<jt;d| D]|}t)t+fdt-jD}t=||D]D\}}| df|z}|| df|zt>j | jf|z| ff||<||| jf|zf|| f|z<EtCj"| || g}tG|| j$| j j}tK||S)aGeneric function for cumulative reduction Parameters ---------- func: callable Cumulative function like np.cumsum or np.cumprod binop: callable Associated binary operator like ``np.cumsum->add`` or ``np.cumprod->mul`` ident: Number Associated identity like ``np.cumsum->0`` or ``np.cumprod->1`` x: dask Array axis: int dtype: dtype method : {'sequential', 'blelloch'}, optional Choose which method to use to perform the cumsum. Default is 'sequential'. * 'sequential' performs the scan of each prior block before the current block. * 'blelloch' is a work-efficient parallel scan. It exposes parallelism by first calling ``preop`` on each block and combines the values via a binary tree. This method may be faster or more memory efficient depending on workload, scheduler, and hardware. More benchmarking is necessary. preop: callable, optional Function used by 'blelloch' method, like ``np.cumsum->np.sum`` or ``np.cumprod->np.prod`` Returns ------- dask array See also -------- cumsum cumprod blellochNz>cumreduction with "blelloch" method required `preop=` argument)rrrzMInvalid method for cumreduction. Expected "sequential" or "blelloch". Got: rrrr:r;rrr(rKcFg|]\}}|krt|ndgSrr)rTrUrrMs r6rz cumreduction..rr8c3TK|]"\}}|krj||ndV#dSrr)rTrUiirMr5s r6rXzcumreduction..s=VVeadahqk"ooVVVVVVr8rhextracFg|]\}}|krt|ngSr`r)rTrrrMrUs r6rz cumreduction..s1XXXvr2rTzz%)))sXXXr8rr)&rmrrerrrr3rr%r;rrdrr$rcrrtr'rrirrrirrar* full_likerYr\rbrrgetitemr)rrrjr)rridentr5rMr;rrrrmr\rfullslcrrr}rh last_indicesr this_slicerrxrUs `` @r6rrs|Z =P #4q$3OOOO <   f\b f f    | IIKK  q}  5 5 }RWT999::GVLL dH % %%%% qv & &D TE 22Am L LhtT5%EJJ L LD DA tT " "D 'D.E"dOO- -16D=1;L0M MCVVVVy?U?UVVVWG &&C--VVVVVyQT~~VVVVV  L WeQW % e  & T7Oc !"  fY_TGcM 1a[[FF  XXXXX1;AWAWXXX    L'22 F FHC3.Jw#%!AF9s?C8C O #(afY_!EC#   F  +D#QC H H HE 5$!' @ @ @F c6 " ""r8ct|tjjst|tjjr}tj|tj|z}tj|tj|S||zSN)rYrdrrIrXgetdata getmaskarrayr@rArgs r6 _cumsum_merger!RU'((FJq"%:L,M,MFq!!BEMM!$4$44u!!&ru/A/A!/D/D!EEE q5Lr8ct|tjjst|tjjr}tj|tj|z}tj|tj|S||zSrrrs r6_cumprod_mergerrr8c dttjtd|||||tj S)acDask added an additional keyword-only argument ``method``. method : {'sequential', 'blelloch'}, optional Choose which method to use to perform the cumsum. Default is 'sequential'. * 'sequential' performs the cumsum of each prior block before the current block. * 'blelloch' is a work-efficient parallel cumsum. It exposes parallelism by first taking the sum of each block and combines the sums via a binary tree. This method may be faster or more memory efficient depending on workload, scheduler, and hardware. More benchmarking is necessary. rr)rrcumsumrrrs r6rrs:      f    r8c dttjtd|||||tj S)anDask added an additional keyword-only argument ``method``. method : {'sequential', 'blelloch'}, optional Choose which method to use to perform the cumprod. Default is 'sequential'. * 'sequential' performs the cumprod of each prior block before the current block. * 'blelloch' is a work-efficient parallel cumprod. It exposes parallelism by first taking the product of each block and combines the products via a binary tree. This method may be faster or more memory efficient depending on workload, scheduler, and hardware. More benchmarking is necessary. rKr)rrcumprodrrrs r6rr6s:      g    r8r(c t||j}ttj|}ttj|}t |||||d|j|t| S)aExtract the k largest elements from a on the given axis, and return them sorted from largest to smallest. If k is negative, extract the -k smallest elements instead, and return them sorted from smallest to largest. This performs best when ``k`` is much smaller than the chunk size. All results will be returned in a single chunk along the given axis. Parameters ---------- x: Array Data being sorted k: int axis: int, optional split_every: int >=2, optional See :func:`reduce`. This parameter becomes very important when k is on the same order of magnitude of the chunk size or more, as it prevents getting the whole or a significant portion of the input array in memory all at once, with a negative impact on network transfer too when running on distributed. Returns ------- Selection of x with size abs(k) along the given axis. Examples -------- >>> import dask.array as da >>> x = np.array([5, 1, 3, 6]) >>> d = da.from_array(x, chunks=2) >>> d.topk(2).compute() array([6, 5]) >>> d.topk(-2).compute() array([1, 3]) rT)rrqrorMrNr;rprW) r$rcrrtopktopk_aggregateryr;abs)r@rrMrp chunk_combineros r6rrPs}H qv & &D EJ!,,,M,222I   gFF    r8ct|jt|j|jft j}|tfdt|jD}| tj |t}ttj|}ttj|}t!t"rd}nt%}|jt jd|j|z dzz} t-||||dt j|dt/|| S) aExtract the indices of the k largest elements from a on the given axis, and return them sorted from largest to smallest. If k is negative, extract the indices of the -k smallest elements instead, and return them sorted from smallest to largest. This performs best when ``k`` is much smaller than the chunk size. All results will be returned in a single chunk along the given axis. Parameters ---------- x: Array Data being sorted k: int axis: int, optional split_every: int >=2, optional See :func:`topk`. The performance considerations for topk also apply here. Returns ------- Selection of np.intp indices of x with size abs(k) along the given axis. Examples -------- >>> import dask.array as da >>> x = np.array([5, 1, 3, 6]) >>> d = da.from_array(x, chunks=2) >>> d.argtopk(2).compute() array([3, 0]) >>> d.argtopk(-2).compute() array([1, 2]) )rjr;c3ZK|]%}|krtdn tjV&dSrE)rirnewaxis)rTrUrMs r6rXzargtopk..s7TT199E$KKK"*TTTTTTr8r:rrKrTF) rrqrorMrNr;rpr]rWrs)r$rcrrhrjrintprarbrrargtopk_preprocessrrargtopkargtopk_aggregaterdr rRrYrrryr) r@rrMrpidx a_plus_idxrronaxisrss ` r6rrsKB qv & &D '8 H H HC eTTTTeAFmmTTTTT UCe66JJJ EMQ///M/1555I$D  7>>"' " " * *416E>A3E+F G GD  gFF     r8cRt||||d|S)N)r{axis1axis2r(r:)rr)r@r{rrr;s r6tracers, AfE ? ? ? C CBe C T TTr8c tdttsffdDtjfdDr3fdt jDtj ||snd|r fdtj Dnd}t||}|S)z This works by automatically chunking the reduced axes to a single chunk if necessary and then calling ``numpy.median`` function across the remaining dimensions NzcThe da.median function only works along an axis. The full algorithm is difficult to do in parallelc4g|]}|dkr |jzn|SrrcrTrr@s r6rzmedian..+ 9 9 9b266BKKr 9 9 9r8c3:K|]}j|dkVdSrrrs r6rXzmedian...77BAKOa'777777r8c"i|] }||vrdnd Sr(autor`rTrrMs r6rzmedian..'PPPBrt22PPPr8c&g|] \}}|vrdn|Srr`rTrrVrMs r6rzmedian..)HHH52qR4ZZQHHHr8rMrN drop_axisrj)NotImplementedErrorrdrr>rrrbrcrrmedianrirjrr@rMrNrrrxs`` r6rrs#  |! @   dH % %w 9 9 9 9D 9 9 9D|7777$77777R IIPPPP%--PPP Q Q \\ &0$$D HHHHIah4G4GHHHH FV $ $F Mr8c tdttsffdDtjfdDr3fdt jDtj ||snd|r fdtj Dnd}t||}|S)z This works by automatically chunking the reduced axes to a single chunk and then calling ``numpy.nanmedian`` function across the remaining dimensions NzzThe da.nanmedian function only works along an axis or a subset of axes. The full algorithm is difficult to do in parallelc4g|]}|dkr |jzn|Srrrs r6rznanmedian.. rr8c3:K|]}j|dkVdSrrrs r6rXznanmedian.. rr8c"i|] }||vrdnd Srr`r s r6rznanmedian..r r8c&g|] \}}|vrdn|Srr`r s r6rznanmedian..r r8r)rrdrr>rrrbrcrr nanmedianrirjrrs`` r6rrs#  |! @   dH % %w 9 9 9 9D 9 9 9D|7777$77777R IIPPPP%--PPP Q Q \\ &0$$D HHHHIah4G4GHHHH FV $ $F Mr8rE) NFNNNNNTrKNN)NNNTN)FNNN)NNFNN)NFNN)NN)rNF)NNFrNN)F)NF)NNN)NNNrN)NNNr)r(N)rrrKN)NFN) __future__rr>rrrrcollections.abcr functoolsr itertoolsrrnumbersrr numpyrtlzr r r r rrdaskr dask.arrayrdask.array.blockwiserdask.array.corerrrrrrrdask.array.creationrrdask.array.dispatchrrrdask.array.utilsrr r!r"r#r$dask.array.wrapr%r& dask.baser'dask.blockwiser(dask.highlevelgraphr) dask.utilsr*r+r,r-r.r/rCrHrJryrnrrrraminrr?amaxrrrrrrrrrrrrrrrr#r0r6rBrMrQrOrUrZr]rarcrorrrrrrrrrrrrrrrrrrrrrrrrr`r8r6r.s """""" $$$$$$%%%%%%%%$$$$$$$$DDDDDDDDDDDDDDDD******10000000LLLLLLLLLL('''''''%%%%%%......    %%%(((       {#{#{#{#H  7777vOSL9L9L9L9^b     b" BFBG b    7777 BFBG b    7777b    b    b"b"b|2b 2b&    b&     9Ee $ $ $ $     $$$$0 - - - -b*b(EIU$u , , , ,       %,%,%,%,T   -/-/-/-/bTXDDDDNb(bLP*      b"bLP&BB     CCCCNRJ#J#J#J#ZQQQRRRb    b    b    b    7776+++&u#u#u#u#z    g#g#g#g#Tb2b28888vEEEEPbUUUUbDbr8