Cd ^ddlmZddlZddlZddlmZmZmZddl m Z m Z m Z m Z mZddlmZddlmZddlmZdZd eZd ed Zd eZd eZd ededZdZdZddddddddddZGddZddZ dS)) annotationsN)concatmergeunique)Arrayapply_infer_dtypeasarray blockwisegetitemmeta_from_array)flatten)HighLevelGraphz\w+z(?:{0:}(?:,{0:})*,?)?z\(z\)z{0:}(?:,{0:})*^->$ctjdd|}tjt|st d||d\}}dtjt|D}dtjt|D}t|dkr|dd kr|d n|}||fS) a Parse string signatures for a generalized universal function. Arguments --------- signature : string Generalized universal function signature, e.g., ``(m,n),(n,p)->(m,p)`` for ``np.matmul``. Returns ------- Tuple of input and output core dimensions parsed from the signature, each of the form List[Tuple[str, ...]], except for one output. For one output core dimension is not a list, but of the form Tuple[str, ...] \s+zNot a valid gufunc signature: rc\g|])}ttjt|*Stuplerefindall_DIMENSION_NAME.0args 1lib/python3.11/site-packages/dask/array/gufunc.py z+_parse_gufunc_signature..0s9   47bj#..//   c\g|])}ttjt|*Srrrs r r!z+_parse_gufunc_signature..3s<     bj#..//   r",r) rsubmatch _SIGNATURE ValueErrorsplitr _ARGUMENTlen) signaturein_txtout_txtinsoutss r _parse_gufunc_signaturer3s vr9--I 8J * *GE)EEFFFood++OFG  ;=:iQW;X;X   C  :i11   DD QWR[C-?-?477tD 9r"cnt|}t|tsdnt|}|td|r$t|tstd|dkr|n|g}tt t|}td|D}|r.|dkrtdt||dgz}||z} jtt std|rD|d} t| dkrtd |D]} | | krtd |fd | D}n1d | D}n$t|tstd d|D}|dkr)|t|kr||zt|ks|dkr%||zt|krtd||d} | r| n d|D} |d|} t t| |D]J\}\}}t|t|kr"td|K|sit t| |D]J\}\}}t|t|kr"td|Kn:|r8|d}|D]}||krtd| dfd|D} | | fS)a Validates logic of `axes`/`axis`/`keepdims` arguments and normalize them. Refer to [1]_ for details Arguments --------- axes: List of tuples axis: int keepdims: bool input_coredimss: List of Tuple of dims output_coredimss: List of Tuple of dims Returns ------- input_axes: List of tuple of int output_axes: List of tuple of int References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html#optional-keyword-arguments r$Nz>Only one of `axis` or `axes` keyword arguments should be givenz`axes` has to be of type listc8g|]}t|dkdS)rTr-rxs r r!z,_validate_normalize_axes..]s##Q#Q#QQc!ffqjjDjjjr"rz.`keepdims` can only be used for scalar outputsz*`axis` argument has to be an integer valuez9`axis` can be used only, if one core dimension is presentz3To use `axis`, all core dimensions have to be equalc6g|]}|rfn tSr)r)rcdaxiss r r!z,_validate_normalize_axes..ws(CCC2r.TGGuwwCCCr"c dg|]-}ttt| d.Srrranger-)ricds r r!z,_validate_normalize_axes..ys2DDD3E%S 1--..DDDr"z `axes` argument has to be a listcBg|]}t|tr|fn|Sr) isinstanceintras r r!z,_validate_normalize_axes..|s- < < )rocds r r!z,_validate_normalize_axes..s2 E E ESeE3s88)Q''(( E E Er"zvThe number of `axes` entries for argument #{} is not equal the number of respective input core dimensions in signaturezwThe number of `axes` entries for argument #{} is not equal the number of respective output core dimensions in signaturez7To use `keepdims`, all core dimensions have to be equalcg|]}Srr)r_iax0s r r!z,_validate_normalize_axes..s:::A4:::r") r-rBlistr*filterrC enumeratezipformat)axesr;keepdimsinput_coredimssoutput_coredimssninnoutfiltered_core_dimsnr_outputs_with_coredims core_dimscd0r: output_axes input_axesidxiaxr@oaxrGicd0rJs ` @r _validate_normalize_axesr`;s1, o  C/66 Q11C@P?@ABCDEFt|tstdtjdd|}t |\} } t| t sdnt| D| |td| |/|rtj ||}n|}t|| | ddD}D7t|tt frt|d kr|d }| r| d ndEDtE| } ntEfd |D} t| } t| t rt| } DAt| tr+t| d kr | d } nqtd nat| tstdDdt| Dkr&tdDdDdt| d|r@t| tr d| Dn| jg}tj |||}|i}t|||| | \}}d| D} t| t| kr.tdt| t| fzg}t!| |D]\}A|jFtFfdADAtAfdt%tF d zd DAz}||}|||} d| D}d| D}dt!|| D}|rt+|ndCdt!||| D}t-|??|Cfd|D}dt!|| D}|rt+|tn t}i}i}t!|||D]~\}F} t!|F| D]f\}!}"}#||!g}$|$|"|$||!<||!g}%|%|#|%||!<g|D]\}!}&t5|&d hzd t+|&hkrtd |!d!|s||!} |!?vr:| d d ?|!kr"td"|!t t9d#t!|&| D}'t|'d krtd |!d$t t;t!| |}(t=||g|(Rd%| d&| })|)j}*D0t|*t tfrJd'| d(|*|*f}*n]t|*t tfsJd)| d(|*t|*DksJd*Dd+t|*d,|)j}+|)j },t tC|)"}-|-d d #d-\}.}/D| g} g}0tIt!| ||*D]\@\}1}2} t?fd.|1D}3t|1d/z>|+|3z}4|,|3z}5d0|.@|/fzB>@BDfd1|-D}6tKj&B|6|)g2}7t| t|4} tO|7B|5|4| 3}8|rKt|8jtQdfzt|2tj)fzz}9|8|9}8dgt|8jz}:t!t%t|2 d |2D] \};}<|;|:|<< d }=t%t|:D]};|:|; |=|:|;<|=d z }=|8|:}8|0|8Drg|0Rn|0d S)4aC Apply a generalized ufunc or similar python function to arrays. ``signature`` determines if the function consumes or produces core dimensions. The remaining dimensions in given input arrays (``*args``) are considered loop dimensions and are required to broadcast naturally against each other. In other terms, this function is like ``np.vectorize``, but for the blocks of dask arrays. If the function itself shall also be vectorized use ``vectorize=True`` for convenience. Parameters ---------- func : callable Function to call like ``func(*args, **kwargs)`` on input arrays (``*args``) that returns an array or tuple of arrays. If multiple arguments with non-matching dimensions are supplied, this function is expected to vectorize (broadcast) over axes of positional arguments in the style of NumPy universal functions [1]_ (if this is not the case, set ``vectorize=True``). If this function returns multiple outputs, ``output_core_dims`` has to be set as well. signature: string Specifies what core dimensions are consumed and produced by ``func``. According to the specification of numpy.gufunc signature [2]_ *args : numeric Input arrays or scalars to the callable function. axes: List of tuples, optional, keyword only A list of tuples with indices of axes a generalized ufunc should operate on. For instance, for a signature of ``"(i,j),(j,k)->(i,k)"`` appropriate for matrix multiplication, the base elements are two-dimensional matrices and these are taken to be stored in the two last axes of each argument. The corresponding axes keyword would be ``[(-2, -1), (-2, -1), (-2, -1)]``. For simplicity, for generalized ufuncs that operate on 1-dimensional arrays (vectors), a single integer is accepted instead of a single-element tuple, and for generalized ufuncs for which all outputs are scalars, the output tuples can be omitted. axis: int, optional, keyword only A single axis over which a generalized ufunc should operate. This is a short-cut for ufuncs that operate over a single, shared core dimension, equivalent to passing in axes with entries of (axis,) for each single-core-dimension argument and ``()`` for all others. For instance, for a signature ``"(i),(i)->()"``, it is equivalent to passing in ``axes=[(axis,), (axis,), ()]``. keepdims: bool, optional, keyword only If this is set to True, axes which are reduced over will be left in the result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core dimensions and with outputs that have no core dimensions , i.e., with signatures like ``"(i),(i)->()"`` or ``"(m,m)->()"``. If used, the location of the dimensions in the output can be controlled with axes and axis. output_dtypes : Optional, dtype or list of dtypes, keyword only Valid numpy dtype specification or list thereof. If not given, a call of ``func`` with a small set of data is performed in order to try to automatically determine the output dtypes. output_sizes : dict, optional, keyword only Optional mapping from dimension names to sizes for outputs. Only used if new core dimensions (not found on inputs) appear on outputs. vectorize: bool, keyword only If set to ``True``, ``np.vectorize`` is applied to ``func`` for convenience. Defaults to ``False``. allow_rechunk: Optional, bool, keyword only Allows rechunking, otherwise chunk sizes need to match and core dimensions are to consist only of one chunk. Warning: enabling this can increase memory usage significantly. Defaults to ``False``. meta: Optional, tuple, keyword only tuple of empty ndarrays describing the shape and dtype of the output of the gufunc. Defaults to ``None``. **kwargs : dict Extra keyword arguments to pass to `func` Returns ------- Single dask.array.Array or tuple of dask.array.Array Examples -------- >>> import dask.array as da >>> import numpy as np >>> def stats(x): ... return np.mean(x, axis=-1), np.std(x, axis=-1) >>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30)) >>> mean, std = da.apply_gufunc(stats, "(i)->(),()", a) >>> mean.compute().shape (10, 20) >>> def outer_product(x, y): ... return np.einsum("i,j->ij", x, y) >>> a = da.random.normal(size=( 20,30), chunks=(10, 30)) >>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40)) >>> c = da.apply_gufunc(outer_product, "(i),(j)->(i,j)", a, b, vectorize=True) >>> c.compute().shape (10, 20, 30, 40) References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html .. [2] https://docs.scipy.org/doc/numpy/reference/c-api/generalized-ufuncs.html z$`signature` has to be of type stringrrNzMOnly one of `meta` and `output_dtypes` should be given (`meta` is preferred).r. apply_gufuncrar$rdtypec3:K|]}t|VdS)riNr )rodtsamples r zapply_gufunc..Ls0UUs;;;UUUUUUr"zhFor a function with one output, must give a single item for `output_dtypes`/`meta`, not a tuple or list.zFor a function with zR outputs, must give a tuple or list for `output_dtypes`/`meta`, not a single item.z' outputs, must give a tuple or list of z' items for `output_dtypes`/`meta`, not .cg|] }|j Srrir7s r r!z apply_gufunc..ks(((a!'(((r")r.otypesc,g|]}t|Sr)r rDs r r!z apply_gufunc..ys % % %1GAJJ % % %r"zDAccording to `signature`, `func` requires %d arguments, but %s givenc3LK|]}|dkr|n|tz VdS)rNr6)rrEshapes r rnzapply_gufunc..s9@@qQAAAE N@@@@@@r"c3$K|] }|v|V dSNr)rir]s r rnzapply_gufunc..s'JJ1Qc\\Q\\\\JJr"cg|] }|j Sr)rtrDs r r!z apply_gufunc..s***AG***r"cg|] }|j Sr)chunksrDs r r!z apply_gufunc..s,,,!QX,,,r"cRg|]$\}}t|t|z %Srr6)rsr:s r r!z apply_gufunc..s-UUUBCFFSWW$UUUr"c `g|]+\}}}tt|||d,Srv)dictrN)rr|nr@s r r!z apply_gufunc..sF Aq# Sae__r"c dg|],}tdt|z D-S)c3 K|] }d|zV dS)z __loopdim%d__Nr)rds r rnz*apply_gufunc...s' Q Qao! Q Q Q Q Q Qr")rr?)rr max_loopdimss r r!z apply_gufunc..sO   Q Q51A<+P+P Q Q QQQr"cg|] \}}||z Srr)rlcs r r!z apply_gufunc..s LLLTQ1q5LLLr")keyz Dimension `'z#'` with different lengths in arrayszCore dimension `'{}'` consists of multiple chunks. To fix, rechunk into a single chunk along this dimension or set `allow_rechunk=True`, but beware that this may increase memory usage significantly.c3,K|]\}}|dk |VdSr$Nr)rr|rs r rnzapply_gufunc..s*EETQq1uuquuuuEEr"z#'` with different chunksize presentT) concatenaterezEmeta changed from single output to multiple output during blockwise: z -> zEmeta changed from multiple output to single output during blockwise: zNumber of outputs changed from z to z during blockwise-c3(K|] }|V dSrvr)rr core_shapess r rnzapply_gufunc..s'!>!>Q+a.!>!>!>!>!>!>r"r=z%s_%d-%scLi|] }f|ddzzr t|fn|!Sr)r )rrcore_chunkindsrw leaf_namerUs r z apply_gufunc..sY   L!""g  48Aa00c   r") dependencies)rzrtre)*rBstr TypeErrorrr'r3rKr-r*nprcrrr rjr`rNrtr? transposeappendmaxrupdategetitemssetrOrrr _metarzr __dask_keys__r+rMrfrom_collectionsrslicenewaxis)Gfuncr.rPr;rQrarbrcrdreargskwargsrRrStempfuncrqr[rZtransposed_argsrtidctransposed_arg input_shapes input_chunkss num_loopdimscore_input_shapesloop_input_dimss input_dimssloop_output_dims dimsizess chunksizessdims chunksizesdimsize chunksizedimsizes chunksizes_sizesrelevant_chunksizesargindstmpmetasloop_output_shapeloop_output_chunkskeysnametoken leaf_arrsrGr^core_output_shape output_shape output_chunksleaf_dskgraphleaf_arrslicestidcsiioajrrrwr]rrrUrmrtsG @@@@@@@@@r rhrhs l i % %@>???vr9--I(? (J(J%O%""2D99 T44sCS?T?TD M5 [    |   < BBB-$M L=5$-88 M""a'')!,M *ad <"6???DDUUUU}UUUUUD 4 D$T{{ | dE " " 4yyA~~Aw +  $&& %t%%%  t99  3|$$$$L\<II*+K|$$$MLS)9?%K%KLLLK9IVs+5555uwwIK#&{L-#P#P++eZ$'eZ$@$@ + + Cy }}S"--H OOD ! ! !%IcN%//#r22K   y ) ) )*K    + oo'' U u:: 3u:: . .TCTTTUU U $S)J{""Aq)9K!>!>!>#!>!>!>>>SD(+<< *->> $5!11             / 8SVRWXXXtS%6%677 9],T     (((E$KK>9CHH }(i,k)"`` appropriate for matrix multiplication, the base elements are two-dimensional matrices and these are taken to be stored in the two last axes of each argument. The corresponding axes keyword would be ``[(-2, -1), (-2, -1), (-2, -1)]``. For simplicity, for generalized ufuncs that operate on 1-dimensional arrays (vectors), a single integer is accepted instead of a single-element tuple, and for generalized ufuncs for which all outputs are scalars, the output tuples can be omitted. axis: int, optional, keyword only A single axis over which a generalized ufunc should operate. This is a short-cut for ufuncs that operate over a single, shared core dimension, equivalent to passing in axes with entries of (axis,) for each single-core-dimension argument and ``()`` for all others. For instance, for a signature ``"(i),(i)->()"``, it is equivalent to passing in ``axes=[(axis,), (axis,), ()]``. keepdims: bool, optional, keyword only If this is set to True, axes which are reduced over will be left in the result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core dimensions and with outputs that have no core dimensions , i.e., with signatures like ``"(i),(i)->()"`` or ``"(m,m)->()"``. If used, the location of the dimensions in the output can be controlled with axes and axis. output_dtypes : Optional, dtype or list of dtypes, keyword only Valid numpy dtype specification or list thereof. If not given, a call of ``func`` with a small set of data is performed in order to try to automatically determine the output dtypes. output_sizes : dict, optional, keyword only Optional mapping from dimension names to sizes for outputs. Only used if new core dimensions (not found on inputs) appear on outputs. vectorize: bool, keyword only If set to ``True``, ``np.vectorize`` is applied to ``func`` for convenience. Defaults to ``False``. allow_rechunk: Optional, bool, keyword only Allows rechunking, otherwise chunk sizes need to match and core dimensions are to consist only of one chunk. Warning: enabling this can increase memory usage significantly. Defaults to ``False``. meta: Optional, tuple, keyword only tuple of empty ndarrays describing the shape and dtype of the output of the gufunc. Defaults to ``None``. Returns ------- Wrapped function Examples -------- >>> import dask.array as da >>> import numpy as np >>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30)) >>> def stats(x): ... return np.mean(x, axis=-1), np.std(x, axis=-1) >>> gustats = da.gufunc(stats, signature="(i)->(),()", output_dtypes=(float, float)) >>> mean, std = gustats(a) >>> mean.compute().shape (10, 20) >>> a = da.random.normal(size=( 20,30), chunks=(10, 30)) >>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40)) >>> def outer_product(x, y): ... return np.einsum("i,j->ij", x, y) >>> guouter_product = da.gufunc(outer_product, signature="(i),(j)->(i,j)", output_dtypes=float, vectorize=True) >>> c = guouter_product(a, b) >>> c.compute().shape (10, 20, 30, 40) >>> a = da.ones((1, 5, 10), chunks=(-1, -1, -1)) >>> def stats(x): ... return np.atleast_1d(x.mean()), np.atleast_1d(x.max()) >>> meta = (np.array((), dtype=np.float64), np.array((), dtype=np.float64)) >>> gustats = da.gufunc(stats, signature="(i,j)->(),()", meta=meta) >>> result = gustats(a) >>> result[0].compute().shape (1,) >>> result[1].compute().shape (1,) References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html .. [2] https://docs.scipy.org/doc/numpy/reference/c-api/generalized-ufuncs.html NF) r.rcrPr;rQrbrardrec ||_||_||_||_||_||_||_||_| |_| |_ d t|j|j|_ dS)Na Bound ``dask.array.gufunc`` func: ``{func}`` signature: ``'{signature}'`` Parameters ---------- *args : numpy/dask arrays or scalars Arrays to which to apply to ``func``. Core dimensions as specified in ``signature`` need to come last. **kwargs : dict Extra keyword arguments to pass to ``func`` Returns ------- Single dask.array.Array or tuple of dask.array.Array )rr.) pyfuncr.rcrPr;rQrbrardrerOr__doc__) selfrr.rcrPr;rQrbrardres r __init__zgufunc.__init__vs ""    (**  FT[!!T^   ! r")rdc t|j|jg|R|j|j|j|j|j|j|j p||j d|S)N)rcrPr;rQrbrardre) rhrr.rcrPr;rQrbrardre)rrdrrs r __call__zgufunc.__call__ss K N     n]*,,=       r")__name__ __module__ __qualname__rrrrr"r rrsvddT   + + + + + Z-2       r"rc hd}|z rtdfd}d|_|S)a  Decorator for ``dask.array.gufunc``. Parameters ---------- signature : String Specifies what core dimensions are consumed and produced by ``func``. According to the specification of numpy.gufunc signature [2]_ axes: List of tuples, optional, keyword only A list of tuples with indices of axes a generalized ufunc should operate on. For instance, for a signature of ``"(i,j),(j,k)->(i,k)"`` appropriate for matrix multiplication, the base elements are two-dimensional matrices and these are taken to be stored in the two last axes of each argument. The corresponding axes keyword would be ``[(-2, -1), (-2, -1), (-2, -1)]``. For simplicity, for generalized ufuncs that operate on 1-dimensional arrays (vectors), a single integer is accepted instead of a single-element tuple, and for generalized ufuncs for which all outputs are scalars, the output tuples can be omitted. axis: int, optional, keyword only A single axis over which a generalized ufunc should operate. This is a short-cut for ufuncs that operate over a single, shared core dimension, equivalent to passing in axes with entries of (axis,) for each single-core-dimension argument and ``()`` for all others. For instance, for a signature ``"(i),(i)->()"``, it is equivalent to passing in ``axes=[(axis,), (axis,), ()]``. keepdims: bool, optional, keyword only If this is set to True, axes which are reduced over will be left in the result as a dimension with size one, so that the result will broadcast correctly against the inputs. This option can only be used for generalized ufuncs that operate on inputs that all have the same number of core dimensions and with outputs that have no core dimensions , i.e., with signatures like ``"(i),(i)->()"`` or ``"(m,m)->()"``. If used, the location of the dimensions in the output can be controlled with axes and axis. output_dtypes : Optional, dtype or list of dtypes, keyword only Valid numpy dtype specification or list thereof. If not given, a call of ``func`` with a small set of data is performed in order to try to automatically determine the output dtypes. output_sizes : dict, optional, keyword only Optional mapping from dimension names to sizes for outputs. Only used if new core dimensions (not found on inputs) appear on outputs. vectorize: bool, keyword only If set to ``True``, ``np.vectorize`` is applied to ``func`` for convenience. Defaults to ``False``. allow_rechunk: Optional, bool, keyword only Allows rechunking, otherwise chunk sizes need to match and core dimensions are to consist only of one chunk. Warning: enabling this can increase memory usage significantly. Defaults to ``False``. meta: Optional, tuple, keyword only tuple of empty ndarrays describing the shape and dtype of the output of the gufunc. Defaults to ``None``. Returns ------- Decorator for `pyfunc` that itself returns a `gufunc`. Examples -------- >>> import dask.array as da >>> import numpy as np >>> a = da.random.normal(size=(10,20,30), chunks=(5, 10, 30)) >>> @da.as_gufunc("(i)->(),()", output_dtypes=(float, float)) ... def stats(x): ... return np.mean(x, axis=-1), np.std(x, axis=-1) >>> mean, std = stats(a) >>> mean.compute().shape (10, 20) >>> a = da.random.normal(size=( 20,30), chunks=(10, 30)) >>> b = da.random.normal(size=(10, 1,40), chunks=(5, 1, 40)) >>> @da.as_gufunc("(i),(j)->(i,j)", output_dtypes=float, vectorize=True) ... def outer_product(x, y): ... return np.einsum("i,j->ij", x, y) >>> c = outer_product(a, b) >>> c.compute().shape (10, 20, 30, 40) References ---------- .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html .. [2] https://docs.scipy.org/doc/numpy/reference/c-api/generalized-ufuncs.html >rPr;rerQrcrbrdraz(Unsupported keyword argument(s) providedc"t|fdiS)Nr.)r)rrr.s r _as_gufunczas_gufunc.._as_gufuncsf<< rs"""""" %%%%%%%%%%QQQQQQQQQQQQQQ,,,,,,...... .55oFF *& * * * *11)<<$++ :! 9 9%6 9 9 9 >r#r#r#r   \3\3\3\3\3~ b b b b b b b b Jrrrrrrr"