d`"dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddlZddlmZmZmZmZmZddlmZdd lmZdd lmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7ddl8m9Z9dZ:dZ;dZdZ?e"ddddZ@e?ZAe@ZBdZCdZDdZEGdd ee(ejFZFGd!d"ee(ZGGd#d$ee(ZHGd%d&ee(ZIdS)'z Here is defined the Index class.N)Path) perf_counter) process_time)calc_chunksize calcoptlevelsget_reduction_level nextafterinftype)indexesextension)NotLoggedMixin)UIntAtomAtom)EArray)CArray)Filters) CacheArray LastRowArray IndexArray)Group) join_path)PerformanceWarning)is_idxidx2longlazyattr) nan_aware_gt nan_aware_ge nan_aware_lt nan_aware_le bisect_left bisect_right) ObjectCachez2.1F) show_stats mergesortTzlib) complevelcomplibshuffle fletcher32) int8int16int32int64uint8uint16uint32uint64float32float64lc"|d}t|D]\}}|drnd|d|dz|ddz}d||dzd}||fS)N/_i_r)split enumerate startswithjoin) indexpathnamenamesiname tablepathname colpathnames ,lib/python3.11/site-packages/tables/index.py_table_column_pathname_of_indexrCOs    $ $EU##4 ??5 ! !  E HHU2A2Y''#-QRR8M((5Q=))K ; ''ceZdZdZdZedZedZedZej dZedZ edZ ed Z ed Z ed Zed Zed ZedZedZedZedZedZedZedZedZ dDfd ZfdZdZdZdEdZdEd ZdEd!Z d"Z!dFd#Z"d$Z#d%Z$d&Z%d'Z&dGd)Z'dHd*Z(dHd+Z)dHd,Z*dHd-Z+d.Z,d/Z-d0Z.dGd1Z/dHd2Z0d3Z1d4Z2d5Z3dId6Z4dId7Z5d8Z6d9Z7d:Z8d;Z9d<Z:d=Z;d>ZdEfdA Z?dBZ@dCZAxZBS)JIndexaRepresents the index of a column in a table. This class is used to keep the indexing information for columns in a Table dataset (see :ref:`TableClassDescr`). It is actually a descendant of the Group class (see :ref:`GroupClassDescr`), with some added functionality. An Index is always associated with one and only one column in the table. .. note:: This class is mainly intended for internal use, but some of its documented attributes and methods may be interesting for the programmer. Parameters ---------- parentnode The parent :class:`Group` object. .. versionchanged:: 3.0 Renamed from *parentNode* to *parentnode*. name : str The name of this node in its parent group. atom : Atom An Atom object representing the shape and type of the atomic objects to be saved. Only scalar atoms are supported. title Sets a TITLE attribute of the Index entity. kind The desired kind for this index. The 'full' kind specifies a complete track of the row position (64-bit), while the 'medium', 'light' or 'ultralight' kinds only specify in which chunk the row is (using 32-bit, 16-bit and 8-bit respectively). optlevel The desired optimization level for this index. filters : Filters An instance of the Filters class that provides information about the desired I/O filters to be applied during the life of this object. tmp_dir The directory for the temporary files. expectedrows Represents an user estimate about the number of row slices that will be added to the growable dimension in the IndexArray object. byteorder The byteorder of the index datasets *on-disk*. blocksizes The four main sizes of the compound blocks in index datasets (a low level parameter). INDEXc&ddddd|jS)zThe kind of this index. ultralightlightmediumfullr)indsizeselfs rBkindz Index.kinds' G(((, 6 6rDc|jS)zSFilter properties for this index - see Filters in :ref:`FiltersClassDescr`.) _v_filtersrRs rBfiltersz Index.filterss rDc.t|jddS)zWhether the index is dirty or not. Dirty indexes are out of sync with column data, so they exist but they are not usable. DIRTYF)getattr_v_attrsrRs rBdirtyz Index.dirtyst}gu555rDc|jt|}}||j_|jj}|s|r||r|s|dSdSdSN)r\boolr[rYtable_condition_cachenailunnail)rSr\wasdirtyisdirtyconditioncaches rBr\z Index.dirtys JU '# 4 "G "    ! ! !  $G $  ! ! # # # # # $ $ $ $rDct|j\}}|j|}|j|}|S)zQThe Column (see :ref:`ColumnClassDescr`) instance for the indexed column.)rC _v_pathname_v_file _get_nodecols_g_col)rS tablepath columnpathr`columns rBroz Index.columnsN !@  !! : &&y11"":.. rDcht|j\}}|j|}|S)z.Accessor for the `Table` object of this index.)rCrhrirj)rSrmrnr`s rBr`z Index.tables9!@  !! : &&y11 rDc |j|jzS)z%The number of blocks in a superblock.)superblocksize blocksizerRs rBnblockssuperblockzIndex.nblockssuperblocks"dn44rDc |j|jzS)z The number of slices in a block.)rs slicesizerRs rB nslicesblockzIndex.nslicesblock~//rDc |j|jzS)z The number of chunks in a slice.)rv chunksizerRs rB nchunkslicezIndex.nchunkslicerxrDc^|j|jz }||jz}||jzdkr|dz }|S)z)The total number of superblocks in index.rr) nelements nelementsILRrrrsrSr}nblockss rB nsuperblockszIndex.nsuperblockss@NT%66 t22 t~ % ) ) qLGrDc^|j|jz }||jz}||jzdkr|dz }|S)z$The total number of blocks in index.rr)r}r~rsrs rBrz Index.nblockss?NT%66 t~- t~ % ) ) qLGrDc |j|jzS)z'The number of complete slices in index.)r}rvrRs rBnslicesz Index.nslicesrxrDc |j|jzS)z'The number of complete chunks in index.)r}rzrRs rBnchunksz Index.nchunksrxrDc|j|jfS)z1The shape of this index (in slices and elements).)nrowsrvrRs rBshapez Index.shapes DN++rDcX|jdko|jdko|jj|jkS)z8Whether a temporary file for indexes is required or not.rr)rQoptlevelr`rrvrRs rB temp_requiredzIndex.temp_requireds6 q 2 !2  4>1 3rDc.|jdko |jdkS)z@Whether we should try to build a completely sorted index or not.rP )rQrrRs rBwant_complete_sortzIndex.want_complete_sorts|q 7T]a%77rDc|jdkrdS|jdkrdSd|jvr |jjS||dd|jdkS)zWhether the index is completely sorted or not. .. versionchanged:: 3.0 The *is_CSI* property has been renamed into *is_csi*. rFrPis_csiN)r}rQr[rcompute_overlaps noverlapsrRs rBrz Index.is_csisg >Q  5 W]1%55557H555>W%:&%<<///1I///E!!!$U_!5!5"4=#6#@AA "4>#7#ABB !"#~b1 + DJ!Z$.8<GDN ,D  ,D a a 4>9J%)j8I1}} NI M\I557DM F tz**, ? ",!4=$,TKKKDO,0? ) dn   2 - 1 1 1, L$-II^t~5 ^t~5 (*y1D'E'E $"$)DN";"; "$)DN";"; "$)DN";"; !% "&. D(D/#T^55 4Hdl$C$C$C-w H H H 44$6!^ - - - - tYdOWe 5 5 5 5&>j844!_)=$gt|'4> C C C C tYdNGe 5 5 5 5tYdL'e 5 5 5 5tYdOWe 5 5 5 5a/13j$ A '**..::: ! {!)4*;+/> ;;; $% $% ! hTZrx@@@ KxdjAAA J             rDctrt}trtd||}|j}|j}|j}trtd||dkr+tjdt|d||zz} n|dkr%tjdt|d} ntj t|d |dzz} |j } t| ||| zz| zz } d| d| <t| || D]} | | zd z | z| | | | z<|d kr||jz|z| z} | | z } |dkra|dkr[t||ksJ||j|d |||j| d |trtd |t%j|| |d}|d krdtrtd||d d |}trtd||}trtd||dkr d|j_trtd|||| fS)z9Compute an initial indices arrays for data to be indexed.zEntering initial_appendzBefore creating idxrPrr1rrOr0zuint%drrNNzBefore keysortrzBefore reductionzAfter reductionzAfter arr <-- reducFzExiting initial_append)profileclockr#poprQrvr~rarangelenrrrrangerw read_slice_lrrrr keysortcopyr[r)rSxarrnrowrtrefarrrQrvr~idxroffsetr>offset2larrreducs rBinitial_appendzIndex.initial_appends  77D  8 0$ 7 7 7hhjj,N (  4 ,d 3 3 3 a<<)As3xxx8884);KKCC \\&)As3xxx888CC(3s88X1%=>>ClGTY-@%AW$LMNNFC&M69g66 B B&''kAo'%AAa'kM""!|| $"33y@GKw qLL\A--s88l****   t}c-<-.@ A A A   t~s=L=/A B B B  / ' . . . c***2w q== 5-t444 ) $))++E 4,d333C 80$777 !88#(DM  7 / 6 6 6S#~rDctrt}trtd||d}||z }||jz}|d}trtd||S)z+Perform final operations in 32-bit indices.zEntering final_idx32r1r0zExiting final_idx32)rrr#astyper)rSrrrs rB final_idx32zIndex.final_idx32hs  77D  5 -t 4 4 4jj"" v   jj""  4 ,d 3 3 3 rDFcztrt}trtd||s|jr |j}d}n |}|j}|j}|j}|j}|j } |j } |j } |j } |j } |j}|j}|j}||||\}}}||d|j|j|z}|j}||d|gg| ||d|g| |dd|| ||dz d|||dzd|}| || ||dzgtrtd|~~trtd||jdkr||||jz}||d|jtrtd |~|dz|_|j|jz|_d|_d|_|j|j_|j|j_d |_trtd |dSdS) z&Append the array to the index objects.zEntering appendrrNrNzBefore deleting arr & smedianzAfter deleting arr & smedianrOzBefore deleting idxTzExiting append)rrr#rtmprrrrrrrrrrrrrappendreshapesizerzr{rQrrvr}rr~rr)rSrupdaterwhererrrrrrrrrrrrrrrcsncssmedians rBrz Index.append|s  77D  0 ($ / / / '$, 'HEIIEI----->O  ,,T5)DDc3 ckk!SX../// ^y ( A~&''' s26r6{m$$$s15b5z"""s26:2:'''bAgkrk"wq)*+++  > 6 = = =   = 5t < < < <1  ""3(>??Cs{{1ch//000  4 ,d 3 3 3 QY dn4$(#4 $($5 !  / ' . . . . . / /rDc&trt}trtd||j}|s|jr |j}d}n |}|j}|j}|j}| |||\} } } t| } t| } |j |z}tj | dd|| gf|_| |j_| |j_t|j}|j|| | |z<| |d| <~ |jdkr|| ||jz} | |dt| <~ |dz|_||jz| z|_| |_| |_d|_trtd|dSdS)z/Append the array to the last row index objects.zEntering appendLRrNrOTzExiting appendLR)rrr#rrrrrrrrrzr concatenaterrr}rQrrvrr~rr)rSrrrrrrrrrrrrr~rrs rBappend_last_rowzIndex.append_last_rows  77D  2 *D 1 1 1  '$, 'HEIIEIO >,,T5)DDc33xx 3xx ^y0 LLjL(9D6'BCC #/ $0 !dm$$8< lW445"%,  <1  ""3(>??C" )3s88) QY /,>((  1 )4 0 0 0 0 0 1 1rDc|jsdS|rd|_n t|_d|_d|_t |j|j|j}|\}}}}trtd||  |rt|D]}| ddrnl|jdkr0| ddrnI| ddrn1| dd rn| dd rnnK|r| ddrn3|r| dd rn|r| dd rn |j rD|jdkr||jdkrt!jd t$|dS) zOptimize an index so as to allow faster searches. verbose If True, messages about the progress of the optimization process are printed out. NTrz optvalues:chunksmedianrslicesstartstopz[OPSI was not able to achieve a completely sorted index. Please report this to the authors.)rverboser last_tover last_noverrrrrQr create_temp2rswaprrdo_complete_sortwarningswarn UserWarning cleanup_temp)rSropts optmedian optstartsoptstopsoptfullniters rBoptimizezIndex.optimizes%!  F  !DLL DLT\4=$,GG26/ 9h  & , % % %   "7^^  Eyy844|a'' 99Xx88"!E99Xx88"!Eyy733yy622yy844yy733yy622   " I~!!%%'''~!! ;Bring an already optimized index into a complete sorted state.Nrrrru%drz3Two slices cannot overlap completely at this stage!zdo_complete_sort()time: .4f . clock: )rrcpuclockrvrrrr~rarrayrrrrrrrrQzerosint_rsearch_item_ltr r update_cachesrrzrr)rSt1c1ssrrr nelementsLRrangeslrrrrrsremainiremainrr>soverioverprev_endjstjnext_begnext_endnrssortedsindicessendsortedlrrs rBrzIndex.do_complete_sort8s < BB ^hAAA,' ??xq!14=3D EFFH^VhZ$899F qLG+<M (2TZ000(2UT\%9:::rw777wK )K )AHRtz222EHRut|';<<gam4e<>>.hvay'<=uEGG>iq +(=>FHH  $Wh 7 7 74<#CRCLq %crc] ""1gcrcl3335zzCLL0!"R$Y,/"2b4i<0LLK''''&! ( "1" #L[L1>.../(2,@BBDL==%@@A ( qyyyy dh(>4          4urDc d|_tjdd|j\}|_t j|||jd|_|jj x|_ }|j }|j }|j }d|f}tj|j}t#|d||d|d|f t#|d t%|j |d |d|f |dz |z}d|f}t#|d ||d|||f d}t#|d||d||f t#|d||d||f t#|d||d||f t#|d|dd||df t#|d|dd||f |dz|zf}t)|d||d||f |f}t)|dt%|j |d||f d S)!9Create some temporary objects for slice sorting purposes.Tz.tmpz pytables-wrrzTemporary sortedrrrrzTemporary indicesrzTemp chunk boundsrrzTemp start boundsrzTemp end boundsrrrrzTemporary range valuesrNrrrz$Temp Last Row sorted values + boundsrzTemp Last Row indicesN)r\tempfilemkstemprrosclosertmpfilerootrrzrvrWrrrrrrQr) rSfdrrr-rWrrrs rBrzIndex.create_temps '/ K / /D   ~~d&6<< **3 ^ ^,Btz**sHdE!72w @ @ @ @sIx>>>"GB A A A A6b.O$sHdE+>B#8 : : : :sItU,?B5 * * * *sItU,=B5 * * * *sItUOB5 * * * * sHdF'b!W F F F FsItTRE ; ; ; ;a/)+sJe5B5 * * * *sK...-B5 * * * * * *rDc l|j}|j}|j}|j|f}t j|j}|j}t|d||d|d|ft|dt|j |d|d|f|dz |z}|j|f}t|d||d |||f|j f}t|d ||d ||ft|d ||d ||ft|d||d||ft|d||jdfd||dft|d||jfd||fdS)rSsorted2zTemporary sorted 2rrUindices2rzTemporary indices 2bounds2zTemp chunk bounds 2abounds2zTemp start bounds 2zbounds2zTemp end bounds 2mbounds2zMedian bounds 2ranges2rNzTemporary range values 2mranges2zMedian ranges 2N) rzrvrWrrrrrrrrQr)rSrr-rWrrrrs rBrzIndex.create_temp2 s^ ^,r"tz**hsItU#W!R B B B BsJ$, ? ? ?$g1b' C C C C6b./sItU,AB#8 : : : :sJe-BB5 * * * *sJe-@B5 * * * *sJe->B5 * * * * sItdlA%6)7Aw H H H HsJt|o 're = = = = = =rDcl|jrtd|j}|j|z}|j}|j}t |jD]}|j|dd| }|j | d|j |j |d|dgg|j ||d|g|j |dd||j ||dz d|||dzd|}|j ||j ||dzg~~|j|}|j | d|j |jdkr|j} |j} |j} |jd| dd| } t1| } | | d| <t3j| dd|| dgf|_t1|j}|j| | | |z<|jdd| dd<| |_|j|jj_|j|jj_|jrtdd|_|jtA|j!"d|_!d|_#d |_$dS) z>Copy the data and delete the temporaries for sorting purposes.zCopying temporary data...NrrrrNzDeleting temporaries...FT)%rrrrzr{rrrrrrrrrrrrrrrrrrr~rrrrrr}rZrYrrunlinkr\r)rSrrrrr>rrrrrr.r=rrs rBrzIndex.cleanup_temp-s < / - . . .N ^y (ht|$$ B BAZ];;Y;/4466F K  v~~a== > > > K  F2J 78 9 9 9 K  rv2v/ 0 0 0 L  qu"u . . . L  rAvzrz 2 3 3 3R1W[b[)G L   ( ( ( L  !2 3 4 4 4k!nG L  7< @ @ A A A A  q }HI+K|L[L1++I+>CCEEHx==L&.H]l] #NHTTrTNXb\N+KLLDM$-((G<@MH\,"88 9=+IaaaL ,D (,(9 %)-):& < - + , , ,  T %%''' rDc|j}|jx} } |j} tj|j|} t |D]} || z}|| kr |j|z} t | D]]}|| | z|z}|| z}||| zz |z}||z }||z}|| kr||||z| |||z<F|||||zf| |||z<^|| kr| d| |z|d| |z<| ||<dS)z)Get sorted & indices values in new order.rN)rzr{rrrrvrr~)rSnewordersrc_disktmp_disklastrowrrrrrncs2 self_nslicesrr>nsr5rinsincncs rB get_neworderzIndex.get_neworderksT^%%d| hT^5999w # #A!B\!!(B.4[[ B Bq3w{+SjS3Y",v V,&&&-c#(l&;C27 OO&.sCbL/@&AC27 OO\!!&)*4"9*o  ## # 7 # #rDr c 6dddd}|j}|j}|j}|j}|j}|j}|j} |j} |j} |j } | | z} | }| ||}|j dko |j dk}t|jD]}|j|| zz }|| kr|}|dkrn|| z}||| z|| z|z}||jdz kr4|r2|dz }|j | z}t!j||jd|f}|t( }t+|| z}||||||||j||||| ||d |j z|d dS) z/Swap & reorder the different chunks in a block.rrr)r rr rPrrNrTr!T)r)rrrr^r_rrrzr{rw _f_get_childrQr~rrrrrrargsortdefsortrrsrreorder_slices)rSrG boundsnamesrrr tmp_sorted tmp_indicesrrrrnsbncbncb2 boundsobjcan_cross_bboundsnblockremainingchunksrrul sbounds_idxrs rBrCzIndex.swap_chunkss YHH h+[ l <M ^Ci$$[%677 !\Q.H43Dq3HDL)) E EF"lVc\9O$$&qyyckGv|FSL4,??@F$,***%+1 &",ss1C(DEE ..g.66K#&&F   k6:x%vtz ; ; ;   k7K%vut|/C E E E E %%%%%rDcLtj||gtj|_tj|dz||jzgtj|_tjdtj|_||j|j|j|dS)=Read a slice from the `where` dataset and put it in `buffer`.rrNrN) rr&r1startlrstoplonesstepl _g_read_slice)rSrnslicebufferr s rB read_slicezIndex.read_slicesh :: Xvz56;+>?KK W1BI666  DKTZHHHHHrDc`tj||gtj|_tj|dz||jzgtj|_tjdtj|_|j|jz }||j|j||dS)z>Write a `slice` to the `where` dataset with the `buffer` data.rrNrN) rr&r1rrrrr_g_write_slice)rSrrrr countls rB write_slicezIndex.write_slicesh :: Xvz56;+>?KK W1BI666 dk) T[$*ffEEEEErDc tj|gtj}tj||jzgtj}tjdgtj}|||||dS)rrrN)rr&r1rr)rSrrr rrrs rBrzIndex.read_slice_lrst5'333%&+-.bi@@@!BI... FE5&99999rDc tj|gtj}tj||jzgtj}tjdgtj}|||||dS)z>Write a slice from the `where` dataset with the `buffer` data.rrN)rr&r1rr)rSrrr rrrs rB write_sliceLRzIndex.write_sliceLRst5'33356;./ryAAA!BI... VUFF;;;;;rDc|j}|||||d|||||dtj|||||dz |d||||dz |d|||dz |d|||d|d|<||d|d|<dS)z8Copy & reorder the slice in source to final destination.Nr)rvrr rrr*) rSrrrr:r;r{r|r-s rB reorder_slicezIndex.reorder_slices^  FGBCCL999  VXbcc];;; (333 !WSbS\::: &1*hssm<<< 6A:wss|444rss|   " rDcd|j}|j}|j}|ddg|j|<||d||j|<|dd||j||z|dz|z<||dz d||j||z|dz|z<||dzd|}||j||z|dz|z<||dz|j|<dS)z%Update the caches for faster lookups.rrNrrN) rzr{rrrrrrr)rSrr:rrrrs rBr*zIndex.update_cachess^h$aW- 6$RVV_ 67>qu"u~ FSL&1*!3347>rAvzrz7J FSL&1*!334"'+2+&7> FSL&1*!334%cQh/ FrDc  |j}|j}|j}|r|j}|j}n|j}|j}|j}|j}|j|jz}|j} |j } |j } tj |dz|j } tj |dztj d|jz} |jdkr||d| d|||d| d|d}t!d|jD]}||||| | ||| dkr|jj}|jj}| d|| z}||d}| d|| z}||d}||||||t-j||||||||tj|dd||dgf}||| | t5|z<||_|||| d||||| d|||| d|dSt!| D]}||z}|||| d||||| d|||z}|| kr| }|}t!|dz|D]}||||| | |||||| d||||| d|||| d|dS) anReorder completely the index at slice level. This method has to maintain the locality of elements in the ambit of ``blocks``, i.e. an element of a ``block`` cannot be sent to another ``block`` during this reordering. This is *critical* for ``light`` indexes to be able to use this. This version of reorder_slices is optimized in that *two* complete slices are taken at a time (including the last row slice) so as to sort them. Then, each new slice that is read is put at the end of this two-slice buffer, while the previous one is moved to the beginning of the buffer. This is in order to better reduce the entropy of the regular part (i.e. all except the last row) of the index. A secondary effect of this is that it takes at least *twice* of memory than a previous version of reorder_slices() that only reorders on a slice-by-slice basis. However, as this is more efficient than the old version, one can configure the slicesize to be smaller, so the memory consumption is barely similar. rNrr!rPrNrr)rrrr^r_rzrvrsrrr~rrrrQrrrrrrrr rrrrrrr*)rSrrrr{r|rr-r}rrr.r:r;rrrssorted2r= sindices2 indiceslrrnbrlrbs rBryzIndex.reorder_slicess0h+  &J,KKJ+K ^ ^n.,,' (atz:::8"q&9M0N0NOOO <1   OOJ73B3< 8 8 8 OOKHSbSM : : :F6<00 < <""667#*H#-{<<<< Q8, H. "#4B$4#45#BCC=$%6b;&6%67 %bccN ""8X666""9i888 (9===""8X666""9i888>8DDbD>HRL>*JKKDL==%@@A (   VVWSbS\ : : :   Wfhssm < < <   vwss| 4 4 4 4 4Gnn 9 9Cx D'#2#,??? T8CRC=AAASj==!C#D1Hc22@@F&&vvw'.'1;@@@@   ">>>  &(3B3-@@@""673B3<8888+ 9 9rDc|j}|j}|j}|j}|j}|j}|j|jz}|} t|j D]} |j | |zz } | |kr| } | dkrd S|dkr|j | |z| |z| zdf} n?|dkr|j | |z| |z| zdf} n|dkr|j | |z| |z| z} | t} | tj| kdz }|dz| kr|jrt'd | | | z}t| D]}| |}||z}||z}||||<||||<|j ||j|<|j ||j|<|j||j|<||z}|dz|z}||z}|dz|z}|j|||j||<|j|||j||<|j|||j||<t| D]}||z}||||<||||<|j||j |<|j||j |<|j||j|<||z}|dz|z}|j|||j||<|j|||j||<|j|||j||<d S) zSwap slices in a superblock.rr rrr rurN2z$skipping reordering of superblock ->N)rrrr^r_r{rrrvrrrrrrwrxrrsumrrrdrerr`rrarrbrrc)rSrGrrrr{r|rnssnss2sblockremainingslicesr sranges_idxndiffror>roioidxr5jnxjxjns rBrDzIndex.swap_slicesosih+[ l !T^3D-..= 7= 7F"lVc\9O$$&qyywFSL#1D$Da$GHFSL#1D$Da$GH!!Vc\&3,2E%EF ..g.66K BIdOO388::Q>ErzD  <J@&III$B4[[ 9 9!!n!VCx!' 2")$- B"%*T"2 B#&;t#4 R "%*T"2 BH1f^CZax3&%([C%8 QrT"%([C%8 QrT"%([C%8 QrT""4[[ 7 7!V'^r )"o !$R 2"%,r"2 B!$R 2H1f^$'L2$6 AbD!$'L2$6 AbD!$'L2$6 AbD!! 7]= 7= 7rDct|d|rt||dsJ|j}|j}|j}||z} ||jkrt |j||| } nt |j|| dz } | dksJ| |z} ||jkr.| t |j || | |zf|z } | |ksJn5| |z} | |kr|} | t |j | | |z } | |ksJ| dksJ| S)z0Search a single item in a specific sorted slice.rr) rrrzrvr~rr rrrr) rSritemrlimitsr rr-r.bstartnchunkposends rBr)zIndex.search_item_ltsJ F1It,,NdF1I1N1NNNN ^ ^' " DL  f!5tVDDFF!f==AF{{{{rk DL ;u|FCbL,@A4HH HC"99999(C[  ! ;t}SW5t<< r4r5r6r7r8rnelemerangers rBcompute_overlaps_finegrainz Index.compute_overlaps_finegrains ^aaa>,' ??xq!14=3D EFFH^VhZ$899F qLGxgV<<< $$&&""$$w N NAad|H1q5'** N NQibyyyy"99t|##%af~HH'}H!!Q$<h&& Q'''1,'''4<''  rCx/ $&q   {S'88 $/q (( Q'''1,'''--xF1Is<> : : :LLNN <1  !9!9!;!; $-NDM "<22rDc ~|jdd}|j}|jdkrHtj|jd|jdg}tj||gf}|dz }d}d}d} tj|d} t|D]} t| dz|D]m} || df|| dfkrU|dz }| | | z xxdz cc<|j dkr2|t|| dft|| dfz z }n|j dkr6t|dt|d z } | dkr|| z } |r#td |z|| t| |j d kr)|j r|dk|j_||_|| | fS) aCompute some statistics about overlaping of slices in index. It returns the following info: noverlaps : int The total number of slices that overlaps in index. multiplicity : array of int The number of times that a concrete slice overlaps with any other. toverlap : float An ovelap index: the sum of the values in segment slices that overlaps divided by the entire range of values. This index is only computed for numerical types. Nrrrr(rrrrzoverlaps (%s):rP)rrr~rr&rrr'rrrrrQrirr[rr)rSrrKrrrr/rrrrr>r5rs rBrzIndex.compute_overlaps8s aaa,  q xq!14=3D EFFH^VhZ$899F qLG xgV<<< w N NA1q5'** N N!Q$<&A,..NI Q'''1,'''yH,,!E&A,$7$7%q!t :M:M$MM N 9 6%=))E&,,?,??Fzz#f,  "W,i B B B ,    <1  !9!9!;!; $-NDM "<22rDc ||||\}}}||krtjd|jS|dkr|}|j|z }|j|z }|dkr-|j}|j}tj||z |j}n/|j}|j}tj||z d|j z}|j } || z} || z} || z} ||z } d}d}t| | dzD]z}| | z }||z| kr| |z }|dkrn^||j kr$| ||||||z| n"||||||z| d} ||z }||z }{|dd|S)z;Return the sorted or indices values in the specified range.rrrr!rN)_process_rangerrrr}rrrrrQrvrrrr)rSrFr rsteprvaluesvaluesLRbuffer_r- nrow_startistart nrow_stoptlenrilenrblens rBread_sorted_indiceszIndex.read_sorted_indicesvs"11%tDDd D==8Atz** * !88CNT)E>C'D 8  [F}Hhte|4:>>>GG\F~Hhte|54<3GHHHG ^b[ BJ e|*i!m44  D;Dd{T!!d{qyydl""D'&$*>"?IIII""gfVd]&:;VEEEF dNF DLDDvvvrDc2|d|||S)zReturn the sorted values of index in the specified range. The meaning of the start, stop and step arguments is the same as in :meth:`Table.read_sorted`. rrrSr rrs rB read_sortedzIndex.read_sorteds''%tDDDrDc2|d|||S)zReturn the indices values of index in the specified range. The meaning of the start, stop and step arguments is the same as in :meth:`Table.read_sorted`. rrrs rB read_indiceszIndex.read_indicess'' 5$EEErDc||t|dz}|d}nt|}|t|j}nt|}|d}nt|}|||fS)z(Get a range specifc for the index usage.Nrr)rr}rs rBrzIndex._process_ranges{  EAI&&D =EEUOOE <DN++DDD>>D <DDD>>DtT""rDc"t|rDtj|}|dkr ||jz }|||dzddSt |t r&||j|j|j SdS)aReturn the indices values of index in the specified range. If key argument is an integer, the corresponding index is returned. If key is a slice, the range of indices determined by it is returned. A negative value of step in slice is supported, meaning that the results will be returned in reverse order. This method is equivalent to :meth:`Index.read_indices`. rrN) roperatorindexr}r isinstanceslicer rr)rSkeys rB __getitem__zIndex.__getitem__s #;; D.%%CQwwt~%$$S#'155a8 8 U # # D$$SY#(CC C D DrDc|jSr^)r}rRs rB__len__z Index.__len__s ~rDcJ|jj}|j|_t |d|dd|j_t |d|dd|j_ t |d|dd|_ t |d|dd |_ tj |j tj |_ tj |j tj |_ |j|d |_d S) z;Clean the limits cache and resize starts and lengths arraysBOUNDS_MAX_SLOTSBOUNDS_MAX_SIZEznon-opt types boundsLIMBOUNDS_MAX_SLOTSLIMBOUNDS_MAX_SIZEzbounding limitsSORTEDLR_MAX_SLOTSSORTEDLR_MAX_SIZEzlast row chunksrFN)riparamsr_sortedr" boundscachelimboundscache sortedLRcacherrrr,rr_init_sorted_slicer)rSrs rB restorecachezIndex.restorecaches$ { #.v6H/I/56G/H/E$G$G  #.f5G.H.45F.G.D#F#F  ))&1F*G*01E*F*;== +(0D)E)/0C)D):<< >hTZrx@@@ xdjAAA  &&t,,,rDcbtrt}trtd||jr||r|d|dkrd|jdd<d|jdd<dSd}|j|}|dkrJ|j |}d|jdd<|D]\}}}||j|<||j|<||z} |S|j } |j dkrO|j |j vr*|j dkr | j|}n)|j dkr | j|}n|j dkr | j|}n|j dkr | j|}n|j d kr | j|}n|j d kr | j|}n|j d kr | j|}n|j d kr | j|}n|j d kr | j|}nx|j dkr | j|}nb|j dkr | j|}nL|j dkr | j|}n6|j dkr | j|}n Jd||| }|jdkr7||\}} ||jd<| |z |jd<|| |z z }|jryg}t=|jD].\} }|dkr#|| |j| |f/tA|dzdzdz} |j!||| trtd||S)-Do a binary search in this index for an item.zEntering searchrrNr,r-float16r2r3float96float128r0r1r*r+r.r/FzThis can't happen!rrPrNzExiting search)"rrr#rrrrrgetslotgetitemrrrr_search_bin_na_i_search_bin_na_ll_search_bin_na_e_search_bin_na_f_search_bin_na_d_search_bin_na_g_search_bin_na_ui_search_bin_na_ull_search_bin_na_b_search_bin_na_s_search_bin_na_ub_search_bin_na_us search_scalarrsearch_last_rowcouldenablecacher9rrsetitem) rSrrrnslot startlengthsnrow2r lengthrrrrs rBsearchz Index.searchs  77D  0 ($ / / / ?        tAwa((DKNDLO1#++D11 A::.66u==LDLO(4 % %$uf%* E"&, U#f}K r rs rBrzIndex.search_scalaresjt|$$ ! !A"..q$77MUD"DKN"UlDLO D5L DD rDcH|\}}|j}|d|d}}|dd}|jj}|j} |j} |j|jz} d} t||rt||rt||} | | } | dkr| | }ne| | z}| | dzz}|| kr| }|j |j||}| | |||z |zt||}|| | zz }n| }nd}t#||rt%||rt'||}|| kr| |} | dkr| | }ne| |z}| |dzz}|| kr| }|j |j||}| ||||z |zt'||}|| |zz }n| }nd}||fS)Nrrr)rrrrrrzrrrr rrr_read_sorted_slicerrrrrr!)rSritem1item2rb0b1rrrhirrrchunkbeginrr nchunk2rs rBrzIndex.search_last_rowqs u=!hrlB!B$:&*  ^t~5  r " " E2&& $VU33%--f55A::)11%88EE&/E$ 3CRxx  M<< UC11E"))&%**,,+.;(*BDDD#E511f,,E r " " E2&& &vu55f$$)11'::Ezz - 5 5e < < *W 4(GaK888"$C $ @ @ K!5!5 &--guzz||/2U{h.FHHH#E511 W,,Dt}rDc*trt}trtd||j}|j}|j}|j}|j}|j|z }tj |j |z }tj |d} |j} |jdz | zdz} |j|jz| z} d| | dk<|j} t%|jD]}| |}| |}||krtj||z d|z}||kr| ||||n|j||||dkr||z}n^|dkr0|d }t1||z|z}||z }n(|dkr"|d }||z|z}||z }d | |<|j}||kr|j }tj ||z }tj |d}||z }| d}||zd } tj |dz|zd } t7| | D] \}}d |||< |} trtd || S) z3Compute a map with the interesting chunks in index.zEntering get_chunkmapr_rrrr!rPrNr(TzExiting get_chunkmap)rrr#rvrwrrrQrsrrr}rr'rrrrrrr_read_index_slicerrrrrEzip)rSrr-r}rrrQbucketsinblockrchunkmaprrstopsrrr rrrrr}tnchunks tchunkmapratios rB get_chunkmapzIndex.get_chunkmaps  77D  6 . 5 5 5 ^,,,'1)DNW4558'888N +/Y.2t|+y8vz,DJ'' % %F6NE=De||hTE\IIIG##--feT3GGGGN44UD#FFFa<<GOCC\\**V,,C &C->!ABBF6MCC\\**V,,C$rkg5F6MC $ ( l " "Iy\!9::Hxv>>>Il*E""$$Q'CEk))&11FGS1W-..55f==E"6511 - - t(, %*%% H  5 -t 4 4 4rDct|dvsJt|dvsJt|t|ksJ|j}|jj}|j}t|dkr|ddvsJ|d}|d}|dkr&t ||dt |d||f}n |dkrt ||d|f}n|d kr%t |d||t ||df}n|d kr|t ||df}n|d kr||f}nt|d kr|dd vr |ddvsJ|\} } | | krdS|dkr%t | d||t | d||f}n@|dkr| t | d||f}n%|dkrt | d||| f}n |dkr| | f}|S)N)rrNrr)ltleeqgegtr"r)signr#r&r%r$rN)r&r%)r"r#)r&r")r%r")r&r#)r%r#)rrorrrr r ) rSopsrrocoldtyperlimitoprange_loweruppers rBget_lookup_rangezIndex.get_lookup_rangesR3xx6!!!!6{{f$$$$3xx3v;;&&&&<$$ v;;!  q6;;;;;1IEQBTzz!(H2>>>#E2xBBDt!(H2>>>!t#E2xBB!(H2>>>@t!(H2>>>@t [[A  q6\))c!f .D.D.D.D!LE5u}}rl""#E2xBB#E2xBBD $$5"h!I!IJ $$#E2xBBEJ $$ rDcJtddS)zRemove this Index object.TN)r _f_remove)rS recursivers rBr2zIndex._f_remove#s# $rDc^g}|jjrq|jjr|d|jjr|d||jjd|jjdd|jd|jd|d|j S)z9This provides a more compact representation than __repr__r( bitshufflerArBzIndex(z, z ).is_csi=) rWr&r(rr5r'rrTr;r)rSrWs rB__str__z Index.__str__*s < ! P|# *y)))|& -|,,, NNdl2NNT\5KNNN O O OIII9I"ii00II;?;II JrDc|jjd|jj}dg|jd|d|jd|jd|jd|jd|j d |j d |j d |j d |j d |jd|jd|jd|jd|jd|jd|jd|j}|S)z2This provides more metainfo than standard __repr__z.cols.rz (Index for column z) optlevel := z kind := z filters := z is_csi := z nelements := z chunksize := z slicesize := z blocksize := z superblocksize := z dirty := z byteorder := z sorted := z indices := z ranges := z bounds := z sortedLR := z indicesLR := )r`rhropathnamer;rrTrWrr}rzrvrsrrr\rrrrrrr)rS cpathnameretstrs rB__repr__zIndex.__repr__8s5z-KKT[5IKK %%%%d&%%%%9%%%%}%%%% 9%%%%l%%%%[ %%%%  %%%%  %%%%%%%%%%%%(%%%% J%%%%%%%%{%%%% %%%%{%%%%{%%%% !%%%%".#%%%%$ rD) NrNNNNrNNT)Fr^)r r)NNN)C__name__ __module__ __qualname____doc__ _c_classidpropertyrTrWr\setterror`rtrwr{rrrrrrrrrrrrrrrrrrrrrrrrsrCrrrrrr*ryrDr)rrrrrrrrrr rrr r0r2r6r; __classcell__)rs@rBrFrFYsl11fJ 66X6 X 66X6 \ $ $\ $XX55X500X000X0XX00X000X0,,X,33X3 88X8##X#,((X( X0#% F@F@F@F@F@F@PXXXXXtMMM^(?/?/?/?/B.1.1.1.1`IIIIVnMnMnM`++++Z2*2*2*h"="="=H<<<|######J+&+&+&+&ZIIIIFFFF::::<<<<&&&$000"i9i9i9VH7H7H7H7T    D[3[3[3z<3<3<3|(((TEEEEFFFF###(DDD,   @aaaH   DDDL:::x...`       J J JrDrFceZdZdZdZdS) IndexesDescGDINDEXcJtjd|jztdS)Nzthe number of indexed columns on a single description group is exceeding the recommended maximum (%d); be ready to see PyTables asking for *lots* of memory and possibly slow I/Orr_v_max_group_widthrrRs rB_g_width_warningzIndexesDescG._g_width_warningT7  $'+&= >       rDN)r<r=r>r@rJr(rDrBrErEQs(J     rDrEceZdZdZedZejdZejdZdZdZ edZ dS) IndexesTableGTINDEXc:d|jvrtS|jjS)N AUTO_INDEX)r[default_auto_indexrPrRs rBautozIndexesTableG.auto`s t} , ,% %}''rDc8t||j_dSr^)r_r[rP)rSrRs rBrRzIndexesTableG.autofs#'::    rDc|j`dSr^)r[rPrRs rBrRzIndexesTableG.autojs M $ $ $rDcJtjd|jztdS)Nzthe number of indexed columns on a single table is exceeding the recommended maximum (%d); be ready to see PyTables asking for *lots* of memory and possibly slow I/OrHrRs rBrJzIndexesTableG._g_width_warningnrKrDcT|dstd|zdS)Nr6z1names of index groups must start with ``_i_``: %s)r: ValueError)rSr?s rB _g_check_namezIndexesTableG._g_check_namevs@u%% LCdJLL L L LrDc|jd}|dd}d|}t ||}|j|}|S)zJAccessor for the `Table` object of this `IndexesTableG` container.r5r7N)rhr8rr;rrirj)rSr= tablenameparentpathnamer@r`s rBr`zIndexesTableG.table{sg &&s++IIKKO %!.)<<  &&}55 rDN) r<r=r>r@rArRrBdeleterrJrXr`r(rDrBrMrM]sJ ((X(  [..[. \%%\%   LLL XrDrMceZdZdZdZdS)OldIndexz4This is meant to hide indexes of PyTables 1.x files.CINDEXN)r<r=r>r?r@r(rDrBr^r^s>>JJJrDr^)Jr?rrrXrrVrpathlibrtimerrrr%numpyridxutilsrrr r r rr rr rrrearrayrcarrayrleafrindexesrrrgrouprpathr exceptionsrutilsrrrutilsextensionrrrrr r!lrucacheextensionr"rrrr#rxrQdefault_index_filtersdefaultAutoIndexdefaultIndexFiltersrmax32rCrFrErMr^r(rDrBrrsx&&  &&&&&&))))))@@@@@@@@@@@@@@ 9999999999******----------8888888888888888+*****     "!!!!!!    !V(,@@@&+* (((uuuuuNE#3#9uuup?      >5    '''''NE'''T~urD