OOIdg ddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ej dZ Gdde Z Gdde ZGdd e ZGd d e ZGd d e ZGdde ZGdde ZGdde ZGddZGdde Zde iZddZe eeeeeeeef D] ZeedS)N)ThreadPoolExecutorfsspecc"eZdZdZdZdZdZdS) BaseCacheagPass-though cache: doesn't keep anything, calls every time Acts as base class for other cachers Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is nonec0||_||_||_dSN) blocksizefetchersize)selfr r r s .lib/python3.11/site-packages/fsspec/caching.py__init__zBaseCache.__init__ s"  cn|d}||j}||jks||krdS|||SNrr)r r )r startstops r_fetchzBaseCache._fetch%sF =E <9D DI  $3||E4(((rN)__name__ __module__ __qualname____doc__namerrrrrrsC   D )))))rrc@eZdZdZdZd fd ZdZdZdZdZ xZ S) MMapCachezmemory-mapped sparse file cache Opens temporary file, which is filled blocks-wise when data is requested. Ensure there is enough disc space in the temporary location. This cache method might only work on posix mmapNct||||tn||_||_||_dSr )superrsetblockslocation _makefilecache)r r r r r#r" __class__s rrzMMapCache.__init__:sN GT222%~ceee6   ^^%% rcHddl}ddl}|jdkrtS|j$t j|js|j(|}t|_ ntj |jd}| |jdz |d|ntj |jd}|||jS)Nrzwb+1zrb+)rtempfiler bytearrayr#ospathexists TemporaryFiler!r"ioopenseekwriteflushfileno)r rr*fds rr$zMMapCache._makefile@s  9>>;;  = t}(E(E }$++--!ee WT]E22 GGDIM " " " HHTNNN HHJJJJ..Byydi000rcBtd|d||d}|j}|jks||krdS|jz}|jz}fdt ||dzD}|r|d}|jz}t |jzj}td|d|d|||j||<j ||j||S) NzMMap cache fetching -rrc&g|] }|jv |Sr)r").0ir s r z$MMapCache._fetch..`s%UUUa@T@T@T@T@Trr(zMMap get block #z () loggerdebugr r rangepopminr r%r"add) r rend start_block end_blockneedr;sstartsends ` rrzMMapCache._fetchVsL 9E99C99::: =E ;)C DI  #3t~- 4>) UUUU5i!m<<UUU  A'Fv. ::D LL@A@@@@$@@ A A A&*ll64&@&@DJvd{ # KOOA    z%)$$rc>|j}|d=|S)Nr%)__dict__copyr states r __getstate__zMMapCache.__getstate__ms! ""$$ 'N rcl|j|||_dSr )rJupdater$r%rLs r __setstate__zMMapCache.__setstate__ss- U###^^%% r)NN) rrrrrrr$rrNrQ __classcell__r&s@rrr/s D&&&&&& 111,%%%. &&&&&&&rrc,eZdZdZdZfdZdZxZS)ReadAheadCachea!Cache which reads only when we get beyond a block of data This is a much simpler version of BytesCache, and does not attempt to fill holes in the cache or keep fragments alive. It is best suited to many small reads in a sequential order (e.g., reading lines from a file). readaheadcxt|||d|_d|_d|_dS)Nrr)r rr%rrCr r r r r&s rrzReadAheadCache.__init__s8 GT222  rcJ|d}| ||jkr|j}||jks||krdS||z }||jkr*||jkr|j||jz ||jz S|j|cxkr |jkr4nn1|j||jz d}|t |z}|j}nd}t |j||jz}||||_||_|jt |jz|_||jd|zSr)r rrCr%lenrAr r )r rrClparts rrzReadAheadCache._fetchs9 =E ;# //)C DI  #3 %K DJ  3$(??:edj033CCD D Z5 + + + +48 + + + + +:edj0223D TNAHEED$)S4>122\\%--  :DJ/dj!n$$rrrrrrrrrRrSs@rrUrUysX D %%%%%%%rrUc,eZdZdZdZfdZdZxZS)FirstChunkCachezCaches the first block of a file only This may be useful for file types where the metadata is stored in the header, but is randomly accessed. firstc\t|||d|_dSr )r rr%rXs rrzFirstChunkCache.__init__s* GT222 rc|pd}|p|j}||jkr|j_||jkr4|d|}|d|j|_||dS|d|j|_|j||}||jkr|||j|z }|S|||SNr)r r r%r )r rrCdatar\s rrzFirstChunkCache._fetchs TY 4> ! !z!''<<3//D!%&6&6!7DJ<'!\\!T^<< :eCi(DT^## T^S999K<<s++ +rr]rSs@rr_r_sX D,,,,,,,rr_cVeZdZdZdZd fd ZdZdZdZdZ d Z fd Z d Z xZ S) BlockCachea Cache holding memory as a set of blocks. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. blockcache ct|||tj||z |_||_t j||j|_ dSr ) r rmathceilnblocks maxblocks functools lru_cache _fetch_block_fetch_block_cachedr r r r rmr&s rrzBlockCache.__init__sb GT222y !122 "#A9#6y#A#A$BS#T#T   rcNd|j|j|jS)Nz.formatr r rlr s r__repr__zBlockCache.__repr__s'?FF NDIt|   rc4|jSz The statistics on the block cache. Returns ------- NamedTuple Returned directly from the LRU Cache used internally. rq cache_inforvs rr{zBlockCache.cache_info'22444rc|j}|d=|S)NrqrJrLs rrNzBlockCache.__getstate__s  ' ( rc|j|tj|d|j|_dS)Nrm)rJrPrnrorprqrLs rrQzBlockCache.__setstate__sH U####J9#6u[7I#J#J  $ $    rc|d}||j}||jks||krdS||jz}||jz}t||dzD]}|||||||S)Nrrr(start_block_numberend_block_number)r r r?rq _read_cache)r rrCrr block_numbers rrzBlockCache._fetchs =E ;)C DI  #3#dn4$.0""46F6JKK 3 3L  $ $\ 2 2 2 2  1-     rc||jkr(td||j||jz}||jz}td|t ||}|S)= Fetch the block of data for `block_number`. ;'block_number={}' is greater than the number of blocks ({})zBlockCache fetching block %drl ValueErrorrur r=infor r)r rrrCblock_contentsr&s rrpzBlockCache._fetch_blocks $, & &MTT $,  t~-dn$ 2LAAAs33rc||jz}||jz}||kr||}|||Sg}||||dt|dz|D]*} ||| +|||d|d|Sz Read from our block cache. Parameters ---------- start, end : int The start and end byte positions. start_block_number, end_block_number : int The start and end block numbers. Nr(rr rqappendr?join r rrCrr start_posend_posblockoutrs rrzBlockCache._read_cache$DN* & !1 1 1,,-?@@E7*+ +C JJt//0BCCIJJO P P P !&&81&<>N O O C C  433LAABBBB JJt//0@AA(7(K L L L88C== rrhrrrrrrrwr{rNrQrrprrRrSs@rrfrfs, DUUUUUU    5 5 5       ." ! ! ! ! ! ! !rrfc4eZdZdZdZdfd ZdZdZxZS) BytesCacheaKCache which holds data in a in-memory bytes object Implements read-ahead by the block size, for semi-random reads progressing through the file. Parameters ---------- trim: bool As we read more data, whether to discard the start of the buffer when we are more than a blocksize ahead of it. bytesTct|||d|_d|_d|_||_dS)Nr)r rr%rrCtrim)r r r r rr&s rrzBytesCache.__init__Vs? GT222   rc|d}||j}||jks||krdS|j<||jkr1|j*||jkr||jz }|j|||z|z S|jrt |j||jz}n|}||ks ||jkrdS|j ||jkr5|j ||jkr#||||_||_n||jkrh|j|z |jkr#||||_||_n|||j}||_||jz|_n|||jkrq|j|jkrn`||jz |jkr#||||_||_n*||j|}|j|z|_|jt|jz|_||jz }|j|||z|z }|jrT|j|jz |jdzz}|dkr4|xj|j|zz c_|j|j|zd|_|S)Nrrr() r rrCr%r rAr rZr)r rrCoffsetbendnewrnums rrzBytesCache._fetch]s =E ;)C DI  #3 J "##$dhTZ'F:fv|e';;< < > ty#"677DDD 5==EDI--3 J %$*"4"4 H dheT22DJDJJ TZ  x#~..!\\%66 " ll5$*55"  4:- DH__x$)##tx$.00!\\%66 " ll48T22!Z#- :DJ/#j&3,"667 9 @8dj(dnq.@ACQww dns22 !Z(<(>(>?  rc*t|jSr )rZr%rvs r__len__zBytesCache.__len__s4:r)T) rrrrrrrrrRrSs@rrrGsm   D;;;zrrc.eZdZdZdZdfd ZdZxZS)AllBytesz!Cache entire contents of the fileallNct|||||d|j}||_dSrc)r rr r rd)r r r r rdr&s rrzAllBytes.__init__sD GT222 <<<49--D rc |j||Sr )rd)r rrCs rrzAllBytes._fetchsys##r)NNNNr]rSs@rrrsW++ D $$$$$$$rrc6eZdZdZdZidffd ZfdZxZS)KnownPartsOfAFilea Cache holding known file parts. Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is data: dict A dictionary mapping explicit `(start, stop)` file-offset tuples with known bytes. strict: bool, default True Whether to fetch reads that go beyond a known byte-range boundary. If `False`, any read that ends outside a known part will be zero padded. Note that zero padding will not be used for reads that begin outside a known byte-range. partsTc tt||||||_|rt t |}|dg}||dg} |ddD]\} } |d\} } | | kr-| | f|d<| dxx|| | fz cc<C|| | f| || | ftt|| |_ dS||_ dS)Nrr() r rrstrictsortedlistkeysr@rdictziprd)r r r r rdr_ old_offsetsoffsetsr"rrstart0stop0r&s rrzKnownPartsOfAFile.__init__sA &&// 7DIII    diikk!2!233K"1~&Ghh{1~../F*122 ; ; t ' E>>#)4.GBK2JJJ$((E4="9"99JJJJNNE4=111MM$((E4="9"9::::S&1122DIIIDIIIrcd}|jD]g\\}}}||cxkr|krRn||z }||||z|z }|jr||cxkr|kr"nn|d||z t|z zz }|cS|}nh|jt d||fdt jd||fdt d|d||t ||zS)Nrz&Read is outside the known file parts: z. z%. IO/caching performance may be poor!z!KnownPartsOfAFile cache fetching r8) rditemsrrZr rwarningswarnr=r>r r) r rrrloc0loc1rdoffr&s rrzKnownPartsOfAFile._fetchsp"&)//"3"3   LT4$u####t#####dl3te!334{ dd&:&:&:&:d&:&:&:&:&: 7dUlSXX&=>>CJJJ !EE$( < WudmWWWXX X   3eT] 3 3 3     GGGGGHHHUWW^^E40000rr]rSs@rrrsl, D68,#1#1#1#1#1#1#1#1#1rrcXeZdZdZejdgdZd dZdZdZ dZ d Z d S) UpdatableLRUzg Custom implementation of LRU cache that allows updating keys Used by BackgroudBlockCache CacheInfo)hitsmissesmaxsizecurrsizectj|_||_||_d|_d|_tj|_ dSrc) collections OrderedDict_cache_func _max_size_hits_misses threadingLock_lock)r funcmax_sizes rrzUpdatableLRU.__init__ sA!-//  !  ^%% rc|j5||jvrC|j||xjdz c_|j|cdddS dddn #1swxYwY|j|}|j5||j|<|xjdz c_t |j|jkr|jddddn #1swxYwY|S)Nr(Flast) rr move_to_endrrrrZrpopitem)r argsresults r__call__zUpdatableLRU.__call__s Z ) )t{"" ''--- a {4(  ) ) ) ) ) ) ) )" ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) T" Z 0 0 &DK  LLA LL4;$.00 ###///  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  s%AA""A&)A&>ACC!$C!cV|j5||jvcdddS#1swxYwYdSr )rr)r rs r is_key_cachedzUpdatableLRU.is_key_cached&sx Z ' '4;& ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' 's ""c|j5||j|<t|j|jkr|jdddddS#1swxYwYdS)NFr)rrrZrr)r rrs radd_keyzUpdatableLRU.add_key*s Z 0 0 &DK 4;$.00 ###/// 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0sAAAAc|j5||jt|j|j|jcdddS#1swxYwYdS)N)rrrr)rrrrZrrrrvs rr{zUpdatableLRU.cache_info0s Z  >>T[))Z| "                  s:AAAN)r) rrrrr namedtuplerrrrrr{rrrrrs ' &>>>I&&&&"'''000 rrcXeZdZdZdZd fd ZdZdZdZdZ d Z dfd Z d Z xZ S)BackgroundBlockCachea Cache holding memory as a set of blocks with pre-loading of the next block in the background. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. If the next block is not in cache, it is loaded in a separate thread in non-blocking way. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. backgroundrhc>t|||tj||z |_||_t |j||_td|_ d|_ d|_ tj|_dS)Nr( max_workers)r rrjrkrlrmrrprqr_thread_executor_fetch_future_block_number _fetch_futurerr_fetch_future_lockrrs rrzBackgroundBlockCache.__init__Vs GT222y !122 "#/0A9#M#M 2q A A A*.'!"+."2"2rcNd|j|j|jS)Nz8rtrvs rrwzBackgroundBlockCache.__repr__as'IPP NDIt|   rc4|jSryrzrvs rr{zBackgroundBlockCache.cache_infofr|rc2|j}|d=|d=|d=|d=|d=|S)Nrqrrrrr~rLs rrNz!BackgroundBlockCache.__getstate__qs:  ' ( $ % . / / " & ' rc|j|t|j|d|_t d|_d|_d|_tj |_ dS)Nrmr(r) rJrPrrprqrrrrrrrrLs rrQz!BackgroundBlockCache.__setstate__zsk U####/0A5CU#V#V 2q A A A*.'!"+."2"2rcD|d}||j}||jks||krdS||jz}||jz}d}d}|j5|j|jr`t d|j|j |j d|_ d|_n@t||j cxko|knc}|r|j }|j}d|_ d|_dddn #1swxYwY|Gt d|j| |t||dzD]}|||dz} |j5|jR| |j krG|j| s-| |_ |j|j| d|_dddn #1swxYwY|||||S)Nrrz3BlockCache joined background fetch without waiting.z(BlockCache waiting for background fetch.r(asyncr)r r rrdoner=rrqrrrboolr?rlrrsubmitrpr) r rrCrrfetch_future_block_number fetch_future must_joinrend_block_plus_1s rrzBackgroundBlockCache._fetchs =E ;)C DI  #3#dn4$.0$(!  $ 2 2!-%**,,2KK UVVV,44*1133T5T7;D3)-D&&!%*:,,,,+,,,,!!I !2594S1'+'9 ;?7-1*5 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2:  # KKB C C C  $ , ,##%%'@    ""46F6JKK 3 3L  $ $\ 2 2 2 2,a/  $  "*$ 440>>?OPP53C/%)%:%A%A%'7&&"                 1-     s&CD  DDAG>>HHsyncc||jkr(td||j||jz}||jz}td||t ||}|S)rrz!BlockCache fetching block (%s) %dr)r rlog_inforrCrr&s rrpz!BackgroundBlockCache._fetch_blocks $, & &MTT $,  t~-dn$ 7<PPPs33rc||jz}||jz}||kr||}|||Sg}||||dt|dz|D]*} ||| +|||d|d|Srrrs rrz BackgroundBlockCache._read_cacherrr)rrrSs@rrr:s2 D 3 3 3 3 3 3   5 5 5333I I I V" ! ! ! ! ! ! !rrFc~|j}|s)|tvr td|dt||t|<dS)z'Register' cache implementation. Parameters ---------- clobber: bool, optional If set to True (default is False) - allow to overwrite existing entry. Raises ------ ValueError zCache with name z is already known: N)rcachesr)clsclobberrs rregister_cachersL 8D Wtv~~UDUUvd|UUVVVF4LLLr)F)rrnr0loggingrjr,rrconcurrent.futuresr getLoggerr=objectrrrUr_rfrrrrrrrcrrrr s  111111  8 $ $))))))))BG&G&G&G&G& G&G&G&T'%'%'%'%'%Y'%'%'%T,,,,,i,,,>B!B!B!B!B!B!B!B!JTTTTTTTTn $ $ $ $ $y $ $ $R1R1R1R1R1 R1R1R1j55555555pD!D!D!D!D!9D!D!D!R )  (     AN1  r