cAdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlmZejddkr ddlZddlmZn ddlmZddlmZejdZejdeZ ejejed ZeZej e!Z"d Z#d Z$d1d Z%dZ&dZ'e#dfdZ(d2dZ)d3dZ*dZ+dZ,dZ-d4dZ.e!dkrAej/dej0ej1ej2ddZ3e34Z5e56d d!d"d d#$d Z7e56d%d&d"d'e7d()e38Z9e9j:0e.e9j:dd*Z;e")d+e;dSe9j)we9j)d,kr'e<ej=e)d-d./dSe9j)e7kr e)ne)e9j)0Z>e<ej=e>d./dSdSdS)5a' This module is an API for downloading, getting information and loading datasets/models. See `RaRe-Technologies/gensim-data `_ repo for more information about models/datasets/how-to-add-new/etc. Give information about available models/datasets: .. sourcecode:: pycon >>> import gensim.downloader as api >>> >>> api.info() # return dict with info about available models/datasets >>> api.info("text8") # return dict with info about "text8" dataset Model example: .. sourcecode:: pycon >>> import gensim.downloader as api >>> >>> model = api.load("glove-twitter-25") # load glove vectors >>> model.most_similar("cat") # show words that similar to word 'cat' Dataset example: .. sourcecode:: pycon >>> import gensim.downloader as api >>> from gensim.models import Word2Vec >>> >>> dataset = api.load("text8") # load dataset as iterable >>> model = Word2Vec(dataset) # train w2v model Also, this API available via CLI:: python -m gensim.downloader --info # same as api.info(dataname) python -m gensim.downloader --info name # same as api.info(name_only=True) python -m gensim.downloader --download # same as api.load(dataname, return_path=True) You may specify the local subdirectory for saving gensim data using the GENSIM_DATA_DIR environment variable. For example: $ export GENSIM_DATA_DIR=/tmp/gensim-data $ python -m gensim.downloader --download By default, this subdirectory is ~/gensim-data. )absolute_importN)partial)urlopenz ~/gensim-dataGENSIM_DATA_DIRz..zPhttps://raw.githubusercontent.com/RaRe-Technologies/gensim-data/master/list.jsonzBhttps://github.com/RaRe-Technologies/gensim-data/releases/downloadcd}t||z}ttj||z|z }t |dz|z d}d|zd||z zz} |dkr}t jd| d|dd t |d z dd t t|d z dd t jdSt jd |dzd |d| d|dd t |d z dd t t|d z dd t jdS)a%Reporthook for :func:`urllib.urlretrieve`, code from [1]_. Parameters ---------- chunks_downloaded : int Number of chunks of data that have been downloaded. chunk_size : int Size of each chunk of data. total_size : int Total size of the dataset/model. part : int, optional Number of current part, used only if `no_parts` > 1. total_parts : int, optional Total number of parts. References ---------- [1] https://gist.github.com/vladignatyev/06860ec2040cb497f0f3 2dr=-z [z] % i/z MB downloadedz Part z [N) floatintmathfloorroundsysstdoutwriteflush) chunks_downloaded chunk_size total_sizepart total_partsbar_lensize_downloaded filled_lenpercent_downloadedbars 1lib/python3.11/site-packages/gensim/downloader.py _progressr%_s,G- :;;OTZ?!:j HIIJJJ3!6* DqII SGj$89 9Ca ''o5q9999eJ'';7;;;; =    q+++sss,>,>o5q9999eJ'';7;;;; =   ctjts tdttjtdS#t$rh}|jtj kr'td ttd td}~wwxYwdS)aCreate the gensim-data directory in home directory, if it has not been already created. Raises ------ Exception An exception is raised when read/write permissions are not available or a file named gensim-data already exists in the home directory. z Creating %sz^Not able to create folder gensim-data in {}. File gensim-data exists in the directory already.z{Can't create {}. Make sure you have the read/write permissions to the directory or you can try creating the folder manuallyN) ospathisdirBASE_DIRloggerinfomakedirsOSErrorerrnoEEXIST Exceptionformat _PARENT_DIR)es r$_create_base_dirr6s 7== " "  KK x 0 0 0 K ! ! ! ! !   w%,& 77=vk7J7J  SVH%%  s9A!! C+A#CCctj}t|d5tfddD]}|| dddn #1swxYwY|S)zCalculate the checksum of the file, exactly same as md5-sum linux util. Parameters ---------- fname : str Path to the file. Returns ------- str MD5-hash of file names as `fname`. rbc.dS)Ni)read)fsr$z)_calculate_md5_checksum..s!&&,,r&r&N)hashlibmd5openiterupdate hexdigest)fnamehash_md5chunkr;s @r$_calculate_md5_checksumrFs{}}H eT  #a....44 # #E OOE " " " " ################     s,AA"%A"zutf-8cZtjtd}t  t |}t|d5}||dddn #1swxYwYn3#ttf$rt d||YnwxYw tj|d|5}tj|cdddS#1swxYwYdS#t$rt!d|zwxYw)zLoad dataset information from the network. If the network access fails, fall back to a local cache. This cache gets updated each time a network request _succeeds_. zinformation.jsonwbNzlcaught non-fatal exception while trying to update gensim-data cache from %r; using local cache at %r insteadr)encodingzPunable to read local cache %r during fallback, connect to the Internet and retry)r(r)joinr+r6rr:r?rr/IOErrorr, exceptioniojsonload ValueError)urlrJ cache_path info_bytesfoutfins r$ _load_inforWs h(:;;J#S\\&&(( *d # # #t JJz " " " # # # # # # # # # # # # # # # W       ./2J         WZx 8 8 8 "C9S>> " " " " " " " " " " " " " " " " " "     02< =    sS!B&BB B -CCD D3 D DD DD D*TFct}|C|d}|d}||vr|d|S||vr|d|Std|s|S|r=t|dt|ddSd|dDd|dDdS)a7Provide the information related to model/dataset. Parameters ---------- name : str, optional Name of model/dataset. If not set - shows all available data. show_only_latest : bool, optional If storage contains different versions for one data/model, this flag allow to hide outdated versions. Affects only if `name` is None. name_only : bool, optional If True, will return only the names of available models and corpora. Returns ------- dict Detailed information about one or all models/datasets. If name is specified, return full information about concrete dataset/model, otherwise, return information about all available datasets/models. Raises ------ Exception If name that has been passed is incorrect. Examples -------- .. sourcecode:: pycon >>> import gensim.downloader as api >>> api.info("text8") # retrieve information about text8 dataset {u'checksum': u'68799af40b6bda07dfa47a32612e5364', u'description': u'Cleaned small sample from wikipedia', u'file_name': u'text8.gz', u'parts': 1, u'source': u'http://mattmahoney.net/dc/text8.zip'} >>> >>> api.info() # retrieve information about all available datasets and models NcorporamodelsIncorrect model/corpus name)rYrZcFi|]\}}|dd||SlatestTget.0namedatas r$ zinfo..s6nnn. s6lll,4SWS[S[\dfjSkSkl4lllr&)rWrQlistkeysitems)rcshow_only_latest name_only informationrYrZs r$r-r-sP,,K <i(X& 7? <y)$/ / V^ <x(. .:;; ; g I 6 ; ; = =>>${[cOdJeJefffon;y3I3O3O3Q3Qnnnll+h2G2M2M2O2Olll  r&cVt}|d}|d}|2||vr|d|dS||vr|d|dSdS||vr'|d|d|S||vr'|d|d|SdS)a0Retrieve the checksum of the model/dataset from gensim-data repository. Parameters ---------- name : str Dataset/model name. part : int, optional Number of part (for multipart data only). Returns ------- str Retrieved checksum of dataset/model. rYrZNchecksumz checksum-{})r-r3)rcrrlrYrZs r$ _get_checksumro$s &&K)$G  "F K 7? ;y)$/ ; ; V^ ;x(.z: : ; ; 7? Ky)$/ 0D0DT0J0JK K V^ Kx(.}/C/CD/I/IJ J K Kr&ct}|d}|d}||vr|d|dS||vr|d|dSdS)zRetrieve the number of parts in which dataset/model has been split. Parameters ---------- name: str Dataset/model name. Returns ------- int Number of parts in which dataset/model has been split. rYrZpartsNr-rcrlrYrZs r$ _get_partsrtCsm&&K)$G  "F w49%d+G44 48$T*73344r&c zdt|}tjt |}|dz}t j}tj|d}tj ||t|}|dkr/d|}tj||}td|D]#} d t|| } d || } tj|| } tj | | tt| | t| t|| kr^t jdt jt(d| dz|t-j|t1dt3|d5} td|D]} tj|d|| }t3|d5}t-j|| dddn #1swxYwYtj| dddn #1swxYwYn dt|} d|} tj|| } tj | | tt| t|krYt jdt jt(d|n#t-j|t1dtj|rtj|t-j||tj||dS)zDownload and extract the dataset/model. Parameters ---------- name: str Dataset/model name which has to be downloaded. Raises ------ Exception If md5sum on client and in repo are different. z{base}/{fname}/__init__.py)baserC_tmpz __init__.pyrz {fname}.gz)rCrz!{base}/{fname}/{fname}.gz_0{part})rvrCrz {f}.gz_0{p})r;p)rr) reporthook zPart %s/%s downloadedz%Checksum comparison failed, try againrHz{fname}.gz_0{part})rCrr8Nz{base}/{fname}/{fname}.gzz %s downloaded)r3DOWNLOAD_BASE_URLr(r)rKr+tempfilemkdtempurllib urlretrievertrangerr%rFrorrrrr,r-shutilrmtreer2r? copyfileobjremoveexistsmoverename)rc url_load_filedata_folder_dirdata_folder_dir_tmptmp_dir init_pathrconcatenated_folder_nameconcatenated_folder_dirrurl_datarCdst_pathwfp part_pathrfps r$ _downloadrZsA177=NVZ7[[Mgll8T22O)F2  G Wm44I }i000T""KQ$E#/#6#6T#6#B#B "$',,w8P"Q"Q![)) I ID:AAGX`dkoAppH!((44(88Ew||GU33H  ("94[QQQ    'x00M$4M4MM I   &&&   """ 3TAX{KKKK g&&& GHHH )4 0 0 %Ca-- % %GLL2F2M2MTX_c2M2d2dee )T**1c&sC000111111111111111 )$$$$  % % % % % % % % % % % % % % % %/55;LTX5YY##$#//7<<//8X)DDDD "8 , , d0C0C C E J  T " " " J      KK . . . . M' " " "CDD D w~~)**' %&&& K,---I!?33333s7 AK8J KJ K!J "KK K ct}|d}|d}||vr|d|dS||vr|d|dSdS)zRetrieve the filename of the dataset/model. Parameters ---------- name: str Name of dataset/model. Returns ------- str: Filename of the dataset/model. rYrZ file_nameNrrrss r$ _get_filenamersm&&K)$G  "F w89%d+K88 88$T*;7788r&ctt|}|tdtjt |}tj||}tj|st||r|Stj dt t|}| S)aDownload (if needed) dataset/model and load it to memory (unless `return_path` is set). Parameters ---------- name: str Name of the model/dataset. return_path: bool, optional If True, return full path to file, otherwise, return loaded model / iterable dataset. Returns ------- Model Requested model, if `name` is model and `return_path` == False. Dataset (iterable) Requested dataset, if `name` is dataset and `return_path` == False. str Path to file with dataset / model, only when `return_path` == True. Raises ------ Exception Raised if `name` is incorrect. Examples -------- Model example: .. sourcecode:: pycon >>> import gensim.downloader as api >>> >>> model = api.load("glove-twitter-25") # load glove vectors >>> model.most_similar("cat") # show words that similar to word 'cat' Dataset example: .. sourcecode:: pycon >>> import gensim.downloader as api >>> >>> wiki = api.load("wiki-en") # load extracted Wikipedia dump, around 6 Gb >>> for article in wiki: # iterate over all wiki script >>> pass Download only example: .. sourcecode:: pycon >>> import gensim.downloader as api >>> >>> print(api.load("wiki-en", return_path=True)) # output: /home/user/gensim-data/wiki-en/wiki-en.gz Nr[r) r6rrQr(r)rKr+rrrinsert __import__ load_data)rc return_pathr folder_dirr)modules r$rPrPsld##I86777h--J 7<< I . .D 7>>* % %$"  8$$$D!!!!!r&__main__z4%(asctime)s : %(name)s : %(levelname)s : %(message)s)r3streamlevelzGensim console APIzCpython -m gensim.api.downloader [-h] [-d data_name | -i data_name]) descriptionusagez-dz --download data_namezFTo download a corpus/model : python -m gensim.downloader -d )metavarnargshelpz-iz--info?zSTo get information about a corpus/model : python -m gensim.downloader -i )rrconstr)rz+Data has been installed and data path is %src)rk)indent)rc)rr)NTF)N)F)?__doc__ __future__rargparser(rNrOloggingrr0r=rrr| functoolsr version_infor~urllib2rurllib.requestrequestr) expanduser_DEFAULT_BASE_DIRenvironr`r+abspathrKr4base_dir getLogger__name__r, DATA_LIST_URLr{r%r6rFrWr-rortrrrP basicConfigrINFOArgumentParserparseradd_mutually_exclusive_groupgroup add_argumentfull_information parse_argsargsdownload data_pathprintdumpsoutputr&r$rs33j'&&&&&   A!'MMM######&&&&&&G&&77 :>>+-> ? ? goobgll8T::;;    8 $ $b X****Z8   *!7$ $ $ $ N====@KKKK>444.?4?4?4D888.D"D"D"D"N z0GEcj`g`l%X $(SF  / / 1 1E  lKq U   h 3>N b     D }0Dq)t<<<  A9MMMMM 0 9  0 E*$*TTD111!<<< = = = = = $ -= =XTTVVVDDdiDXDXDXF E*$*VA... / / / / /=00200r&