U ėžŌdØ5ć@s”dZddlZddlZddlZddlZddlZddlZddlm Z ddl Z ddl Z ddl mZddl mZddlmZmZe e”ZGdd„dƒZdS) a .. module:: BuscoDownloadManager :synopsis: BuscoDownloadManager manage the version and download the most recent file .. versionadded:: 4.0.0 .. versionchanged:: 5.4.0 Copyright (c) 2016-2023, Evgeny Zdobnov (ez@ezlab.org) Licensed under the MIT license. See LICENSE.md file. éN)ŚURLError)Ś BuscoLogger)Ś LogDecorator)ŚBatchFatalErrorŚ BuscoErrorc@sžeZdZdZdZdd„Zdd„Zedeƒdd „ƒZ d d „Z e d d „ƒZ dd„Z dd„Ze dd„ƒZededddd„ƒZe dd„ƒZededddd„ƒZdS)ŚBuscoDownloadManagera3 This class obtains and manages the version of data files required to run a BUSCO analysis. When the config parameter `offline` is set, no attempt to download is made. When the config parameter `auto_update_files` is set, new versions replace old versions.s Else, a warning is produced. Nc Cs¬| dd”|_| dd”|_| dd”|_| dd”|_| ”t|ƒjdkrØ|jsØz |  ”WnFt k r¦}z(d|_t   d  |j|j””|  ”W5d}~XYnXdS) zz :param config: Values of all parameters to be used during the analysis :type config: BuscoConfig Z busco_runŚofflinez update-dataŚdownload_base_urlZ download_pathNz&https://busco-data2.ezlab.org/v5/data/z{}. Retrying with backup url {})Z getbooleanrŚ update_dataŚgetr Ślocal_download_pathŚ_create_main_download_dirŚtypeŚ version_filesŚ_obtain_versions_filerŚloggerŚinfoŚformatŚvalue)ŚselfZconfigZbfe©rś9lib/python3.8/site-packages/busco/BuscoDownloadManager.pyŚ__init__)s" ’’zBuscoDownloadManager.__init__cCs"tj |j”stj|jdddS©NT)Śexist_ok)ŚosŚpathŚexistsr Śmakedirs)rrrrr ?sz.BuscoDownloadManager._create_main_download_dirz;Downloading information on latest versions of BUSCO data...c CsŽtj |jd”}tj |jd”}dD]“}z4tj ||”tj |ddddddgdd t |ƒ_ Wq$t k r~t d  |”ƒ‚Yq$tk rÖ|jr¤t d ”YqŚn.|rÄt d  |””t |”nt d  |”ƒ‚Yq$Xq$dS)Nzfile_versions.tsv©é édNś ZdatasetŚdateŚhashŚdomainr)ŚsepŚnamesZ index_colzPInvalid URL. Cannot reach {}. Please provide a valid URL for --base_download_urlz7Unable to verify BUSCO datasets because of offline modeś3Download connection problem. Retrying in {} secondsśCannot reach {})rrŚjoinr r ŚurllibŚrequestŚ urlretrieveŚpdZread_csvrrŚ ValueErrorrrrrrŚwarningrŚtimeŚsleep)rŚremote_filepathŚlocal_filepathŚtsleeprrrrDs@ ü’’’’’ z*BuscoDownloadManager._obtain_versions_filecCs.tj |j|”}tj |”s*tj|dddSr)rrr*r rr)rŚcategoryZcategory_folderrrrŚ_create_category_dirjs z)BuscoDownloadManager._create_category_dirc CsZd}t|dƒ6}|D]*}| ” d”}|ddkr|d}q@qW5QRX|sVtdƒ‚|S)NŚrś=rZ creation_dateéz;Creation date could not be extracted from dataset.cfg file.)ŚopenŚstripŚsplitr)Zdataset_config_fileŚ dataset_dateZ data_configŚlinerrrŚ_extract_creation_daters  ’z+BuscoDownloadManager._extract_creation_datec Cs2zt|ƒjj|d}Wn$tk r<td ||”ƒ‚YnXtj |”\}}|dkrØd  ||g”}z$|  tj  |d””}||k} d} Wnt k r¤d} d} YnXnhd  |||  d”g”}|  ||”}tj |”} tj |”\} } tt d | d d …| d d…””ƒd k} t|ƒjj|d } | | ||| fS) Nr#z!{} is not a valid option for '{}'ŚlineagesŚ.z dataset.cfgTFz{}.*.{}riõ’’’r:r$)rrŚlocŚKeyErrorrrrrŚsplitextr*r@ŚFileNotFoundErrorŚlstripŚreplacerŚlenŚglob)rr4r6Ś data_basenameZ latest_updateZ path_basenameŚ extensionŚlatest_versionr>Ś up_to_dateŚpresentZ path_to_checkZextension_to_checkr$rrrŚ_check_existing_versionsD ’  ’ ’  ’’ū’ z,BuscoDownloadManager._check_existing_versionc Csōtj |”r*d|kr*t d ||””|Sd|kr@td |”ƒ‚|jrŲ|dkr€tj |j ||”}tj |”rp|Std |”ƒ‚nXtj  |”\}}t t   tj |j |d ||”””ƒ}t |ƒdkrČ|dStd  |j ”ƒ‚tj |”}tj |j ||”}| |||”\} } } }} | s|js"| sŚ| |”d } tj |j|| | ”}| r`|dkr`| |”d D]j}| ||| | ”}|r¬| || ”}| r¤t d  |””qšn |rdt d  |””t |”qdtdƒ‚n| sšt d |””|S)Nś/zUsing local {} directory {}z{} does not existrAz?Unable to run BUSCO in offline mode. Dataset {} does not exist.z{}.*{}ré’’’’zWUnable to run BUSCO placer in offline mode. Cannot find necessary placement files in {}z.tar.gzrz0The file or folder {} was updated automatically.r(zVFile download unsuccessful. Check the download url or consider running in offline modez˜The file or folder {} is not the last available version. To update all data files to the last version, add the parameter --update-data in your next run.)rrrrrrrrr*r rEŚsortedrJrIŚbasenamerPr r7r Ś_rename_old_versionŚ_download_fileŚ_decompress_filer0r1r2)rZ data_namer6Z local_datasetrTrLZplacement_filesrKr4rOrNrMr$Zcompression_extensionr3r5Zdownload_successrrrr Ŗs²’ ’’ ż’’ ’’  ś ’ ’’’’’’’ž’zBuscoDownloadManager.getc Cs˜tj |”r”z(t |d |””t d ||””Wn^tk r’z4t ”}t |d ||””t d |||””Wntk rŒYnXYnXdS)Nz{}.oldzRenaming {} into {}.oldz {}.old.{}zRenaming {} into {}.old.{}) rrrŚrenamerrrŚOSErrorr1)r4Z timestamprrrrUs.  ’ ’’’ z(BuscoDownloadManager._rename_old_versionzDownloading file {}r:)Zfunc_argcCs¢zvtj ||”t|ƒ |”}||krdt d t|ƒt|ƒ””t  d |””t   |”t dƒ‚nt  d |””Wn&tk rœt d |””YdSXdS)Nz+md5 hash is incorrect: {} while {} expectedzdeleting corrupted file {}z:BUSCO was unable to download or update all necessary fileszmd5 hash is {}r)FT)r+r,r-rŚ_md5rŚerrorrŚstrrrŚremoverŚdebugr)rr3r4Z expected_hashZ observed_hashrrrrVs(’’ ’z#BuscoDownloadManager._download_filec sHt ”}t|dƒ(‰t‡fdd„dƒD]}| |”q&W5QRX| ”S)NŚrbcs ˆ d”S)Ni)Śreadr©ŚfrrŚ8óz+BuscoDownloadManager._md5..rd)ŚhashlibZmd5r;ŚiterŚupdateZ hexdigest)ŚfnameZhash_md5ŚchunkrrarrZ4s  zBuscoDownloadManager._md5zDecompressing file {}c Cs| dd”}tj |”ddkrœt tj |”d”0}t|dƒ}|D]}| |”qFW5QRXW5QRXzt |”Wnt k r–t   d”YnX|}tj |”ddkr| dd”}t  |”}|  tj |””W5QRXzt |”Wn t k rt   d”YnX|}|S) Nz.gzŚr:r_ŚwbzGDownloaded gzip file was removed before this BUSCO run could remove it.z.tarzEDownloaded tarball was removed before this BUSCO run could remove it.)rHrrrEŚgzipr;r*Śwriter]rFrr0ŚtarfileŚ extractallŚdirname)rr4Zunzipped_filenameZcompressed_fileZdecompressed_filer?Zuntarred_filenameZtar_filerrrrW<s2   ’   ’ z%BuscoDownloadManager._decompress_file)Ś__name__Ś __module__Ś __qualname__Ś__doc__rrr Ślogrrr7Ś staticmethodr@rPr rUrVrZrWrrrrrs& % )\     r)rtrr1rJrnreZurllib.requestr+Z urllib.errorrrlZpandasr.Zbusco.BuscoLoggerrrruZbusco.ExceptionsrrZ get_loggerrqrrrrrrŚs