U WcW @snddlZddlZddlZddlmZmZddlmZddl m Z m Z ddl Z ddl mZmZdZdZddd d d d gZd ZdZdZdZeeeeeedZdZddhZddhZddddhZddddddd d!d"d#dddh Zd$d%Z d&d'Z!d(d)Z"d*d+Z#d,d-Z$d.d/Z%d0d1Z&d2d3Z'd4d5Z(dId8d9Z)d:d;Z*Gdd?Z-d@dAZ.dJdCdDZ/dKdEdFZ0dGdHZ1dS)LN)BiomParseExceptionUnknownAxisError)Table) biom_open __version__) defaultdict OrderedDictzJustin Kuczynskiz5Copyright 2011-2020, The BIOM Format Development TeamzDaniel McDonaldz Greg CaporasozJose Carlos Clemente LitranzAdam Robbins-PiankazJose Antonio Navas MolinaZBSDzhttp://biom-format.orgzdaniel.mcdonald@colorado.edu)intfloatunicoder r r "[{]}   ,0123456789cCs|d|}|dkrdS|t|d}|}||tkrD|d7}q.||tkrh||dkrf|d7}qPn||g}|d7}|r ||}|tkr|dtkr|n ||nN|tkrz |Wn"tk r|d8}Yq YnXn|tkr|||d7}qz|||S)z|Returns key:value from the biom string, or "" This method pulls an arbitrary key/value pair out from a BIOM string z"%s":)rrr) findlen JSON_START JSON_OPENQUOTEpopappend JSON_CLOSE IndexError)biom_strkeyZbase_idxZ start_idxZcur_idxstackZcur_charr/)lib/python3.8/site-packages/biom/parse.pydirect_parse_key9s8             r1c Csr|dkrtdt|d}|dkr*tdt|d}|dkrDtdt|d}|dkr^td|dd d dd d}ttt|d \}}|d d } || t |d }t |dkrtdd} |dkrt ||krtd| t ||f} n0|dkr,t ||krtd| |t |f} t |}g} |dkrNt ||} n|dkrbt||} d| d| S)a0Pull out specific slices from a BIOM string biom_str : JSON-formatted BIOM string to_keep : indices to keep axis : either 'samples' or 'observations' Will raise IndexError if the inices are out of bounds. Fully zerod rows or columns are possible and this is _not_ checked.  observationsamplezUnknown axis typeshaper .biom_str does not appear to be in BIOM format!dataZ matrix_type:rr rrr"rz'Observations to keep are out of bounds!z[%d, %d]r3r4z"Samples to keep are out of bounds!z"data": z , "shape": )r+r1 ValueErrorsplitreplacelistmapr r#r$minmaxset_direct_slice_data_sparse_obs_direct_slice_data_sparse_samp) r,to_keepaxisZ shape_kv_pairZ data_fieldsZmatrix_type_kv_pairZ raw_shapeZn_rowsZn_colsZ data_startZ new_shapenew_datar/r/r0direct_slice_datagsB           rFcCs |dS)Nz[] stripxr/r/r0strip_fsrKcCs2ttt|d\}}}||d|d|S)zRemap a sparse observation axisrr<r=rKr:rcvlookuprowcolvaluer/r/r0_remap_axis_sparse_obssrScCs2ttt|d\}}}|d||d|S)zRemap a sparse sample axisrrLrMr/r/r0_remap_axis_sparse_sampsrTcCsdg}ddtt|D}|dD]0}t|d\}}}||kr$|t||q$dd|S)zislice observations from data data : raw data string from a biom file to_keep : rows to keep cSsi|]\}}t||qSr/str.0ivr/r/r0 sz1_direct_slice_data_sparse_obs..],r[[%s]]],[) enumeratesortedr:rKr)rSjoinr7rCrEZ remap_lookuprNrcrZr/r/r0rAsrAcCs`g}ddtt|D}|dD],}|d\}}}||kr$|t||q$dd|S)zgslice samples from data data : raw data string from a biom file to_keep : columns to keep cSsi|]\}}t||qSr/rUrWr/r/r0r[sz2_direct_slice_data_sparse_samp..r\rr]r^)r_r`r:r)rTrarbr/r/r0rBsrBc st|dkr d}t||}n |dkr8d}t||}ntd|dkrPtdtd|}d d ||D}|std fd d t||D}t|}|gi}t||D]\} } | |kr||| q|t |ddfS)zReturns the indices for the associated ids to keep biom_str : a BIOM formatted JSON string to_keep : a list of IDs to get indices for axis : either 'samples' or 'observations' Raises KeyError if unknown key is specified r3Zrowsr4columnsz Unknown axis!r r6z{%s}cSsh|] }|dqSidr/)rXrZr/r/r0 sz#get_axis_indices..z+Not all of the to_keep ids are in biom_str!cs g|]\}}|dkr|qSrfr/rWrCr/r0 s z$get_axis_indices..r"r) r@r1r9jsonloadsissubsetKeyErrorr_r)dumps) r,rCrDZaxis_keyZ axis_dataZall_idsZidxsZ idxs_lookupZsubsetrYrZr/rir0get_axis_indicess*    rpc CsHtt}i}g}i}g}td}|D]}|}|s8q$|d}|d} | |krTq$|dd} |dd} | dkr| } | |kr|| } nt|} || | || <| dks| dkr$z| d } Wntk rtd YnX| d | }||kr||}nt|}|||<|||| |fd 7<q$q$t |||d S)a Create a Table object from a uclust/usearch/vsearch uc file. Parameters ---------- fh : file handle The ``.uc`` file to be parsed. Returns ------- biom.Table : The resulting BIOM table. Raises ------ ValueError If a sequence identifier is encountered that doesn't have at least one underscore in it (see Notes). Notes ----- This function assumes sequence identifiers in this file are in QIIME's "post-split-libraries" format, where the identifiers are of the form ``_``. Everything before the first underscore will be used as the sample identifier in the resulting ``Table``. The information after the first underscore is not used directly, though the full identifiers of seeds will be used as the observation identifier in the resulting ``Table``. ZHSLrr *HS_zA query sequence was encountered that does not have an underscore. An underscore is required in all query sequence identifiers to indicate the sample identifier.Nr")observation_ids sample_ids) rr r@rHr:r$r)rindexr9r)Zfhr7Z sample_idxsrxZobservation_idxsrwZ line_typeslineZfieldsZ line_typeZobservation_idZquery_idZobservation_idxZunderscore_indexZ sample_idZ sample_idxr/r/r0parse_ucsN         r{r4Fc s|dkrt|ztj||dWStk r6Yntk rHYnXt|dr|}|d}|rz|d}qf|dkr| |tj t j |t d|d}n| |t|ddd d }ntt|tr"z"tj t jd |t d|d}Wn*tk rt|ddd d }YnXntj t j|t d|d}fd d}dd}dk r|j||d|dkrtdnd}|j||d|S)aParses the biom table stored in `file_obj` Parameters ---------- file_obj : file-like object, or list file-like object storing the BIOM table (tab-delimited or JSON), or a list of lines of the BIOM table in tab-delimited or JSON format ids : iterable The sample/observation ids of the samples/observations that we need to retrieve from the biom table axis : {'sample', 'observation'}, optional The axis to subset on input_is_dense : boolean Indicates if the BIOM table is dense or sparse. Valid only for JSON tables. Returns ------- Table The BIOM table stored at file_obj Raises ------ ValueError If `samples` and `observations` are provided. Notes ----- Subsetting from the BIOM table is only supported in one axis Examples -------- Parse a hdf5 biom table >>> from h5py import File # doctest: +SKIP >>> from biom.parse import parse_biom_table >>> f = File('rich_sparse_otu_table_hdf5.biom') # doctest: +SKIP >>> t = parse_biom_table(f) # doctest: +SKIP Parse a hdf5 biom table subsetting observations >>> from h5py import File # doctest: +SKIP >>> from biom.parse import parse_biom_table >>> f = File('rich_sparse_otu_table_hdf5.biom') # doctest: +SKIP >>> t = parse_biom_table(f, ids=["GG_OTU_1"], ... axis='observation') # doctest: +SKIP r2)idsrDreadr"r)Zobject_pairs_hook)input_is_denseNcSs|SNr/rIr/r/r0z"parse_biom_table..r cSs|Srr/rIr/r/r0rrcs|kSrr/)r7id_mdr|r/r0 subset_idssz$parse_biom_table..subset_idscSs t|Sr)npany)valsrrr/r/r0gt_zerosz!parse_biom_table..gt_zerorDr4r3)rrZ from_hdf5r9 RuntimeErrorhasattrtellr}isspaceseekZ from_jsonrkloadrfrom_tsv isinstancer<rlrafilter) Zfile_objr|rDr~Zold_posrdtrrr/rr0parse_biom_tableYsV/           rcCsDg}|dD]0}g}|dD]}||q ||q|S)N|;)r:r)rH)rJZcomplex_metadatayZsimple_metadataer/r/r0sc_pipe_separateds rcs*eZdZedddZfddZZS) MetadataMapTFNc Cst|dr4z t|}Wntk r2tdYnX|rP|rFdd}qfdd}n|r^dd}ndd}|dkrri}g}|p|g}g}|D]} || } | r|r| sq| d r| d d} |s| d }n || qtt || d } t | t |kr| d gt |t | || q|s6td |sDtddd|D} t | t t | krptdi} |D]p} i}t |d d| d dD]>\}}z|||||<Wntk r|||<YnXq|| | d<qx|| S)a,Parse mapping file that relates samples or observations to metadata. Format: header line with fields optionally other comment lines starting with # tab-delimited fields process_fns: a dictionary of functions to apply to metadata categories. the keys should be the column headings, and the values should be functions which take a single value. For example, if the values in a column called "taxonomy" should be split on semi-colons before being added as metadata, and all other columns should be left as-is, process_fns should be: {'taxonomy': lambda x: x.split(';')} Assumes the first column in the mapping file is the id. This method is ported from QIIME (http://www.qiime.org), previously named parse_mapping_file/parse_mapping_file_to_dict. QIIME is a GPL project, but we obtained permission from the authors of this method to port it to the BIOM Format project (and keep it under BIOM's BSD license). upperzAA string was passed that doesn't refer to an accessible filepath.cSs |ddSNr r )r;rIr/r/r0rKsz&MetadataMap.from_file..strip_fcSs|ddSrr;rHrIr/r/r0rKscSs|Srr/rIr/r/r0rKscSs|SrrGrIr/r/r0rKsN#r"rr z)No header line was found in mapping file.zNo data found in mapping file.cSsg|] }|dqS)rr/)rXrYr/r/r0rjsz)MetadataMap.from_file..z2First column values are not unique! Cannot be ids.r)ropenOSErrorrrH startswithr:r)r<r=r$extendr@ziprn)clslinesZ strip_quotesZsuppress_strippingheaderZ process_fnsrKZ mapping_dataZcommentsrzZtmp_lineZ first_colmappingrZ current_dkrZr/r/r0 from_files^         "zMetadataMap.from_filecst|dS)zAccepts dictionary mapping IDs to metadata. ``mapping`` should be a dictionary mapping an ID to a dictionary of metadata. For example: {'Sample1': {'Treatment': 'Fast'}, 'Sample2': {'Treatment': 'Control'}} N)super__init__)selfr __class__r/r0r/szMetadataMap.__init__)TFNN)__name__ __module__ __qualname__ classmethodrr __classcell__r/r/rr0rs  hrcCsdtS)zReturns a generated by stringzBIOM-Format %s)rr/r/r/r0 generatedby:srcKs tj||||f|}|tS)aConvert a contigency table to a biom table sample_mapping : dict of {'sample_id':metadata} or None obs_mapping : dict of {'obs_id':metadata} or None process_func: a function to transform observation metadata dtype : type of table data )rrZto_jsonr)Ztable_fZsample_mappingZ obs_mappingZ process_funckwargsZ otu_tabler/r/r0convert_table_to_biom?s rr8cst|tr|dSt|trtddgt|dtr~g}|D]&}fdd|D}|d|qLd|Sdfd d |DSd S) zCDetermine which format the metadata is and then convert to a stringrz;|r rcsg|]}|qSr/)rHZtrans)rXr)transtabr/r0rj\sz'biom_meta_to_string..z; rc3s|]}|dVqdS)rNr)rXrJ) replace_strr/r0 csz&biom_meta_to_string..N)rrVr;r<bytes maketransrar))metadatarZ new_metadatarJvaluesr/)rrr0biom_meta_to_stringMs    rcCs\t|}|dkrt}|jdddkr,|S||jdddkrP|j|||dS|SdS)z*Convert a biom table to a contigency tableNr3rr) header_key header_valueZmetadata_formatter) load_tablerrZdelimited_self)Zbiom_frrZ md_formattabler/r/r0convert_biom_to_tablehsrc Cst|tjtjfrFz t|}Wqttfk rBtd|YqXnFt|8}z t|}Wn$ttfk rtd|YnXW5QRX|S)a\Load a `Table` from a path Parameters ---------- f : str or file-like object The entity to parse Returns ------- Table Raises ------ IOError If the path does not exist TypeError If the data in the path does not appear to be a BIOM table Examples -------- Parse a table from a path. BIOM will attempt to determine if the fhe file is either in TSV, HDF5, JSON, gzip'd JSON or gzip'd TSV and parse accordingly: >>> from biom import load_table >>> table = load_table('path/to/table.biom') # doctest: +SKIP z%%s does not appear to be a BIOM file!) rioIOBaseh5pyZFilerr+ TypeErrorr)frfpr/r/r0rzs   r)Nr4F)r8)NNN)2ZnumpyrrrZbiom.exceptionrrZ biom.tablerZ biom.utilrrrk collectionsrr __author__Z __copyright__ __credits__Z __license__Z__url__Z__maintainer__Z __email__r r rVZMATRIX_ELEMENT_TYPEr'r&r*Z JSON_SKIPr%r1rFrKrSrTrArBrpr{rrdictrrrrrrr/r/r/r0 sx  .>'\ a v