U e@c@sd!ddZd"ddZd#ddZd$d d Zd%d d Zd&d dZd'ddZd(ddZd)ddZd*ddZ d+ddZ d,ddZ d-ddZ ddZ d.dd ZdS)/NcCsjddl}ddl}ddl}|dkr4|}|j|}|j||||dd}|j|\}}}}t |S)aFetch the unique tags within a context Parameters ---------- context : str The context to obtain tags from. Returns ------- set The set of sample identifers within a context. Raises ------ ValueError If the requested context is not known. Redis Command Summary --------------------- SMEMBERS :samples-represented NSMEMBERSsamples-represented redbiomredbiom._requests redbiom.util get_config _requestsmake_getvalidutilpartition_samples_by_tagsset)contextgetrconfigobs_tagsrl/mounts/lovelace/software/anaconda3/envs/qiime2-amplicon-2023.9/lib/python3.8/site-packages/redbiom/fetch.pytags_in_contexts  rcCszddl}ddl}ddl}|dkr4|}|j|}|j||||dd}|sn|j|\}}}}t |St |SdS)aKFetch samples in a context Parameters ---------- context : str The context to obtain samples from. unambiguous : bool If True, return unambiguous identifiers, if false return ambiguated identifiers. get : a make_get instance, optional A constructed get method. Returns ------- set The set of sample identifers within a context. Raises ------ ValueError If the requested context is not known. Redis Command Summary --------------------- SMEMBERS :samples-represented rNrrr)rZ unambiguousrrrrr tagged_cleanrrrsamples_in_context(s  rcCsVddl}ddl}ddl}|dkr4|}|j|}|j||||dd}t|S)aFeatures in a context Parameters ---------- context : str The context to obtain samples from. get : a make_get instance, optional A constructed get method. Returns ------- set The set of features within a context. Raises ------ ValueError If the requested context is not known. Redis Command Summary --------------------- SMEMBERS :features-represented rNrzfeatures-represented)rrrrr r r r)rrrrrrrrfeatures_in_contextVs  rTFc"Csddl}ddlm}ddl}ddl}ddl}|} |j| } |j |\} } } } | | }i}|dk r|j ||| \} } }}|rdd|D}dd|D}qdd| D}ndd|D}|st d g}g}|jjt|d d d | d d}|D]@\}}|||D](}|dk r||}|t|qqt|d}|ddD],}|dk sf|sr||}n ||}qR|dk rt||stdt|t||}|t}|D]"}||D]}|||d<qΐq|D]8}t||D]&\}}||D]}||||<qqq||j}|dk rxg}|dD](}|dd\} }!|d|!| fqF||d<||fS)aLFetch metadata for the corresponding samples Parameters ---------- samples : iterable of str The samples to obtain the metadata for. common : bool, optional If True (default), only the columns of the metadata common across all samples is returned. If False, all columns for all samples are returned. If value is missing for a given column and sample, None is stored in its place in the resulting DataFrame. context : str, optional If provided, resolve possible ambiguities in the sample identifiers relative to a context. restrict_to : Iterable of str, optional Restrict the retrieval of metadata to a subset of columns. If this parameter is specified, it will override the use of `common`. tagged : bool, optional Retrieve tagged metadata (e.g., preparation information). Returns ------- pandas.DataFrame A DataFrame indexed by the sample IDs, with the sample metadata dict ambiguous associations {sample_id: [tagged_sample_ids]} Raises ------ KeyError If a key in `restrict_to` is not found. Redis command summary --------------------- MGET metadata:categories: ... metadata:categories: HMGET metadata:category: ... rN defaultdictcSsi|] }||gqSrr.0Zrbidrrr sz#sample_metadata..cSsi|] }||qSrrrrrrrscSs"i|]\}}||dddqS)r)split)rkvrrrrscSsi|] }||gqSrr)rr"rrrrsz-None of the samples were found in the context categoriesMGETmetadatar buffer_sizer z+The following columns were not observed: %s #SampleIDrz%s.%s)json collectionsrpandasrrrr r r r resolve_ambiguitiesitems ValueErrorbufferedlistextendloadsappendrunion intersectionissubsetKeyErrordictget_sample_values DataFrameTr!)"samplescommonr restrict_toZtaggedr+rpdrrruntaggedrr ambig_map ambig_assocZrbid_mapZ all_columnsZ all_samplesgetterZcolumns_by_sample column_setZcolumns_to_getcolumnsr&sampleZsample_ambiguitycategoryvaluemdZnew_idsitagid_rrrsample_metadata}s|'                rOcCsbddl}ddl}ddl}|}|j|}|j|||j||d|g}t |t |||dS)aFetch sample data from an iterable of features. Parameters ---------- context : str The name of the context to retrieve sample data from. features : Iterable of str The features of interest. exact : bool If True, only samples in which all features exist are obtained. Otherwise, all samples with at least one feature are obtained. skip_taxonomy : bool, optional If true, do not resolve taxonomy. This greatly reduces fetch time. Default is false. Returns ------- biom.Table A Table populated with the found samples. dict A map of {sample_id_in_table: original_id}. This map can be used to identify what samples are ambiguous based off their original IDs. rNfeature)r skip_taxonomy) rrrrr r r r Zids_from_biom_from_samplesiter)rfeaturesexactrQrrrr>rrrdata_from_featuress  rVcCst|||dS)a`Fetch sample data from an iterable of samples. Paramters --------- context : str The name of the context to retrieve sample data from. samples : Iterable of str The samples of interest. skip_taxonomy : bool, optional If true, do not resolve taxonomy. This greatly reduces fetch time. Default is false. Returns ------- biom.Table A Table populated with the found samples. dict A map of {sample_id_in_table: original_id}. This map can be used to identify what samples are ambiguous based off their original IDs. )rQ)rR)rr>rQrrrdata_from_samplessrWc$Csddlm}ddlm}ddl}ddl}ddl}ddl}ddl}| } |dkr\|j | }|j | } |j ||t|}|j|||\} } } }g}t}|jjd}|D]*}| |d||}|||f||qddt|D}ddt||d d D}d d|D}|t|t|f}t|D]2\}\}}|D]\}}|||||f<qNq:|rzd}nt||||d }|dk rd d|D}nd}|||||}||i} |D]"\}!}"|!dd \}#}|| |"<q|| fS)aCreate a BIOM table from an iterable of samples Parameters ---------- context : str The context to obtain sample data from. samples : iterable of str The samples to fetch. get : a make_get instance, optional A constructed get method. normalize_taxonomy : list, optional The ranks to normalize a lineage too (e.g., [k, p, c, o, f, g, s]) skip_taxonomy : bool, optional If true, do not resolve taxonomy. This greatly reduces fetch time. Default is false. Returns ------- biom.Table A Table populated with the found samples. dict A map of {sample_id_in_table: original_id}. This map can be used to identify what samples are ambiguous based off their original IDs. Redis command summary --------------------- HMGET :feature-index-inverted EVALSHA 0 context r) itemgetterNz fetch-samplecSsi|]\}}||qSrr)rindexZobservedrrrrusz&_biom_from_samples..cSsg|] \}}|qSrrrrNrrrr ysz&_biom_from_samples..r )keycSsg|] \}}|qSrrrZrrrr[{s) normalizecSsg|] }d|iqS)Ztaxonomyr)rlineagerrrr[sr) operatorrXZ scipy.sparsesparsebiomZ redbiom.adminrrrrr r Zmake_script_execr r2r r.rZadminZ ScriptManagerrr5update enumeratesortedr/Z lil_matrixlentaxon_ancestorsTable update_idsr!)$rr>rZnormalize_taxonomyrQrXssrarrseZ stable_idsZ unobservedrDZrimapZ table_dataZunique_indicesZ fetch_samplerNdataZunique_indices_mapZobs_ids sample_idsmatcolrHZcol_dataZobs_idrJlineagesZobs_mdtableZ ambiguity_mapr"r#rMrrrrR5s`          rRc Csfddlm}ddl}|dkr8ddl}|}|j|}|jj}|t|dd||ddd}dd |D} t | } i} | rd } |t| dd||d| d} t }| D]2}t |D]$\}}|dkrq|| |<| |qq|} qx| sdSg}|D]r}g}| |}|dk r&| |}|dk r||q|ddd }|dk rVd d |||ddD}||q|S)a4Fetch the taxonomy information for a set of IDs Parameters ---------- context : str The context to operate in ids : list or tuple of str The IDs to retreive get : function, optional A get method normalize : list, optional The ranks to normalize a lineage too (e.g., [k, p, c, o, f, g, s]) Returns ------- list of list The lineage information for each ID in order with ids Redis Command Summary --------------------- HMGET :taxonomy-parents ... r) zip_longestNHMGETr'z feature-indexrr)ZmultikeycSs6i|].\}}t||D]\}}||dk r,|n|qqSNzip)rnamesidxnamerNrrrrs  z#taxon_ancestors..ztaxonomy-parentscSs g|]\}}|r|nd|qS)z%s__r)rZlinZrnkrrrr[sz#taxon_ancestors..F) fillvalue)Zfuture.moves.itertoolsrqrrrr r r1rSr2valuesrrvaddrr5)ridsrr]rqrrhmgetter remapped_bulkremappedto_getZ child_parentr\rE new_to_getblockchildparentrorNr^currentrrrrfsh         rfc sddl}|dkr,ddl}|}|j|}|jj}dfg}t}|rg}|D]X\} dkr||dd| } |t| qP||dd} |fdd| DqP|}qD||dd ||d d d } d d| D} d| krt d| S)aGet tips associated with a taxon Parameters ---------- context : str The context to operate in taxon : str The taxon to search for get : function, optional A get method Returns ------- set The set of feature IDs found Redis Command Summary --------------------- SMEMBERS :taxonomy-children: rNz has-terminalrzterminal-of:%sztaxonomy-children:%scsg|] }|fqSrr)rrtaxonrrr[sz%taxon_descendents..rrr'zfeature-index-invertedrscSs(h|] \}}t||D] \}}|qqSrru)rrxrwrNryrrr s z$taxon_descendents..z$An unassociated index has been found) rrrr r r1rrbr3r0)rrrrrrrto_keeprrZtipsZgottenrrrrrtaxon_descendentss<    rc Csddl}ddl}ddl}|j|}|dk rL|j|\}}}}||}t|||d}dd|D} dd|D} |j | | dS)aZObtain the samples and their corresponding category values Parameters ---------- category : str A metadata column of interest. samples : Iterable of str, optional If provided, only the specified samples and their values are obtained. Returns ------- pandas.Series A Series indexed by the Sample ID and valued by the metadata value for that sample for the specified category. Redis command summary --------------------- HGETALL metadata:category: HMGET metadata:category: ... rN)rcss|]}|dVqdS)rNrrr#rrr Msz)category_sample_values..css|]}|dVqdS)r NrrrrrrNs)rkrY) rrr-r r rr r r;Series) rIr>rrArrBrrZ keys_valsrYrkrrrcategory_sample_values+s rcCszddl}ddl}ddl}|j|}|dkr@t|ddd}g}|D]"}d|}|t|dd|qH|j ||dS) a(Get the number of samples with usable metadata per category Paramaters ---------- categories : list of str, optional The specific categories to summarize. If None, then summarize all of the categories in the database. Returns ------- pandas.Series A series keyed by the category and valued by the number of samples which have metadata for that category. Redis command summary --------------------- SMEMBERS metadata:categories-represented HLEN metadata:category: rNr&rcategories-represented category:%sZHLENrY) rrr-r r rr2r5intr)r$rrArresultsrIr\rrrsample_counts_per_categoryRsrcsddl}ddlm}ddl}ddl}ddl}|dkr ... metadata:categories: HMGET metadata:category: ... rNrzrestrict_to must be setr&rrz8The following requested categories are not not found: %s,rcSsh|]}d|kr|qS)rrrsrrrrszmetadata..csh|]}|dr|qS)z%s_) startswithrrMrrrsr$r%r'r(r*)where)r+r,rr-rrr0rr r rr8r9joinr1rvr4r7r5r:r;r<r=rerGr&ZMetadataZ set_indexr~)rrMr@r+rrArrrr$diffr>rEZsamples_to_getchunkrHrFr&rIrJrKrrrr&ws\        r&c Cslddl}|dkr$|}|j|}d|}|dkr@|dd|}|jjt|ddd|d|d}d d |DS) a Obtain the metadata values associated with the requested samples Parameters ---------- samples : Iterable of str or None The samples to obtain category : str The category to obtain values for. get : function, optional A get method Returns ------- [(str, str), ...] A list of (sample, value) tuples Redis command summary --------------------- HMGET metadata:category: ... HMKEYS metadata:category: rNrr&ZHKEYSrrr'rscSsg|]}t|D]}|qqSrru)rritemrrrr[s z%get_sample_values..)rrr r r1rS)r>rIrrrr\rErrrr;s  r;csttt|krtdfdd}|j|ddd}t}g}D]$\}}||krJ||||qJ|S)aMerge ambiguous samples Parameters ---------- table : biom.Table The table obtained from redbiom collapse_map : dict A mapping of a sample ID in the table to its collapse target name. Raises ------ ValueError If the IDs present in the table are not a perfect match to the keys of the collapse map. Returns ------- biom.Table A table of the merged data with updated sample identifiers IDs are inconsistentcs|Srtr)rLm collapse_maprr collapsersz#_ambiguity_merge..collapserrHF)ZaxisZnorm)rr~r0Zcollapser/r5r})rprrZcollapsed_tableseenZkeepr"r#rrr_ambiguity_merges   rcCsddl}t|t|kr$td|j|d|d}i}|D]&\}}||krdg||<|||qLg}|D]h\} } t | dkr| d} || } | ddD]} || }|| kr| } |} q|| q|| dq|j t|dd }|s|j |d d|S) aKeep the ambiguous sample with the most reads Parameters ---------- table : biom.Table The table obtained from redbiom ambig_map : dict A mapping of a sample ID in the table to its ambiguous form. retain_artifact_id : boolean, default False If True, do not strip the artifact ID Returns ------- biom.Table A table of the most volumous data with updated sample identifiers rNrrHrr F)ZinplaceT) r-rr~r0rsumto_dictr/r5refilterZ remove_emptyrh)rprCZretain_artifact_idrAZ sample_countsZambigsr"r#rZ sample_namerlbestZbest_cntrLZcntZ subset_tablerrr_ambiguity_keep_most_reads s2  r)N)N)N)TNNF)F)F)NNF)NN)N)N)N)NNN)N)F)rrrrOrVrWrRrfrrrr&r;rrrrrrs$ ' . ' x (  d X : ' % X )(