чoe8dZddlZddlZddlZddlmZddlmZddl m Z m Z ddl mZmZddlmZmZddlmZdd lmZmZdd lmZmZmZmZdd lmZmZmZm Z m!Z!m"Z"m#Z#ddl$Z%ddl&Z&dd l'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4ddl5m6Z6ddl7m7Z7ddl8m8Z8ddl9m9Z9dZ:dZ;gdZdZ?dZ@eAeBeCeAeBeCdZDGdde ZEee eEZ dZFdZGdZHd ZId!ZJGd"d#ZKejLdfd$ZMeBdfd%ZNeBfd&ZOeBfd'ZPeBfd(ZQeBfd)ZReBdfd*ZSdS)+aj BIOM Table (:mod:`biom.table`) ============================== The biom-format project provides rich ``Table`` objects to support use of the BIOM file format. The objects encapsulate matrix data (such as OTU counts) and abstract the interaction away from the programmer. .. currentmodule:: biom.table Classes ------- .. autosummary:: :toctree: generated/ Table Examples -------- First, let's create a toy table to play around with. For this example, we're going to construct a 10x4 `Table`, or one that has 10 observations and 4 samples. Each observation and sample will be given an arbitrary but unique name. We'll also add on some metadata. >>> import numpy as np >>> from biom.table import Table >>> data = np.arange(40).reshape(10, 4) >>> sample_ids = ['S%d' % i for i in range(4)] >>> observ_ids = ['O%d' % i for i in range(10)] >>> sample_metadata = [{'environment': 'A'}, {'environment': 'B'}, ... {'environment': 'A'}, {'environment': 'B'}] >>> observ_metadata = [{'taxonomy': ['Bacteria', 'Firmicutes']}, ... {'taxonomy': ['Bacteria', 'Firmicutes']}, ... {'taxonomy': ['Bacteria', 'Proteobacteria']}, ... {'taxonomy': ['Bacteria', 'Proteobacteria']}, ... {'taxonomy': ['Bacteria', 'Proteobacteria']}, ... {'taxonomy': ['Bacteria', 'Bacteroidetes']}, ... {'taxonomy': ['Bacteria', 'Bacteroidetes']}, ... {'taxonomy': ['Bacteria', 'Firmicutes']}, ... {'taxonomy': ['Bacteria', 'Firmicutes']}, ... {'taxonomy': ['Bacteria', 'Firmicutes']}] >>> table = Table(data, observ_ids, sample_ids, observ_metadata, ... sample_metadata, table_id='Example Table') Now that we have a table, let's explore it at a high level first. >>> table 10 x 4 with 39 nonzero entries (97% dense) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S0 S1 S2 S3 O0 0.0 1.0 2.0 3.0 O1 4.0 5.0 6.0 7.0 O2 8.0 9.0 10.0 11.0 O3 12.0 13.0 14.0 15.0 O4 16.0 17.0 18.0 19.0 O5 20.0 21.0 22.0 23.0 O6 24.0 25.0 26.0 27.0 O7 28.0 29.0 30.0 31.0 O8 32.0 33.0 34.0 35.0 O9 36.0 37.0 38.0 39.0 >>> print(table.ids()) # doctest: +NORMALIZE_WHITESPACE ['S0' 'S1' 'S2' 'S3'] >>> print(table.ids(axis='observation')) # doctest: +NORMALIZE_WHITESPACE ['O0' 'O1' 'O2' 'O3' 'O4' 'O5' 'O6' 'O7' 'O8' 'O9'] >>> print(table.nnz) # number of nonzero entries 39 While it's fun to just poke at the table, let's dig deeper. First, we're going to convert `table` into relative abundances (within each sample), and then filter `table` to just the samples associated with environment 'A'. The filtering gets fancy: we can pass in an arbitrary function to determine what samples we want to keep. This function must accept a sparse vector of values, the corresponding ID and the corresponding metadata, and should return ``True`` or ``False``, where ``True`` indicates that the vector should be retained. >>> normed = table.norm(axis='sample', inplace=False) >>> filter_f = lambda values, id_, md: md['environment'] == 'A' >>> env_a = normed.filter(filter_f, axis='sample', inplace=False) >>> print(env_a) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S0 S2 O0 0.0 0.01 O1 0.0222222222222 0.03 O2 0.0444444444444 0.05 O3 0.0666666666667 0.07 O4 0.0888888888889 0.09 O5 0.111111111111 0.11 O6 0.133333333333 0.13 O7 0.155555555556 0.15 O8 0.177777777778 0.17 O9 0.2 0.19 But, what if we wanted individual tables per environment? While we could just perform some fancy iteration, we can instead just rely on `Table.partition` for these operations. `partition`, like `filter`, accepts a function. However, the `partition` method only passes the corresponding ID and metadata to the function. The function should return what partition the data are a part of. Within this example, we're also going to sum up our tables over the partitioned samples. Please note that we're using the original table (ie, not normalized) here. >>> part_f = lambda id_, md: md['environment'] >>> env_tables = table.partition(part_f, axis='sample') >>> for partition, env_table in env_tables: ... print(partition, env_table.sum('sample')) A [ 180. 200.] B [ 190. 210.] For this last example, and to highlight a bit more functionality, we're going to first transform the table such that all multiples of three will be retained, while all non-multiples of three will get set to zero. Following this, we'll then collpase the table by taxonomy, and then convert the table into presence/absence data. First, let's setup the transform. We're going to define a function that takes the modulus of every value in the vector, and see if it is equal to zero. If it is equal to zero, we'll keep the value, otherwise we'll set the value to zero. >>> transform_f = lambda v,i,m: np.where(v % 3 == 0, v, 0) >>> mult_of_three = tform = table.transform(transform_f, inplace=False) >>> print(mult_of_three) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S0 S1 S2 S3 O0 0.0 0.0 0.0 3.0 O1 0.0 0.0 6.0 0.0 O2 0.0 9.0 0.0 0.0 O3 12.0 0.0 0.0 15.0 O4 0.0 0.0 18.0 0.0 O5 0.0 21.0 0.0 0.0 O6 24.0 0.0 0.0 27.0 O7 0.0 0.0 30.0 0.0 O8 0.0 33.0 0.0 0.0 O9 36.0 0.0 0.0 39.0 Next, we're going to collapse the table over the phylum level taxon. To do this, we're going to define a helper variable for the index position of the phylum (see the construction of the table above). Next, we're going to pass this to `Table.collapse`, and since we want to collapse over the observations, we'll need to specify 'observation' as the axis. >>> phylum_idx = 1 >>> collapse_f = lambda id_, md: '; '.join(md['taxonomy'][:phylum_idx + 1]) >>> collapsed = mult_of_three.collapse(collapse_f, axis='observation') >>> print(collapsed) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S0 S1 S2 S3 Bacteria; Firmicutes 7.2 6.6 7.2 8.4 Bacteria; Proteobacteria 4.0 3.0 6.0 5.0 Bacteria; Bacteroidetes 12.0 10.5 0.0 13.5 Finally, let's convert the table to presence/absence data. >>> pa = collapsed.pa() >>> print(pa) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S0 S1 S2 S3 Bacteria; Firmicutes 1.0 1.0 1.0 1.0 Bacteria; Proteobacteria 1.0 1.0 1.0 1.0 Bacteria; Bacteroidetes 1.0 1.0 0.0 1.0 N)deepcopy)datetime)dumps JSONEncoder)reducepartial) itemgetteror_) defaultdict)HashableIterable)ndarrayasarrayzerosnewaxis) coo_matrix csc_matrix csr_matrix isspmatrixvstackhstack dok_matrix)TableExceptionUnknownAxisErrorUnknownIDErrorDisjointIDError)get_biom_format_version_stringget_biom_format_url_stringflattennatsort prefer_self index_list H5PY_VLEN_STR__format_version__)errcheck)_filter) _transform) _subsampleDaniel McDonaldz5Copyright 2011-2020, The BIOM Format Development Team) r*zJai Ram Rideoutz Greg Caporasoz Jose ClementezJustin KuczynskizAdam Robbins-PiankazJoshua ShorensteinzJose Antonio Navas MolinauJorge Cañardo Alastueyz Steven BrownBSDhttp://biom-format.orgzdaniel.mcdonald@colorado.edu)intfloatunicoder-r.r/ceZdZfdZxZS) NpEncodercTt|tjrt|St|tjrt |St|tjr|Stt| |SN) isinstancenpintegerr-floatingr.rtolistsuperr1default)selfobj __class__s _/mounts/lovelace/software/anaconda3/envs/kraken-biom/lib/python3.11/site-packages/biom/table.pyr:zNpEncoder.defaults c2: & & s88O c2; ' ' ::  c2: & & ::<< Y%%--c222)__name__ __module__ __qualname__r: __classcell__)r=s@r>r1r1s8333333333r?r1)clscld}d}t|D]\}} ||#|}|}YnxYw||fS)aRIdentify the first value which cannot be cast Paramters --------- dtype : type A type to cast to fields : Iterable of str A series of str to cast into dtype Returns ------- str or None A value that cannot be cast int or None The index of the value that cannot be cast N) enumerate)dtypefieldsbadvalbadidxidxvs r>_identify_bad_valuerMsb"F FF##Q  E!HHHH FF EE F s &/cZt|tr|d}|SNutf8r4bytesdecodexs r>general_parserrVs*!U HHV   Hr?cg}|D]C}|r?t|tr|d}||D|r|ndS)zParses the taxonomy valuerPN)r4rRrSappend)value new_valuerLs r>vlen_list_of_str_parserr[ seI     !U## %HHV$$   Q   ! +99t+r?cJt|f}fd|D}dd}d|z}t|thr-|||t fd|D|dSt|tthrt|||dSg}g} t||D]b\} } | } | d} t} | tkr| d } | | | | ct| thrt } nd} ||t|f| ||dS) z4Creates a dataset for a general atomic type categoryc:g|]}t|Stype.0mheaders r> z%general_formatter..s# * * *!d1V9oo * * *r?/ @@SLASH@@ metadata/%scFg|]}|dSrPencoderas r>rez%general_formatter..&s+ F F Fa6!1!1&!9!9 F F Fr?shaperGdata compressionNrP) lenreplacesetissubsetstrcreate_datasetr#listtuplevlen_list_of_str_formatterziprlrX)grprdmdrprndtypes sanitizedname formatted dtypes_useddtrcval dtype_to_uses ` r>general_formatterrs WWJE * * * *r * * *F sK00I 9 $D 6{{SE"" % 4u!. F F F F2 F F F'2  4 4 4 4 4 V  tUm , ,%"3K@@@@@  __ # #EBF)C{Syyjj((   S ! ! !   r " " " " {   $ $cU + + (LLL  R #  % % % % %r?c g}g}|D]}|||d t||tr|d^|t||gt|t ||t j|s|dkrd} g}g}|D]L}|||} ||| i|t | M|}n8#td|d|d|xYwtd |zt|} t || f} t j | t } t|D]H\} }||t j ||}d |D| | dt |f<It j| t jdkd | } |d |z| t"| |dS)z!Creates a (N, ?) vlen str datasetNTFtaxonomycD|d}d|DS)N;c6g|]}|Sr^)strip)rbps r>rezGvlen_list_of_str_formatter..split_and_strip..^s 111a 111r?)split)ipartss r>split_and_stripz3vlen_list_of_str_formatter..split_and_strip\s% 1151111r?z Category 'a ' is not formatted properly. The most common issue is when 'taxonomy' is represented as a flat string instead of a list. An attempt was made to split this field on a ';' to coerce it into a list but it failed. An example entry (which is not assured to be the problematic entry) is below: rzCategory %s not formatted correctly. Did you pass --process-obs-metadata taxonomy when converting from tsv? Please see Table.to_hdf5 docstring for more informationrGc8g|]}|dSrjrkrbrLs r>rez.vlen_list_of_str_formatter..~s$@@@Q 0 0@@@r?rqrhrm)rXr4getrvr rrr5all TypeErrormaxemptyobjectrFrwherearrayrwr#)r|rdr}rpiterable_checkslengthsrcrnew_mdr max_list_lenrnrorrYs r>rzrzFs OG ++ V9   " "4 ( ( ( ( f s + + +  " "5 ) ) ) )  " "155,,h77 9 9 9 NN3qy>> * * * * 6/ " ". Z   2 2 2 H//A+OAfI66EMM65/222NN3u::.... Hi171f !GHHH$'--.. . w<>#E!cdeZdZdZ dedZdZdfdZdgd Zed Z ede ddfd Z d Z e d Ze dZe dZe dZdfdZdfdZdhdZdfdZdZdZdZdfdZdidZdZdjdZdZdkd Zdfd!Zdfd"Z dld#Z!dfd$Z"dmd%Z#d&Z$d'Z%d(Z&d)Z'dfd*Z(d+dde)d,dfd-Z*d.Z+d/Z,d0Z-d1Z.d2Z/d3Z0d4Z1d5Z2d6Z3dnd7Z4d8Z5dod9Z6dod:Z7dpd;Z8dfd<Z9e:dfd=Z;dqd>ZdDZ?dEZ@dfdFZAdfdGZB dsdHZCdtdIZDdndJZEdudLZFdndMZGdNZHdtdOZIdPZJdQZKdvdRZLdwdTZMdfdUZNdVZOdWdWePePfdXZQeR dxdYZSdydZZTdzd\ZUd]ZV d{d^ZWeR d|d_ZXd}d`ZYedaZZedbZ[ed+e dfdcZ\dde)d,dfddZ]dS)~Tablea The (canonically pronounced 'teh') Table. Give in to the power of the Table! Creates an in-memory representation of a BIOM file. BIOM version 1.0 is based on JSON to provide the overall structure for the format while versions 2.0 and 2.1 are based on HDF5. For more information see [1]_ and [2]_ Paramaters ---------- data : array_like An (N,M) sample by observation matrix represented as one of these types: * An 1-dimensional array of values * An n-dimensional array of values * An empty list * A list of numpy arrays * A list of dict * A list of sparse matrices * A dictionary of values * A list of lists * A sparse matrix of values observation_ids : array_like of str A (N,) dataset of the observation IDs, where N is the total number of IDs sample_ids : array_like of str A (M,) dataset of the sample IDs, where M is the total number of IDs observation_metadata : list of dicts, optional per observation dictionary of annotations where every key represents a metadata field that contains specific metadata information, ie taxonomy, KEGG pathway, etc sample_metadata : array_like of dicts, optional per sample dictionary of annotations where every key represents a metadata field that contains sample specific metadata information, ie table_id : str, optional A field that can be used to identify the table type : str, see notes The type of table represented create_date : str, optional Date that this table was built generated_by : str, optional Individual who built the table observation_group_metadata : list, optional group that contains observation specific group metadata information (e.g., phylogenetic tree) sample_group_metadata : list, optional group that contains sample specific group metadata information (e.g., relationships between samples) Attributes ---------- shape dtype nnz matrix_data type table_id create_date generated_by format_version Notes ----- Allowed table types are None, "OTU table", "Pathway table", "Function table", "Ortholog table", "Gene table", "Metabolite table", "Taxon table" Raises ------ TableException When an invalid table type is provided. References ---------- .. [1] http://biom-format.org/documentation/biom_format.html .. [2] D. McDonald, et al. "The Biological Observation Matrix (BIOM) format or: how I learned to stop worrying and love the ome-ome" GigaScience 2012 1:7 NTc ||_||_||_| |_t|_t |sWt|t|f}|dd}t ||||_ n| |_ |j t|_ tj||_tj||_|.d|Ddhkrd|_nt)||_nd|_|.d|Ddhkrd|_nt)||_nd|_| |_| |_| rt1|d|_d|_||| |dS)Ninput_is_denseF)rrnch|]}| Sr^r^rbrcs r> z!Table.__init__..s///!A///r?Tch|]}| Sr^r^rs r>rz!Table.__init__..s444!A444r?)r`table_id create_date generated_byr$format_versionrrrrr _to_sparse_datatocsrastyper.r5r _sample_ids_observation_ids_sample_metadatary_observation_metadata_sample_group_metadata_observation_group_metadatar% _sample_index _obs_index_cast_metadata _index_ids)r;roobservation_ids sample_idsobservation_metadatasample_metadatarr`rrobservation_group_metadatasample_group_metadatavalidateobservation_index sample_indexkwargsrnrs r>__init__zTable.__init__s   &(0$ &))3z??;E#ZZ(8%@@N))$~05*77DJJDJZ&&u-- :j11 " ? ; ;  &0////D8;;(,%%(-o(>(>%%$(D !  +543444@@-1**-23G-H-H**)-D &&;#+E(   TNNN"  )<88888r?c|t|j|_n||_|t|j|_dS||_dS)zpSets lookups {id:index in _data}. Should only be called in constructor as this modifies state. N)r"rrrr)r;rrs r>rzTable._index_idssO  !+D,??DOOO/DOOOr?samplecT|dkr|jS|dkr|jSt|)a}Return the index lookups of the given axis Parameters ---------- axis : {'sample', 'observation'}, optional Axis to get the index dict. Defaults to 'sample' Returns ------- dict lookups {id:index} Raises ------ UnknownAxisError If provided an unrecognized axis. r observation)rrrr;axiss r>_indexz Table._index"s9$ 8  % % ] " "? ""4(( (r?Fcn||j}t|r|St|||S)z0For converting vectors to a compatible self type)rGrrr)r;vals transposerGs r>_conv_to_self_typezTable._conv_to_self_type;s; =JE d   <K##D)U;; ;r?c|}|jdkr|dStj|S)zConverts a row/col vector to a dense numpy array. Always returns a 1-D row vector for consistency with numpy iteration over arrays. )r&r&r&)toarrayrnreshaper5squeeze)vec dense_vecs r> _to_densezTable._to_denseEsDKKMM 9  $$Q'' ':i(( (r?c.t|trLt|jdkr4|r t |ddt f|}nt ||}|St|tr*|rt |j|}nt ||}|St|tr"t|dkrtdSt|tr6t|dtrt||}|r|j}|St|tr6t|dtrt||}|r|j}|St|tr0t|drt||}|r|j}|St|trt|||}|r|j}|St|trft|dtrK|r5t|}t|j|j|jff||}nt'|||}|St|r|}|r|}|St+d)zTry to return a populated scipy.sparse matrix. NOTE: assumes the max value observed in row and col defines the size of the matrix. r&Nrrr)rGrnrnzUnknown input type)r4rrrrnnparray_to_sparserTrxrlist_nparray_to_sparsedictlist_dict_to_sparserlist_sparse_to_sparsedict_to_sparsecoo_arrays_to_sparserorowcollist_list_to_sparserr)valuesrrGrrnmatds r>rzTable._to_sparseVs fg & & 3v|+<+<+A+A 7'qqq'z(:EBB'66J fg & &. 7 7'%88'66J  % %' 7#f++*:*:f%% %  % %$ 7*VAY*H*H$ 7(77C eJ  % % 7*VAY*E*E 7%fe44C eJ  % % 7*VAY*?*? 7'66C eJ  % % 7 66C eJ  % % 7*VAY*E*E 7 Fv&&*AFQUAEN+C16eEEE*&%uEEEJ    7C &mmooJ !566 6r?cd}||j|_||j|_|jr|jnd|_|jr|jnd|_dS)zCasts all metadata to defaultdict to support default values. Should be called after any modifications to sample/observation metadata. cng}|(|dt|krdS||D]t}td}t|tr||n"|nt dt|z||ut|S|S)zDo the actual castingNcdSr3r^r^r?r>z=Table._cast_metadata..cast_metadata..sDr?zUnable to cast metadata: %s) countrrr r4rupdaterreprrXry)r} default_mditemrs r> cast_metadataz+Table._cast_metadata..cast_metadatasJ~88D>>SWW,,4~ ) )D#LL11A!$--9,-J-1$ZZ.8999%%a((((Z(((Ir?N)rrrr)r;rs r>rzTable._cast_metadatas    ,!. d.C D D%2]43M%N%N"* 5D ' '04 # / :D , ,59 (((r?c|jjS)z.The shape of the underlying contingency matrix)rrnr;s r>rnz Table.shapezr?c|jjS)zrGz Table.dtyperr?cL|j|jjS)z@Number of non-zero elements of the underlying contingency matrix)reliminate_zerosnnzrs r>rz Table.nnzs" ""$$$z~r?c|jS)zThe sparse matrix object)rrs r> matrix_datazTable.matrix_datas zr?ch|dvrt||dkr |jdn |jdS)aReturn the length of an axis Parameters ---------- axis : {'sample', 'observation'}, optional The axis to operate on Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> from biom import example_table >>> print(example_table.length(axis='sample')) 3 >>> print(example_table.length(axis='observation')) 2 rrrr&r)rrnrs r>lengthz Table.lengths>* 0 0 0"4(( ( $ 0 0tz!}}djmCr?c|dkr,|j|j|dS||_dS|dkr,|j|j|dS||_dSt|)aTake a dict of group metadata and add it to an axis Parameters ---------- group_md : dict of tuples `group_md` should be of the form ``{category: (data type, value)`` axis : {'sample', 'observation'}, optional The axis to operate on Raises ------ UnknownAxisError If provided an unrecognized axis. rNr)rrrr)r;group_mdrs r>add_group_metadatazTable.add_group_metadatas 8  *6+228<<<<<.6+++ ] " "/;077AAAAA3;000"4(( (r?wholec|dkrddg}n|dvr|g}ntd|z|,|dkrd|_d|_n|dkrd|_nd|_dS|D]}||t ||||D]\}}|D] }||vr||= d||D}|d hkr|dkrd|_d|_dS) aIRemove metadata from an axis Parameters ---------- keys : list of str, optional The keys to remove from metadata. If None, all keys from the axis are removed. axis : {'sample', 'observation', 'whole'}, optional The axis to operate on. If 'whole', the operation is applied to both the sample and observation axes. Raises ------ UnknownAxisError If the requested axis does not exist. Examples -------- >>> from biom import Table >>> import numpy as np >>> tab = Table(np.array([[1, 2], [3, 4]]), ... ['O1', 'O2'], ... ['S1', 'S2'], ... sample_metadata=[{'barcode': 'ATGC', 'env': 'A'}, ... {'barcode': 'GGTT', 'env': 'B'}]) >>> tab.del_metadata(keys=['env']) >>> for id, md in zip(tab.ids(), tab.metadata()): ... print(id, list(md.items())) S1 [('barcode', 'ATGC')] S2 [('barcode', 'GGTT')] rrrrz%s is not recognizedNrch|]}|sdnd S)TFr^)rbr}s r>rz%Table.del_metadata..As1999$&0tt5999r?T)rrrmetadatar{ids) r;keysraxesaxrr}kemptiess r> del_metadatazTable.del_metadatask@ 7??m,DD . . .6DD"#9D#@AA A <w(,%-1**!!(,%%-1* F 6 6B}}"}%%-TXX2X.. 2 0F0FGG " "2""ABwwqE" 99!%B!7!7999G4("">>,0D))15D.! 6 6r?c||}|dD]N\}}|||r2|||}|||Ons||}|dkr!t fd|D|_n6|dkr!t fd|D|_nt|| dS)aTake a dict of metadata and add it to an axis. Parameters ---------- md : dict of dict `md` should be of the form ``{id: {dict_of_metadata}}`` axis : {'sample', 'observation'}, optional The axis to operate on rNrc34K|]}|vr|ndVdSr3r^rbid_r}s r> z%Table.add_metadata..\sL.C.C7:sbyyBsGGd.C.C.C.C.C.Cr?rc34K|]}|vr|ndVdSr3r^rs r>rz%Table.add_metadata.._sL3C3C7:sbyyBsGGd3C3C3C3C3C3Cr?) ritemsexistsindexrrryrrrr)r;r}rrrmd_entryrKrs ` r> add_metadatazTable.add_metadataIs_==d=++  !# 3 3 X;;s;..3**St*44CSM((222 3 (((%%Cx(-.C.C.C.C>A.C.C.C)C)C%%&&-23C3C3C3C>A3C3C3C.C.C**'t,,, r?c|rtd |\}}n#tdxYwt|tr$t|trtdt|tr2|j|j||Stdt|tr2|j|j||Std|j dkr|j |_|j||fS)a?Handles row or column slices Slicing over an individual axis is supported, but slicing over both axes at the same time is not supported. Partial slices, such as `foo[0, 5:10]` are not supported, however full slices are supported, such as `foo[0, :]`. Parameters ---------- args : tuple or slice The specific element (by index position) to return or an entire row or column of the data. Returns ------- float or spmatrix A float is return if a specific element is specified, otherwise a spmatrix object representing a vector of sparse data is returned. Raises ------ IndexError - If the matrix is empty - If the arguments do not appear to be a tuple - If a slice on row and column is specified - If a partial slice is specified Notes ----- Switching between slicing rows and columns is inefficient. Slicing of rows requires a CSR representation, while slicing of columns requires a CSC representation, and transforms are performed on the data if the data are not in the required representation. These transforms can be expensive if done frequently. .. shownumpydoc z4Cannot retrieve an element from an empty/null table.zMust specify (row, col).zCan only slice a single axis.Nz'Can only handle full : slices per axis.coo) is_empty IndexErrorr4slicestartstop_get_col_get_rowr getformatr)r;argsrrs r> __getitem__zTable.__getitem__esIL ==?? '&'' ' 9HC 9788 8 c5 ! ! >je&<&< ><== = c5 ! ! (y SX%5}}S))) !JKKK U # # (y SX%5}}S))) !JKKKz##%%..!Z--// :c3h' 's+<cr|j|_|j|S)aReturn the row at ``row_idx``. A row vector will be returned as a scipy.sparse matrix in csr format. Notes ----- Switching between slicing rows and columns is inefficient. Slicing of rows requires a CSR representation, while slicing of columns requires a CSC representation, and transforms are performed on the data if the data are not in the required representation. These transforms can be expensive if done frequently. )rrgetrow)r;row_idxs r>r!zTable._get_rows/Z%%'' z  )))r?cr|j|_|j|S)aReturn the column at ``col_idx``. A column vector will be returned as a scipy.sparse matrix in csc format. Notes ----- Switching between slicing rows and columns is inefficient. Slicing of rows requires a CSR representation, while slicing of columns requires a CSC representation, and transforms are performed on the data if the data are not in the required representation. These transforms can be expensive if done frequently. )rtocscgetcol)r;col_idxs r>r zTable._get_cols/Z%%'' z  )))r?cXt||t|jz}t|dkrt d|||d}||j||}||fS)a8 Aligns dataframe against biom table, only keeping common ids. Parameters ---------- metadata : pd.DataFrame The metadata, either respect to the sample metadata or observation metadata. axis : {'sample', 'observation'} The axis on which to operate. Returns ------- biom.Table A filtered biom table. pd.DataFrame A filtered metadata table. Examples -------- >>> from biom import Table >>> import numpy as np >>> import pandas as pd >>> table = Table(np.array([[0, 0, 1, 1], ... [2, 2, 4, 4], ... [5, 5, 3, 3], ... [0, 0, 0, 1]]), ... ['o1', 'o2', 'o3', 'o4'], ... ['s1', 's2', 's3', 's4']) >>> metadata = pd.DataFrame([['a', 'control'], ... ['c', 'diseased'], ... ['b', 'control']], ... index=['s1', 's3', 's2'], ... columns=['Barcode', 'Treatment']) >>> res_table, res_metadata = table.align_to_dataframe(metadata) >>> print(res_table) # Constructed from biom file #OTU ID s1 s2 s3 o1 0.0 0.0 1.0 o2 2.0 2.0 4.0 o3 5.0 5.0 3.0 >>> print(res_metadata) Barcode Treatment s1 a control s2 b control s3 c diseased rrz*No common ids between table and dataframe.Frinplace)rtrrrrrfilter remove_emptyloc)r;rrrtr}s r>align_to_dataframezTable.align_to_dataframes^$(((%%&&X^)<)<< s88q== !MNN N KK$K 6 6  \!%%T%** +"u r?rcd|D}|t||z}t|dkrt d||}|||d}||d|D}| ||}||fS) al Aligns biom table against tree, only keeping common ids. Parameters ---------- tree : skbio.TreeNode The tree object, either respect to the sample metadata or observation metadata. axis : {'sample', 'observation'} The axis on which to operate. Returns ------- biom.Table A filtered biom table. skbio.TreeNode A filtered skbio TreeNode object. Examples -------- >>> from biom import Table >>> import numpy as np >>> from skbio import TreeNode >>> table = Table(np.array([[0, 0, 1, 1], ... [2, 2, 4, 4], ... [5, 5, 3, 3], ... [0, 0, 0, 1]]), ... ['o1', 'o2', 'o3', 'o4'], ... ['s1', 's2', 's3', 's4']) >>> tree = TreeNode.read([u"((o1,o2)f,o3)r;"]) >>> res_table, res_tree = table.align_tree(tree) >>> print(res_table) # Constructed from biom file #OTU ID s1 s2 s3 s4 o1 0.0 0.0 1.0 1.0 o2 2.0 2.0 4.0 4.0 o3 5.0 5.0 3.0 3.0 >>> print(res_tree.ascii_art()) /-o1 /f-------| -r-------| \-o2 | \-o3 ch|] }|j Sr^r)rbrUs r>rz#Table.align_tree...s,,,1,,,r?rrz%No common ids between table and tree.)namesFr-cg|] }|j Sr^r6)rbns r>rez$Table.align_tree..6s...A...r?) tipsrtrrrrshearr/r0prune sort_order)r;treerr: common_tips_tree_tableorders r> align_treezTable.align_treesX-, ,,,St!4!4555 {  q  !HII I  --UCC .....""5t"44u}r?c|rtdtfd||DS)aReduce over axis using function `f` Parameters ---------- f : function The function to use for the reduce operation axis : {'sample', 'observation'} The axis on which to operate Returns ------- numpy.array A one-dimensional array representing the reduced rows (observations) or columns (samples) of the data matrix Raises ------ UnknownAxisError If `axis` is neither "sample" nor "observation" TableException If the table's data matrix is empty Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 table >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... [{'foo': 'bar'}, {'x': 'y'}], None) Create a reduce function >>> func = lambda x, y: x + y Reduce table on samples >>> table.reduce(func, 'sample') # doctest: +NORMALIZE_WHITESPACE array([ 1., 3., 43.]) Reduce table on observations >>> table.reduce(func, 'observation') # doctest: +NORMALIZE_WHITESPACE array([ 1., 46.]) zCannot reduce an empty tablec0g|]}t|Sr^)r)rbrLfs r>rez Table.reduce..os!HHHq! HHHr?r)rrr iter_data)r;rFrs ` r>rz Table.reduce:sY` ==?? A !?@@ @HHHHdnn$n.G.GHHHIIIr?c|dkrd}n!|dkrd}n|dkrd}nt|tjtj|j|}| |jdkr|d}|S) a_Returns the sum by axis Parameters ---------- axis : {'whole', 'sample', 'observation'}, optional The axis on which to operate. Returns ------- numpy.array or float If `axis` is "whole", returns an float representing the whole table sum. If `axis` is either "sample" or "observation", returns a numpy.array that holds a sum for each sample or observation, respectively. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3']) Add all values in the table: >>> table.sum() array(47.0) Add all values per sample: >>> table.sum(axis='sample') # doctest: +NORMALIZE_WHITESPACE array([ 1., 3., 43.]) Add all values per observation: >>> table.sum(axis='observation') # doctest: +NORMALIZE_WHITESPACE array([ 1., 46.]) rNrrrr&rr^)rr5rrrsumrnr)r;r matrix_sums r>rIz Table.sumqsR 7??DD X  DD ] " "DD"4(( (Z 4:>>t>+D+D E EFF    0B 6 6#++A..Jr?ct|}t|d}|jdkr|j|_||jd|dd|ddd|||jS)a0Transpose the contingency table The returned table will be an entirely new table, including copies of the (transposed) data, sample/observation IDs and metadata. Returns ------- Table Return a new table that is the transpose of caller table. rrlilTcopyN) rrrr"rr=rrr)r;sample_md_copy obs_md_copys r>rzTable.transposes"$--//22t}}-}@@AA :   ! !U * *))++DJ~~dj222=="hhjjmTXX=X-I-I!!!-L,k4=JJ Jr?c,|dkrtd|dkrtd|dd|}|dd|}||dd }||dS) avGet the first n rows and m columns from self Parameters ---------- n : int, optional The number of rows (observations) to get. This number must be greater than 0. If not specified, 5 rows will be retrieved. m : int, optional The number of columns (samples) to get. This number must be greater than 0. If not specified, 5 columns will be retrieved. Notes ----- Like `head` for Linux like systems, requesting more rows (or columns) than exists will silently work. Raises ------ IndexError If `n` or `m` are <= 0. Returns ------- Table The subset table. Examples -------- >>> import numpy as np >>> from biom.table import Table >>> data = np.arange(100).reshape(5, 20) >>> obs_ids = ['O%d' % i for i in range(1, 6)] >>> samp_ids = ['S%d' % i for i in range(1, 21)] >>> table = Table(data, obs_ids, samp_ids) >>> print(table.head()) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 S4 S5 O1 0.0 1.0 2.0 3.0 4.0 O2 20.0 21.0 22.0 23.0 24.0 O3 40.0 41.0 42.0 43.0 44.0 O4 60.0 61.0 62.0 63.0 64.0 O5 80.0 81.0 82.0 83.0 84.0 rzn cannot be <= 0.zm cannot be <= 0.rrNrFr-)rrr/)r;r9rcrow_idscol_idstables r>headz Table.heads^ 66011 1 66011 1(( (..rr2((())"1"- G- GG||G(|333r?cT|dkr|jS|dkr|jSt|)azReturn the group metadata of the given axis Parameters ---------- axis : {'sample', 'observation'}, optional Axis to search for the group metadata. Defaults to 'sample' Returns ------- dict The corresponding group metadata for the given axis Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table, with group observation metadata and no group sample metadata: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> group_observation_md = {'tree': ('newick', '(O1:0.3,O2:0.4);')} >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... observation_group_metadata=group_observation_md) Get the observation group metadata: >>> table.group_metadata(axis='observation') {'tree': ('newick', '(O1:0.3,O2:0.4);')} Get the sample group metadata: >> table.group_metadata() None rr)rrrrs r>group_metadatazTable.group_metadatas;R 8  . . ] " "3 3"4(( (r?cT|dkr|jS|dkr|jSt|)akReturn the ids along the given axis Parameters ---------- axis : {'sample', 'observation'}, optional Axis to return ids from. Defaults to 'sample' Returns ------- 1-D numpy array The ids along the given axis Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3']) Get the ids along the observation axis: >>> print(table.ids(axis='observation')) ['O1' 'O2'] Get the ids along the sample axis: >>> print(table.ids()) ['S1' 'S2' 'S3'] rr)rrrrs r>rz Table.ids.s;L 8  # # ] " "( ("4(( (r?cdtd|Dz}t||j|}t ||D]:\}}|r||vrt d|d|d|||||<;|r>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3']) Define a mapping of old to new sample ids: >>> id_map = {'S1':'s1.1', 'S2':'s2.2', 'S3':'s3.3'} Get the ids along the sample axis in the table: >>> print(table.ids(axis='sample')) ['S1' 'S2' 'S3'] Update the sample ids and get the ids along the sample axis in the updated table: >>> updated_table = table.update_ids(id_map, axis='sample') >>> print(updated_table.ids(axis='sample')) ['s1.1' 's2.2' 's3.3'] U%dc,g|]}t|Sr^rrrs r>rez$Table.update_ids..s A A AAQ A A Ar?rrzMapping not provided for z identifier: z>. If this identifier should not be updated, pass strict=False.zDuplicate IDs observedrN)rrrrsizerFrrrrrtrNrrrr%) r;id_maprstrictr. str_dtype updated_idsrKold_idresults r> update_idszTable.update_ids[snnC A A A A ABBB DHH$H//4IFFF $TXX4X%8%899 : :KC &&..$nttVVV%&&& &zz&&99K    ?;3s;'7'7#8#888$%=>>>!1diikk 8  !,F  &1F #$%%%  r?c|dkr|jS|dkr|jSt|)auReturns the internal data in the correct sparse representation Parameters ---------- axis : {'sample', 'observation'}, optional Axis to search for `id`. Defaults to 'sample' Returns ------- sparse matrix The data in csc (axis='sample') or csr (axis='observation') representation rr)rr)rrrs r>_get_sparse_datazTable._get_sparse_datasP 8  :##%% % ] " ":##%% %"4(( (r?c|dkr|j}n|dkr|j}nt|||S|||}|||ndS)a9Return the metadata of the identified sample/observation. Parameters ---------- id : str ID of the sample or observation whose index will be returned. axis : {'sample', 'observation'} Axis to search for `id`. Returns ------- defaultdict or None The corresponding metadata ``defaultdict`` or ``None`` of that axis does not have metadata. Raises ------ UnknownAxisError If provided an unrecognized axis. UnknownIDError If provided an unrecognized sample/observation ID. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table, with observation metadata and no sample metadata: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... [{'foo': 'bar'}, {'x': 'y'}], None) Get the metadata of the observation with ID "O2": >>> # casting to `dict` as the return is `defaultdict` >>> dict(table.metadata('O2', 'observation')) {'x': 'y'} Get the metadata of the sample with ID "S1": >>> table.metadata('S1', 'sample') is None True rrNr)rrrr)r;idrr}rKs r>rzTable.metadatasn\ 8  &BB ] " "+BB"4(( ( :Ijj$j''.r#wwd2r?cf||}||vrt||||S)aReturn the index of the identified sample/observation. Parameters ---------- id : str ID of the sample or observation whose index will be returned. axis : {'sample', 'observation'} Axis to search for `id`. Returns ------- int Index of the sample/observation identified by `id`. Raises ------ UnknownAxisError If provided an unrecognized axis. UnknownIDError If provided an unrecognized sample/observation ID. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3']) Get the index of the observation with ID "O2": >>> table.index('O2', 'observation') 1 Get the index of the sample with ID "S1": >>> table.index('S1', 'sample') 0 r)rr)r;rir idx_lookups r>rz Table.indexs=T[[d[++ Z   T** *"~r?cf|||d||dfS)a)Return value in the matrix corresponding to ``(obs_id, samp_id)`` Parameters ---------- obs_id : str The ID of the observation samp_id : str The ID of the sample Returns ------- float The data value corresponding to the specified matrix position Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'Z3']) Retrieve the number of counts for observation `O1` in sample `Z3`. >>> print(table.get_value_by_ids('O2', 'Z3')) 42.0 See Also -------- Table.data rrr)r;obs_idsamp_ids r>get_value_by_idszTable.get_value_by_ids5s8DDJJv}55JJw1123 3r?c*|S)zcStringify self Default str output for a Table is just row/col ids and data values delimited_selfrs r>__str__z Table.__str__Zs ""$$$r?c|j\}}d||t|j|j|dzfzS)zReturns a high-level summary of the table's properties Returns ------- str A string detailing the shape, class, number of nonzero entries, and table density z/%d x %d %s with %d nonzero entries (%d%% dense)d)rnrr=rget_table_density)r;rowscolss r>__repr__zTable.__repr__asKZ d@ $T^,,dh  " " $ $s *D   r?c2|||vS)aReturns whether id exists in axis Parameters ---------- id: str id to check if exists axis : {'sample', 'observation'}, optional The axis to check Returns ------- bool ``True`` if `id` exists, ``False`` otherwise Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3']) Check whether sample ID is in the table: >>> table.exists('S1') True >>> table.exists('S4') False Check whether an observation ID is in the table: >>> table.exists('O1', 'observation') True >>> table.exists('O3', 'observation') False r)r)r;rirs r>rz Table.existspsNT[[d[++++r? #OTU IDc 2d|rtd|fd|D}||td||td|rd|||d|g}n d|||g}||d|D|d } |d } |d nd } t | |D]\} } |tt| | }| } |ro| m| |j | }|| |d}| ||d|| }|| |||| ||| }|| |||d |S) aReturn self as a string in a delimited form Default str output for the Table is just row/col ids and table data without any metadata Including observation metadata in output: If ``header_key`` is not ``None``, the observation metadata with that name will be included in the delimited output. If ``header_value`` is also not ``None``, the observation metadata will use the provided ``header_value`` as the observation metadata name (i.e., the column header) in the delimited output. ``metadata_formatter``: a function which takes a metadata entry and returns a formatted version that should be written to file ``observation_column_name``: the name of the first column in the output table, corresponding to the observation IDs. For example, the default will look something like: #OTU ID Sample1 Sample2 OTU1 10 2 OTU2 4 8 ctt|tr|dSt|SrO)r4rRrSrv)rs r>to_utf8z%Table.delimited_self..to_utf8s1!U## xx'''1vv r?z+Cannot delimit self if I don't have data...c&g|] }|Sr^r^)rbrrs r>rez(Table.delimited_self..s!>>>awwqzz>>>r?Nz4You need to specify both header_key and header_valuez# Constructed from biom filer|cg|]}|dzS) r^rbrs r>rez(Table.delimited_self..s!9!9!9Q!D&!9!9!9r?rrrqr)rrjoinr writelinesrr{ _iter_obsmaprvrrrrXwrite)r;delim header_key header_valuemetadata_formatterobservation_column_name direct_iosamp_idsoutput obs_metadataiterableend_linern obs_values str_obs_valsr}md_out output_rowrs @r>rszTable.delimited_selfs4    ==?? P !NOO O::>>>>488::>>>??  !#$JLLL  #!$JLLL  E.*MEM8MM|MMFF 50C%CCCEF   !9!9&!9!9!9 : : :}}-}88 888//"*22"%h&*nn&6&6#8#8 0 0 FJ ::c#t~~j/I/I&J&JKKLWV__F 0l6!$/&"9:++BFF:t,D,DEEVUULLL&&((D $MM*----OOJ////$VUULL((D $MM*----OOJ////yy   r?cr|jr|djsdSdS)zCheck whether the table is empty Returns ------- bool ``True`` if the table is empty, ``False`` otherwise rrTF)rr^rs r>rzTable.is_emptys8xxzz dhhMh&B&B&G 45r?c*|S)zSee ``biom.table.Table.iter``)iterrs r>__iter__zTable.__iter__syy{{r?c#Kt|jdD]/}||}|dV0dS)z,Return sample vectors of data matrix vectorsr&TrMN)rangernr r)r;ccolvecs r> _iter_sampzTable._iter_samps`tz!}%% . .A]]1%%F"""-- - - - -  . .r?c#rKt|jdD]}||VdS)z)Return observation vectors of data matrixrN)rrnr!)r;rs r>rzTable._iter_obs sHtz!}%% # #A--"" " " " " # #r?cd}|sM|jt|t|dzz }|S)zReturns the fraction of nonzero elements in the table. Returns ------- float The fraction of nonzero elements in the table rr)rrrrr)r;densitys r>rwzTable.get_table_densitysZ}} NxDHHJJ#dhhMh.J.J*K*KKMGr?cxt||jsdS|j|jksdStj|d|dsdStj||sdStj|d|dsdStj||sdS||jsd Sd S) z9For use in testing, describe how the tables are not equalz$Tables are not of comparable classeszTables are not the same typerrz Observation IDs are not the samezSample IDs are not the samez%Observation metadata are not the samez Sample metadata are not the samezData elements are not the samezTables appear equal r4r=r`r5 array_equalrr_data_equalityrr;others r>descriptive_equalityzTable.descriptive_equalitys#%00 :99yEJ&&11~dhhMh::#ii]i;;== 655~dhhjj%))++66 100~dmmm??#nn-n@@BB ;::~dmmoou~~/?/?@@ 655""5;// 433$$r?cxt||jsdS|j|jkrdStj|d|dsdStj||sdStj|d|dsdStj||sdS||jsdSdS)z__eq__z Table.__eq__3s%00 5 9 " "5~dhhMh::#ii]i;;== 5~dhhjj%))++66 5~dmmm??#nn-n@@BB 5~dmmoou~~/?/?@@ 5""5;// 5tr?c |jj|jkrdS|jj|jkrdS|jj|jkrdS|j|_|}|j|kjdkrdSdS)aReturn ``True`` if both matrices are equal. Matrices are equal iff the following items are equal: - shape - dtype - size (nnz) - matrix data (more expensive, so checked last) The sparse format does not need to be the same between the two matrices. ``self`` and ``other`` will be converted to csr format if necessary before performing the final comparison. FrT)rrnrGrrrs r>rzTable._data_equalityHs : u{ * *5 : u{ * *5 :>UY & &5Z%%''   J%  $q ( (5tr?c||k Sr3r^rs r>__ne__z Table.__ne__gsEM""r?c|dkr!|dd||df}n6|dkr!|||dddf}nt||r||S|S)a Returns data associated with an `id` Parameters ---------- id : str ID of the sample or observation whose data will be returned. axis : {'sample', 'observation'} Axis to search for `id`. dense : bool, optional If ``True``, return data as dense Returns ------- np.ndarray or scipy.sparse.spmatrix np.ndarray if ``dense``, otherwise scipy.sparse.spmatrix Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> from biom import example_table >>> example_table.data('S1', axis='sample') array([ 0., 3.]) See Also -------- Table.get_value_by_ids rNr)rrr)r;rirdenseros r>roz Table.datajsB 8  4::b(3334DD ] " " 2}55qqq89DD"4(( (  >>$'' 'Kr?c ||j|d|t |dt ||j|jS)zReturns a copy of the tablerrr_)r=rrNrrrrr`rs r>rNz Table.copys~~djoo//"hhMh::??AA"hhjjoo//&t}}-}'H'HII&t}}77"m#'9 .. .r?c#K|dkr7|D] }|r||V|V!dS|dkr7|D] }|r||V|V!dSt|)aYields axis values Parameters ---------- dense : bool, optional Defaults to ``True``. If ``False``, yield compressed sparse row or compressed sparse columns if `axis` is 'observation' or 'sample', respectively. axis : {'sample', 'observation'}, optional Axis to iterate over. Returns ------- generator Yields list of values for each value in `axis` Raises ------ UnknownAxisError If axis other than 'sample' or 'observation' passed Examples -------- >>> import numpy as np >>> from biom.table import Table >>> data = np.arange(30).reshape(3,10) # 3 X 10 OTU X Sample table >>> obs_ids = ['o1', 'o2', 'o3'] >>> sam_ids = ['s%i' %i for i in range(1,11)] >>> bt = Table(data, observation_ids=obs_ids, sample_ids=sam_ids) Lets find the sample with the largest sum >>> sample_gen = bt.iter_data(axis='sample') >>> max_sample_count = max([sample.sum() for sample in sample_gen]) >>> print(max_sample_count) 57.0 rrN)rrrr)r;rrsamp_vobs_vs r>rGzTable.iter_datasL 8  //++ ! !!..000000 LLLL  ! ! ] " "))   ..//////KKKK   #4(( (r?c\||}||}|dkr|}n*|dkr|}nt ||dt |z}|||}t|||S)aYields ``(value, id, metadata)`` Parameters ---------- dense : bool, optional Defaults to ``True``. If ``False``, yield compressed sparse row or compressed sparse columns if `axis` is 'observation' or 'sample', respectively. axis : {'sample', 'observation'}, optional The axis to iterate over. Returns ------- GeneratorType A generator that yields (values, id, metadata) Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'Z3']) Iter over samples and keep those that start with an Z: >>> [(values, id, metadata) ... for values, id, metadata in table.iter() if id[0]=='Z'] [(array([ 1., 42.]), 'Z3', None)] Iter over observations and add the 2nd column of the values >>> col = [values[1] for values, id, metadata in table.iter()] >>> sum(col) 46.0 rrrNr3rr)rrrrrrrrGr{)r;rrrriter_s r>rz Table.itersPhhDh!!==d=++ 8  OO%%EE ] " "NN$$EE"4(( (  S)HD665#x(((r?c#K||}||}|dt|z}tjt|d|z |rfd}nfd}t D]o\}} || } || } || ||} ||D]6} || }|| }||||}| | | f|||ffV7pdS)aPairwise iteration over self Parameters ---------- dense : bool, optional Defaults to ``True``. If ``False``, yield compressed sparse row or compressed sparse columns if `axis` is 'observation' or 'sample', respectively. axis : {'sample', 'observation'}, optional The axis to iterate over. tri : bool, optional If ``True``, just yield [i, j] and not [j, i] diag : bool, optional If ``True``, yield [i, i] Returns ------- GeneratorType Yields [(val_i, id_i, metadata_i), (val_j, id_j, metadata_j)] Raises ------ UnknownAxisError Examples -------- >>> from biom import example_table By default, only the upper triangle without the diagonal of the resulting pairwise combinations is yielded. >>> iter_ = example_table.iter_pairwise() >>> for (val_i, id_i, md_i), (val_j, id_j, md_j) in iter_: ... print(id_i, id_j) S1 S2 S1 S3 S2 S3 The full pairwise combinations can also be yielded though. >>> iter_ = example_table.iter_pairwise(tri=False, diag=True) >>> for (val_i, id_i, md_i), (val_j, id_j, md_j) in iter_: ... print(id_i, id_j) S1 S1 S1 S2 S1 S3 S2 S1 S2 S2 S2 S3 S3 S1 S3 S2 S3 S3 rNr3r&c|zdSr3r^rKdiag_vinds r>tri_fz"Table.iter_pairwise..tri_fOs3v:;;''r?cVtjd||zdgSr3)r5rrs r>rz"Table.iter_pairwise..tri_fRs-y#dsd)SV-=!>???r?r)rrrrr5arangerFro)r;rrtridiagrrrrKrid_imd_idata_ijid_jmd_jdata_jrrs @@r> iter_pairwisezTable.iter_pairwisesqn==d=++hhDh!!  S)HiC!!T  @ ( ( ( ( ( ( ( @ @ @ @ @ @ nn C CFCq6DA;DYYt$eY<>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[1, 0, 4], [1, 3, 0]]) >>> table = Table(data, ['O2', 'O1'], ['S2', 'S1', 'S3']) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S2 S1 S3 O2 1.0 0.0 4.0 O1 1.0 3.0 0.0 Sort the table using a list of samples: >>> sorted_table = table.sort_order(['S2', 'S3', 'S1']) >>> print(sorted_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S2 S3 S1 O2 1.0 4.0 0.0 O1 1.0 0.0 3.0 Additionally you could sort the table's observations: >>> sorted_table = table.sort_order(['O1', 'O2'], axis="observation") >>> print(sorted_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S2 S1 S3 O1 1.0 3.0 0.0 O2 1.0 0.0 4.0 c>g|]}|Srrm)rbrrr;s r>rez$Table.sort_order..s)BBBq$**QT*22BBBr?rrNrr) r5rr-rrr=rrr`r)r;rBrfancyrrs` ` r>r=zTable.sort_orderasUhBBBBBEBBB#NNN==d=++  x))%0H 8  "111e8,C>>#"&(( (">">qqq"A58"&--]-"C"CX"&-<< < ] " ""5!!!8,C>>#"'(DHHJJqqqM"*DMMOOT]"&)-- - #4(( (r?cj|||||S)afReturn a table sorted along axis Parameters ---------- sort_f : function, optional Defaults to ``biom.util.natsort``. A function that takes a list of values and sorts it axis : {'sample', 'observation'}, optional The axis to operate on Returns ------- biom.Table A table whose samples or observations are sorted according to the `sort_f` function Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table: >>> data = np.asarray([[1, 0, 4], [1, 3, 0]]) >>> table = Table(data, ['O2', 'O1'], ['S2', 'S1', 'S3']) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S2 S1 S3 O2 1.0 0.0 4.0 O1 1.0 3.0 0.0 Sort the order of samples in the table using the default natural sorting: >>> new_table = table.sort() >>> print(new_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O2 0.0 1.0 4.0 O1 3.0 1.0 0.0 Sort the order of observations in the table using the default natural sorting: >>> new_table = table.sort(axis='observation') >>> print(new_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S2 S1 S3 O1 1.0 3.0 0.0 O2 1.0 0.0 4.0 Sort the samples in reverse order using a custom sort function: >>> sort_f = lambda x: list(sorted(x, reverse=True)) >>> new_table = table.sort(sort_f=sort_f) >>> print(new_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S3 S2 S1 O2 4.0 1.0 0.0 O1 0.0 1.0 3.0 r)r=r)r;sort_frs r>sortz Table.sorts3|vvdhhDh&9&9::FFFr?c Z|r|n|}||}||}||}||}|j} t | ||||||\} }}| |_|dkr<||_||_| |j dnA|dkr;||_ ||_ | d|j t||S)a Filter a table based on a function or iterable. Parameters ---------- ids_to_keep : iterable, or function(values, id, metadata) -> bool If a function, it will be called with the values of the sample/observation, its id (a string) and the dictionary of metadata of each sample/observation, and must return a boolean. If it's an iterable, it must be a list of ids to keep. axis : {'sample', 'observation'}, optional It controls whether to filter samples or observations and defaults to "sample". invert : bool, optional Defaults to ``False``. If set to ``True``, discard samples or observations where `ids_to_keep` returns True inplace : bool, optional Defaults to ``True``. Whether to return a new table or modify itself. Returns ------- biom.Table Returns itself if `inplace`, else returns a new filtered table. Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table, with observation metadata and sample metadata: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... [{'full_genome_available': True}, ... {'full_genome_available': False}], ... [{'sample_type': 'a'}, {'sample_type': 'a'}, ... {'sample_type': 'b'}]) Define a function to keep only samples with sample_type == 'a'. This will drop sample S3, which has sample_type 'b': >>> filter_fn = lambda val, id_, md: md['sample_type'] == 'a' Get a filtered version of the table, leaving the original table untouched: >>> new_table = table.filter(filter_fn, inplace=False) >>> print(table.ids()) ['S1' 'S2' 'S3'] >>> print(new_table.ids()) ['S1' 'S2'] Using the same filtering function, discard all samples with sample_type 'a'. This will keep only sample S3, which has sample_type 'b': >>> new_table = table.filter(filter_fn, inplace=False, invert=True) >>> print(table.ids()) ['S1' 'S2' 'S3'] >>> print(new_table.ids()) ['S3'] Filter the table in-place using the same function (drop all samples where sample_type is not 'a'): >>> table.filter(filter_fn) 2 x 2 with 2 nonzero entries (50% dense) >>> print(table.ids()) ['S1' 'S2'] Filter out all observations in the table that do not have full_genome_available == True. This will filter out observation O2: >>> filter_fn = lambda val, id_, md: md['full_genome_available'] >>> table.filter(filter_fn, axis='observation') 1 x 2 with 0 nonzero entries (0% dense) >>> print(table.ids(axis='observation')) ['O1'] r)invertr&Nr)rNrrr _axis_to_numrr'rrrrrrrr%) r; ids_to_keeprrr.rUrrrarrs r>r/z Table.filters7n 0TYY[[>>t>,,iiTi""  &&!!t!,,k$S%(%-%*%0%),2 444S( 199 #E %-E "   T_1133T : : : : QYY%(E "*2E '   T4#5#:#:#<#< = = = r?c #Ki}|d|D]\}}}|||}t|tst|}||vrgggg||<||d|||d|||d||||}|D]\}\}} } |dkrc|| d } |} | } | d d d }| |d d nd }d |j i}nf|d kr`|| d } |}| }| d d } | |d d nd } d |j i}|t| || || |jf|jdd|fVd S)aYields partitions Parameters ---------- f : function `f` is given the ID and metadata of the vector and must return what partition the vector is part of. axis : {'sample', 'observation'}, optional The axis to iterate over Returns ------- GeneratorType A generator that yields (partition, `Table`) Examples -------- >>> import numpy as np >>> from biom.table import Table >>> from biom.util import unzip Create a 2x3 BIOM table, with observation metadata and sample metadata: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... [{'full_genome_available': True}, ... {'full_genome_available': False}], ... [{'sample_type': 'a'}, {'sample_type': 'a'}, ... {'sample_type': 'b'}]) Define a function to bin by sample_type >>> f = lambda id_, md: md['sample_type'] Partition the table and view results >>> bins, tables = table.partition(f) >>> print(bins[1]) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 0.0 0.0 O2 1.0 3.0 >>> print(tables[1]) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S3 O1 1.0 O2 42.0 Frrrr&rrTrrNrr)r`r)rr4r ryrXr _invert_axisrrrrrNrrrr`)r;rFr partitionsrrr}partrrrrorsamp_mdobs_idsobs_mdindicess r> partitionzTable.partition^ sed "YYUY>> + +MD#r1S"::DdH-- #T{{:%%$&B< 4 t Q  & &s + + + t Q  & &t , , , t Q  & &r * * * * ]] 1 1$ 7 7] 8 8-7-=-=-?-? ) ) )D)3x..v.FF"(( (66qqq9"$.AAAd.0D0D0F0FG&&..v.GG!88::aaa=#%>"QQQ%%t)4+=+B+B+D+DEdGXvw"m)26)e)) '))) ) ) ) )# ) )r?r&addPathc * g} g} |rg} nd} |dvrtd|z fd} dkrd}d}n dkrd }d }nt |r|rtd i}i}t||D]l\}}|||}d } t |\}}n4#t $r| rd |d|}t |Y:t $rYnwxYw|||<|dz }S|||<mdtt|D}|dkr tj n|j } dkrBtt|dt|f|}nAtt|dt|f|}| d D]\}}}|||} t |\}}n4#t $r| rd |d|}t |Y:t $rYnwxYw||}|dkr3t|j|jD]\} }!|| |fxx|!z cc<n?||}"||"z }#t|#j|#jD]\} }!|| |fxx|!z cc<Ɍ|rSt|t'dD]"\}$}%| |||$i#dt|t'dD} dkrt+|j}nt/|}||}&n|d}|| D]\}}'||'\}(})t|(|kr'||'| }*|r|*t|(z}*| ||*| ||r)| d|(i|| |}&t9|d|| } dkr)| }+| },|ddd}-||nd}.n&|dd}+| }-| }.||nd},t=|&|-|+|.|,|j|j S)aCollapse partitions in a table by metadata or by IDs Partition data by metadata or IDs and then collapse each partition into a single vector. If `include_collapsed_metadata` is ``True``, the metadata for the collapsed partition will be a category named 'collapsed_ids', in which a list of the original ids that made up the partition is retained The remainder is only relevant to setting `one_to_many` to ``True``. If `one_to_many` is ``True``, allow vectors to collapse into multiple bins if the metadata describe a one-many relationship. Supplied functions must allow for iteration support over the metadata key and must return a tuple of (path, bin) as to describe both the path in the hierarchy represented and the specific bin being collapsed into. The uniqueness of the bin is _not_ based on the path but by the name of the bin. The metadata value for the corresponding collapsed column may include more (or less) information about the collapsed data. For example, if collapsing "FOO", and there are vectors that span three associations A, B, and C, such that vector 1 spans A and B, vector 2 spans B and C and vector 3 spans A and C, the resulting table will contain three collapsed vectors: - A, containing original vectors 1 and 3 - B, containing original vectors 1 and 2 - C, containing original vectors 2 and 3 If a vector maps to the same partition multiple times, it will be counted multiple times. There are two supported modes for handling one-to-many relationships via `one_to_many_mode`: ``add`` and `divide`. ``add`` will add the vector counts to each partition that the vector maps to, which may increase the total number of counts in the output table. ``divide`` will divide a vectors's counts by the number of metadata that the vector has before adding the counts to each partition. This will not increase the total number of counts in the output table. If `one_to_many_md_key` is specified, that becomes the metadata key that describes the collapsed path. If a value is not specified, then it defaults to 'Path'. If `strict` is specified, then all metadata pathways operated on must be indexable by `metadata_f`. `one_to_many` and `norm` are not supported together. `one_to_many` and `collapse_f` are not supported together. `one_to_many` and `min_group_size` are not supported together. A final note on space consumption. At present, the `one_to_many` functionality requires a temporary dense matrix representation. Parameters ---------- f : function Function that is used to determine what partition a vector belongs to collapse_f : function, optional Function that collapses a partition in a one-to-one collapse. The expected function signature is: dense or sparse_vector <- collapse_f(Table, axis) Defaults to a pairwise add. norm : bool, optional Defaults to ``True``. If ``True``, normalize the resulting table min_group_size : int, optional Defaults to ``1``. The minimum size of a partition when performing a one-to-one collapse include_collapsed_metadata : bool, optional Defaults to ``True``. If ``True``, retain the collapsed metadata keyed by the original IDs of the associated vectors one_to_many : bool, optional Defaults to ``False``. Perform a one-to-many collapse one_to_many_mode : {'add', 'divide'}, optional The way to reduce two vectors in a one-to-many collapse one_to_many_md_key : str, optional Defaults to "Path". If `include_collapsed_metadata` is ``True``, store the original vector metadata under this key strict : bool, optional Defaults to ``False``. Requires full pathway data within a one-to-many structure axis : {'sample', 'observation'}, optional The axis to collapse Returns ------- Table The collapsed table Examples -------- >>> import numpy as np >>> from biom.table import Table Create a ``Table`` >>> dt_rich = Table( ... np.array([[5, 6, 7], [8, 9, 10], [11, 12, 13]]), ... ['1', '2', '3'], ['a', 'b', 'c'], ... [{'taxonomy': ['k__a', 'p__b']}, ... {'taxonomy': ['k__a', 'p__c']}, ... {'taxonomy': ['k__a', 'p__c']}], ... [{'barcode': 'aatt'}, ... {'barcode': 'ttgg'}, ... {'barcode': 'aatt'}]) >>> print(dt_rich) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID a b c 1 5.0 6.0 7.0 2 8.0 9.0 10.0 3 11.0 12.0 13.0 Create Function to determine what partition a vector belongs to >>> bin_f = lambda id_, x: x['taxonomy'][1] >>> obs_phy = dt_rich.collapse( ... bin_f, norm=False, min_group_size=1, ... axis='observation').sort(axis='observation') >>> print(obs_phy) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID a b c p__b 5.0 6.0 7.0 p__c 19.0 21.0 23.0 N)rdividezEUnrecognized one-to-many mode '%s'. Must be either 'add' or 'divide'.c\||fS)Nr)rrr2rs r> axis_ids_mdz#Table.collapse..axis_ids_mdN s*EEtE$$ajjdj&;&;< axis_updatez#Table.collapse..axis_updateT s ((r?rFc ||fSr3r^rs r>rz#Table.collapse..axis_updateZ s ((r?z/norm and one_to_many are not supported togetherrzIncomplete pathway, ID: z , metadata: r&ci|]\}}|| Sr^r^)rbrrs r> z"Table.collapse.. sKKKga$KKKr?rrrrrkeycg|]\}}|Sr^r^)rbr rs r>rez"Table.collapse.. s.FFF41aQFFFr?c,||Sr3rIrs r> collapse_fz"Table.collapse..collapse_f s55;;&r? collapsed_idsrrr_)! ValueErrorrAttributeErrorr{nextr StopIterationrFsortedr5float64rGrrrrrrrorr rXrrrrrrr8r%rrrr`)/r;rFrnormmin_group_sizeinclude_collapsed_metadata one_to_manyone_to_many_modeone_to_many_md_keyr`rcollapsed_datar collapsed_mdrrrrmd_countrr}md_iternum_mdpathwayrerrrkrGnew_datarrcolumnvidxrLdvtmpr rrorUaxis_idsaxis_md redux_datar sample_mdrrs/ ` r>collapsezTable.collapse sN % LLL #4 4 49;KLMM M  = = = = = 8  I ) ) ) )] " "I ) ) ) )#4(( ( ~ P G$EGGG FH D 1 12 ' 'R!C**  -1']]*% % % %!%%$'33#,C",S//1%H()0F9%aKF# &!' KK6&>>1J1JKKKJ#3h">">BJJDJE}$$%s4888+B+B'C'C'*6{{'4,1333&s4888+G+G'H'H'*6{{'4;@BBB "&E!B!B# 8# 8 c2!C** 8 (,W % % % %!%%$'33#,C",S//1%H((-F'500'*4<'C'C88GD!$T6\222a722228&c]"Ri'*3;'A'A88GD!$T6\222a72222A 8(* I":#3#3#5#5:a==IIIIIDAq '');VAY(GHHHHFF6*2B2B2D2D6@mm,E,E,EFFFM}$$%hj11%h//**844DD!''' $~~ad~;; N N e$/K$6$6!'x==>11'Zt/@/@/F/FGG 0#h--/J%%d&=&=j&I&IJJJ$$T***-N ''(//:K:K(LMMM**>Y*OOD w ]] 1 1$ 7 7] 8 8 8  &J$IhhMh221115G>RRtFFAAAJ#G!F n$IT7J ]444 4s0B!!#C CCG,,#H HHc@|dkrdS|dkrdSt|S)zInvert an axisrrrrs r>rzTable._invert_axis s0 8   = ] " "8#D)) )r?c@|dkrdS|dkrdSt|)z"Convert str axis to numerical axisrr&rrrrs r>rzTable._axis_to_num s0 8  1 ] " "1"4(( (r?c|dvrt||dkrMtj}|dD])}t ||j}*n}t t|||j }t|d|D]!\}}|j||<"|S)a8Get the minimum nonzero value over an axis Parameters ---------- axis : {'sample', 'observation', 'whole'}, optional Defaults to "sample". The axis over which to calculate minima. Returns ------- scalar of self.dtype or np.array of self.dtype Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> from biom import example_table >>> print(example_table.min(axis='sample')) [ 3. 1. 2.] rrrrFrrrr) rr5infrGminrorrrrrGrF)r;rmin_valrorKs r>r z Table.min s0 9 9 9"4(( ( 7??fGU33 8 8gty}}77 8Cd 3 344DJGGGG&t~~E~'M'MNN / / T#y}} r?c|dvrt||dkrNtj }|dD])}t ||j}*ntjt|||j }t|d|D]!\}}|j||<"|S)a9Get the maximum nonzero value over an axis Parameters ---------- axis : {'sample', 'observation', 'whole'}, optional Defaults to "sample". The axis over which to calculate maxima. Returns ------- scalar of self.dtype or np.array of self.dtype Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> from biom import example_table >>> print(example_table.max(axis='observation')) [ 2. 5.] rrFrrrr) rr5rrGrrorrrrrGrF)r;rmax_valrorKs r>rz Table.max, s0 9 9 9"4(( ( 7??vgGU33 8 8gty}}77 8hs4888#6#677tzJJJG&t~~E~'M'MNN / / T#y}} r?c |dkrtd|r|rtd|}tj|}|ro||}||t|d| | fd|nE| } t| |||| |_ |d|| |} |d| |S) a Randomly subsample without replacement. Parameters ---------- n : int Number of items to subsample from `counts`. axis : {'sample', 'observation'}, optional The axis to sample over by_id : boolean, optional If `False`, the subsampling is based on the counts contained in the matrix (e.g., rarefaction). If `True`, the subsampling is based on the IDs (e.g., fetch a random subset of samples). Default is `False`. with_replacement : boolean, optional If `False` (default), subsample without replacement. If `True`, resample with replacement via the multinomial distribution. Should not be `True` if `by_id` is `True`. seed : int, optional If provided, set the numpy random seed with this value Returns ------- biom.Table A subsampled version of self Raises ------ ValueError - If `n` is less than zero. - If `by_id` and `with_replacement` are both True. Notes ----- Subsampling is performed without replacement. If `n` is greater than the sum of a given vector, that vector is omitted from the result. Adapted from `skbio.math.subsample`, see biom-format/licenses for more information about scikit-bio. This code assumes absolute abundance if `by_id` is False. Examples -------- >>> import numpy as np >>> from biom.table import Table >>> table = Table(np.array([[0, 2, 3], [1, 0, 2]]), ['O1', 'O2'], ... ['S1', 'S2', 'S3']) Subsample 1 item over the sample axis by value (e.g., rarefaction): >>> print(table.subsample(1).sum(axis='sample')) [ 1. 1. 1.] Subsample 2 items over the sample axis, note that 'S1' is filtered out: >>> ss = table.subsample(2) >>> print(ss.sum(axis='sample')) [ 2. 2.] >>> print(ss.ids()) ['S2' 'S3'] Subsample by IDs over the sample axis. For this example, we're going to randomly select 2 samples and do this 100 times, and then print out the set of IDs observed. >>> ids = set([tuple(table.subsample(2, by_id=True).ids()) ... for i in range(100)]) >>> print(sorted(ids)) [('S1', 'S2'), ('S1', 'S3'), ('S2', 'S3')] rzn cannot be negative.z.by_id and with_replacement cannot both be TruerNc |vSr3r^)rLrr}subsets r>rz!Table.subsample.. s !v+r?c2|dkSNrrrLrr}s r>rz!Table.subsample.. s!%%''A+r?c2|dkSr(rr)s r>rz!Table.subsample.. saeeggkr?) rrNr5random default_rngrshufflertr/rgr)rr) r;r9rby_idwith_replacementseedrUrngrroinv_axisr&s @r> subsamplezTable.subsampleT sIR q55455 5  O OMNN N i##D))  B)))&&++--C KK   RaR\\F LL5555DL A A A A))++D tQ 0# 6 6 6EK LL55DL A A A$$T** 11 AAA r?c6d}|||S)aConvert the table to presence/absence data Parameters ---------- inplace : bool, optional Defaults to ``True`` Returns ------- Table Returns itself if `inplace`, else returns a new presence/absence table. Examples -------- >>> from biom.table import Table >>> import numpy as np Create a 2x3 BIOM table >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3']) Convert to presence/absence data >>> _ = table.pa() >>> print(table.data('O1', 'observation')) [ 0. 0. 1.] >>> print(table.data('O2', 'observation')) [ 1. 1. 1.] c6tj|dkddS)Nrg?r)r5r)rorrs r> transform_fzTable.pa..transform_f s8DAIr2.. .r?)r. transform)r;r.r6s r>pazTable.pa s*@ / / /~~k7~;;;r?c@|r|n|}||}||}||}||}t ||||||||_|S)a Iterate over `axis`, applying a function `f` to each vector. Only non null values can be modified and the density of the table can't increase. However, zeroing values is fine. Parameters ---------- f : function(data, id, metadata) -> new data A function that takes three values: an array of nonzero values corresponding to each observation or sample, an observation or sample id, and an observation or sample metadata entry. It must return an array of transformed values that replace the original values. axis : {'sample', 'observation'}, optional The axis to operate on. Can be "sample" or "observation". inplace : bool, optional Defaults to ``True``. Whether to return a new table or modify itself. Returns ------- biom.Table Returns itself if `inplace`, else returns a new transformed table. Raises ------ UnknownAxisError If provided an unrecognized axis. Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 table >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... [{'foo': 'bar'}, {'x': 'y'}], None) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O1 0.0 0.0 1.0 O2 1.0 3.0 42.0 Create a transform function >>> f = lambda data, id_, md: data / 2 Transform to a new table on samples >>> table2 = table.transform(f, 'sample', False) >>> print(table2) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O1 0.0 0.0 0.5 O2 0.5 1.5 21.0 `table` hasn't changed >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O1 0.0 0.0 1.0 O2 1.0 3.0 42.0 Tranform in place on observations >>> table3 = table.transform(f, 'observation', True) `table` is different now >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O1 0.0 0.0 0.5 O2 0.5 1.5 21.0 but the table returned (`table3`) is the same as `table` >>> print(table3) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O1 0.0 0.0 0.5 O2 0.5 1.5 21.0 r)rNrrrgrr(rr)r;rFrr.rUrrrs r>r8zTable.transform sp 0TYY[[>>t>,,iiTi""$$$$//!!$''3Xq$///   r?averagec>fd}||||S)aConvert values to rank abundances from smallest to largest Parameters ---------- axis : {'sample', 'observation'}, optional The axis to use for ranking. inplace : bool, optional Defaults to ``True``. If ``True``, performs the ranking in place. Otherwise, returns a new table with ranking applied. method : str, optional The method for handling ties in counts. This can be any valid string that can be passed to `scipy.stats.rankdata`. Returns ------- biom.Table The rank-abundance-transformed table. Raises ------ ValueError If unknown ``method`` is provided. See Also -------- scipy.stats.rankdata Examples -------- >>> import numpy as np >>> from biom import Table >>> data = np.array([[ 99, 12, 8], [ 0, 42, 7], ... [112, 42, 6], [ 5, 75, 5]]) >>> t = Table(data, sample_ids=['s1', 's2', 's3'], ... observation_ids=['o1', 'o2', 'o3', 'o4']) Convert observation counts to their ranked abundance, from smallest to largest. >>> print(t.rankdata()) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID s1 s2 s3 o1 2.0 1.0 4.0 o2 0.0 2.5 3.0 o3 3.0 2.5 2.0 o4 1.0 4.0 1.0 cFtj|S)N)method)scipystatsrankdata)rr_r>s r>rFzTable.rankdata..fu s;''F';; ;r?r-r7)r;rr.r>rFs ` r>rAzTable.rankdataD s5b < < < < <~~adG~<<>> import numpy as np >>> from biom.table import Table Create a 2x2 table: >>> data = np.asarray([[2, 0], [6, 1]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2']) Get a version of the table normalized on the 'sample' axis, leaving the original table untouched: >>> new_table = table.norm(inplace=False) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 2.0 0.0 O2 6.0 1.0 >>> print(new_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 0.25 0.0 O2 0.75 1.0 Get a version of the table normalized on the 'observation' axis, again leaving the original table untouched: >>> new_table = table.norm(axis='observation', inplace=False) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 2.0 0.0 O2 6.0 1.0 >>> print(new_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 1.0 0.0 O2 0.857142857143 0.142857142857 Do the same normalization on 'observation', this time in-place: >>> table.norm(axis='observation') 2 x 2 with 3 nonzero entries (75% dense) >>> print(table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 1.0 0.0 O2 0.857142857143 0.142857142857 cJ|t|z Sr3)r.rI)rrrBs r>rFzTable.norm..f sswwyy))) )r?r-r7)r;rr.rFs r>rz Table.normy s,F * * *~~adG~<<nonzeroz Table.nonzero sj  88::(( (..+V[1_-- 2 2G7OE#CW%F"59- 2 2x011111 2  2 2r?c|rd}d}n |j}d}|dvrjtt|||}t ||D]\}}||||<nAtd|}|D]}|dxx||z cc<|S) aGet nonzero summaries about an axis Parameters ---------- axis : {'sample', 'observation', 'whole'} The axis on which to count nonzero entries binary : bool, optional Defaults to ``True``. If ``True``, return number of nonzero entries. If ``False``, sum the values of the entries. Returns ------- numpy.array Counts in index order to the axis r-c@|djSr()rIr^rTs r>opz Table.nonzero_counts..op syy{{1~**r?c*|Sr3rrTs r>rLz Table.nonzero_counts..op suuwwr?rrrr&r)rGrrrrrFrG)r;rbinaryrGrLrdrKrs r>nonzero_countszTable.nonzero_counts s  E + + + +JE    , , ,3txxTx22335AAAF&t~~4~'@'@AA ' ' T bhhs  '1E***F(( & &q RRXX%  r?ct|dd}||ddi}d}|D]}||vr |||<|dz }|S)z+Determines merge order for id lists A and BNrr&)rxextend)r;aball_ids new_orderrKrs r>_union_id_orderzTable._union_id_order slqt**qt   C)##!$ #qr?cbt|dd}i}d}|D]}||vr |||<|dz }|S)z/Determines the merge order for id lists A and BNrr&)rt)r;rRrSall_brUrKrs r>_intersect_id_orderzTable._intersect_id_order sPAaaaD    Ce||!$ #qr?c|dvrt||r|}n|}|dkrddg}n|g}|D]K}|||||dk|L|S)axRemove empty samples or observations from the table Parameters ---------- axis : {'whole', 'sample', 'observation'}, optional The axis on which to operate. inplace : bool, optional If ``True`` vectors are removed in ``self``; if ``False`` the vectors are removed in a new table is returned. Raises ------ UnknownAxisError If the axis is not recognized. Returns ------- Table A table object with the zero'd rows, or columns removed as specified by the `axis` parameter. rrrrrr)rrNr/rrI)r;rr.rUr r s r>r0zTable.remove_empty s, 9 9 9"4(( (  EEIIKKE 7??m,DD6D N NB LL++EII2I,>,>,BC"L M M M M r?detectc"t|d}t|}t|d}t|}||k}||k}|dkr|r|std|dkr|std|dkr|std|dkr|s|std |dkrddg} n]|dkr1g} |r| d|r| dn&|dkrdg} n|dkrdg} nt d |z|} | D]-} | || | } .| S) adAlign self to other over a requested axis Parameters ---------- other : biom.Table The table to align too axis : str, optional, {sample, observation, both, detect} If 'sample' or 'observation', align to that axis. If 'both', align both axes. If 'detect', align what can be aligned. Raises ------ DisjointIDError If the requested axis can't be aligned. UnknownAxisError If an unrecognized axis is specified. Examples -------- Align one table to another, for instance a table of 16S data to a table of metagenomic data. In this example, we're aligning the samples of the two tables. >>> from biom import Table >>> import numpy as np >>> amplicon = Table(np.array([[0, 1, 2], [3, 4, 5]]), ... ['Ecoli', 'Staphylococcus'], ... ['S1', 'S2', 'S3']) >>> metag = Table(np.array([[6, 7, 8], [9, 10, 11]]), ... ['geneA', 'geneB'], ... ['S3', 'S2', 'S1']) >>> amplicon = amplicon.align_to(metag) >>> print(amplicon) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S3 S2 S1 Ecoli 2.0 1.0 0.0 Staphylococcus 5.0 4.0 3.0 rrbothzCannot align both axesrzCannot align sampleszCannot align observationsr[zNeither axis appears alignablezUnrecognized axis: %s)rtrrrXrr=) r;rrself_oself_sother_oother_s alignable_o alignable_srBrUaln_axiss r>align_tozTable.align_to= sNTXX=X1122TXXZZeii]i3344eiikk""' ' 6>>;>;>!":;; ; X  k !"899 9 ] " "; "!"=>> > X  { k !"BCC C 6>>"H-EE X  E ' X&&& , ]+++ X  JEE ] " ""OEE"#:T#ABB B 4 4H$$UYYHY%=%=*2%44EE r?c 4 "t||jr|g}|}dkrtd}t}t }ntd}t }t}t }t }i"|dd} | d|| D]} | } | |} t | } | | std| | t | |z rDt | |z D]}| ||"|<| | t|}g}| D]%} t|t | |z }|r}t|}t| }dkr||f}n||f}t!|}|| j|g}t| |}||| |}|(dgt| |z}nt|}|"fd|Dt| }| }dkr||||||}n||||||}n| }|||kr||||||'|d|D}t-jfd |D}g}|D]C} | }|dg|| jz}||D|d |} dkr!||||| ||j }!n |||||| |j }!|!S) aConcatenate tables if axis is disjoint Parameters ---------- others : iterable of biom.Table, or a single biom.Table instance Tables to concatenate axis : {'sample', 'observation'}, optional The axis to concatenate on. i.e., if axis is 'sample', then tables will be joined such that the set of sample IDs in the resulting table will be the union of sample IDs across all tables in others. Raises ------ DisjointIDError If IDs over the axis are not disjoint. Notes ----- The type of the table is inherited from self. Examples -------- Concatenate three tables in which the sample IDs are disjoint. Note the observation IDs in this example are not disjoint (although they can be): >>> from biom import Table >>> import numpy as np >>> a = Table(np.array([[0, 1, 2], [3, 4, 5]]), ['O1', 'O2'], ... ['S1', 'S2', 'S3'], ... [{'taxonomy': 'foo'}, {'taxonomy': 'bar'}]) >>> b = Table(np.array([[6, 7, 8], [9, 10, 11]]), ['O3', 'O4'], ... ['S4', 'S5', 'S6'], ... [{'taxonomy': 'baz'}, {'taxonomy': 'foobar'}]) >>> c = Table(np.array([[12, 13, 14], [15, 16, 17]]), ['O1', 'O5'], ... ['S7', 'S8', 'S9'], ... [{'taxonomy': 'foo'}, {'taxonomy': 'biz'}]) >>> d = a.concat([b, c]) >>> print(d) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 S4 S5 S6 S7 S8 S9 O1 0.0 1.0 2.0 0.0 0.0 0.0 12.0 13.0 14.0 O2 3.0 4.0 5.0 0.0 0.0 0.0 0.0 0.0 0.0 O3 0.0 0.0 0.0 6.0 7.0 8.0 0.0 0.0 0.0 O4 0.0 0.0 0.0 9.0 10.0 11.0 0.0 0.0 0.0 O5 0.0 0.0 0.0 0.0 0.0 0.0 15.0 16.0 17.0 rr&rNrzIDs are not disjointc g|] }| Sr^r^)rbrinvaxis_metadatas r>rez Table.concat..s"L"L"L1#3A#6"L"L"Lr?cg|] }|j Sr^)rrbr2s r>rez Table.concat..sAAAaAMAAAr?c<g|]}|Sr)r)rbr2rs r>rez Table.concat..s'$M$M$M!QUUU%5%5$M$M$Mr?r_)r4r=rr rrrtinsertr isdisjointrrrrrxrrrrrQrrXr=r5 concatenaternr`)#r;othersrinvaxis dim_getterstackinvstackr invaxis_ids all_tablesrUtable_axis_idstable_invaxis_order table_invaxisr invaxis_order padded_tables missing_ids n_invaxisn_axisrnzerodtmp_mat tmp_inv_ids tmp_inv_mdtmp_idstmp_md tmp_table concat_mat concat_ids concat_mdrinv_mdconcatrhs# ` @r>rz Table.concat s'b fdn - - ZF##D)) 8  #AJEHH#AJEH55ee AAAY !T"""  2 2E"YYDY11N"')))"9"9  344M&&~66 >%&<=== OON + + +=!!K/ 2m,,{:JJA*/...*I*I$Q''""=111{++  . I. IE{S1H1H-I-IIJJK$ " ,, UYYDY11228##&/EE#Y/E#5))"(E$5u#=>>#599'9#:#:;; "";///"^^^99 %"&#eiiWi.E.E*F*F!FJJ!%j!1!1J!!"L"L"L"L "L"L"LMMMuyydy3344T228## $w W/96!C!CII!%w/5z!C!CII"  7 ++}<AACC I$$Y////$$Y%9%9-?F&:&H&HIIIIUAA=AAABB ^$M$M$M$M}$M$M$MNN  " ' 'E~~4~00H 6JJu{$;$;;   X & & & &q!***88 8  ^^J z$*IDI$GGFF^^J M$-vDI$GGF r?c|g|z}ttd|D}ttd|D}dtt|Ddtt|Dd}dt|D}dt|D}g}g} |D]} | j} | d } | } fd t| D}fd t| D}| D]@\\}}}|||||f| |Atj |}tj | | }| ddg}d|D}||||S)a For simple merge operations it is faster to aggregate using pandas Parameters ---------- others : Table, or Iterable of Table If a Table, then merge with that table. If an iterable, then merge all of the tables cTg|]%}t|d&S)rrrtrrjs r>rez%Table._fast_merge..9s>$5$5$5()%(=(A(A$B$B$5$5$5r?cPg|]#}t|$Sr^rrjs r>rez%Table._fast_merge..;s&"@"@"@A3quuww<<"@"@"@r?ci|]\}}|| Sr^r^rbrKrs r>rz%Table._fast_merge..?sLLL&#qq#LLLr?ci|]\}}|| Sr^r^rs r>rz%Table._fast_merge..@sJJJaaJJJr?c|dS)Nr&r^rTs r>rz#Table._fast_merge..Cs 1r?cg|]\}}|Sr^r^rbr rLs r>rez%Table._fast_merge..DsMMMtq!MMMr?rcg|]\}}|Sr^r^rs r>rez%Table._fast_merge..EsKKKdaKKKr?rrc(i|]\}}||Sr^r^)rbrKr feature_maps r>rz%Table._fast_merge..Qs9AAA"(#q";q>AAAr?c(i|]\}}||Sr^r^)rbrKr sample_maps r>rz%Table._fast_merge..Ss9@@@!'a!*Q-@@@r?rmrr&)levelc$g|] \\}}}|||gSr^r^)rbrrrLs r>rez%Table._fast_merge..fs&FFF96Aq1aAYFFFr?)rr rFrrrtodokrrXpd MultiIndex from_tuplesSeriesgroupbyrIr=)r;rotables all_features all_samplesget1 feature_order sample_ordermirrU data_as_dokfeat_idsrtable_features table_samplesrFsrLgrouped collapsed_rcv list_listrrs @@r> _fast_mergezTable._fast_merge-s&c$5$5-3$5$5$566 S"@"@"@"@"@AA MLIf\6J6J,K,KLLL JJ9VK5H5H+I+IJJJ ~MMvk.?.?.A.At'L'L'LMMM KKfZ-=-=-?-?T&J&J&JKKK  ! !E+1133Kyymy44Hyy{{HAAAA,5h,?,?AAAN@@@@+4X+>+>@@@M)..00 ! ! A >!,mA.>?@@@ a     !] & &r * *)F"--- q!f5599;; GF 0C0C0E0EFFF ~~i EEEr?unioncX |}|d}|duo|du}|duo|du} |s| rY|dkrS|dkrMt|tttfr||S||gS|dkr;|||} nS|dkr;|||} ntd|z|dkr?||d|d} nW|dkr?||d|d} ntd|zt| td } t| td } | std | std |j } |j } |j}|j}g}g}| D][\}}||||df||||df\d t#t%| D}g}g}|}|}| D]\}}|||||sd}n|||}|||sd}n|||}||||g}g}|d}|d} | D]\}}|||||dsd}n|| |}| ||dsd}n| | |}||||t%| }!| D]\}"}#t)|!d }$||"dr||"d}%nd}%||"dr||"d}&nd}&|%|D]\}'}(|( |&|(|$|'<nR|&|D]\}'})|) |%|)|$|'<n:| D]7\}}*||vrd}+n|&||}+||vrd},n|%||},|+|,z|$|*<8||$||#<||||dd|dd||S)a` Merge two tables together The axes, samples and observations, can be controlled independently. Both can work on either "union" or "intersection". `sample_metadata_f` and `observation_metadata_f` define how to merge metadata between tables. The default is to just keep the metadata associated to self if self has metadata otherwise take metadata from other. These functions are given both metadata dicts and must return a single metadata dict Parameters ---------- other : biom.Table or Iterable of Table The other table to merge with this one. If an iterable, the tables are expected to not have metadata. sample : 'union', 'intersection', optional How the sample axis is handled observation : 'union', 'intersection', optional How the observation axis is handled sample_metadata_f : function, optional Defaults to ``biom.util.prefer_self``. Defines how to handle sample metadata during merge. obesrvation_metadata_f : function, optional Defaults to ``biom.util.prefer_self``. Defines how to handle observation metdata during merge. Returns ------- biom.Table The merged table Notes ----- - If ``sample_metadata_f`` and ``observation_metadata_f`` are None, then a fast merge is applied. - There is an implicit type conversion to ``float``. - The return type is always that of ``self`` Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x2 table and a 3x2 table: >>> d_a = np.asarray([[2, 0], [6, 1]]) >>> t_a = Table(d_a, ['O1', 'O2'], ['S1', 'S2']) >>> d_b = np.asarray([[4, 5], [0, 3], [10, 10]]) >>> t_b = Table(d_b, ['O1', 'O2', 'O3'], ['S1', 'S2']) Merging the table results in the overlapping samples/observations (see `O1` and `S2`) to be summed and the non-overlapping ones to be added to the resulting table (see `S3`). >>> merged_table = t_a.merge(t_b) >>> print(merged_table) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 O1 6.0 5.0 O2 6.0 4.0 O3 10.0 10.0 rrNr intersectionzUnknown sample merge type: %sz"Unknown observation merge type: %sr&rzNo samples in resulting table!z#No observations in resulting table!cg|]}dSr3r^rs r>rezTable.merge..s888888r?r.rr)rr4rxrtryrrVrrYrrrr rrrXrrrrrrrorr=)-r;rrrsample_metadata_fobservation_metadata_fs_mdo_mdno_md ignore_mdnew_samp_order new_obs_order other_obs_idx self_obs_idxother_samp_idx self_samp_idxother_samp_orderself_samp_orderronsirrrself_sample_mdother_sample_mdrrKself_mdother_mdrr self_obs_md other_obs_md vec_lengthrn new_obs_idxnew_vec other_vecself_vecn_idxs_idxo_idx new_samp_idxself_vec_valueother_vec_values- r>mergez Table.mergejsH}}}}-}001DDL&$.- #t +   7I 7  [G%;%;edC%7887++E222++UI666 W  !11$((**eiikkJJNN ~ % %!55dhhjj%))++NNNN !@6!IJJ J ' ! ! 00m,,eii]i.K.KMMMM N * * 44m,,eii]i.K.KMMMM!4    4 4 6 6JqMMJJJ}2244*Q--HHH  C !ABB B H !FGG G(  ,* * L LLGS  # #S.*<*#+>?   ..wAA B B B Bmmm77 ~~=~99 % E EHC NN3   "$++c +*N*N"%l3&78# S} ==$' c(:; MM00(CC D D D D(( $1= A= A FKJg666G||F|77 !!JJv}==  {{6 {66 99V];; &599NUE()1%9 !&6::NUE()25)9:.<MM)G\ m33)*)1-2H)In44*+*3N74K*L,:_,LGL))!% 7 7 @ @D  ~~d55d;;WQQQZ(mVY@@ @r?c d *+,-./0t|tjtjfst d|dvrt |i|sB?t |d|z/|d|z0|d|z.|d|zdd}tjfdt|Dt }0fd |D}tj t|d ztj } d | d <tjd |D| d d<tj.fd|D} tj/fd|D} |dkrK|ddd} ||} t| t|f}t!| | | f|}nJ|ddd} ||} t|t| f}t#| | | f|}dt%d| Dz}dt%d| Dz}tj| | } tj| | } t)|| | S|jd}|jd}|jd}t-t.dr- t/j|}n#t2tf$rYnwxYw|jd}|jddkrdn |jd}t|t4r|d}t|t4r|d}d**fd }||d!\} }}||d\} }}||d"}|d#+|d$,|d%-Wd&}|dkrdfndf\}}|| |\} }|| |\} } t| t| f}d'}!|!||}|!|| }|dkr| n|}"tj|"d }#t;-fd(|#D}$tjd)|$D}%tj t|#d ztj } d | d <|%| d d<tj+fd*|$D} tj,fd+|$D} n+} ,} -} | | | f}&|dkrt!|&|}'nt#|&|}'t)|'| | |pd|pd||||||, }($d-})|dkrd!nd}|(|)|.|(S)/a.Parse an HDF5 formatted BIOM table If ids is provided, only the samples/observations listed in ids (depending on the value of axis) will be loaded The expected structure of this group is below. A few basic definitions, N is the number of observations and M is the number of samples. Data are stored in both compressed sparse row (for observation oriented operations) and compressed sparse column (for sample oriented operations). Notes ----- The expected HDF5 group structure is below. An example of an HDF5 file in DDL can be found here [1]_. - ./id : str, an arbitrary ID # noqa - ./type : str, the table type (e.g, OTU table) # noqa - ./format-url : str, a URL that describes the format # noqa - ./format-version : two element tuple of int32, major and minor # noqa - ./generated-by : str, what generated this file # noqa - ./creation-date : str, ISO format # noqa - ./shape : two element tuple of int32, N by M # noqa - ./nnz : int32 or int64, number of non zero elems # noqa - ./observation : Group # noqa - ./observation/ids : (N,) dataset of str or vlen str # noqa - ./observation/matrix : Group # noqa - ./observation/matrix/data : (nnz,) dataset of float64 # noqa - ./observation/matrix/indices : (nnz,) dataset of int32 # noqa - ./observation/matrix/indptr : (M+1,) dataset of int32 # noqa - ./observation/metadata : Group # noqa - [./observation/metadata/foo] : Optional, (N,) dataset of any valid HDF5 type in index order with IDs. # noqa - ./observation/group-metadata : Group # noqa - [./observation/group-metadata/foo] : Optional, (?,) dataset of group metadata that relates IDs # noqa - [./observation/group-metadata/foo.attrs['data_type']] : attribute of the foo dataset that describes contained type (e.g., newick) # noqa - ./sample : Group # noqa - ./sample/ids : (M,) dataset of str or vlen str # noqa - ./sample/matrix : Group # noqa - ./sample/matrix/data : (nnz,) dataset of float64 # noqa - ./sample/matrix/indices : (nnz,) dataset of int32 # noqa - ./sample/matrix/indptr : (N+1,) dataset of int32 # noqa - ./sample/metadata : Group # noqa - [./sample/metadata/foo] : Optional, (M,) dataset of any valid HDF5 type in index order with IDs. # noqa - ./sample/group-metadata : Group # noqa - [./sample/group-metadata/foo] : Optional, (?,) dataset of group metadata that relates IDs # noqa - [./sample/group-metadata/foo.attrs['data_type']] : attribute of the foo dataset that describes contained type (e.g., newick) # noqa The '?' character on the dataset size means that it can be of arbitrary length. The expected structure for each of the metadata datasets is a list of atomic type objects (int, float, str, ...), where the index order of the list corresponds to the index order of the relevant axis IDs. Special metadata fields have been defined, and they are stored in a specific way. Currently, the available special metadata fields are: - taxonomy: (N, ?) dataset of str or vlen str - KEGG_Pathways: (N, ?) dataset of str or vlen str - collapsed_ids: (N, ?) dataset of str or vlen str Parameters ---------- h5grp : a h5py ``Group`` or an open h5py ``File`` The object to load from ids : iterable The sample/observation ids of the samples/observations that we need to retrieve from the hdf5 biom table axis : 'sample', 'observation', optional The axis to subset on parse_fs : dict, optional Specify custom parsing functions for metadata fields. This dict is expected to be {'metadata_field': function}, where the function signature is (object) corresponding to a single row in the associated metadata dataset. The return from this function an object as well, and is the parsed representation of the metadata. subset_with_metadata : bool, optional When subsetting (i.e., `ids` is `not None`), whether to also parse the metadata. By default, the metadata are also subset. The reason for exposing this functionality is that, for large tables, there exists a very large overhead for this metadata manipulation. Returns ------- biom.Table A BIOM ``Table`` object Raises ------ ValueError If `ids` are not a subset of the samples or observations ids present in the hdf5 biom table If h5grp is not a HDF5 file or group References ---------- .. [1] http://biom-format.org/documentation/format_versions/biom-2.1.html See Also -------- Table.to_hdf5 Examples -------- >>> from biom.table import Table >>> from biom.util import biom_open >>> with biom_open('rich_sparse_otu_table_hdf5.biom') as f # doctest: +SKIP >>> t = Table.from_hdf5(f) # doctest: +SKIP Parse a hdf5 biom table subsetting observations >>> from biom.util import biom_open # doctest: +SKIP >>> from biom.parse import parse_biom_table >>> with biom_open('rich_sparse_otu_table_hdf5.biom') as f # doctest: +SKIP >>> t = Table.from_hdf5(f, ids=["GG_OTU_1"], ... axis='observation') # doctest: +SKIP z1h5grp does not appear to be an HDF5 file or grouprNz%s/matrix/indicesz%s/matrix/indptrz%s/matrix/dataz%s/idsc"g|] \}}|v | Sr^r^)rbrrrs r>rez#Table.from_hdf5..s- / / /fa#&#::!"#-::r?rc6g|]}||dzfSr&r^)rbr raw_indptrs r>rez#Table.from_hdf5..s*KKKa*Q-AaC9KKKr?r&rcg|] \}}||z  Sr^r^)rbres r>rez#Table.from_hdf5..s "?"?"?TQ1q5"?"?"?r?c*g|]\}}||Sr^r^)rbrrraw_datas r>rez#Table.from_hdf5..s%"G"G"GTQ8AaC="G"G"Gr?c*g|]\}}||Sr^r^)rbrr raw_indicess r>rez#Table.from_hdf5..s&%M%M%M41ak!A#&6%M%M%Mr?rzobservation/idsrz sample/idsr[c,g|]}t|Sr^r]rs r>rez#Table.from_hdf5..s(A(A(AAQ(A(A(Ar?c,g|]}t|Sr^r]rs r>rez#Table.from_hdf5..s)C)C)CQ#a&&)C)C)Cr?ri creation-date generated-by fromisoformatrnr`rqasciicZt|tr|dSdSrOrQrTs r> ensure_utf8z$Table.from_hdf5..ensure_utf8s+!U## xx'''r?c|ddd}|jdkr2dtd|Dz}tj||}t d}t |d<t |d <t |d <| d tt|D}|d  D]Q\}}| d d}||}|dd}t||D]\} } || | |<Rt|r|nd} fd|d D} ||| fS)z%Loads all the data of the given grouprNrr[c,g|]}t|Sr^r]rs r>rez6Table.from_hdf5..axis_load..)s(=(=(=AQ(=(=(=r?rctSr3)rVr^r?r>rz4Table.from_hdf5..axis_load..,sr?r KEGG_Pathwaysrcg|]}iSr^r^rs r>rez6Table.from_hdf5..axis_load..3s..."...r?rrgrfc:i|]\}}||dSrr^)rbcatrrs r>rz6Table.from_hdf5..axis_load..?sCEEE"#s;;s1v..EEEr?group-metadata) r^rr5rr r[rrrrrrsr{any)r|r ids_dtypeparserr}categorydsetparse_fromd_dictdata_rowgrp_mdrparse_fss r> axis_loadz"Table.from_hdf5..axis_load#se*QQQ-Cx!||!C(=(=(=(=(=$>$>> jI666 !7!788F!8F: &=F? #&=F? # MM( # # #/.eCHHoo...B"%j/"7"7"9"9 : :$#++K== *AAAw),R::%GX(/(9(9GH%%:2ww(DBEEEE&)*:&;&A&A&C&CEEEFF? "r?rmatrixrorrFcD|+|dd}tj|jt}nptj|}tj||}||}|j|jkr/t dt|t|z z||fS)zIf desired_ids is not None, makes sure that it is a subset of source_ids and returns the desired_ids array-like and a boolean array indicating where the desired_ids can be found in source_idsNrz:The following ids could not be found in the biom table: %s)r5onesrnboolrin1drrt) source_ids desired_idsrrKs r>_get_idsz!Table.from_hdf5.._get_idsNs &$QQQ-C'*"2$???CC"$*["9"9K'*k::C$S/CyK$555(*G*-k*:*:SXX*E*GHHHCxr?c||r9ttj|tj|}|S)zTIf md has data, returns the subset indicated by idx, a boolean array)rxr5rr)r}rKs r>_subset_metadataz)Table.from_hdf5.._subset_metadatams3=bjnnRXc]];<K|]}||dzfVdS)r&Nr^)rbr h5_indptrs r>rz"Table.from_hdf5..zsE$$341y1~.$$$$$$r?cg|] \}}||z  Sr^r^)rbrrHs r>rez#Table.from_hdf5..~s6&H&H&H*4%'*Ek&H&H&Hr?c*g|]\}}||Sr^r^)rbrrHh5_datas r>rez#Table.from_hdf5..s<@@@",%&eCi0@@@r?c*g|]\}}||Sr^r^)rbrrH h5_indicess r>rez#Table.from_hdf5..s<!C!C!C%/UC",E#I!6!C!C!Cr?)r`rrrrrc*tj|Sr3)r5r)rrr}s r> any_valuez"Table.from_hdf5..any_valuesvd||#r?r) r4h5pyGroupFilerrrtr5rrFr-rrrint32cumsumrnrrrrrattrshasattrrrrrRrSrrrr/)1rDh5grprrrsubset_with_metadatarto_keep start_endrFrorrrrnr obs_ids_dtypesamp_ids_dtyperrrtype_rr obs_grp_mdr samp_grp_mddata_grprsampobsobs_idxsamp_idxrrKkeepindptr_indices indptr_subsetcsrr2rrr r rrrrs1 ` ` @@@@@@@r> from_hdf5zTable.from_hdf5cs6r%$*di!899 &%&& & 0 0 0"4(( (  H#" 1c((C 3d :;K1D89J-45HX_-aaa0Hh / / / / (0C0C / / /69;;;GKKKK7KKKIXc'llQ.bh???FF1I"?"?Y"?"?"?@@GGIIF122J>"G"G"G"GY"G"G"GHHDn%M%M%M%M9%M%M%MNNGx 121115#G,Ws7||4 $!8FFF .qqq1"7+Ws8}}5 $!8FFF"C(A(A(A(A(A$B$BBM"S)C)C()C)C)C%D%DDNj >>>Gz(.AAAHgx00 0k$k/2 {>2 8_ - -  &4[AA z*     G$ F+r11u{67J c5 ! ! &**W%%C eU # # *LL))E     # # # # # #@'0im0D&E&E#)25?)C)C&';;x(6"i( X&  ?   .(,x'7'7d dC[ID#'x55 GW!)(D!9!9 Hh\\3x==1E   &%fg66F&&w99G#h..((GC8C==#D#$$$$8<$$$NH&H&H8F&H&H&HIIMXc$ii!m28<<@@@AADi!C!C!C!C3A!C!C!CDDGGD GFGV $ 8  %000FF%000F &'8V^t/T;+c-7(3  5 5 5 ? $ $ $%)H$4$4==(D HHYTH * * *sK--LLc:|d}|}|r&|j}tj}n<|j}t tjjj}||||S)aConvert matrix data to a Pandas SparseDataFrame or DataFrame Parameters ---------- dense : bool, optional If True, return pd.DataFrame instead of pd.SparseDataFrame. Returns ------- pd.DataFrame or pd.SparseDataFrame A DataFrame indexed on the observation IDs, with the column names as the sample IDs. Notes ----- Metadata are not included. Examples -------- >>> from biom import example_table >>> df = example_table.to_dataframe() >>> df S1 S2 S3 O1 0.0 1.0 2.0 O2 3.0 4.0 5.0 rrrcolumns) rrrr DataFramerNrsparse from_spmatrix)r;rrr,r constructors r> to_dataframezTable.to_dataframes6m,,((**  E"**,,C,KK"''))C!","5"CDDK{3eW====r?float32c4 ddl}n#t$rtdwxYw|j}|r|}|d}|d}|||||}|}|S)ugConvert Table to AnnData format Parameters ---------- dense : bool, optional If True, set adata.X as np.ndarray instead of sparse matrix. dtype: str, optional dtype used for storage in anndata object. tranpose: bool, optional If True, transpose the anndata so that observations are columns Returns ------- anndata.AnnData AnnData with matrix data and associated observation and sample metadata. Notes ----- Nested metadata are not included. Examples -------- >>> from biom import example_table >>> adata = example_table.to_anndata() >>> adata AnnData object with n_obs × n_vars = 3 × 2 obs: 'environment' var: 'taxonomy_0', 'taxonomy_1' rNz7Please install anndata package -- `pip install anndata`rr)r"varrG)anndata ImportErrorrrmetadata_to_dataframeAnnDatar) r;rrGrr5rr4r"adatas r> to_anndatazTable.to_anndatas>  NNNN   I    ++--C((22((77#UCC!! s!c||}|td|zg}|D]}g}i}|D]z\}}t|tt fr?d||<t t|D]} |d|| fz`d||<||{t|t|kr|}g} |D]k} g} | D]=\}}||r|D]} | | (| |>| | ltj | | ||S)aConvert axis metadata to a Pandas DataFrame Parameters ---------- axis : {'sample', 'observation'} The axis to operate on. Returns ------- pd.DataFrame A DataFrame indexed by the ids of the desired axis, columns by the metadata keys over that axis. Raises ------ UnknownAxisError If the requested axis isn't recognized KeyError IF the requested axis does not have metadata TypeError If a metadata column is a list or tuple, but is jagged over the axis. Notes ----- Nested metadata (e.g., KEGG_Pathways) is not supported. Metadata which are lists or tuples (e.g., taxonomy) are expanded such that each index position is a unique column. For instance, the key taxonomy will become "taxonomy_0", "taxonomy_1", etc where "taxonomy_0" corresponds to the 0th index position of the taxonomy. Examples -------- >>> from biom import example_table >>> example_table.metadata_to_dataframe('observation') taxonomy_0 taxonomy_1 O1 Bacteria Firmicutes O2 Bacteria Bacteroidetes rNz%s does not have metadataTz%s_%dFr+) rKeyErrorrr4ryrxrrrrXrr-r)r;rr}mcolstestr,expandrrYrKrxrcrrLs r>r7zTable.metadata_to_dataframesR]]] % % :6=>> > DGF"jjll ( ( UeeT]33("&F3K$SZZ00==w#s';<<<<=#(F3KNN3''''7||c%jj((  ACggii & & U#;&"&& 1 &JJu%%%% KK    |Dd(;(;UKKKKr?c|i}|j}|jr|jnd|jd<|jr|jnd|jd<d|jd<|j|jd<||jd <|.t j|jd <n||jd <|j|jd <||jd <d}|d urd}td}t|d<t|d<t|d<| |tddgddgD]\} } | | } |j| |_|| } t#| } t#|jj}|}|| }| d|rt)|d}t| dd|ddD]^\}}t)||krFt+|d| dd|dt-|d| ddt-| _t-|dD]}||| ||||| }| d|rK|D]6\}}|\}}| d |zd!t4||"}||jd#<7| d$| d%|ft6j|jj|"| d&|ft6j|jj|"| d'|ft6j|jj|"| dkr,| d(| ft4d)| D|"| d(d*g|+dS),apStore CSC and CSR in place The resulting structure of this group is below. A few basic definitions, N is the number of observations and M is the number of samples. Data are stored in both compressed sparse row [1]_ (CSR, for observation oriented operations) and compressed sparse column [2]_ (CSC, for sample oriented operations). Notes ----- This method does not return anything and operates in place on h5grp. The expected HDF5 group structure is below. An example of an HDF5 file in DDL can be found here [3]_. - ./id : str, an arbitrary ID - ./type : str, the table type (e.g, OTU table) - ./format-url : str, a URL that describes the format - ./format-version : two element tuple of int32, major and minor - ./generated-by : str, what generated this file - ./creation-date : str, ISO format - ./shape : two element tuple of int32, N by M - ./nnz : int32 or int64, number of non zero elems - ./observation : Group - ./observation/ids : (N,) dataset of str or vlen str - ./observation/matrix : Group - ./observation/matrix/data : (nnz,) dataset of float64 - ./observation/matrix/indices : (nnz,) dataset of int32 - ./observation/matrix/indptr : (M+1,) dataset of int32 - ./observation/metadata : Group - [./observation/metadata/foo] : Optional, (N,) dataset of any valid HDF5 type in index order with IDs. - ./observation/group-metadata : Group - [./observation/group-metadata/foo] : Optional, (?,) dataset of group metadata that relates IDs - [./observation/group-metadata/foo.attrs['data_type']] : attribute of the foo dataset that describes contained type (e.g., newick) - ./sample : Group - ./sample/ids : (M,) dataset of str or vlen str - ./sample/matrix : Group - ./sample/matrix/data : (nnz,) dataset of float64 - ./sample/matrix/indices : (nnz,) dataset of int32 - ./sample/matrix/indptr : (N+1,) dataset of int32 - ./sample/metadata : Group - [./sample/metadata/foo] : Optional, (M,) dataset of any valid HDF5 type in index order with IDs. - ./sample/group-metadata : Group - [./sample/group-metadata/foo] : Optional, (?,) dataset of group metadata that relates IDs - [./sample/group-metadata/foo.attrs['data_type']] : attribute of the foo dataset that describes contained type (e.g., newick) The '?' character on the dataset size means that it can be of arbitrary length. The expected structure for each of the metadata datasets is a list of atomic type objects (int, float, str, ...), where the index order of the list corresponds to the index order of the relevant axis IDs. Special metadata fields have been defined, and they are stored in a specific way. Currently, the available special metadata fields are: - taxonomy: (N, ?) dataset of str or vlen str - KEGG_Pathways: (N, ?) dataset of str or vlen str - collapsed_ids: (N, ?) dataset of str or vlen str Parameters ---------- h5grp : `h5py.Group` or `h5py.File` The HDF5 entity in which to write the BIOM formatted data. generated_by : str A description of what generated the table compress : bool, optional Defaults to ``True`` means fields will be compressed with gzip, ``False`` means no compression format_fs : dict, optional Specify custom formatting functions for metadata fields. This dict is expected to be {'metadata_field': function}, where the function signature is (h5py.Group, str, dict, bool) corresponding to the specific HDF5 group the metadata dataset will be associated with, the category being operated on, the metadata for the entire axis being operated on, and whether to enable compression on the dataset. Anything returned by this function is ignored. creation_date : datetime, optional If provided, use this specific datetime on write as the creation timestamp See Also -------- Table.from_hdf5 References ---------- .. [1] http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.sparse.csr_matrix.html .. [2] http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.sparse.csc_matrix.html .. [3] http://biom-format.org/documentation/format_versions/biom-2.1.html Examples -------- >>> from biom.util import biom_open # doctest: +SKIP >>> from biom.table import Table >>> from numpy import array >>> t = Table(array([[1, 2], [3, 4]]), ['a', 'b'], ['x', 'y']) >>> with biom_open('foo.biom', 'w') as f: # doctest: +SKIP ... t.to_hdf5(f, "example") Nz No Table IDrirqr`r,z format-urlzformat-versionrrrnrTgzipctSr3)rr^r?r>rzTable.to_hdf5..s(9r?rrrrrrGcscrrrr&z+ has inconsistent metadata categories with z: z: rrzgroup-metadata/%srrm data_typerz matrix/datazmatrix/indicesz matrix/indptrrc8g|]}|dSrjrkrs r>rez!Table.to_hdf5..#s$(G(G(Ga&)9)9(G(G(Gr?r)rnrorp) rrrr`rrnow isoformatrnr rzrr{ create_grouprasformatrrrrFrrtrrxrXrrwr#r5rrorr)r;rrcompress format_fs creation_daterrp formatterrrBr|rlen_ids len_indptrlen_datar}expother_idrrrrrYdatatyper grp_datasets r>to_hdf5z Table.to_hdf5Is~  Ih-1]MDMM  D+/9>+C+C+E+EEK ( (+8+B+B+D+DEK (#z G  E t   K 9 9:: : *%? /"%? /"### x85%.IID <D *>II&Hh8}}++(6>XXs1vvv5=XXtH~~~~58VVVT#YYY *HIII,!%RU HHH(Ih'Xr;GGGG**400H   - . . . >"*.."2"2>>JC$)MHc"%"4"4+c1"- k#5#;#;K6>K%k22   X & & &   }XK%'Z$(JO+6  8 8 8   /{%'X$(J$6+6  8 8 8   zm%'X$(J$5+6  8 8 8 {{""5 )6(G(G3(G(G(G/:#<<<< ""5B/:#<<<>> from biom import Table >>> json_obj = {"id": "None", ... "format": "Biological Observation Matrix 1.0.0", ... "format_url": "http://biom-format.org", ... "generated_by": "foo", ... "type": "OTU table", ... "date": "2014-06-03T14:24:40.884420", ... "matrix_element_type": "float", ... "shape": [5, 6], ... "data": [[0,2,1.0], ... [1,0,5.0], ... [1,1,1.0], ... [1,3,2.0], ... [1,4,3.0], ... [1,5,1.0], ... [2,2,1.0], ... [2,3,4.0], ... [2,5,2.0], ... [3,0,2.0], ... [3,1,1.0], ... [3,2,1.0], ... [3,5,1.0], ... [4,1,1.0], ... [4,2,1.0]], ... "rows": [{"id": "GG_OTU_1", "metadata": None}, ... {"id": "GG_OTU_2", "metadata": None}, ... {"id": "GG_OTU_3", "metadata": None}, ... {"id": "GG_OTU_4", "metadata": None}, ... {"id": "GG_OTU_5", "metadata": None}], ... "columns": [{"id": "Sample1", "metadata": None}, ... {"id": "Sample2", "metadata": None}, ... {"id": "Sample3", "metadata": None}, ... {"id": "Sample4", "metadata": None}, ... {"id": "Sample5", "metadata": None}, ... {"id": "Sample6", "metadata": None}] ... } >>> t = Table.from_json(json_obj) cg|] }|d Srir^rbrs r>rez#Table.from_json..esAAACc$iAAAr?r,cg|] }|d Srr^rYs r>rez#Table.from_json..fsLLLs3z?LLLr?cg|] }|d SrXr^rbrs r>rez#Table.from_json..gs;;;3t9;;;r?rxcg|] }|d Sr[r^r]s r>rez#Table.from_json..hsFFFCJFFFr?matrix_element_type matrix_typerTFr`Nrordaternr)rnrGr`rrr)MATRIX_ELEMENT_TYPErrrrrr) r; json_table data_pumprrrrrrGrror table_objs r> from_jsonzTable.from_json*sYvBA:i+@AAA LLj6KLLL;; 6(:;;;FF:f3EFFF #J/D$EF J & &-(G33!%!&6"  f%DDD 8_ - -  &4Z5GHH z*    $& *7 3 %$&1'1.'A)7999 sB99C  C cF t|tstd|&tj}n|}|r|d|dt|jz|dtdz|dtz|d|z|d |znDdt|jz}dtdz}dtz}d|z}d |z} |j \}} n #d x}} YnxYw|d kr| d krd nd } d } | r|d } t| trd} n?t| trd} n't| trd} ntd|r3|d| z|d|| fzn d| z} d|| fz} |j d}n d|j z}|r|||r+|d|dnd}dg}t|ddz }t|dz }dg}d }t!|dD]b\}}||krC|dt'|ddt'|ddnB|dt'|ddt'|dd g}t!|d D]3\}}t|d!kr|d"|||fz4|r|r-|r|d#n|d#|r)|d#|n(|d#|d }d|r|d$n|d$d%g}t!|D]\}}||krP|d&t'|dt'|d[|d't'|dt'|d|d dkrt|dkrd(g}d)g}d*|}d*|}|rA|||||d+dSd,d*|||||||| | d*|||g zS)-aReturns a JSON string representing the table in BIOM format. Parameters ---------- generated_by : str a string describing the software used to build the table direct_io : file or file-like object, optional Defaults to ``None``. Must implementing a ``write`` function. If `direct_io` is not ``None``, the final output is written directly to `direct_io` during processing. creation_date : datetime, optional If provided, use this datetime as the creation date on write. Returns ------- str A JSON-formatted string representing the biom table z"Must specify a generated_by stringN{z "id": "%s",z"format": "%s",r&rz"format_url": "%s",z"generated_by": "%s",z "date": "%s",rTFrr-r.rvzUnsupported matrix data type.z"matrix_element_type": "%s",z"shape": [%d, %d],z "type": null,z "type": "%s",z"matrix_type": "sparse",z "data": [rrr&z "rows": [z{"id": z, "metadata": rz},z}],rz [%d,%d,%r],z],z "columns": [z{{"id": {}, "metadata": {}}},z{{"id": {}, "metadata": {}}}]z "rows": [],z "columns": []rq}z{%s})r4rvrrrFrGrrrrrnr-r.r`rrrrFrrXrrformat)r;rrrLrformat_ format_urlranum_rowsnum_colshas_data test_elementr_rnrr`ro max_row_idx max_col_idxrx have_written obs_indexr" built_row col_indexrr, samp_indexr!s r>to_jsonz Table.to_jsons&,,, G !EFF F  $LNN4466MM)3355M  3 OOC OOMC ,>,>> ? ? ? OO!.v667 8 8 8 OO%*,,- . . . OO3lB C C C OOOm; < < < <#dm"4"44C'*H++G.1K1M1MMJ2\AL"]2D $!% Hhh $"# #Hxxx#a<EE$q'NNE$q'NN 4 45555>EE$q'NNE$q'NN 4 45555 7k ! !c$ii1nn!?D&'Gwwt}}'''""   OOD ! ! ! OOG $ $ $ OOC BGG#  %    s ! E,,E4ct|ttfsYt|dr|}n4t|dr|}nt dd}|dd}t|dkrt d|gd krd }n#||d rd }nt d|s |d d}g}g}g}|D]}|d}t|dksJ| |d| |d | t|d tt|} tt|} dt| Ddt| Dtjfd|Dt"} tjfd|Dt"} tj|} t'| | | ff}t)|| | S)aParse an adjacency format into BIOM Parameters ---------- lines : list, str, or file-like object The tab delimited data to parse Returns ------- biom.Table A BIOM ``Table`` object Notes ----- The input is expected to be of the form: observation, sample, value. A header is not required, but if present, it must be of the form: #OTU IDSampleIDvalue Raises ------ ValueError If the input is not an iterable or file-like object. ValueError If the data is incorrectly formatted. Examples -------- Parse tab separated adjacency data into a table: >>> from biom.table import Table >>> from io import StringIO >>> data = 'a\tb\t1\na\tc\t2\nd\tc\t3' >>> data_fh = StringIO(data) >>> test_table = Table.from_adjacency(data_fh) readlines splitlinesz!Not sure how to handle this inputctjd}||}|\}}||z t |krdSdS)Nz0(?=.)([+-]?([0-9]*)(\.([0-9]+))?)([eE][+-]?\d+)?TF)recompilematchspanrr)rnumericrrrs r>is_numz$Table.from_adjacency..is_numjsSj!TUUGMM$''E**,,KE4u T**tur?rr|z)Does not appear to be an adjacency format)r}SampleIDrYFrTr&Nci|]\}}|| Sr^r^)rbros r>rz(Table.from_adjacency..s;;;daQ;;;r?ci|]\}}|| Sr^r^)rbrrs r>rz(Table.from_adjacency..s===tq!a===r?c g|] }| Sr^r^)rbr"rvs r>rez(Table.from_adjacency..s???3 #???r?rc g|] }| Sr^r^)rbr!rys r>rez(Table.from_adjacency..s===T 4(===r?)r4rxryrr|r}rrrrrrXr.rrtrFr5rr-rrr)linesrlhinclude_line_zero observationssamplesrliner obs_order samp_orderrrrorrvrys @@r>from_adjacencyzTable.from_adjacency<sL%$// Fuk** F)) -- F((** !DEEE   1X^^   # #D ) ) r77a<<HII I 333 3 3 %   VBqE]] J!%  HII I  !""IE  + +DJJt$$Eu::????   a ) ) ) NN58 $ $ $ MM%a// * * * *3|,,-- CLL)) ;;i &:&:;;; ==y'<'<=== h????,???sKKKh====W===SIIIz&!!$c +,,S)Z000r?c  tj|fi|\}}}} |d} n fd|D} d} nfd|D} fd|D} t|||| | S)ayParse a tab separated (observation x sample) formatted BIOM table Parameters ---------- lines : list, or file-like object The tab delimited data to parse obs_mapping : dict or None The corresponding observation metadata sample_mapping : dict or None The corresponding sample metadata process_func : function A function to transform the observation metadata Returns ------- biom.Table A BIOM ``Table`` object Examples -------- Parse tab separated data into a table: >>> from biom.table import Table >>> from io import StringIO >>> tsv = 'a\tb\tc\n1\t2\t3\n4\t5\t6' >>> tsv_fh = StringIO(tsv) >>> func = lambda x : x >>> test_table = Table.from_tsv(tsv_fh, None, None, func) Nc*g|]}|iSr^r^)rbrL process_func t_md_names r>rez"Table.from_tsv..s&GGGQY Q8GGGr?c g|] }| Sr^r^)rb sample_idsample_mappings r>rez"Table.from_tsv..s.<<<#, .i8<<rez"Table.from_tsv..sFFFFK/FFFr?)r_extract_data_from_tsv) rrrrrrrrot_mdrrrs ``` @r>from_tsvzTable.from_tsvsB25CCFCC WdD  <LLGGGGG$GGGL  !"OO<<<<0:<<isfloatz-Table._extract_data_from_tsv..isfloats9 e t   uu s  !!seekz+Input needs to support seek or be indexableFr#r&Ncg|]}|Sr^r^)rbrs r>rez0Table._extract_data_from_tsv..&s333TD333r?clg|]0}|dd1S)r&)rsplitr)rbrrs r>rez0Table._extract_data_from_tsv..)sH111{{5!,,R06688111r?c&g|] }|Sr^r^)rbrrs r>rez0Table._extract_data_from_tsv..+s!%F%F%FQggajj%F%F%Fr?rz-Invalid value on line %d, column %d, value %s)r4rxrr RuntimeErrorr startswithrstriprrrreadlinerrFrXrrrMrrr)rrrGmd_parserd list_index data_startr value_checksr last_valueslast_column_is_numericmd_namerrror row_numberlinenorHrrIrJmsg column_numberrs ` @r>rzTable._extract_data_from_tsvsR   %&& C Cv&&&&! C C C"ACCC C    3 3D::<< ??3'' ,![[]]0077;F!+aJJ!+J !OJZZ\\''..qrr2FF eT " " 4 -LL JJqMMMq*-- ! !    33U333L1111#/111 !$%F%F%F%F+%F%F%F!G!G " #Z1__GHaaayHHRjGHcrc{H  5& ! ! ' JJqMMMq*-- ( (~~'' (*++&E%eZ88! ! LFD::<< s## ZZ&&F))++F2J NN6!9 % % %% 0F!#eVABBZ"8"899FF!FFF%8qrr %K%KNFFIC#C66!8V*D$DEEEF F!#eVAbD\":":;;FF!FFF%8qrr %K%KNFFIC#C66!8V*D$DEEEF 'OOHHVBZ$8$89999OOF2J///!&q#f++!6!6 9 9 -(EE!HH44KK]!' !6!8999 !OJJ$'99s#-A%J>>AK>%L((AM(c8|d|||||S)aReturn self as a string in tab delimited form Default ``str`` output for the ``Table`` is just row/col ids and table data without any metadata Parameters ---------- header_key : str or ``None``, optional Defaults to ``None`` header_value : str or ``None``, optional Defaults to ``None`` metadata_formatter : function, optional Defaults to ``str``. a function which takes a metadata entry and returns a formatted version that should be written to file observation_column_name : str, optional Defaults to "#OTU ID". The name of the first column in the output table, corresponding to the observation IDs. direct_io : file or file-like object, optional Defaults to ``None``. Must implement a ``write`` function. If `direct_io` is not ``None``, the final output is written directly to `direct_io` during processing. Returns ------- str tab delimited representation of the Table Examples -------- >>> import numpy as np >>> from biom.table import Table Create a 2x3 BIOM table, with observation metadata and no sample metadata: >>> data = np.asarray([[0, 0, 1], [1, 3, 42]]) >>> table = Table(data, ['O1', 'O2'], ['S1', 'S2', 'S3'], ... [{'foo': 'bar'}, {'x': 'y'}], None) >>> print(table.to_tsv()) # doctest: +NORMALIZE_WHITESPACE # Constructed from biom file #OTU ID S1 S2 S3 O1 0.0 0.0 1.0 O2 1.0 3.0 42.0 >>> with open("result.tsv", "w") as f: table.to_tsv(direct_io=f) r|)rrr)r;rrrrrs r>to_tsvz Table.to_tsvgs1f""4\#5#:-6#88 8r?) NNNNNNNNTNN)r)FN)Nr)r)r)rQrQ)rTT)Nr)rT)Tr)TrTF)rFT) NTr&TFrrFr)rFFN)T)rTr;)rT)r[)NrNT)F)Fr2T)TNN)NF)NN)^r@rArB__doc__rrrr staticmethodrr.rrpropertyrnrGrrrrr rr$r!r r3rCrrIrrVrXrrergrrrprtrzrrvrsrrrrrwrrrrrorNrGrrr=r rr/rrrrr rr3r9r8rArrIrOrVrYr0rerrr!r classmethodr)r1r:r7rUrfrzrrrrr^r?r>rrsNNb=AJNHLEI 99999999v 0 0 0))))2<<<<))\) %*%<7<7<7\<7|$;$;$;L  X   X X XDDDD4))))8A6A6A6A6F8@(@(@(D***"***$6666p6666p5J5J5Jn8888tJJJ094949494v.).).).)`+)+)+)+)ZVVVVp))))*:3:3:3:3x///b#3#3#3J%%%    ',',',',R$(Dt*-/8DU!U!U!U!n   ...###  %%%**>###++++Z...3)3)3)3)j6)6)6)6)pQCQCQCQCfG)G)G)G)R">G>G>G>G@rrrrhX)X)X)X)tFG>C%>%>%>N1111fGLGLGLREI"_<_<_<_rrs }# tTQTQVV$4E B B BF \\^^F  Mr?ct|\}}}|%t|dz}t|dz}n|\}}t|||ff||f|}|}||S)aConvert a list of lists into a scipy.sparse matrix. Parameters ---------- data : iterable of iterables `data` should be in the format [[row, col, value], ...] dtype : type, optional defaults to ``float`` shape : tuple or ``None``, optional Defaults to ``None``. If `shape` is ``None``, shape will be determined automatically from `data`. Returns ------- scipy.csr_matrix The newly generated matrix Nr&r)r{rrrr) rorGrnrxryrrrrs r>rrs$dD$ }TQTQ $.vv6F#%%%F \\^^F  Mr?cZ|jdkrtd|S|jdvr|jdkrtd|St|jdkrd|jdf}n|j}t |||}|}||S)aJConvert a numpy array to a scipy.sparse matrix. Parameters ---------- data : numpy.array The data to convert into a sparse matrix dtype : type, optional Defaults to ``float``. The type of data to be represented. Returns ------- scipy.csr_matrix The newly generated matrix rrr)ri)rr&rr&r)rnrr^rrrrr)rorGrnrs r>rrs zT&.... ' ' 'DINN &.... TZA  DJqM"  E 7 7 7F \\^^F  Mr?ct|t|t|df|}|}||S)afTakes a list of numpy arrays and creates a scipy.sparse matrix. Parameters ---------- data : iterable of numpy.array The data to convert into a sparse matrix dtype : type, optional Defaults to ``float``. The type of data to be represented. Returns ------- scipy.csr_matrix The newly generated matrix rr)rrrrr)rorGrs r>rrsQSYYDG $=U K K KF \\^^F  Mr?ct|drn|djd|djdkr#t|}|djd}nt|}|djd}ntd|D}t |t dddz}t |t dddz}||krt|}nt|}t |}t|||f|}|}| |S)avTakes a list of scipy.sparse matrices and creates a scipy.sparse mat. Parameters ---------- data : iterable of scipy.sparse matrices The data to convert into a sparse matrix dtype : type, optional Defaults to ``float``. The type of data to be represented. Returns ------- scipy.csr_matrix The newly generated matrix rr&c6g|]}|Sr^rrbrs r>rez)list_sparse_to_sparse..4 333AFFHH333r?rr) rrnrrrrr rrrr)rorGrrall_keysrs r>rrsC$q' 7= d1gmA. . .YYF!W]1%FFYYF!W]1%FF33d33344X:a==111!4q8X:a==111!4q8 F??YYFFYYF $<rez'list_dict_to_sparse..]rr?rr) rrnrrrrr rFrrXrrr)rorGis_colrrrrxryrr'rrow_valr+rrs r>rrDs$q' 7= d1gmA. . .FYYF!W]1%FFFYYF!W]1%FF33d33344X:a==111!4q8X:a==111!4q8 F??FYYFFFYYF D D D!$ ! ! '*yy{{ ! ! # Wg ! G$$$ G$$$ C     G$$$ G$$$ C     !d|,VV4D#%%%F \\^^F  Mr?c|st|tdddz}t|tdddz}n|\}}g}g}g}|D]G\\}} } |||| || Ht |||ff||f|S)aSTakes a dict {(row,col):val} and creates a scipy.sparse matrix. Parameters ---------- data : dict The data to convert into a sparse matrix dtype : type, optional Defaults to ``float``. The type of data to be represented. Returns ------- scipy.csr_matrix The newly generated matrix Nrrr&r)rrr rrXr) rorGrnrrrxryrrrrLs r>rr}s }TYY[[jmm444Q7!;TYY[[jmm444Q7!; D D DZZ\\ A A A A d| 4'-v&6e E E EEr?)Trnumpyr5 scipy.statsr?rrNrrjsonr _json_dumpsr functoolsrroperatorr r collectionsr collections.abcr r rrrr scipy.sparserrrrrrrpandasrrbiom.exceptionrrrr biom.utilrrrr r!r"r#r$biom.errr%r'r(r) __author__ __copyright__ __credits__ __license____url____maintainer__ __email__r-r.rvrbr1rMrVr[rrzrrrrrrrrrr^r?r>rs-bbX 22222222%%%%%%%%$$$$$$$$######........222222222222666666666666666666 ------------++++++++++++++++++++"""""""""""" G :::  "" * "Ec!EcCC 33333 333  +++:   , , ,+%+%+%\>!>!>!BVL8VL8VL8VL8VL8VL8VL8VL8rX&(ZtB%*B#(####L(-*',$$$$N%*6666r %DEEEEEEr?