""" Taxdump format (:mod:`skbio.io.format.taxdump`) =============================================== .. currentmodule:: skbio.io.format.taxdump The NCBI Taxonomy database dump (``taxdump``) format stores information of organism names, classifications and other properties. It is a tabular format with a delimiter: ```` between columns, and a line end ```` after all columns. The file name usually ends with .dmp. Format Support -------------- **Has Sniffer: No** +------+------+---------------------------------------------------------------+ |Reader|Writer| Object Class | +======+======+===============================================================+ |Yes |No |:mod:`pandas.DataFrame` | +------+------+---------------------------------------------------------------+ Format Specification -------------------- **State: Experimental as of 0.5.8.** The NCBI taxonomy database [1]_ [2]_ hosts organism names and classifications. It has a web portal [3]_ and an FTP download server [4]_. It is also accessible using E-utilities [5]_. The database is being updated daily, and an archive is generated every month. The data release has the file name ``taxdump``. It consists of multiple .dmp files. These files serve different purposes, but they follow a common format pattern: - It is a tabular format. - Column delimiter is ````. - Line end is ````. - The first column is a numeric identifier, which usually represent taxa (i.e., "TaxID"), but can also be genetic codes, citations or other entries. The two most important files of the data release are ``nodes.dmp`` and ``names.dmp``. They store the hierarchical structure of the classification system (i.e., taxonomy) and the names of organisms, respectively. They can be used to construct the taxonomy tree of organisms. The definition of columns of each .dmp file type are taken from [6]_ and [7]_. ``nodes.dmp`` ^^^^^^^^^^^^^ +----------------+-------------------------------------+ |Name |Description | +================+=====================================+ |tax_id |node id in GenBank taxonomy database | +----------------+-------------------------------------+ |parent tax_id |parent node id in GenBank taxonomy | | |database | +----------------+-------------------------------------+ |rank |rank of this node (superkingdom, | | |kingdom, ...) | +----------------+-------------------------------------+ |embl code |locus-name prefix; not unique | +----------------+-------------------------------------+ |division id |see division.dmp file | +----------------+-------------------------------------+ |inherited div |1 if node inherits division from | |flag (1 or 0) |parent | +----------------+-------------------------------------+ |genetic code id |see gencode.dmp file | +----------------+-------------------------------------+ |inherited GC |1 if node inherits genetic code from | |flag (1 or 0) |parent | +----------------+-------------------------------------+ |mitochondrial |see gencode.dmp file | |genetic code id | | +----------------+-------------------------------------+ |inherited MGC |1 if node inherits mitochondrial | |flag (1 or 0) |gencode from parent | +----------------+-------------------------------------+ |GenBank hidden |1 if name is suppressed in GenBank | |flag (1 or 0) |entry lineage | +----------------+-------------------------------------+ |hidden subtree |1 if this subtree has no sequence | |root flag |data yet | |(1 or 0) | | +----------------+-------------------------------------+ |comments |free-text comments and citations | +----------------+-------------------------------------+ Since 2018, NCBI releases "new taxonomy files" [8]_ (``new_taxdump``). The new ``nodes.dmp`` format is compatible with the classical format, plus five extra columns after all aforementioned columns. +----------------+-------------------------------------+ |Name |Description | +================+=====================================+ |plastid genetic |see gencode.dmp file | |code id | | +----------------+-------------------------------------+ |inherited PGC |1 if node inherits plastid gencode | |flag (1 or 0) |from parent | +----------------+-------------------------------------+ |specified\\_ |1 if species in the node's lineage | |species |has formal name | +----------------+-------------------------------------+ |hydrogenosome |see gencode.dmp file | |genetic code id | | +----------------+-------------------------------------+ |inherited HGC |1 if node inherits hydrogenosome | |flag (1 or 0) |gencode from parent | +----------------+-------------------------------------+ ``names.dmp`` ^^^^^^^^^^^^^ +----------------+-------------------------------------+ |Name |Description | +================+=====================================+ |tax_id |the id of node associated with this | | |name | +----------------+-------------------------------------+ |name_txt |name itself | +----------------+-------------------------------------+ |unique name |the unique variant of this name if | | |name not unique | +----------------+-------------------------------------+ |name class |(synonym, common name, ...) | +----------------+-------------------------------------+ ``division.dmp`` ^^^^^^^^^^^^^^^^ +----------------+-------------------------------------+ |Name |Description | +================+=====================================+ |division id |taxonomy database division id | +----------------+-------------------------------------+ |division cde |GenBank division code (three | | |characters) | +----------------+-------------------------------------+ |division name |e.g. BCT, PLN, VRT, MAM, PRI... | +----------------+-------------------------------------+ |comments | | +----------------+-------------------------------------+ ``gencode.dmp`` ^^^^^^^^^^^^^^^ +----------------+-------------------------------------+ |Name |Description | +================+=====================================+ |genetic code id |GenBank genetic code id | +----------------+-------------------------------------+ |abbreviation |genetic code name abbreviation | +----------------+-------------------------------------+ |name |genetic code name | +----------------+-------------------------------------+ |cde |translation table for this genetic | | |code | +----------------+-------------------------------------+ |starts |start codons for this genetic code | +----------------+-------------------------------------+ Other types of .dmp files are currently not supported by scikit-bio. However, the user may customize column definitions in using this utility. See below for details. Format Parameters ----------------- The following format parameters are available in ``taxdump`` format: - ``scheme``: The column definition scheme name of the input .dmp file. Available options are listed below. Alternatively, one can provide a custom scheme as defined in a name-to-data type dictionary. 1. ``nodes``: The classical ``nodes.dmp`` scheme. It is also compatible with new ``nodes.dmp`` format, in which case only the columns defined by the classical format will be read. 2. ``nodes_new``: The new ``nodes.dmp`` scheme. 3. ``nodes_slim``: Only the first three columns: tax_id, parent_tax_id and rank, which are the minimum required information for constructing the taxonomy tree. It can be applied to both classical and new ``nodes.dmp`` files. It can also handle custom files which only contains these three columns. 4. ``names``: The ``names.dmp`` scheme. 5. ``division``: The ``division.dmp`` scheme. 6. ``gencode``: The ``gencode.dmp`` scheme. .. note:: scikit-bio will read columns from leftmost till the number of columns defined in the scheme. Extra columns will be cropped. Examples -------- >>> from io import StringIO >>> import skbio.io >>> import pandas as pd >>> fs = '\\n'.join([ ... '1\\t|\\t1\\t|\\tno rank\\t|', ... '2\\t|\\t131567\\t|\\tsuperkingdom\\t|', ... '6\\t|\\t335928\\t|\\tgenus\\t|' ... ]) >>> fh = StringIO(fs) Read the file into a ``pd.DataFrame`` and specify that the "nodes_slim" scheme should be used: >>> df = skbio.io.read(fh, format="taxdump", into=pd.DataFrame, ... scheme="nodes_slim") >>> df # doctest: +NORMALIZE_WHITESPACE parent_tax_id rank tax_id 1 1 no rank 2 131567 superkingdom 6 335928 genus References ---------- .. [1] Federhen, S. (2012). The NCBI taxonomy database. Nucleic acids research, 40(D1), D136-D143. .. [2] Schoch, C. L., Ciufo, S., Domrachev, M., Hotton, C. L., Kannan, S., Khovanskaya, R., ... & Karsch-Mizrachi, I. (2020). NCBI Taxonomy: a comprehensive update on curation, resources and tools. Database, 2020. .. [3] https://www.ncbi.nlm.nih.gov/taxonomy .. [4] https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/ .. [5] Kans, J. (2022). Entrez direct: E-utilities on the UNIX command line. In Entrez Programming Utilities Help [Internet]. National Center for Biotechnology Information (US). .. [6] https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_readme.txt .. [7] https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/new_taxdump/taxdump_readme.txt .. [8] https://ncbiinsights.ncbi.nlm.nih.gov/2018/02/22/new-taxonomy-files- available-with-lineage-type-and-host-information/ """ # ---------------------------------------------------------------------------- # Copyright (c) 2013--, scikit-bio development team. # # Distributed under the terms of the Modified BSD License. # # The full license is in the file LICENSE.txt, distributed with this software. # ---------------------------------------------------------------------------- import pandas as pd from skbio.io import create_format taxdump = create_format('taxdump') _taxdump_column_schemes = { 'nodes_slim': { 'tax_id': int, 'parent_tax_id': int, 'rank': str }, 'nodes': { 'tax_id': int, 'parent_tax_id': int, 'rank': str, 'embl_code': str, 'division_id': int, 'inherited_div_flag': bool, 'genetic_code_id': int, 'inherited_GC_flag': bool, 'mitochondrial_genetic_code_id': int, 'inherited_MGC_flag': bool, 'GenBank_hidden_flag': bool, 'hidden_subtree_root_flag': bool, 'comments': str }, 'names': { 'tax_id': int, 'name_txt': str, 'unique_name': str, 'name_class': str }, 'division': { 'division_id': int, 'division_cde': str, 'division_name': str, 'comments': str }, 'gencode': { 'genetic_code_id': int, 'abbreviation': str, 'name': str, 'cde': str, 'starts': str } } _taxdump_column_schemes['nodes_new'] = dict( _taxdump_column_schemes['nodes'], **{ 'plastid_genetic_code_id': bool, 'inherited_PGC_flag': bool, 'specified_species': bool, 'hydrogenosome_genetic_code_id': int, 'inherited_HGC_flag': bool }) @taxdump.reader(pd.DataFrame, monkey_patch=False) def _taxdump_to_data_frame(fh, scheme): '''Read a taxdump file into a data frame. Parameters ---------- fh : file handle Input taxdump file scheme : str Name of column scheme Returns ------- pd.DataFrame Parsed table ''' if isinstance(scheme, str): if scheme not in _taxdump_column_schemes: raise ValueError(f'Invalid taxdump column scheme: "{scheme}".') scheme = _taxdump_column_schemes[scheme] names = list(scheme.keys()) try: return pd.read_csv( fh, sep='\t\\|(?:\t|$)', engine='python', index_col=0, names=names, dtype=scheme, usecols=range(len(names))) except ValueError: raise ValueError('Invalid taxdump file format.')