\documentclass[10pt]{article}
\usepackage{color}
\definecolor{gray}{rgb}{0.7,0.7,0.7}
\usepackage{framed}
\usepackage{enumitem}
\usepackage{longtable}
\usepackage[pdfborder={0 0 0},hyperfootnotes=false]{hyperref}

\addtolength{\textwidth}{3.4cm}
\addtolength{\hoffset}{-1.7cm}
\addtolength{\textheight}{4cm}
\addtolength{\voffset}{-2cm}


\begin{document}

\title{metadamage(?) formats}
\author{tsk}
\maketitle
\vspace*{1em}

This document describe some of the internal formats used by the metadamage software. These are at the current time.
\begin{itemize}
\item .bdamage.gz
\item .lca
\item .stat
\item .dfit.txt.gz
\item .agggregate.stat.txt.gz
\item .rlens.gz
\end{itemize}

\section{bdamage format}
bdamage files are files that contain counts of mismatchs conditional on strand and cycle (position within read). These are generated with metadamage lca or metadamage getdamage. 
The first 8 bytes magic number determines which bdamage version. If no magic number is present then version0 is assumed. 

\subsection{version 0}
First version of the bdamage file is a single bgzf compressed file. MAXLENGTH occurs once in the beginning of the file, followed by succesive blocks of data[1-8]. Block[3-5] indicates the actual mismatch counts for the forward which we have MAXLENGTH times. Block[6-8] indicates the actual mismatch counts for the reverse strand which will also occur MAXLENGTH times. 
\begin{table}[h!]
\begin{tabular}{rllll}
  \hline
  {\bf Col} & {\bf Field} & {\bf Type} & {\bf Brief description} \\
  \hline
  0 & {\sf MAXLENGTH} & int &  Number of cycles\\\hline
  1 & {\sf ID} & int &  Id for mismatch type$^1$\\
  2 & {\sf NREADS} & int & Number of reads used supporting the mismatch matrix\\\hline\hline
  3 & {\sf 1} & float[16] & mismatch rate for first cycle from the 5prime\\
  4 & {\sf $i$} & float[16] & mismatch rate for the $i$'th cycle from the 5prime\\
  5 & {\sf MAXLENGTH} & float[16] & mismatch rate for the last cycle from the 5prime\\\hline
  6 & {\sf 1} & float[16] & mismatch rate for first cycle from the 3prime\\
  7 & {\sf $i$} & float[16] & mismatch rate for the $i$'th cycle from the 3prime\\
  8 & {\sf MAXLENGTH} & float[16] & mismatch rate for the last cycle from the 3prime\\\hline
  \hline
\end{tabular}\label{tab1}
\caption{Content of bdamage.gz file. Note$^1$ This is either the taxidID or the referenceID relative to the SAM/BAM header for single species resequencing projects or it is the \textit{taxid} if output has been generated with metadamage lca 3) Order is given by AA,AC,AG,AT,CA,CC,CG,CT,GA,GC,GG,GT,TA,TC,TG,TT, with first base indicatting reference nucleotide and second base indicating observed nucleotide}.
\end{table}

\subsection{version 1}
Second iteration of the bdamage file as magicnumber[8] = "bdam1".
\clearpage
\section{.lca} \label{sec:lca}
This section describes the test output generated by a metadamage lca subfunctionality and contains information at the readlevel regarding both taxonomic information and statistics pertaining usefull readinformation.\\

First line of the file begins with a hashtag followed by the actual command used for generating the file. Last entry of the line is again a hashtag followed by the git commit id which will serve as a primitive versioncontrol.

Each line consists of a number of items seperated by tabspace. First entry contains readID together with other information seperated by colon. After the first entry succesive blocks of the type taxid:name:''taxlevel'' from the lca toward the root. The complete specification is seen in table below.

\begin{table}[h]
\begin{tabular}{rll}
  \hline
  {\bf Col} & {\bf Brief description} \\
  \hline
  1 & {\sf readID} & readID, this might contains colon\\
  2 & {\sf seq} & The actual sequence\\
  3 & {\sf length(seq)} & The length of the sequence\\
  4 & {\sf nAlignments} & The number of alignments used for inferring the lca\\
  5 & {\sf gc-content} & The GC content for the sequence\\\hline\hline
  6 & {\sf lca taxid} & the taxomic id (integer)\\
  7 & {\sf lca taxid} & the taxonomic name(string)\\
  8 & {\sf lca ``taxlevel''} & the taxonomic level\\\hline\hline
  9 & {\sf taxid} & the taxomic id (integer)\\
  10 & {\sf taxid} & the taxonomic name(string)\\
  11 & {\sf ``taxlevel''} & the taxonomic level\\\hline\hline
\end{tabular}\label{tab2}
\caption{Content of a .lca file. Note that 1) seperate between
  fields[1-6,8-9] is tab, readID might also contain colon. 2) the
  quotes around field[7,8] is intentional since taxlevels and names
  might contain spaces andor colons. 3) Number of tab seperated
  entries is consistent across different reads, since seperator
  between block 9-11 is semicolon. Seperator between 6-9,9-11 is colon}
\end{table}
\section{.stat}\label{sec:stat}
Very simple tabseperated flatfile
\begin{enumerate}
\item taxid
\item Number of supporting reads
\item Mean lengths of supporting reads
\item Variance of the lengths of the supporting reads
\item Mean gccontent of supporting reads
\item Variance of the gccontent of supporting reads
\item name of lca in quotes (if relevant, otherwise NA)
\item name of taxomic level of lca in quotes (if relevant, otherwise NA)
\end{enumerate}
\clearpage
\section{.dfit.txt.gz}
The output format will be heavily dependent on the provided runmode and supplied parameters. It might contain the
per file, the per reference or the per species estimate of damage.\\ \noindent
Only the columns described in \textit{showfits 0} remains the same across the runmode, with the \textit{showfits 1} or \textit{showfits 2} describes the unique columns and order which is appended to the \textit{showfits 0} columns.\\ \noindent
\textbf{Non-boostrapping optimization:} \\ \noindent

\begin{table}[h]
\begin{tabular}{rll}
  \hline
  {\bf Column name} & {\bf Brief description} \\
  \hline
  {\sf } & \textit{--showfits} 0 \\ \hline  
  {\sf id} & identifier see paragraph for details\\
  {\sf A} & Dfit statistic. Damage at position one, taking into account offset\\
  {\sf q} & per cycle decrease\\
  {\sf c} & background substitution rate or noise baseline\\ 
  {\sf $\phi$} & Variance between beta-binomial and binomial model\\
  {\sf llh} & Likelihood for our MLE\\
  {\sf ncall} & Number of optimization calls used for obtaining our MLE\\
  {\sf $\sigma_D$} & Z value\\
  {\sf Zfit} & significance\\\hline 
  {\sf } & \textit{--showfits} 1, \textit{i} signifies position inferred from \textit{.bdamage.gz}\\ \hline  
  {\sf $fwdxi$} & damage estimates forward strand \\
  {\sf $fwdConfi$} & forward strand confidence interval $\pm$\\
  {\sf $bwdxi$} & damage estimates backward strand\\
  {\sf $bwdConfi$} & backward strand confidence interval $\pm$\\ \hline
  {\sf } & \textit{--showfits} 2, \textit{i} signifies position inferred from \textit{.bdamage.gz} \\ \hline  
  {\sf $fwKi$} & Number of deamination substitution (C$\rightarrow$T) observations forward strand \\
  {\sf $fwNi$} & Total number of  C for forward strand\\
  {\sf $fwdxi$} & Damage estimates forward strand \\
  {\sf $fwfi$} & Calculated damage frequency $fwKi$/$fwNi$\\
  {\sf $fwdConfi$} & forward strand confidence interval $\pm$\\
  {\sf $bwKi$} & Number of deamination substitution (G$\rightarrow$A for ds, C$\rightarrow$T for ss) observations forward strand \\
  {\sf $bwNi$} & Total number of G for forward strand (ds) and C (ss)\\
  {\sf $bwdxi$} & Damage estimates backward strand \\
  {\sf $bwfi$} & Calculated damage frequency $fwKi$/$fwNi$\\
  {\sf $bwdConfi$} & backward strand confidence interval $\pm$\\ \hline
\end{tabular}\label{tab3}
\caption{Content of a .dfit.txt.gz file. Note that entry nine to 13 is
  repeated for each cycle first of the 5\' and then from the 3\'. With
  the total number of times repeated is given by \ref{tab1}.} 
\end{table}
\pagebreak
\noindent
\textbf{Boostrapping optimization:} \\ \noindent
Providing \textit{-nbootstrap} $>$ 1 numerical optimizations of the binomial distribution will also be conducted using bootstrapping methods
\begin{table}[h]
\begin{tabular}{rll}
  \hline
  {\bf Column name} & {\bf Brief description} \\
  \hline
  {\sf } & \textit{--showfits} 0 \\ \hline  
  {\sf id} & identifier see paragraph for details\\
  {\sf A} & Dfit statistic. Damage at position one, taking into account offset\\
  {\sf q} & per cycle decrease\\
  {\sf c} & background substitution rate or noise baseline\\ 
  {\sf $\phi$} & Variance between beta-binomial and binomial model\\
  {\sf llh} & Likelihood for our MLE\\
  {\sf ncall} & Number of optimization calls used for obtaining our MLE\\
  {\sf $\sigma_D$} & Z value\\
  {\sf Zfit} & significance\\
  {\sf A\_b} & Dfit statistic from bootstrap estimate. Damage at position one, taking into account offset\\
  {\sf q\_b} & per cycle decrease from bootstrap estimate\\
  {\sf c\_b} & background substitution rate or noise baseline from bootstrap estimate\\ 
  {\sf $\phi$\_b} & Variance between beta-binomial and binomial model from bootstrap estimate\\
  {\sf A\_CI\_l} & Lower bound of CI for A estimate calculated from all bootstrap values \\
  {\sf A\_CI\_h} & Upper bound of CI for A estimate calculated from all bootstrap values \\
  {\sf q\_CI\_l} & Lower bound of CI for q estimate calculated from all bootstrap values \\
  {\sf q\_CI\_h} & Upper bound of CI for q estimate calculated from all bootstrap values \\
  {\sf c\_CI\_l} & Lower bound of CI for c estimate calculated from all bootstrap values \\
  {\sf c\_CI\_h} & Upper bound of CI for c estimate calculated from all bootstrap values \\
  {\sf $\phi$\_CI\_l} & Lower bound of CI for $\phi$ estimate calculated from all bootstrap values \\
  {\sf $\phi$\_CI\_h} & Upper bound of CI for $\phi$ estimate calculated from all bootstrap values \\ \hline
  {\sf } & \textit{--showfits} 1, \textit{i} signifies position inferred from \textit{.bdamage.gz}\\ \hline  
  {\sf $fwdxi$} & damage estimates forward strand \\
  {\sf $fwdConfi$} & forward strand confidence interval $\pm$\\
  {\sf $bwdxi$} & damage estimates backward strand\\
  {\sf $bwdConfi$} & backward strand confidence interval $\pm$\\ \hline
  {\sf } & \textit{--showfits} 2, \textit{i} signifies position inferred from \textit{.bdamage.gz} \\ \hline  
  {\sf $fwKi$} & Number of deamination substitution (C$\rightarrow$T) observations forward strand \\
  {\sf $fwNi$} & Total number of  C for forward strand\\
  {\sf $fwdxi$} & Damage estimates forward strand \\
  {\sf $fwfi$} & Calculated damage frequency $fwKi$/$fwNi$\\
  {\sf $fwdConfi$} & forward strand confidence interval $\pm$\\
  {\sf $bwKi$} & Number of deamination substitution (G$\rightarrow$A for ds, C$\rightarrow$T for ss) observations forward strand \\
  {\sf $bwNi$} & Total number of G for forward strand (ds) and C (ss)\\
  {\sf $bwdxi$} & Damage estimates backward strand \\
  {\sf $bwfi$} & Calculated damage frequency $fwKi$/$fwNi$\\
  {\sf $bwdConfi$} & backward strand confidence interval $\pm$\\ \hline
\end{tabular}\label{tab4}
\caption{Content of a .dfit.txt.gz file. Note that entry nine to 13 is
  repeated for each cycle first of the 5\' and then from the 3\'. With
  the total number of times repeated is given by \ref{tab1}.} 
\end{table}

Depending on which parameters and runmode (local,global or lca)that was supplied to both
\emph{getdamage} and \emph{dfit}, the content of the id will be
different. If a bamfile is supplied (with --bam) then each line will be
the information associated with the different refids in the bam file
and the id will be the referenceids from the bam file. The case
scenario for this would be either obtaining perchromosome estimates of
damage or per reference damage which could be relevant for metagenomic
studies. If user are computing the damagesignal in the context of the
lca. Then the id column will contain the taxid. If -names has been
supplied to the dfit program, then the id column will the
taxid:\emph{scientific name}. If -nodes has not been defined the
dfit.txt.gz will only contain information for the observed
references. If -nodes has been defined the program will aggregate the
summary statitics for the internal nodes.


\section{.boot.stat.txt.gz}
Providing \textit{dfit} command with \textit{nbootstrap $>$ 1} and
\textit{doboot 1} the boostrapping values
(id,A\_b,q\_b,c\_b,$\phi$\_b) for each iteration are stored in separate file

\section{.agggregate.stat.txt.gz}
Aggregates the information stored within the lca \textit{.stat} format, described in section \ref{sec:lca}. The aggregated file has the prefix \textit{.aggregate.stat.txt.gz}, with a similar format to the format presented for the lca output in section \ref{sec:lca}. 
\begin{table}[h]
\begin{tabular}{rll}
  \hline
  {\bf Col} & {\bf Brief description} \\
  \hline
  1 & {\sf taxid} & The lca taxomic id (integer)\\
  2 & {name} & The lca taxonomic name(string)\\
  3 & {rank} & The lca taxonomic rank/level (string)\\
  4 & {nalign} & The aggregated number of alignments used \\ & & for inferring the lca across the taxonomic levels\\
  5 & {nreads} & The aggregated number of reads \\ & & for inferring the lca across the taxonomic levels\\
  6 & {mean\_rlen} & The weighted mean of read length \\ & & when transvering through the taxonomic levels\\
  7 & {var\_rlen} & The pooled variance of read length \\ & & when transvering through the taxonomic levels\\
  8 & {mean\_gc} & The weighted mean of gc content \\ & & when transvering through the taxonomic levels\\
  9 & {var\_gc} & The pooled variance of gc content \\ & & when transvering through the taxonomic levels\\
  10 & {lca} & The lca rank \\
  11 & {taxa\_path} & The taxonimcal path from lca to root \\ \hline
\end{tabular}\label{aggregate}
\caption{Content of a .aggregate.stat.txt.gz file. With column 10 containing taxid and name for the lca node separated by colon, and column 11 contains the taxid":"name for all nodes from the lca to the root, with each node separated by comma, i.e. taxid":"name,taxid":"name,taxid":"name}
\end{table}

\section{.rlens.gz}
Readlength distribution. Distribution is count of alignments of
specific readlengths. Depending on runmode there might be multiple
groups as taxid/refs and there will be group specific
distributions. Distributions for different groups are split by
newlines. First entry on each line is the identifer
(chromosomename,taxid). The remaining entries are the number of times
we have observed an alignment of length (columnnumber). Notice that
the first 30 column of counts is likely to be zero since reads shorter
than 30basepairs are normally discarded.
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
