U  e@sdZddlmZddlmZmZmZddlmZ ddl m Z m Z m Z deeeedd d Zdeeeedd d ZdeeeedddZdeeeeeeefdddZdS)a  Chunked reading of FASTA and FASTQ files This can be used to very quickly split up the input file into similarly-sized chunks, without actually parsing the records. The chunks can then be distributed to worker threads or subprocess and be parsed and processed there. ) RawIOBase)OptionalIteratorTuple)paired_fastq_heads)FileFormatErrorFastaFormatErrorUnknownFileFormatN)bufendreturncCsx|dd|}|dkr|dS|dddks>|dddkrBdSt|dkrRdSt|d}tdt|dd dS) z Search for the end of the last complete FASTA record within buf[:end] Return an integer length such that buf[:length] contains the highest possible number of complete FASTA records. s >rr>#z1FASTA file expected to start with '>', but found Nline)rfindlenchrr repr)r r poscr+lib/python3.8/site-packages/dnaio/chunks.py _fasta_heads    rcCs>|dd|}|}t|ddD]}|dd|}q"|dS)z Search for the end of the last complete *two* FASTQ records in buf[:end]. Two FASTQ records are required to ensure that read pairs in interleaved paired-end data are not split.  rr)countranger)r r Z linebreaksright_rrr _fastq_head%s r"@)f buffer_sizer ccsBt|}|t|dd}|dkr*dS|dks6t|dddkrLt}n@|dddksl|dddkrrt}ntdtt|d|t |krt d|t||d|}||krƐq"|||}||kst|dkrt|d|V||}|dks t||||d|<q|dkr>t|d|VdS) aE Read chunks of complete FASTA or FASTQ records from a file. If the format is detected to be FASTQ, all chunks except possibly the last contain an even number of records such that interleaved paired-end reads remain in sync. The yielded memoryview objects are only valid for one iteration because the internal buffer is re-used in the next iteration. Arguments: f: File with FASTA or FASTQ reads; must have been opened in binary mode buffer_size: Largest allowed chunk size Yields: memoryview representing the chunk. This becomes invalid on the next iteration. Raises: ValueError: A FASTQ record was encountered that is larger than *buffer_size*. UnknownFileFormat: The file format could not be detected (the first byte must be "@", ">" or "#") rrN@rrzZCannnot determine input file format: First character expected to be '>' or '@', but found z+FASTA/FASTQ record does not fit into buffer) bytearrayreadinto memoryviewAssertionErrorr"rr rrr OverflowError)r$r%r startheadZbufendr rrr read_chunks6s4      r.)r$f2r%r c csN|dkrtdt|}t|}|t|dd}|t|dd}|dkrd|dddks||dkr|dddkrtddd|t|kr|t|krtd ||t||d|}|t||d|}||kr||krqt||||\} } | |kst| |ks$t| dks8| dkr\t|d| t|d| fVnb| dkrp| dksttd } |dks|dkr|dkrdnd } d | d } td| ddd|| }|dkst|| ||d|<|| }|dkst|| ||d|<q|dks(|dkrJt|d|t|d|fVdS)aR Read chunks of paired-end FASTQ reads from two files. A pair of chunks (memoryview objects) is yielded on each iteration, and both chunks are guaranteed to have the same number of sequences. That is, the paired-end reads will stay in sync. The memoryviews are only valid for one iteration because the internal buffer is re-used in the next iteration. This is similar to `read_chunks`, but for paired-end data. Unlike `read_chunks`, this only works for FASTQ input. Args: f: File with R1 reads; must have been opened in binary mode f2: File with R2 reads; must have been opened in binary mode buffer_size: Largest allowed chunk size Yields: Pairs of memoryview objects. Raises: ValueError: A FASTQ record was encountered that is larger than *buffer_size*. zBuffer size too smallrrr&zAPaired-end data must be in FASTQ format when using multiple coresNrz-FASTQ records do not fit into buffer of size z. File z- ended, but more data found in the other filez#Premature end of paired FASTQ input.) ValueErrorr'r(r)rr_paired_fastq_headsr*) r$r/r%Zbuf1Zbuf2Zstart1Zstart2Zbufend1Zbufend2Zend1Zend2Zextrairrrread_paired_chunks~sR0$  r7)N)N)r#)r#)__doc__iortypingrrrZ_corerr5 exceptionsrr r bytesintrr"r)r.r7rrrrs  K