U \)_f!@sdZddlmZddlmZmZmZddlmZ ddl m Z m Z m Z deeeedd d Zeeeeeeefd d d ZdeeeedddZdeeeedddZdeeeeeeefdddZdS)a  Chunked reading of FASTA and FASTQ files This can be used to very quickly split up the input file into similarly-sized chunks, without actually parsing the records. The chunks can then be distributed to worker threads or subprocess and be parsed and processed there. ) RawIOBase)OptionalIteratorTuple)paired_fastq_heads)FileFormatErrorFastaFormatErrorUnknownFileFormatN)bufendreturncCsx|dd|}|dkr|dS|dddks>|dddkrBdSt|dkrRdSt|d}tdt|dd dS) z Search for the end of the last complete FASTA record within buf[:end] Return an integer length such that buf[:length] contains the highest possible number of complete FASTA records.  >rr>#z1FASTA file expected to start with '>', but found Nline)rfindlenchrr repr)r r poscr+lib/python3.8/site-packages/dnaio/chunks.py _fasta_heads    r)buf1buf2end1end2r c Cs|dks|dkrdS|dkr,|dddksD|dkrP|dddkrPtddd|dd|}|dd|}t||}d}}|dkr|d||d}|d||d}|d8}q~||fS) z Return positions pos1, pos2 where right1 <= end1 and right2 <= end2 such that buf1[:pos1] and buf2[:pos2] contain the same number of complete FASTA records. r)rrNrrz%FASTA file expected to start with '>'rr)r countminfind) rrrr Z n_records1Z n_records2Z n_recordsZpos1Zpos2rrr_paired_fasta_heads%s0   r$cCs>|dd|}|}t|ddD]}|dd|}q"|dS)z Search for the end of the last complete *two* FASTQ records in buf[:end]. Two FASTQ records are required to ensure that read pairs in interleaved paired-end data are not split.  rr)r!ranger)r r Z linebreaksright_rrr _fastq_headAs r*@)f buffer_sizer ccsBt|}|t|dd}|dkr*dS|dks6t|dddkrLt}n@|dddksl|dddkrrt}ntdtt|d|t |krt d|t||d|}||krƐq"|||}||kst|dkrt|d|V||}|dks t||||d|<q|dkr>t|d|VdS) aE Read chunks of complete FASTA or FASTQ records from a file. If the format is detected to be FASTQ, all chunks except possibly the last contain an even number of records such that interleaved paired-end reads remain in sync. The yielded memoryview objects are only valid for one iteration because the internal buffer is re-used in the next iteration. Arguments: f: File with FASTA or FASTQ reads; must have been opened in binary mode buffer_size: Largest allowed chunk size Yields: memoryview representing the chunk. This becomes invalid on the next iteration. Raises: ValueError: A FASTQ record was encountered that is larger than *buffer_size*. UnknownFileFormat: The file format could not be detected (the first byte must be "@", ">" or "#") rrN@rrzZCannnot determine input file format: First character expected to be '>' or '@', but found z+FASTA/FASTQ record does not fit into buffer) bytearrayreadinto memoryviewAssertionErrorr*rr rrr OverflowError)r,r-r startheadZbufendr rrr read_chunksRs4      r6)r,f2r-r ccs|dkrtdt|}t|}|t|dd}|t|dd}|dkrl|dkrltdtdfS|dk|dkkr|dkrdnd}td|dd d |d dd kr|d dkrd krnn td d d |d dd krd }t} n6|d ddkrd}t} ntd|d dd d |t|krX|t|krXtd||t||d |} |t||d |} || kr|| krq| ||| | \} } | | kst| | kst| dks| dks|dkrt|d| t|d| fVnb| dkr(| dks,td}| dksD| dkrb| dkrRdnd}d|d}td|dd d | | }|dkst|| | |d|<| | }|dkst|| | |d|<q.|dks|dkrt|d|t|d|fVd S)a+ Read chunks of paired-end FASTA or FASTQ records from two files. A pair of chunks (memoryview objects) is yielded on each iteration, and both chunks are guaranteed to have the same number of sequences. That is, the paired-end reads will stay in sync. The memoryviews are only valid for one iteration because the internal buffer is re-used in the next iteration. This is similar to `read_chunks`, but for paired-end data. Args: f: File with R1 reads; must have been opened in binary mode f2: File with R2 reads; must have been opened in binary mode buffer_size: Largest allowed chunk size Yields: Pairs of memoryview objects. Raises: ValueError: A FASTA or FASTQ record was encountered that is larger than *buffer_size*. zBuffer size too smallrrz)Paired-end reads not in sync: File with Rz$ reads is empty and the other is notNrr.zAPaired-end data must be in FASTQ format when using multiple coresZFASTQrZFASTAzLFirst character in input file must be '@' (FASTQ) or '>' (FASTA), but found z3FASTA/FASTQ records do not fit into buffer of size z. File z- ended, but more data found in the other filez!Premature end of paired-end input.) ValueErrorr/r0r1r_paired_fastq_headsr$rr2)r,r7r-rrZstart1Zstart2iZ file_formatZ paired_headsZbufend1Zbufend2rr Zextrarrrread_paired_chunkssv 2$  r@)N)N)r+)r+)__doc__iortypingrrrZ_corerr> exceptionsrr r bytesintrr$r*r1r6r@rrrrs(   K