c*b` ddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z ddlmZGddeZy) N)xrange) LineNumbers)Token match_tokenis_non_coding_token) MarkTokensceZdZdZddZdZdZedZedZ edZ ed Z d Z d Z d Zdd ZddZddZddZddZdZdZy) ASTTokensa ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and as tokens, and is used to mark and access token and position information. ``source_text`` must be a unicode or UTF8-encoded string. If you pass in UTF8 bytes, remember that all offsets you'll get are to the unicode text, which is available as the ``.text`` property. If ``parse`` is set, the ``source_text`` will be parsed with ``ast.parse()``, and the resulting tree marked with token info and made available as the ``.tree`` property. If ``tree`` is given, it will be marked and made available as the ``.tree`` property. In addition to the trees produced by the ``ast`` module, ASTTokens will also mark trees produced using ``astroid`` library . If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST tree created separately. Nc||_|rtj||n||_t |t j r|jd}||_t||_ t|j||_ |jDcgc]}|jc}|_|jr|j!|jyycc}w)Nutf8) _filenameastparse_tree isinstancesix binary_typedecode_textr _line_numberslist_generate_tokens_tokensstartpos_token_offsets mark_tokens)self source_textrtreefilenametoks 3lib/python3.12/site-packages/asttokens/asttokens.py__init__zASTTokens.__init__-sDN5:;1DJ +s/&&v.kDJ$[1D--k:;DL48<<@>r(cb|jtj|j|dz S)z Returns the token containing the given character offset (0-based position in source text), or the preceeding token if the position is between tokens. r)rbisectr)roffsets r#get_token_from_offsetzASTTokens.get_token_from_offsetos) << d&9&96BQF GGr(cX|j|jj||S)z Returns the token containing the given (lineno, col_offset) position, or the preceeding token if the position is between tokens. )rBrr0rlineno col_offsets r# get_tokenzASTTokens.get_tokenvs)  % %d&8&8&G&GPZ&[ \\r(cZ|j||jj||S)zd Same as get_token(), but interprets col_offset as a UTF8 offset, which is what `ast` uses. )rGr from_utf8_colrDs r#get_token_from_utf8zASTTokens.get_token_from_utf8s( >>&$"4"4"B"B6:"V WWr(c|jdz}|sJt|j|jr(|dz }t|j|jr(|j|S)z Returns the next token after the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. rr2rrtyperr" include_extrais r# next_tokenzASTTokens.next_tokenY A A   Q 4 4 5 Q  Q 4 4 5 <<?r(c|jdz }|sJt|j|jr(|dz}t|j|jr(|j|S)z Returns the previous token before the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. rrLrNs r# prev_tokenzASTTokens.prev_tokenrRr(c|}|r |jn |j}t|||sVtj|j s7||d}t|||s tj|j s7|S)z Looks for the first token, starting at start_token, that matches tok_type and, if given, the token string. Searches backwards if reverse is True. Returns ENDMARKER token if not found (you can check it with `token.ISEOF(t.type)`. TrO)rTrQrtokenISEOFrM)r start_tokenr3r4reversetadvances r# find_tokenzASTTokens.find_tokensc A!(doodooG!Xw/ AFF8K !4 (a!Xw/ AFF8K Hr(c#Kt|j|jdzD]8}|s#t|j|jr(|j|:yw)z Yields all tokens in order from first_token through and including last_token. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. rN)rr2rrrM)r first_token last_tokenrOrPs r# token_rangezASTTokens.token_rangesQ K%%z'7'7!'; < 1$,,q/2F2FGll1o=s A A"A"cR|j|j|j|S)z Yields all tokens making up the given node. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. rV)rar_r`)rnoderOs r# get_tokenszASTTokens.get_tokenss&   D,,doo]  [[r(ct|dsy|jj}td|j |Dr |j j dd|dz}||jjfS)z After mark_tokens() has been called, returns the (startpos, endpos) positions in source text corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text. r_)rrc3PK|]}t|tj yw)N)rrWNEWLINE).0r[s r# z+ASTTokens.get_text_range..s H2GQ;q%-- (2Gs$& rr) hasattrr_ranyrdrrfindr`endpos)rrcr5s r#get_text_rangezASTTokens.get_text_rangesk 4 '     % %E H$//$2G HHjjtQ.2e 4??)) **r(cH|j|\}}|j||S)z After mark_tokens() has been called, returns the text corresponding to the given node. Returns '' for nodes (like `Load`) that don't correspond to any particular text. )ror)rrcr5r6s r#get_textzASTTokens.get_texts) $$T*JE3 ::ec ""r()FNz )F)NF)__name__ __module__ __qualname____doc__r$rrpropertyr1r<r r!rBrGrJrQrTr]rardrorqr(r#r r s$#.+ E        H]X    \+ #r(r )rr@rWr+r-r six.movesr line_numbersrutilrrrrr objectr rwr(r#r|s6  %99#t#t#r(