c*b` ddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z ddlmZGddeZdS) N)xrange) LineNumbers)Token match_tokenis_non_coding_token) MarkTokensceZdZdZddZdZdZedZed Z ed Z ed Z d Z d Z dZddZddZddZddZddZdZdZdS) ASTTokensa ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and as tokens, and is used to mark and access token and position information. ``source_text`` must be a unicode or UTF8-encoded string. If you pass in UTF8 bytes, remember that all offsets you'll get are to the unicode text, which is available as the ``.text`` property. If ``parse`` is set, the ``source_text`` will be parsed with ``ast.parse()``, and the resulting tree marked with token info and made available as the ``.tree`` property. If ``tree`` is given, it will be marked and made available as the ``.tree`` property. In addition to the trees produced by the ``ast`` module, ASTTokens will also mark trees produced using ``astroid`` library . If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST tree created separately. FN c||_|rtj||n||_t |t jr|d}||_t||_ t| ||_ d|j D|_|jr||jdSdS)Nutf8cg|] }|j S)startpos).0toks 3lib/python3.11/site-packages/asttokens/asttokens.py z&ASTTokens.__init__..>s@@@C3<@@@) _filenameastparse_tree isinstancesix binary_typedecode_textr _line_numberslist_generate_tokens_tokens_token_offsets mark_tokens)self source_textrtreefilenames r__init__zASTTokens.__init__-sDN5:D;111DJ +s///&&v..kDJ$[11D--k::;;DLA@4<@@@D z# tz"""""##rcJt||dS)ap Given the root of the AST or Astroid tree produced from source_text, visits all nodes marking them with token and position information by adding ``.first_token`` and ``.last_token``attributes. This is done automatically in the constructor when ``parse`` or ``tree`` arguments are set, but may be used manually with a separate AST or Astroid tree. N)r visit_tree)r& root_nodes rr%zASTTokens.mark_tokensDs&t *****rc#ZKttjtj|jD]o\}}|\}}}}}t |||||||j|d|d|j|d|dVpdS)z. Generates tokens for the given code. rrN) enumeratetokenizegenerate_tokensioStringIOreadlinerr line_to_offset) r&textindexrtok_typetok_strstartendlines rr"zASTTokens._generate_tokensOs   8T9J9J9S T TUUEE s,/)hT (GUCu$33E!HeAhGG$33CFCFCC E EEEEEEErc|jS)z,The source code passed into the constructor.)rr&s rr6zASTTokens.text[  :rc|jS)zIThe list of tokens corresponding to the source code from the constructor.)r#r>s rtokenszASTTokens.tokens`s  <rc|jS)zTThe root of the AST tree passed into the constructor or parsed from the source code.)rr>s rr(zASTTokens.treeer?rc|jS)zThe filename that was parsed)rr>s rr)zASTTokens.filenamejs  >rcR|jtj|j|dz S)z Returns the token containing the given character offset (0-based position in source text), or the preceeding token if the position is between tokens. r)r#bisectr$)r&offsets rget_token_from_offsetzASTTokens.get_token_from_offsetos% < d&96BBQF GGrc^||j||S)z Returns the token containing the given (lineno, col_offset) position, or the preceeding token if the position is between tokens. )rGr r5r&lineno col_offsets r get_tokenzASTTokens.get_tokenvs,  % %d&8&G&GPZ&[&[ \ \\rc`|||j||S)zd Same as get_token(), but interprets col_offset as a UTF8 offset, which is what `ast` uses. )rLr from_utf8_colrIs rget_token_from_utf8zASTTokens.get_token_from_utf8s+ >>&$"4"B"B6:"V"V W WWrc|jdz}|sCt|j|jr$|dz }t|j|j$|j|S)z Returns the next token after the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. rr7rr#typer&r include_extrais r next_tokenzASTTokens.next_token` A A   Q 4 5 5 Q  Q 4 5 5 <?rc|jdz }|sCt|j|jr$|dz}t|j|j$|j|S)z Returns the previous token before the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. rrQrSs r prev_tokenzASTTokens.prev_tokenrWrc|}|r|jn|j}t|||sPtj|js7||d}t|||stj|j7|S)z Looks for the first token, starting at start_token, that matches tok_type and, if given, the token string. Searches backwards if reverse is True. Returns ENDMARKER token if not found (you can check it with `token.ISEOF(t.type)`. TrT)rYrVrtokenISEOFrR)r& start_tokenr8r9reversetadvances r find_tokenzASTTokens.find_tokens A!(=doodoG!Xw//) AF8K8K) '!4 ( ( (a!Xw//) AF8K8K) Hrc#Kt|j|jdzD]2}|st|j|js|j|V3dS)z Yields all tokens in order from first_token through and including last_token. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. rN)rr7rr#rR)r& first_token last_tokenrTrUs r token_rangezASTTokens.token_rangesj K%z'7!'; < < 1$,q/2FGGl1orcF||j|j|S)z Yields all tokens making up the given node. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. r[)rfrdre)r&noderTs r get_tokenszASTTokens.get_tokenss$   D,do]  [ [[rct|dsdS|jj}td||Dr|jdd|dz}||jjfS)z After mark_tokens() has been called, returns the (startpos, endpos) positions in source text corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text. rd)rrc3JK|]}t|tjVdS)N)rr\NEWLINE)rr`s r z+ASTTokens.get_text_range..s. H HQ;q%- ( ( H H H H H Hr rr) hasattrrdranyrirrfindreendpos)r&rhr:s rget_text_rangezASTTokens.get_text_ranges 4 ' ' V   %E H H$//$2G2G H H HHH3jtQ..2e 4?) **rcP||\}}|j||S)z After mark_tokens() has been called, returns the text corresponding to the given node. Returns '' for nodes (like `Load`) that don't correspond to any particular text. )rsr)r&rhr:r;s rget_textzASTTokens.get_texts, $$T**JE3 :eck ""r)FNr )F)NF)__name__ __module__ __qualname____doc__r*r%r"propertyr6rAr(r)rGrLrOrVrYrbrfrirsrurrrr r s{$####.+++ E E E  8  8  8  8HHH]]]XXX             \\\\+++ #####rr )rrEr\r0r2r six.movesr line_numbersrutilrrrr%r objectr rrrrs  %%%%%%9999999999######t#t#t#t#t#t#t#t#t#t#r