U
|
Þ=®d t ã @ s U d Z ddlZddlZddlmZ ddlmZ ddlm Z ddl
|
m Z ddl mZ dd lmZ ejrxddlZdd
|
lmZ ed Zejejd f ed < e d¡Ze d¡Z e dej ¡Z e dej ej B ¡Z!e dej ej B ¡Z"e dZ#e dZ$e dZ%e dZ&e dZ'e dZ(e dZ)e dZ*e dZ+e d Z,e d Z-e d Z.e d Z/e d Z0e d!Z1e d"Z2e d#Z3e d$Z4e d%Z5e d&Z6e d'Z7e d(Z8e d)Z9e d*Z:e d+Z;e d,Z<e d-Z=e d.Z>e d/Z?e d0Z@e d1ZAe d2ZBe d3ZCe d4ZDe d5ZEe d6ZFe d7ZGe d8ZHe d9ZIe d:ZJe d;ZKe d<ZLe d=ZMe d>ZNe d?ZOe d@ZPe dAZQe dBZRe dCZSe#e;e'e*e3e2e6e<e.e8e/e9e-e7e)e4e+e,e0e1e$e(e%e5e&e:dDZTdEdF eT U¡ D ZVeWeTeWeVksÌtXdGe dHdI YdJdK eZeTdLdM dND ¡ dO¡Z[e\eIeKeJe=eNeOePgZ]e\e=eQeKePgZ^e_e_dPdQdRZ`dSe_dTdUdVZae_e_dWdXdYZbe_ecdZd[d\Zdd]ejeeje_e_f d^d_d`ZfG dadb dbZgG dcdS dSejhZiG ddde deZjG dfdg dgZkd]d d^dhdiZlG djdk dkemZnG dldm dmejhZoG dnd d ZpdS )ozúImplements a Jinja / Python combination lexer. The ``Lexer`` class
|
is used to do some preprocessing. It filters out invalid operators like
|
the bitshift operators we don't allow in templates. It separates
|
template code and python code in expressions.
|
é N)Ú literal_eval)Údeque)Úinterné )Úpattern)ÚTemplateSyntaxError)ÚLRUCache)Ú Environmenté2 ÚLexerÚ _lexer_cachez\s+z (\r\n|\r|\n)z7('([^'\\]*(?:\\.[^'\\]*)*)'|"([^"\\]*(?:\\.[^"\\]*)*)")z¿
|
(
|
0b(_?[0-1])+ # binary
|
|
|
0o(_?[0-7])+ # octal
|
|
|
0x(_?[\da-f])+ # hex
|
|
|
[1-9](_?\d)* # decimal
|
|
|
0(_?0)* # decimal zero
|
)
|
zü
|
(?<!\.) # doesn't start with a .
|
(\d+_)*\d+ # digits, possibly _ separated
|
(
|
(\.(\d+_)*\d+)? # optional fractional part
|
e[+\-]?(\d+_)*\d+ # exponent part
|
|
|
\.(\d+_)*\d+ # required fractional part
|
)
|
ÚaddZassignÚcolonÚcommaÚdivÚdotÚeqÚfloordivÚgtZgteqZlbraceZlbracketZlparenÚltZlteqÚmodÚmulÚneÚpipeÚpowÚrbraceZrbracketZrparenZ semicolonÚsubÚtildeÚ
|
whitespaceÚfloatÚintegerÚnameÚstringÚoperatorZ block_beginZ block_endZvariable_beginZ variable_endZ raw_beginZraw_endZ comment_beginZ comment_endÚcommentZlinestatement_beginZlinestatement_endZlinecomment_beginZlinecomment_endZ linecommentÚdataÚinitialÚeof)ú+ú-ú/z//Ú*ú%z**ú~ú[ú]ú(ú)Ú{Ú}z==z!=ú>z>=ú<z<=ú=Ú.ú:ú|ú,ú;c C s i | ]\}}||qS © r< )Ú.0ÚkÚvr< r< úCd:\z\workplace\vscode\pyvenv\venv\Lib\site-packages\jinja2/lexer.pyÚ
|
<dictcomp> s rA zoperators droppedr0 r9 c c s | ]}t |¡V qd S ©N)ÚreÚescape©r= Úxr< r< r@ Ú <genexpr> s rG c C s
|
t | S rB )Úlen©rF r< r< r@ Ú<lambda> ó rJ )Úkeyr1 )Ú
|
token_typeÚreturnc C sL | t krt | S tdtdtdtdtdtdtdtdt dt
|
d t d
|
t d i | | ¡S ) Nzbegin of commentzend of commentr$ zbegin of statement blockzend of statement blockzbegin of print statementzend of print statementzbegin of line statementzend of line statementztemplate data / textzend of template)Úreverse_operatorsÚTOKEN_COMMENT_BEGINÚTOKEN_COMMENT_ENDÚ TOKEN_COMMENTÚTOKEN_LINECOMMENTÚTOKEN_BLOCK_BEGINÚTOKEN_BLOCK_ENDÚTOKEN_VARIABLE_BEGINÚTOKEN_VARIABLE_ENDÚTOKEN_LINESTATEMENT_BEGINÚTOKEN_LINESTATEMENT_ENDÚ
|
TOKEN_DATAÚ TOKEN_EOFÚget)rM r< r< r@ Ú_describe_token_type£ s< ô ór] ÚToken©ÚtokenrN c C s | j tkr| jS t| j S )z#Returns a description of the token.)ÚtypeÚ
|
TOKEN_NAMEÚvaluer] )r` r< r< r@ Údescribe_token· s
|
rd ©ÚexprrN c C s2 d| kr&| dd¡\}}|tkr*|S n| }t|S )z0Like `describe_token` but for token expressions.r8 r )Úsplitrb r] )rf ra rc r< r< r@ Údescribe_token_expr¿ s rh ©rc rN c C s t t | ¡S )zsCount the number of newline characters in the string. This is
|
useful for extensions that filter a stream.
|
)rH Ú
|
newline_reÚfindall)rc r< r< r@ Úcount_newlinesÌ s rl r ©Ú environmentrN c C s° t j}t| jt|| jft| jt|| jft| jt|| jfg}| j dk rp|
|
t| j t d|| j f¡ | j dk r|
|
t| j t d|| j f¡ dd t|ddD S )zACompiles all the rules from the environment into a list of rules.Nz ^[ \t\v]*z(?:^|(?<=\S))[^\S\r\n]*c S s g | ]}|d d
qS )r Nr< rE r< r< r@ Ú
|
<listcomp>ù s z!compile_rules.<locals>.<listcomp>T)Úreverse)rC rD rH Úcomment_start_stringrP Úblock_start_stringrT Úvariable_start_stringrV Úline_statement_prefixÚappendrX Úline_comment_prefixÚTOKEN_LINECOMMENT_BEGINÚsorted)rn ÚeÚrulesr< r< r@ Ú compile_rulesÓ s: ýýýõ
|
ýÿ
|
ýÿr{ c @ s> e Zd ZdZefeeje ddddZe edddd Z
|
dS )
|
ÚFailurezjClass that raises a `TemplateSyntaxError` if called.
|
Used by the `Lexer` to specify known errors.
|
N)ÚmessageÚclsrN c C s || _ || _d S rB )r} Ú error_class)Úselfr} r~ r< r< r@ Ú__init__ s zFailure.__init__z te.NoReturn)ÚlinenoÚfilenamerN c C s | | j||¡d S rB )r r} )r r r r< r< r@ Ú__call__ s zFailure.__call__) Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r ÚstrÚtÚTyper Úintr r< r< r< r@ r| ü s ÿ þ r| c @ sT e Zd ZU eed< eed< eed< edddZeeddd Zeed
|
d d Z d S )r^ r ra rc ©rN c C s t | S rB )rd ©r r< r< r@ Ú__str__ s z Token.__str__re c C s2 | j |krdS d|kr.| dd¡| j | jgkS dS )z¸Test a token against a token expression. This can either be a
|
token type or ``'token_type:token_value'``. This can only test
|
against string values and types.
|
Tr8 r F)ra rg rc ©r rf r< r< r@ Útest s
|
|
z
|
Token.test)ÚiterablerN c s t fdd|D S )z(Test against multiple token expressions.c 3 s | ]} |¡V qd S rB )r )r= rf r r< r@ rG $ s z!Token.test_any.<locals>.<genexpr>)Úany)r r r< r r@ Útest_any" s zToken.test_anyN)
|
r
r r r Ú__annotations__r r Úboolr r r< r< r< r@ r^ s
|
c @ s< e Zd ZdZdddddZd ddd Zedd
|
d ZdS ) ÚTokenStreamIteratorz`The iterator for tokenstreams. Iterate over the stream
|
until the eof token is reached.
|
Ú TokenStreamN)ÚstreamrN c C s
|
|| _ d S rB )r )r r r< r< r@ r , s z TokenStreamIterator.__init__r c C s | S rB r< r r< r< r@ Ú__iter__/ s z TokenStreamIterator.__iter__c C s. | j j}|jtkr | j ¡ tt| j |S rB )r Úcurrentra r[ ÚcloseÚ StopIterationÚnext©r r` r< r< r@ Ú__next__2 s
|
|
|
z TokenStreamIterator.__next__)r
r r r r r r^ r r< r< r< r@ r ' s r c @ sä e Zd ZdZeje eje eje dddZ e
|
dddZ e ddd Z ee dd
|
d Zed d ddZedddZd!ed dddZeeje dddZee dddZeddd Zd dd d Zeedd d Zd S )"r zÛA token stream is an iterable that yields :class:`Token`\s. The
|
parser however does not iterate over it but calls :meth:`next` to go
|
one token ahead. The current active token is stored as :attr:`current`.
|
)Ú generatorr! r c C s> t || _t | _|| _|| _d| _tdtd| _ t
|
| d S )NFr Ú ) ÚiterÚ_iterr Ú_pushedr! r Úclosedr^ Ú TOKEN_INITIALr r )r r¡ r! r r< r< r@ r C s
|
zTokenStream.__init__r c C s t | S rB )r r r< r< r@ r Q s zTokenStream.__iter__c C s t | jp| jjtk S rB )r r¥ r ra r[ r r< r< r@ Ú__bool__T s zTokenStream.__bool__c C s | S )z Are we at the end of the stream?r< r r< r< r@ ÚeosW s zTokenStream.eosNr_ c C s | j |¡ dS )z Push a token back to the stream.N)r¥ ru r r< r< r@ Úpush\ s zTokenStream.pushc C s" t | }| j}| |¡ || _|S )zLook at the next token.)r r rª )r Z old_tokenÚresultr< r< r@ Úlook` s
|
|
zTokenStream.lookr )ÚnrN c C s t |D ] }t| qdS )zGot n tokens ahead.N)Úranger )r r Ú_r< r< r@ Úskiph s zTokenStream.skipre c C s | j |¡rt| S dS )zqPerform the token test and return the token if it matched.
|
Otherwise the return value is `None`.
|
N)r r r r r< r< r@ Únext_ifm s zTokenStream.next_ifc C s | |¡dk S )z8Like :meth:`next_if` but only returns `True` or `False`.N)r± r r< r< r@ Úskip_ifv s zTokenStream.skip_ifc C sX | j }| jr| j ¡ | _ n:| j jtk rTzt| j| _ W n tk
|
rR | ¡ Y nX |S )z|Go one token ahead and return the old one.
|
|
Use the built-in :func:`next` instead of calling this directly.
|
) r r¥ Úpopleftra r[ r r¤ r r )r Úrvr< r< r@ r z s zTokenStream.__next__c C s& t | jjtd| _td| _d| _dS )zClose the stream.r¢ r< TN)r^ r r r[ r£ r¤ r¦ r r< r< r@ r s
|
zTokenStream.closec C sn | j |¡sft|}| j jtkr>td|d| j j| j| jtd|dt | j | j j| j| jt
|
| S )z}Expect a given token type and return it. This accepts the same
|
argument as :meth:`jinja2.lexer.Token.test`.
|
z%unexpected end of template, expected r7 zexpected token z, got ) r r rh ra r[ r r r! r rd r r r< r< r@ Úexpect s
|
üüzTokenStream.expect)r )r
r r r r ÚIterabler^ ÚOptionalr r r r r r¨ Úpropertyr© rª r¬ r r° r± r² r r rµ r< r< r< r@ r = s" ü r c C sZ | j | j| j| j| j| j| j| j| j| j | j
|
| j f }t |¡}|dkrVt| t |< }|S )z(Return a lexer which is probably cached.N)rr Úblock_end_stringrs Úvariable_end_stringrq Úcomment_end_stringrt rv Ú trim_blocksÚ lstrip_blocksÚnewline_sequenceÚkeep_trailing_newliner r\ r )rn rL Úlexerr< r< r@ Ú get_lexerª s" ô
|
rÁ c s$ e Zd ZdZdZ fddZ ZS )ÚOptionalLStripzWA special tuple for marking a point in the state that can have
|
lstrip applied.
|
r< c s t | |¡S rB )ÚsuperÚ__new__)r~ ÚmembersÚkwargs©Ú __class__r< r@ rÄ Ë s zOptionalLStrip.__new__)r
r r r Ú __slots__rÄ Ú __classcell__r< r< rÇ r@ r  s r c @ sN e Zd ZU eje ed< ejeejedf eje f ed< ej
|
e ed< dS )Ú_Ruler .ÚtokensÚcommandN) r
r r r ÚPatternr r ÚUnionÚTupler| r· r< r< r< r@ rË Ï s
|
$rË c
|
@ sÐ e Zd ZdZdddddZeeddd Zdeeje eje eje e d
|
d d Z
|
dej ej e eef eje eje eje d ddZdeeje eje eje ejej e eef d
|
ddZdS )r a Class that implements a lexer for a given environment. Automatically
|
created by the environment class, usually you don't have to do that.
|
|
Note that the lexer is not automatically bound to an environment.
|
Multiple environments can share the same lexer.
|
r Nrm c C s$ t j}ttjt ddd}tttd ttt d tt
|
t d tt t d tttd tttd g}t|}||j}||j}||j}||j} |jrdnd}
|
|j| _|j| _|j| _d| d| d| d } d
|
| gd d |D ¡} d t|d| dt t ddt|dt d gt t|d| d| d| |
|
d t t fdt|dt!dfd gt"t|d| d| d| |
|
dt#dg| t$t|d| d| t%dg| t&t|d| d| d| d| |
|
d
|
t t t'dt|dt!dfd gt(t|d t)dg| t*t|d t+t,fdgi| _-d S ) N)rF rN c S s t | t jt jB ¡S rB )rC ÚcompileÚMÚSrI r< r< r@ Úcá s zLexer.__init__.<locals>.cz\n?r¢ z(?P<raw_begin>z(\-|\+|)\s*raw\s*(?:\-z\s*|z))r9 c S s" g | ]\}}d | d| dqS )z(?P<r4 z (\-|\+|))r< )r= r Úrr< r< r@ ro s z"Lexer.__init__.<locals>.<listcomp>Úrootz(.*?)(?:r1 ú#bygroupz.+z (.*?)((?:\+z|\-ú#popz(.)zMissing end of comment tagz(?:\+z\-z (.*?)((?:z(\-|\+|))\s*endraw\s*(?:\+z Missing end of raw directivez \s*(\n|$)z(.*?)()(?=\n|$)).rC rD r r rÎ rË Ú whitespace_reÚTOKEN_WHITESPACEÚfloat_reÚ TOKEN_FLOATÚ
|
integer_reÚ TOKEN_INTEGERÚname_rerb Ú string_reÚ TOKEN_STRINGÚ operator_reÚTOKEN_OPERATORr{ rr r¹ r» rº r¼ r½ r¾ r¿ Újoinr rZ rP rR rQ r| rT rU rV rW ÚTOKEN_RAW_BEGINÚ TOKEN_RAW_ENDrX rY rw rS ÚTOKEN_LINECOMMENT_ENDrz ) r rn ry rÔ Z tag_rulesZroot_tag_rulesZblock_start_reZ block_end_reZcomment_end_reZvariable_end_reZblock_suffix_reZ root_raw_reZ root_parts_rer< r< r@ r Ý s¨
|
|
|
|
|
|
ú
|
|
|
|
ÿÿýø ÿú÷ ÿúÿ
|
ö ýÿù ÿù ö ÿýýÿÁzLexer.__init__ri c C s t | j|¡S )z`Replace all newlines with the configured sequence in strings
|
and template data.
|
)rj r r¾ )r rc r< r< r@ Ú_normalize_newlinesT s zLexer._normalize_newlines)Úsourcer! r ÚstaterN c C s& | ||||¡}t| |||¡||S )z:Calls tokeniter + tokenize and wraps it in a token stream.)Ú tokeniterr Úwrap)r ré r! r rê r r< r< r@ ÚtokenizeZ s zLexer.tokenize)r r! r rN c
|
c sr |D ]f\}}}|t krq|}|tkr.t}n0|tkr>t}n |ttfkrPqn|tkrd| |¡}nú|dkrr|}nì|t kr|}|
|
¡ st d|||nÈ|t krz$| |dd
¡ dd¡ d¡}W nJ tk
|
r } z*t| d¡d ¡ } t | ||||W 5 d }~X Y nX nL|tkr0t| d
|
d ¡d }n.|tkrLt| d
|
d ¡}n|tkr^t| }t|||V qd S ) zThis is called with the stream as returned by `tokenize` and wraps
|
every token in a :class:`Token` and converts the value.
|
Úkeywordz Invalid character in identifierr éÿÿÿÿÚasciiÚbackslashreplacezunicode-escaper8 Nr¯ r¢ r )Úignored_tokensrX rT rY rU rå ræ rZ rè rb Ú isidentifierr rá ÚencodeÚdecodeÚ Exceptionr rg ÚstriprÞ r ÚreplacerÜ r rã Ú operatorsr^ )
|
r r r! r r r` Z value_strrc ry Úmsgr< r< r@ rì e sT ÿ
|
ÿþÿ$
|
|
|
z
|
Lexer.wrapc c s@ t |¡ddd
}| js,|d dkr,|d= d |¡}d}d}dg}|dk rr|dkrr|d ksdtd
|
| |d ¡ | j|d } t|}
|
g } d} d } | D ]r\}}}| ||¡}|dkrºq| rÎ|t t
|
t fkrÎqt |t rz| ¡ }t |tr¶|d }td d |ddd
D }|dkrN| ¡ }|t|d
d¡} |f|dd
}nh|dkr¶| jr¶| ¡ t¡s¶| d¡d }|dks| r¶t ||¡r¶|d|
f|dd
}t|D ]¸\}}|jt krÞ|||n|dkr:| ¡ ¡ D ]4\}}|dk rô|||fV || d¡7 } qtqôt |dn:|| }|sR|t kr^|||fV || d¡| 7 }d} q¾nÒ| ¡ }|t!kr"|dkr¢| d¡ n|dkr¸| d¡ nj|dkrÎ| d¡ nT|dkr"| sôt"d| d|||| #¡ } | |kr"t"d| d | d||||s2|t kr>|||fV || d¡7 }| ¡ dd
dk} | $¡ } |dk rì|d kr| #¡ nV|dkrÒ| ¡ ¡ D ]$\}}|dk r| |¡ qÜqt |d n
|
| |¡ | j|d } n| |krt |d | } qq||
|
kr dS t"d || d!| |||qdS )"a This method tokenizes the text and returns the tokens in a
|
generator. Use this method if you just want to tokenize a template.
|
|
.. versionchanged:: 3.0
|
Only ``\n``, ``\r\n`` and ``\r`` are treated as line
|
breaks.
|
Né rï r¢ Ú
|
r r rÖ )ÚvariableÚblockz invalid stateZ_beginTc s s | ]}|d k r|V qd S rB r< )r= Úgr< r< r@ rG Ü s z"Lexer.tokeniter.<locals>.<genexpr>r) r( r× z= wanted to resolve the token dynamically but no group matchedr2 r3 r0 r1 r. r/ )r3 r1 r/ z unexpected 'ú'z ', expected 'rØ zA wanted to resolve the new state dynamically but no group matchedz* yielded empty string without stack changezunexpected char z at )%rj rg r¿ rä ÚAssertionErrorru rz rH ÚmatchrW rU rY Ú
|
isinstanceÚtupleÚgroupsr r ÚrstripÚcountr½ Ú groupdictr\ rV ÚrfindrÙ Ú fullmatchÚ enumeraterÈ r| ÚitemsÚ RuntimeErrorÚignore_if_emptyÚgrouprã r ÚpopÚend) r ré r! r rê ÚlinesÚposr ÚstackZ statetokensZ source_lengthZbalancing_stackZnewlines_strippedZ line_startingÚregexrÌ Z new_stateÚmr ÚtextZ
|
strip_signÚstrippedZl_posÚidxr` rL rc r% Z expected_opÚpos2r< r< r@ rë sê
|
ý
|
þü ú
|
|
|
ÿ
|
|
|
|
|
|
|
ÿ
|
ü
|
|
|
|
|
|
|
ÿ
|
|
ÿ
|
ÿzLexer.tokeniter)NNN)NN)NN)r
r r r r r rè r r· r rí r¶ rÐ r ÚIteratorr^ rì rë r< r< r< r@ r Õ s< w ûú üû : ûú)qr rC Útypingr Úastr Ú collectionsr Úsysr Z _identifierr rß Ú
|
exceptionsr Úutilsr Ú TYPE_CHECKINGZtyping_extensionsÚtern r r ÚMutableMappingrÐ r rÑ rÙ rj rÓ rà Ú
|
IGNORECASEÚVERBOSErÝ rÛ Z TOKEN_ADDZ TOKEN_ASSIGNZ TOKEN_COLONZ TOKEN_COMMAZ TOKEN_DIVZ TOKEN_DOTZTOKEN_EQZTOKEN_FLOORDIVZTOKEN_GTZ
|
TOKEN_GTEQZ TOKEN_LBRACEZTOKEN_LBRACKETZ TOKEN_LPARENZTOKEN_LTZ
|
TOKEN_LTEQZ TOKEN_MODZ TOKEN_MULZTOKEN_NEZ
|
TOKEN_PIPEZ TOKEN_POWZ TOKEN_RBRACEZTOKEN_RBRACKETZ TOKEN_RPARENZTOKEN_SEMICOLONZ TOKEN_SUBZ TOKEN_TILDErÚ rÜ rÞ rb rá rã rT rU rV rW rå ræ rP rQ rR rX rY rw rç rS rZ r§ r[ rù r rO rH r rä rx râ Ú frozensetrò r r r] rd rh r rl ÚListr{ r| Ú
|
NamedTupler^ r r rÁ r r rË r r< r< r< r@ Ú<module> s
|
|
|
ÿ
|
ò
|
|
õæ &ÿùÿ
|
ÿ ) m
|
