a :c@slddlmZmZmZmZmZddlmZmZm Z m Z m Z m Z m Z mZmZmZmZmZddlZddlZddlZddlZddlZddlZddlmZdagaddZGdd d eZGd d d e Z!d d Z"e#dej$Z%ddZ&ddZ'ddZ(da)d)ddZ*ddZ+e,gdZ-gdZ.gdZ/d0dd e.De/Z1d!Z2d"Z3Gd#d$d$e Z4Gd%d&d&e Z5d'd(Z6dS)*)divisionabsolute_importwith_statementprint_functionunicode_literals) PY2 basestringbchrbordchropenpystrrangeroundstrtobytesunicodeN)match_logical_wordcCsddl}tj|}|tkr|azdt|d }|dd}Wdn1sR0Y|drpt |d}|d7}| da Wnt yga Yn0|t t krt |d dSdSdS) z Gets the text of a line, in a best-effort way, for debugging purposes. May return just a newline, if the line doesn't exist. rNrbutf-8 python_strict_ren.py  ) linecacherenpyexportsunelide_filenameline_text_filenamer readdecodeendswith ren_py_to_rpysplitline_text_cache Exceptionlen)filenamelinenorZ full_filenamefdatar-!/home/tom/ab/renpy/renpy/lexer.py get_line_text,s   .     r/c@seZdZdddZddZdS) ParseErrorNFc Cs6dt|||f}|r t|tr,d|}|d}t|dkrd} d} | t|dkr|d| } | dkrx| d7} n0| | krd} n"| rn| dks| dks| d kr| } | d7} qJ| r|d | 7}|D]X} |d | 7}|dur|t| kr|d d |d 7}d}n |t| 8}|rq q||_t||dS)NzFile "%s", line %d: %srrrr\`'"z: (Perhaps you left out a %s at the end of the first line.)z  ^) unicode_filename isinstancelistjoinr%r(messager'__init__) selfr)numbermsglineposfirstr;linesZ open_stringiclr-r-r.r<Ts@           zParseError.__init__cCs|jSN)r;r=r-r-r. __unicode__szParseError.__unicode__)NNF)__name__ __module__ __qualname__r<rIr-r-r-r.r0Rs /r0c@seZdZdZddZdS)LineNumberHolderz) Holds the expected line number. cCs d|_dSNr)r@rHr-r-r.r<szLineNumberHolder.__init__N)rJrKrL__doc__r<r-r-r-r.rMsrMcCsXt|tr|Sz |dWSty,Yn0z |dWStyLYn0|dS)z4 Converts the supplied filename to unicode. mbcsrzlatin-1)r8rr"r')fnr-r-r.r7s     r7z__(\w+)|\w+| +|.cCsJtj|}tj|d}|dd}dd}td||}d|dS) Nrr5_cSstt|dSrN)hexordgroup)mr-r-r. munge_charsz"munge_filename..munge_charz [^a-zA-Z0-9_]Z_m1___)ospathbasenamesplitextreplaceresub)rQrvrWr-r-r.munge_filenames   racCs|dd}tjtjjddd}tjtjjddd}||rX||g}n||g}|D]"}||rd|t |d}qqd|}|S)zy Returns a version of fn that is either relative to the base directory, or relative to the Ren'Py directory. r1/N) r]rYrZabspathrconfigbasedir renpy_base startswithr()rQrerfdirsdr`r-r-r.elide_filenames    rjcCstj|}tjjdur:tjtjj|}tj|r:|Stjtjj|}tj|r\|Stjtjj |}tj|r~|S|SrG) rYrZnormpathrrdalternate_unelide_pathr:existsrerf)rQZfn0Zfn1Zfn2r-r-r.rs     rrFcsfdd}|a|r|}n:t|d }|dd}Wdn1sJ0Y|drht||}t|}t||d7}g}|}d } t|r|d d kr| d 7} |st j j rt j j} ni} t|} t j j|d } d } | | kr|} g} d }|| f}t j t| | | |<d}| | kr| }|| }|d krDt||d |dkr|sd| } td| s|||| | f|dur| }|d | |_||d dvr|d 8}q|| |_|| |j| |j| |_|| |j| |j| |_| d 7} |d 7}d}g} q|dkr,|d 7}d}|dkrB| d 7} q|dkr||| d dkr|| d7} |d 7}| dq|dvr|d 7}|dvr|r|d 8}|dkr| }|| dkr| d 7} qq|dvr|}| || d 7} d}d}| | d krH|| |krH|| d |krH| || || d7} d}g}| | krP|| }|dkrp|d 7}|dkr| d 7} qL|rd}| d 7} ||qL||kr(|s| d 7} ||qP| | dkr(|| d |kr(|| d|kr(| d7} ||||||qP|dkr6d}||| d 7} qLqLd|}d|vrrtd||}| |qt|| \}}}|r|dd}d|vr|}| ||} | |d krt|| d!| dd"qq| rt|| d#| dd"|S)$a Reads `filename`, and divides it into logical lines. Returns a list of (filename, line number, line text) triples. If `filedata` is given, it should be a unicode string giving the file contents. In that case, `filename` need not exist. csN|d}t|d@dkr$|dSd|dvr<|dS||dS)NrrrX)rUr()rVZbracketsprefixr-r. munge_strings    z(list_logical_lines..munge_stringrrrNrrrur z1Tab characters are not allowed in Ren'Py scripts.rrz^\s*$z  r1rnz\ z([{z}])#z"'`FTz[__z(\.|\[+)__(\w+)rXiz:Overly long logical line. (Check strings and parenthesis.))r@rBzBis not terminated with a newline. (Check strings and parenthesis.))original_filenamer r!r"r#r$rjrar(rgamecontext init_phaseZ scripteditrCfilesaddZLiner0r:r^matchappendZ end_delimendstarttextZ full_textr_r)r)Zfiledata linenumberZ add_linesrqr,r+r`r>rArCZlen_datar@Z start_numberZ parendepthlocendposstartposrEdelimescapeZ triplequoteswordmagicr~restr-ror.list_logical_liness  .                  .        2           rcsTddfddrFd\}}}|ddkrFt||ddddS)aE This takes as input the list of logical line triples output from list_logical_lines, and breaks the lines into blocks. Each block is represented as a list of (filename, line number, line text, block) triples, where block is a block list (which may be empty if no block is associated with this line.) cSs:d}d}||dkr*|d7}|d7}qq*q|||dfS)Nrr5rr-)rFdepthindexr-r-r. depth_splits z(group_logical_lines..depth_splitc sg}d}|tkr|\}}}|\}}||kr8q|durD|}||krXt||d|d7}||d\} }||||| fq||fS)NzIndentation mismatch.r)r(r0r}) rDZ min_depthr`rr)r>rZ line_depthrblockrgll_corerCr-r.rs   z%group_logical_lines..gll_corerz(Unexpected indentation at start of file.)r0)rCr)r>rr-rr.group_logical_liness  r)$asatbehindcall expressionhideifinimageinitjumpmenuZonlayerpythonreturnsceneshowwithwhilezorder transform)z<>z<>z>=>z!=z==|r6&+-z***z//rb%~)z\bor\bz\band\bz\bnot\bz\bin\bz\bis\brcCsg|]}t|qSr-)r^r).0rDr-r-r. grz0[a-zA-Z_\u00a0-\ufffd][0-9a-zA-Z_\u00a0-\ufffd]*z5[-0-9a-zA-Z_\u00a0-\ufffd][-0-9a-zA-Z_\u00a0-\ufffd]*c@s eZdZdZddZddZdS)SubParsezs This represents the information about a subparse that can be provided to a creator-defined statement. cCs ||_dSrG)r)r=rr-r-r.r<sszSubParse.__init__cCs*|js dSd|jdj|jdjSdS)Nzzr)rformatr)rrHr-r-r.__repr__vszSubParse.__repr__N)rJrKrLrOr<rr-r-r-r.rmsrc@seZdZdZdaddZdd Zd d Zd d ZddZddZ ddZ e j ddZ ddZddZddZddZddZd d!Zdbd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1Zd2d3Zdcd4d5Zd6d7Zd8d9Zd:d;Z dd?Z"dddAdBZ#dedCdDZ$dEdFZ%dfdGdHZ&dIdJZ'dKdLZ(dMdNZ)dOdPZ*dQdRZ+dgdSdTZ,dUdVZ-dWdXZ.dYdZZ/d[d\Z0d]d^Z1dhd_d`Z2dS)iLexerz The lexer that is used to lex script files. This works on the idea that we want to lex each line in a block individually, and use sub-lexers to lex sub-blocks. FrNrcCsd||_||_||_d|_d|_d|_d|_d|_g|_||_ d|_ d|_ d|_ d|_ ||_||_dS)NFrr)r init_offsetreobr@r)rr>subblock global_labelrAword_cache_posword_cache_newpos word_cachemonologue_delimiter subparses)r=rrrrrrr-r-r.r<s zLexer.__init__cCsT|jd7_|jt|jkr(d|_dS|j|j\|_|_|_|_d|_d|_ dS)a' Advances this lexer to the next line in the block. The lexer starts off before the first line, so advance must be called before any matching can be done. Returns True if we've successfully advanced to a line in the block, or False if we have advanced beyond all lines in the block. In general, once this method has returned False, the lexer is in an undefined state, and it doesn't make sense to call any method other than advance (which will always return False) on the lexer. rTFrr) r@r(rrr)r>rrrArrHr-r-r.advances z Lexer.advancecCsF|jd8_d|_|j|j\|_|_|_|_t|j|_d|_ dS)z Puts the parsing point at the end of the previous line. This is used after renpy_statement to prevent the advance that Ren'Py statements do. rFrN) r@rrr)r>rrr(rArrHr-r-r. unadvances  zLexer.unadvancecCsT|jr dS|jt|jkrdSt|tj|j|j}|s@dS||_| dS)a Tries to match the given regexp at the current location on the current line. If it succeds, it returns the matched text (if any), and updates the current position to be after the match. Otherwise, returns None and the position is unchanged. Nr) rrAr(rr^compileDOTALLr|r~rU)r=regexprVr-r-r. match_regexps zLexer.match_regexpcCs|ddS)zQ Advances the current position beyond any contiguous whitespace. z (\s+|\\\n)+N)rrHr-r-r.skip_whitespaceszLexer.skip_whitespacecCs|||S)z Matches something at the current position, skipping past whitespace. Even if we can't match, the current position is still skipped past the leading whitespace. )rr)r=rr-r-r.r|sz Lexer.matchcCs |j}||kr|S||_dS)z Matches a keyword at the current position. A keyword is a word that is surrounded by things that aren't words, like whitespace. (This prevents a keyword from matching a prefix.) r)rAr)r=roldposr-r-r.keywords  z Lexer.keywordc csDz dVWn4ty>}ztjj|jWYd}~n d}~00dS)zm Catches errors, then causes the line to advance if it hasn't been advanced already. N)r0rparser parse_errorsr}r;)r=er-r-r. catch_errors zLexer.catch_errorcCsF|jdkr*|jr*|jd\|_|_|_|_t|j|j||j|jdS)zc Convenience function for reporting a parse error at the current location. rrN)r@rr)r>rrr0rA)r=r?r-r-r.error sz Lexer.errorcCs||jt|jkS)z Returns True if, after skipping whitespace, the current position is at the end of the end of the current line, or False otherwise. )rrAr(rrHr-r-r.eolsz Lexer.eolcCs|s|ddS)zG If we are not at the end of the line, raise an error. zend of line expected.N)rrrHr-r-r. expect_eolszLexer.expect_eolcCs(|jr$|}||d|dS)zz Called to indicate this statement does not expect a block. If a block is found, raises an error. zoLine is indented, but the preceding %s statement does not expect a block. Please check this line's indentation.N)rsubblock_lexerrr)r=stmtllr-r-r.expect_noblock'szLexer.expect_noblockcCs|js|d|dS)zk Called to indicate that the statement requires that a non-empty block is present. z%s expects a non-empty block.N)rr)r=rr-r-r. expect_block2szLexer.expect_blockcCs t|jS)zL Called to check if the current line has a non-empty block. )boolrrHr-r-r. has_block;szLexer.has_blockcCs(|jp|}t|j||j|j|j|jdS)zk Returns a new lexer object, equiped to parse the block associated with this line. )rrrrr)rrrrrrr)r=rr-r-r.rAs zLexer.subblock_lexercCs|d}|dur|d}|dur.|d}|dur:dS|ddkrXd}|dd}nd }|dd }d d }|std d|}td||}|S)a' Lexes a string, and returns the string to the user, or None if no string could be found. This also takes care of expanding escapes and collapsing whitespace. Be a little careful, as this can return an empty string, which is different than None. zr?"([^\\"]|\\.)*"Nzr?'([^\\']|\\.)*'zr?`([^\\`]|\\.)*`rrTrFrcSsr|d}|dkrdS|dkr"dS|dkr.dS|dkr:d S|d d krj|d }|rntt|d d Sn|SdSNr{z{{[z[[rz%%nrrurnrUr intrVrEZgroup2r-r-r.dequoteis   zLexer.string..dequote\s+r5\\(u([0-9a-fA-F]{1,4})|.))r|r^r_)r=rrawrr-r-r.stringKs"     z Lexer.stringcCs|d}|dur|d}|dur.|d}|dur:dS|ddkrXd}|dd}nd }|d d }d d }|stdd|}g}||jD]8}|}|sqtdd|}td||}||q|S|S)z Lexes a triple quoted string, intended for use with monologue mode. This is about the same as the double-quoted strings, except that runs of whitespace with multiple newlines are turned into a single newline. zr?"""([^\\"]|\\.|"(?!""))*"""Nzr?'''([^\\']|\\.|'(?!''))*'''zr?```([^\\`]|\\.|`(?!``))*```rrTrFrucSsr|d}|dkrdS|dkr"dS|dkr.dS|dkr:d S|d d krj|d }|rntt|d d Sn|SdSrrrr-r-r.rs   z$Lexer.triple_string..dequotez *\n *rrr5r)r|r^r_r%rstripr})r=rrrr`r-r-r. triple_strings2      zLexer.triple_stringcCs |dS)zf Tries to parse an integer. Returns a string containing the integer, or None. z (\+|\-)?\d+r|rHr-r-r.integersz Lexer.integercCs |dS)zk Tries to parse a number (float). Returns a string containing the number, or None. z((\+|\-)?(\d+\.?\d*|\.\d+)([eE][-+]?\d+)?rrHr-r-r.floatsz Lexer.floatcCs |dS)zG Matches the characters in an md5 hash, and then some. z\w+rrHr-r-r.hashsz Lexer.hashcCs>|j|jkr|j|_|jS|j|_|t}||_|j|_|S)z? Parses a name, which may be a keyword or not. )rArrrr| word_regexp)r=r`r-r-r.rs  z Lexer.wordcCs`|j}|}|dks&|dks&|dkrJ|j|j|jddvrJ||_dS|tvr\||_dS|S)zG This tries to parse a name. Returns the name or None. rrurrr4r3r2N)rArrKEYWORDSr=rr`r-r-r.namesz Lexer.namecCs$|r |ddkr |dd|_dS)z Set current global_label, which is used for label_name calculations. label can be any valid label or None, but this has only effect if label has global part. r.N)r%r)r=labelr-r-r.set_global_labelszLexer.set_global_labelcCs|j}d}|}|sN|dr&|js0||_dS|j}|}|s||_dSn8|dr|rp||jkrp||_dS|}|s||_dS|s|S|d|S)a Try to parse label name. Returns name in form of "global.local" if local is present, "global" otherwise; or None if it doesn't parse. If declare is True, allow only such names that are valid for declaration (e.g. forbid global name mismatch) N\.r)rArr|r)r=declareold_posZ local_nameZ global_namer-r-r. label_name s.  zLexer.label_namecCs |jddS)z> Same as label_name, but set declare to True. T)r)rrHr-r-r.label_name_declare4szLexer.label_name_declarecCsZ|j}|t}|dks |dkrD|j|j|jddvrD||_dS|tvrV||_dS|S)z Matches a word that is a component of an image name. (These are strings of numbers, letters, and underscores.) rrrrN)rAr|image_word_regexprrrr-r-r.image_name_component:s zLexer.image_name_componentcCs|r dS|j}|j|j}dD]D}||kr0q"|jd7_|jt|jkrZ||_dS|j|j}q"|dvrz||_dS|}|jd7_|r|d|j|j}||krq|dkr~|jd7_q~|jd7_dS)z This tries to match a python string at the current location. If it matches, it returns True, and the current position is updated to the end of the string. Otherwise, returns False. F)rrr+r)r4r3z)end of line reached while parsing string.r1T)rrArr(r)r=rrEmodrr-r-r. python_stringNs4   zLexer.python_stringcCsB|}|sdS|dr>|}|s0|d|d|7}q|S)aQ This tries to match a dotted name, which is one or more names, separated by dots. Returns the dotted name if it can, or None if it cannot. Once this sees the first name, it commits to parsing a dotted_name. It will report an error if it then sees a dot without a name behind it. Nrzexpecting name.r)rr|r)r=r`rr-r-r. dotted_names   zLexer.dotted_namecCs|s|Stj||j|jSrG)rastPyExprr)r>)r=rexprr-r-r.rsz Lexer.exprTcCsx|j}|sf|j|j}||vr:||j||j|S|dvrL|q|rVq|jd7_q|d|dS)a# This matches python code up to, but not including, the non-whitespace delimiter characters. Returns a string containing the matched code, which may be empty if the first thing is the delimiter. Raises an error if EOL is reached before the delimiter. z'"rz(reached end of line when expecting '%s'.N)rArrrrparenthesised_pythonr)r=rrrrEr-r-r.delimited_pythons zLexer.delimited_pythoncCs.|dd}|s|d|||}|S)zk Returns a python expression, which is arbitrary python code extending to a colon. :Fzexpected python_expression)rrrr)r=rZper`r-r-r.python_expressions   zLexer.python_expressioncCs|j|j}|dkr@|jd7_|dd|jd7_dS|dkrt|jd7_|dd|jd7_dS|dkr|jd7_|d d|jd7_dSdS) z Tries to match a parenthesised python expression. If it can, returns true and updates the current position to be after the closing parenthesis. Returns False otherwise. (r)FTr]r})rrAr)r=rEr-r-r.rs"    zLexer.parenthesised_pythoncCs|j}|trq|rq|s>|s>|s>|s>q||rPq|drr| }|s>| dq>|rq>qq>|r|trq|r|drqqq|j ||j }|sdSt j||j|jS)zo Tries to parse a simple_expression. Returns the text if it can, or None if it cannot. rzexpecting name after dot.,N)rAr|operator_regexprrrrrrrrrrrrrr)r>)r=commaoperatorrrrr-r-r.simple_expressionsB   zLexer.simple_expressioncCs |jddS)zt One or more simple expressions, separated by commas, including an optional trailing comma. T)rrrHr-r-r.comma_expression$szLexer.comma_expressioncCs |jddS)z= Parses the name portion of a say statement. F)rrrHr-r-r.say_expression,szLexer.say_expressioncCs&|j|j|j|j|j|jtjj fS)z Returns an opaque representation of the lexer state. This can be passed to revert to back the lexer up. ) r@r)r>rrrArrr checkpointrHr-r-r.r2szLexer.checkpointcCsT|\|_|_|_|_|_|_}tjj |d|_ |jt |j krJd|_ nd|_ dS)z Reverts the lexer to the given state. State must have been returned by a previous checkpoint operation on this lexer. rFTN)r@r)r>rrrArrrrevertrr(rr)r=stateZpyexpr_checkpointr-r-r.r:s z Lexer.revertcCs |j|jfS)z Returns a (filename, line number) tuple representing the current physical location of the start of the current logical line. )r)r>rHr-r-r. get_locationJszLexer.get_locationcCsJt|tr|p|}||}n|p(|jj}|}|durF|d||S)z Tries to parse thing, and reports an error if it cannot be done. If thing is a string, tries to parse it using self.match(thing). Otherwise, thing must be a method on this lexer object, which is called directly. Nzexpected '%s' not found.)r8rr|__func__rJr)r=thingrr`r-r-r.requireRs   z Lexer.requirecCs<||j}t|j|_tj|j|d|j|j S)z Skips whitespace, then returns the rest of the current line, and advances the current position to the end of the current line. N) rrAr(rrrrrr)r>r=rAr-r-r.rgs z Lexer.restcCs$|j}t|j|_|j|dS)zG Like rest, but returns a string rather than a PyExpr. N)rAr(rrrr-r-r.rest_statementts zLexer.rest_statementcs8gt|j_fdd|jddS)z Returns the subblock of this code, and subblocks of that subblock, as indented python code. This tries to insert whitespace to ensure line numbers match up. csv|D]l\}}}}j|kr8|djd7_q||d}|j|d7_||dqdS)Nrr )r@r}count)rindentZ_fnZlnrrZlinetextoprocessr`r-r.r#s   z#Lexer.python_block..processr)rMr>r@rr:rHr-r!r. python_block}s  zLexer.python_blockcCs tj|S)zo Returns an Argument object if there is a list of arguments, or None there is not one. )rrZparse_argumentsrHr-r-r. argumentsszLexer.argumentscCsN|jdurtdtj|}|t|ts6|g}t|}|j ||S)z Parses the remainder of the current line as a statement in the Ren'Py script language. Returns a SubParse corresponding to the AST node generated by that statement. NzHA renpy_statement can only be parsed inside a creator-defined statement.) rr'rrparse_statementrr8r9rr})r=rspr-r-r.renpy_statements    zLexer.renpy_statementc Cs|jdurtd|jdkr$|g}|jsz0tj|}t|t rR| |n | |Wq(t y}z$tjj |j|WYd}~q(d}~00q(|s|r| tj|n |dt|}|j ||S)NzDA renpy_block can only be parsed inside a creator-defined statement.rz*At least one Ren'Py statement is expected.)rr'r@rrrrr&r8r9extendr}r0rr;rPassrrr)r=emptyrrrr'r-r-r. renpy_blocks*        zLexer.renpy_block)FrNrN)F)F)T)T)FT)N)F)3rJrKrLrOr<rrrrr|r contextlibcontextmanagerrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr$r%r(r,r-r-r-r.r~s^          9E (2   =   rcCsf|}g}d}d}d}d}|}t|ddD]\} } ||krZ| drZ|}|d| } q,||kr| dkrz|}|dq,| } | s|| q,| ddkr|| q,d}| D]} | d krq|d 7}q| d d kr|d 7}|| q,||kr||| q,||kr,|dq,q,|d urX||kr>td|||krXtd|| d|}|S)a Transforms an _ren.py file into the equivalent .rpy file. This should retain line numbers. `filename` If not None, and an error occurs, the error is reported with the given filename. Otherwise, errors are ignored and a a best effort is used. rrrrn)rz"""renpyz"""rtr5rrrNz@In {!r}, there are no """renpy blocks, so every line is ignored.zLIn {!r}, there is a """renpy block at line {} that is not terminated by """.r) splitlines enumeratergr}rr'rr:)rr)rCresultrpZIGNOREZRENPYZPYTHONrrrFZopen_linenumberslrDr`r-r-r.r$sd                r$)NrF)7 __future__rrrrr renpy.compatrrr r r r r rrrrrcodecsr^rYtimer-rZrenpy.lexersupportrr r&r/r'r0objectrMr7rSZlllwordrarjrrvrrsetrZ OPERATORSZESCAPED_OPERATORSr:rrrrrr$r-r-r-r.sJ8 &7  qN d