07:59 <cfbolz> korvo: more context needed

07:59 <cfbolz> korvo: we have a peg generator for pypy3, based on the cpython stuff

08:00 <cfbolz> but it's even more adhoc

15:29 <korvo> Yeah, it'd be nice if PyPy were already nicely factored into generic RPython tools.

15:30 <korvo> cfbolz: Concretely, I don't know how to convince RPLY's lexer that a '}' can occur at the start of two different tokens, but which token to match depends on the current parser state. I could do it if the lexer had a stack of states.

15:30 <korvo> Maybe I should just hack RPLY.

15:30 <cfbolz> korvo: which language is that?

15:31 <cfbolz> korvo: "nicely factored into generic rpython tools" hah (rpython has like five users? maybe ten?)

15:31 <korvo> Nix. Here are two legal Nix strings: "{ x = 2; }" and '"x + ${ y } + z"'

15:32 <cfbolz> korvo: and the other implementations have a stateful lexer?

15:33 <cfbolz> is that like f-strings?

15:34 <nikolar> can't you introduce a token for ${

15:34 <korvo> Yeah, it's like f-strings; it's a quasiliteral syntax. The reference implementation is YACC + Bison and uses both lexer and parser states.

15:35 <nikolar> korvo: wouldn't creating a separate token for "${" solve that problem

15:35 <korvo> nikolar: Yes. That doesn't help with recognizing the end of the splice, though. RPLY doesn't have stateful lexing, so the lexer doesn't know whether it's currently in a splice or a string literal.

15:35 <nikolar> ah right

15:37 <cfbolz> korvo: so what I would do is probably the following: hack rply to allow writing your own lexer, then write a stateful lexer by hand

15:38 <korvo> Yeah, I guess so. I still want a better version of this grammar, but I should duplicate the already-working grammar first.

15:38 <cfbolz> the other option could be what pypy does for f-strings: always parse the string as a full token, then later (during bytecode compilation/ast construction) parse the insides of the quasi quotes

15:42 <korvo> Hm. I wonder if that actually forms a regular language (and thus could be lexed); I'm thinking nasty cases like '"x + ${ { y = "z + ${ w }"; } }"' might make it hard to defer parsing. Fun idea though.

15:48 <cfbolz> korvo: that kind of construction used to be forbidden in python

15:49 <nikolar> but not anymore kek

15:50 <cfbolz> thankfully we don't support that version yet

15:51 <nikolar> which version changed that

15:52 <cfbolz> 3.12

15:52 <cfbolz> https://docs.python.org/3/whatsnew/3.12.html#whatsnew312-pep701

15:53 <nikolar> yeah that seems like a pain to parse

15:54 <cfbolz> the old approach is also annoying, so win some lose some. but first we'd have to do 3.11