Skip Menu |
Report information
Id: 124220
Status: open
Priority: 0/
Queue: perl6

Owner: Nobody
Requestors: drforr [at] pobox.com
Cc:
AdminCc:

Severity: (no value)
Tag: (no value)
Platform: (no value)
Patch Status: (no value)
VM: (no value)



Subject: Grammar waterbed-style issue
To: Rakudobug <rakudobug [...] perl.org>
Date: Tue, 31 Mar 2015 22:17:21 +0200
From: drforr [...] pobox.com
Download (untitled) / with headers
text/plain 1.9k
OS: Ubuntu 14.04 LTS on VirtualBox Host OS: Windows 8 Rakudo version: Current as of 25/03/2015 This is a simple parser for function argument syntax. With this there are two surprising behaviors for the price of one. The first is in the token TOP. As the script stands, the test passes. token <term> expands to either a <compound-term> or an <integer>, and of course the first alternative matches, as this trace shows: --cut here-- 「foo(1)」 term => 「foo(1)」 compound-term => 「foo(1)」 atom => 「foo」 argument-list => 「1」 integer => 「1」 --cut here-- The commented-out line should simply bypass expanding <term> into <compound-term>, but instead parsing fails. Note that it's using the same quantifiers in both cases. The other waterbed-style issue is in the second set of commented-out lines. Just like above, the uncommented line works correctly, and expands to the match tree shown above. However, if you write out the <integer>* % ',' inline in the compound-term directly, the match fails. Since actions don't run on a failed parse (a good thing from the point of view of side effects) I don't have much of a way to debug the situation, but I'll look at parser internals later. Something like a regex debugger and/or REPL would be an excellent idea, and I've already started binding Linux libreadline in perl6. Anyway, thoughts for consideration. I'm not certain why the behavior manifests itself, but I'm going to spend some time poking around. --cut here-- use v6; grammar Bug { #token TOP { <compound-term>* % \n } token TOP { <term>* % \n } token term { <compound-term> | <integer> } token atom { <[a..z]>+ } token integer { <[0..9]>+ } token argument-list { <integer>* % ',' } token compound-term { #<atom> '(' <integer>* % ',' ')' # This term should be the expanded form of <atom> '(' <argument-list> ')' # This term here, yet the above generates an error. } } use Test; ok Bug.parse('foo(1)'); --cut here--
Subject: [BUG] Grammar waterbed-style issue
On Tue, 31 Mar 2015 13:17:41 -0700, drforr@pobox.com wrote: Show quoted text
> OS: Ubuntu 14.04 LTS on VirtualBox > Host OS: Windows 8 > Rakudo version: Current as of 25/03/2015 > > This is a simple parser for function argument syntax. > > With this there are two surprising behaviors for the price of one. The > first is in the token TOP. As the script stands, the test passes. > > token <term> expands to either a <compound-term> or an <integer>, and of > course the first alternative matches, as this trace shows: > > --cut here-- > 「foo(1)」 > term => 「foo(1)」 > compound-term => 「foo(1)」 > atom => 「foo」 > argument-list => 「1」 > integer => 「1」 > --cut here-- > > The commented-out line should simply bypass expanding <term> into > <compound-term>, but instead parsing fails. Note that it's using the > same quantifiers in both cases. > > The other waterbed-style issue is in the second set of commented-out > lines. > > Just like above, the uncommented line works correctly, and expands to > the match tree shown above. However, if you write out the <integer>* % > ',' inline in the compound-term directly, the match fails. Since actions > don't run on a failed parse (a good thing from the point of view of side > effects) I don't have much of a way to debug the situation, but I'll > look at parser internals later. Something like a regex debugger and/or > REPL would be an excellent idea, and I've already started binding Linux > libreadline in perl6. > > Anyway, thoughts for consideration. I'm not certain why the behavior > manifests itself, but I'm going to spend some time poking around. > > --cut here-- > use v6; > grammar Bug { > > #token TOP { <compound-term>* % \n } > token TOP { <term>* % \n } > > token term { > <compound-term> > | <integer> > } > > token atom { <[a..z]>+ } > token integer { <[0..9]>+ } > > token argument-list { <integer>* % ',' } > > token compound-term { > #<atom> '(' <integer>* % ',' ')' # This term should be the expanded > form of > <atom> '(' <argument-list> ')' # This term here, yet the above > generates an error. > } > } > > use Test; > ok Bug.parse('foo(1)'); > --cut here--
Update: I was only able to replicate this when keeping the first comment and uncommenting the second one. No other combination failed the parse. The failing case golfs down to: $ perl6 -e 'grammar Bug { token TOP { f "(" 1* % "," ")" | 1 } }; Bug.parse("f(1)"); $/.say;' Nil $ perl6 -e 'grammar NoBug { token TOP { f "(" <b> ")" | 1 }; token b { 1* % "," } }; NoBug.parse("f(1)"); $/.say;' 「f(1)」 b => 「1」 A sequence point will "work around" the bug: $ perl6 -e 'grammar Bug { token TOP { f "(" 1* % "," {} ")" | 1 } }; Bug.parse("f(1)"); $/.say;' 「f(1)」 ...so will using 1+ instead of 1*. This would make me suspect that this was merely a problem with backtracking... ISTR there was some sort of issue when using patterns that can match 0 chars. However, I tried changing <b> to a regex instead of a token to remove the ratchet, and that failed to reproduce the problem. So -^o^-


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org