From @drforr

OS​: Ubuntu 14.04 LTS on VirtualBox
Host OS​: Windows 8
Rakudo version​: Current as of 25/03/2015

This is a simple parser for function argument syntax.

With this there are two surprising behaviors for the price of one. The
first is in the token TOP. As the script stands, the test passes.

token <term> expands to either a <compound-term> or an <integer>, and of
course the first alternative matches, as this trace shows​:

--cut here--
｢foo(1)｣
  term => ｢foo(1)｣
  compound-term => ｢foo(1)｣
  atom => ｢foo｣
  argument-list => ｢1｣
  integer => ｢1｣
--cut here--

The commented-out line should simply bypass expanding <term> into
<compound-term>, but instead parsing fails. Note that it's using the
same quantifiers in both cases.

The other waterbed-style issue is in the second set of commented-out
lines.

Just like above, the uncommented line works correctly, and expands to
the match tree shown above. However, if you write out the <integer>* %
',' inline in the compound-term directly, the match fails. Since actions
don't run on a failed parse (a good thing from the point of view of side
effects) I don't have much of a way to debug the situation, but I'll
look at parser internals later. Something like a regex debugger and/or
REPL would be an excellent idea, and I've already started binding Linux
libreadline in perl6.

Anyway, thoughts for consideration. I'm not certain why the behavior
manifests itself, but I'm going to spend some time poking around.

--cut here--
use v6;
grammar Bug {

  #token TOP { <compound-term>* % \n }
  token TOP { <term>* % \n }

  token term {
  <compound-term>
  | <integer>
  }

  token atom { <[a..z]>+ }
  token integer { <[0..9]>+ }

  token argument-list { <integer>* % ',' }

  token compound-term {
  #<atom> '(' <integer>* % ',' ')' # This term should be the expanded
form of
  <atom> '(' <argument-list> ')' # This term here, yet the above
generates an error.
  }
}

use Test;
ok Bug.parse('foo(1)');
--cut here--

From @skids

On Tue, 31 Mar 2015 13​:17​:41 -0700, drforr@​pobox.com wrote​:

OS​: Ubuntu 14.04 LTS on VirtualBox
Host OS​: Windows 8
Rakudo version​: Current as of 25/03/2015

This is a simple parser for function argument syntax.

With this there are two surprising behaviors for the price of one. The
first is in the token TOP. As the script stands, the test passes.

token <term> expands to either a <compound-term> or an <integer>, and of
course the first alternative matches, as this trace shows​:

--cut here--
｢foo(1)｣
term => ｢foo(1)｣
compound-term => ｢foo(1)｣
atom => ｢foo｣
argument-list => ｢1｣
integer => ｢1｣
--cut here--

The commented-out line should simply bypass expanding <term> into
<compound-term>, but instead parsing fails. Note that it's using the
same quantifiers in both cases.

The other waterbed-style issue is in the second set of commented-out
lines.

Just like above, the uncommented line works correctly, and expands to
the match tree shown above. However, if you write out the <integer>* %
',' inline in the compound-term directly, the match fails. Since actions
don't run on a failed parse (a good thing from the point of view of side
effects) I don't have much of a way to debug the situation, but I'll
look at parser internals later. Something like a regex debugger and/or
REPL would be an excellent idea, and I've already started binding Linux
libreadline in perl6.

Anyway, thoughts for consideration. I'm not certain why the behavior
manifests itself, but I'm going to spend some time poking around.

--cut here--
use v6;
grammar Bug {

#token TOP { <compound-term>* % \n }
token TOP { <term>* % \n }

token term {
<compound-term>
| <integer>
}

token atom { <[a..z]>+ }
token integer { <[0..9]>+ }

token argument-list { <integer>* % ',' }

token compound-term {
#<atom> '(' <integer>* % ',' ')' # This term should be the expanded
form of
<atom> '(' <argument-list> ')' # This term here, yet the above
generates an error.
}
}

use Test;
ok Bug.parse('foo(1)');
--cut here--

Update​:

I was only able to replicate this when keeping the first comment
and uncommenting the second one. No other combination failed the
parse.

The failing case golfs down to​:

$ perl6 -e 'grammar Bug { token TOP { f "(" 1* % "," ")" | 1 } }; Bug.parse("f(1)"); $/.say;'
Nil
$ perl6 -e 'grammar NoBug { token TOP { f "(" <b> ")" | 1 }; token b { 1* % "," } }; NoBug.parse("f(1)"); $/.say;'
｢f(1)｣
b => ｢1｣

A sequence point will "work around" the bug​:

$ perl6 -e 'grammar Bug { token TOP { f "(" 1* % "," {} ")" | 1 } }; Bug.parse("f(1)"); $/.say;'
｢f(1)｣

...so will using 1+ instead of 1*. This would make me suspect that this was merely a
problem with backtracking... ISTR there was some sort of issue when using patterns
that can match 0 chars.

However, I tried changing <b> to a regex instead of a token to remove the ratchet,
and that failed to reproduce the problem. So -^o^-

The RT System itself - Status changed from 'new' to 'open'