From martin@martinkjeldsen.dk

When using rules at least with <ws> overwritten, the rule also captures
whitespace. Please see attached file for example.

It is tested with revision 33823.

Martin

From martin@martinkjeldsen.dk

grammar NG {
  rule TOP { ^ <line>+ $ };

  #whitespace rules is borrowed from pipp
  # Whitespace and comments
  token ws_char { \h | \v }

  token singlelinecomment { [ '#' || '//' ] \N* } # end of line comment like Unix shell and C++

  token multilinecomment { '/*' .*? '*/' } # C-like multiline comment

  token ws_all {
  <.ws_char>
  | <.singlelinecomment>
  | <.multilinecomment>
  }

  token ws {
  <.ws_all>*
  }

  rule line {
  'default' \w+
  }
};

my $str = 'default testtesttest
// this is a comment shouldn\'t be outputted';

$str ~~ NG;

die("No match") unless ~$/;

say $/<line>[0];

From @moritz

Martin Kjeldsen (via RT) wrote​:

# New Ticket Created by Martin Kjeldsen
# Please include the string​: [perl #​61308]
# in the subject line of all future correspondence about this issue.
# <URL​: http://rt.perl.org/rt3/Ticket/Display.html?id=61308 >

When using rules at least with <ws> overwritten, the rule also captures
whitespace. Please see attached file for example.

You're right that there's a bug, but the real issue is that 'token ws {
... }' isn't used by implicit <.ws> in rules yet​:

grammar A {
  token ws { 'a' };
  rule b {x y};
};

if 'xab' ~~ m/ ^ <A​::b> $/ {
  say "match";
} else {
  say "no match";
}
# output​: no match\n

Thank you for your report,
Moritz

The RT System itself - Status changed from 'new' to 'open'

From @chrisdolan

On Dec 13, 2008, at 3​:44 AM, Moritz Lenz wrote​:

Martin Kjeldsen (via RT) wrote​:

# New Ticket Created by Martin Kjeldsen
# Please include the string​: [perl #​61308]
# in the subject line of all future correspondence about this issue.
# <URL​: http://rt.perl.org/rt3/Ticket/Display.html?id=61308 >

When using rules at least with <ws> overwritten, the rule also
captures
whitespace. Please see attached file for example.

You're right that there's a bug, but the real issue is that 'token
ws {
... }' isn't used by implicit <.ws> in rules yet​:

grammar A {
token ws { 'a' };
rule b {x y};
};

if 'xab' ~~ m/ ^ <A​::b> $/ {
say "match";
} else {
say "no match";
}
# output​: no match\n

Thank you for your report,
Moritz

In which case, the root cause may be the same as
  [perl #​57864] Calling a token "text", "null" or "ws" in Rakudo
makes matching fail

Chris

From @moritz

Moritz Lenz wrote​:

Martin Kjeldsen (via RT) wrote​:

# New Ticket Created by Martin Kjeldsen
# Please include the string​: [perl #​61308]
# in the subject line of all future correspondence about this issue.
# <URL​: http://rt.perl.org/rt3/Ticket/Display.html?id=61308 >

When using rules at least with <ws> overwritten, the rule also captures
whitespace. Please see attached file for example.

You're right that there's a bug, but the real issue is that 'token ws {
... }' isn't used by implicit <.ws> in rules yet​:

grammar A {
token ws { 'a' };
rule b {x y};
};

if 'xab' ~~ m/ ^ <A​::b> $/ {
say "match";
} else {
say "no match";
}
# output​: no match\n

The output is still "no match" for me, but the tests I've added to
t/spec/S05-grammar/ws.t all pass, and they seem to test for exactly the
same thing. Or maybe I'm too tired to figure it out right now... any ideas?

Cheers,
Moritz

From martin@martinkjeldsen.dk

Moritz Lenz via RT (09​:23 2008-12-14)​:

Martin Kjeldsen (via RT) wrote​:

# New Ticket Created by Martin Kjeldsen
# Please include the string​: [perl #​61308]
# in the subject line of all future correspondence about this issue.
# <URL​: http://rt.perl.org/rt3/Ticket/Display.html?id=61308 >

When using rules at least with <ws> overwritten, the rule also captures
whitespace. Please see attached file for example.

You're right that there's a bug, but the real issue is that 'token ws {
... }' isn't used by implicit <.ws> in rules yet​:

grammar A {
token ws { 'a' };
rule b {x y};
};

if 'xab' ~~ m/ ^ <A​::b> $/ {
say "match";
} else {
say "no match";
}
# output​: no match\n

Thank you for your report,
Moritz

Hi Moritz,

you're right, I can't get your example to work. But I think you meant

if 'xay' ~~ m/ ^ <A​::b> $/ { # y instead of b in string

and shouldn't it be rule b {'x' 'y'}; or doesn't that matter (I don't know).

But no matter it still doesn't work for some reason.

But more importantly if the whole problem is the <ws> is not overwritten I don't
understand why my example matches, but doesn't match if I remove the <ws> token.
Surely <ws> shouldn't default match C style comments.

Martin

From @pmichaud

On Mon Dec 15 03​:43​:26 2008, baest wrote​:

Moritz Lenz via RT (09​:23 2008-12-14)​:

Martin Kjeldsen (via RT) wrote​:

# New Ticket Created by Martin Kjeldsen
# Please include the string​: [perl #​61308]
# in the subject line of all future correspondence about this
issue.
# <URL​: http://rt.perl.org/rt3/Ticket/Display.html?id=61308 >

When using rules at least with <ws> overwritten, the rule also
captures
whitespace. Please see attached file for example.

You're right that there's a bug, but the real issue is that 'token
ws {
... }' isn't used by implicit <.ws> in rules yet​:

grammar A {
token ws { 'a' };
rule b {x y};
};

if 'xab' ~~ m/ ^ <A​::b> $/ {
say "match";
} else {
say "no match";
}
# output​: no match\n

Thank you for your report,
Moritz

Hi Moritz,

you're right, I can't get your example to work. But I think you meant

if 'xay' ~~ m/ ^ <A​::b> $/ { # y instead of b in string

and shouldn't it be rule b {'x' 'y'}; or doesn't that matter (I don't
know).

But no matter it still doesn't work for some reason.

The following version works fine for me​:

$ cat x
grammar A {
  token ws { 'a' };
  rule b {x y};
};

if 'xay' ~~ m/^ <A​::b> $/ { say 'match'; }
else { say 'nomatch'; }

$ ./parrot perl6.pbc x
match
$

So, clearly the <ws> rule is being called.

See next reply for the other example.

Pm

From @pmichaud

On Fri Dec 12 04​:52​:12 2008, baest wrote​:

When using rules at least with <ws> overwritten, the rule also captures
whitespace. Please see attached file for example.

The problem is with trying to invoke the rule using "$str ~~ NG​::TOP" --
that's not a valid syntax for invoking a rule in a grammar.

Using "$str ~~ /<NG​::TOP>/" instead gets the correct match -- see
attached example.

Marking ticket as resolved, thanks!

Pm

From @pmichaud

$ cat NG.pl
grammar NG {
  rule TOP { ^ <line>+ $ };

  #whitespace rules is borrowed from pipp
  # Whitespace and comments
  token ws_char { \h | \v }

  # end of line comment like Unix shell and C++
  token singlelinecomment { [ '#' || '//' ] \N* }

  # C-like multiline comment
  token multilinecomment { '/*' .*? '*/' }

  token ws_all {
  <.ws_char>
  | <.singlelinecomment>
  | <.multilinecomment>
  }

  token ws { <.ws_all>* }

  rule line {
  'default' \w+
  }
};

my $str = 'default testtesttest
// this is a comment shouldn\'t be outputted';

$str ~~ /<NG​::TOP>/;

die("No match") unless ~$/;

say $<NG​::TOP><line>[0];

$ ./parrot perl6.pbc NG.pl
default testtesttest
// this is a comment shouldn't be outputted

@pmichaud - Status changed from 'open' to 'resolved'