From rubio.terra@gmail.com

I can declare non-ascii tokens without problems on a grammar but I get an encoding error if I refer to the token using the <> syntax. This happens on MoarVM, it works fine on JVM.

This runs OK​:

  perl6 -c -e 'grammar { token TOP { <numero>+ }; token número {<< \d+ >>} }'
  Syntax OK

Now, if I reference the new token (número) I get an error​:

  perl6 -c -e 'grammar { token TOP { <número>+ }; token número {<< \d+ >>} }'
  ===SORRY!===
  Error encoding ASCII string​: could not encode codepoint 250

The same code on JVM runs OK​:

  perl6-j -c -e 'grammar { token TOP { <número>+ }; token número {<< \d+ >>} }'
  Syntax OK

I'm running Linux Fedora 22. Both Rakudo 2015.11 and 2015.12 exhibit the error.

From @lizmat

On 29 Dec 2015, at 17​:35, Rúbio Terra (via RT) <perl6-bugs-followup@​perl.org> wrote​:

# New Ticket Created by Rúbio Terra
# Please include the string​: [perl #​127075]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl6/Ticket/Display.html?id=127075 >

I can declare non-ascii tokens without problems on a grammar but I get an encoding error if I refer to the token using the <> syntax. This happens on MoarVM, it works fine on JVM.

This runs OK​:

perl6 -c -e 'grammar { token TOP { <numero>+ }; token número {<< \d+ >>} }'
Syntax OK

Now, if I reference the new token (número) I get an error​:

perl6 -c -e 'grammar { token TOP { <número>+ }; token número {<< \d+ >>} }'
===SORRY!===
Error encoding ASCII string​: could not encode codepoint 250

The same code on JVM runs OK​:

perl6-j -c -e 'grammar { token TOP { <número>+ }; token número {<< \d+ >>} }'
Syntax OK

I'm running Linux Fedora 22. Both Rakudo 2015.11 and 2015.12 exhibit the error.

This looks like it is happening deep in the bowels of NQP​:

$ 6l 'grammar { token TOP { <número>+ } }'
Error encoding ASCII string​: could not encode codepoint 250
  at gen/moar/stage2/NQPP6QRegex.nqp​:1382 (/Users/liz/Github/rakudo.moar/install/share/nqp/lib/NQPP6QRegex.moarvm​:capnames​:293)
from gen/moar/stage2/NQPP6QRegex.nqp​:1397 (/Users/liz/Github/rakudo.moar/install/share/nqp/lib/NQPP6QRegex.moarvm​:capnames​:478)
from gen/moar/stage2/NQPP6QRegex.nqp​:1361 (/Users/liz/Github/rakudo.moar/install/share/nqp/lib/NQPP6QRegex.moarvm​:capnames​:52)
from gen/moar/stage2/NQPP6QRegex.nqp​:1313 (/Users/liz/Github/rakudo.moar/install/share/nqp/lib/NQPP6QRegex.moarvm​:qbuildsub​:112)

Hope somebody with NQP internals knowledge can pick this up.

Liz

The RT System itself - Status changed from 'new' to 'open'

@coke - Status changed from 'open' to 'new'

From @MasterDuke17

Fixed in Raku/nqp@4fd4b48 and rakudo/rakudo@20ed9e2e2c

Tests added in Raku/roast@e82a3df483

The RT System itself - Status changed from 'new' to 'open'

@AlexDaniel - Status changed from 'open' to 'resolved'