Skip Menu |
Queue is disabled
This queue is disabled and you may not create new tickets in it. Disabled queues are usually because the distribution was merged with another or changed names. Sometimes they are the end result of a bad autocreate from PAUSE data before anyone noticed.

Subject: Windows REPL cannot handle «, » characters
Date: Tue, 14 Nov 2017 04:29:36 +0000
From: Dan Zwell <dan [...] zwell.net>
To: "rakudobug [...] perl.org" <rakudobug [...] perl.org>
Download (untitled) / with headers
text/plain 968b
Note: results are from Windows 10. Same results from CMD and from Cmder.
Rakudo version 2017.09-355-g27131ed8d built on MoarVM version 2017.09.1-575-gd4e230a6

Working:
perl6 -e "<a b>».say" # output is as expected

But in the REPL, I tried to run:
> <a b>».say

Without Linenoise installed, Rakudo endlessly repeats "> Decoder may not be used concurrently"

With Linenoise installed, I get this error:

Malformed UTF-8 at line 1 col 6
  in sub nativecast at D:\rakudo\share\perl6\sources\51E302443A2C8FF185ABC10CA1E5520EFEE885A1 (NativeCall::Types) line 5
  in method deref at D:\rakudo\share\perl6\sources\51E302443A2C8FF185ABC10CA1E5520EFEE885A1 (NativeCall::Types) line 58
  in sub linenoise at D:\rakudo\share\perl6\site\sources\0BDF8C54D33921FEA066491D8D13C96A7CB144B9 (Linenoise) line 86

I got similar error when using these characters as quoting operators, such as: q«text».

Note: the obvious workaround is to substitute "<<" and ">>" for "«" and "»".
Marked as 「SEVERE」 because it's very unfortunate that non-ascii stuff does not work. I don't know why it happens though, so people should feel free to retag as needed.

On 2017-11-13 20:29:52, dan@zwell.net wrote:
Show quoted text
> Note: results are from Windows 10. Same results from CMD and from Cmder.
> Rakudo version 2017.09-355-g27131ed8d built on MoarVM version
> 2017.09.1-575-gd4e230a6
>
> Working:
> perl6 -e "<a b>».say" # output is as expected
>
> But in the REPL, I tried to run:
> > <a b>».say
>
> Without Linenoise installed, Rakudo endlessly repeats "> Decoder may not be
> used concurrently"
>
> With Linenoise installed, I get this error:
>
> Malformed UTF-8 at line 1 col 6
> in sub nativecast at
> D:\rakudo\share\perl6\sources\51E302443A2C8FF185ABC10CA1E5520EFEE885A1
> (NativeCall::Types) line 5
> in method deref at
> D:\rakudo\share\perl6\sources\51E302443A2C8FF185ABC10CA1E5520EFEE885A1
> (NativeCall::Types) line 58
> in sub linenoise at
> D:\rakudo\share\perl6\site\sources\0BDF8C54D33921FEA066491D8D13C96A7CB144B9
> (Linenoise) line 86
>
> I got similar error when using these characters as quoting operators, such
> as: q«text».
>
> Note: the obvious workaround is to substitute "<<" and ">>" for "«" and "»".


RT-Send-CC: perl6-compiler [...] perl.org
On Mon, 13 Nov 2017 20:29:52 -0800, dan@zwell.net wrote: Show quoted text
> With Linenoise installed, I get this error: > > Malformed UTF-8 at line 1 col 6
That looks like it might be working correctly, except cmd.exe wasn't told to UTF-8 code page (need to run `chcp 65001`). Show quoted text
> But in the REPL, I tried to run:
> > <a b>».say
> > Without Linenoise installed, Rakudo endlessly repeats "> Decoder may not be > used concurrently"
On 2017.07 on Win7 with 65001 code page enabled, the » char doesn't show up at all. Just seems to get removed from the content if I paste it into the terminal. The default REPL uses `prompt` to read stuff and here's another datapoint: prompt() returns Nil if the data it reads has fancy Unicode chars like » or 「」: C:\Users\zoffi>perl6 -e "say prompt" » Nil C:\Users\zoffi>perl6 -e "say prompt" 42 42 C:\Users\zoffi>perl6 -e "say prompt" 42» Nil C:\Users\zoffi>perl6 -e "say prompt" »42 Nil Looks like it might be a good place to start looking at for the OP bug.
Download (untitled) / with headers
text/plain 478b
On Thu, 16 Nov 2017 09:53:46 -0800, cpan@zoffix.com wrote: Show quoted text
> The default REPL uses `prompt` to read stuff and here's another > datapoint: prompt() returns Nil if the data it reads has fancy Unicode > chars like » or 「」: > > C:\Users\zoffi>perl6 -e "say prompt" > » > Nil
looks to be a problem in moarvm. perl6 -e "say $*IN.read-internal(1000)" gives an empty buf (Buf[uint8]:0x<>) if the input got fancypants chars https://irclog.perlgeek.de/perl6/2017-11-21#i_15478971
RT-Send-CC: perl6-compiler [...] perl.org
Download (untitled) / with headers
text/plain 1.7k
On Thu, 16 Nov 2017 09:53:46 -0800, cpan@zoffix.com wrote: Show quoted text
> On 2017.07 on Win7 with 65001 code page enabled, the » char doesn't > show up at all. Just seems to get removed from the content if I paste > it into the terminal.
Starting to think this might be a limitation of cmd.exe. Though strangely, I'm failing to find anyone mentioning this problem on Google... For example, Perl 5 has the exact same problem that the fancy chars get stripped: C:\rakudo>perl -wlE "say '[' . scalar(readline) . ']'" e »♥ b [e b ] I also cobbled together this C program from MSDN's code examples and the same problem is present in it as well: #include <fcntl.h> #include <io.h> #include <stdlib.h> #include <stdio.h> #include <share.h> int main( void ) { int fh, i; unsigned char buffer[60000]; unsigned int nbytes = 60000, bytesread; int result; result = _setmode(_fileno(stdin), _O_BINARY); if( result == -1 ) perror( "Cannot set mode" ); else printf( "'stdin' successfully changed to binary mode\n" ); if( ( bytesread = _read( _fileno( stdin ), buffer, nbytes ) ) <= 0 ) perror( "Problem reading file" ); else printf( "Read %u bytes from file\n", bytesread ); printf("Read this: `"); for (i = 0; i < bytesread; i++) printf("%u ", buffer[i]); printf("`\n================\n"); return 1; } Anything fancy gets read as a nul byte instead of the proper bytes for that char: C:\rakudo>gcc test.c && a.exe 'stdin' successfully changed to binary mode e »♥ b Read 8 bytes from file Read this: `101 32 0 0 32 98 13 10 ` ================ C:\rakudo>
From: Brandon Allbery <allbery.b [...] gmail.com>
Date: Tue, 26 Dec 2017 00:15:41 -0500
To: Carl Mäsak via RT <perl6-bugs-followup [...] perl.org>
CC: perl6-compiler <perl6-compiler [...] perl.org>
Subject: Re: [perl #132441] [SEVERE][WINDOWS][IO] IO::Handle.read-internal cannot handle fancy Unicode chars on TTY handles
Download (untitled) / with headers
text/plain 815b
On Mon, Dec 25, 2017 at 1:07 AM, Zoffix Znet via RT <perl6-bugs-followup@perl.org> wrote:
Show quoted text
On Thu, 16 Nov 2017 09:53:46 -0800, cpan@zoffix.com wrote:
> On 2017.07 on Win7 with 65001 code page enabled, the » char doesn't
> show up at all. Just seems to get removed from the content if I paste
> it into the terminal.

Starting to think this might be a limitation of cmd.exe. Though strangely,
I'm failing to find anyone mentioning this problem on Google...

IIRC this is known, and not really fixable. It's not even cmd.exe but a Windows console mode limitation.

--
brandon s allbery kf8nh                               sine nomine associates
allbery.b@gmail.com                                  ballbery@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net
Subject: Re: [perl #132441] [SEVERE][WINDOWS][IO] IO::Handle.read-internal cannot handle fancy Unicode chars on TTY handles
To: Carl Mäsak via RT <perl6-bugs-followup [...] perl.org>
CC: dan [...] zwell.net
Date: Tue, 26 Dec 2017 00:23:46 -0500
From: Brandon Allbery <allbery.b [...] gmail.com>
Download (untitled) / with headers
text/plain 577b
On Tue, Dec 26, 2017 at 12:15 AM, Brandon Allbery via RT <perl6-bugs-followup@perl.org> wrote:
Show quoted text
IIRC this is known, and not really fixable. It's not even cmd.exe but a
Windows console mode limitation.

Come to think of it, there should be existing mention of this on the moarvm bug tracker (ticket may have been closed as unfixable).
 
--
brandon s allbery kf8nh                               sine nomine associates
allbery.b@gmail.com                                  ballbery@sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org