Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode handling on Windows command line #5244

Closed
p6rt opened this issue Apr 18, 2016 · 16 comments
Closed

Unicode handling on Windows command line #5244

p6rt opened this issue Apr 18, 2016 · 16 comments

Comments

@p6rt
Copy link

p6rt commented Apr 18, 2016

Migrated from rt.perl.org#127925 (status was 'resolved')

Searchable as RT127925$

@p6rt
Copy link
Author

p6rt commented Apr 18, 2016

From joerg.thomas.de@googlemail.com

I hope Unicode passes through via e-mail, if not please refer to the
identical description at https://stackoverflow.com/q/36648940/1529709

Unicode handling on the Windows command line fails​:

C​:\Windows\System32>perl6 -e "'Я'.say"
?

Interestingly, this works​:

C​:\Windows\System32>perl6 -e "Buf.new(0xD0, 0xAF).decode('UTF-8').say"
Я

Seen on Rakudo version 2016.01.1 built on MoarVM version 2016.01
Windows 8.1 64-bit

@p6rt
Copy link
Author

p6rt commented Apr 19, 2016

From @zoffixznet

On Mon Apr 18 13​:51​:31 2016, joerg.thomas.de@​googlemail.com wrote​:

Unicode handling on the Windows command line fails​:

C​:\Windows\System32>perl6 -e "'Я'.say"
?

Does it still fail if you change cmd.exe to use unicode code page? Run chcp 65001 in the terminal first.

@p6rt
Copy link
Author

p6rt commented Apr 19, 2016

The RT System itself - Status changed from 'new' to 'open'

@p6rt
Copy link
Author

p6rt commented Apr 19, 2016

From @zoffixznet

On Tue Apr 19 11​:00​:00 2016, cpan@​zoffix.com wrote​:

Does it still fail if you change cmd.exe to use unicode code page? Run
chcp 65001 in the terminal first.

To answer my own question​: yes, it still fails. It also fails if you spurt it into a file.

Looking around, I found this StackOverflow answer with one of the commenters claiming there are bugs with cmd.exe and programs that use IO from C standard lib​: http://stackoverflow.com/questions/388490/unicode-characters-in-windows-command-line-how/388500#​388500

@p6rt
Copy link
Author

p6rt commented Feb 7, 2017

From @zoffixznet

Another report in NQP repo​: Raku/nqp#346

------------->8----------------------

It is entirely possible that I am missing something obvious, but while trying to figure out what happens between typing

C​:\> perl6 -e "say 'yağmur'"

and getting the output

yagmur

@p6rt
Copy link
Author

p6rt commented Feb 7, 2017

From @zoffixznet

FWIW, nqp -e "say('yağmur')" and perl -C -wlE "use utf8; say('yağmur')" have the same issue (the output comes out as `yagmur`, without the thingie on `g`). So, perhaps this is a Windows command prompt issue?

@p6rt
Copy link
Author

p6rt commented Feb 7, 2017

From 1parrota@gmail.com

A quick look at Stackoverflow suggests that Windows isn't being
terribly helpful.

On 2/7/17, Zoffix Znet via RT <perl6-bugs-followup@​perl.org> wrote​:

Another report in NQP repo​: Raku/nqp#346

------------->8----------------------

It is entirely possible that I am missing something obvious, but while
trying to figure out what happens between typing

C​:\> perl6 -e "say 'yağmur'"

and getting the output

yagmur

@p6rt
Copy link
Author

p6rt commented Feb 7, 2017

From @nanis

The problem is caused by the fact that moar.exe uses main rather than
wmain, so it has no hope of getting the right characters if anything
outside of the code page is used.

However, I wrote a wrapper around moar.exe which basically does what
perl6.bat does but UTF-8 encodes the command line arguments which
moar.exe relays to whatever executes Perl 6 code.

In that case, I get incorrectly encoded output as in​:

C​:\> p6run -e "say 'yağmur'"
yaÄŸmur

and

C​:\> p6run yağmur.pl6
Could not open yaÄŸmur.pl6. Failed to stat file​: no such file or directory

perl and a bunch of other programs also use main instead of wmain, so
they also end up echoing "yagmur" instead of "yağmur" and attempt to
open "yagmur.pl" instead of "yağmur.pl6", but that is beside the
point.

I would like to know what encoding is assumed for command line
arguments to perl6 and where that is determined/implemented/handled.

Thank you.

-- Sinan

@p6rt
Copy link
Author

p6rt commented Feb 7, 2017

From @nanis

I created this report by mistake when I was hastily trying to follow-up on

https://rt.perl.org/Public/Bug/Display.html?id=127925

The reply belongs there. I would appreciate it if you could merge this
ticket with the correct one.

Apologies and thank you.

@p6rt
Copy link
Author

p6rt commented Feb 7, 2017

From @nanis

@​Parrot Raiser, please see

Raku/nqp#346 (comment)

Raku/nqp#346 (comment)

Raku/nqp#346 (comment)

-- Sinan

@p6rt
Copy link
Author

p6rt commented Feb 8, 2017

From @nanis

See also MoarVM/MoarVM#527

@p6rt
Copy link
Author

p6rt commented Feb 10, 2017

From @nanis

I think my pull request has reached the point where it should work on
others' machines, too ;-)

Please try it out​:

https://github.com/MoarVM/MoarVM/pull/528/files?diff=split

@p6rt
Copy link
Author

p6rt commented Feb 11, 2017

From @nanis

https://github.com/MoarVM/MoarVM/pull/528/files?diff=split

was merged so this ticket can be closed. Thank you.

-- Sinan

@p6rt
Copy link
Author

p6rt commented Feb 11, 2017

From @zoffixznet

Marking as testsneeded.

@p6rt
Copy link
Author

p6rt commented Apr 26, 2017

From @zoffixznet

Tests added Raku/roast@5eb61ccaf2

@p6rt p6rt closed this as completed Apr 26, 2017
@p6rt
Copy link
Author

p6rt commented Apr 26, 2017

@zoffixznet - Status changed from 'open' to 'resolved'

@p6rt p6rt added the testneeded label Jan 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant