Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

utf8-c8 mangles by NFC #5409

Closed
p6rt opened this issue Jul 1, 2016 · 6 comments
Closed

utf8-c8 mangles by NFC #5409

p6rt opened this issue Jul 1, 2016 · 6 comments

Comments

@p6rt
Copy link

p6rt commented Jul 1, 2016

Migrated from rt.perl.org#128512 (status was 'resolved')

Searchable as RT128512$

@p6rt
Copy link
Author

p6rt commented Jul 1, 2016

From zefram@fysh.org

A decode-then-encode cycle through the utf8-c8 encoding is meant to
round-trip an octet string. But if the input is the UTF-8 encoding
of a string that's not NFC normalised, the output ends up different,
because this normalisation got performed somewhere in the middle​:

Blob[uint8].new(101, 204, 129).decode("utf8-c8").encode("utf8-c8").perl
Blob[uint8].new(195,169)
Blob[uint8].new(195, 169).decode("utf8-c8").encode("utf8-c8").perl
Blob[uint8].new(195,169)

This is of particular concern for things like access to command-line
arguments​:

$ perl6 -e 'say @​*ARGS[0].encode("utf8-c8")' $'e\xcc\x81'
Blob[uint8]​:0x<c3 a9>
$ perl6 -e 'say @​*ARGS[0].encode("utf8-c8")' $'\xc3\xa9'
Blob[uint8]​:0x<c3 a9>

-zefram

@p6rt
Copy link
Author

p6rt commented Feb 27, 2017

From zefram@fysh.org

This problem no longer occurs.

-zefram

@p6rt
Copy link
Author

p6rt commented Mar 14, 2018

From @dogbert17

On Sun, 26 Feb 2017 23​:01​:18 -0800, zefram@​fysh.org wrote​:

This problem no longer occurs.

-zefram

Fixed with commit (2017-01-02) rakudo/rakudo@7c8b705

@p6rt
Copy link
Author

p6rt commented Mar 14, 2018

The RT System itself - Status changed from 'new' to 'open'

@p6rt
Copy link
Author

p6rt commented Mar 17, 2018

From @dogbert17

On Wed, 14 Mar 2018 13​:28​:05 -0700, jan-olof.hendig@​bredband.net wrote​:

On Sun, 26 Feb 2017 23​:01​:18 -0800, zefram@​fysh.org wrote​:

This problem no longer occurs.

-zefram

Fixed with commit (2017-01-02)
rakudo/rakudo@7c8b705

Test added with commit Raku/roast@d0d4609

@p6rt p6rt closed this as completed Mar 17, 2018
@p6rt
Copy link
Author

p6rt commented Mar 17, 2018

@dogbert17 - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant