Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode handling of warnings appears broken in 5.18.2 and 5.24.0 #15469

Closed
p5pRT opened this issue Jul 22, 2016 · 8 comments
Closed

Unicode handling of warnings appears broken in 5.18.2 and 5.24.0 #15469

p5pRT opened this issue Jul 22, 2016 · 8 comments

Comments

@p5pRT
Copy link

p5pRT commented Jul 22, 2016

Migrated from rt.perl.org#128701 (status was 'resolved')

Searchable as RT128701$

@p5pRT
Copy link
Author

p5pRT commented Jul 22, 2016

From ajs@ajs.com

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line 4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line 4.

Or at least an error about the use of an invalid punctuation character as a
delimiter (perhaps "what do you think this is, Perl 6?!"). But definitely
not munged latin-1...

Also tried this under 5.24.0 and got the same output.

--
Aaron Sherman, M.​:
P​: 617-440-4332 Google Talk, Email and Google Plus​: ajs@​ajs.com
Toolsmith, developer, gamer and life-long student.

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @cpansprout

On Fri Jul 22 10​:36​:25 2016, ajs wrote​:

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line 4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line 4.

Or at least an error about the use of an invalid punctuation character as a
delimiter (perhaps "what do you think this is, Perl 6?!"). But definitely
not munged latin-1...

Also tried this under 5.24.0 and got the same output.

The same bug occurs with the ‘(Might be a runaway multi-line...)’ hint that sometimes accompanies a syntax error.

In a utf-8 terminal, with 5.12.4​:

$ perl -CS -e 'use utf8; q Ω' -e 'Ω >'
syntax error at -e line 2, at EOF
  (Might be a runaway multi-line ?????????????????????????? string starting on line 1)
Execution of -e aborted due to compilation errors.

And 5.25.3​:

$ ./perl -Ilib -CS -e 'use utf8; q Ω' -e 'Ω >'
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
Use of code point 0xFFFFFFFFFFFFFFCE is deprecated; the permissible max is 0x7FFFFFFFFFFFFFFF at -e line 2.
syntax error at -e line 2, at EOF
  (Might be a runaway multi-line ?????????????????????????? string starting on line 1)
Execution of -e aborted due to compilation errors.

Ow!

(Another thing I might note is that the runaway quote hint should include the operator if the delimiter is non-standard, because this is unhelpful​:

$ perl -e 'q x' -e 'x >'
syntax error at -e line 2, at EOF
  (Might be a runaway multi-line xx string starting on line 1)
Execution of -e aborted due to compilation errors.
)
--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @cpansprout

On Tue Jul 26 00​:04​:19 2016, sprout wrote​:

On Fri Jul 22 10​:36​:25 2016, ajs wrote​:

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line 4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line 4.

This is fixed in cb65013. But there are problems with heredoc terminators (handled by the same code, more or less), so there is still some more fixing to do​:

$ ./perl -Ilib -CS -e 'use utf8; <<"«"'
Can't find string terminator "«" anywhere before EOF at -e line 1.

The same bug occurs with the ‘(Might be a runaway multi-line...)’ hint
that sometimes accompanies a syntax error.

In a utf-8 terminal, with 5.12.4​:

$ perl -CS -e 'use utf8; q Ω' -e 'Ω >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line ?????????????????????????? string
starting on line 1)
Execution of -e aborted due to compilation errors.

That is not yet fixed.

(Another thing I might note is that the runaway quote hint should
include the operator if the delimiter is non-standard, because this is
unhelpful​:

$ perl -e 'q x' -e 'x >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line xx string starting on line 1)
Execution of -e aborted due to compilation errors.
)

Neither is that.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 27, 2016

From @cpansprout

On Tue Jul 26 01​:55​:10 2016, sprout wrote​:

On Tue Jul 26 00​:04​:19 2016, sprout wrote​:

On Fri Jul 22 10​:36​:25 2016, ajs wrote​:

$ perl -C x.pl
Can't find string terminator "Â" anywhere before EOF at x.pl line
4.
$ cat x.pl
use v5.18.0;
use utf8;

say q« foo4 » ;
$perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for
x86_64-linux-gnu-thread-multi

Expected​:
Can't find string terminator "«" anywhere before EOF at x.pl line
4.

This is fixed in cb65013. But there are problems with heredoc
terminators (handled by the same code, more or less), so there is
still some more fixing to do​:

$ ./perl -Ilib -CS -e 'use utf8; <<"«"'
Can't find string terminator "«" anywhere before EOF at -e line 1.

That one is fixed now in commit 7f7f7d0.

The same bug occurs with the ‘(Might be a runaway multi-line...)’
hint
that sometimes accompanies a syntax error.

In a utf-8 terminal, with 5.12.4​:

$ perl -CS -e 'use utf8; q Ω' -e 'Ω >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line ?????????????????????????? string
starting on line 1)
Execution of -e aborted due to compilation errors.

That is not yet fixed.

Actually, cb65013 *did* fix it. I didn’t realise that Perl_sv_catpvf supports %c with numbers > 255, but apparently it does. However, it only goes up to 32 bits (at least on a Mac), because %c takes an int (which is 32 bits).

That means you get garbled output for heredoc delimiters above \x{7fffffff}, but those are not valid Unicode characters anyway, and hence cannot display in a terminal. So I’m not going to worry about it. (I started to write a patch, but it adds to much complexity to the code.)

(Another thing I might note is that the runaway quote hint should
include the operator if the delimiter is non-standard, because this
is
unhelpful​:

$ perl -e 'q x' -e 'x >'
syntax error at -e line 2, at EOF
(Might be a runaway multi-line xx string starting on line 1)
Execution of -e aborted due to compilation errors.
)

Neither is that.

I have decided to leave this alone. It’s not easy to fix, and nobody has complained about it.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 27, 2016

@cpansprout - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

p5pRT commented May 30, 2017

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Author

p5pRT commented May 30, 2017

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant