Skip Menu |

Subject: RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
Download (untitled) / with headers
text/plain 3.5k
One of the few remaining warts[1] in Perl's Unicode support is how sysread() and syswrite() behave on streams with a unicode layer. First sysread(): For example: open my $fh, "<:utf8", "filewithutf8.txt" or die; my $buf; sysread $fh, $buf, 1000; will reads up to 1000 unvalidated UTF-8[2] *characters* from the stream. That seems all fine and good, but the following: open my $fh, "<:encoding(UCS-2BE)", "filewithucs2be.txt" or die; my $buf; sysread $fh, $buf, 1000; does exactly the same thing - only the fact that the stream is unicode flagged (i.e., has the PERLIO_F_UTF8 flag) is referenced, the actual layers are ignored. This behaviour is mostly documented by: Note that if the filehandle has been marked as C<:utf8> Unicode characters are read instead of bytes (the LENGTH, OFFSET, and the return value of sysread() are in Unicode characters). The C<:encoding(...)> layer implicitly introduces the C<:utf8> layer. See L</binmode>, L</open>, and the C<open> pragma, L<open>. which skips mentioning that the "Unicode characters" read are always UTF-8 encoded. This, beyond the broken :utf8 layer itself, is one of the few pure perl vectors for badly encoded SVf_UTF8 strings in the perl interpreter. Also it can be confusing, even an experienced CPAN author managed to get it wrong[3]. My suggestion is that (eventually) sysread() on a file with the PERLIO_F_UTF8 flag on should either do a simple octet read, as it does without that flag, or fail. For the transition sysread() would warn when passed a handle with PERLIO_F_UTF8, presumably something like "sysread() on a unicode handle is deprecated". So what's the desired behaviour after the transition: 1) sysread() would act as if the flag was not there, completely ignoring the layers rather than ignoring the layers *except* for the flag. This has the advantage that sysread() behaves consistently after the change. It may however make code that depends on the old behaviour silently misbehave. 2) sysread() fails, probably with EINVAL. While sysread() becomes no longer useful on handles with the flag, mixing low- and high-level I/O is generally unsafe anyway, and PerlIO layers are pretty much a high-level construct, so there isn't much lost. It prevents most silent mis-behaviour while remaining true to sysread()'s contract to read bytes from a file. 3) sysread() croaks. Similar to 2), but with more emphasis. Then syswrite(): Unlike sysread(), syswrite() doesn't act a a vector for producing corrupt internal perl data structures, but it does have the same issue that it pays attention to only part of the layer state for the handle. For example: open my $fh, ">:utf8", "filetobeutf8.txt" or die; my $data = "\x{101}"; syswrite $fh, $data; will write UTF-8 encoded data to the file, which is fine, but: open my $fh, ">:encoding(UCS-2BE)", "filetobeucs2.txt" or die; my $data = "\x{101}"; syswrite $fh, $data; does the same thing. I believe if syswrite() is going to ignore any of the layer state of the handle, it should ignore it all, so the examples above would throw an exception, just as they do for handles without the flag when syswrite() is called with wide characters. Again, for the transition, syswrite() should produce a deprecation warning. Tony [1] the other I can think of is that the :utf8 PerlIO layer doesn't validate, it's just a flag [2] or utf8, perl's internal encoding [3] https://rt.cpan.org/Public/Bug/Display.html?id=83126 and a few other tickets for the same distribution, and https://rt.perl.org/Ticket/Display.html?id=121870
Date: Thu, 06 Aug 2015 09:12:26 +0200
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
To: perl5-porters [...] perl.org
From: Peter Rabbitson <rabbit-p5p [...] rabbit.us>
Download (untitled) / with headers
text/plain 759b
On 08/06/2015 09:00 AM, Tony Cook (via RT) wrote: Show quoted text
> # New Ticket Created by Tony Cook > # Please include the string: [perl #125760] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=125760 > > > > One of the few remaining warts[1] in Perl's Unicode support is how > sysread() and syswrite() behave on streams with a unicode layer. >
Having read the excellent analysis, my 2c is that both failure cases for sysread and syswrite should ultimately croak. I do not have an informed opinion on what the deprecation cycle would look like, as it is likely very beneficial to exercise the croaking as early as 5.23.x on a CPAN smoke, yet it is clearly too early for code in the wild.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 101b
I think I am the original vector of these vectors... and I think I was wrong. Just make them croak.
CC: perl5-porters [...] perl.org
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
From: Ricardo Signes <perl.p5p [...] rjbs.manxome.org>
To: Peter Rabbitson <rabbit-p5p [...] rabbit.us>
Date: Thu, 6 Aug 2015 19:11:59 -0400
Download (untitled) / with headers
text/plain 837b
* Peter Rabbitson <rabbit-p5p@rabbit.us> [2015-08-06T03:12:26] Show quoted text
> Having read the excellent analysis, my 2c is that both failure cases for > sysread and syswrite should ultimately croak.
Yes. Thanks, Tony, and I agree. Show quoted text
> I do not have an informed opinion on what the deprecation cycle would look > like, as it is likely very beneficial to exercise the croaking as early as > 5.23.x on a CPAN smoke, yet it is clearly too early for code in the wild.
We should definitely get the warnings in place soon. I think it would be beneficial if we had a way to mark any deprecation warning as fatal, process wide, for the purpose of smoking (and other places, like integration testing), but I think it needs more thought than a "hey it would be neat" from me. But if we're going to make it croak in 5.28, time to make it warn now. -- rjbs
Download signature.asc
application/pgp-signature 473b

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 280b
On Thu Aug 06 16:12:45 2015, perl.p5p@rjbs.manxome.org wrote: Show quoted text
> But if we're going to make it croak in 5.28, time to make it warn now.
Patch attached. As chansen mentioned in #p5p, send() and recv() have the same issue, so the patch also deprecates them on :utf8 handles. Tony
Subject: 0001-perl-125760-deprecate-sys-read-write-send-recv-on-ut.patch
From 549d60629b72c2b689e97815e582d6bc355d24db Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Mon, 10 Aug 2015 16:15:48 +1000 Subject: [PATCH] [perl #125760] deprecate sys(read|write)(), send(), recv() on :utf8 --- pod/perldiag.pod | 21 +++++++++++++++++++++ pp_sys.c | 8 ++++++++ t/lib/warnings/pp_sys | 22 ++++++++++++++++++++++ t/op/gmagic.t | 1 + t/uni/overload.t | 1 + 5 files changed, 53 insertions(+) diff --git a/pod/perldiag.pod b/pod/perldiag.pod index 4f21dbe..f47fd3e 100644 --- a/pod/perldiag.pod +++ b/pod/perldiag.pod @@ -2619,6 +2619,27 @@ provides a list context to its subscript, which can do weird things if you're expecting only one subscript. When called in list context, it also returns the key in addition to the value. +=item %s() is deprecated on :utf8 handles + +(W deprecated) The sysread(), recv(), syswrite() and send() operators +are deprecated on handles that have the C<:utf8> layer, either +explicitly, or implicitly, eg., with the C<:encoding(UTF-16LE)> layer. + +Both sysread() and recv() currently use only the C<:utf8> flag for the +stream, ignoring the actual layers. Since sysread() and recv() do no +UTF-8 validation they can end up creating invalidly encoded scalars. + +Similarly, syswrite() and send() use only the C<:utf8> flag, otherwise +ignoring any layers. If the flag is set, both write the value UTF-8 +encoded, even if the layer is some different encoding, such as the +example above. + +Ideally, all of these operators would completely ignore the C<:utf8> +state, working only with bytes, but this would result in silently +breaking existing code. To avoid this a future version of perl will +throw an exception when any of sysread(), recv(), syswrite() or send() +are called on handle with the C<:utf8> layer. + =item Insecure dependency in %s (F) You tried to do something that the tainting mechanism didn't like. diff --git a/pp_sys.c b/pp_sys.c index ebd675b..dc1b3ce 100644 --- a/pp_sys.c +++ b/pp_sys.c @@ -1691,6 +1691,11 @@ PP(pp_sysread) fd = PerlIO_fileno(IoIFP(io)); if ((fp_utf8 = PerlIO_isutf8(IoIFP(io))) && !IN_BYTES) { + if (PL_op->op_type == OP_SYSREAD || PL_op->op_type == OP_RECV) { + Perl_ck_warner(aTHX_ packWARN(WARN_DEPRECATED), + "%s() is deprecated on :utf8 handles", + OP_DESC(PL_op)); + } buffer = SvPVutf8_force(bufsv, blen); /* UTF-8 may not have been set if they are all low bytes */ SvUTF8_on(bufsv); @@ -1950,6 +1955,9 @@ PP(pp_syswrite) doing_utf8 = DO_UTF8(bufsv); if (PerlIO_isutf8(IoIFP(io))) { + Perl_ck_warner(aTHX_ packWARN(WARN_DEPRECATED), + "%s() is deprecated on :utf8 handles", + OP_DESC(PL_op)); if (!SvUTF8(bufsv)) { /* We don't modify the original scalar. */ tmpbuf = bytes_to_utf8((const U8*) buffer, &blen); diff --git a/t/lib/warnings/pp_sys b/t/lib/warnings/pp_sys index a1e07f8..ea18bac 100644 --- a/t/lib/warnings/pp_sys +++ b/t/lib/warnings/pp_sys @@ -939,3 +939,25 @@ sleep(-1); EXPECT sleep() with negative argument at - line 2. +######## +# NAME sysread() deprecated on :utf8 +use warnings 'deprecated'; +open my $fh, "<", "../harness" or die "# $!"; +my $buf; +sysread $fh, $buf, 10; +binmode $fh, ':utf8'; +sysread $fh, $buf, 10; +EXPECT +sysread() is deprecated on :utf8 handles at - line 6. +######## +# NAME syswrite() deprecated on :utf8 +my $file = "syswwarn.tmp"; +use warnings 'deprecated'; +open my $fh, ">", $file or die "# $!"; +syswrite $fh, 'ABC'; +binmode $fh, ':utf8'; +syswrite $fh, 'ABC'; +close $fh; +unlink $file; +EXPECT +syswrite() is deprecated on :utf8 handles at - line 6. diff --git a/t/op/gmagic.t b/t/op/gmagic.t index bcf1322..94e164e 100644 --- a/t/op/gmagic.t +++ b/t/op/gmagic.t @@ -77,6 +77,7 @@ expected_tie_calls(tied $c, 1, 2, 'chomping a ref'); # Do this again, with a utf8 handle $c = *foo; # 1 write open $h, "<:utf8", $outfile; + no warnings 'deprecated'; sysread $h, $c, 3, 7; # 1 read; 1 write is $c, "*main::bar", 'what sysread wrote'; # 1 read expected_tie_calls(tied $c, 2, 2, 'calling sysread with tied buf'); diff --git a/t/uni/overload.t b/t/uni/overload.t index 66cd5b8..ff89b08 100644 --- a/t/uni/overload.t +++ b/t/uni/overload.t @@ -169,6 +169,7 @@ foreach my $operator ('print', 'syswrite', 'syswrite len', 'syswrite off', my $trail = $operator =~ /\blen\b/ ? "!" : ""; my $u = UTF8Toggle->new("$pad$E_acute\n$trail"); my $l = UTF8Toggle->new("$pad$e_acute\n$trail", 1); + no warnings 'deprecated'; if ($operator eq 'print') { no warnings 'utf8'; print $fh $u; -- 1.7.10.4
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 447b
On Sun Aug 09 23:23:05 2015, tonyc wrote: Show quoted text
> On Thu Aug 06 16:12:45 2015, perl.p5p@rjbs.manxome.org wrote:
> > But if we're going to make it croak in 5.28, time to make it warn > > now.
> > Patch attached. > > As chansen mentioned in #p5p, send() and recv() have the same issue, > so the patch > also deprecates them on :utf8 handles.
Applied as fb10a8a78bba7573de4629b739bfe81cd42e78c9. Leaving open, as I expect *something* will break. Tony
Date: Mon, 17 Aug 2015 10:54:50 +0200
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
CC: Perl5 Porters <perl5-porters [...] perl.org>
To: perlbug <perlbug-followup [...] perl.org>
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 712b
On Mon, Aug 17, 2015 at 8:40 AM, Tony Cook via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Sun Aug 09 23:23:05 2015, tonyc wrote:
> On Thu Aug 06 16:12:45 2015, perl.p5p@rjbs.manxome.org wrote:
> > But if we're going to make it croak in 5.28, time to make it warn
> > now.
>
> Patch attached.
>
> As chansen mentioned in #p5p, send() and recv() have the same issue,
> so the patch
> also deprecates them on :utf8 handles.

Applied as fb10a8a78bba7573de4629b739bfe81cd42e78c9.

Leaving open, as I expect *something* will break.

Tony

Among commonly used modules, I don't expect File::Slurp to like this, then again the beast is so broken by design that that may be a feature.

Leon

CC: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Date: Wed, 19 Aug 2015 09:43:14 +0100
To: Tony Cook via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
Download (untitled) / with headers
text/plain 2.9k
On Sun, Aug 16, 2015 at 11:40:06PM -0700, Tony Cook via RT wrote: Show quoted text
> On Sun Aug 09 23:23:05 2015, tonyc wrote:
> > On Thu Aug 06 16:12:45 2015, perl.p5p@rjbs.manxome.org wrote:
> > > But if we're going to make it croak in 5.28, time to make it warn > > > now.
> > > > Patch attached. > > > > As chansen mentioned in #p5p, send() and recv() have the same issue, > > so the patch > > also deprecates them on :utf8 handles.
> > Applied as fb10a8a78bba7573de4629b739bfe81cd42e78c9.
It's causing lib/warnings.t to fail in smokes that do PERL_UNICODE="" e.g. [davem@robin t]$ PERL_UNICODE="" ./perl harness ../lib/warnings.t ../lib/warnings.t .. 617/876 PROG: # pp_sys.c [pp_sysread] use warnings 'io' ; if ($^O eq 'dos') { print <<EOM ; SKIPPED # skipped on dos EOM exit ; } my $file = "./xcv" ; open(F, ">$file") ; my $a = sysread(F, $a,10) ; no warnings 'io' ; my $a = sysread(F, $a,10) ; close F ; use warnings 'io' ; sysread(F, $a, 10); read(F, $a, 10); sysread(NONEXISTENT, $a, 10); read(NONEXISTENT, $a, 10); unlink $file ; EXPECTED: Filehandle F opened only for output at - line 12. sysread() on closed filehandle F at - line 17. read() on closed filehandle F at - line 18. sysread() on unopened filehandle NONEXISTENT at - line 19. read() on unopened filehandle NONEXISTENT at - line 20. GOT: sysread() is deprecated on :utf8 handles at - line 12. Filehandle F opened only for output at - line 12. sysread() is deprecated on :utf8 handles at - line 14. sysread() on closed filehandle F at - line 17. read() on closed filehandle F at - line 18. sysread() on unopened filehandle NONEXISTENT at - line 19. read() on unopened filehandle NONEXISTENT at - line 20. # Failed test 673 - at lib/warnings/pp_sys line 625 PROG: use warnings 'deprecated'; open my $fh, "<", "../harness" or die "# $!"; my $buf; sysread $fh, $buf, 10; binmode $fh, ':utf8'; sysread $fh, $buf, 10; EXPECTED: sysread() is deprecated on :utf8 handles at - line 6. GOT: sysread() is deprecated on :utf8 handles at - line 4. sysread() is deprecated on :utf8 handles at - line 6. # Failed test 689 - sysread() deprecated on :utf8 at lib/warnings/pp_sys line 943 PROG: my $file = "syswwarn.tmp"; use warnings 'deprecated'; open my $fh, ">", $file or die "# $!"; syswrite $fh, 'ABC'; binmode $fh, ':utf8'; syswrite $fh, 'ABC'; close $fh; unlink $file; EXPECTED: syswrite() is deprecated on :utf8 handles at - line 6. GOT: syswrite() is deprecated on :utf8 handles at - line 4. syswrite() is deprecated on :utf8 handles at - line 6. # Failed test 690 - syswrite() deprecated on :utf8 at lib/warnings/pp_sys line 953 ../lib/warnings.t .. Failed 3/876 subtests (less 2 skipped subtests: 871 okay) Test Summary Report ------------------- ../lib/warnings.t (Wstat: 0 Tests: 876 Failed: 3) Failed tests: 673, 689-690 Files=1, Tests=876, 11 wallclock secs ( 0.31 usr 0.02 sys + 8.64 cusr 2.42 csys = 11.39 CPU) Result: FAIL [davem@robin t]$ -- Indomitable in retreat, invincible in advance, insufferable in victory -- Churchill on Montgomery
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 661b
On Wed Aug 19 01:43:50 2015, davem wrote: Show quoted text
> On Sun, Aug 16, 2015 at 11:40:06PM -0700, Tony Cook via RT wrote:
> > On Sun Aug 09 23:23:05 2015, tonyc wrote:
> > > On Thu Aug 06 16:12:45 2015, perl.p5p@rjbs.manxome.org wrote:
> > > > But if we're going to make it croak in 5.28, time to make it warn > > > > now.
> > > > > > Patch attached. > > > > > > As chansen mentioned in #p5p, send() and recv() have the same > > > issue, > > > so the patch > > > also deprecates them on :utf8 handles.
> > > > Applied as fb10a8a78bba7573de4629b739bfe81cd42e78c9.
> > It's causing lib/warnings.t to fail in smokes that do PERL_UNICODE=""
Thanks, fixed in 60e67244ff. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 599b
On Sun, 16 Aug 2015 23:40:06 -0700, tonyc wrote: Show quoted text
> On Sun Aug 09 23:23:05 2015, tonyc wrote:
> > On Thu Aug 06 16:12:45 2015, perl.p5p@rjbs.manxome.org wrote:
> > > But if we're going to make it croak in 5.28, time to make it warn > > > now.
> > > > Patch attached. > > > > As chansen mentioned in #p5p, send() and recv() have the same issue, > > so the patch > > also deprecates them on :utf8 handles.
> > Applied as fb10a8a78bba7573de4629b739bfe81cd42e78c9. > > Leaving open, as I expect *something* will break.
Added to the 5.30 blockers ticket, since these should start croaking then. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 193b
On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote: Show quoted text
> Added to the 5.30 blockers ticket, since these should start croaking then.
Patch to make them fatal attached, I'll apply this Soon(tm). Tony
Subject: 0001-perl-133170-fatalize-sysread-syswrite-recv-send-on-u.patch

Message body is not shown because it is too large.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 289b
On Mon, 24 Sep 2018 18:26:12 -0700, tonyc wrote: Show quoted text
> On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote:
> > Added to the 5.30 blockers ticket, since these should start croaking then.
> > Patch to make them fatal attached, I'll apply this Soon(tm).
Ignore that, I misread a test report. Tony
From: Leon Timmermans <fawaka [...] gmail.com>
CC: Perl5 Porters <perl5-porters [...] perl.org>
To: perlbug <perlbug-followup [...] perl.org>
Date: Tue, 25 Sep 2018 19:27:09 +0200
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
Download (untitled) / with headers
text/plain 620b
On Tue, Sep 25, 2018 at 3:26 AM Tony Cook via RT <perlbug-followup@perl.org> wrote: Show quoted text
> > On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote:
> > Added to the 5.30 blockers ticket, since these should start croaking then.
> > Patch to make them fatal attached, I'll apply this Soon(tm).
I should point out that File::Slurp still hasn't been fix to not use this misfeature (despite a ticket being open about it for half a decade). File::Slurp currently has 636 direct dependents (and an unknown bug likely high number of indirect dependencies). This change will break CPAN given the current state of File::Slurp. Leon Leon
RT-Send-CC: perl5-porters [...] perl.org
On Mon, 24 Sep 2018 21:50:54 -0700, tonyc wrote: Show quoted text
> On Mon, 24 Sep 2018 18:26:12 -0700, tonyc wrote:
> > On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote:
> > > Added to the 5.30 blockers ticket, since these should start > > > croaking then.
> > > > Patch to make them fatal attached, I'll apply this Soon(tm).
> > Ignore that, I misread a test report.
The attached should be better. sigtrap.pm in particular was a problem. It has a signal handler (from the 5.000 commit) that appears to try to behave safely when called as an unsafe signal handler, and uses syswrite() to write to STDERR. This is broken if STDERR happens to have any non-trivial layers on it. I've modified the sigtrap code to try to use syswrite() in an eval block initially and fallback to print, and then check PerlIO::get_layers() for any non-default layers to decide whether to continue to use syswrite() if it happened to succeed the first time. This means if the layers don't produce something vaguely like ASCII, the initial output will be garbled, but that was true of the original code. Tony
Subject: 0001-perl-133170-fatalize-sysread-syswrite-recv-send-on-u.patch

Message body is not shown because it is too large.

Subject: 0002-perl-133170-adapt-sigtrap-for-layers-on-STDERR.patch
From 63745c81b519a507576574c9897eccc5b1ab9291 Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Wed, 26 Sep 2018 11:12:34 +1000 Subject: (perl #133170) adapt sigtrap for layers on STDERR. sigtrap defines a signal handler apparently intended to be called under unsafe signals, since a) the code was written before safe signals were implemented and b) it uses syswrite() for output and avoid creating new SVs where it can. Unfortunately syswrite() doesn't handle PerlIO layers, *and* with syswrite() being disallowed for :utf8 handlers, throws an exception. This causes the sigtrap tests to fail if PERL_UNICODE is set and the current locale is a UTF-8 locale. I want to avoid allocating new SVs until the point where the code originally did so, so the code now attempts a syswrite() under eval, falling back to print, and then at the point where the original code started allocating SVs uses PerlIO::get_layers() to check if any layers might make a difference to the output. --- lib/sigtrap.pm | 56 +++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 47 insertions(+), 9 deletions(-) diff --git a/lib/sigtrap.pm b/lib/sigtrap.pm index 7d801461d4..11d670942b 100644 --- a/lib/sigtrap.pm +++ b/lib/sigtrap.pm @@ -8,7 +8,7 @@ sigtrap - Perl pragma to enable simple signal handling use Carp; -$VERSION = 1.08; +$VERSION = 1.09; $Verbose ||= 0; sub import { @@ -81,16 +81,49 @@ sub handler_die { sub handler_traceback { package DB; # To get subroutine args. + my $use_print; $SIG{'ABRT'} = DEFAULT; kill 'ABRT', $$ if $panic++; - syswrite(STDERR, 'Caught a SIG', 12); - syswrite(STDERR, $_[0], length($_[0])); - syswrite(STDERR, ' at ', 4); + + # This function might be called as an unsafe signal handler, so it + # tries to delay any memory allocations as long as possible. + # + # Unfortunately with PerlIO layers, using syswrite() here has always + # been broken. + # + # Calling PerlIO::get_layers() here is tempting, but that does + # allocations, which we're trying to avoid for this early code. + if (eval { syswrite(STDERR, 'Caught a SIG', 12); 1 }) { + syswrite(STDERR, $_[0], length($_[0])); + syswrite(STDERR, ' at ', 4); + } + else { + print STDERR 'Caught a SIG', $_[0], ' at '; + ++$use_print; + } + ($pack,$file,$line) = caller; - syswrite(STDERR, $file, length($file)); - syswrite(STDERR, ' line ', 6); - syswrite(STDERR, $line, length($line)); - syswrite(STDERR, "\n", 1); + unless ($use_print) { + syswrite(STDERR, $file, length($file)); + syswrite(STDERR, ' line ', 6); + syswrite(STDERR, $line, length($line)); + syswrite(STDERR, "\n", 1); + } + else { + print STDERR $file, ' line ', $line, "\n"; + } + + # we've got our basic output done, from now on we can be freer with allocations + # find out whether we have any layers we need to worry about + unless ($use_print) { + my @layers = PerlIO::get_layers(*STDERR); + for my $name (@layers) { + unless ($name =~ /^(unix|perlio)$/) { + ++$use_print; + last; + } + } + } # Now go for broke. for ($i = 1; ($p,$f,$l,$s,$h,$w,$e,$r) = caller($i); $i++) { @@ -116,7 +149,12 @@ sub handler_traceback { } $f = "file '$f'" unless $f eq '-e'; $mess = "$w$s$a called from $f line $l\n"; - syswrite(STDERR, $mess, length($mess)); + if ($use_print) { + print STDERR $mess; + } + else { + syswrite(STDERR, $mess, length($mess)); + } } kill 'ABRT', $$; } -- 2.11.0
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 789b
On Tue, 25 Sep 2018 10:27:33 -0700, LeonT wrote: Show quoted text
> On Tue, Sep 25, 2018 at 3:26 AM Tony Cook via RT > <perlbug-followup@perl.org> wrote:
> > > > On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote:
> > > Added to the 5.30 blockers ticket, since these should start > > > croaking then.
> > > > Patch to make them fatal attached, I'll apply this Soon(tm).
> > I should point out that File::Slurp still hasn't been fix to not use > this misfeature (despite a ticket being open about it for half a > decade). File::Slurp currently has 636 direct dependents (and an > unknown bug likely high number of indirect dependencies). > > This change will break CPAN given the current state of File::Slurp.
Code using File::Slurp with :utf8 is already broken, it just won't be silently broken now. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.2k
On Tue, 25 Sep 2018 18:38:58 -0700, tonyc wrote: Show quoted text
> On Mon, 24 Sep 2018 21:50:54 -0700, tonyc wrote:
> > On Mon, 24 Sep 2018 18:26:12 -0700, tonyc wrote:
> > > On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote:
> > > > Added to the 5.30 blockers ticket, since these should start > > > > croaking then.
> > > > > > Patch to make them fatal attached, I'll apply this Soon(tm).
> > > > Ignore that, I misread a test report.
> > The attached should be better. > > sigtrap.pm in particular was a problem. > > It has a signal handler (from the 5.000 commit) that appears to try to > behave safely when called as an unsafe signal handler, and uses > syswrite() > to write to STDERR. > > This is broken if STDERR happens to have any non-trivial layers on it. > > I've modified the sigtrap code to try to use syswrite() in an eval > block initially and fallback to print, and then check > PerlIO::get_layers() for > any non-default layers to decide whether to continue to use syswrite() > if > it happened to succeed the first time. > > This means if the layers don't produce something vaguely like ASCII, > the > initial output will be garbled, but that was true of the original > code.
Applied as 1ed4b7762a858fb9c71bc209fe868060f3774cb5 and 5c0551aafb45d343b720500fd9560ffedd9607fa. Tony
To: perlbug <perlbug-followup [...] perl.org>
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
Date: Wed, 10 Oct 2018 16:33:02 +0200
CC: Perl5 Porters <perl5-porters [...] perl.org>
From: Leon Timmermans <fawaka [...] gmail.com>
On Wed, Oct 10, 2018 at 3:14 AM Tony Cook via RT <perlbug-followup@perl.org> wrote: Show quoted text
> > On Tue, 25 Sep 2018 10:27:33 -0700, LeonT wrote:
> > On Tue, Sep 25, 2018 at 3:26 AM Tony Cook via RT > > <perlbug-followup@perl.org> wrote:
> > > > > > On Sun, 15 Oct 2017 22:23:32 -0700, tonyc wrote:
> > > > Added to the 5.30 blockers ticket, since these should start > > > > croaking then.
> > > > > > Patch to make them fatal attached, I'll apply this Soon(tm).
> > > > I should point out that File::Slurp still hasn't been fix to not use > > this misfeature (despite a ticket being open about it for half a > > decade). File::Slurp currently has 636 direct dependents (and an > > unknown bug likely high number of indirect dependencies). > > > > This change will break CPAN given the current state of File::Slurp.
> > Code using File::Slurp with :utf8 is already broken, it just won't be > silently broken now. > > Tony
True, but it's now also uninstallable for all File::Slurp users that don't use :utf8 (which is probably the majority). Leon
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.4k
Am Sun, 09 Aug 2015 23:23:05 -0700, tonyc schrieb: Show quoted text
> if ((fp_utf8 = PerlIO_isutf8(IoIFP(io))) && !IN_BYTES) { > + if (PL_op->op_type == OP_SYSREAD || PL_op->op_type == OP_RECV) { > + Perl_ck_warner(aTHX_ packWARN(WARN_DEPRECATED), > + "%s() is deprecated on :utf8 handles", > + OP_DESC(PL_op)); > + }
Could someone please explain me if the following change is valid: Show quoted text
> binmode($DB::OUT, ':utf-8')
to Show quoted text
> binmode($DB::OUT, ':encoding(UTF-8)')
From my understanding it is not because of statements like the following: Show quoted text
> +(W deprecated) The sysread(), recv(), syswrite() and send() operators > +are deprecated on handles that have the C<:utf8> layer, either > +explicitly, or implicitly, eg., with the C<:encoding(UTF-16LE)> layer.
https://rt.perl.org/Public/Ticket/Attachment/1360169/728149/0001-perl-125760-deprecate-sys-read-write-send-recv-on-ut.patch Show quoted text
> +Note that if the filehandle has been marked as C<:utf8>, C<sysread> will > +throw an exception. The C<:encoding(...)> layer implicitly > introduces the C<:utf8> layer.</sysread>
https://rt.perl.org/Public/Ticket/Attachment/1583747/827814/0001-perl-133170-fatalize-sysread-syswrite-recv-send-on-u.patch But the change seems to fix the warnings. Shouldn't that change still result in the ":utf8"-flag being applied and therefore being still wrong? The other attached patches of this bug e.g. regarding tests use ":raw" only, like I understand this bug as well. Thanks for any clarification!
CC: perl5-porters [...] perl.org
From: Tony Cook <tony [...] develop-help.com>
To: Thorsten Schöning via RT <perlbug-followup [...] perl.org>
Date: Mon, 12 Nov 2018 14:36:08 +1100
Subject: Re: [perl #125760] RFC remove strange behaviour of sysread()/syswrite() on UTF-8 streams
Download (untitled) / with headers
text/plain 2.1k
On Sun, Nov 11, 2018 at 05:24:51AM -0800, Thorsten Schöning via RT wrote: Show quoted text
> Am Sun, 09 Aug 2015 23:23:05 -0700, tonyc schrieb:
> > if ((fp_utf8 = PerlIO_isutf8(IoIFP(io))) && !IN_BYTES) { > > + if (PL_op->op_type == OP_SYSREAD || PL_op->op_type == OP_RECV) { > > + Perl_ck_warner(aTHX_ packWARN(WARN_DEPRECATED), > > + "%s() is deprecated on :utf8 handles", > > + OP_DESC(PL_op)); > > + }
> > Could someone please explain me if the following change is valid: >
> > binmode($DB::OUT, ':utf-8')
> to
> > binmode($DB::OUT, ':encoding(UTF-8)')
No, that won't prevent the warning. For example, with a default 5.28.0 build: $ ./perl -Ilib -e 'open my $f, ">", "foo"; binmode $f; binmode $f, ":encoding(UTF8)"; syswrite($f, "foo")' syswrite() is deprecated on :utf8 handles. This will be a fatal error in Perl 5.30 at -e line 1. In blead this throws an error instead. Show quoted text
>
> >From my understanding it is not because of statements like the following:
>
> > +(W deprecated) The sysread(), recv(), syswrite() and send() operators > > +are deprecated on handles that have the C<:utf8> layer, either > > +explicitly, or implicitly, eg., with the C<:encoding(UTF-16LE)> layer.
> > https://rt.perl.org/Public/Ticket/Attachment/1360169/728149/0001-perl-125760-deprecate-sys-read-write-send-recv-on-ut.patch >
> > +Note that if the filehandle has been marked as C<:utf8>, C<sysread> will > > +throw an exception. The C<:encoding(...)> layer implicitly > > introduces the C<:utf8> layer.</sysread>
> > https://rt.perl.org/Public/Ticket/Attachment/1583747/827814/0001-perl-133170-fatalize-sysread-syswrite-recv-send-on-u.patch > > But the change seems to fix the warnings. Shouldn't that change still result in the ":utf8"-flag being applied and therefore being still wrong? The other attached patches of this bug e.g. regarding tests use ":raw" only, like I understand this bug as well.
I'm not sure what you're saying here, if you could provide simple but complete code that doesn't behave how you expect, the result you expect and the result you got, we can decide if we have a code bug or perhaps a documentation bug. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.3k
On Sun, 11 Nov 2018 19:36:38 -0800, tonyc wrote: Show quoted text
> On Sun, Nov 11, 2018 at 05:24:51AM -0800, Thorsten Schöning via RT > wrote:
> > Am Sun, 09 Aug 2015 23:23:05 -0700, tonyc schrieb:
> > > if ((fp_utf8 = PerlIO_isutf8(IoIFP(io))) && !IN_BYTES) { > > > + if (PL_op->op_type == OP_SYSREAD || PL_op->op_type == > > > OP_RECV) { > > > + Perl_ck_warner(aTHX_ packWARN(WARN_DEPRECATED), > > > + "%s() is deprecated on :utf8 handles", > > > + OP_DESC(PL_op)); > > > + }
> > > > Could someone please explain me if the following change is valid: > >
> > > binmode($DB::OUT, ':utf-8')
> > to
> > > binmode($DB::OUT, ':encoding(UTF-8)')
> > No, that won't prevent the warning. > > For example, with a default 5.28.0 build: > > $ ./perl -Ilib -e 'open my $f, ">", "foo"; binmode $f; binmode $f, > ":encoding(UTF8)"; syswrite($f, "foo")' > syswrite() is deprecated on :utf8 handles. This will be a fatal error > in Perl 5.30 at -e line 1. > > In blead this throws an error instead. >
> >
> > > From my understanding it is not because of statements like the > > > following:
> >
> > > +(W deprecated) The sysread(), recv(), syswrite() and send() > > > operators > > > +are deprecated on handles that have the C<:utf8> layer, either > > > +explicitly, or implicitly, eg., with the C<:encoding(UTF-16LE)> > > > layer.
> > > > https://rt.perl.org/Public/Ticket/Attachment/1360169/728149/0001- > > perl-125760-deprecate-sys-read-write-send-recv-on-ut.patch > >
> > > +Note that if the filehandle has been marked as C<:utf8>, > > > C<sysread> will > > > +throw an exception. The C<:encoding(...)> layer implicitly > > > introduces the C<:utf8> layer.</sysread>
> > > > https://rt.perl.org/Public/Ticket/Attachment/1583747/827814/0001- > > perl-133170-fatalize-sysread-syswrite-recv-send-on-u.patch > > > > But the change seems to fix the warnings. Shouldn't that change still > > result in the ":utf8"-flag being applied and therefore being still > > wrong? The other attached patches of this bug e.g. regarding tests > > use ":raw" only, like I understand this bug as well.
> > I'm not sure what you're saying here, if you could provide simple but > complete code that doesn't behave how you expect, the result you > expect and the result you got, we can decide if we have a code bug or > perhaps a documentation bug. > > Tony
Is this ticket closable? -- Karl Williamson
I've moved this ticket from 5.30.0 blocker to 5.32.0 blocker
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 260b
On Tue, 23 Apr 2019 14:57:24 GMT, davem wrote: Show quoted text
> I've moved this ticket from 5.30.0 blocker to 5.32.0 blocker
Dave, What issues still remain such that this ticket remains open and is a 5.32 blocker? Thank you very much. -- James E Keenan (jkeenan@cpan.org)
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 375b
On Fri, 31 May 2019 05:18:40 -0700, jkeenan wrote: Show quoted text
> On Tue, 23 Apr 2019 14:57:24 GMT, davem wrote:
> > I've moved this ticket from 5.30.0 blocker to 5.32.0 blocker
> > Dave, > > What issues still remain such that this ticket remains open and is a > 5.32 blocker?
None. I only left it open to track related issues, which were all resolved to my knowledge. Closing. Tony


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org