Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong aliasing of $_ #17119

Closed
p5pRT opened this issue Aug 4, 2019 · 12 comments
Closed

Wrong aliasing of $_ #17119

p5pRT opened this issue Aug 4, 2019 · 12 comments

Comments

@p5pRT
Copy link

p5pRT commented Aug 4, 2019

Migrated from rt.perl.org#134337 (status was 'rejected')

Searchable as RT134337$

@p5pRT
Copy link
Author

p5pRT commented Aug 4, 2019

From @kroki

This is a bug report for perl from tomash.brechko@​gmail.com,
generated with the help of perlbug 1.41 running under perl 5.28.2.

Not tested with v5.30, sorry. With v5.28.2 the code below dies with
"Modification of a read-only value attempted at -e line 9."​:

  $ echo 1 | perl -e '
  use strict;
  use warnings;

  f($_) for "a";

  sub f {
  my ($v) = @​_;
  while (<STDIN>) { # line 9.
  print "$v $_";
  }
  }
  '

However the problem is more severe​: in a real-world program (that I
can't provide, sorry again) in the code pattern

  $/ = "\n";
  ...
  foreach (@​a) {
  warn "$_\n"; # Expect values of @​a, but also get values read from
$fh.
  f($_);
  }
  ...
  sub f {
  my ($v) = @​_;
  while (<$fh>) {
  ...
  $/ = "END\n";
  <$fh>;
  $/ = "\n";
  last;
  }
  }

warn() along with the values from @​a also prints values read from $fh
($v in f() also gets values read from $fh, and the whole thing explodes).

Still I believe both cases are instances of the same problem​:
something is wrong with how $_ aliases its values.


Flags​:
  category=core
  severity=high


Site configuration information for perl 5.28.2​:

Configured by Red Hat, Inc. at Tue Apr 23 08​:23​:13 UTC 2019.

Summary of my perl5 (revision 5 version 28 subversion 2) configuration​:

  Platform​:
  osname=linux
  osvers=5.0.6-200.fc29.x86_64
  archname=x86_64-linux-thread-multi
  uname='linux buildvm-07.phx2.fedoraproject.org 5.0.6-200.fc29.x86_64 #1
smp wed apr 3 15​:09​:51 utc 2019 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-fexceptions -fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-Dldflags=-Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld
-Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dlddlflags=-shared
-Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dshrpdir=/usr/lib64
-DDEBUGGING=-g -Dversion=5.28.2 -Dmyhostname=localhost
-Dperladmin=root@​localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr
-Dvendorprefix=/usr -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5
-Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl
-Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl
-Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64
/usr/lib64 -Duseshrplib -Dusethreads -Duseithreads
-Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db
-Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio
-Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less
-isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto
-Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto
-Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize
-Duse64bitint'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=define
  usemultiplicity=define
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='gcc'
  ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-fexceptions -fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64'
  optimize=' -g'
  cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
-fexceptions -fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-fwrapv -fno-strict-aliasing -I/usr/local/include'
  ccversion=''
  gccversion='9.0.1 20190312 (Red Hat 9.0.1-0.10)'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='gcc'
  ldflags ='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong
-L/usr/local/lib'
  libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib
/lib/../lib64 /usr/lib/../lib64 /lib
  libs=-lpthread -lresolv -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
-lgdbm_compat
  perllibs=-lpthread -lresolv -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.29.so
  so=so
  useshrplib=true
  libperl=libperl.so
  gnulibc_version='2.29'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro -Wl,--as-needed
-Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
  cccdlflags='-fPIC'
  lddlflags='-lpthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib
-fstack-protector-strong'

Locally applied patches​:
  Fedora Patch1​: Removes date check, Fedora/RHEL specific
  Fedora Patch2​: support for libdir64
  Fedora Patch3​: use libresolv instead of libbind
  Fedora Patch4​: USE_MM_LD_RUN_PATH
  Fedora Patch5​: Provide MM​::maybe_command independently (bug #1129443)
  Fedora Patch6​: Dont run one io test due to random builder failures
  Fedora Patch8​: Define SONAME for libperl.so
  Fedora Patch9​: Install libperl.so to -Dshrpdir value
  Fedora Patch10​: Document Math​::BigInt​::CalcEmu requires Math​::BigInt
(CPAN RT#85015)
  Fedora Patch11​: Make *DBM_File desctructors thread-safe (RT#61912)
  Fedora Patch12​: Replace EU​::MakeMaker dependency with EU​::MM​::Utils in
IPC​::Cmd (bug #1129443)
  Fedora Patch13​: Fix executing arybase​::_tie_it() in Safe compartement
(RT#131588)
  Fedora Patch14​: Link XS modules to pthread library to fix linking with
-z defs
  Fedora Patch17​: Fix printing a warning about a wide character when
matching a regular expression while ISO-8859-1 locale is in effect
  Fedora Patch18​: Fix invoking a check for wide characters while
ISO-8859-1 locale is in effect
  Fedora Patch20​: Fix build conditions in locale.c
  Fedora Patch21​: Fix a file descriptor leak in in-place edits (RT#133314)
  Fedora Patch23​: Fix a buffer overrun in deprecated S_is_utf8_common()
  Fedora Patch24​: Fix a time race in Time-HiRes/t/itimer.t test
  Fedora Patch26​: Fix Time​::Piece to handle objects in overloaded methods
correctly
  Fedora Patch27​: Fix an assignment to a lexical variable in
multiconcatenation expressions (RT#133441)
  Fedora Patch28​: Fix a spurious warning about uninitialized value in
warn (RT#132683)
  Fedora Patch30​: Pass the correct CFLAGS to dtrace
  Fedora Patch33​: Fix PathTools tests to cope with ESTALE error
(RT#133534)
  Fedora Patch34​: Fix an undefined behaviour in S_hv_delete_common()
  Fedora Patch38​: Fix compiling regular expressions that contain both
compile- and run-time compiled code blocks (RT#133687)
  Fedora Patch39​: Adjust tests to gdbm-1.15 (RT#133295)
  Fedora Patch44​: Fix reporting a line number for non-terminated
prototypes (RT#133524)
  Fedora Patch45​: Fix first eof() return value (RT#133721)
  Fedora Patch49​: Prevent long jumps from clobbering local variables
(RT#133575)
  Fedora Patch50​: Fix a mismatch with a case-insesitive regular
expression on a text with ligatures (RT#133756)
  Fedora Patch53​: Fix setting magic when changing $^R (RT#133782)
  Fedora Patch54​: Fix a race when loading XS modules
  Fedora Patch56​: Fix a leak when compiling a typed hash dereference
  Fedora Patch58​: Fix a buffer overread when handling a scope error in
qr/\(?{/ (RT#133879)
  Fedora Patch59​: Fix a buffer overread when parsing a regular expression
with an unknown character name (RT#133880)
  Fedora Patch60​: Fix mbstate_t initialization in POSIX​::mblen (RT#133928)
  Fedora Patch61​: Fix a memory leak when cloning a regular expression
  Fedora Patch62​: Fix a memory leak when spawning threads in a BEGIN phase
  Fedora Patch63​: Fix a memory leak when assigning a regular expression
to a non-copy-on-write string
  Fedora Patch64​: Fix a memory leak when assignig to a localized
${^WARNING_BITS}
  Fedora Patch65​: Fix a memory leak when parsing misindented
here-documents
  Fedora Patch66​: Fix a memory leak in package name lookup (RT#133977)
  Fedora Patch68​: Fix a memory leak when deletion in a tied hash dies
  Fedora Patch69​: Fix a crash when matching case insensitively (RT#133892)
  Fedora Patch70​: Fix a memory leak when warning about malformed UTF-8
string
  Fedora Patch200​: Link XS modules to libperl.so with EU​::CBuilder on
Linux
  Fedora Patch201​: Link XS modules to libperl.so with EU​::MM on Linux


@​INC for perl 5.28.2​:
  /usr/local/lib64/perl5
  /usr/local/share/perl5
  /usr/lib64/perl5/vendor_perl
  /usr/share/perl5/vendor_perl
  /usr/lib64/perl5
  /usr/share/perl5


Environment for perl 5.28.2​:
  HOME=/home/tomash
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)

PATH=/usr/share/Modules/bin​:/usr/local/bin​:/usr/local/sbin​:/usr/bin​:/usr/sbin​:/home/tomash/.local/bin​:/home/tomash/bin​:/home/tomash/NOBACKUP/ANDROID/android-studio/bin​:/home/tomash/.local/bin​:/home/tomash/bin​:/home/tomash/NOBACKUP/ANDROID/android-studio/bin
  PERL_BADLANG (unset)
  SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Aug 5, 2019

From @tonycoz

On Sun, 04 Aug 2019 02​:23​:49 -0700, tomash.brechko@​gmail.com wrote​:

This is a bug report for perl from tomash.brechko@​gmail.com,
generated with the help of perlbug 1.41 running under perl 5.28.2.

Not tested with v5.30, sorry. With v5.28.2 the code below dies with
"Modification of a read-only value attempted at -e line 9."​:

$ echo 1 | perl -e '
use strict;
use warnings;

f($_) for "a";

sub f {
my ($v) = @​_;
while (<STDIN>) { # line 9.
print "$v $_";
}
}
'

$_ is global, when you alias $_ it's aliased for all code.

When you alias $_ to a read-only value, and try to modify $_ the "Modification of a read-only value attempted at -e line 9." error is expected, not unusual.

Adding "local $_;" before the while loop in f() fixes this.

However the problem is more severe​: in a real-world program (that I
can't provide, sorry again) in the code pattern

$/ = "\n";
...
foreach (@​a) {
warn "$_\n"; # Expect values of @​a, but also get values read from
$fh.
f($_);
}
...
sub f {
my ($v) = @​_;
while (<$fh>) {
...
$/ = "END\n";
<$fh>;
$/ = "\n";
last;
}
}

warn() along with the values from @​a also prints values read from $fh
($v in f() also gets values read from $fh, and the whole thing
explodes).

Still I believe both cases are instances of the same problem​:
something is wrong with how $_ aliases its values.

Similarly here the while (<$fh>) in f() is modifying the elements of @​a in the caller, again local fixes it.

$_ can be convenient, but it's like any other global, it can lead to unexpected action at a distance.

For complex code I tend to only use $_ with the aliasing done with operators like grep and map, and rarely with the statement modifier foreach.

This isn't a bug.

Tony

@p5pRT
Copy link
Author

p5pRT commented Aug 5, 2019

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Aug 5, 2019

From @kroki

On Sun, 04 Aug 2019 17​:06​:52 -0700, tonyc wrote​:

This isn't a bug.

Ah, I see you point, thanks! Somehow I though that while() should cancel old aliasing, not merely assign through $_, but that was my misunderstanding. Then indeed, postfix for/foreach are quite dangerous.

It seems I can't close this myself though...

@p5pRT
Copy link
Author

p5pRT commented Aug 5, 2019

From @tonycoz

On Sun, 04 Aug 2019 23​:05​:07 -0700, tomash.brechko@​gmail.com wrote​:

On Sun, 04 Aug 2019 17​:06​:52 -0700, tonyc wrote​:

This isn't a bug.

Ah, I see you point, thanks! Somehow I though that while() should
cancel old aliasing, not merely assign through $_, but that was my
misunderstanding. Then indeed, postfix for/foreach are quite
dangerous.

It seems I can't close this myself though...

Closed.

Tony

@p5pRT
Copy link
Author

p5pRT commented Aug 5, 2019

@tonycoz - Status changed from 'open' to 'rejected'

@p5pRT p5pRT closed this as completed Aug 5, 2019
@p5pRT
Copy link
Author

p5pRT commented Aug 7, 2019

From Eirik-Berg.Hanssen@allverden.no

On Wed, Aug 7, 2019 at 5​:59 AM Tomash Brechko via RT <
perlbug-followup@​perl.org> wrote​:

On Sun, 04 Aug 2019 17​:06​:52 -0700, tonyc wrote​:

This isn't a bug.

Ah, I see you point, thanks! Somehow I though that while() should cancel
old aliasing, not merely assign through $_, but that was my
misunderstanding. Then indeed, postfix for/foreach are quite dangerous.

  If I may, that seems to be the wrong lesson. There are other constructs
that alias $_, notably map and grep.

  The common factor that's dangerous is the C<< while (<...>) { ... } >>
idiom. If used inside a subroutine, it should always be guarded with a
C<< local $_ >> to render it (relatively) harmless.

Eirik

@p5pRT
Copy link
Author

p5pRT commented Aug 7, 2019

From @Grinnz

On Wed, Aug 7, 2019 at 11​:46 AM Eirik Berg Hanssen <
Eirik-Berg.Hanssen@​allverden.no> wrote​:

On Wed, Aug 7, 2019 at 5​:59 AM Tomash Brechko via RT <
perlbug-followup@​perl.org> wrote​:

On Sun, 04 Aug 2019 17​:06​:52 -0700, tonyc wrote​:

This isn't a bug.

Ah, I see you point, thanks! Somehow I though that while() should cancel
old aliasing, not merely assign through $_, but that was my
misunderstanding. Then indeed, postfix for/foreach are quite dangerous.

If I may, that seems to be the wrong lesson. There are other constructs
that alias $_, notably map and grep.

The common factor that's dangerous is the C<< while (<...>) { ... } >>
idiom. If used inside a subroutine, it should always be guarded with a
C<< local $_ >> to render it (relatively) harmless.

Harmless to code that calls it, but code it calls can still be dangerous as
it's still a globally aliased variable within the scope. It is easiest to
avoid it altogether for while-readline by assigning to a lexical.

-Dan

@p5pRT
Copy link
Author

p5pRT commented Aug 7, 2019

From Eirik-Berg.Hanssen@allverden.no

On Wed, Aug 7, 2019 at 5​:52 PM Dan Book <grinnz@​gmail.com> wrote​:

On Wed, Aug 7, 2019 at 11​:46 AM Eirik Berg Hanssen <
Eirik-Berg.Hanssen@​allverden.no> wrote​:

On Wed, Aug 7, 2019 at 5​:59 AM Tomash Brechko via RT <
perlbug-followup@​perl.org> wrote​:

On Sun, 04 Aug 2019 17​:06​:52 -0700, tonyc wrote​:

This isn't a bug.

Ah, I see you point, thanks! Somehow I though that while() should
cancel old aliasing, not merely assign through $_, but that was my
misunderstanding. Then indeed, postfix for/foreach are quite dangerous.

If I may, that seems to be the wrong lesson. There are other
constructs that alias $_, notably map and grep.

The common factor that's dangerous is the C<< while (<...>) { ... } >>
idiom. If used inside a subroutine, it should always be guarded with a
C<< local $_ >> to render it (relatively) harmless.

Harmless to code that calls it, but code it calls can still be dangerous
as it's still a globally aliased variable within the scope.

  Globals are always trouble, hence the "(relatively)". This particular
global can't be entirely avoided though, as it's used all over the place.

It is easiest to avoid it altogether for while-readline by assigning to a
lexical.

  But sure, I can agree C<< while (my $l = <...>) { ... } >> (or similar)
is easier than C<< local $_; while (<...>) { ... } >>.

  So, fair enough. When next I encounter this, I'll recommend the lexical
instead of the localization. :)

Eirik

@p5pRT
Copy link
Author

p5pRT commented Aug 8, 2019

From @kroki

ср, 7 авг. 2019 г. в 19​:18, Eirik Berg Hanssen via RT <
perlbug-followup@​perl.org>​:

But sure, I can agree C<< while (my $l = <...>) { ... } >> (or similar)
is easier than C<< local $_; while (<...>) { ... } >>.

For good or bad, we don't always control every function we call, so any
instance of

  f($_) for @​a;

is a potential risk unless you ensured that f() is safe.

The following works​:

  foreach ("a", "b") {
  my $v = $_;
  foreach (1, 2) {
  print "$v $_\n";
  }
  }

Likewise, you can safely nest map/grep/whatever, so I still think it's fair
to expect
that you could nest while(<FH>) as well. I understand that foreach()
aliases while
while() doesn't, but that's a "because it works so" argument, $_ being
global has
nothing to do with this​: while(<FH>) is a special case anyways, so it
_could_ allow
safe nesting similar to foreach(). Yet it doesn't.

I appreciate your advise, but in the end it boils down to "the feature is
there, but
don't use it and hope the code you call doesn't use it either" :). I will
:D

@p5pRT
Copy link
Author

p5pRT commented Aug 8, 2019

From @Grinnz

On Thu, Aug 8, 2019 at 12​:17 AM Tomash Brechko <tomash.brechko@​gmail.com>
wrote​:

ср, 7 авг. 2019 г. в 19​:18, Eirik Berg Hanssen via RT <
perlbug-followup@​perl.org>​:

But sure, I can agree C<< while (my $l = <...>) { ... } >> (or similar)
is easier than C<< local $_; while (<...>) { ... } >>.

For good or bad, we don't always control every function we call, so any
instance of

f($_) for @​a;

is a potential risk unless you ensured that f() is safe.

Just for the sake of driving the point home, "f() for @​a" is the potential
risk, no need to appear to use $_. That's the danger of the global alias.

-Dan

@p5pRT
Copy link
Author

p5pRT commented Aug 8, 2019

From @kroki

On Wed, 07 Aug 2019 21​:29​:03 -0700, grinnz@​gmail.com wrote​:

Just for the sake of driving the point home, "f() for @​a" is the potential
risk, no need to appear to use $_. That's the danger of the global alias.

I see what you mean, yet I'd like to disagree. $_ is not just like any other global var, it is special and is meant to be used implicitly. In previous posts I provided some code for the sake of example, but in real life it often looks like

  f(split /.../) for @​a;
  @​b = grep { /.../ } @​c;

i.e. $_ is assumed but never mentioned. Even when we do mention it, like

  @​a = map { ">$_<" } @​b;

then it is an rvalue, and nested foreach/map/grep/etc. do localize $_, so no danger here. And if one ever finds him/herself writing $_ = ...; then I agree that it should be replaced with a real variable (and hopefully not global).

Now, when I write

  sub f {
  while (<$fh>) {
  last if /.../;
  next if /.../;
  my @​s = split /.../;
  ...
  <$_ is never used here>
  ...
  }
  }

I do not violate "The Protocol" of using $_ implicitly, this is what while(<>) is for. However, as other pointed out, in order to allow f() to be nested in other loops I _have_ to "patch" such _implicit_ use of $_ with preceding _explicit_ local $_, which circumvents the implicit nature of $_ and shows that while(<>) violates "Do The Right Thing" motto. Other implicit looping construct do not have this flaw because they alias, or perhaps otherwise​: while(<>) wished to alias too, but it doesn't have rvalues to alias to, so it assigns to $_ *without localizing it first as other constructs do*​:

  for ("old") {
  for ("new") {
  print "$_\n";
  }
  print "$_\n";
  }
-- output --
  new
  old <-- old value restored after inner loop.

Thus I judge it is a bug, but as with some other real bugs in their early denial stage it too may be rejected for a while, not argue against that :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant