Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eval in package DB can't see caller's lexicals in certain cases #16230

Open
p5pRT opened this issue Nov 8, 2017 · 4 comments
Open

eval in package DB can't see caller's lexicals in certain cases #16230

p5pRT opened this issue Nov 8, 2017 · 4 comments

Comments

@p5pRT
Copy link

p5pRT commented Nov 8, 2017

Migrated from rt.perl.org#132414 (status was 'open')

Searchable as RT132414$

@p5pRT
Copy link
Author

p5pRT commented Nov 8, 2017

From @jimav

This is a bug report for perl from jim.avera@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.24.1.


I have a debugging library which uses eval in package DB to
access variables in the caller's context (per perldoc -f eval).

However lexicals are sometimes not accessible, and it seems to
depend on "irrelevant" details such as whether some not-yet-executed
code exists following the call in the same block.   I don't know if this
is a regression,
but I noticed it shortly after upgrading to perl 5.24.1

In the example below, if there is any statement following
the call into package DB, then it works as expected, i.e., the
lexicals $key and $value are visible.  But as shown the
code dies with an eval error.

Can someone shed light on this?
Is there anything my code inside the DB function can do to avoid this
problem?

#!/usr/bin/perl
use strict; use warnings;

package DB;

sub my_interpolate($) {
  my($s) = @​_;
  my $result = eval "$s"; die "eval '$s' Failed​: $@​" if $@​;
  print "eval '$s' => $result\n";
}

package main;

my %hash = (A => 100, B => 200);

while (my ($key, $value) = each %hash) {
  # **DIES HERE** with
  #    eval '$key' Failed​: Global symbol "$key" requires explicit
package name...
  DB​::my_interpolate '$key';

  #1;    # un-comment this and the problem vanishes!
}



Flags​:
    category=core
    severity=low


Site configuration information for perl 5.24.1​:

Configured by Debian Project at Sat Mar 18 17​:00​:39 UTC 2017.

Summary of my perl5 (revision 5 version 24 subversion 1) configuration​:

  Platform​:
    osname=linux, osvers=3.16.0, archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc
-Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN
-Wdate-time -D_FORTIFY_SOURCE=2 -g -O2
-fdebug-prefix-map=/build/perl-gaNj0t/perl-5.24.1=.
-fstack-protector-strong -Wformat -Werror=format-security -Dldflags=
-Wl,-Bsymbolic-functions -Wl,-z,relro -Dlddlflags=-shared
-Wl,-Bsymbolic-functions -Wl,-z,relro -Dcccdlflags=-fPIC
-Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.24
-Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.24 -Dvendorprefix=/usr
-Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.24
-Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.24.1
-Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.24.1
-Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3
-Dusesitecustomize -Duse64bitint -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio
-Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs
-Duseshrplib -Dlibperl=libperl.so.5.24.1'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler​:
    cc='x86_64-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE
-DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv
-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='6.3.0 20170316', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678,
doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries​:
    ld='x86_64-linux-gnu-gcc', ldflags =' -fstack-protector-strong
-L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib
/usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.24.so, so=so, useshrplib=true, libperl=libperl.so.5.24
    gnulibc_version='2.24'
  Dynamic Linking​:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib
-fstack-protector-strong'

Locally applied patches​:
    DEBPKG​:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS
default for modules installed from CPAN.
    DEBPKG​:debian/db_file_ver - https://bugs.debian.org/340047 Remove
overly restrictive DB_File version check.
    DEBPKG​:debian/doc_info - Replace generic man(1) instructions with
Debian-specific information.
    DEBPKG​:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak
enc2xs to follow symlinks and ignore missing @​INC directories.
    DEBPKG​:debian/errno_ver - https://bugs.debian.org/343351 Remove
Errno version check due to upgrade problems with long-running processes.
    DEBPKG​:debian/libperl_embed_doc - https://bugs.debian.org/186778
Note that libperl-dev package is required for embedded linking
    DEBPKG​:fixes/respect_umask - Respect umask during installation
    DEBPKG​:debian/writable_site_dirs - Set umask approproately for site
install directories
    DEBPKG​:debian/extutils_set_libperl_path - EU​:MM​: set location of
libperl.a under /usr/lib
    DEBPKG​:debian/no_packlist_perllocal - Don't install .packlist or
perllocal.pod for perl or vendor
    DEBPKG​:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the
binary targets.
    DEBPKG​:debian/instmodsh_doc - Debian policy doesn't install
.packlist files for core or vendor.
    DEBPKG​:debian/ld_run_path - Remove standard libs from LD_RUN_PATH
as per Debian policy.
    DEBPKG​:debian/libnet_config_path - Set location of libnet.cfg to
/etc/perl/Net as /usr may not be writable.
    DEBPKG​:debian/mod_paths - Tweak @​INC ordering for Debian
    DEBPKG​:debian/prune_libs - https://bugs.debian.org/128355 Prune the
list of libraries wanted to what we actually need.
    DEBPKG​:fixes/net_smtp_docs - [rt.cpan.org #36038]
https://bugs.debian.org/100195 Document the Net​::SMTP 'Port' option
    DEBPKG​:debian/perlivp - https://bugs.debian.org/510895 Make perlivp
skip include directories in /usr/local
    DEBPKG​:debian/deprecate-with-apt - https://bugs.debian.org/747628
Point users to Debian packages of deprecated core modules
    DEBPKG​:debian/squelch-locale-warnings -
https://bugs.debian.org/508764 Squelch locale warnings in Debian package
maintainer scripts
    DEBPKG​:debian/skip-upstream-git-tests - Skip tests specific to the
upstream Git repository
    DEBPKG​:debian/patchlevel - https://bugs.debian.org/567489 List
packaged patches for 5.24.1-2ubuntu1 in patchlevel.h
    DEBPKG​:debian/skip-kfreebsd-crash - https://bugs.debian.org/628493
[perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
    DEBPKG​:fixes/document_makemaker_ccflags -
https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that
CCFLAGS should include $Config{ccflags}
    DEBPKG​:debian/find_html2text - https://bugs.debian.org/640479
Configure CPAN​::Distribution with correct name of html2text
    DEBPKG​:debian/perl5db-x-terminal-emulator.patch -
https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than
xterm in perl5db.pl
    DEBPKG​:debian/cpan-missing-site-dirs -
https://bugs.debian.org/688842 Fix CPAN​::FirstTime defaults with
nonexisting site dirs if a parent is writable
    DEBPKG​:fixes/memoize_storable_nstore - [rt.cpan.org #77790]
https://bugs.debian.org/587650 Memoize​::Storable​: respect 'nstore'
option not respected
    DEBPKG​:debian/regen-skip - Skip a regeneration check in unrelated
git repositories
    DEBPKG​:debian/makemaker-pasthru - https://bugs.debian.org/758471
Pass LD settings through to subdirectories
    DEBPKG​:debian/makemaker-manext - https://bugs.debian.org/247370
Make EU​::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG​:debian/devel-ppport-reproducibility -
https://bugs.debian.org/801523 Sort the list of XS code files when
generating RealPPPort.xs
    DEBPKG​:debian/encode-unicode-bom-doc -
https://bugs.debian.org/798727 Document Debian backport of
Encode​::Unicode fix
    DEBPKG​:debian/kfreebsd-softupdates - https://bugs.debian.org/796798
Work around Debian Bug#796798
    DEBPKG​:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a
scoping issue with "no autodie" and the "system" sub
    DEBPKG​:fixes/crosscompile-no-targethost - [23695c0] [perl #127234]
Fix the Configure escape with usecrosscompile but no targethost
    DEBPKG​:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in
Memoize
    DEBPKG​:fixes/ok-pod - Added encoding for pod.
    DEBPKG​:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix
t/op/stat.t failures on hurd
    DEBPKG​:fixes/nntp_docs - https://bugs.debian.org/51962 Net​::NNTP​:
Correct innd/nnrpd confusion in relation to Reader option
    DEBPKG​:fixes/math_complex_doc_great_circle -
https://bugs.debian.org/697567 [rt.cpan.org #114104] Math​::Trig​: clarify
definition of great_circle_midpoint
    DEBPKG​:fixes/math_complex_doc_see_also -
https://bugs.debian.org/697568 [rt.cpan.org #114105] Math​::Trig​: add
missing SEE ALSO
    DEBPKG​:fixes/math_complex_doc_angle_units -
https://bugs.debian.org/731505 [rt.cpan.org #114106] Math​::Trig​:
document angle units
    DEBPKG​:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN​:
Add link to main CPAN web site
    DEBPKG​:fixes/time_piece_doc - https://bugs.debian.org/817925
Time​::Piece​: Improve documentation for add_months and add_years
    DEBPKG​:fixes/perlbug-refactor - https://bugs.debian.org/822463
[perl #128020] perlbug​: Refactor duplicated file reading code
    DEBPKG​:fixes/perlbug-linewrap - https://bugs.debian.org/822463
[perl #128020] perlbug​: wrap overly long lines
    DEBPKG​:fixes/hurd_sigaction - https://bugs.debian.org/825016
[d54f4ed] ext/POSIX/t/sigaction.t​: Skip uid and pid tests on GNU/Hurd
    DEBPKG​:fixes/hurd_hints - [4694301] https://bugs.debian.org/825020
[perl #128279] Modify hints for Hurd per Debian ticket 825020.
    DEBPKG​:fixes/extutils-parsexs-reproducibility - [perl #128517]
https://bugs.debian.org/829296 Make the output of ExtUtils​::ParseXS
reproducible
    DEBPKG​:debian/CVE-2016-1238/sitecustomize-in-etc - Look for
sitecustomize.pl in /etc/perl rather than sitelib on Debian systems
    DEBPKG​:debian/CVE-2016-1238/test-suite-without-dot - [perl #127810]
Patch unit tests to explicitly insert "." into @​INC when needed.
    DEBPKG​:debian/CVE-2016-1238/eumm-without-dot - [perl #127810] Add
PERL_USE_UNSAFE_INC support to EU​::MM for fortify_inc support.
    DEBPKG​:debian/CVE-2016-1238/cpan-without-dot - [perl #127810] Set
PERL_USE_UNSAFE_INC for cpan usage
    DEBPKG​:debian/document_inc_removal - Document in perlvar that we
remove '.' from @​INC by default
    DEBPKG​:fixes/extutils_makemaker_reproducible -
https​://bugs.debian.org/835815 https://bugs.debian.org/834190 Make
perllocal.pod files reproducible
    DEBPKG​:debian/CVE-2016-1238/remove-inc-test - Remove test for '.'
in @​INC as it might not be
    DEBPKG​:debian/customized - Update customized.dat for files patched
in Debian
    DEBPKG​:fixes/file_path_hurd_errno - File-Path​: Fix test failure in
Hurd due to hard-coded ENOENT
    DEBPKG​:debian/hppa_op_optimize_workaround -
https://bugs.debian.org/838613 Temporarily lower the optimization of
op.c on hppa due to gcc-6 problems
    DEBPKG​:fixes/test-builder-warning - https://bugs.debian.org/840968
Silence a 'used only once' warning in Test​::Builder
    DEBPKG​:fixes/longdblinf-randomness - [dd68853] [perl #130133]
https://bugs.debian.org/844752 Configure​: fix garbage filtering with
80-bit long doubles
    DEBPKG​:debian/installman-utf8 - https://bugs.debian.org/840211
Generate man pages with UTF-8 characters
    DEBPKG​:fixes/list_assign_leak - [1050723] [perl #130766]
https://bugs.debian.org/855064 avoid a leak in list assign from/to magic
values
    DEBPKG​:fixes/perlfunc_inc_doc - [a03e9f8]
https://bugs.debian.org/839536 [perl #130832] Documentation fixes for
'.' possibly no longer being in @​INC
DEBPKG​:fixes/Compress-Raw-Zlib-2.071-Adapt-tests-to-zlib-1.2.11.patch -
[PATCH] Adapt tests to zlib 1.2.11
    DEBPKG​:fixes/0001-Adapt-tests-to-zlib-1.2.11.patch - [PATCH] Adapt
tests to zlib 1.2.11


@​INC for perl 5.24.1​:
/home/jima/perl5/lib/perl5/5.24.1/x86_64-linux-gnu-thread-multi
    /home/jima/perl5/lib/perl5/5.24.1
    /home/jima/perl5/lib/perl5/x86_64-linux-gnu-thread-multi
    /home/jima/perl5/lib/perl5
    /home/jima/lib/perl
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
    /usr/local/share/perl/5.24.1
    /usr/lib/x86_64-linux-gnu/perl5/5.24
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.24
    /usr/share/perl/5.24
    /usr/local/lib/site_perl
    /usr/lib/x86_64-linux-gnu/perl-base


Environment for perl 5.24.1​:
    HOME=/home/jima
    LANG=en_US.UTF-8
    LANGUAGE=en_US
    LC_COLLATE=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
PATH=/home/jima/perl5/bin​:/home/jima/bin​:/home/jima/jima_tools/x86_64/bin​:/home/jima/jima_tools/bin​:/usr/bin​:/bin​:/usr/sbin​:/sbin​:/usr/bin/X11​:/usr/local/bin​:/usr/local/sbin​:/usr/games​:/usr/local/games​:/usr/lib/jvm/java-8-oracle/bin​:/usr/lib/jvm/java-8-oracle/db/bin​:/usr/lib/jvm/java-8-oracle/jre/bin​:.
    PERL5LIB=/home/jima/perl5/lib/perl5​:/home/jima/lib/perl
    PERL_BADLANG (unset)
    PERL_LOCAL_LIB_ROOT=/home/jima/perl5
    PERL_MB_OPT=--install_base "/home/jima/perl5"
    PERL_MM_OPT=INSTALL_BASE=/home/jima/perl5
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Nov 10, 2017

From zefram@fysh.org

I don't know if this
is a regression,
but I noticed it shortly after upgrading to perl 5.24.1

It's a regression in 5.13.7. Bisect fingers commit
eae48c8 "refactor and regularise
label/statement grammar". That commit was not intended to change this
eval behaviour.

Is there anything my code inside the DB function can do to avoid this
problem?

No.

-zefram

@p5pRT
Copy link
Author

p5pRT commented Nov 10, 2017

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From zefram@fysh.org

This issue is about nextstate ops being elided. A simple failing case is​:

$ perl5.26.0 -lwe '{ package DB; sub f { print eval "\$x" ? "ok" : "fail"; } } if(++(my $x)) { DB​::f(); }'
fail

If you look at the optree, you'll see that the body of the if() had a
nextstate op at compile time, but it got nulled out​:

6 <;> nextstate(main 5 -e​:1) v​:{ ->7
- <1> null vK/1 ->d
9 <|> and(other->a) vK/1 ->d
8 <1> preinc sK/1 ->9
7 <0> padsv[$x​:6,9] sPRM/LVINTRO ->8
- <@​> scope vK ->-
- <;> ex-nextstate(main 8 -e​:1) v ->a
c <1> entersub vKS ->d
- <1> ex-list K ->c
a <0> pushmark s ->b
- <1> ex-rv2cv sK/1 ->-
b <#> gv[*DB​::f] s ->c

This matters because the eval from DB-land goes looking up the call chain
for the last call from a non-DB location, and uses the lexical state
represented by whatever COP was current (i.e., most recently executed)
at the point of that call. Without the elision, the current COP at
the time of the entersub would be the immediately preceding nextstate
op with sequence number 8, which would mean that $x would be in scope
(sequence range (6, 9]). But with that COP elided, the current COP
is the one preceding the whole if() construct, with sequence number 5,
leaving $x out of scope.

Note that $x is nevertheless in scope for the subroutine call as judged
at compile time of that call. You can pass $x to the sub, for example,
and that'll be the lexical $x. That works because at compile time lexical
scoping is determined on the basis of the COPs as they're generated,
before elision happens. So you can happily pass the lexical $x to a
sub that at runtime doesn't see that $x being in scope at the call site.

Adding a no-op statement such as "1;" inside the block, either before
or after the sub call, can prevent elision of the nextstate op, making
the eval behave correctly.

I said this was a regression in 5.13.7, but that's not entirely correct.
The test case provided by the bug reporter, with a while() loop, starts
failing in 5.13.7, and that can be seen by changing "if(++(my $x))" in the
above one-liner to "while(++(my $x) > $a++)" or similar. But the if()
example fails all the way back to pre-5.6 perls. It's manifesting by
different routes, but there's basically a problem that our compilation
code doesn't appreciate the importance of having the correct COP current
at the time of a sub call.

If the downside of this problem were only evals from the DB package
going wrong, we might accept that as the price of an optimisation,
especially since it only applies when not in debug mode. (With -d on
the command line, the nextstate ops become dbstate ops, and are never
elided, avoiding the problem.) But there are other things controlled
by COPs. Runtime warnings seem to be OK​: changing lexical warning state
prevents COP elision, afaics. But since 5.9.4 COPs have stored a shadow
copy of the compile-time %^H, accessible by called subroutines through
(caller(0))[10]. It's semantically important for that to be correct,
but it's not​:

$ perl5.26.0 -lwe 'sub f { print +((caller(0))[10] || {})->{foo} || "fail"; } if(++(my $x)) { BEGIN { $^H{foo} = "ok"; } f(); }'
fail
$ perl5.10.0 -lwe 'sub f { print +((caller(0))[10] || {})->{foo} || "fail"; } if(++(my $x)) { BEGIN { $^H{foo} = "ok"; } f(); }'
fail

Weirdly, although the while() version of the eval problem didn't occur
prior to 5.13.7, the while() version of the %^H problem *does* occur
prior to 5.13.7. There are multiple nextstate ops in play here, and
the one that the called sub sees pre-5.13.7 has the lexical variable
but not the lexical hint.

So generally there's going to have to be less nulling of nextstate ops,
at least around entersub ops.

-zefram

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants