From zefram@fysh.org

Created by zefram@fysh.org

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; } BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line 1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I described this on p5p last week, mainly concerned with PL_compcv
which is one of the other things that gets localised. I'd found a
handful of situations with this behaviour​: do "FILE", require, and
string eval. I asked whether the behaviour was intentional, but got
no reply. On further investigation and pondering, I've found that all
the situations doing this are based on doeval() in pp_ctl.c, and I've
decided that it's a bug.

I had a go at modifying doeval() and its callers to avoid the problem,
but I've so far failed to produce a fix. The current block arrangement is
deeply embedded in the structure of the code, such that it is not amenable
to making the compiled op tree survive outside the compilation scope.
Fixing it will be a bigger job than I initially anticipated.

Marginally relevant​: my Parse​::Perl module, if used as a direct
replacement for string eval, does not exhibit this problem. It fully
separates the parsing and execution stages, using code that is derived
from pp_entereval() and doeval() but handles the resulting op tree in
a completely different way.

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.10.0:

Configured by Debian Project at Fri Aug 28 22:30:10 UTC 2009.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.26-2-amd64, archname=i486-linux-gnu-thread-multi
    uname='linux puccini 2.6.26-2-amd64 #1 smp fri aug 14 07:12:04 utc 2009 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.3.2', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /usr/lib64
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0
    gnulibc_version='2.7'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.10.0:
    /etc/perl
    /usr/local/lib/perl/5.10.0
    /usr/local/share/perl/5.10.0
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.10
    /usr/share/perl/5.10
    /usr/local/lib/site_perl
    .


Environment for perl 5.10.0:
    HOME=/home/zefram
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/zefram/pub/i686-pc-linux-gnu/bin:/home/zefram/pub/common/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/local/bin:/usr/games
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zsh

From @khwilliamson

Zefram (via RT) wrote​:

# New Ticket Created by Zefram
# Please include the string​: [perl #70151]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=70151 >

This is a bug report for perl from zefram@​fysh.org,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
[Please enter your report here]

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; } BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line 1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I described this on p5p last week, mainly concerned with PL_compcv
which is one of the other things that gets localised. I'd found a
handful of situations with this behaviour​: do "FILE", require, and
string eval. I asked whether the behaviour was intentional, but got
no reply. On further investigation and pondering, I've found that all
the situations doing this are based on doeval() in pp_ctl.c, and I've
decided that it's a bug.

I had a go at modifying doeval() and its callers to avoid the problem,
but I've so far failed to produce a fix. The current block arrangement is
deeply embedded in the structure of the code, such that it is not amenable
to making the compiled op tree survive outside the compilation scope.
Fixing it will be a bigger job than I initially anticipated.

Marginally relevant​: my Parse​::Perl module, if used as a direct
replacement for string eval, does not exhibit this problem. It fully
separates the parsing and execution stages, using code that is derived
from pp_entereval() and doeval() but handles the resulting op tree in
a completely different way.

I wonder if this is has any bearing on [perl #56444] and [perl #62056]
in which the hint is 0 for charnames when it shouldn't be for regcomp.c

The RT System itself - Status changed from 'new' to 'open'

From @demerphq

2009/11/4 karl williamson <public@​khwilliamson.com>​:

Zefram (via RT) wrote​:

# New Ticket Created by  Zefram # Please include the string​:  [perl
#70151]
# in the subject line of all future correspondence about this issue. #
<URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=70151 >

This is a bug report for perl from zefram@​fysh.org,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
[Please enter your report here]

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line 1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I described this on p5p last week, mainly concerned with PL_compcv
which is one of the other things that gets localised.  I'd found a
handful of situations with this behaviour​: do "FILE", require, and
string eval.  I asked whether the behaviour was intentional, but got
no reply.  On further investigation and pondering, I've found that all
the situations doing this are based on doeval() in pp_ctl.c, and I've
decided that it's a bug.

I had a go at modifying doeval() and its callers to avoid the problem,
but I've so far failed to produce a fix.  The current block arrangement is
deeply embedded in the structure of the code, such that it is not amenable
to making the compiled op tree survive outside the compilation scope.
Fixing it will be a bigger job than I initially anticipated.

Marginally relevant​: my Parse​::Perl module, if used as a direct
replacement for string eval, does not exhibit this problem.  It fully
separates the parsing and execution stages, using code that is derived
from pp_entereval() and doeval() but handles the resulting op tree in
a completely different way.

I wonder if this is has any bearing on [perl #56444] and [perl #62056]
in which the hint is 0 for charnames when it shouldn't be for regcomp.c

Im pretty sure not, im pretty sure that charnames problem is just a
fundamental probelm of doing late resolution of the charnames.

I think i explained this before somewhere....

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

From @khwilliamson

demerphq wrote​:

2009/11/4 karl williamson <public@​khwilliamson.com>​:

Zefram (via RT) wrote​:

# New Ticket Created by Zefram # Please include the string​: [perl
#70151]
# in the subject line of all future correspondence about this issue. #
<URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=70151 >

This is a bug report for perl from zefram@​fysh.org,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
[Please enter your report here]

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line 1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I described this on p5p last week, mainly concerned with PL_compcv
which is one of the other things that gets localised. I'd found a
handful of situations with this behaviour​: do "FILE", require, and
string eval. I asked whether the behaviour was intentional, but got
no reply. On further investigation and pondering, I've found that all
the situations doing this are based on doeval() in pp_ctl.c, and I've
decided that it's a bug.

I had a go at modifying doeval() and its callers to avoid the problem,
but I've so far failed to produce a fix. The current block arrangement is
deeply embedded in the structure of the code, such that it is not amenable
to making the compiled op tree survive outside the compilation scope.
Fixing it will be a bigger job than I initially anticipated.

Marginally relevant​: my Parse​::Perl module, if used as a direct
replacement for string eval, does not exhibit this problem. It fully
separates the parsing and execution stages, using code that is derived
from pp_entereval() and doeval() but handles the resulting op tree in
a completely different way.

I wonder if this is has any bearing on [perl #56444] and [perl #62056]
in which the hint is 0 for charnames when it shouldn't be for regcomp.c

Im pretty sure not, im pretty sure that charnames problem is just a
fundamental probelm of doing late resolution of the charnames.

I think i explained this before somewhere....

cheers,
Yves

What I know is that looking at this with gdb, the bit was 0 when I,
perhaps in my ignorance, thought it should be 1.

From @demerphq

2009/11/5 karl williamson <public@​khwilliamson.com>​:

demerphq wrote​:

2009/11/4 karl williamson <public@​khwilliamson.com>​:

Zefram (via RT) wrote​:

# New Ticket Created by  Zefram # Please include the string​:  [perl
#70151]
# in the subject line of all future correspondence about this issue. #
<URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=70151 >

This is a bug report for perl from zefram@​fysh.org,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
[Please enter your report here]

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line 1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I described this on p5p last week, mainly concerned with PL_compcv
which is one of the other things that gets localised.  I'd found a
handful of situations with this behaviour​: do "FILE", require, and
string eval.  I asked whether the behaviour was intentional, but got
no reply.  On further investigation and pondering, I've found that all
the situations doing this are based on doeval() in pp_ctl.c, and I've
decided that it's a bug.

I had a go at modifying doeval() and its callers to avoid the problem,
but I've so far failed to produce a fix.  The current block arrangement
is
deeply embedded in the structure of the code, such that it is not
amenable
to making the compiled op tree survive outside the compilation scope.
Fixing it will be a bigger job than I initially anticipated.

Marginally relevant​: my Parse​::Perl module, if used as a direct
replacement for string eval, does not exhibit this problem.  It fully
separates the parsing and execution stages, using code that is derived
from pp_entereval() and doeval() but handles the resulting op tree in
a completely different way.

I wonder if this is has any bearing on [perl #56444] and [perl #62056]
in which the hint is 0 for charnames when it shouldn't be for regcomp.c

Im pretty sure not, im pretty sure that charnames problem is just a
fundamental probelm of doing late resolution of the charnames.

I think i explained this before somewhere....
...
What I know is that looking at this with gdb, the bit was 0 when I, perhaps
in my ignorance, thought it should be 1.

The hints problem probably does lead to problems with charnames in
some situations.

But even if it didn't we would still have problems with charnames due
the late resolution of the escape.

If we fix the escape problem the hints problem will be irrelevant to
the regex engine.

In other words the hints problem exposes an aspect of the underlying
bug in the regex engine* but is not its cause.

Cheers,
Yves
* although im a tiny bit reluctant to call it a bug, it is a feature
conflict at a conceptual level

--
perl -Mre=debug -e "/just|another|perl|hacker/"

From @cpansprout

On Sun Nov 01 08​:03​:47 2009, zefram@​fysh.org wrote​:

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line
1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I described this on p5p last week, mainly concerned with PL_compcv
which is one of the other things that gets localised. I'd found a
handful of situations with this behaviour​: do "FILE", require, and
string eval. I asked whether the behaviour was intentional, but got
no reply. On further investigation and pondering, I've found that all
the situations doing this are based on doeval() in pp_ctl.c, and I've
decided that it's a bug.

I’ve started looking at this, because I think it *might* get in the way
of some evalbytes bugs that I’ve discovered (good thing I haven’t merged
it yet!).

I have a question​: Why does pp_hintseval have to store a copy of
PL_hintgv? Why can’t pp_entereval simply reconstruct it from PL_curcop?
In either case you have to make a whole new HV.

Then that leads on to another question​: If we eliminate the hintseval
op, can we also remove the op type, or does it have to live on forever,
for the sake of code that explicitly passes its name to Opcode.pm?

--

Father Chrysostomos

From zefram@fysh.org

Father Chrysostomos via RT wrote​:

I have a question​: Why does pp_hintseval have to store a copy of
PL_hintgv? Why can't pp_entereval simply reconstruct it from PL_curcop?

Because string eval needs the compile-time form of the hints hash,
because it's going to compile more code in that lexical environment.
PL_curcop only stores the runtime form of the hints hash, which is a
pale shadow of what existed at compile time.

-zefram

From @cpansprout

On Sun Nov 01 08​:03​:47 2009, zefram@​fysh.org wrote​:

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line
1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I’m trying to fix this by setting up a new scope for hint localisation,
but I’m slightly mystified by this line in S_doeval​:

  yystatus = (!in_require && CATCH_GET) ? S_try_yyparse(aTHX_
GRAMPROG) : yyparse(GRAMPROG);

In particular, what are the possible values of yystatus? Which values
indicate that scopes have been left, and how many scopes/how far?

--

Father Chrysostomos

From @cpansprout

On Sun Nov 01 08​:03​:47 2009, zefram@​fysh.org wrote​:

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line
1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

Interestingly, there are tests for this buggy behaviour in t/comp/hints.t​:

  # op_entereval should keep the pragmas it was compiled with
  eval q*
  print "not " if $^H{foo} ne "a";
  print "ok 13 - \$^H{foo} is 'a' at eval-\"\" time\n";
  print "not " unless $^H & 0x00020000;
  print "ok 14 - \$^H contains HINT_LOCALIZE_HH at eval\"\"-time\n";
  *;

I think the tests should change.

--

Father Chrysostomos

From @iabyn

On Wed, Nov 16, 2011 at 03​:51​:11PM -0800, Father Chrysostomos via RT wrote​:

On Sun Nov 01 08​:03​:47 2009, zefram@​fysh.org wrote​:

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line
1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I’m trying to fix this by setting up a new scope for hint localisation,
but I’m slightly mystified by this line in S_doeval​:

yystatus = \(\!in\_require && CATCH\_GET\) ? S\_try\_yyparse\(aTHX\_

GRAMPROG) : yyparse(GRAMPROG);

In particular, what are the possible values of yystatus? Which values
indicate that scopes have been left, and how many scopes/how far?

I introduced this code with this commit​:

  27e9045

but have already mostly forgotten it.

I depends on what sort of scopes you are talking about. That commit fixed
an issue where the number of pushed JMPENV scopes got out of sync with the
number of pushed EVAL contexts, so the longjump went back to a place that
wasn't suitable.

The return values of S_try_yyparse are documented just above that function​:

* 0​: yyparse() successful
* 1​: yyparse() failed
* 3​: yyparse() died

so 0/1 are normal execution, with yyparse() having returned true or false​:
no exceptions have been raised, and no scopes have been popped.
3 indicates that the compiling code raised an exception, e.g. BEGIN {die },
so a longjump will have been done, one JMPENV will have been popped
(S_try_yyparse both pushes and pops a JMPENV, so the net loss is zero),
and I can't off the top of my head remember whether the die code (which
does the longjmp) pops an EVAL context.

--
Atheism is a religion like not collecting stamps is a hobby

From @cpansprout

On Thu Nov 17 03​:20​:15 2011, davem wrote​:

On Wed, Nov 16, 2011 at 03​:51​:11PM -0800, Father Chrysostomos via RT
wrote​:

On Sun Nov 01 08​:03​:47 2009, zefram@​fysh.org wrote​:

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line
1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I’m trying to fix this by setting up a new scope for hint localisation,
but I’m slightly mystified by this line in S_doeval​:

yystatus = \(\!in\_require && CATCH\_GET\) ? S\_try\_yyparse\(aTHX\_

GRAMPROG) : yyparse(GRAMPROG);

In particular, what are the possible values of yystatus? Which values
indicate that scopes have been left, and how many scopes/how far?

I introduced this code with this commit​:

27e904532594b7fb224bdf9a05bf3b5336b8a39e

Interesting. The same thing affects UNITCHECK{die}, which crashes
inside FETCH. I’ll make a separate ticket for that.

--

Father Chrysostomos

From @cpansprout

On Thu Nov 17 03​:20​:15 2011, davem wrote​:

On Wed, Nov 16, 2011 at 03​:51​:11PM -0800, Father Chrysostomos via RT
wrote​:

On Sun Nov 01 08​:03​:47 2009, zefram@​fysh.org wrote​:

$ perl -lwe 'BEGIN { $^H{foo}=1; } BEGIN { eval q{ $^H{bar} = 1; }; }
BEGIN { print "foo=$^H{foo}, bar=$^H{bar}"; }'
Use of uninitialized value in concatenation (.) or string at -e line
1.
foo=1, bar=

The execution of the string-evaled code takes place with %^H, and a
bunch
of other things, localised in a block that was set up to localise them
for *compilation* of the string-evaled code.

I’m trying to fix this by setting up a new scope for hint localisation,
but I’m slightly mystified by this line in S_doeval​:

yystatus = \(\!in\_require && CATCH\_GET\) ? S\_try\_yyparse\(aTHX\_

GRAMPROG) : yyparse(GRAMPROG);

In particular, what are the possible values of yystatus? Which values
indicate that scopes have been left, and how many scopes/how far?

I introduced this code with this commit​:

27e904532594b7fb224bdf9a05bf3b5336b8a39e

but have already mostly forgotten it.

I depends on what sort of scopes you are talking about. That commit fixed
an issue where the number of pushed JMPENV scopes got out of sync with the
number of pushed EVAL contexts, so the longjump went back to a place that
wasn't suitable.

The return values of S_try_yyparse are documented just above that
function​:

* 0​: yyparse() successful
* 1​: yyparse() failed
* 3​: yyparse() died

so 0/1 are normal execution, with yyparse() having returned true or false​:
no exceptions have been raised, and no scopes have been popped.
3 indicates that the compiling code raised an exception, e.g. BEGIN
{die },
so a longjump will have been done, one JMPENV will have been popped
(S_try_yyparse both pushes and pops a JMPENV, so the net loss is zero),
and I can't off the top of my head remember whether the die code (which
does the longjmp) pops an EVAL context.

It seems it does, as that was the only assumption that worked. I’ve now
pushed a fix as commit f45b078.

--

Father Chrysostomos

@cpansprout - Status changed from 'open' to 'resolved'