From david.hillman@livetext.com

This is a bug report for perl from david.hillman@​livetext.com,
generated with the help of perlbug 1.35 running under perl v5.8.6.

  The following script demonstrates the problem, which has been
replicated on the platforms listed in the POD.
When a hashref being used as a 'while condition' runs out, the
resulting undefined error is reported from the last
line of the loop, instead of the first.

#!/usr/bin/perl

use strict; # error is not thrown unless strict is used

if (not $ARGV[0] or $ARGV[0] =~ 'bad_error') {
  bad_error();
} else {
  good_error();
}

sub good_error {
  my %hash;
  my $hash_ref = { name => 'value' };
  while ( 1 )
  {
  %hash = %{ $hash_ref }; # error correctly reported on this line
  last unless %hash;

  print "in good_error\n";
  $hash_ref = undef;
  }
}

sub bad_error {
  my %hash;
  my $hash_ref = { name => 'value' };
  while ( %hash = %{ $hash_ref } ) # error occurs here
  {
  print "in bad_error\n";
  $hash_ref = undef;
  1; # error reported on this line
  }
}

=pod

=head1 Undefined Hashref at Wrong Line Number

Script dies with this error message​:
Can't use an undefined value as a HASH reference at ./
undefined_hashref_line_num line 31.

The error message should read​:
Can't use an undefined value as a HASH reference at ./
undefined_hashref_line_num line 16.

The actual error message reports the line number of the last
statement of the loop instead of the assignment in the while().

The error only occurs when the strict pragma is in use.

Discovered when doing a this with DBI​:

  while( %results = %{ $dbh->fetrow_hashref() } ) {

  # Do some stuff with %results

  some statement (error was reported on this line
  when the statement handle returned undef)
  }

However, the script above proves that this issue is not DBI related.

=head1 Script Usage

  undefined_hashref_line_num.pl --good-error|--bad-error

Option --good-error dies with a diagnostic message and correct line
number.
Option --bad-error dies with a diagnostic message and incorrect line
number.

=head1 Reproduced On

=over

=item *
Mac OS X 10.3 with the default Perl 5.8.1

=item *
Mac OS X 10.4.* with 5.8.1, 5.8.6, 5.8.7, and 5.9.2

=item *
FreeBSD 4.8 with 5.8.3

=back

=cut

Flags​:
  category=core
  severity=low

Site configuration information for perl v5.8.6​:

Configured by root at Sun Mar 20 16​:31​:42 PST 2005.

Summary of my perl5 (revision 5 version 8 subversion 6) configuration​:
  Platform​:
  osname=darwin, osvers=8.0, archname=darwin-thread-multi-2level
  uname='darwin b28.apple.com 8.0 darwin kernel version 7.5.0​: thu
mar 3 18​:48​:46 pst 2005; root​:xnuxnu-517.99.13.obj~1release_ppc power
macintosh powerpc '
  config_args='-ds -e -Dprefix=/usr -Dccflags=-g -pipe -
Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-g -pipe -fno-common -DPERL_DARWIN -no-cpp-
precomp -fno-strict-aliasing -I/usr/local/include',
  optimize='-Os',
  cppflags='-no-cpp-precomp -g -pipe -fno-common -DPERL_DARWIN -no-
cpp-precomp -fno-strict-aliasing -I/usr/local/include'
  ccversion='', gccversion='3.3 20030304 (Apple Computer, Inc.
build 1809)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
  ivtype='long', ivsize=4, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='-L/usr/
local/lib'
  libpth=/usr/local/lib /usr/lib
  libs=-ldbm -ldl -lm -lc
  perllibs=-ldl -lm -lc
  libc=/usr/lib/libc.dylib, so=dylib, useshrplib=true,
libperl=libperl.dylib
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags='-bundle -undefined dynamic_lookup -L/
usr/local/lib'

Locally applied patches​:
  23953 - fix for File​::Path​::rmtree CAN-2004-0452 security issue
  33990 - fix for setuid perl security issues

@​INC for perl v5.8.6​:
  /Applications/livetext/branches/misk5.cms/conf/test
  /Applications/livetext/branches/misk5.cms/mod
  /System/Library/Perl/5.8.6/darwin-thread-multi-2level
  /System/Library/Perl/5.8.6
  /Library/Perl/5.8.6/darwin-thread-multi-2level
  /Library/Perl/5.8.6
  /Library/Perl
  /Network/Library/Perl/5.8.6/darwin-thread-multi-2level
  /Network/Library/Perl/5.8.6
  /Network/Library/Perl
  /System/Library/Perl/Extras/5.8.6/darwin-thread-multi-2level
  /System/Library/Perl/Extras/5.8.6
  /Library/Perl/5.8.1
  .

Environment for perl v5.8.6​:
  DYLD_LIBRARY_PATH (unset)
  HOME=/Users/hillman
  LANG (unset)
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/bin​:/sbin​:/usr/bin​:/usr/sbin​:/usr/local/bin​:/Users/hillman/
bin​:/usr/local/mysql/bin
  PERL5LIB=/Applications/livetext/branches/misk5.cms/conf/test​:/
Applications/livetext/branches/misk5.cms/mod
  PERL_BADLANG (unset)
  SHELL=/bin/bash

--
David Hillman
LiveText, Inc
1.866.LiveText x235

From BQW10602@nifty.com

On Fri, 18 Nov 2005 17​:08​:05 -0800, David Hillman (via RT) <perlbug-followup@​perl.org> wrote

The following script demonstrates the problem\, which has been  

replicated on the platforms listed in the POD.
When a hashref being used as a 'while condition' runs out, the
resulting undefined error is reported from the last
line of the loop, instead of the first.

#!/usr/bin/perl

use strict; # error is not thrown unless strict is used

if (not $ARGV[0] or $ARGV[0] =~ 'bad_error') {
  ^
bad_error();
} else {
good_error();
}

(snip)

=head1 Script Usage

undefined_hashref_line_num.pl --good-error|--bad-error
  ^

We should tell a hyphen from a underscore, right?
If --bad_error used, no problem is reproduced.

Regards,
SADAHIRO Tomoyuki

The RT System itself - Status changed from 'new' to 'open'

From @iabyn

On Fri, Nov 18, 2005 at 05​:08​:05PM -0800, David Hillman wrote​:

When a hashref being used as a 'while condition' runs out, the
resulting undefined error is reported from the last
line of the loop, instead of the first.

Thanks for the report. There are a number of similar bugs present in the
perl interpreter.

p5pers​:

A generic fix to this would be attach the line number to *every* op rather
than just to control ops (we'd still get the file name from the last cop).
The cost of this is an extra 4 bytes per op. Is this worth the memory
cost?

--
"You're so sadly neglected, and often ignored.
A poor second to Belgium, When going abroad."
  -- Monty Python - "Finland"

From @smpeters

On Sat, Nov 19, 2005 at 11​:39​:58PM +0000, Dave Mitchell wrote​:

On Fri, Nov 18, 2005 at 05​:08​:05PM -0800, David Hillman wrote​:

When a hashref being used as a 'while condition' runs out, the
resulting undefined error is reported from the last
line of the loop, instead of the first.

Thanks for the report. There are a number of similar bugs present in the
perl interpreter.

p5pers​:

A generic fix to this would be attach the line number to *every* op rather
than just to control ops (we'd still get the file name from the last cop).
The cost of this is an extra 4 bytes per op. Is this worth the memory
cost?

Considering the number of bugs currently like this, and the number of questions
generated about problems like this, yes, I think it'd be worth it.

Steve Peters
steve@​fisharerojo.org

From @jimc

Dave Mitchell wrote​:

On Fri, Nov 18, 2005 at 05​:08​:05PM -0800, David Hillman wrote​:

When a hashref being used as a 'while condition' runs out, the
resulting undefined error is reported from the last
line of the loop, instead of the first.

Thanks for the report. There are a number of similar bugs present in the
perl interpreter.

p5pers​:

A generic fix to this would be attach the line number to *every* op rather
than just to control ops (we'd still get the file name from the last cop).
The cost of this is an extra 4 bytes per op. Is this worth the memory
cost?

hmm. this may border on the grotesque, but hopefully I will have the
good sense to consider killing this msg b4 actually sending it. (or not..)

assumptions & observations;

- rarely need extra info. make it a pragma, ?
  and use only when/where needed.

- only tokenizer could know line-numbers with sufficient detail to
  collect them inside an expression.

- only machine-generated expressions will exceed 255 lines (this is
debug support after all)
  a tiny-int / char would be sufficient to remember lines

- assume line-num per opcode is useful.
  ideally users want to know line numbers of terms and operators,
  how to compute line-nums for terms based on those of opcodes

- denser encodings are possible​:
struct extra_cop_line_info {
  int MaxOPs_per_COP;
  bool incvec[MaxOPs_per_COP]; // loosely 1 flag per opcode.
  unsigned char nextline[MaxOPs_per_COP];
};
so. if an op is on same line as the previous (*), then its incvec bit is
zero.
else the bit ==1 says that next byte in nexline[] field is an offset
from the COP line-num.
And MaxOPs_per_COP can be dynamically set to the number of opcodes
attached to a COP, once its known for each statement.

- hang this optional cop-related info off the cop; then pointer overhead
is reduced.

- create new cop sub-type that has the room for the extra info.
  this is vaguely similar to nextstate vs dbstate,
  and avoids unused storage (if it can be properly pragma controlled)

ugly is as ugly does ;-)
jimc

From @iabyn

On Sat, Nov 19, 2005 at 11​:39​:58PM +0000, Dave Mitchell wrote​:

A generic fix to this would be attach the line number to *every* op rather
than just to control ops (we'd still get the file name from the last cop).
The cost of this is an extra 4 bytes per op. Is this worth the memory
cost?

(replying to myself..)

A data point​:

adding the extra field to BASEOP shows that on a 32-bit system, adding a 'use
POSIX' to an empty program causes 1184K rather than 1048K extra memory to
be allocated (13%), while on a 64-bit Opteron system there was no change
(possibly due to pre-existing alignment wastage). In neither case was
there a detectable change in performance on 'make test'.

Note that the 13% is an upper limit​: it so happens that this respresents a
single brk() call; the malloc library may not yet have actually allocated
all this extra memory yet.

--
The optimist believes that he lives in the best of all possible worlds.
As does the pessimist.

From @Tux

On Sat, 19 Nov 2005 19​:15​:22 -0700, Jim Cromie <jim.cromie@​gmail.com> wrote​:

Dave Mitchell wrote​:

On Fri, Nov 18, 2005 at 05​:08​:05PM -0800, David Hillman wrote​:

When a hashref being used as a 'while condition' runs out, the
resulting undefined error is reported from the last
line of the loop, instead of the first.

Thanks for the report. There are a number of similar bugs present in the
perl interpreter.

p5pers​:

A generic fix to this would be attach the line number to *every* op rather
than just to control ops (we'd still get the file name from the last cop).
The cost of this is an extra 4 bytes per op. Is this worth the memory
cost?

hmm. this may border on the grotesque, but hopefully I will have the
good sense to consider killing this msg b4 actually sending it. (or not..)

assumptions & observations;

- rarely need extra info.

yes

make it a pragma, ?
and use only when/where needed.

Will people remember?

- only tokenizer could know line-numbers with sufficient detail to
collect them inside an expression.

- only machine-generated expressions will exceed 255 lines (this is
debug support after all)
a tiny-int / char would be sufficient to remember lines

What a ver very very wrong assumption!

- assume line-num per opcode is useful.
ideally users want to know line numbers of terms and operators,
how to compute line-nums for terms based on those of opcodes

- denser encodings are possible​:
struct extra_cop_line_info {
int MaxOPs_per_COP;
bool incvec[MaxOPs_per_COP]; // loosely 1 flag per opcode.

Oh, C++ comment.
Bad bad boy!

  unsigned char nextline\[MaxOPs\_per\_COP\];
  \};

so. if an op is on same line as the previous (*), then its incvec bit is
zero.
else the bit == 1 says that next byte in nexline[] field is an offset
from the COP line-num.
And MaxOPs_per_COP can be dynamically set to the number of opcodes
attached to a COP, once its known for each statement.

- hang this optional cop-related info off the cop; then pointer overhead
is reduced.

- create new cop sub-type that has the room for the extra info.
this is vaguely similar to nextstate vs dbstate,
and avoids unused storage (if it can be properly pragma controlled)

--
H.Merijn Brand Amsterdam Perl Mongers (http​://amsterdam.pm.org/)
using Perl 5.6.2, 5.8.0, 5.8.5, & 5.9.2 on HP-UX 10.20, 11.00 & 11.11,
AIX 4.3 & 5.2, SuSE 9.2 & 9.3, and Cygwin. http​://www.cmve.net/~merijn
Smoking perl​: http​://www.test-smoke.org, perl QA​: http​://qa.perl.org
reports to​: smokers-reports@​perl.org, perl-qa@​perl.org

From @nwc10

On Sat, Nov 19, 2005 at 07​:15​:22PM -0700, Jim Cromie wrote​:

Dave Mitchell wrote​:

A generic fix to this would be attach the line number to *every* op rather
than just to control ops (we'd still get the file name from the last cop).
The cost of this is an extra 4 bytes per op. Is this worth the memory
cost?

Yes, I believe it would be, based on the number of (valid) bug reports we
get about this (and therefore inferring the number of people who find or
are confused by the bug, but don't report it)

- denser encodings are possible​:
struct extra_cop_line_info {
int MaxOPs_per_COP;
bool incvec[MaxOPs_per_COP]; // loosely 1 flag per opcode.
unsigned char nextline[MaxOPs_per_COP];
};
so. if an op is on same line as the previous (*), then its incvec bit is
zero.
else the bit ==1 says that next byte in nexline[] field is an offset
from the COP line-num.
And MaxOPs_per_COP can be dynamically set to the number of opcodes
attached to a COP, once its known for each statement.

Well, that use of 'bool' goes wrong because it's probably char, not a bit, the
way perl's C source defines it​:

#ifndef HAS_BOOL
# if defined(UTS) || defined(VMS)
# define bool int
# else
# define bool char
# endif
# define HAS_BOOL 1
#endif

but that's a comment on implementation details. It might work to use the
FD_SET macros, but however it's done, bitvectors are possible.

- create new cop sub-type that has the room for the extra info.
this is vaguely similar to nextstate vs dbstate,
and avoids unused storage (if it can be properly pragma controlled)

Well, or something possibly more sick. But based on all the SV pointer
manipulations will work. For every current op type, define a variant type that
has an extra member at the *start* that stores the line number. There are 5
spare bits in the base OP structure currently. Use 1 bit to signal that the
variant type is being used. After allocating the variant type, adjust the
pointer by the correct offset so that the members line up. Reverse that
offset before free()ing the OP. (See how PVs, PVIVs, AVs and HVs now work)
I don't know the minimum number of places line numbers need to be stored
in addition to COPs to make all the error reporting correct, but this
approach would minimise the extra memory requirements.

Nicholas Clark