Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perl segfaults when freeing deeply nested structures #8978

Closed
p5pRT opened this issue Jul 28, 2007 · 24 comments
Closed

perl segfaults when freeing deeply nested structures #8978

p5pRT opened this issue Jul 28, 2007 · 24 comments

Comments

@p5pRT
Copy link

p5pRT commented Jul 28, 2007

Migrated from rt.perl.org#44225 (status was 'resolved')

Searchable as RT44225$

@p5pRT
Copy link
Author

p5pRT commented May 3, 2003

From perl-5.8.0@ton.iguana.be

Created by perl-5.8.0@ton.iguana.be

The following program builds a double linked list, and then
destroys it again. On exit it dumps core. Effect happens in
many but not all versions off 5.8.0, no problems in 5.6.1.

Increasing the 10000 to 1000000 makes more perl versions
crash (e.g. my 5.6.1 begins crashing too). That version
uses about 160M for me, so it's no out of memory effect.

I was unable to reduce the program further.

#! /usr/bin/perl -w
my $head = {};
$head->{"next"} = $head;
$head->{"previous"} = $head;

for (1..10000) {
  my $node = {};

  # attach node in front
  $prev = $head;
  my $next = $prev->{"next"};
  $prev->{"next"} = $node;
  $node->{"next"} = $next;
  $node->{"previous"} = $prev;
  $next->{"previous"} = $node;
}

# Make nodes single connected
$prev = $head;
my $next = $prev->{"next"};
while ($next != $head) {
  undef($next->{"previous"});
  $prev = $next;
  $next = $prev->{"next"};
}
undef($prev->{"next"});

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl v5.8.0:

Configured by ton at Tue Nov 12 01:56:18 CET 2002.

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.19, archname=i686-linux-thread-multi-64int-ld
    uname='linux quasar 2.4.19 #5 wed oct 2 02:34:25 cest 2002 i686 unknown '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=undef uselongdouble=define
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -fomit-frame-pointer',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='2.95.3 20010315 (release)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='long double', nvsize=12, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -ldb -ldl -lm -lpthread -lc -lposix -lcrypt -lutil
    perllibs=-lnsl -ldl -lm -lpthread -lc -lposix -lcrypt -lutil
    libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.0:
    /usr/lib/perl5/5.8.0/i686-linux-thread-multi-64int-ld
    /usr/lib/perl5/5.8.0
    /usr/lib/perl5/site_perl/5.8.0/i686-linux-thread-multi-64int-ld
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    .


Environment for perl v5.8.0:
    HOME=/home/ton
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/ton/bin.Linux:/home/ton/bin:/home/ton/bin.SampleSetup:/usr/local/bin:/usr/local/sbin:/usr/local/jre/bin:/home/oracle/product/9.0.1/bin:/usr/local/ar/bin:/usr/games/bin:/usr/X11R6/bin:/usr/share/bin:/usr/bin:/usr/sbin:/bin:/sbin:.
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented May 5, 2003

From enache@rdslink.ro

On Sat, May 03, 2003 at 02​:53​:08PM -0000, perl-5.8.0@​ton.iguana.be (via RT) wrote​:

The following program builds a double linked list, and then
destroys it again. On exit it dumps core. Effect happens in
many but not all versions off 5.8.0, no problems in 5.6.1.

Increasing the 10000 to 1000000 makes more perl versions
crash (e.g. my 5.6.1 begins crashing too). That version
uses about 160M for me, so it's no out of memory effect.

Simple hard stack overflow caused be deep recursion.
(22095.pl below is your test with 10000 changed to 100000).

Solution​: adjust your limits.

ex.

$ perl 22095.pl
Segmentation fault
$ ulimit -s 40000
$ perl 22095.pl
$

Regards,
Adi

@p5pRT
Copy link
Author

p5pRT commented May 5, 2003

From perl5-porters@ton.iguana.be

In article <20030505152443.GA2169@​ratsnest.hole>,
  Enache Adrian <enache@​rdslink.ro> writes​:

On Sat, May 03, 2003 at 02​:53​:08PM -0000, perl-5.8.0@​ton.iguana.be (via RT) wrote​:

The following program builds a double linked list, and then
destroys it again. On exit it dumps core. Effect happens in
many but not all versions off 5.8.0, no problems in 5.6.1.

Increasing the 10000 to 1000000 makes more perl versions
crash (e.g. my 5.6.1 begins crashing too). That version
uses about 160M for me, so it's no out of memory effect.

Simple hard stack overflow caused be deep recursion.
(22095.pl below is your test with 10000 changed to 100000).

Solution​: adjust your limits.

Mmm, since this happens in a module that should be always useable without
doing external things, that's not so great as a general solution. I'll
sever both links while walking the structure, which should also solve it
though at a speed hit.

Since my program has no recursion, I assume the recursion is
happening inside perl, probabably code doing something like​:

dec_thingy {
  return if --self->refcount > 0;
  for (self->all_references) {
  dec_thingy($_);
  }
  free(self);
}

which would then be called recursively for as many times as my linear
list is long ?

Since e.g. on linux the default stack limit I get is only 8192, maybe
worth considering making that iterative ? Otherwise this will generally
bite if you drop a deep structure.

This also begs the question why 5.6.1 on the same machine can handle
much bigger structures before failing.

@p5pRT
Copy link
Author

p5pRT commented May 9, 2003

From sky@nanisky.com

On måndag, maj 5, 2003, at 18​:50 Europe/Stockholm, Ton Hospel wrote​:

Since e.g. on linux the default stack limit I get is only 8192, maybe
worth considering making that iterative ? Otherwise this will generally
bite if you drop a deep structure.

Patches welcome :)

Arthur

@p5pRT
Copy link
Author

p5pRT commented May 9, 2003

From perl5-porters@ton.iguana.be

In article <61BDB378-81FE-11D7-9B8A-000393CB5BC4@​nanisky.com>,
  Arthur Bergman <sky@​nanisky.com> writes​:

On måndag, maj 5, 2003, at 18​:50 Europe/Stockholm, Ton Hospel wrote​:

Since e.g. on linux the default stack limit I get is only 8192, maybe
worth considering making that iterative ? Otherwise this will generally
bite if you drop a deep structure.

Patches welcome :)

Probably this thing should be looked into anyways even if it's kept
recursive. 8192 (the default stack limit on my linux box) is 8 Meg,
and the structure in my example is only 10000 entries. That seems
to imply about 800 bytes per frame, which is excessive.

@p5pRT
Copy link
Author

p5pRT commented Mar 22, 2006

From reto.stamm@xilinx.com

Created by reto@xilinx.com

The following script core dumps as it tries to collect $hash.

# start of script
{
  my $hash = {};
  my $in_hash = $hash;
  for (my $i = 0; $i < 250000; $i ++) {
  $in_hash->{INSIDE} = {};
  $in_hash = $in_hash->{INSIDE};
  }
}
print "All cleaned up now.";
# end of script

This seems to happen on 32 bit Linux and 32 bit Solaris.

Looking at the stack trace, it looks like it's analyzing the hash
recursively.

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.0:

Configured by bhcompile'
cf_email='bhcompile at Mon Sep 15 10:01:56 EDT 2003.

Summary of my rderl (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.21-1.1931.2.393.entsmp, 
archname=i386-linux-thread-multi
    uname='linux por'
    config_args='-des -Doptimize=-O2 -g -pipe -march=i386 -mcpu=i686 
-Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red 
Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux 
-Dvendorprefix=/usr -Dsiteprefix=/usr 
-Dotherlibdirs=/usr/lib/perl5/5.8.0 -Duseshrplib -Dusethreads 
-Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db 
-Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio 
-Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less 
-isr'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef'
 useithreads=define usemultiplicity=
    useperlio= d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=un uselongdouble=
    usemymalloc=, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS 
-DDEBUGGING -fno-strict-aliasing -I/usr/local/include 
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING 
-fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.2.3 20030502 (Red Hat Linux 3.2.3-19)', 
gccosandvers=''
gccversion='3.2.3 200305'
    intsize=o, longsize=s, ptrsize=l, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long'
k', ivsize=4'
ivtype, nvtype='double'
o_no', nvsize=, Off_t='', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc'
l', ldflags =' -L/usr/local/lib'
ldflags_use'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libper
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so', d_dlsymun=undef, ccdlflags='-rdynamic 
-Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC'
ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5', lddlflags='s 
Unicode/Normalize XS/A'

Locally applied patches:
    MAINT18379


@INC for perl v5.8.0:
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    .


Environment for perl v5.8.0:
    HOME=/home/reto
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LANGVAR=en_US.UTF-8
    
LD_LIBRARY_PATH=/home/reto/sb/HEAD/rtf/bin/lin:/build/xfndry/HEAD/rtf/bin/lin:/build/xfndry/HEAD/rtf/bin/lin
    LOGDIR (unset)
    
PATH=.:/devl/swtools:/home/reto/gnu/bin:/home/reto/gnu/firefox:/usr/X/bin:/usr/kerberos/bin:/home/reto/sb/HEAD/rtf/bin/lin:/build/xfndry/HEAD/rtf/bin/lin:/home/reto/sb/HEAD/env/bin/lin:/build/xfndry/HEAD/env/bin/lin:/home/reto/sb/HEAD/rtf:/build/xfndry/HEAD/rtf:/build/xfndry/HEAD/rtf/bin/lin:/devl/swtools:/bin:/usr/bin:/usr/local/bin
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/tcsh
    dlflags='-share (unset)



@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2006

From chromatic@wgz.org

On Wednesday 22 March 2006 14​:43, Reto Stamm wrote​:

The following script core dumps as it tries to collect $hash.

# start of script
{
my $hash = {};
my $in_hash = $hash;
for (my $i = 0; $i < 250000; $i ++) {
$in_hash->{INSIDE} = {};
$in_hash = $in_hash->{INSIDE};
}
}
print "All cleaned up now.";
# end of script

This seems to happen on 32 bit Linux and 32 bit Solaris.

Looking at the stack trace, it looks like it's analyzing the hash
recursively.

Looking at the stack trace shows tens of thousands of calls. I think it's
recursing past what the C stack can handle. There may be no easy fix, though
rewriting the functions S_hfreeentries, Perl_hv_undef, Perl_sv_clear,
Perl_sv_free2, and Perl_hv_free_ent for iteration instead of recursion might
do it.

-- c

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2006

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Apr 28, 2006

From christian@pflanze.mine.nu

Created by christian@pflanze.mine.nu

See the code below which segfaults.

I don't see any cyclic reference, and running LengthOfStream
(SequenceStream 15000) in an endless loop runs just fine without
leaking, so cyclic references are not the problem.

It is clearly related to the stack size, increasing the stack limit
using ulimit raises the length of streams that can be processed.

It should be possible to create such streams of infinite length, being
built and deallocated while being traversed. This bug means that the
functional stream programming paradigm (or probably anything else
which creates closures in a repeated fashion?) cannot be applied to
perl 5.

use strict;

sub Delay ( & ) {
  [ 0, $_[0] ]
}

sub Force ( $ ) {
  my $p=shift;
  $$p[0] ? $$p[1]
  : do {
  $$p[1]= &{ $$p[1] };
  $$p[0]= 1;
  $$p[1]
  };
}

sub Cons ( $ $ ) {
  [ @​_ ]
}

sub Car ( $ ) {
  $_[0][0]
}

sub Cdr ( $ ) {
  $_[0][1]
}

our $Nil= [];

sub LengthOfStream ( $ ) {
  my ($strm)=@​_;
  my $len=0;
  while(1) {
  my $p= Force($strm);
  if ($p eq $Nil) {
  return $len;
  } else {
  $len++;
  $strm= Cdr($p);
  }
  }
}

sub SequenceStream ( $ ) {
  my ($n)=@​_;
  Delay {
  ($n < 0) ? $Nil
  #​: Cons $n, SequenceStream($n-1);
  : [ $n, SequenceStream($n-1) ];#no change
  };
}

if (0) {
  while(1) {
  print LengthOfStream (SequenceStream 15000),"\n";
  }
}
# above is no problem. but​:
if (0) {
  print LengthOfStream (SequenceStream 15000),"\n";
  print LengthOfStream (SequenceStream 15000),"\n";
  print LengthOfStream (SequenceStream 15000),"\n";
  print LengthOfStream (SequenceStream 25000),"\n";
  # now here a segfault happens
  print LengthOfStream (SequenceStream 25000),"\n";
}
if (1) {
  my $s= (SequenceStream 100000);
  print LengthOfStream $s ,"\n";
  print "Still alive\n";
  undef $s;# <-- segfaults
  print "Still alive?\n";
}

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl v5.8.7:

Configured by Debian Project at Thu Jun  9 00:28:22 EST 2005.

Summary of my perl5 (revision 5 version 8 subversion 7) configuration:
  Platform:
    osname=linux, osvers=2.4.27-ti1211, archname=i386-linux-thread-multi
    uname='linux kosh 2.4.27-ti1211 #1 sun sep 19 18:17:45 est 2004 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i386-linux -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.7 -Dsitearch=/usr/local/lib/perl/5.8.7 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.7 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='3.3.6 (Debian 1:3.3.6-6)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so.5.8.7
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.7:
    /etc/perl
    /usr/local/lib/perl/5.8.7
    /usr/local/share/perl/5.8.7
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    /usr/local/lib/perl/5.8.4
    /usr/local/share/perl/5.8.4
    /usr/local/lib/perl/5.8.3
    /usr/local/share/perl/5.8.3
    .


Environment for perl v5.8.7:
    HOME=/home/chris
    LANG=de_CH
    LANGUAGE (unset)
    LC_CTYPE=de_CH
    LC_NUMERIC=C
    LD_LIBRARY_PATH=:/home/chris/lib
    LOGDIR (unset)
    PATH=/usr/local/Gambit-C/bin:/opt/j2sdk_nb/j2sdk1.4.2/bin/:/home/chris/local/bin:/home/chris/bin:/root/local/bin:/root/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/bin/X11:/usr/local/sbin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Apr 28, 2006

From christian@pflanze.mine.nu

  (It looks like the below text which I already sent as followup somehow
got lost, so I'm posting it through RT again.)

It seems like it might be a problem with parameter passing through @​_.
The following change prevents the segfault in all cases.

# either use this definition instead​:

# sub LengthOfStream ( $ ) {
# my ($strm)=@​_;
# #@​_=();# does not help
# undef $_[0];# <------ helps
# my $len=0;
# while(1) {
# my $p= Force($strm);
# if ($p eq $Nil) {
# return $len;
# } else {
# $len++;
# $strm= Cdr($p);
# }
# }
# }

# or this​:

sub LengthOfStream_ {
  my ($strm,$len)=@​_;
  #@​_=(); # does not help
  undef $_[0];# helps.
  my $p= Force($strm);
  if ($p eq $Nil) {
  $len
  } else {
  @​_=(Cdr($p), $len+1);
  goto \&LengthOfStream; # tail call
  }
}

sub LengthOfStream ( $ ) {
  push @​_,0;
  goto \&LengthOfStream_;
}

But it has, of course, the ugly side effect of erasing the variable
being passed by the caller​:

my $s= SequenceStream 100000;
print LengthOfStream $s,"\n"; # -> 100001
# but now $s is undef

I find it strange, that this case also requires the undef $_[0] (and
@​_=() is not enough!)​:

print LengthOfStream SequenceStream 100000,"\n";

Any idea?

What I also don't understand is why this leaks memory and eventually
terminates because of an out of memory condition (it should just run
forever)​:

sub InfiniteStream {
  my ($startnumber)=@​_;
  #undef $_[0]; Modification of a read-only value attempted -- of course.
  #@​_=();
  Delay {
  Cons $startnumber, InfiniteStream($startnumber+1);
  };
}

LengthOfStream InfiniteStream(0);

Any idea here?

Christian.

@p5pRT
Copy link
Author

p5pRT commented Apr 28, 2006

christian@pflanze.mine.nu - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Apr 28, 2006

From @iabyn

On Fri, Apr 28, 2006 at 04​:23​:49AM -0700, Christian wrote​:

See the code below which segfaults.

It's not actually closure-related; it's just that perl currently uses a
recursive method to free recursive data structures, which can blow the C
stack on deeply nested ones. For example the following will crash perl
just as happily as your example​:

  my $s;
  for (1..100_000) {
  $s = [$_, $s ];
  }
  undef $s;

It's on our list of things to fix one day...

--
Spock (or Data) is fired from his high-ranking position for not being able
to understand the most basic nuances of about one in three sentences that
anyone says to him.
  -- Things That Never Happen in "Star Trek" #19

@p5pRT
Copy link
Author

p5pRT commented Apr 29, 2006

From @iabyn

On Fri, Apr 28, 2006 at 02​:25​:25PM +0200, Christian wrote​:

It seems like it might be a problem with parameter passing through
@​_. The following change prevents the segfault in all cases.
sub LengthOfStream_ {
my ($strm,$len)=@​_;
#@​_=(); # does not help
undef $_[0];# helps.

This is avoiding the segfault by avoidly building a recursive data
structure.

--
The optimist believes that he lives in the best of all possible worlds.
As does the pessimist.

@p5pRT
Copy link
Author

p5pRT commented Apr 29, 2006

From christian@pflanze.mine.nu

It seems like it might be a problem with parameter passing through
@​_. The following change prevents the segfault in all cases.

# either use this definition instead​:

# sub LengthOfStream ( $ ) {
# my ($strm)=@​_;
# #@​_=();# does not help
# undef $_[0];# <------ helps
# my $len=0;
# while(1) {
# my $p= Force($strm);
# if ($p eq $Nil) {
# return $len;
# } else {
# $len++;
# $strm= Cdr($p);
# }
# }
# }

# or this​:

sub LengthOfStream_ {
  my ($strm,$len)=@​_;
  #@​_=(); # does not help
  undef $_[0];# helps.
  my $p= Force($strm);
  if ($p eq $Nil) {
  $len
  } else {
  @​_=(Cdr($p), $len+1);
  goto \&LengthOfStream; # tail call
  }
}

sub LengthOfStream ( $ ) {
  push @​_,0;
  goto \&LengthOfStream_;
}

But it has, of course, the ugly side effect of erasing the variable
being passed by the caller​:

my $s= SequenceStream 100000;
print LengthOfStream $s,"\n"; # -> 100001
# but now $s is undef

I find it strange, that this case also requires the undef $_[0] (and
@​_=() is not enough!)​:

print LengthOfStream SequenceStream 100000,"\n";

Any idea?

What I also don't understand is why this leaks memory and eventually
terminates because of an out of memory condition (it should just run
forever)​:

sub InfiniteStream {
  my ($startnumber)=@​_;
  #undef $_[0]; Modification of a read-only value attempted -- of course.
  #@​_=();
  Delay {
  Cons $startnumber, InfiniteStream($startnumber+1);
  };
}

LengthOfStream InfiniteStream(0);

Any idea here?

Christian.

@p5pRT
Copy link
Author

p5pRT commented Jul 28, 2007

From xmltwig@gmail.com

Created by xmltwig@gmail.com

I create a tree with a single root and thousands of children. The link
from the root to the first child is strong, as is the link from each
node to its next sibling. The link from each node to the root is weakened.

The program crashes when the root goes out of scope.

See the attached code which tries to find the exact number. Change the
$NB_ELT_INIT and the $STEP values to find it.

Note that the limit can vary (23790 and 23810 in my limited tests).
I had the same problem with XML​::Twig, which also builds a tree, at
about the same number of nodes. Twig uses weaken several times for each
node though, so that might be an element. Note that it is not the total
number of weaken that counts, just the number in the block in which the
variables are in scope.

This behaviour happens in Scalar​::Util 1.18 and 1.19
perl -v​:
This is perl, v5.8.8 built for i486-linux-gnu-thread-multi

I have also had reports of bugs in XML​::Twig that look suspiciously
like they could come from this problem, on various machines (just today​:
Solaris in c.l.p.m, perl 5.8.5 and 5.8.8, and perl v5.8.7 built for
i386-freebsd-64int by mail).

I hope that helps

--
mirod

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl v5.8.8:

Configured by Debian Project at Tue Mar  6 01:52:23 UTC 2007.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.15.7, archname=i486-linux-gnu-thread-multi
    uname='linux rothera 2.6.15.7 #1 smp sat sep 30 10:21:42 utc 2006
i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.8.8
-Dsitearch=/usr/local/lib/perl/5.8.8 -Dman1dir=/usr/share/man/man1
-Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
-Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm
-Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN
-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.1.2 (Ubuntu 4.1.2-0ubuntu4)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.5.so, so=so, useshrplib=true, libperl=libperl.so.5.8.8
    gnulibc_version='2.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:



@INC for perl v5.8.8:
    /etc/perl
    /usr/local/lib/perl/5.8.8
    /usr/local/share/perl/5.8.8
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    /usr/local/lib/perl/5.8.7
    /usr/local/share/perl/5.8.7
    .


Environment for perl v5.8.8:
    HOME=/home/mrodrigu
    LANG=en_US.UTF-8
    LANGUAGE=en_IT:en
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/mrodrigu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games:/home/mrodrigu/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Jul 28, 2007

@p5pRT
Copy link
Author

p5pRT commented Jul 29, 2007

From @andk

On Sat, 28 Jul 2007 11​:56​:28 -0700, "Michel Rodriguez" (via RT) <perlbug-followup@​perl.org> said​:

mr> The program crashes when the root goes out of scope.

mr> See the attached code which tries to find the exact number. Change the
mr> $NB_ELT_INIT and the $STEP values to find it.

Might be architecture dependent. I cannot reproduce under linux. I
increased $NB_ELT_INIT to 100000. Program ran to an end without a
crash with 5.6.2, 5.8.7, 5.8.8, maint-5.8, blead@​31663.

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Jul 29, 2007

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jul 29, 2007

From nospam-abuse@bloodgate.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash​: SHA1

Moin,

On Sunday 29 July 2007 11​:29​:42 Andreas J. Koenig wrote​:

On Sat, 28 Jul 2007 11​:56​:28 -0700, "Michel Rodriguez" (via RT)
<perlbug-followup@​perl.org> said​:

mr> The program crashes when the root goes out of scope.

mr> See the attached code which tries to find the exact number. Change
the mr> $NB_ELT_INIT and the $STEP values to find it.

Might be architecture dependent. I cannot reproduce under linux. I
increased $NB_ELT_INIT to 100000. Program ran to an end without a
crash with 5.6.2, 5.8.7, 5.8.8, maint-5.8, blead@​31663.

Note that the submitter had threads :) Reproducible here​:

  # perl bug.pl
  tree created
  tree out of scope OK (10000)
  tree created
  tree out of scope OK (15000)
  tree created
  tree out of scope OK (20000)
  tree created
  Segmentation fault

  # te@​linux​:~> perl -v

  This is perl, v5.8.8 built for x86_64-linux-thread-multi
 
  Copyright 1987-2006, Larry Wall

All the best,

Tels

Summary of my perl5 (revision 5 version 8 subversion 8) configuration​:
  Platform​:
  osname=linux, osvers=2.6.16, archname=x86_64-linux-thread-multi
  uname='linux dvorak 2.6.16 #1 smp mon apr 10 04​:51​:13 utc 2006 x86_64
x86_64 x86_64 gnulinux '
 
config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=true -Doptimize=-O2 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -g -Wall -pipe'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=define use64bitall=define uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags
='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
 
optimize='-O2 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -g -Wall -pipe',
 
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -Wdeclaration-after-statement'
  ccversion='', gccversion='4.1.0 (SUSE Linux)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib64'
  libpth=/lib64 /usr/lib64 /usr/local/lib64
  libs=-lm -ldl -lcrypt -lpthread
  perllibs=-lm -ldl -lcrypt -lpthread
  libc=/lib64/libc-2.4.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.4'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/x86_64-linux-thread-multi/CORE'
  cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING MULTIPLICITY PERL_IMPLICIT_CONTEXT
  PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_64_BIT_ALL
  USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
  USE_PERLIO USE_REENTRANT_API
  Built under linux
  Compiled at Apr 22 2006 23​:33​:01
  @​INC​:
  /usr/lib/perl5/5.8.8/x86_64-linux-thread-multi
  /usr/lib/perl5/5.8.8
  /usr/lib/perl5/site_perl/5.8.8/x86_64-linux-thread-multi
  /usr/lib/perl5/site_perl/5.8.8
  /usr/lib/perl5/site_perl
  /usr/lib/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi
  /usr/lib/perl5/vendor_perl/5.8.8
  /usr/lib/perl5/vendor_perl
  .

- --
Signed on Sun Jul 29 11​:36​:29 2007 with key 0x93B84C15.
View my photo gallery​: http​://bloodgate.com/photos
PGP key on http​://bloodgate.com/tels.asc or per email.

"Most people, I think, don't even know what a rootkit is, so why should
they care about it?"

  -- Thomas Hesse, President of Sony BMG's global digital business division,
2005.
-----BEGIN PGP SIGNATURE-----
Version​: GnuPG v1.4.2 (GNU/Linux)

iQEVAwUBRqxgDXcLPEOTuEwVAQIUggf+LZevVglAJVgseFy2lG46cmJ3Tu8una/V
RE9tFLSOWYAwNdj1RFspbmRT5qETWtkPGCfND4FpMnCna6Wv06BgdxlMli5RmJcT
ynnYya7c3Sat3jxF2tSk9yxIHphgr2TJ8IqB4M/9VmB10rugYcPBF9lU0jI2U0of
+MkbRQIHYEjLNJqEyr8BvXCnRE5OVrHJzkWOwy8kdlrsTFuFnzn6b8l+cz6FVk4H
pt819cijRz3QAqdlwfcJNwue0+cRdfKsCpkdoSsOR24KaUZXSFDVr5V47mhRCJio
f/UWTGWJrkPPFs5QLbvLEIOSXl0ZnlUhppfBBMTtbN1FaB7r9ssBtw==
=p8c9
-----END PGP SIGNATURE-----

@p5pRT
Copy link
Author

p5pRT commented Jul 29, 2007

From @andk

On Sun, 29 Jul 2007 11​:38​:21 +0200, Tels <nospam-abuse@​bloodgate.com> said​:

  > Note that the submitter had threads :)

I cannot reproduce with threaded perls either.

  > gnulibc_version='2.4'

I have 2.6 and this seems to be the key.

I can reproduce it with bleadperl@​31663 (without threads) on a box
with gnulibc 2.5. The stack trace on that other box says​:

#0 0x080e538b in Perl_hv_undef (hv=0x850c600) at hv.c​:1850
#1 0x0813363b in Perl_sv_clear (sv=0x850c600) at sv.c​:5156
#2 0x081341d9 in Perl_sv_free2 (sv=0x850c600) at sv.c​:5338
#3 0x0813413b in Perl_sv_free (sv=0x850c600) at sv.c​:5317
#4 0x08133f3f in Perl_sv_clear (sv=0x850c628) at sv.c​:5206
#5 0x081341d9 in Perl_sv_free2 (sv=0x850c628) at sv.c​:5338
#6 0x0813413b in Perl_sv_free (sv=0x850c628) at sv.c​:5317
#7 0x080e4866 in Perl_hv_free_ent (hv=0x850c5c4, entry=0x850b290) at hv.c​:1527
#8 0x080e51c2 in S_hfreeentries (hv=0x850c5c4) at hv.c​:1783
#9 0x080e5390 in Perl_hv_undef (hv=0x850c5c4) at hv.c​:1850
#10 0x0813363b in Perl_sv_clear (sv=0x850c5c4) at sv.c​:5156
#11 0x081341d9 in Perl_sv_free2 (sv=0x850c5c4) at sv.c​:5338
#12 0x0813413b in Perl_sv_free (sv=0x850c5c4) at sv.c​:5317
#13 0x08133f3f in Perl_sv_clear (sv=0x850c5ec) at sv.c​:5206
#14 0x081341d9 in Perl_sv_free2 (sv=0x850c5ec) at sv.c​:5338
#15 0x0813413b in Perl_sv_free (sv=0x850c5ec) at sv.c​:5317
#16 0x080e4866 in Perl_hv_free_ent (hv=0x850c588, entry=0x850b278) at hv.c​:1527
#17 0x080e51c2 in S_hfreeentries (hv=0x850c588) at hv.c​:1783
#18 0x080e5390 in Perl_hv_undef (hv=0x850c588) at hv.c​:1850
#19 0x0813363b in Perl_sv_clear (sv=0x850c588) at sv.c​:5156
#20 0x081341d9 in Perl_sv_free2 (sv=0x850c588) at sv.c​:5338
#21 0x0813413b in Perl_sv_free (sv=0x850c588) at sv.c​:5317
#22 0x08133f3f in Perl_sv_clear (sv=0x850c5b0) at sv.c​:5206
#23 0x081341d9 in Perl_sv_free2 (sv=0x850c5b0) at sv.c​:5338
#24 0x0813413b in Perl_sv_free (sv=0x850c5b0) at sv.c​:5317
#25 0x080e4866 in Perl_hv_free_ent (hv=0x850c54c, entry=0x850b260) at hv.c​:1527
#26 0x080e51c2 in S_hfreeentries (hv=0x850c54c) at hv.c​:1783
#27 0x080e5390 in Perl_hv_undef (hv=0x850c54c) at hv.c​:1850
#28 0x0813363b in Perl_sv_clear (sv=0x850c54c) at sv.c​:5156
#29 0x081341d9 in Perl_sv_free2 (sv=0x850c54c) at sv.c​:5338
#30 0x0813413b in Perl_sv_free (sv=0x850c54c) at sv.c​:5317
#31 0x08133f3f in Perl_sv_clear (sv=0x850c574) at sv.c​:5206
#32 0x081341d9 in Perl_sv_free2 (sv=0x850c574) at sv.c​:5338
#33 0x0813413b in Perl_sv_free (sv=0x850c574) at sv.c​:5317
#34 0x080e4866 in Perl_hv_free_ent (hv=0x850c510, entry=0x850b248) at hv.c​:1527

[da capo ad libitum]

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Jul 29, 2007

From blgl@hagernas.com

This looks like a stack overflow caused by the recursive implementation
of the reference counting scheme. You can evoke it with any sufficiently
long chain of references, no weakening required​:

  for (my $len=1; ; $len<<=1) {
  my $list;

  print STDERR "list length $len\n";
  foreach (1..$len) {
  $list={
  next => $list,
  };
  }
  }

/Bo Lindbergh

@p5pRT
Copy link
Author

p5pRT commented May 16, 2008

p5p@spam.wizbit.be - Status changed from 'open' to 'stalled'

@p5pRT
Copy link
Author

p5pRT commented May 19, 2011

From @iabyn

The issues related to perl seg-faulting when freeing deeply nested
structures
should now be resolved; first by the series of commits in oct 2010 that
made AVs and RVs non-recursive​:

  5239d5c
  df90f6a
  b98b62b

followed in May 2011 by the series of commits that made HVs
non-recursive too​:

  afbbf21
through
  c2217cd

@p5pRT
Copy link
Author

p5pRT commented May 19, 2011

@iabyn - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant