From abw@wardley.org

This is a bug report for perl from abw@​wardley.org
generated with the help of perlbug 1.39 running under perl 5.11.1.

I've got some code that's making Perl segfault (see attachment). A little
investigation suggests that the stack is being blown during the memory
cleanup of a large linked list. The av_undef(), av_clear(), sv_free2() and
sv_clear() functions call each other recursively and eventually exhaust the
stack space.

The bug is reliably reproducible on all the versions of Perl that I've
tried (5.8, 5.10.0, 5.10.1 and 5.11.1) and it doesn't appear to be platform
specific (confirmed on OSX and Linux).

More detail follows...

I'm creating a linked list using array references as nodes. The first element
in the array ref contains some data, the second item contains a reference to
the next item. Think "cons" lists.

  while (++$n < $max) {
  $token = ["token $n"];
  $last->[1] = $token if $last;
  $last = $token;
  }

Using the above code I can create a linked list of 100 million nodes and
everything works just fine (assuming you don't mind twiddling your thumbs
for a few minutes).

However, if I also stuff the nodes into a container list then Perl segfaults
at cleanup time, either when the tokens go out of scope or during global
cleanup.

  while (++$n < $max) {
  $token = ["token $n"];
  $last->[1] = $token if $last;
  $last = $token;
  push(@​tokens, $token); # add this line -> BOOM!
  }

In this case Perl will reliably segfault with a mere 30,000 nodes.

If I just push them onto @​tokens and don't create the linked list then it
also works fine. It's the combination of linked list + container list that
farks things up.

A ready-to-run version of the test script is attached. Also available here​:

  http​://wardley.org/perl/linked_list_segfault.pl

I tested this on my Macbook using versions 5.8, 5.10.0, 5.10.1 and 5.11.1,
and 5.10 on Linux. They all fail somewhere between the 25k and 40k mark.

Now convinced that this was a bug in Perl, I enlisted the help of London.pm
to try and track the problem down. The relevant responses are included below.

On 04/11/2009 10​:45, Matthew Boyle wrote​:

mine seems to be running out of stack space​:

[chemnitz@​10​:36 ~]$ gdb perl
[...]
(gdb) run downloads/linked_list_segfault.pl
Starting program​: /usr/bin/perl downloads/linked_list_segfault.pl

[snip]

starting
creating linked list of 40000 tokens
created 40000 tokens
creating linked list of 40000 tokens in container
created 40000 tokens
[New Thread 0xb7fe26c0 (LWP 9178)]

Program received signal SIGSEGV, Segmentation fault.
0x026e6105 in Perl_sv_free2 () from
/usr/lib/perl5/5.10.0/i386-linux-thread-multi/CORE/libperl.so
(gdb) bt -1
#229310 0x08048a2e in main ()
(gdb)

quarter of a million frames is quite a lot :-)

On 04/11/2009 10​:55, Matthew Boyle wrote​:

looks like it is a stack issue​:

[chemnitz@​10​:52 ~]$ ulimit -s
10240
[chemnitz@​10​:53 ~]$ perl downloads/linked_list_segfault.pl
# segfaults
[chemnitz@​10​:53 ~]$ ulimit -s 20480
[chemnitz@​10​:53 ~]$ perl downloads/linked_list_segfault.pl
# is fine
[chemnitz@​10​:53 ~]$ perl downloads/linked_list_segfault.pl 80000
# segfaults again

On 04/11/2009 10​:46, Dagfinn Ilmari Mannsåker wrote​:

It seems to be recursing when freeing @​tokens, I'm see the following
stacktrace​:

[fuckloads of reapeating calls snipped]
#192663 0x0000000000492bcc in Perl_sv_clear (my_perl=0x78a010, sv=0x7b5a80)
at sv.c​:5236
#192664 0x00000000004931a1 in Perl_sv_free2 (my_perl=0x78a010, sv=0x7b5a80)
at sv.c​:5368
#192665 0x00000000004719e8 in Perl_av_undef (my_perl=0x78a010, av=0x78ee88)
at av.c​:485
#192666 0x0000000000492c38 in Perl_sv_clear (my_perl=0x78a010, sv=0x78ee88)
at sv.c​:5193
#192667 0x00000000004931a1 in Perl_sv_free2 (my_perl=0x78a010, sv=0x78ee88)
at sv.c​:5368
#192668 0x0000000000492bcc in Perl_sv_clear (my_perl=0x78a010, sv=0x78ecd8)
at sv.c​:5236
#192669 0x00000000004931a1 in Perl_sv_free2 (my_perl=0x78a010, sv=0x78ecd8)
at sv.c​:5368
#192670 0x00000000004700e8 in Perl_av_clear (my_perl=0x78a010, av=0x7d65b8)
at av.c​:453
#192671 0x00000000004b5ca6 in Perl_leave_scope (my_perl=0x78a010, base=5)
at scope.c​:799
#192672 0x000000000047e199 in Perl_pp_leavesub (my_perl=0x78a010) at
pp_hot.c​:2475
#192673 0x0000000000454ce4 in Perl_runops_debug (my_perl=0x78a010) at
dump.c​:1931
#192674 0x0000000000478aa4 in S_run_body (my_perl=<value optimised out>) at
perl.c​:2391
#192675 perl_run (my_perl=<value optimised out>) at perl.c​:2309
#192676 0x000000000042177c in main (argc=2, argv=0x7fffffffe198,
env=0x7fffffffe1b0) at perlmain.c​:113

I think it's time to take this to p5p.

Flags​:
  category=core
  severity=medium

Site configuration information for perl 5.11.1​:

Configured by abw at Tue Nov 3 19​:06​:07 GMT 2009.

Summary of my perl5 (revision 5 version 11 subversion 1) configuration​:

  Platform​:
  osname=darwin, osvers=9.8.0, archname=darwin-2level
  uname='darwin shiny 9.8.0 darwin kernel version 9.8.0​: wed jul 15
16​:55​:01 pdt 2009; root​:xnu-1228.15.4~1release_i386 i386 '
  config_args='-Dusedevel -Dprefix=/usr/local/perl/5.11.1 -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -no-cpp-precomp
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include',
  optimize='-O3',
  cppflags='-no-cpp-precomp -fno-common -DPERL_DARWIN -no-cpp-precomp
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include'
  ccversion='', gccversion='4.0.1 (Apple Inc. build 5465)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector
-L/usr/local/lib -L/opt/local/lib'
  libpth=/usr/local/lib /opt/local/lib /usr/lib
  libs=-ldbm -ldl -lm -lutil -lc
  perllibs=-ldl -lm -lutil -lc
  libc=/usr/lib/libc.dylib, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-L/usr/local/lib -L/opt/local/lib -fstack-protector'

Locally applied patches​:

@​INC for perl 5.11.1​:
  /usr/local/perl/5.11.1/lib/site_perl/5.11.1/darwin-2level
  /usr/local/perl/5.11.1/lib/site_perl/5.11.1
  /usr/local/perl/5.11.1/lib/5.11.1/darwin-2level
  /usr/local/perl/5.11.1/lib/5.11.1
  .

Environment for perl 5.11.1​:
  DYLD_LIBRARY_PATH=/usr/local/ImageMagick/lib
  HOME=/Users/abw
  LANG=en_GB.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)

PATH=/usr/local/git/bin​:/usr/bin​:/bin​:/usr/sbin​:/sbin​:/usr/local/bin​:/usr/X11/bin​:/usr/local/git/bin​:/usr/local/bin​:/opt/local/bin​:/opt/local/sbin​:/Users/abw/bin​:/usr/local/ImageMagick/bin​:.
  PERL_BADLANG (unset)
  SHELL=/bin/bash

From abw@wardley.org

linked_list_segfault.pl

abw@wardley.org - Status changed from 'new' to 'open'

From zefram@fysh.org

Andy Wardley wrote​:

I've got some code that's making Perl segfault (see attachment). A little
investigation suggests that the stack is being blown during the memory
cleanup of a large linked list.

I have a fairly clear idea of how to fix this. I'll look into it.

-zefram

From andrew@pimlott.net

Created by andrew@pimlott.net

The following seg faults after printing "built"​:

  my $h = {};
  for (1 .. 100000) {
  $h = { h => $h };
  }
  print "built\n";
  $h = undef;
  print "gone\n";

I originally encountered this with a data structure built by XML​::Twig, but
I think the root problem is the same​: perl's garbage collector uses the C
stack to recur into data structures. Here's a backtrace​:

#0 0x00000000004a7942 in Perl_sv_clear (my_perl=0xe9a010, sv=0x1d2b8f8)
  at sv.c​:5391
#1 0x00000000004a8222 in Perl_sv_free2 (my_perl=0xe9a010, sv=0x1d2b8f8)
  at sv.c​:5694
#2 0x00000000004a7d77 in Perl_sv_clear (my_perl=0xe9a010, sv=0x1d2b940)
  at sv.c​:5551
#3 0x00000000004a8222 in Perl_sv_free2 (my_perl=0xe9a010, sv=0x1d2b940)
  at sv.c​:5694
#4 0x000000000048d011 in Perl_hv_free_ent (my_perl=0xe9a010, hv=0x1d2b928,
  entry=0x1d28618) at hv.c​:1473
#5 0x000000000048e261 in S_hfreeentries (my_perl=0xe9a010, hv=0x1d2b928)
  at hv.c​:1749
#6 0x000000000049020b in Perl_hv_undef (my_perl=0xe9a010, hv=0x1d2b928)
  at hv.c​:1816
#7 0x00000000004a7f73 in Perl_sv_clear (my_perl=0xe9a010, sv=0x1d2b928)
  at sv.c​:5500
...

(Line numbers are from the Debian perl_5.10.1-12 source package.)

The 7-step cycle repeats over and over​:

  Perl_sv_clear ->
  Perl_hv_undef ->
  S_hfreeentries ->
  Perl_hv_free_ent ->
  Perl_sv_free2 ->
  Perl_sv_clear ->
  Perl_sv_free2 -> Perl_sv_clear

Is there any way around this? It's a shame that perl can manipulate a large
data structure without sweating, then breaks down when you're finished with
it.

Andrew

Flags:
    category=core
    severity=low

Site configuration information for perl 5.10.0:

Configured by Debian Project at Fri Aug 28 22:23:22 UTC 2009.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.30.5-dsa-amd64, archname=x86_64-linux-gnu-thread-multi
    uname='linux brahms 2.6.30.5-dsa-amd64 #1 smp mon aug 17 02:18:43 cest 2009 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.3.2', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0
    gnulibc_version='2.7'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.10.0:
    /home/andrew/stable/perl
    /home/andrew/lib/perl
    /home/andrew/local/lib/perl
    /etc/perl
    /usr/local/lib/perl/5.10.0
    /usr/local/share/perl/5.10.0
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.10
    /usr/share/perl/5.10
    /usr/local/lib/site_perl
    .


Environment for perl 5.10.0:
    HOME=/home/andrew
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/andrew/stable/bin:/home/andrew/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
    PERL5LIB=/home/andrew/stable/perl:/home/andrew/lib/perl:/home/andrew/local/lib/perl
    PERLDB_OPTS=HistFile=/home/andrew/.perldb.history HistSize=1000
    PERL_BADLANG (unset)
    SHELL=/bin/zsh

From @avar

On Wed, May 5, 2010 at 00​:03, andrew@​pimlott.net
<perlbug-followup@​perl.org> wrote​:

   my $h = {};
   for (1 .. 100000) {
       $h = { h => $h };
   }
   print "built\n";
   $h = undef;
   print "gone\n";

It also segfaults on blead (89abef2).

The RT System itself - Status changed from 'new' to 'open'

From zefram@fysh.org

andrew@​pimlott.net wrote​:

The following seg faults after printing "built"​:

It's essentially a duplicate of [perl #70253].

-zefram

From andrew@pimlott.net

Zefram wrote​:

It's essentially a duplicate of [perl #70253].

Do you intend to fix the garbage collector to use its own stack? Has it
always worked this way? I was pretty shocked, especially since I know
perl doesn't use the C stack for Perl function calls, nor does it allow
recursion on deep data structures in Perl code. But I'm glad to hear
your looking at the problem.

Andrew

From @cpansprout

This was resolved in 5.14.0 as part of the fix for #44225.

--

Father Chrysostomos

@cpansprout - Status changed from 'open' to 'resolved'