Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data::Dumper -- dumping large hash returns empty string, segfaults perl #10292

Closed
p5pRT opened this issue Apr 9, 2010 · 17 comments
Closed

Data::Dumper -- dumping large hash returns empty string, segfaults perl #10292

p5pRT opened this issue Apr 9, 2010 · 17 comments

Comments

@p5pRT
Copy link

p5pRT commented Apr 9, 2010

Migrated from rt.perl.org#74170 (status was 'resolved')

Searchable as RT74170$

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2010

From @sitaramc

Created by @sitaramc

I'm the author of a little tool called "gitolite" which is basically an access
control wrapper around git.

The basic mode of operation of gitolite is that whenever the ACL rules change,
we "compile" them into a hash that gets Data​::Dumper-ed. [More details
available on request].

This dumping seems to fail if the hash is too large and you have a custom sort
sub. One work around seems to be a very unusual piece of code, that actually
does nothing meaningful but does seem to make the Dumper work!

  for my $key (sort keys %repos) {
  my @​wtf = sort keys %{ $repos{$key} };
  }

Finally, this seems to happen only when the program builds up the hash step by
step; if you load the hash in one shot it doesn't happen! Of course in my
application this is not a useful option (I have to build the hash as I parse
the ACL rules line by line!)

http​://github.com/sitaramc/gitolite/raw/temp-br--data-dumper-problem-demo/data-dumper-returns-undef.pl
is a completely standalone program I wrote (based on recording the actual
sequence of operations that a real run takes) to demonstrate the problem. The
top 20 or so lines tell you what it's all about.

Perl Info

Flags:
    category=core
    severity=low

This perlbug was built using Perl 5.10.0 in the Fedora build system.
It is being executed now by Perl 5.10.0 - Thu Oct 23 11:13:18 EDT 2008.

Site configuration information for perl 5.10.0:

Configured by Red Hat, Inc. at Thu Oct 23 11:13:18 EDT 2008.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.18-92.1.10.el5, archname=i386-linux-thread-multi
    uname='linux x86-4.fedora.phx.redhat.com 2.6.18-92.1.10.el5 #1 smp wed jul 23 03:56:11 edt 2008 i686 i686 i386 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -DPERL_USE_SAFE_PUTENV -Dversion=5.10.0 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Dprivlib=/usr/lib/perl5/5.10.0 -Dsitelib=/usr/local/lib/perl5/site_perl/5.10.0 -Dvendorlib=/usr/lib/perl5/vendor_perl/5.10.0 -Darchlib=/usr/lib/perl5/5.10.0/i386-linux-thread-multi -Dsitearch=/usr/local/lib/perl5/site_perl/5.10.0/i386-linux-thread-multi -Dvendorarch=/usr/lib/perl5/vendor_perl/5.10.0/i386-linux-thread-multi -Darchname=i386-linux-thread-multi -Dotherlibdirs=/usr/local/lib/perl5/site_perl:/usr/lib/perl5/site_perl -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinst!
 allusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -DPERL_USE_SAFE_PUTENV',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='4.3.0 20080428 (Red Hat 4.3.0-8)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.8.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.8'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.10.0/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -DPERL_USE_SAFE_PUTENV -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.10.0:
    /usr/lib/perl5/5.10.0/i386-linux-thread-multi
    /usr/lib/perl5/5.10.0
    /usr/local/lib/perl5/site_perl/5.10.0/i386-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/lib/perl5/vendor_perl/5.10.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.10.0
    /usr/lib/perl5/vendor_perl
    /usr/local/lib/perl5/site_perl/5.10.0/i386-linux-thread-multi
    /usr/local/lib/perl5/site_perl/5.10.0
    /usr/local/lib/perl5/site_perl
    /usr/lib/perl5/site_perl
    .


Environment for perl 5.10.0:
    HOME=/home/sitaram
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/sitaram/._s/bin:/home/sitaram/bin:/home/sitaram/._s/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2010

From @avar

Thanks for the report. I'm attaching a copy of your test script for reference.

I can confirm that this happens in 5.10.1 and blead. Perl segfaults in
do_clean_objs​:

  (gdb) bt full
  #0 0x081aa5d0 in do_clean_objs (my_perl=0x83f7008, ref=0x83fc9c4)
at sv.c​:489
  target = 0x0
  __PRETTY_FUNCTION__ = "do_clean_objs"
  #1 0x081aa378 in S_visit (my_perl=0x83f7008, f=0x81aa413
<do_clean_objs>, flags=2048, mask=2048) at sv.c​:440
  svend = 0x8d810fa4
  sv = 0x83fc9c4
  sva = 0x83fc5e4
  visited = 0
  __PRETTY_FUNCTION__ = "S_visit"
  #2 0x081ab3d2 in Perl_sv_clean_objs (my_perl=0x83f7008) at sv.c​:548
  No locals.
  #3 0x0808c1e1 in perl_destruct (my_perl=0x83f7008) at perl.c​:770
  destruct_level = 0 '\000'
  hv = 0x0
  __PRETTY_FUNCTION__ = "perl_destruct"
  #4 0x08064942 in main (argc=2, argv=0xbffff2f4, env=0xbffff300)
at perlmain.c​:126
  exitstatus = 0
  i = 69

At that point Perl is working with a scalar that seems corrupted. It
claims to be a reference but isn't​:

  (gdb) p SvFLAGS(ref) & SVf_ROK
  $14 = 2048

(Equivalent to SvRV() but gdb doesn't grok it)​:

  (gdb) p ref->sv_u.svu_rv
  $16 = (SV *) 0x0

Or in another session​:

  (gdb) p ref->sv_u.svu_rv
  $5 = (SV *) 0x1

Or​:

  (gdb) p target
  $1 = (SV * const) 0xabababab

Which then segfaults on​:

  SvOBJECT(SvRV(ref))

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2010

From @avar

data-dumper-returns-undef.pl

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2010

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2010

From @avar

On Fri, Apr 9, 2010 at 15​:31, Ævar Arnfjörð Bjarmason <avarab@​gmail.com> wrote​:

I can confirm that this happens in 5.10.1 and blead.

It's OK in 5.8.9, but bisecting below 5.10.0 is too much of a PITA due
to random build failures that I couldn't find out what commit broke
this.

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2010

From @avar

On Fri, Apr 9, 2010 at 17​:13, Ævar Arnfjörð Bjarmason <avarab@​gmail.com> wrote​:

On Fri, Apr 9, 2010 at 15​:31, Ævar Arnfjörð Bjarmason <avarab@​gmail.com> wrote​:

I can confirm that this happens in 5.10.1 and blead.

It's OK in 5.8.9, but bisecting below 5.10.0 is too much of a PITA due
to random build failures that I  couldn't find out what commit broke
this.

Fixed those​:

d7aa538 is the first bad commit
commit d7aa538
Author​: John Peacock <jpeacock@​rowman.com>
Date​: Tue Aug 3 18​:23​:57 2004 -0400

  Final version object core patch?
  Message-ID​: <411048BD.3080700@​rowman.com>

  p4raw-id​: //depot/perl@​23190

:100644 100644 2c6641d6f6a9947812cd4c2b3801159ae717ba92
d9d16ed0a47cbdc4eb9bb501991a78c3b09d40d6 M gv.c
:100644 100644 4415d8d9259727cbadc0952fee6728a023eac9fd
4af4e06e91c7d6f0789787e2e80be9b69e2b25ee M perl.c
:100644 100644 7fd4c4ed2d8478c3ddf2b2ba3ac666402af16bbf
4ba11712f1993b40d42eb7e1a1e2613da8f9cc62 M pp_ctl.c
:100644 100644 2cdebd66dc1a0db5bf052d74f480b786bd850964
e71c03c68b4dceadd77670336641c644af90ce28 M sv.c
:040000 040000 e61d2627813019b6f03824a9c01249f67de67e8e
8ea962a860cfa956e86e60a06eed9f0bc0f6d0b4 M t
:100644 100644 02d65a683acfda0eb8fdef9e604495368152c2d3
8d4c13e0a197d44eb68023b10dda44aa12f0b597 M util.c

@p5pRT
Copy link
Author

p5pRT commented May 17, 2010

From @cpansprout

Data_Dumper_Dumpxs is missing a SPAGAIN after the call to DD_dump. It
needs it, as DD_Dump might call a custom sort routine, which might
reallocate the stack. This patch adds it, along with the test which
I’ve reduced to half the size, yet put in its own file since it’s
still big.

@p5pRT
Copy link
Author

p5pRT commented May 17, 2010

From @cpansprout

Inline Patch
diff -Nurp blead/dist/Data-Dumper/Dumper.xs blead-74170/dist/Data-Dumper/Dumper.xs
--- blead/dist/Data-Dumper/Dumper.xs	2010-04-15 12:45:10.000000000 -0700
+++ blead-74170/dist/Data-Dumper/Dumper.xs	2010-05-12 22:31:57.000000000 -0700
@@ -1192,6 +1192,10 @@ Data_Dumper_Dumpxs(href, ...)
 			    postav, &level, indent, pad, xpad, newapad, sep, pair,
 			    freezer, toaster, purity, deepcopy, quotekeys,
 			    bless, maxdepth, sortkeys);
+		    /* DD_dump might have called a custom sort routine that
+		       might have reallocated the stack (see
+		       [perl #74170]). */
+		    SPAGAIN;
 		
 		    if (indent >= 2 && !terse)
 			SvREFCNT_dec(newapad);
diff -Nurp blead/dist/Data-Dumper/t/perl-74170.t blead-74170/dist/Data-Dumper/t/perl-74170.t
--- blead/dist/Data-Dumper/t/perl-74170.t	1969-12-31 16:00:00.000000000 -0800
+++ blead-74170/dist/Data-Dumper/t/perl-74170.t	2010-05-12 22:25:45.000000000 -0700
@@ -0,0 +1,143 @@
+#!perl -X
+#
+# Regression test for [perl #74170] (missing SPAGAIN after DD_Dump(...)):
+# Since it’s so large, it gets its own file.
+
+BEGIN {
+    require Config; import Config;
+    no warnings 'once';
+    if ($Config{'extensions'} !~ /\bData\/Dumper\b/) {
+	print "1..0 # Skip: Data::Dumper was not built\n";
+	exit 0;
+    }
+}
+
+use strict;
+use Test::More tests => 1;
+use Data::Dumper;
+
+our %repos = ();
+&real_life_setup();
+
+$Data::Dumper::Indent = 1;
+# A custom sort sub is necessary for reproducing the bug, as this is where
+# the stack gets reallocated.
+$Data::Dumper::Sortkeys = sub { return [ reverse sort keys %{$_[0]} ]; }
+    unless exists $ENV{NO_SORT_SUB};
+
+ok +Data::Dumper->Dumpxs([\%repos], [qw(*repos)]);
+
+sub real_life_setup {
+    # set up the %repos hash in a manner that reflects a real run of
+    # gitolite's "compiler" script:
+    # Yes, all this is necessary to get the stack in such a state that the
+    # custom sort sub will trigger a reallocation.
+    push @{ $repos{''}{'@all'} }, ();
+    push @{ $repos{''}{'guser86'} }, ();
+    push @{ $repos{''}{'guser87'} }, ();
+    push @{ $repos{''}{'user88'} }, ();
+    push @{ $repos{''}{'grussell'} }, ();
+    push @{ $repos{''}{'guser0'} }, ();
+    push @{ $repos{''}{'guser1'} }, ();
+    push @{ $repos{''}{'guser10'} }, ();
+    push @{ $repos{''}{'guser11'} }, ();
+    push @{ $repos{''}{'guser12'} }, ();
+    push @{ $repos{''}{'guser13'} }, ();
+    push @{ $repos{''}{'guser14'} }, ();
+    push @{ $repos{''}{'guser15'} }, ();
+    push @{ $repos{''}{'guser16'} }, ();
+    push @{ $repos{''}{'guser17'} }, ();
+    push @{ $repos{''}{'guser18'} }, ();
+    push @{ $repos{''}{'guser19'} }, ();
+    push @{ $repos{''}{'guser2'} }, ();
+    push @{ $repos{''}{'guser20'} }, ();
+    push @{ $repos{''}{'guser21'} }, ();
+    push @{ $repos{''}{'guser22'} }, ();
+    push @{ $repos{''}{'guser23'} }, ();
+    push @{ $repos{''}{'guser24'} }, ();
+    push @{ $repos{''}{'guser25'} }, ();
+    push @{ $repos{''}{'guser26'} }, ();
+    push @{ $repos{''}{'guser27'} }, ();
+    push @{ $repos{''}{'guser28'} }, ();
+    push @{ $repos{''}{'guser29'} }, ();
+    push @{ $repos{''}{'guser3'} }, ();
+    push @{ $repos{''}{'guser30'} }, ();
+    push @{ $repos{''}{'guser31'} }, ();
+    push @{ $repos{''}{'guser32'} }, ();
+    push @{ $repos{''}{'guser33'} }, ();
+    push @{ $repos{''}{'guser34'} }, ();
+    push @{ $repos{''}{'guser35'} }, ();
+    push @{ $repos{''}{'guser36'} }, ();
+    push @{ $repos{''}{'guser37'} }, ();
+    push @{ $repos{''}{'guser38'} }, ();
+    push @{ $repos{''}{'guser39'} }, ();
+    push @{ $repos{''}{'guser4'} }, ();
+    push @{ $repos{''}{'guser40'} }, ();
+    push @{ $repos{''}{'guser41'} }, ();
+    push @{ $repos{''}{'guser42'} }, ();
+    push @{ $repos{''}{'guser43'} }, ();
+    push @{ $repos{''}{'guser44'} }, ();
+    push @{ $repos{''}{'guser45'} }, ();
+    push @{ $repos{''}{'guser46'} }, ();
+    push @{ $repos{''}{'guser47'} }, ();
+    push @{ $repos{''}{'guser48'} }, ();
+    push @{ $repos{''}{'guser49'} }, ();
+    push @{ $repos{''}{'guser5'} }, ();
+    push @{ $repos{''}{'guser50'} }, ();
+    push @{ $repos{''}{'guser51'} }, ();
+    push @{ $repos{''}{'guser52'} }, ();
+    push @{ $repos{''}{'guser53'} }, ();
+    push @{ $repos{''}{'guser54'} }, ();
+    push @{ $repos{''}{'guser55'} }, ();
+    push @{ $repos{''}{'guser56'} }, ();
+    push @{ $repos{''}{'guser57'} }, ();
+    push @{ $repos{''}{'guser58'} }, ();
+    push @{ $repos{''}{'guser59'} }, ();
+    push @{ $repos{''}{'guser6'} }, ();
+    push @{ $repos{''}{'guser60'} }, ();
+    push @{ $repos{''}{'guser61'} }, ();
+    push @{ $repos{''}{'guser62'} }, ();
+    push @{ $repos{''}{'guser63'} }, ();
+    push @{ $repos{''}{'guser64'} }, ();
+    push @{ $repos{''}{'guser65'} }, ();
+    push @{ $repos{''}{'guser66'} }, ();
+    push @{ $repos{''}{'guser67'} }, ();
+    push @{ $repos{''}{'guser68'} }, ();
+    push @{ $repos{''}{'guser69'} }, ();
+    push @{ $repos{''}{'guser7'} }, ();
+    push @{ $repos{''}{'guser70'} }, ();
+    push @{ $repos{''}{'guser71'} }, ();
+    push @{ $repos{''}{'guser72'} }, ();
+    push @{ $repos{''}{'guser73'} }, ();
+    push @{ $repos{''}{'guser74'} }, ();
+    push @{ $repos{''}{'guser75'} }, ();
+    push @{ $repos{''}{'guser76'} }, ();
+    push @{ $repos{''}{'guser77'} }, ();
+    push @{ $repos{''}{'guser78'} }, ();
+    push @{ $repos{''}{'guser79'} }, ();
+    push @{ $repos{''}{'guser8'} }, ();
+    push @{ $repos{''}{'guser80'} }, ();
+    push @{ $repos{''}{'guser81'} }, ();
+    push @{ $repos{''}{'guser82'} }, ();
+    push @{ $repos{''}{'guser83'} }, ();
+    push @{ $repos{''}{'guser84'} }, ();
+    push @{ $repos{''}{'guser85'} }, ();
+    push @{ $repos{''}{'guser9'} }, ();
+    push @{ $repos{''}{'user1'} }, ();
+    push @{ $repos{''}{'user10'} }, ();
+    push @{ $repos{''}{'user11'} }, ();
+    push @{ $repos{''}{'user12'} }, ();
+    push @{ $repos{''}{'user13'} }, ();
+    push @{ $repos{''}{'user14'} }, ();
+    push @{ $repos{''}{'user15'} }, ();
+    push @{ $repos{''}{'user16'} }, ();
+    push @{ $repos{''}{'user2'} }, ();
+    push @{ $repos{''}{'user3'} }, ();
+    push @{ $repos{''}{'user4'} }, ();
+    push @{ $repos{''}{'user5'} }, ();
+    push @{ $repos{''}{'user6'} }, ();
+    push @{ $repos{''}{'user7'} }, ();
+    $repos{''}{R}{'user8'} = 1;
+    $repos{''}{W}{'user8'} = 1;
+    push @{ $repos{''}{'user8'} }, ();
+}
--- blead/MANIFEST	2010-04-26 02:44:11.000000000 -0700
+++ blead-74170/MANIFEST	2010-05-12 22:30:08.000000000 -0700
@@ -2614,6 +2614,7 @@ dist/Data-Dumper/t/freezer.t	See if $Dat
 dist/Data-Dumper/Todo		Data pretty printer, futures
 dist/Data-Dumper/t/overload.t	See if Data::Dumper works for overloaded data
 dist/Data-Dumper/t/pair.t	See if Data::Dumper pair separator works
+dist/Data-Dumper/t/perl-74170.t	Regression test for [perl #74170]
 dist/Data-Dumper/t/terse.t	See if Data::Dumper terse option works
 dist/ExtUtils-Install/Changes				ExtUtils-Install change log
 dist/ExtUtils-Install/lib/ExtUtils/Installed.pm		Information on installed extensions

@p5pRT
Copy link
Author

p5pRT commented May 17, 2010

From @tsee

Hi,

Father Chrysostomos wrote​:

Data_Dumper_Dumpxs is missing a SPAGAIN after the call to DD_dump. It
needs it, as DD_Dump might call a custom sort routine, which might
reallocate the stack. This patch adds it, along with the test which I’ve
reduced to half the size, yet put in its own file since it’s still big.

thanks for the patch! Unfortunately, I see some test failures when
running core tests with this change​:

Test Summary Report


op/stash.t (Wstat​:
0 Tests​: 31 Failed​: 0)
  TODO passed​: 26

../cpan/CGI/t/http.t (Wstat​:
256 Tests​: 7 Failed​: 1)
  Failed test​: 7

  Non-zero exit status​: 1

../cpan/ExtUtils-MakeMaker/t/min_perl_version.t (Wstat​:
256 Tests​: 33 Failed​: 1)
  Failed test​: 17

  Non-zero exit status​: 1

../cpan/ExtUtils-MakeMaker/t/prereq_print.t (Wstat​:
256 Tests​: 11 Failed​: 1)
  Failed test​: 5

  Non-zero exit status​: 1

../cpan/Test-Simple/t/explain.t (Wstat​:
512 Tests​: 5 Failed​: 2)
  Failed tests​: 4-5

  Non-zero exit status​: 2

Files=1806, Tests=358509, 437 wallclock secs (100.08 usr 7.54 sys +
742.75 cusr 47.70 csys = 898.07 CPU)
Result​: FAIL

The CGI failure is not relevant (CGI.pm test bug related to environment).

The Test​::Simple one reads​:

===( 109147;321 2298/? 1/5 0/? )====================================

# Failed test at t/explain.t line 25.

# Structures begin differing at​:

# $got->[0] = undef

# $expected->[0] = ARRAY(0x2299e40)

# Failed test at t/explain.t line 27.
# Structures begin differing at​:
# $got->[1] = undef
# $expected->[1] = ARRAY(0x2299a38)
# Looks like you failed 2 tests of 5.
../cpan/Test-Simple/t/explain.t ...................................
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/5 subtests

MakeMaker​:
===( 22881;210 1/11 1/912 0/? )=================================

# Failed test 'prereqs dumped'

# at t/prereq_print.t line 53.

# Structures begin differing at​:

# $got = undef

# $expected = HASH(0x28c5478)

===( 22895;210 8/11 7/912 1/13
)===============================# Looks like you failed 1 test of 11.

../cpan/ExtUtils-MakeMaker/t/prereq_print.t .......................
Dubious, test returned 1 (wstat 256, 0x100)

Failed 1/11 subtests

more of the same​:
===( 17665;198 5/8 15/33 1/19 )===================================

# Failed test ' and talking like we expect'

# at t/min_perl_version.t line 120.

# $BUILD_REQUIRES = {};

../cpan/IO-Compress/t/010examples-bzip2.t ......................... ok

===( 17695;198 6/8 26/33
)==========================================# Looks like you failed 1
test of 33.

../cpan/ExtUtils-MakeMaker/t/min_perl_version.t ...................
Dubious, test returned 1 (wstat 256, 0x100)

Failed 1/33 subtests

  (less 3 skipped subtests​: 29 okay)

Before applying the patch, these all passed (minus CGI.pm).

Could you have a look at these?

Best regards,
Steffen

@p5pRT
Copy link
Author

p5pRT commented May 17, 2010

From @tonycoz

On Sun May 16 18​:03​:05 2010, sprout@​cpan.org wrote​:

Data_Dumper_Dumpxs is missing a SPAGAIN after the call to DD_dump. It
needs it, as DD_Dump might call a custom sort routine, which might
reallocate the stack. This patch adds it, along with the test which
I’ve reduced to half the size, yet put in its own file since it’s
still big.

When applied to blead several tests that otherwise pass, fail​:

Script started on Mon 17 May 2010 22​:07​:20 EST
tony@​mars​:.../perl/t$ ./perl harness -v
../cpan/ExtUtils-MakeMaker/t/min_perl_version.t
../cpan/ExtUtils-MakeMaker/t/prereq_print.t ../cpan/Test-Simple/t/explain.t
../cpan/ExtUtils-MakeMaker/t/min_perl_version.t ..
1..33
ok 1 - setup
ok 2 - entering dir Min-PerlVers
ok 3 - capturing stdout
ok 4 - MIN_PERL_VERSION=5 does not trigger a warning
ok 5 - nor a hard failure
ok 6 - MIN_PERL_VERSION=X.Y.Z does not trigger a warning
ok 7 - nor a hard failure
ok 8 - MIN_PERL_VERSION=999999 triggers a warning
ok 9 - with expected message text
ok 10 - and without a hard failure
ok 11 - MIN_PERL_VERSION=999999 and PREREQ_FATAL​: no warning
ok 12 - correct exception
ok 13 - MIN_PERL_VERSION=foobar triggers a warning
ok 14 - with expected message text
ok 15 - and without a hard failure

ok 16 - PREREQ_PRINT exiting normally
not ok 17 - and talking like we expect
# Failed test ' and talking like we expect'
# at t/min_perl_version.t line 120.
# $BUILD_REQUIRES = {};
ok 18 # skip not going to evaluate rubbish
ok 19 # skip not going to evaluate rubbish
ok 20 # skip not going to evaluate rubbish
ok 21 - PRINT_PREREQ exiting normally
ok 22 - and not complaining loudly
ok 23 - dump has prereqs and perl version
ok 24 - Makefile.PL exiting normally
ok 25 - Makefile present
ok 26 - Make ppd exiting normally
ok 27 - .ppd file present
ok 28 - .ppd file content good
# Looks like you failed 1 test of 33.
ok 29 - Make metafile exiting normally
ok 30 - META.yml present
ok 31 - META.yml content good
ok 32 - leaving dir
ok 33 - teardown
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/33 subtests
  (less 3 skipped subtests​: 29 okay)
../cpan/ExtUtils-MakeMaker/t/prereq_print.t ......
1..11
ok 1 - setup
ok 2 - chdir'd to Big-Dummy

# Failed test 'prereqs dumped'
# at t/prereq_print.t line 53.
# Structures begin differing at​:
# $got = undef
# $expected = HASH(0x1d49688)
ok 3 - PREREQ_PRINT produces no Makefile
ok 4 - exited normally
not ok 5 - prereqs dumped
ok 6 - without error
# Looks like you failed 1 test of 11.
ok 7 - PRINT_PREREQ produces no Makefile
ok 8 - exited normally
ok 9 - prereqs dumped
ok 10
ok 11 - teardown
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/11 subtests
../cpan/Test-Simple/t/explain.t ..................
1..5
ok 1 - main->can('explain')
ok 2
ok 3

# Failed test at t/explain.t line 25.
not ok 4# Structures begin differing at​:
# $got->[0] = undef
# $expected->[0] = ARRAY(0x1aeb900)

# Failed test at t/explain.t line 27.
not ok 5
# Structures begin differing at​:
# $got->[1] = undef
# $expected->[1] = ARRAY(0x1aeb6d8)
# Looks like you failed 2 tests of 5.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/5 subtests

Test Summary Report


../cpan/ExtUtils-MakeMaker/t/min_perl_version.t (Wstat​: 256 Tests​: 33
Failed​: 1)
  Failed test​: 17
  Non-zero exit status​: 1
../cpan/ExtUtils-MakeMaker/t/prereq_print.t (Wstat​: 256 Tests​: 11
Failed​: 1)
  Failed test​: 5
  Non-zero exit status​: 1
../cpan/Test-Simple/t/explain.t (Wstat​: 512 Tests​: 5
Failed​: 2)
  Failed tests​: 4-5
  Non-zero exit status​: 2
Files=3, Tests=49, 1 wallclock secs ( 0.02 usr 0.00 sys + 0.77 cusr
0.10 csys = 0.89 CPU)
Result​: FAIL
tony@​mars​:.../perl/t$ exit

Script done on Mon 17 May 2010 22​:07​:57 EST

@p5pRT
Copy link
Author

p5pRT commented May 25, 2010

From @cpansprout

On May 17, 2010, at 4​:53 AM, Steffen Mueller wrote​:

Hi,

Father Chrysostomos wrote​:

Data_Dumper_Dumpxs is missing a SPAGAIN after the call to DD_dump.
It needs it, as DD_Dump might call a custom sort routine, which
might reallocate the stack. This patch adds it, along with the test
which I’ve reduced to half the size, yet put in its own file since
it’s still big.

thanks for the patch! Unfortunately, I see some test failures when
running core tests with this change​:

...

Before applying the patch, these all passed (minus CGI.pm).

Could you have a look at these?

Ah, I should have run all of perl’s tests, not just DD’s.

Attached is a new patch, which works this time (hopefully), and also
adds explicit tests for Dumpxs in list context.

@p5pRT
Copy link
Author

p5pRT commented May 25, 2010

From @cpansprout

Inline Patch
diff -Nurp blead/dist/Data-Dumper/Dumper.xs blead-74170/dist/Data-Dumper/Dumper.xs
--- blead/dist/Data-Dumper/Dumper.xs	2010-04-15 12:45:10.000000000 -0700
+++ blead-74170/dist/Data-Dumper/Dumper.xs	2010-05-17 09:50:17.000000000 -0700
@@ -1133,6 +1133,7 @@ Data_Dumper_Dumpxs(href, ...)
 		valstr = newSVpvn("",0);
 		for (i = 0; i <= imax; ++i) {
 		    SV *newapad;
+		    I32 diff;
 		
 		    av_clear(postav);
 		    if ((svp = av_fetch(todumpav, i, FALSE)))
@@ -1188,10 +1189,16 @@ Data_Dumper_Dumpxs(href, ...)
 		    else
 			newapad = apad;
 		
+		    diff = SP - PL_stack_sp;
 		    DD_dump(aTHX_ val, SvPVX_const(name), SvCUR(name), valstr, seenhv,
 			    postav, &level, indent, pad, xpad, newapad, sep, pair,
 			    freezer, toaster, purity, deepcopy, quotekeys,
 			    bless, maxdepth, sortkeys);
+		    /* DD_dump might have called a custom sort routine that
+		       might have reallocated the stack (see
+		       [perl #74170]). */
+		    SPAGAIN;
+		    SP += diff;
 		
 		    if (indent >= 2 && !terse)
 			SvREFCNT_dec(newapad);
diff -Nurp blead/dist/Data-Dumper/t/dumper.t blead-74170/dist/Data-Dumper/t/dumper.t
--- blead/dist/Data-Dumper/t/dumper.t	2009-11-19 08:51:38.000000000 -0800
+++ blead-74170/dist/Data-Dumper/t/dumper.t	2010-05-17 12:43:32.000000000 -0700
@@ -83,7 +83,7 @@ sub SKIP_TEST {
 $Data::Dumper::Useperl = 1;
 if (defined &Data::Dumper::Dumpxs) {
   print "### XS extension loaded, will run XS tests\n";
-  $TMAX = 363; $XS = 1;
+  $TMAX = 366; $XS = 1;
 }
 else {
   print "### XS extensions not loaded, will NOT run XS tests\n";
@@ -1429,4 +1429,13 @@ EOT
     TEST q(Data::Dumper->Dumpxs([\@foo])) if $XS;
 }
 
+############# 364
+# Make sure $obj->Dumpxs returns the right thing in list context. This was
+# broken by the initial attempt to fix [perl #74170].
+$WANT = <<'EOT';
+#$VAR1 = [];
+EOT
+TEST q(join " ", new Data::Dumper [[]],[] =>->Dumpxs),
+    '$obj->Dumpxs in list context'
+ if $XS;
 
diff -Nurp blead/dist/Data-Dumper/t/perl-74170.t blead-74170/dist/Data-Dumper/t/perl-74170.t
--- blead/dist/Data-Dumper/t/perl-74170.t	1969-12-31 16:00:00.000000000 -0800
+++ blead-74170/dist/Data-Dumper/t/perl-74170.t	2010-05-12 22:25:45.000000000 -0700
@@ -0,0 +1,143 @@
+#!perl -X
+#
+# Regression test for [perl #74170] (missing SPAGAIN after DD_Dump(...)):
+# Since it’s so large, it gets its own file.
+
+BEGIN {
+    require Config; import Config;
+    no warnings 'once';
+    if ($Config{'extensions'} !~ /\bData\/Dumper\b/) {
+	print "1..0 # Skip: Data::Dumper was not built\n";
+	exit 0;
+    }
+}
+
+use strict;
+use Test::More tests => 1;
+use Data::Dumper;
+
+our %repos = ();
+&real_life_setup();
+
+$Data::Dumper::Indent = 1;
+# A custom sort sub is necessary for reproducing the bug, as this is where
+# the stack gets reallocated.
+$Data::Dumper::Sortkeys = sub { return [ reverse sort keys %{$_[0]} ]; }
+    unless exists $ENV{NO_SORT_SUB};
+
+ok +Data::Dumper->Dumpxs([\%repos], [qw(*repos)]);
+
+sub real_life_setup {
+    # set up the %repos hash in a manner that reflects a real run of
+    # gitolite's "compiler" script:
+    # Yes, all this is necessary to get the stack in such a state that the
+    # custom sort sub will trigger a reallocation.
+    push @{ $repos{''}{'@all'} }, ();
+    push @{ $repos{''}{'guser86'} }, ();
+    push @{ $repos{''}{'guser87'} }, ();
+    push @{ $repos{''}{'user88'} }, ();
+    push @{ $repos{''}{'grussell'} }, ();
+    push @{ $repos{''}{'guser0'} }, ();
+    push @{ $repos{''}{'guser1'} }, ();
+    push @{ $repos{''}{'guser10'} }, ();
+    push @{ $repos{''}{'guser11'} }, ();
+    push @{ $repos{''}{'guser12'} }, ();
+    push @{ $repos{''}{'guser13'} }, ();
+    push @{ $repos{''}{'guser14'} }, ();
+    push @{ $repos{''}{'guser15'} }, ();
+    push @{ $repos{''}{'guser16'} }, ();
+    push @{ $repos{''}{'guser17'} }, ();
+    push @{ $repos{''}{'guser18'} }, ();
+    push @{ $repos{''}{'guser19'} }, ();
+    push @{ $repos{''}{'guser2'} }, ();
+    push @{ $repos{''}{'guser20'} }, ();
+    push @{ $repos{''}{'guser21'} }, ();
+    push @{ $repos{''}{'guser22'} }, ();
+    push @{ $repos{''}{'guser23'} }, ();
+    push @{ $repos{''}{'guser24'} }, ();
+    push @{ $repos{''}{'guser25'} }, ();
+    push @{ $repos{''}{'guser26'} }, ();
+    push @{ $repos{''}{'guser27'} }, ();
+    push @{ $repos{''}{'guser28'} }, ();
+    push @{ $repos{''}{'guser29'} }, ();
+    push @{ $repos{''}{'guser3'} }, ();
+    push @{ $repos{''}{'guser30'} }, ();
+    push @{ $repos{''}{'guser31'} }, ();
+    push @{ $repos{''}{'guser32'} }, ();
+    push @{ $repos{''}{'guser33'} }, ();
+    push @{ $repos{''}{'guser34'} }, ();
+    push @{ $repos{''}{'guser35'} }, ();
+    push @{ $repos{''}{'guser36'} }, ();
+    push @{ $repos{''}{'guser37'} }, ();
+    push @{ $repos{''}{'guser38'} }, ();
+    push @{ $repos{''}{'guser39'} }, ();
+    push @{ $repos{''}{'guser4'} }, ();
+    push @{ $repos{''}{'guser40'} }, ();
+    push @{ $repos{''}{'guser41'} }, ();
+    push @{ $repos{''}{'guser42'} }, ();
+    push @{ $repos{''}{'guser43'} }, ();
+    push @{ $repos{''}{'guser44'} }, ();
+    push @{ $repos{''}{'guser45'} }, ();
+    push @{ $repos{''}{'guser46'} }, ();
+    push @{ $repos{''}{'guser47'} }, ();
+    push @{ $repos{''}{'guser48'} }, ();
+    push @{ $repos{''}{'guser49'} }, ();
+    push @{ $repos{''}{'guser5'} }, ();
+    push @{ $repos{''}{'guser50'} }, ();
+    push @{ $repos{''}{'guser51'} }, ();
+    push @{ $repos{''}{'guser52'} }, ();
+    push @{ $repos{''}{'guser53'} }, ();
+    push @{ $repos{''}{'guser54'} }, ();
+    push @{ $repos{''}{'guser55'} }, ();
+    push @{ $repos{''}{'guser56'} }, ();
+    push @{ $repos{''}{'guser57'} }, ();
+    push @{ $repos{''}{'guser58'} }, ();
+    push @{ $repos{''}{'guser59'} }, ();
+    push @{ $repos{''}{'guser6'} }, ();
+    push @{ $repos{''}{'guser60'} }, ();
+    push @{ $repos{''}{'guser61'} }, ();
+    push @{ $repos{''}{'guser62'} }, ();
+    push @{ $repos{''}{'guser63'} }, ();
+    push @{ $repos{''}{'guser64'} }, ();
+    push @{ $repos{''}{'guser65'} }, ();
+    push @{ $repos{''}{'guser66'} }, ();
+    push @{ $repos{''}{'guser67'} }, ();
+    push @{ $repos{''}{'guser68'} }, ();
+    push @{ $repos{''}{'guser69'} }, ();
+    push @{ $repos{''}{'guser7'} }, ();
+    push @{ $repos{''}{'guser70'} }, ();
+    push @{ $repos{''}{'guser71'} }, ();
+    push @{ $repos{''}{'guser72'} }, ();
+    push @{ $repos{''}{'guser73'} }, ();
+    push @{ $repos{''}{'guser74'} }, ();
+    push @{ $repos{''}{'guser75'} }, ();
+    push @{ $repos{''}{'guser76'} }, ();
+    push @{ $repos{''}{'guser77'} }, ();
+    push @{ $repos{''}{'guser78'} }, ();
+    push @{ $repos{''}{'guser79'} }, ();
+    push @{ $repos{''}{'guser8'} }, ();
+    push @{ $repos{''}{'guser80'} }, ();
+    push @{ $repos{''}{'guser81'} }, ();
+    push @{ $repos{''}{'guser82'} }, ();
+    push @{ $repos{''}{'guser83'} }, ();
+    push @{ $repos{''}{'guser84'} }, ();
+    push @{ $repos{''}{'guser85'} }, ();
+    push @{ $repos{''}{'guser9'} }, ();
+    push @{ $repos{''}{'user1'} }, ();
+    push @{ $repos{''}{'user10'} }, ();
+    push @{ $repos{''}{'user11'} }, ();
+    push @{ $repos{''}{'user12'} }, ();
+    push @{ $repos{''}{'user13'} }, ();
+    push @{ $repos{''}{'user14'} }, ();
+    push @{ $repos{''}{'user15'} }, ();
+    push @{ $repos{''}{'user16'} }, ();
+    push @{ $repos{''}{'user2'} }, ();
+    push @{ $repos{''}{'user3'} }, ();
+    push @{ $repos{''}{'user4'} }, ();
+    push @{ $repos{''}{'user5'} }, ();
+    push @{ $repos{''}{'user6'} }, ();
+    push @{ $repos{''}{'user7'} }, ();
+    $repos{''}{R}{'user8'} = 1;
+    $repos{''}{W}{'user8'} = 1;
+    push @{ $repos{''}{'user8'} }, ();
+}

@p5pRT
Copy link
Author

p5pRT commented May 31, 2010

From @tonycoz

On Sun, May 23, 2010 at 01​:55​:08PM -0700, Father Chrysostomos wrote​:

         newapad = apad;

+ diff = SP - PL_stack_sp;
DD_dump(aTHX_ val, SvPVX_const(name), SvCUR(name), valstr, seenhv,
postav, &level, indent, pad, xpad, newapad, sep, pair,
freezer, toaster, purity, deepcopy, quotekeys,
bless, maxdepth, sortkeys);
+ /* DD_dump might have called a custom sort routine that
+ might have reallocated the stack (see
+ [perl #74170]). */
+ SPAGAIN;
+ SP += diff;

Couldn't that just be​:

  PUTBACK;
...
  SPAGAIN;

?

Tony

@p5pRT
Copy link
Author

p5pRT commented Jun 16, 2010

From @cpansprout

I forgot to CC this to the list.

Begin forwarded message​:

From​: Father Chrysostomos
Date​: June 6, 2010 12​:59​:22 PM PDT
To​: Tony Cook
Subject​: Re​: [PATCH] Re​: [perl #74170] Data​::Dumper -- dumping large
hash returns empty string, segfaults perl

On May 31, 2010, at 6​:19 AM, Tony Cook wrote​:

On Sun, May 23, 2010 at 01​:55​:08PM -0700, Father Chrysostomos wrote​:

        newapad = apad;

+ diff = SP - PL_stack_sp;
DD_dump(aTHX_ val, SvPVX_const(name), SvCUR(name), valstr,
seenhv,
postav, &level, indent, pad, xpad, newapad, sep, pair,
freezer, toaster, purity, deepcopy, quotekeys,
bless, maxdepth, sortkeys);
+ /* DD_dump might have called a custom sort routine that
+ might have reallocated the stack (see
+ [perl #74170]). */
+ SPAGAIN;
+ SP += diff;

Couldn't that just be​:

PUTBACK;
...
SPAGAIN;

?

I have no idea. :-)

I believe that will cause PL_stack_sp to have a different value when
the custom sort sub is called, but I don’t know enough about perl’s
stack to say whether that will make any difference.

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 3, 2010

From @tonycoz

On Tue, Jun 15, 2010 at 07​:11​:43PM -0700, Father Chrysostomos wrote​:

I forgot to CC this to the list.

Begin forwarded message​:

From​: Father Chrysostomos
Date​: June 6, 2010 12​:59​:22 PM PDT
To​: Tony Cook
Subject​: Re​: [PATCH] Re​: [perl #74170] Data​::Dumper -- dumping large
hash returns empty string, segfaults perl

On May 31, 2010, at 6​:19 AM, Tony Cook wrote​:

On Sun, May 23, 2010 at 01​:55​:08PM -0700, Father Chrysostomos wrote​:

        newapad = apad;

+ diff = SP - PL_stack_sp;
DD_dump(aTHX_ val, SvPVX_const(name), SvCUR(name), valstr,
seenhv,
postav, &level, indent, pad, xpad, newapad, sep, pair,
freezer, toaster, purity, deepcopy, quotekeys,
bless, maxdepth, sortkeys);
+ /* DD_dump might have called a custom sort routine that
+ might have reallocated the stack (see
+ [perl #74170]). */
+ SPAGAIN;
+ SP += diff;

Couldn't that just be​:

PUTBACK;
...
SPAGAIN;

?

I have no idea. :-)

I believe that will cause PL_stack_sp to have a different value when
the custom sort sub is called, but I don’t know enough about perl’s
stack to say whether that will make any difference.

Father Chrysostomos

Could someone with a better internals understanding take a look at
this?

I wouldn't expect any problems with a simple PUTBACK/SPAGAIN, but then
what do I know?

Tony

@p5pRT
Copy link
Author

p5pRT commented Sep 6, 2010

@rafl - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Sep 6, 2010
@p5pRT
Copy link
Author

p5pRT commented Sep 6, 2010

From @tonycoz

On Tue, Jun 15, 2010 at 07​:11​:43PM -0700, Father Chrysostomos wrote​:

I forgot to CC this to the list.

Begin forwarded message​:

From​: Father Chrysostomos
Date​: June 6, 2010 12​:59​:22 PM PDT
To​: Tony Cook
Subject​: Re​: [PATCH] Re​: [perl #74170] Data​::Dumper -- dumping large
hash returns empty string, segfaults perl

On May 31, 2010, at 6​:19 AM, Tony Cook wrote​:

On Sun, May 23, 2010 at 01​:55​:08PM -0700, Father Chrysostomos wrote​:

        newapad = apad;

+ diff = SP - PL_stack_sp;
DD_dump(aTHX_ val, SvPVX_const(name), SvCUR(name), valstr,
seenhv,
postav, &level, indent, pad, xpad, newapad, sep, pair,
freezer, toaster, purity, deepcopy, quotekeys,
bless, maxdepth, sortkeys);
+ /* DD_dump might have called a custom sort routine that
+ might have reallocated the stack (see
+ [perl #74170]). */
+ SPAGAIN;
+ SP += diff;

Couldn't that just be​:

PUTBACK;
...
SPAGAIN;

?

I have no idea. :-)

I believe that will cause PL_stack_sp to have a different value when
the custom sort sub is called, but I don’t know enough about perl’s
stack to say whether that will make any difference.

Father Chrysostomos

I applied the PUTBACK/SPAGAIN alternative as e3ec229 and your tests
as fe64260.

Parameters supplied to a function are delimited by PUSHMARK(), which
the custom sort function call does.

If you were wondering there were two reasons I didn't want to apply
your patch​:

a) it seemed over complex

b) it used an I32 offset, which could be a problem in the future with
64-bit systems.

Tony

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant