Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak with use feature; and s///ee #9643

Closed
p5pRT opened this issue Feb 9, 2009 · 14 comments
Closed

Memory leak with use feature; and s///ee #9643

p5pRT opened this issue Feb 9, 2009 · 14 comments

Comments

@p5pRT
Copy link

p5pRT commented Feb 9, 2009

Migrated from rt.perl.org#63110 (status was 'resolved')

Searchable as RT63110$

@p5pRT
Copy link
Author

p5pRT commented Feb 9, 2009

From cary.millsap@gmail.com

Created by cary.millsap@gmail.com

This is a bug report for perl from cary.millsap@​gmail.com,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------

Here is a test case that reproduces a memory leak. Repro instructions are
within the introductory comment.

#!/usr/bin/perl

# Demonstrate an apparent memory leak when using use feature qw(​:5.10) with
# /gee. To reproduce the problem, run this script with a large input file.
On
# my system (Perl v5.10.0 Binary build 1004 [287188] on Mac OS X v10.5.6), I
# can run RSIZE1GB, VSIZE1GB with a 468MB input file. When I run with the
# 'use feature' line commented out, I can run with RSIZE=1.66MB,
VSIZE=18.97MB.

use strict;
use warnings;
use feature qw(​:5.10); # Remove this line, and the memory leak does not
occur.
my $group_expr = '$1';
while (<>) {
  my $bucket;
  if (/^(\w+)/) {
  ($bucket = $group_expr) =~ s/(\$\w+)/$1/gee;
  }
}

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl 5.10.0:

Configured by ActiveState at Wed Sep  3 12:30:38 PDT 2008.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=darwin, osvers=9.4.0, archname=darwin-thread-multi-2level
    uname='darwin kitty 9.4.0 darwin kernel version 9.4.0: mon jun 9
19:30:53 pdt 2008; root:xnu-1228.5.20~1release_i386 i386 '
    config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads -Ud_sigsetjmp
-Uinstallusrbinperl -Ulocincpth= -Uloclibpth= -Ud_poll -Ui_poll
-Ud_sockatmark -Ud_sockatmarkproto -Ud_pthread_atfork -Ud_msg -Ud_lchown
-Ud_attribute_format -Ud_attribute_malloc -Ud_attribute_nonnull
-Ud_attribute_noreturn -Ud_attribute_pure -Ud_attribute_unused
-Ud_attribute_warn_unused_result -Ui_libutil
-Aldflags=-Wl,-search_paths_first -Alddlflags=-Wl,-search_paths_first
-Accflags=-arch ppc -arch i386 -Aldflags=-arch ppc -arch i386
-Alddlflags=-arch ppc -arch i386 -Accflags=-nostdinc
-B/Developer/SDKs/MacOSX10.4u.sdk/usr/include/gcc
-B/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/gcc
-isystem/Developer/SDKs/MacOSX10.4u.sdk/usr/include
-F/Developer/SDKs/MacOSX10.4u.sdk/System/Library/Frameworks
-mmacosx-version-min=10.3
-Aldflags=-Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk
-mmacosx-version-min=10.3
-Alddlflags=-Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk
-mmacosx-version-min=10.3 -Accflags=-DUSE_SITECUSTOMIZE -Duselargefiles
-Accflags=-DPRIVLIB_LAST_IN_INC -Dprefix=/usr/local/ActivePerl-5.10
-Dprivlib=/usr/local/ActivePerl-5.10/lib
-Darchlib=/usr/local/ActivePerl-5.10/lib
-Dsiteprefix=/usr/local/ActivePerl-5.10/site
-Dsitelib=/usr/local/ActivePerl-5.10/site/lib
-Dsitearch=/usr/local/ActivePerl-5.10/site/lib -Dsed=/usr/bin/sed
-Duseshrplib -Dcf_by=ActiveState -Dcf_email=support@ActiveState.com'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-fno-common -DPERL_DARWIN -no-cpp-precomp -arch ppc
-arch i386 -nostdinc -B/Developer/SDKs/MacOSX10.4u.sdk/usr/include/gcc
-B/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/gcc
-isystem/Developer/SDKs/MacOSX10.4u.sdk/usr/include
-F/Developer/SDKs/MacOSX10.4u.sdk/System/Library/Frameworks
-mmacosx-version-min=10.3 -DUSE_SITECUSTOMIZE -DPRIVLIB_LAST_IN_INC
-fno-strict-aliasing -pipe',
    optimize='-O3',
    cppflags='-no-cpp-precomp -fno-common -DPERL_DARWIN -no-cpp-precomp
-arch ppc -nostdinc -B/Developer/SDKs/MacOSX10.4u.sdk/usr/include/gcc
-B/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/gcc
-isystem/Developer/SDKs/MacOSX10.4u.sdk/usr/include
-F/Developer/SDKs/MacOSX10.4u.sdk/System/Library/Frameworks
-mmacosx-version-min=10.3 -DUSE_SITECUSTOMIZE -DPRIVLIB_LAST_IN_INC
-fno-strict-aliasing -pipe'
    ccversion='', gccversion='i686-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1
(Apple Inc. build 5488)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='
-Wl,-search_paths_first -arch ppc -arch i386
-Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.3'
    libpth=/usr/lib
    libs=-ldbm -ldl -lm -lc
    perllibs=-ldl -lm -lc
    libc=, so=dylib, useshrplib=true, libperl=libperl.dylib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-Wl,-search_paths_first -arch ppc -arch i386
-Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk -mmacosx-version-min=10.3'

Locally applied patches:
    ACTIVEPERL_LOCAL_PATCHES_ENTRY
    33741 avoids segfaults invoking S_raise_signal() (on Linux)
    33763 Win32 process ids can have more than 16 bits
    32809 Load 'loadable object' with non-default file extension
    32728 64-bit fix for Time::Local


@INC for perl 5.10.0:
    /Users/carymillsap/Documents/Development/TFK-workspace/trace-tools/
    /usr/local/ActivePerl-5.10/site/bin
    /Users/carymillsap/Library/ActivePerl-5.10/lib
    /usr/local/ActivePerl-5.10/site/lib
    /usr/local/ActivePerl-5.10/lib
    .


Environment for perl 5.10.0:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/carymillsap
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)

PATH=.:/usr/local/ActivePerl-5.10/bin:/usr/local/ActivePerl-5.10/site/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

PERL5LIB=/Users/carymillsap/Documents/Development/TFK-workspace/trace-tools/:/usr/local/ActivePerl-5.10/site/bin
    PERLDIR=/usr/local/ActivePerl-5.10
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Feb 9, 2009

From p5p@perl.wizbit.be

Here is a test case that reproduces a memory leak. Repro instructions are
within the introductory comment.

Easier test script​:

#!/usr/bin/perl -l

use strict;
use warnings;
use feature qw(​:5.10);

system("ps uh $$");
for (0 .. 100_000) {
  system("ps uh $$") if $_ % 10_000 == 0;
  my $s = "a";
  $s =~ s/a/qq("aa")/ee;
}
system("ps uh $$");

__END__
$ perl-5.10.0 rt-63110.pl
bram 25508 0.0 0.0 3304 1644 pts/0 S+ 13​:31 0​:00
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 0.0 0.0 3304 1652 pts/0 S+ 13​:31 0​:00
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 25.0 0.1 5416 3752 pts/0 S+ 13​:31 0​:00
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 49.0 0.1 7396 5828 pts/0 S+ 13​:31 0​:00
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 73.0 0.2 9508 7912 pts/0 S+ 13​:31 0​:00
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 97.0 0.3 11620 9992 pts/0 S+ 13​:31 0​:00
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 60.5 0.3 13732 12072 pts/0 S+ 13​:31 0​:01
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 73.0 0.4 15844 14152 pts/0 S+ 13​:31 0​:01
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 85.5 0.5 17824 16232 pts/0 S+ 13​:31 0​:01
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 97.5 0.5 19936 18312 pts/0 S+ 13​:31 0​:01
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 73.3 0.6 22048 20396 pts/0 S+ 13​:31 0​:02
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 81.3 0.7 24160 22472 pts/0 S+ 13​:31 0​:02
/opt/perl/bin/perl5100 rt-63110.pl
bram 25508 81.3 0.7 24160 22472 pts/0 S+ 13​:31 0​:02
/opt/perl/bin/perl5100 rt-63110.pl

Two things affect the memory leak​:
- use feature
- /ee​: if /e is used there is no leak

(I also reproduced this on blead but the one that is tested is a
fairly old blead... (.patch = 34076))

Kind regards,

Bram

@p5pRT
Copy link
Author

p5pRT commented Feb 9, 2009

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Feb 14, 2009

From p5p@spam.wizbit.be

On Mon Feb 09 04​:35​:54 2009, p5p@​perl.wizbit.be wrote​:

Here is a test case that reproduces a memory leak. Repro
instructions are
within the introductory comment.

Two things affect the memory leak​:
- use feature
- /ee​: if /e is used there is no leak

Did a binary search​:

----Program----
#!/usr/bin/perl -l

use strict;
use warnings;
use feature qw(​:5.10);

my $out1 = qx(ps uh $$);
my ($mem1) = $out1 =~ m/^\S+\s+\d+\s+[\d.]+\s+[\d.]+\s+(\d+)\s+/;

for (0 .. 100_000) {
  my $s = "a";
  $s =~ s/a/qq("aa")/ee;
}

my $out2 = qx(ps uh $$);
my ($mem2) = $out2 =~ m/^\S+\s+\d+\s+[\d.]+\s+[\d.]+\s+(\d+)\s+/;

if ($mem2 > $mem1 + 2048) {
  print "not ok";
}
else {
  print "ok";
}

----Output of .../pjYRGC9/perl-5.9.3@​27665/bin/perl----
ok

----EOF ($?='0')----
----Output of .../pNCKy6I/perl-5.9.3@​27666/bin/perl----
not ok

----EOF ($?='0')----

http​://public.activestate.com/cgi-bin/perlbrowse/p/27666

Change 27666 by nicholas@​nicholas-saigo on 2006/04/01 21​:17​:46

  Automatically set HINT_LOCALIZE_HH whenever %^H is modified.

Kind regards,

Bram

@p5pRT
Copy link
Author

p5pRT commented May 30, 2009

From alex@chmrr.net

At Sun Feb 08 23​:49​:18 -0500 2009, Cary Millsap (via RT) wrote​:

Here is a test case that reproduces a memory leak. Repro instructions are
within the introductory comment.

Fix is attached.
- Alex
--
Networking -- only one letter away from not working

@p5pRT
Copy link
Author

p5pRT commented May 30, 2009

From alex@chmrr.net

0001-Fix-RT-63110-two-small-memory-leaks-were-introduced-.patch
From a05e910190864dbe2c98bc149f6fc067c16072a6 Mon Sep 17 00:00:00 2001
From: Alex Vandiver <alexmv@mit.edu>
Date: Fri, 29 May 2009 20:17:36 -0400
Subject: [PATCH] Fix [RT#63110] -- two small memory leaks were introduced in 5b9c067

---
 hv.c     |    2 +-
 pp_ctl.c |    5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/hv.c b/hv.c
index 1776053..2cefbcb 100644
--- a/hv.c
+++ b/hv.c
@@ -1444,7 +1444,7 @@ Perl_hv_copy_hints_hv(pTHX_ HV *const ohv)
 	while ((entry = hv_iternext_flags(ohv, 0))) {
 	    SV *const sv = newSVsv(HeVAL(entry));
 	    sv_magic(sv, NULL, PERL_MAGIC_hintselem,
-		     (char *)newSVhek (HeKEY_hek(entry)), HEf_SVKEY);
+		     (char *)sv_2mortal(newSVhek (HeKEY_hek(entry))), HEf_SVKEY);
 	    (void)hv_store_flags(hv, HeKEY(entry), HeKLEN(entry),
 				 sv, HeHASH(entry), HeKFLAGS(entry));
 	}
diff --git a/pp_ctl.c b/pp_ctl.c
index b4f0ade..2c80627 100644
--- a/pp_ctl.c
+++ b/pp_ctl.c
@@ -3636,8 +3636,11 @@ PP(pp_entereval)
     SAVEDELETE(PL_defstash, safestr, len);
     SAVEHINTS();
     PL_hints = PL_op->op_targ;
-    if (saved_hh)
+    if (saved_hh) {
+	/* SAVEHINTS created a new HV in PL_hintgv, which we need to GC */
+	SvREFCNT_dec(GvHV(PL_hintgv));
 	GvHV(PL_hintgv) = saved_hh;
+    }
     SAVECOMPILEWARNINGS();
     PL_compiling.cop_warnings = DUP_WARNINGS(PL_curcop->cop_warnings);
     if (PL_compiling.cop_hints_hash) {
-- 
1.6.3.204.g8c948

@p5pRT
Copy link
Author

p5pRT commented May 30, 2009

@rgs - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed May 30, 2009
@p5pRT
Copy link
Author

p5pRT commented May 30, 2009

From @rgs

2009/5/30 Alex Vandiver <alex@​chmrr.net>​:

At Sun Feb 08 23​:49​:18 -0500 2009, Cary Millsap (via RT) wrote​:

Here is a test case that reproduces a memory leak. Repro instructions are
within the introductory comment.

Fix is attached.

Thanks, applied as cda5537

@p5pRT
Copy link
Author

p5pRT commented May 31, 2009

From p5p@perl.wizbit.be

Citeren Rafael Garcia-Suarez <rgarciasuarez@​gmail.com>​:

2009/5/30 Alex Vandiver <alex@​chmrr.net>​:

At Sun Feb 08 23​:49​:18 -0500 2009, Cary Millsap (via RT) wrote​:

Here is a test case that reproduces a memory leak. Repro instructions are
within the introductory comment.

Fix is attached.

Thanks, applied as cda5537

Attached is a test case for this bug.

I added the test in a new file t/lib/feature/leak because I could not
find a good place to add it...

Best regards,

Bram

@p5pRT
Copy link
Author

p5pRT commented May 31, 2009

From p5p@perl.wizbit.be

patch_63110

@p5pRT
Copy link
Author

p5pRT commented May 31, 2009

From @rgs

2009/5/31 Bram <p5p@​perl.wizbit.be>​:

Citeren Rafael Garcia-Suarez <rgarciasuarez@​gmail.com>​:

2009/5/30 Alex Vandiver <alex@​chmrr.net>​:

At Sun Feb 08 23​:49​:18 -0500 2009, Cary Millsap (via RT) wrote​:

Here is a test case that reproduces a memory leak. Repro instructions
are
within the introductory comment.

Fix is attached.

Thanks, applied as cda5537

Attached is a test case for this bug.

I added the test in a new file t/lib/feature/leak because I could not find a
good place to add it...

It's technically not a feature.pm bug, but it's related to hints,
scopes and eval.
I added it in eval.t instead (fa13de9)

@p5pRT
Copy link
Author

p5pRT commented Jul 20, 2009

From @jbenjore

p5p@​perl.wizbit.be (Bram) writes​:

Attached is a test case for this bug.

I added the test in a new file t/lib/feature/leak because I could not
find a good place to add it...

Best regards,

Bram

Thanks, applied as cda5e2e

@p5pRT
Copy link
Author

p5pRT commented Jul 20, 2009

From @iabyn

On Mon, Jul 20, 2009 at 09​:52​:40AM -0700, Josh ben Jore wrote​:

p5p@​perl.wizbit.be (Bram) writes​:

Attached is a test case for this bug.

I added the test in a new file t/lib/feature/leak because I could not
find a good place to add it...

Best regards,

Bram

Thanks, applied as cda5e2e

That test had already been applied by Rafael (in a different form)​:

commit fa13de9
Author​: Rafael Garcia-Suarez <rgarciasuarez@​gmail.com>
AuthorDate​: Sun May 31 15​:09​:56 2009 +0200
Commit​: Rafael Garcia-Suarez <rgarciasuarez@​gmail.com>
CommitDate​: Sun May 31 15​:09​:56 2009 +0200

  Add regression test for [RT#63110]
 
  This is based on a stand-alone test by Bram
 

Affected files ...
 
  M t/op/eval.t

Differences ...

Inline Patch
diff --git a/t/op/eval.t b/t/op/eval.t
index 23725d5..4c01434 100755
--- a/t/op/eval.t
+++ b/t/op/eval.t
@@ -6,7 +6,7 @@ BEGIN {
     require './test.pl';
 }
 
-print "1..98\n";
+print "1..99\n";
 
 eval 'print "ok 1\n";';
 
@@ -539,10 +539,20 @@ END_EVAL_TEST
    my $ok = runperl(progfile => $tempfile);
    print "not " unless $ok;
    print "ok $test # eval { 1 } completly resets \$@\n";
-
-   $test++;
 }
 else {
   print "ok $test # skipped - eval { 1 } completly resets \$@";
 }
+$test++;
 
+# Test that "use feature" and other hint transmission in evals and s///ee
+# don't leak memory
+{
+    use feature qw(:5.10);
+    my $t;
+    my $s = "a";
+    $s =~ s/a/$t = \%^H;  qq( qq() );/ee;
+    print "not " if Internals::SvREFCNT(%$t) != 1;
+    print "ok $test - RT 63110\n";
+    $test++;
+}


-- 

A major Starfleet emergency breaks out near the Enterprise, but
fortunately some other ships in the area are able to deal with it to
everyone's satisfaction.
  -- Things That Never Happen in "Star Trek" #13

@p5pRT
Copy link
Author

p5pRT commented Jul 20, 2009

From @jbenjore

On Mon, Jul 20, 2009 at 9​:52 AM, Josh ben Jore<twists@​gmail.com> wrote​:

p5p@​perl.wizbit.be (Bram) writes​:

Attached is a test case for this bug.

I added the test in a new file t/lib/feature/leak because I could not
find a good place to add it...

Best regards,

Bram

Thanks, applied as cda5e2e

Reverted as a5326e8 because
cda5e2e duplicates a commit in
fa13de9.

Josh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant