Memory leak in nested shared data structures in 5.8.4 #7342

p5pRT · 2004-06-06T05:04:33Z

Migrated from rt.perl.org#30066 (status was 'resolved')

Searchable as RT30066$

p5pRT · 2004-06-06T05:04:33Z

From perlbug@ericgarland.com

Created by perlbug@ericgarland.com

The following code snippet leaks memory. I've tested it
on perl-5.8.4 on RedHat 7.2, ActivePerl-5.8.3 on Windows XP and
perl-5.8.2 on cygwin. They all leaked.

use threads;
use threads::shared;

share (@q);
$q[0] = &share([]);

while (1) {
push (@{ $q[0] }, $_);
shift(@{ $q[0] });
}

I'm using a shared array stored as element 0 of another shared
array. I found the issue when I was trying to create a library
for doing a multi-priority queue and with much testing I boiled
it down to this.

I'm using a nested shared array which may not be supported.
I tried to find somewhere in the documentation where this was
mentioned as working or not working but I didn't find anything.

I also tried accomplishing the same task with the shared array
stored in a hash and it leaked memory as well. Below is the code
for that test.

use threads;
use threads::shared;

share (%q);
$q{0} = &share({});

while (1) {
push (@{ $q{0} }, $_);
shift(@{ $q{0} });
}

Perl Info


Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.4:

Configured by Red Hat, Inc. at Fri Jun  4 15:51:16 EDT 2004.

Summary of my perl5 (revision 5 version 8 subversion 4) configuration:
  Platform:
    osname=linux, osvers=2.4.18-19.7.xsmp, archname=i386-linux-thread-multi
    uname='linux fsdev2.ericgarland.com 2.4.18-19.7.xsmp #1 smp thu dec 12 07:56:58 est 2002 i686 unknown '
    config_args='-des -Doptimize=-O2 -march=i386 -mcpu=i686 -Dversion=5.8.4 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dinc_version_list=5.8.2 5.8.1 5.8.0'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -march=i386 -mcpu=i686',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.1 2.96-98)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.2.4.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.2.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.4/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.4:
    /usr/lib/perl5/5.8.4/i386-linux-thread-multi
    /usr/lib/perl5/5.8.4
    /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.4
    /usr/lib/perl5/site_perl/5.8.2
    /usr/lib/perl5/site_perl/5.8.1
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.4
    /usr/lib/perl5/vendor_perl/5.8.2
    /usr/lib/perl5/vendor_perl/5.8.1
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    .


Environment for perl v5.8.4:
    HOME=/root
    LANG=en_US
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/egarland/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2004-06-23T06:15:39Z

From perlbug@ericgarland.com

I also tried accomplishing the same task with the shared array
stored in a hash and it leaked memory as well. Below is the code
for that test.

use threads;
use threads::shared;

share (%q);
$q{0} = &share({});

while (1) {
push (@{ $q{0} }, $_);
shift(@{ $q{0} });
}

Oops. I typo'd that.
The line $q{0} = &share({});
should be $q{0} = &share([]);
Full script:

use threads;
use threads::shared;

share (%q);
$q{0} = &share([]);

while (1) {
push (@{ $q{0} }, $_);
shift(@{ $q{0} });
}

FYI, I've tested both scripts on ActivePerl 5.8.4. They both leak there
as well.

p5pRT · 2004-06-23T06:15:41Z

perlbug@ericgarland.com - Status changed from 'new' to 'open'

p5pRT · 2004-09-12T22:40:35Z

From @iabyn

On Sun, Jun 06, 2004 at 05:04:34AM -0000, Eric Garland wrote:

The following code snippet leaks memory. I've tested it
on perl-5.8.4 on RedHat 7.2, ActivePerl-5.8.3 on Windows XP and
perl-5.8.2 on cygwin. They all leaked.

use threads;
use threads::shared;

share (@q);
$q[0] = &share([]);

while (1) {
push (@{ $q[0] }, $_);
shift(@{ $q[0] });
}

Thanks for the report. I've fixed it with the patch below to the
development branch of Perl.

P5Pers:

You may remember that there was a refcount bug with version objects, which
I tracked down to sv_upgrade(sv, SVt_PVAV) not setting the AVf_REAL flag.
I suggested then getting sv_upgrade() to set this flag rather than getting
each individual caller to do it, but the response was lukewarm, especially
on the grounds that upgrading a scalar to an AV was probably a bad thing
to be doing in the first place.

Well, it turns out that threads.xs has the same bug, and its usage of
sv_upgrade() seems quite reasonable. The code in Perl_sharedsv_associate()
is along the lines of

if (!newsv)
newsv = newSV(0);
sv_upgrade(newsv, SvTYPE(oldsv));
switch(SvTYPE(oldsv)) {
case SVt_PVAV:
...

Since this has now cropped up 3 times, I've made an executive decision
and fixed sv_upgrade().

Dave.

--
But Pity stayed his hand. "It's a pity I've run out of bullets",
he thought. - "Bored of the Rings"

Change 23320 by davem@davem-percy on 2004/09/12 22:09:51

[perl #30066] Memory leak in nested shared data structures in 5.8.4
A pop of an item from a shared array ref causes a leak due to
AVf_REAL not having been set after an sv_upgrade(sv, SVt_PVAV).
Make sv_upgrade() set always this flag.

Affected files ...

... //depot/perl/sv.c#762 edit

Differences ...

==== //depot/perl/sv.c#762 (text) ====

@@ -1756,6 +1756,7 @@
bool
Perl_sv_upgrade(pTHX_ register SV *sv, U32 mt)
{
+
char* pv = NULL;
U32 cur = 0;
U32 len = 0;
@@ -1953,7 +1954,7 @@
SvSTASH(sv) = stash;
AvALLOC(sv) = 0;
AvARYLEN(sv) = 0;
- AvFLAGS(sv) = 0;
+ AvFLAGS(sv) = AVf_REAL;
break;
case SVt_PVHV:
SvANY(sv) = new_XPVHV();

p5pRT · 2004-09-12T22:48:49Z

@iabyn - Status changed from 'open' to 'resolved'

p5pRT · 2004-09-15T23:30:26Z

From perl5porters@ericgarland.com

Thanks Dave! You da man! Hopefully this will fix the general
leakiness I've seen in shared variables in 5.8.

I have no idea how to download the current development tree for
Perl so I applied your patch to 5.9.1 and it worked on both the
shared array and the shared hash leaks. I also tried it on
5.8.4 and it worked there too. Do you think there's a chance
of getting it in 5.8.5?

Thanks again.

-Eric
Dave Mitchell wrote:

On Sun, Jun 06, 2004 at 05:04:34AM -0000, Eric Garland wrote:

The following code snippet leaks memory. I've tested it
on perl-5.8.4 on RedHat 7.2, ActivePerl-5.8.3 on Windows XP and
perl-5.8.2 on cygwin. They all leaked.

use threads;
use threads::shared;

share (@q);
$q[0] = &share([]);

while (1) {
push (@{ $q[0] }, $_);
shift(@{ $q[0] });
}

Thanks for the report. I've fixed it with the patch below to the
development branch of Perl.

P5Pers:

You may remember that there was a refcount bug with version objects, which
I tracked down to sv_upgrade(sv, SVt_PVAV) not setting the AVf_REAL flag.
I suggested then getting sv_upgrade() to set this flag rather than getting
each individual caller to do it, but the response was lukewarm, especially
on the grounds that upgrading a scalar to an AV was probably a bad thing
to be doing in the first place.

Well, it turns out that threads.xs has the same bug, and its usage of
sv_upgrade() seems quite reasonable. The code in Perl_sharedsv_associate()
is along the lines of
if $\!newsv$
newsv = newSV$0$;
sv\_upgrade$newsv\, SvTYPE\(oldsv$\);
switch$SvTYPE\(oldsv$\) \{
case SVt\_PVAV&#8203;:
\.\.\.
Since this has now cropped up 3 times, I've made an executive decision
and fixed sv_upgrade().

Dave.

p5pRT · 2004-09-16T18:21:42Z

From perl5porters@ericgarland.com

I should say is there any chance of getting it in 5.8.6 since
5.8.5 has already been released.

-Eric

Eric Garland wrote:

Thanks Dave! You da man! Hopefully this will fix the general
leakiness I've seen in shared variables in 5.8.

I have no idea how to download the current development tree for
Perl so I applied your patch to 5.9.1 and it worked on both the
shared array and the shared hash leaks. I also tried it on
5.8.4 and it worked there too. Do you think there's a chance
of getting it in 5.8.5?

Thanks again.
\-Eric
Dave Mitchell wrote:
On Sun, Jun 06, 2004 at 05:04:34AM -0000, Eric Garland wrote:

The following code snippet leaks memory. I've tested it
on perl-5.8.4 on RedHat 7.2, ActivePerl-5.8.3 on Windows XP and
perl-5.8.2 on cygwin. They all leaked.

use threads;
use threads::shared;

share (@q);
$q[0] = &share([]);

while (1) {
push (@{ $q[0] }, $_);
shift(@{ $q[0] });
}

Thanks for the report. I've fixed it with the patch below to the
development branch of Perl.

P5Pers:

You may remember that there was a refcount bug with version objects,
which
I tracked down to sv_upgrade(sv, SVt_PVAV) not setting the AVf_REAL flag.
I suggested then getting sv_upgrade() to set this flag rather than
getting
each individual caller to do it, but the response was lukewarm,
especially
on the grounds that upgrading a scalar to an AV was probably a bad thing
to be doing in the first place.

Well, it turns out that threads.xs has the same bug, and its usage of
sv_upgrade() seems quite reasonable. The code in
Perl_sharedsv_associate()
is along the lines of
if $\!newsv$
newsv = newSV$0$;
sv\_upgrade$newsv\, SvTYPE\(oldsv$\);
switch$SvTYPE\(oldsv$\) \{
case SVt\_PVAV&#8203;:
\.\.\.
Since this has now cropped up 3 times, I've made an executive decision
and fixed sv_upgrade().

Dave.

p5pRT · 2004-09-20T22:44:22Z

From @iabyn

On Thu, Sep 16, 2004 at 02:20:49PM -0400, Eric Garland wrote:

I have no idea how to download the current development tree for
Perl so I applied your patch to 5.9.1 and it worked on both the
shared array and the shared hash leaks. I also tried it on
5.8.4 and it worked there too. Do you think there's a chance
of getting it in 5.8.5?

I should say is there any chance of getting it in 5.8.6 since
5.8.5 has already been released.

Well, that's up to the release manager for the 5.8.x branch, Nicholas;
but I can't see any reason for him not to integrate the patch (hint hint
:-)

Dave.

--
This is a great day for France!
-- Nixon at Charles De Gaulle's funeral

p5pRT closed this as completed Sep 12, 2004

p5pRT added Severity High distro-All type-core labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory leak in nested shared data structures in 5.8.4 #7342

Memory leak in nested shared data structures in 5.8.4 #7342

p5pRT commented Jun 6, 2004

p5pRT commented Jun 6, 2004

p5pRT commented Jun 23, 2004

p5pRT commented Jun 23, 2004

p5pRT commented Sep 12, 2004

p5pRT commented Sep 12, 2004

p5pRT commented Sep 15, 2004

p5pRT commented Sep 16, 2004

p5pRT commented Sep 20, 2004

Memory leak in nested shared data structures in 5.8.4 #7342

Memory leak in nested shared data structures in 5.8.4 #7342

Comments

p5pRT commented Jun 6, 2004

p5pRT commented Jun 6, 2004

From perlbug@ericgarland.com

Created by perlbug@ericgarland.com

p5pRT commented Jun 23, 2004

From perlbug@ericgarland.com

p5pRT commented Jun 23, 2004

p5pRT commented Sep 12, 2004

From @iabyn

p5pRT commented Sep 12, 2004

p5pRT commented Sep 15, 2004

From perl5porters@ericgarland.com

p5pRT commented Sep 16, 2004

From perl5porters@ericgarland.com

p5pRT commented Sep 20, 2004

From @iabyn