Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Out of memory ... sbrk()" on FreeBSD-6.x for v.5.8.x but not v.5.6.x #9061

Closed
p5pRT opened this issue Oct 12, 2007 · 7 comments
Closed

"Out of memory ... sbrk()" on FreeBSD-6.x for v.5.8.x but not v.5.6.x #9061

p5pRT opened this issue Oct 12, 2007 · 7 comments

Comments

@p5pRT
Copy link

p5pRT commented Oct 12, 2007

Migrated from rt.perl.org#46381 (status was 'resolved')

Searchable as RT46381$

@p5pRT
Copy link
Author

p5pRT commented Oct 12, 2007

From zeus@dn.farlep.net

Created by zeus@dn.farlep.net

i've got reproduceble picture​:

FreeBSD-6.x + Perl5.8.8 on i386, are affected
while Linux/Windows are not ...

the script using Image​::ExifTool to manipulate EXIF data in batch mode

the problems looks so​:
---
Out of memory during "large" request for 2101248 bytes, total sbrk() is
533565440 bytes at
/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl line 2925,
<EXIFTOOL_FILE2> chunk 12.
---

process memory grows and finaly it is kicked :(

yes, ulimit -a shows​: data seg size is 524288

but why run under linux the process frees memory? and under M$Win it works?

on amd64 the picture differs, top shows​:
---
last pid​: 1019; load averages​: 0.49, 0.25, 0.10 up 0+02​:04​:25 10​:06​:21
43 processes​: 1 running, 42 sleeping
CPU states​: 28.6% user, 0.0% nice, 1.1% system, 0.8% interrupt, 69.5%
idle
Mem​: 293M Active, 27M Inact, 73M Wired, 18M Cache, 54M Buf, 892K Free
Swap​: 2048M Total, 1034M Used, 1014M Free, 50% Inuse, 460K Out

  PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
  988 zeus 1 -8 0 1865M 301M biord 1 1​:11 34.18% perl
nd the process finishes successfully

while trying to meditate upon the cause, i've launched the debuger
and here the last lines of output​:

---
...
Image​::ExifTool​::WriteJPEG(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3446)​:
3446​: Write($outfile, $buff) or $err = 1, last;
Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3002)​:
3002​: my $outfile = shift;
Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3003)​:
3003​: if (UNIVERSAL​::isa($outfile,'GLOB')) {
Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3006)​:
3006​: $$outfile .= join('', @​_);
Out of memory during request for 16480 bytes, total sbrk() is 533696512
bytes!
Debugged program terminated. Use q to quit or R to restart,
  use o inhibit_exit to avoid stopping after program termination,
  h q, h R or h o to get additional info.
  DB<10> T
$ = DB​::fake​::at_exit() called from file `/usr/local/lib/perl5/5.8.8/perl5db.pl' line 8993
$ = DB​::END() called from file `/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl' line
3006
$ = eval {...} called from file `/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl' line
3006
  DB<10>
---

i would like to point out it again, the problem present only on FreeBSD 6.x with Perl5.8.x (Perl5.6.x not affected)

Perl Info

Flags:
    category=core
    severity=critical

Site configuration information for perl v5.8.8:

Configured by root at Sun Aug  5 02:09:25 EEST 2007.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=freebsd, osvers=6.2-stable, archname=i386-freebsd-64int
    uname='freebsd zeus.dn.farlep.net 6.2-stable freebsd 6.2-stable #0: sat aug 4 18:08:24 eest 2007 root@zeus.dn.farlep.net:usrobjusrsrcsyszeus i386 '
    config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.8.8/mach -Dprivlib=/usr/local/lib/perl5/5.8.8 -Dman3dir=/usr/local/lib/perl5/5.8.8/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.8.8/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.8.8 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.8.8/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.8/BSDPAN" -Doptimize=-O2 -fno-strict-aliasing -pipe  -Ud_dosuid -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=undef uselongdouble=undef
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.8/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include',
    optimize='-O2 -fno-strict-aliasing -pipe ',
    cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.8/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include'
    ccversion='', gccversion='3.4.6 [FreeBSD] 20060305', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -Wl,-E -L/usr/local/lib'
    libpth=/usr/lib /usr/local/lib
    libs=-lm -lcrypt -lutil
    perllibs=-lm -lcrypt -lutil
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -Wl,-R/usr/local/lib/perl5/5.8.8/mach/CORE'
    cccdlflags='-DPIC -fPIC', lddlflags='-shared  -L/usr/local/lib'

Locally applied patches:
    defined-or


@INC for perl v5.8.8:
    /usr/local/lib/perl5/5.8.8/BSDPAN
    /usr/local/lib/perl5/site_perl/5.8.8/mach
    /usr/local/lib/perl5/site_perl/5.8.8
    /usr/local/lib/perl5/site_perl
    /usr/local/lib/perl5/5.8.8/mach
    /usr/local/lib/perl5/5.8.8
    .


Environment for perl v5.8.8:
    HOME=/home/zeus
    LANG=ru_RU.KOI8-R
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/home/zeus/bin
    PERL_BADLANG (unset)
    SHELL=/bin/tcsh

@p5pRT
Copy link
Author

p5pRT commented Oct 25, 2007

From @eserte

"zeus@​dn.farlep.net (via RT)" <perlbug-followup@​perl.org> writes​:

PATCH below!

# New Ticket Created by zeus@​dn.farlep.net
# Please include the string​: [perl #46381]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=46381 >

This is a bug report for perl from zeus@​dn.farlep.net,
generated with the help of perlbug 1.35 running under perl v5.8.8.

-----------------------------------------------------------------
[Please enter your report here]

i've got reproduceble picture​:

FreeBSD-6.x + Perl5.8.8 on i386, are affected
while Linux/Windows are not ...

the script using Image​::ExifTool to manipulate EXIF data in batch mode

the problems looks so​:
---
Out of memory during "large" request for 2101248 bytes, total sbrk() is
533565440 bytes at
/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl line 2925,
<EXIFTOOL_FILE2> chunk 12.
---

process memory grows and finaly it is kicked :(

yes, ulimit -a shows​: data seg size is 524288

but why run under linux the process frees memory? and under M$Win it works?

on amd64 the picture differs, top shows​:
---
last pid​: 1019; load averages​: 0.49, 0.25, 0.10 up 0+02​:04​:25 10​:06​:21
43 processes​: 1 running, 42 sleeping
CPU states​: 28.6% user, 0.0% nice, 1.1% system, 0.8% interrupt, 69.5%
idle
Mem​: 293M Active, 27M Inact, 73M Wired, 18M Cache, 54M Buf, 892K Free
Swap​: 2048M Total, 1034M Used, 1014M Free, 50% Inuse, 460K Out

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
988 zeus 1 -8 0 1865M 301M biord 1 1​:11 34.18% perl
nd the process finishes successfully

while trying to meditate upon the cause, i've launched the debuger
and here the last lines of output​:

---
...
Image​::ExifTool​::WriteJPEG(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3446)​:
3446​: Write($outfile, $buff) or $err = 1, last;
Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3002)​:
3002​: my $outfile = shift;
Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3003)​:
3003​: if (UNIVERSAL​::isa($outfile,'GLOB')) {
Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3006)​:
3006​: $$outfile .= join('', @​_);
Out of memory during request for 16480 bytes, total sbrk() is 533696512
bytes!
Debugged program terminated. Use q to quit or R to restart,
use o inhibit_exit to avoid stopping after program termination,
h q, h R or h o to get additional info.
DB<10> T
$ = DB​::fake​::at_exit() called from file `/usr/local/lib/perl5/5.8.8/perl5db.pl' line 8993
$ = DB​::END() called from file `/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl' line
3006
$ = eval {...} called from file `/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl' line
3006
DB<10>
---

i would like to point out it again, the problem present only on FreeBSD 6.x with Perl5.8.x (Perl5.6.x not affected)

The problem can be reproduced with the script below, a low memory
limit (say, 30MB) and a perl compiled with usemymalloc=y​:

#!perl
my $block=" "x100_000;
for (1..50) {
  my $bla = "";
  for (1..10) {
  $bla .= $block;
  }
  undef $bla;
# warn join(",",currmem());
}
__END__

The memory growth looks like this​:
$ limits -d 30000000 perl ~/trash/memleak_usemymalloc.pl
8642560,1880064 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
9695232,2932736 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
10747904,3985408 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
11800576,5038080 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
12853248,6090752 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
13905920,7143424 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
14958592,8196096 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
16011264,9248768 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
17063936,10301440 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
18116608,11354112 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
19169280,12406784 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
20221952,13459456 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
21274624,14512128 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
22327296,15564800 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
23379968,16617472 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
24432640,17670144 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
25485312,18722816 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
26537984,19775488 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
27590656,20828160 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
28643328,21880832 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
29696000,22933504 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
30748672,23986176 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
31801344,25038848 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
32862208,26099712 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
33914880,27152384 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
34967552,28205056 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
36020224,29257728 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
36548608,29786112 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
36814848,30052352 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
36950016,30187520 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
37019648,30257152 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
37056512,30294016 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
Out of memory during request for 10016 bytes, total sbrk() is 29753344 bytes!

With a system-malloc (here​: FreeBSD-6.2) there's no leak​:

$ limits -d 30000000 perl ~/trash/memleak_usemymalloc.pl
8646656,1875968 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
9699328,2928640 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
9699328,2928640 at /home/e/eserte/trash/memleak_usemymalloc.pl line 14.
... etc. until end of script ...

The bug is triggered by the special allocation pattern​: the initial
memory block is reallocated many times and gets so large that it is
transferred from one bucket into the next higher bucket. At the end of
the loop iteration this memory block is completely freed.
Unfortunately the max_bucket variable is not updated in the bucket
transfer step. This variable is used as an upper limit in the
get_from_bigger_buckets() function in malloc.c. So large amounts of
free memory sit there and will never be used (this can be seen by
looking at Perl_dump_mstats).

The problem can be avoided even with pure perl​: just allocate
the largest expected memory block at the beginning, say using
$x=" "x$memory_size;

And finally the patch​:

Inline Patch
--- ../bleedperl-amd64/malloc.c	Mon Apr 16 01:25:50 2007
+++ ./malloc.c	Thu Oct 25 23:37:43 2007
@@ -2285,6 +2285,7 @@ Perl_realloc(void *mp, size_t nbytes)
 		nmalloc[bucket]--;
 		nmalloc[pow * BUCKETS_PER_POW2]++;
 #endif 	    
+		max_bucket = pow * BUCKETS_PER_POW2;
 		*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
 		MALLOC_UNLOCK;
 		goto inplace_label;


Regards,   Slaven

--
Slaven Rezic - slaven <at> rezic <dot> de

  tkruler - Perl/Tk program for measuring screen distances
  http​://ptktools.sourceforge.net/#tkruler

@p5pRT
Copy link
Author

p5pRT commented Oct 25, 2007

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Oct 26, 2007

From @eserte

On Do. 25. Okt. 2007, 15​:22​:51, slaven@​rezic.de wrote​:

"zeus@​dn.farlep.net (via RT)" <perlbug-followup@​perl.org> writes​:

PATCH below!

# New Ticket Created by zeus@​dn.farlep.net
# Please include the string​: [perl #46381]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=46381 >

This is a bug report for perl from zeus@​dn.farlep.net,
generated with the help of perlbug 1.35 running under perl v5.8.8.

-----------------------------------------------------------------
[Please enter your report here]

i've got reproduceble picture​:

FreeBSD-6.x + Perl5.8.8 on i386, are affected
while Linux/Windows are not ...

the script using Image​::ExifTool to manipulate EXIF data in batch
mode

the problems looks so​:
---
Out of memory during "large" request for 2101248 bytes, total sbrk()
is
533565440 bytes at
/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl line
2925,
<EXIFTOOL_FILE2> chunk 12.
---

process memory grows and finaly it is kicked :(

yes, ulimit -a shows​: data seg size is 524288

but why run under linux the process frees memory? and under M$Win it
works?

on amd64 the picture differs, top shows​:
---
last pid​: 1019; load averages​: 0.49, 0.25, 0.10 up
0+02​:04​:25 10​:06​:21
43 processes​: 1 running, 42 sleeping
CPU states​: 28.6% user, 0.0% nice, 1.1% system, 0.8% interrupt,
69.5%
idle
Mem​: 293M Active, 27M Inact, 73M Wired, 18M Cache, 54M Buf, 892K
Free
Swap​: 2048M Total, 1034M Used, 1014M Free, 50% Inuse, 460K Out

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
988 zeus 1 -8 0 1865M 301M biord 1 1​:11 34.18%
perl
nd the process finishes successfully

while trying to meditate upon the cause, i've launched the debuger
and here the last lines of output​:

---
...

Image​::ExifTool​::WriteJPEG(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3446)​:

3446​: Write($outfile, $buff) or $err = 1,
last;

Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3002)​:

3002​: my $outfile = shift;

Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3003)​:

3003​: if (UNIVERSAL​::isa($outfile,'GLOB')) {

Image​::ExifTool​::Write(/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl​:3006)​:

3006​: $$outfile .= join('', @​_);
Out of memory during request for 16480 bytes, total sbrk() is
533696512
bytes!
Debugged program terminated. Use q to quit or R to restart,
use o inhibit_exit to avoid stopping after program termination,
h q, h R or h o to get additional info.
DB<10> T
$ = DB​::fake​::at_exit() called from file
`/usr/local/lib/perl5/5.8.8/perl5db.pl' line 8993
$ = DB​::END() called from file
`/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl' line
3006
$ = eval {...} called from file
`/usr/local/lib/perl5/site_perl/5.8.8/Image/ExifTool/Writer.pl' line
3006
DB<10>
---

i would like to point out it again, the problem present only on
FreeBSD 6.x with Perl5.8.x (Perl5.6.x not affected)

The problem can be reproduced with the script below, a low memory
limit (say, 30MB) and a perl compiled with usemymalloc=y​:

#!perl
my $block=" "x100_000;
for (1..50) {
my $bla = "";
for (1..10) {
$bla .= $block;
}
undef $bla;
# warn join(",",currmem());
}
__END__

The memory growth looks like this​:
$ limits -d 30000000 perl ~/trash/memleak_usemymalloc.pl
8642560,1880064 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
9695232,2932736 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
10747904,3985408 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
11800576,5038080 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
12853248,6090752 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
13905920,7143424 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
14958592,8196096 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
16011264,9248768 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
17063936,10301440 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
18116608,11354112 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
19169280,12406784 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
20221952,13459456 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
21274624,14512128 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
22327296,15564800 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
23379968,16617472 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
24432640,17670144 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
25485312,18722816 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
26537984,19775488 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
27590656,20828160 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
28643328,21880832 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
29696000,22933504 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
30748672,23986176 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
31801344,25038848 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
32862208,26099712 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
33914880,27152384 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
34967552,28205056 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
36020224,29257728 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
36548608,29786112 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
36814848,30052352 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
36950016,30187520 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
37019648,30257152 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
37056512,30294016 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
Out of memory during request for 10016 bytes, total sbrk() is 29753344
bytes!

With a system-malloc (here​: FreeBSD-6.2) there's no leak​:

$ limits -d 30000000 perl ~/trash/memleak_usemymalloc.pl
8646656,1875968 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
9699328,2928640 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
9699328,2928640 at /home/e/eserte/trash/memleak_usemymalloc.pl line
14.
... etc. until end of script ...

The bug is triggered by the special allocation pattern​: the initial
memory block is reallocated many times and gets so large that it is
transferred from one bucket into the next higher bucket. At the end of
the loop iteration this memory block is completely freed.
Unfortunately the max_bucket variable is not updated in the bucket
transfer step. This variable is used as an upper limit in the
get_from_bigger_buckets() function in malloc.c. So large amounts of
free memory sit there and will never be used (this can be seen by
looking at Perl_dump_mstats).

The problem can be avoided even with pure perl​: just allocate
the largest expected memory block at the beginning, say using
$x=" "x$memory_size;

And finally the patch​:

--- ../bleedperl-amd64/malloc.c Mon Apr 16 01​:25​:50 2007
+++ ./malloc.c Thu Oct 25 23​:37​:43 2007
@​@​ -2285,6 +2285,7 @​@​ Perl_realloc(void *mp, size_t nbytes)
nmalloc[bucket]--;
nmalloc[pow * BUCKETS_PER_POW2]++;
#endif
+ max_bucket = pow * BUCKETS_PER_POW2;
*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
MALLOC_UNLOCK;
goto inplace_label;

Regards,
Slaven

Don't apply this patch. There must be a condition before the newly
inserted line. I'll make another one tonight...

Regards,
  Slaven

@p5pRT
Copy link
Author

p5pRT commented Oct 26, 2007

From @eserte

Slaven Rezic <slaven@​rezic.de> writes​:

"zeus@​dn.farlep.net (via RT)" <perlbug-followup@​perl.org> writes​:

PATCH below!

[...]

And finally the patch​:

--- ../bleedperl-amd64/malloc.c Mon Apr 16 01​:25​:50 2007
+++ ./malloc.c Thu Oct 25 23​:37​:43 2007
@​@​ -2285,6 +2285,7 @​@​ Perl_realloc(void *mp, size_t nbytes)
nmalloc[bucket]--;
nmalloc[pow * BUCKETS_PER_POW2]++;
#endif
+ max_bucket = pow * BUCKETS_PER_POW2;
*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
MALLOC_UNLOCK;
goto inplace_label;

Here the revised patch​:

Inline Patch
--- ../bleedperl-amd64/malloc.c	Mon Apr 16 01:25:50 2007
+++ ./malloc.c	Fri Oct 26 16:14:10 2007
@@ -2285,6 +2285,8 @@ Perl_realloc(void *mp, size_t nbytes)
 		nmalloc[bucket]--;
 		nmalloc[pow * BUCKETS_PER_POW2]++;
 #endif 	    
+		if (pow * BUCKETS_PER_POW2 > max_bucket)
+		    max_bucket = pow * BUCKETS_PER_POW2;
 		*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
 		MALLOC_UNLOCK;
 		goto inplace_label;


-- 

Slaven Rezic - slaven <at> rezic <dot> de

Tk-AppMaster​: a perl/Tk module launcher designed for handhelds
  http​://tk-appmaster.sf.net

@p5pRT
Copy link
Author

p5pRT commented Jun 21, 2008

From @rgs

2007/10/26 Slaven Rezic <slaven@​rezic.de>​:

Here the revised patch​:

--- ../bleedperl-amd64/malloc.c Mon Apr 16 01​:25​:50 2007
+++ ./malloc.c Fri Oct 26 16​:14​:10 2007
@​@​ -2285,6 +2285,8 @​@​ Perl_realloc(void *mp, size_t nbytes)
nmalloc[bucket]--;
nmalloc[pow * BUCKETS_PER_POW2]++;
#endif
+ if (pow * BUCKETS_PER_POW2 > max_bucket)
+ max_bucket = pow * BUCKETS_PER_POW2;
*(cp - M_OVERHEAD) = pow * BUCKETS_PER_POW2; /* Fill index. */
MALLOC_UNLOCK;
goto inplace_label;

Thanks, applied as #34076.

@p5pRT
Copy link
Author

p5pRT commented Jun 21, 2008

@rgs - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant