From nico-kw-perlbug.be9b7b@gianniotis.org

The following code fragment, from the FAQ, exhibits a memory leak for
large numbers of input files​:

open(FIND, "find /terrabyte1/ro -type f -print |") or die;;
while (<FIND>)
{
  chop;
  open(PLAIN_FILE, $_) or die;
  $checksum = do
  {
  # something here leaks memory
  local $/; # slurp!
  unpack("%32C*", <PLAIN_FILE>) % 0xFFFF;
  };
  print "$checksum $_\n";
  close(PLAIN_FILE);
}
close(FIND);

My data set is approx 161,000 files totaling slightly over 1 Tb. When
this script attempts to checksum each file, I notice the process size
and RSS grows monotonically until vm is exhausted.

If I refactor the do {} block to not "slurp" the file in but compute
the checksum via a loop with multiple calls to sysread(), the process
maintains a stable size (approx 3Mb) throughout its entire life.

Platform details​:

% perl -V
Summary of my perl5 (revision 5 version 8 subversion 4) configuration​:
  Platform​:
  osname=solaris, osvers=2.8, archname=i86pc-solaris-64int
  uname='sunos kamakura 5.8 generic_108529-12 i86pc i386 i86pc '
  config_args='-de -Dcc=gcc -Dprefix=/usr/local -Duse64bitint -Duselargefiles'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=define use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O',
  cppflags='-fno-strict-aliasing -I/usr/local/include'
  ccversion='', gccversion='2.95.3 20010315 (release)', gccosandvers='solaris2.8'
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -L/usr/local/lib '
  libpth=/usr/local/lib /usr/lib /usr/ccs/lib
  libs=-lsocket -lnsl -ldl -lm -lc
  perllibs=-lsocket -lnsl -ldl -lm -lc
  libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
  cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​: USE_64_BIT_INT USE_LARGE_FILES
  Built under solaris
  Compiled at Jun 25 2004 19​:08​:06
  %ENV​:
  PERL5LIB="/home/nico/work/ascentuate/cvs/usma/lib/perl"
  @​INC​:
  /home/nico/work/ascentuate/cvs/usma/lib/perl
  /usr/local/lib/perl5/5.8.4/i86pc-solaris-64int
  /usr/local/lib/perl5/5.8.4
  /usr/local/lib/perl5/site_perl/5.8.4/i86pc-solaris-64int
  /usr/local/lib/perl5/site_perl/5.8.4
  /usr/local/lib/perl5/site_perl/5.6.1
  /usr/local/lib/perl5/site_perl/5.005
  /usr/local/lib/perl5/site_perl
  .

From @smpeters

[nico-kw-perlbug.be9b7b@​gianniotis.org - Sat Mar 12 16​:59​:56 2005]​:

The following code fragment, from the FAQ, exhibits a memory leak for
large numbers of input files​:

open(FIND, "find /terrabyte1/ro -type f -print |") or die;;
while (<FIND>)
{
chop;
open(PLAIN_FILE, $_) or die;
$checksum = do
{
# something here leaks memory
local $/; # slurp!
unpack("%32C*", <PLAIN_FILE>) % 0xFFFF;
};
print "$checksum $_\n";
close(PLAIN_FILE);
}
close(FIND);

My data set is approx 161,000 files totaling slightly over 1 Tb. When
this script attempts to checksum each file, I notice the process size
and RSS grows monotonically until vm is exhausted.

If I refactor the do {} block to not "slurp" the file in but compute
the checksum via a loop with multiple calls to sysread(), the process
maintains a stable size (approx 3Mb) throughout its entire life.

Valgrind seems to confirm the leaks.

==9320== warning​: Valgrind's siglongjmp is incomplete
==9320== (it ignores cleanup handlers)
==9320== your program may misbehave as a result
==9320==
==9320== ERROR SUMMARY​: 0 errors from 0 contexts (suppressed​: 20 from 1)
--9320--
--9320-- supp​: 20 dl_relocate_object/dl_main
==9320== malloc/free​: in use at exit​: 225479 bytes in 607 blocks.
==9320== malloc/free​: 16541 allocs, 15934 frees, 131130187 bytes allocated.
==9320==
==9320== searching for pointers to 607 not-freed blocks.
==9320== checked 3329276 bytes.
==9320==
==9320== 5 bytes in 1 blocks are definitely lost in loss record 1 of 9
==9320== at 0x1B904A90​: malloc (vg_replace_malloc.c​:131)
==9320== by 0x32C6E94​: Perl_savesharedpv (in
/usr/lib/perl5/5.8.5/i386-linux-
thread-multi/CORE/libperl.so)
==9320== by 0x3274353​: (within
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/C
ORE/libperl.so)
==9320== by 0x327827B​: perl_parse (in
/usr/lib/perl5/5.8.5/i386-linux-thread-
multi/CORE/libperl.so)
==9320==
==9320==
==9320== 1192 bytes in 1 blocks are definitely lost in loss record 5 of 9
==9320== at 0x1B9054FA​: realloc (vg_replace_malloc.c​:197)
==9320== by 0x32C892B​: Perl_safesysrealloc (in
/usr/lib/perl5/5.8.5/i386-linu
x-thread-multi/CORE/libperl.so)
==9320== by 0x330B818​: Perl_savestack_grow (in
/usr/lib/perl5/5.8.5/i386-linu
x-thread-multi/CORE/libperl.so)
==9320== by 0x330B9B8​: Perl_save_sptr (in
/usr/lib/perl5/5.8.5/i386-linux-thr
ead-multi/CORE/libperl.so)
==9320==
==9320==
==9320== 8968 bytes in 9 blocks are possibly lost in loss record 7 of 9
==9320== at 0x1B904A90​: malloc (vg_replace_malloc.c​:131)
==9320== by 0x32C7C1A​: Perl_safesysmalloc (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x32E4FCA​: (within
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/C
ORE/libperl.so)
==9320== by 0x32E70F8​: Perl_sv_upgrade (in
/usr/lib/perl5/5.8.5/i386-linux-th
read-multi/CORE/libperl.so)
==9320==
==9320==
==9320== 132726 bytes in 11 blocks are definitely lost in loss record 9 of 9
==9320== at 0x1B904A90​: malloc (vg_replace_malloc.c​:131)
==9320== by 0x32C7C1A​: Perl_safesysmalloc (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x330B4BF​: Perl_new_stackinfo (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x3271DAA​: Perl_init_stacks (in
/usr/lib/perl5/5.8.5/i386-linux-t
hread-multi/CORE/libperl.so)
==9320==
==9320== LEAK SUMMARY​:
==9320== definitely lost​: 133923 bytes in 13 blocks.
==9320== possibly lost​: 8968 bytes in 9 blocks.
==9320== still reachable​: 82388 bytes in 584 blocks.
==9320== suppressed​: 200 bytes in 1 blocks.
==9320== Reachable blocks (those to which a pointer was found) are not
shown.
==9320== To see them, rerun with​: --show-reachable=yes
--9320-- TT/TC​: 0 tc sectors discarded.
--9320-- 56488 tt_fast misses.
--9320-- translate​: new 9010 (153873 -> 1997185; ratio 129​:10)
--9320-- discard 1 (23 -> 320; ratio 139​:10).
--9320-- chainings​: 6458 chainings, 2 unchainings.
--9320-- dispatch​: 373000000 jumps (bb entries); of them 16024882 (4%)
unchaine
d.
--9320-- 121827/1722921 major/minor sched events.
--9320-- reg-alloc​: 1835 t-req-spill, 367423+12618 orig+spill uis,
--9320-- 46852 total-reg-rank
--9320-- sanity​: 117523 cheap, 4701 expensive checks.
--9320-- ccalls​: 36820 C calls, 56% saves+restores avoided (123274 bytes)
--9320-- 49527 args, avg 0.89 setup instrs each (10700 bytes)
--9320-- 0% clear the stack (110208 bytes)
--9320-- 15121 retvals, 34% of reg-reg movs avoided (10196 bytes)

The RT System itself - Status changed from 'new' to 'open'

From @smpeters

[stmpeters - Mon Mar 21 08​:17​:22 2005]​:

[nico-kw-perlbug.be9b7b@​gianniotis.org - Sat Mar 12 16​:59​:56 2005]​:

The following code fragment, from the FAQ, exhibits a memory leak for
large numbers of input files​:

open(FIND, "find /terrabyte1/ro -type f -print |") or die;;
while (<FIND>)
{
chop;
open(PLAIN_FILE, $_) or die;
$checksum = do
{
# something here leaks memory
local $/; # slurp!
unpack("%32C*", <PLAIN_FILE>) % 0xFFFF;
};
print "$checksum $_\n";
close(PLAIN_FILE);
}
close(FIND);

Valgrind seems to confirm the leaks.

==9320== warning​: Valgrind's siglongjmp is incomplete
==9320== (it ignores cleanup handlers)
==9320== your program may misbehave as a result
==9320==
==9320== ERROR SUMMARY​: 0 errors from 0 contexts (suppressed​: 20 from 1)
--9320--
--9320-- supp​: 20 dl_relocate_object/dl_main
==9320== malloc/free​: in use at exit​: 225479 bytes in 607 blocks.
==9320== malloc/free​: 16541 allocs, 15934 frees, 131130187 bytes
allocated.
==9320==
==9320== searching for pointers to 607 not-freed blocks.
==9320== checked 3329276 bytes.
==9320==
==9320== 5 bytes in 1 blocks are definitely lost in loss record 1 of 9
==9320== at 0x1B904A90​: malloc (vg_replace_malloc.c​:131)
==9320== by 0x32C6E94​: Perl_savesharedpv (in
/usr/lib/perl5/5.8.5/i386-linux-
thread-multi/CORE/libperl.so)
==9320== by 0x3274353​: (within
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/C
ORE/libperl.so)
==9320== by 0x327827B​: perl_parse (in
/usr/lib/perl5/5.8.5/i386-linux-thread-
multi/CORE/libperl.so)
==9320==
==9320==
==9320== 1192 bytes in 1 blocks are definitely lost in loss record 5 of 9
==9320== at 0x1B9054FA​: realloc (vg_replace_malloc.c​:197)
==9320== by 0x32C892B​: Perl_safesysrealloc (in
/usr/lib/perl5/5.8.5/i386-linu
x-thread-multi/CORE/libperl.so)
==9320== by 0x330B818​: Perl_savestack_grow (in
/usr/lib/perl5/5.8.5/i386-linu
x-thread-multi/CORE/libperl.so)
==9320== by 0x330B9B8​: Perl_save_sptr (in
/usr/lib/perl5/5.8.5/i386-linux-thr
ead-multi/CORE/libperl.so)
==9320==
==9320==
==9320== 8968 bytes in 9 blocks are possibly lost in loss record 7 of 9
==9320== at 0x1B904A90​: malloc (vg_replace_malloc.c​:131)
==9320== by 0x32C7C1A​: Perl_safesysmalloc (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x32E4FCA​: (within
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/C
ORE/libperl.so)
==9320== by 0x32E70F8​: Perl_sv_upgrade (in
/usr/lib/perl5/5.8.5/i386-linux-th
read-multi/CORE/libperl.so)
==9320==
==9320==
==9320== 132726 bytes in 11 blocks are definitely lost in loss record
9 of 9
==9320== at 0x1B904A90​: malloc (vg_replace_malloc.c​:131)
==9320== by 0x32C7C1A​: Perl_safesysmalloc (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x330B4BF​: Perl_new_stackinfo (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x3271DAA​: Perl_init_stacks (in
/usr/lib/perl5/5.8.5/i386-linux-t
hread-multi/CORE/libperl.so)
==9320==
==9320== LEAK SUMMARY​:
==9320== definitely lost​: 133923 bytes in 13 blocks.
==9320== possibly lost​: 8968 bytes in 9 blocks.
==9320== still reachable​: 82388 bytes in 584 blocks.
==9320== suppressed​: 200 bytes in 1 blocks.
==9320== Reachable blocks (those to which a pointer was found) are not
shown.
==9320== To see them, rerun with​: --show-reachable=yes
--9320-- TT/TC​: 0 tc sectors discarded.
--9320-- 56488 tt_fast misses.
--9320-- translate​: new 9010 (153873 -> 1997185; ratio 129​:10)
--9320-- discard 1 (23 -> 320; ratio 139​:10).
--9320-- chainings​: 6458 chainings, 2 unchainings.
--9320-- dispatch​: 373000000 jumps (bb entries); of them 16024882 (4%)
unchaine
d.
--9320-- 121827/1722921 major/minor sched events.
--9320-- reg-alloc​: 1835 t-req-spill, 367423+12618 orig+spill uis,
--9320-- 46852 total-reg-rank
--9320-- sanity​: 117523 cheap, 4701 expensive checks.
--9320-- ccalls​: 36820 C calls, 56% saves+restores avoided (123274
bytes)
--9320-- 49527 args, avg 0.89 setup instrs each (10700 bytes)
--9320-- 0% clear the stack (110208 bytes)
--9320-- 15121 retvals, 34% of reg-reg movs avoided (10196
bytes)

Wow! I had been periodically checking this, but something has changed
dramatically within the past few days in bleadperl. Here's what it
looks like now.

==22372== ERROR SUMMARY​: 0 errors from 0 contexts (suppressed​: 24 from 1)
==22372== malloc/free​: in use at exit​: 73921 bytes in 572 blocks.
==22372== malloc/free​: 36725 allocs, 36153 frees, 37204524 bytes allocated.
==22372== For counts of detected errors, rerun with​: -v
==22372== searching for pointers to 572 not-freed blocks.
==22372== checked 608816 bytes.
==22372==
==22372== 1324 (92 direct, 1232 indirect) bytes in 1 blocks are
definitely lost in loss record 1 of 5
==22372== at 0x1B900896​: malloc (vg_replace_malloc.c​:149)
==22372== by 0x80C9397​: Perl_safesysmalloc (util.c​:91)
==22372== by 0x80CC8BC​: Perl_my_setenv (util.c​:1494)
==22372== by 0x80620FA​: perl_parse (perl.c​:1502)
==22372== by 0x805E799​: main (perlmain.c​:101)
==22372==
==22372== LEAK SUMMARY​:
==22372== definitely lost​: 92 bytes in 1 blocks.
==22372== indirectly lost​: 1232 bytes in 21 blocks.
==22372== possibly lost​: 0 bytes in 0 blocks.
==22372== still reachable​: 72597 bytes in 550 blocks.
==22372== suppressed​: 0 bytes in 0 blocks.
==22372== Reachable blocks (those to which a pointer was found) are not
shown.
==22372== To see them, rerun with​: --show-reachable=yes

@smpeters - Status changed from 'open' to 'resolved'