Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sv arenas are never freeded #13610

Closed
p5pRT opened this issue Feb 18, 2014 · 11 comments
Closed

sv arenas are never freeded #13610

p5pRT opened this issue Feb 18, 2014 · 11 comments

Comments

@p5pRT
Copy link

p5pRT commented Feb 18, 2014

Migrated from rt.perl.org#121274 (status was 'rejected')

Searchable as RT121274$

@p5pRT
Copy link
Author

p5pRT commented Feb 18, 2014

From @bulk88

Created by @bulk88

From an IRC discussion today about why perl doesn't free any mem after
allocing 100,000,000 SVs. I've attached the script in the gist for
archive reasons.

------------------------------------------------------------
[07​:48] <azawawi> Is there any particular reason why Perl does not
release memory back to the OS as in
https://gist.github.com/azawawi/9070264#file-test_big_array-pl
------------------------------------------------------------

I wrote up the following test xsub. ran as "perl -MLocal​::XS2
-E"Local​::XS​::SVAllocTest()"".
------------------------------------------------------------
void
SVAllocTest()
PREINIT​:
  SV ** svarr;
  const U32 svcnt = 50000000;
  U32 i;
CODE​:
  system("echo Check mem usage now, this is before minimum mem usage &
pause"); /* windows only */
  Newxz(svarr, svcnt, SV *);
  for(i=0; i < svcnt; i++) {
  svarr[i] = newSV(0);
  }
  system("echo Check mem usage now, this is peak mem & pause"); /*
windows only */
  for(i=0; i < svcnt; i++) {
  SvREFCNT_dec_NN(svarr[i]);
  }
  system("echo Check mem usage now, this should be less mem, SVs r
free, arr not & pause"); /* windows only */
  Safefree(svarr);
  system("echo Check mem usage now, this should be less mem,
everything freed & pause"); /* windows only */
-----------------------------------------------------------

I will be using Process Explorer "private bytes" for ram usage statistic
and blead perl (see botton of this post).

At "Check mem usage now, this is before minimum mem usage", 920 KB.
At "Check mem usage now, this is peak mem", 989,300 KB.
At "Check mem usage now, this should be less mem, SVs r free, arr not",
989,300 KB.
At "Check mem usage now, this should be less mem, everything freed",
793,796 KB.

About the Newxz'd array, array was 50 000 000 * 4 = 200 000 000 bytes
long. Actual memory drop when Safefree()ing that array 989300 KB -
793796 KB = 195504 KB, 195504 * 1024 = 200 196 096 bytes, close enough.

I have a suspect there is no code path to ever free() SV arena block
except by interp exit. I dont really have an idea how to GC arenas.
refcount each arena's usage, free() it the moment it hits zero. Have a
global live SV counter, run the GC reaper every time "(PL_svlivecnt &
0xffff) == 0"? I see some algorithm problems if some PP code
continously stradles that line triggering the GC arena reaper on every
pp_nextstate. Or have a counter called PL_svallocfreecount that ticks
upwards on every SV free and SV alloc, and overflows eventually, and
every 65K ticks reaps the arenas?

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.19.9:

Configured by Owner at Wed Feb 12 06:47:30 2014.

Summary of my perl5 (revision 5 version 19 subversion 9) configuration:
  Derived from: 633f0fd2ca244ca83cc99b3af3a7d3ac2931850b
  Platform:
    osname=MSWin32, osvers=5.1, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -Od -MD -Zi -DDEBUGGING -DWIN32 
-D_CONSOLE -DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS 
-DPERL_HASH_FUNC_ONE_AT_A_TIME -DPERL_IMPLICIT_CONTEXT 
-DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T',
    optimize='-Od -MD -Zi -DDEBUGGING',
    cppflags='-DWIN32'
    ccversion='13.10.6030', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', 
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug  
-libpath:"c:\perl519\lib\CORE"  -machine:x86'
    libpth="C:\Program Files\Microsoft Visual Studio .NET 2003\VC7\lib"
    libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib 
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib 
winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib 
oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  
version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl519.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug  
-libpath:"c:\perl519\lib\CORE"  -machine:x86'

Locally applied patches:
    uncommitted-changes


@INC for perl 5.19.9:
    C:/perl519/site/lib
    C:/perl519/lib
    .


Environment for perl 5.19.9:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\perl519\bin;C:\Program Files\Microsoft Visual Studio .NET 
2003\Common7\IDE;C:\Program Files\Microsoft Visual Studio .NET 
2003\VC7\BIN;C:\Program Files\Microsoft Visual Studio .NET 
2003\Common7\Tools;C:\Program Files\Microsoft Visual Studio .NET 
2003\Common7\Tools\bin\prerelease;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\wbem;
    PERL_BADLANG (unset)
    SHELL (unset)


@p5pRT
Copy link
Author

p5pRT commented Feb 18, 2014

From @bulk88

test_big_array.pl

@p5pRT
Copy link
Author

p5pRT commented Feb 19, 2014

From zefram@fysh.org

bulk88 wrote​:

for(i=0; i < svcnt; i++) {
svarr[i] = newSV(0);
}
system("echo Check mem usage now, this is peak mem & pause"); /*
windows only */
for(i=0; i < svcnt; i++) {
SvREFCNT_dec_NN(svarr[i]);
}

This is a very unusual pattern of memory usage, and it's not worth
optimising for it. More commonly these quickly-freed SVs would be
interspersed with some that have a longer lifetime and so prevent the
freeing of that block of pages.

-zefram

@p5pRT
Copy link
Author

p5pRT commented Feb 19, 2014

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Feb 27, 2014

From @bulk88

On Wed Feb 19 02​:20​:46 2014, zefram@​fysh.org wrote​:

This is a very unusual pattern of memory usage, and it's not worth
optimising for it. More commonly these quickly-freed SVs would be
interspersed with some that have a longer lifetime and so prevent the
freeing of that block of pages.

-zefram

I disagree. If you allocate an AV with atleast (2^16)/16 = 4096 slices/SV*s, then free the AV. The memory won't be released. There are numerous public complaints through out Perl 5's history about "perl never releases memory" to the OS. Loading large data files (>100 KB) like CSVs, JSON, and the like, into perl data structures and transforming or searching them, with the operation taking atleast 1 second of full throttle I/O and/or CPU usage, is a common operation.

To address what I think will be the response. Processing each data file/task in a fork is not a solution. It is a bandaid to a design bug in Perl. Perl should be able to goto 1 or 2 GB of memory to process a huge data set, then scale back down to, below, 150% of the memory usage before the task/data set job began. Not stay at 1 GB or 400% even though all variables/data belonging to the task were freed on a PP level.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Feb 27, 2014

From @iabyn

On Wed, Feb 26, 2014 at 11​:35​:09PM -0800, bulk88 via RT wrote​:

To address what I think will be the response. Processing each data
file/task in a fork is not a solution. It is a bandaid to a design bug
in Perl. Perl should be able to goto 1 or 2 GB of memory to process a
huge data set, then scale back down to, below, 150% of the memory usage
before the task/data set job began. Not stay at 1 GB or 400% even though
all variables/data belonging to the task were freed on a PP level.

But that is likely to be almost entirely down to the system's malloc()
implementation rather than perl's SV arena usage.

Yes, perl will keep SV head and bodies around for future use, but it
free()s any PVX() structures etc. If the system's malloc() can consolidate
and unmap() freed chunks, then good for it. But that's outside our
control.

--
My Dad used to say 'always fight fire with fire', which is probably why
he got thrown out of the fire brigade.

@p5pRT
Copy link
Author

p5pRT commented Feb 27, 2014

From @bulk88

On Thu Feb 27 09​:09​:47 2014, davem wrote​:

On Wed, Feb 26, 2014 at 11​:35​:09PM -0800, bulk88 via RT wrote​:

To address what I think will be the response. Processing each data
file/task in a fork is not a solution. It is a bandaid to a design bug
in Perl. Perl should be able to goto 1 or 2 GB of memory to process a
huge data set, then scale back down to, below, 150% of the memory usage
before the task/data set job began. Not stay at 1 GB or 400% even though
all variables/data belonging to the task were freed on a PP level.

But that is likely to be almost entirely down to the system's malloc()
implementation rather than perl's SV arena usage.

No, AFAIK perl's SV arena usage never calls free on any arena blocks until interp exit. So this memory usage has nothing to do with malloc implementation since free() was never called on it. Calling free() is perl's domain, not libc's.

Yes, perl will keep SV head and bodies around for future use

Forever? That is a leak.

but it
free()s any PVX() structures etc. If the system's malloc() can consolidate
and unmap() freed chunks, then good for it. But that's outside our
control.

This bug isn't about PVX buffers. None were allocated in the sample XS code in this ticket.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Feb 28, 2014

From @iabyn

On Thu, Feb 27, 2014 at 09​:36​:46AM -0800, bulk88 via RT wrote​:

On Thu Feb 27 09​:09​:47 2014, davem wrote​:

On Wed, Feb 26, 2014 at 11​:35​:09PM -0800, bulk88 via RT wrote​:

To address what I think will be the response. Processing each data
file/task in a fork is not a solution. It is a bandaid to a design bug
in Perl. Perl should be able to goto 1 or 2 GB of memory to process a
huge data set, then scale back down to, below, 150% of the memory usage
before the task/data set job began. Not stay at 1 GB or 400% even though
all variables/data belonging to the task were freed on a PP level.

But that is likely to be almost entirely down to the system's malloc()
implementation rather than perl's SV arena usage.

No, AFAIK perl's SV arena usage never calls free on any arena blocks
until interp exit. So this memory usage has nothing to do with malloc
implementation since free() was never called on it. Calling free() is
perl's domain, not libc's.

I'm referring to the fact that in the real world, most code that uses
large amounts of memory have something having off each SV, like a PVX. So
in the real world, the issue of whether most of this memory is released to
the OS is dependent on whether the OS consolidates and unmmaps free()d
strings rather than whether perl free()s SV arenas.

Yes, perl will keep SV head and bodies around for future use

Forever? That is a leak.

No, that's efficiency. Its a chosen trade-off. Like PADTMPs;

This bug isn't about PVX buffers. None were allocated in the sample XS
code in this ticket.

And I feel that the sample XS code isn't representative of typical large
memory footprint perl applications.

Look, if you can come up with a scheme that can detect completely
empty arenas and free them, that *doesn't* slow down the allocation or
freeing of each SV, and doesn't have un-perlish random pauses for GC
sweeps, then I would consider including in core. But I just think for the
reasons I and Zefram mentioned, the reality is that in practice users
won't see more memory reclaimed very often.

--
My Dad used to say 'always fight fire with fire', which is probably why
he got thrown out of the fire brigade.

@p5pRT
Copy link
Author

p5pRT commented Mar 1, 2014

From @bulk88

On Fri Feb 28 03​:30​:36 2014, davem wrote​:

Look, if you can come up with a scheme that can detect completely
empty arenas and free them, that *doesn't* slow down the allocation or
freeing of each SV, and doesn't have un-perlish random pauses for GC
sweeps, then I would consider including in core. But I just think for
the
reasons I and Zefram mentioned, the reality is that in practice users
won't see more memory reclaimed very often.

Should I write a GC for http​://perl5.git.perl.org/perl.git/blob/HEAD​:/lib/less.pm ?

eval "use less 'memory'";

will actually do something. GCing arenas is 1 thing for less 'memory' to do. Another idea I've had was deallocating unused C stack. The perl compiler has horrible C stack recursion during the optimizer phase, and a dozen pages can be shaved off the C stack (see commented out code at https​://rt.perl.org/Ticket/Attachment/1235621/643133/perlmain.c ) and returned to the OS after the compiler returns, one example of C stack https​://rt.perl.org/Ticket/Attachment/1064560/528860/plstk1.txt which was made changing VM settings so C stack can't grow beyond whatever was allocated at main(). Whole bug https://rt-archive.perl.org/perl5/Ticket/Display.html?id=108276 .

If use less 'memory' runs GC code, then its upto the user to decide when to run the GC and the performance implication with it, not P5P's decision.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Mar 4, 2014

From @iabyn

On Fri, Feb 28, 2014 at 09​:12​:33PM -0800, bulk88 via RT wrote​:

On Fri Feb 28 03​:30​:36 2014, davem wrote​:

Look, if you can come up with a scheme that can detect completely
empty arenas and free them, that *doesn't* slow down the allocation or
freeing of each SV, and doesn't have un-perlish random pauses for GC
sweeps, then I would consider including in core. But I just think for
the
reasons I and Zefram mentioned, the reality is that in practice users
won't see more memory reclaimed very often.

Should I write a GC for http​://perl5.git.perl.org/perl.git/blob/HEAD​:/lib/less.pm ?

I'm really not convinced of its utility.

eval "use less 'memory'";

Why the 'eval'? Are you proposing that each 'use' this module triggers a
one-off GC sweep? Or that code within the scope of "use less 'memory'"
triggers occasional sweeps, e.g. at scope exit?

If the former, then this seems to be a misuse of the semantics of a pragma.

Another idea I've had was deallocating unused C stack. The perl compiler
has horrible C stack recursion during the optimizer phase, and a dozen
pages can be shaved off the C stack

The best thing to do here is to avoid the recursion in the first place. If
the scheme to have the last op_sibling point back to the parent works out
for 5.21, then removing recursion from the depth-first tree walks of
Perl_scalarvoid() etc becomes trivial. Removing recursion from the op_next
chain walking of Perl_rpeep() is more problematic. I've put a mechanism in
place that reduces the likelihood of recursion being needed, but it still
recurses in pathological examples.

If use less 'memory' runs GC code, then its upto the user to decide when
to run the GC and the performance implication with it, not P5P's
decision.

No one's stopping anyone from putting up a CPAN module that does this.

--
Overhead, without any fuss, the stars were going out.
  -- Arthur C Clarke

@p5pRT p5pRT closed this as completed Mar 29, 2017
@p5pRT
Copy link
Author

p5pRT commented Mar 29, 2017

@iabyn - Status changed from 'open' to 'rejected'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant