Skip Menu |
Report information
Id: 130494
Status: open
Priority: 0/
Queue: perl6

Owner: jnthn [at] jnthn.net
Requestors: ronaldxs <ronaldxs [at] software-path.com>
Cc:
AdminCc:

Severity: (no value)
Tag: Bug
Platform: (no value)
Patch Status: (no value)
VM: (no value)



Subject: [BUG] Using Proc::Async with tap leads to memory leak
From: Ronald Schmidt <ronaldxs [...] software-path.com>
To: rakudobug [...] perl.org
Date: Tue, 03 Jan 2017 19:33:04 +0100
Download (untitled) / with headers
text/plain 1.1k

The example below is believed to be a simplification of memory leak Issue #1104 for the htmlify.p6 program of http://github.com/perl6/doc.

The program just echoes numbered lines through the UNIX 'cat' program and uses Perl 6 multiprocessing mechanisms to read back the echoed lines.  It leaks 1 Gig of memory on my system fairly quickly.

This issue occurs with Perl 6 doc generation of pod code blocks.  At Pod::To::HTML line 300 a callback mechanism is used which htmlify.p6 uses to format some pod code blocks using Node.js.  The code below does a simplified simulation of those callbacks.

my $proc = Proc::Async.new('/usr/bin/env', 'cat', :r, :w);
my $proc-supply = $proc.stdout.lines;
my $proc-prom = $proc.start;

my $last;
loop (my $i = 0; $i < 10_000; $i++) {

    say $i if $i %% 1_000;

    my $p = Promise.new;

    my $tap = $proc-supply.tap( -> $cat-echo {
        $p.keep($cat-echo);
        $tap.close();
    } );

    $proc.say('line ' ~ $i); # $tmp_fname);
    await $p;
    $last = $p.result;
}

say $last;

 
RT-Send-CC: perl6-compiler [...] perl.org
Download (untitled) / with headers
text/plain 382b
On Tue, 03 Jan 2017 10:33:31 -0800, ronaldxs wrote: Show quoted text
Confirmed on Rakudo version 2016.12-52-g9eed276 built on MoarVM version 2016.12-6-g65acd55 on running on Linux. Memory keeps slowly increasing; got up to 1.1GB by 48000 iterations.
RT-Send-CC: perl6-compiler [...] perl.org
Download (untitled) / with headers
text/plain 581b
On Tue, 03 Jan 2017 12:36:10 -0800, cpan@zoffix.com wrote: Show quoted text
> On Tue, 03 Jan 2017 10:33:31 -0800, ronaldxs wrote:
> > Links: > > ------ > > [1] https://github.com/perl6/doc/issues/1104 > > [2] > > https://github.com/perl6/Pod-To- > > HTML/blob/master/lib/Pod/To/HTML.pm#L300
> > > Confirmed on Rakudo version 2016.12-52-g9eed276 built on MoarVM > version 2016.12-6-g65acd55 on running on Linux. Memory keeps slowly > increasing; got up to 1.1GB by 48000 iterations.
Worth retrying on HEAD, which has a number of fixes and improvements resulting from analyzing this issue. /jnthn
Download (untitled) / with headers
text/plain 344b
Show quoted text
> Worth retrying on HEAD, which has a number of fixes and improvements > resulting from analyzing this issue. > > /jnthn
Both code sample and htmlify.p6 still leak for me. Sorry. ron@ron-laptop:~/.rakudobrew/moar-nom$ perl6 -v This is Rakudo version 2016.12-272-g9a11eac built on MoarVM version 2016.12-100-gf9e14e9 implementing Perl 6.c.
Just to try to quantify for future reference or anyone interested ... my $proc-prom = $proc.start; my $last; loop (my $i = 0; $i < 10_000; $i++) { if ($i == 2 || $i == 10 || $i %% 1_000) { say "pmap after $i iterations"; my $pmap = run('pmap', $*PID, :out); say | $pmap.out.lines.grep(/^\s*total/); } my $p = Promise.new; my $tap = $proc-supply.tap( -> $cat-echo { $p.keep($cat-echo); $tap.close(); } ); $proc.say('line ' ~ $i); # $tmp_fname); await $p; $last = $p.result; } say $last; /*** prints ***/ pmap after 0 iterations total 94952K pmap after 2 iterations total 141824K pmap after 10 iterations total 241568K pmap after 1000 iterations total 475952K pmap after 2000 iterations total 578520K pmap after 3000 iterations total 684888K pmap after 4000 iterations total 797292K pmap after 5000 iterations total 899428K pmap after 6000 iterations total 1012452K pmap after 7000 iterations total 1114832K pmap after 8000 iterations total 1229764K pmap after 9000 iterations total 1333620K line 9999
RT-Send-CC: perl6-compiler [...] perl.org
Download (untitled) / with headers
text/plain 668b
On Tue, 10 Jan 2017 14:30:27 -0800, ronaldxs wrote: Show quoted text
> Both code sample and htmlify.p6 still leak for me. Sorry. >
Yes, I wasn't entirely clear - there was more than one issue, and so fixing the first couple of issues only improved things rather than fully resolved them. I just bumped MOAR_REVISION again to get another set of fixes, which almost completely address the leakage. There is a tiny leak remaining that I'm aware of (about 8 bytes per write), so I'll leave this open for now, but by my measurements it seems to converge around 1000 iterations and leak very little beyond then. (This kind of initial growth and convergence is normal behavior.) /jnthn
Download (untitled) / with headers
text/plain 1.1k
Hi. I now have 2 ubuntu/Rakudo computers, a desktop and an old laptop, and after upgrading both to a current Rakudo, Rakudo version 2017.01-52-g42a1eff built on MoarVM, I see differing results. On the older laptop that showed the problem before I see fairly identical behavior to before the fix. Memory goes from 100k to 1.3Gig at 10_000 iterations and when I step to 18_000 iterations I get 2.3Gig. On the new ubuntu/rakudo running on an i7 under a Windows 10 hypervisor I get the behavior you describe of going to 1Gig before 2000 iterations and then staying stable up through 18_000 iterations. So AFAIK both machines are running rakudo from today / this afternoon ET, and both are running ubuntu 14.04. The computer that gets the results you describe is an intel i7 with 16Gig Ram running Windows 10 and running ubuntu under a hypervisor. The hypervisor has some say in memory management is configured for dynamic memory allocation. The computer that still seems to show a memory leak is a dual pentium laptop with 2Gig of Ram. Both systems have swap. "make html" still runs out of memory in the doc repository on the notebook. If you want me to run any test or diagnostics on either computer please let me know.
One other difference between the laptop and desktop computers that had differing memory leak behavior is that the laptop was running 32 bit Ubuntu and the desktop 64 bit Ubuntu. I retested after noticing the following release note for 2017-02: Fixed overflow during full GC collection on 32-bit systems The fix made me aware of the possibility that 32 bit and 64 bit systems might have different memory management issues. Running with rakudo 2017-02 (slightly different versions) 64 bit shows the behavior I described on my last note of stabilizing at 1 Gig. With 2017-02 32 bit now stabilizes around 460k. I sort of understand that doubling the size of a memory word may double the amount of memory but wonder if there may not be some further room for optimization on 64 bit given 32 bit can get by with half the memory. There is still some slow leakage of a few (~3%) over ten thousand iterations. Also htmlify.p6 can now run on my 2Gig laptop without running out of memory. At this point this leak seams to be at least lower priority and possibly fixed.
Download (untitled) / with headers
text/plain 149b
jnthn mentioned on irc awareness of at least one more leak (presumably) related to this ticket and so ticket waits on news of fixing further leak(s).


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org