Skip Menu |
Report information
Id: 119937
Status: rejected
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: vsespb <victor [at] vsespb.ru>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: notabug
Perl Version: (no value)
Fixed In: (no value)



Subject: Global destruction sometimes makes linux COW useless
Date: Sun, 22 Sep 2013 01:41:55 +0400
To: perlbug [...] perl.org
From: Victor Efimov <victor [...] vsespb.ru>
Download (untitled) / with headers
text/plain 976b
Below example (with $N high enough - 100 in my case) works fine if I terminate process group with Ctrl-C
but start swapping if I wait for graceful process termination (i.e. with USR1, when exit() called).

I think that's because Linux COW feature - after huge process forks, memory is not copied in real, until modified.
And, it seems, that when Perl global destruction happens, all memory is modified in this particular example.

So I am wondering, maybe it's possible to free simple data structures without modifying them.

=============
#!/usr/bin/perl

use strict;
use warnings;

my $N=20;

my $x={};

$x->{a} = [map { $_ } 1..10_000_000 ];

$SIG{USR1} = sub { print "exiting $$\n"; exit(0); };

my @pids;
for (1..$N) {
  if (my $pid = fork) {
    push @pids, $pid;
  } else {
    while() { sleep 1;}
  }
}

print "PRECC CTRL-C TO EXIT\n";
sleep 5;
print "GRACEFUL EXIT\n";
kill 'USR1', $_ for @pids;
=============

(tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)

Subject: Re: [perl #119937] Global destruction sometimes makes linux COW useless
Date: Mon, 30 Sep 2013 21:00:51 +0100
To: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
On Sat, Sep 21, 2013 at 02:42:18PM -0700, Victor Efimov wrote: Show quoted text
> # New Ticket Created by Victor Efimov > # Please include the string: [perl #119937] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=119937 > > > > Below example (with $N high enough - 100 in my case) works fine if I > terminate process group with Ctrl-C > but start swapping if I wait for graceful process termination (i.e. with > USR1, when exit() called). > > I think that's because Linux COW feature - after huge process forks, memory > is not copied in real, until modified. > And, it seems, that when Perl global destruction happens, all memory is > modified in this particular example. > > So I am wondering, maybe it's possible to free simple data structures > without modifying them.
I can't see how this could be done. Every SV is reference counted, and in the case of your code below, the lexical $x goes out of scope, which refcnt_dec()s the AV pointed to by it, which triggers refcnt_dec()ing each of its members, which frees them, adding them back to the pool of free SVs. Note that this is happening earlier than global destruction. If you made $x a package var, then (assuming it didn't contain any objects, that PERL_DESTRUCT_LEVEL isn't set, and that you're not in a thread), then all the SVs are just abandoned rather than being freed. Your other option would be to use POSIX::_exit. Show quoted text
> > ============= > #!/usr/bin/perl > > use strict; > use warnings; > > my $N=20; > > my $x={}; > > $x->{a} = [map { $_ } 1..10_000_000 ]; > > $SIG{USR1} = sub { print "exiting $$\n"; exit(0); }; > > my @pids; > for (1..$N) { > if (my $pid = fork) { > push @pids, $pid; > } else { > while() { sleep 1;} > } > } > > print "PRECC CTRL-C TO EXIT\n"; > sleep 5; > print "GRACEFUL EXIT\n"; > kill 'USR1', $_ for @pids; > ============= > > (tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)
-- Little fly, thy summer's play my thoughtless hand has terminated with extreme prejudice. (with apologies to William Blake)
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.2k
Indeed, if replace "my $x={};" with "our $x={};", problem seems to go away (RAM usage spike is smaller). Sorry, didn't notice this earlier. I think ticket can be closed then. Thanks. On Mon Sep 30 13:01:38 2013, davem wrote: Show quoted text
> On Sat, Sep 21, 2013 at 02:42:18PM -0700, Victor Efimov wrote:
> > # New Ticket Created by Victor Efimov > > # Please include the string: [perl #119937] > > # in the subject line of all future correspondence about this issue. > > # <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=119937 > > > > > > > Below example (with $N high enough - 100 in my case) works fine if I > > terminate process group with Ctrl-C > > but start swapping if I wait for graceful process termination (i.e.
> with
> > USR1, when exit() called). > > > > I think that's because Linux COW feature - after huge process forks,
> memory
> > is not copied in real, until modified. > > And, it seems, that when Perl global destruction happens, all memory
> is
> > modified in this particular example. > > > > So I am wondering, maybe it's possible to free simple data
> structures
> > without modifying them.
> > I can't see how this could be done. Every SV is reference counted, > and in the case of your code below, the lexical $x goes out of scope, > which refcnt_dec()s the AV pointed to by it, which triggers > refcnt_dec()ing each of its members, which frees them, adding them > back to > the pool of free SVs. > > Note that this is happening earlier than global destruction. If you > made $x a package var, then (assuming it didn't contain any objects, > that PERL_DESTRUCT_LEVEL isn't set, and that you're not in a thread), > then all the SVs are just abandoned rather than being freed. > > Your other option would be to use POSIX::_exit. >
> > > > ============= > > #!/usr/bin/perl > > > > use strict; > > use warnings; > > > > my $N=20; > > > > my $x={}; > > > > $x->{a} = [map { $_ } 1..10_000_000 ]; > > > > $SIG{USR1} = sub { print "exiting $$\n"; exit(0); }; > > > > my @pids; > > for (1..$N) { > > if (my $pid = fork) { > > push @pids, $pid; > > } else { > > while() { sleep 1;} > > } > > } > > > > print "PRECC CTRL-C TO EXIT\n"; > > sleep 5; > > print "GRACEFUL EXIT\n"; > > kill 'USR1', $_ for @pids; > > ============= > > > > (tested with 5.10.1 and 5.19.4. Linux 2.6.38, 16GB ram)
> >


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org