Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EvalServer leaks threads and memory when using Proc::Async #6527

Open
p6rt opened this issue Sep 16, 2017 · 5 comments
Open

EvalServer leaks threads and memory when using Proc::Async #6527

p6rt opened this issue Sep 16, 2017 · 5 comments
Labels
JVM Related to Rakudo-JVM

Comments

@p6rt
Copy link

p6rt commented Sep 16, 2017

Migrated from rt.perl.org#132104 (status was 'open')

Searchable as RT132104$

@p6rt
Copy link
Author

p6rt commented Sep 16, 2017

From @AlexDaniel

From this discussion https://irclog.perlgeek.de/perl6-dev/2017-09-16#i_15171820

4) the EvalServer seems to leak memory. it's no longer possible to run 'make spectest', even with -Xmx6000m

@p6rt
Copy link
Author

p6rt commented Sep 23, 2017

From @usev6

On Sat, 16 Sep 2017 07​:31​:18 -0700, alex.jakimenko@​gmail.com wrote​:

From this discussion https://irclog.perlgeek.de/perl6-dev/2017-09-
16#i_15171820

4) the EvalServer seems to leak memory. it's no longer possible to run
'make spectest', even with -Xmx6000m

My findings so far​:

1) The EvalServer does not leak memory per se. Assuming a clean EvalServer instance has been started with './perl6-eval-server -cookie TESTTOKEN -app ./perl6.jar' the following does not leak memory​:

$ echo 'say 42;' > foo.p6
$ for i in {1..500}; do ./eval-client.pl TESTTOKEN run ./foo.p6; done

2) Using 'run' (or calling Proc​::Async directly) does not leak memory​:

$ ./perl6-j -e 'for ^5000 { run("echo", "42") }'

3) Feeding the EvalServer with a program that calls 'run' (or Proc​::Async directly) does heavily leak memory, threads and what not (again assuming a clean EvalServer instance started as above)​:

$ echo 'run("echo", "42");' > foo.p6
$ for i in {1..500}; do ./eval-client.pl TESTTOKEN run foo.p6; done

@p6rt
Copy link
Author

p6rt commented Sep 23, 2017

The RT System itself - Status changed from 'new' to 'open'

@p6rt
Copy link
Author

p6rt commented Oct 12, 2017

From @usev6

On Sat, 23 Sep 2017 12​:28​:53 -0700, bartolin@​gmx.de wrote​:

3) Feeding the EvalServer with a program that calls 'run' (or
Proc​::Async directly) does heavily leak memory, threads and what not
(again assuming a clean EvalServer instance started as above)​:

$ echo 'run("echo", "42");' > foo.p6
$ for i in {1..500}; do ./eval-client.pl TESTTOKEN run foo.p6; done

I'm still trying to find the cause of the memory leak. Part of the problem seems to be that with each invokation of './eval-client.pl TESTTOKEN run foo.p6' three or four additional threads are started. Those threads are never stopped.

Some debug statements in src/core/ThreadPoolScheduler.pm indicate that with each invokation (s.a.) a new AffinityWorker instance and a new GeneralWorker instance are created (and at least one other thread). I guess those newly started Threads stay around until the EvalServer is killed.

@p6rt
Copy link
Author

p6rt commented Jan 25, 2018

From @peschwa

On Thu, 12 Oct 2017 13​:35​:27 -0700, bartolin@​gmx.de wrote​:

On Sat, 23 Sep 2017 12​:28​:53 -0700, bartolin@​gmx.de wrote​:

3) Feeding the EvalServer with a program that calls 'run' (or
Proc​::Async directly) does heavily leak memory, threads and what not
(again assuming a clean EvalServer instance started as above)​:

$ echo 'run("echo", "42");' > foo.p6
$ for i in {1..500}; do ./eval-client.pl TESTTOKEN run foo.p6; done

I'm still trying to find the cause of the memory leak. Part of the
problem seems to be that with each invokation of './eval-client.pl
TESTTOKEN run foo.p6' three or four additional threads are started.
Those threads are never stopped.

Some debug statements in src/core/ThreadPoolScheduler.pm indicate that
with each invokation (s.a.) a new AffinityWorker instance and a new
GeneralWorker instance are created (and at least one other thread). I
guess those newly started Threads stay around until the EvalServer is
killed.

I've pushed some improvements for this issue in NQP commit b88de49aad. As the commit message notes, there's probably more out there, but this gets us through spectest on hack.p6c.org at least, with a max RES of around 4.5gb.

@p6rt p6rt added the JVM Related to Rakudo-JVM label Jan 5, 2020
usev6 added a commit to usev6/rakudo that referenced this issue Feb 4, 2021
Since the EvalServer is not in a good shape
(Raku/old-issue-tracker#6527) it makes more
sense to just start a new JVM for each test file. Using the new option
makes it possible to get the old behaviour.
usev6 added a commit to rakudo/rakudo that referenced this issue Feb 8, 2021
Since the EvalServer is not in a good shape
(Raku/old-issue-tracker#6527) it makes more
sense to just start a new JVM for each test file. Using the new option
makes it possible to get the old behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
JVM Related to Rakudo-JVM
Projects
None yet
Development

No branches or pull requests

1 participant