From gthomas@netstarnetworks.com

Hi,
  I've attached a program that simulates the problem I'm experiencing. Basically, as I spawn and join threads in a long
running program, I appear to leak away memory. The program shows how this leak manifests itself. It shows the leak on version 5.8.6.

Thanks
Garry

From gthomas@netstarnetworks.com

leaktest.pl

From @iabyn

On Sat, Sep 10, 2005 at 07​:10​:26PM -0700, Garry wrote​:

I've attached a program that simulates the problem I'm experiencing\. Basically\, as I spawn and join threads in a long

running program, I appear to leak away memory. The program shows how this leak manifests itself. It shows the leak on version 5.8.6.

I can reduce the leak to this astonishingly small test case​:

  use threads;

  sub ThreadRoutine {}

  while (1) {
  $thread = threads->new(\&ThreadRoutine);
  $thread->join;
  }

This leaks like a sieve on 5.8.1+ and bleedperl (but not on 5.8.0 funnily
enough).

I haven't got time at the moment to investigate further (lots of other
things I should be doing).

--
Standards (n). Battle insignia or tribal totems.

The RT System itself - Status changed from 'new' to 'open'

From gthomas@netstarnetworks.com

If I put an "undef $thread;" in your example, the leak "seems" to be
plugged....so to speak...

So your example becomes...

  use threads;

  sub ThreadRoutine {}

  while (1) {
  $thread = threads->new(\&ThreadRoutine);
  $thread->join;
  undef $thread;
  }

Is this the expected behaviour? Shouldn't the old thread object be
destroyed automagically? Could this be effecting other object
creation/deletion - ie not just thread objects? Should I always be
doing a "undef $object" in my code?

I'm yet to try this in my code, so I hope its the same leak that I've
been experiencing.

[davem@​iabyn.com - Sun Sep 11 06​:18​:34 2005]​:

On Sat, Sep 10, 2005 at 07​:10​:26PM -0700, Garry wrote​:

I've attached a program that simulates the problem I'm

experiencing. Basically, as I spawn and join threads in a long

running program, I appear to leak away memory. The program shows how
this leak manifests itself. It shows the leak on version 5.8.6.

I can reduce the leak to this astonishingly small test case​:

use threads;

sub ThreadRoutine \{\}

while \(1\) \{
$thread = threads\->new\(\\&ThreadRoutine\);
$thread\->join;
\}

This leaks like a sieve on 5.8.1+ and bleedperl (but not on 5.8.0
funnily
enough).

I haven't got time at the moment to investigate further (lots of other
things I should be doing).

From gthomas@netstarnetworks.com

I've modified the example small script to closer emulate my actual
program. I've moved the thr->join into a subroutine. This seems to make
the thread object go out of scope, and seems to destroy it
automatically - I don't need the explicit "undef", and it doesn't leak.

My feeling is that the leak I'm getting in my actual program is more a
symptom of either highly fragmented memory (memory used by a defunct
thread is not reusable due to fragmentation??), or some other leak I
can't spot. How do I rule out the former?

#!/usr/bin/perl
use strict;
use threads;
use threads​::shared;

our $thr : shared;

sub ThreadRoutine {
  $thr=threads->tid();
}

sub jointhread
{
  while ($thr==0){}; # make sure child thread has started
  my $throbj = threads->object($thr);
  $throbj->join;
  $thr=0;
}

$thr=0;
while (1){
  my $thread = threads->new(\&ThreadRoutine);
  jointhread();
}

[thoga01 - Fri Sep 30 16​:23​:50 2005]​:

If I put an "undef $thread;" in your example, the leak "seems" to be
plugged....so to speak...

So your example becomes...

 use threads;

 sub ThreadRoutine \{\}

 while \(1\) \{
 $thread = threads\->new\(\\&ThreadRoutine\);
 $thread\->join;
    undef $thread;
 \}

Is this the expected behaviour? Shouldn't the old thread object be
destroyed automagically? Could this be effecting other object
creation/deletion - ie not just thread objects? Should I always be
doing a "undef $object" in my code?

I'm yet to try this in my code, so I hope its the same leak that I've
been experiencing.

[davem@​iabyn.com - Sun Sep 11 06​:18​:34 2005]​:

On Sat, Sep 10, 2005 at 07​:10​:26PM -0700, Garry wrote​:

I've attached a program that simulates the problem I'm

experiencing. Basically, as I spawn and join threads in a long

running program, I appear to leak away memory. The program shows
how
this leak manifests itself. It shows the leak on version 5.8.6.

I can reduce the leak to this astonishingly small test case​:

use threads;

sub ThreadRoutine \{\}

while \(1\) \{
$thread = threads\->new\(\\&ThreadRoutine\);
$thread\->join;
\}

This leaks like a sieve on 5.8.1+ and bleedperl (but not on 5.8.0
funnily
enough).

I haven't got time at the moment to investigate further (lots of
other
things I should be doing).

From @iabyn

On Sun, Sep 11, 2005 at 02​:18​:06PM +0100, Dave Mitchell wrote​:

On Sat, Sep 10, 2005 at 07​:10​:26PM -0700, Garry wrote​:

I've attached a program that simulates the problem I'm experiencing\. Basically\, as I spawn and join threads in a long

running program, I appear to leak away memory. The program shows how this leak manifests itself. It shows the leak on version 5.8.6.

I can reduce the leak to this astonishingly small test case​:

use threads;

sub ThreadRoutine \{\}

while \(1\) \{
$thread = threads\->new\(\\&ThreadRoutine\);
$thread\->join;
\}

This leaks like a sieve on 5.8.1+ and bleedperl (but not on 5.8.0 funnily
enough).

I've fixed this by changing when the old interpreter is freed from when
$thread is freed to when the join happens, ie

  {
  my $thread = thread->new(...);
  $thread-join(); # interpreter now freed here
  } # used to be freed here

What was happening was that the second time round the loop, a new thread
was created containing a clone of $thread; then $thread was assigned to in
the parent thread, but the cloned copy kept the old interpreter alive.

(Actually the code above still leaks slighty but not nearly as much. I'm
now looking into that too).

--
Red sky at night - gerroff my land!
Red sky at morning - gerroff my land!
  -- old farmers' sayings #14

Change 25769 by davem@​davem-splatty on 2005/10/16 14​:53​:45

  free a thread's interpreter after $t->join() rather than after undef $t
  This should fix some ithreads memory leaks.

Affected files ...

... //depot/perl/ext/threads/threads.xs#88 edit

Differences ...

==== //depot/perl/ext/threads/threads.xs#88 (xtext) ====

@​@​ -86,14 +86,49 @​@​
}

+/* free any data (such as the perl interpreter) attached to an
+ * ithread structure. This is a bit like undef on SVs, where the SV
+ * isn't freed, but the PVX is.
+ * Must be called with thread->mutex already held
+ */
+
+static void
+Perl_ithread_clear(pTHX_ ithread* thread)
+{
+ PerlInterpreter *interp;
+ assert(thread->state & PERL_ITHR_FINISHED &&
+ (thread->state & PERL_ITHR_DETACHED ||
+ thread->state & PERL_ITHR_JOINED));
+
+ interp = thread->interp;
+ if (interp) {
+ dTHXa(interp);
+ ithread* current_thread;
+#ifdef OEMVS
+ void *ptr;
+#endif
+ PERL_SET_CONTEXT(interp);
+ current_thread = Perl_ithread_get(aTHX);
+ Perl_ithread_set(aTHX_ thread);
+
+ SvREFCNT_dec(thread->params);

+ thread->params = Nullsv;
+ perl_destruct(interp);
+ thread->interp = NULL;
+ }
+ if (interp)
+ perl_free(interp);
+ PERL_SET_CONTEXT(aTHX);
+}
+
+
/*
- * Clear up after thread is done with
+ * free an ithread structure and any attached data if its count == 0
  */
void
Perl_ithread_destruct (pTHX_ ithread* thread, const char *why)
{
- PerlInterpreter *freeperl = NULL;
  MUTEX_LOCK(&thread->mutex);
  if (!thread->next) {
  MUTEX_UNLOCK(&thread->mutex);
@​@​ -127,28 +162,7 @​@​
  MUTEX_UNLOCK(&create_destruct_mutex);
  /* Thread is now disowned */

- if(thread->interp) {
- dTHXa(thread->interp);
- ithread* current_thread;
-#ifdef OEMVS
- void *ptr;
-#endif
- PERL_SET_CONTEXT(thread->interp);
- current_thread = Perl_ithread_get(aTHX);
- Perl_ithread_set(aTHX_ thread);
-
-
-
-
- SvREFCNT_dec(thread->params);
-
-
-
- thread->params = Nullsv;
- perl_destruct(thread->interp);
- freeperl = thread->interp;
- thread->interp = NULL;
- }
+ Perl_ithread_clear(aTHX_ thread);
  MUTEX_UNLOCK(&thread->mutex);
  MUTEX_DESTROY(&thread->mutex);
#ifdef WIN32
@​@​ -157,10 +171,6 @​@​
  thread->handle = 0;
#endif
  PerlMemShared_free(thread);
- if (freeperl)
- perl_free(freeperl);
-
- PERL_SET_CONTEXT(aTHX);
}

int
@​@​ -650,6 +660,7 @​@​
  }
  /* We are finished with it */
  thread->state |= PERL_ITHR_JOINED;
+ Perl_ithread_clear(aTHX_ thread);
  MUTEX_UNLOCK(&thread->mutex);
 
  return retparam;

From @nwc10

On Sun, Oct 16, 2005 at 04​:52​:43PM +0100, Dave Mitchell wrote​:

Good stuff nailing this one.

+static void
+Perl_ithread_clear(pTHX_ ithread* thread)

Traditionally aren't static functions named S_* rather than Perl_* ?

Nicholas Clark

From @iabyn

On Sun, Oct 16, 2005 at 06​:12​:14PM +0100, Nicholas Clark wrote​:

On Sun, Oct 16, 2005 at 04​:52​:43PM +0100, Dave Mitchell wrote​:

Good stuff nailing this one.

+static void
+Perl_ithread_clear(pTHX_ ithread* thread)

Traditionally aren't static functions named S_* rather than Perl_* ?

Ah yes. Lazy cut-n-paste. Fixed by #25770.

--
My Dad used to say 'always fight fire with fire', which is probably why
he got thrown out of the fire brigade.

@iabyn - Status changed from 'open' to 'resolved'

From gthomas@netstarnetworks.com

Hi Dave,
  Does this patch have the same effect as me using "undef $thread" in terms of memory consumption, or does this patch do more
than this?

Thanks
Garry

-----Original Message-----
From​: Dave Mitchell via RT [mailto​:perlbug-followup@​perl.org]
Sent​: Monday, October 17, 2005 4​:16 AM
To​: gthomas@​netstarnetworks.com
Subject​: Re​: [perl #37134] Memory Leaking when joining threads?

On Sun, Oct 16, 2005 at 06​:12​:14PM +0100, Nicholas Clark wrote​:

On Sun, Oct 16, 2005 at 04​:52​:43PM +0100, Dave Mitchell wrote​:

Good stuff nailing this one.

+static void
+Perl_ithread_clear(pTHX_ ithread* thread)

Traditionally aren't static functions named S_* rather than Perl_* ?

Ah yes. Lazy cut-n-paste. Fixed by #25770.

--
My Dad used to say 'always fight fire with fire', which is probably why
he got thrown out of the fire brigade.

From @iabyn

On Mon, Oct 17, 2005 at 01​:26​:52PM +1000, Garry Thomas wrote​:

Hi Dave,
Does this patch have the same effect as me using "undef $thread"
in terms of memory consumption

Yes, essentially.

--
Justice is when you get what you deserve.
Law is when you get what you pay for.