From @eserte

The test t/06words.t in the String-Lookup distribution exits with an Out of memory error.

See http​://matrix.cpantesters.org/?dist=String-Lookup-0.12
and the only fail report so far is http​://www.cpantesters.org/cpan/report/eea073a6-d54b-11e3-a891-875aeea79835 (on a FreeBSD system), but there will come more error reports for FreeBSD and Linux.

To reproduce the problem on a Debian Linux system a large wordlist needs to be installed​:

  sudo aptitude install wamerican-huge
  sudo select-default-wordlist
  (and select the huge list)

The maximum memory has to be limited to 2GB or so.

The same test works fine with perl 5.18.2 and memory limited to 0.5GB.

Maybe related to https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121815

From @lizmat

On 08 May 2014, at 08​:41, slaven@​rezic.de (via RT) <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by slaven@​rezic.de
# Please include the string​: [perl #121826]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121826 >

The test t/06words.t in the String-Lookup distribution exits with an Out of memory error.

See http​://matrix.cpantesters.org/?dist=String-Lookup-0.12
and the only fail report so far is http​://www.cpantesters.org/cpan/report/eea073a6-d54b-11e3-a891-875aeea79835 (on a FreeBSD system), but there will come more error reports for FreeBSD and Linux.

To reproduce the problem on a Debian Linux system a large wordlist needs to be installed​:

sudo aptitude install wamerican-huge
sudo select-default-wordlist
(and select the huge list)

The maximum memory has to be limited to 2GB or so.

The same test works fine with perl 5.18.2 and memory limited to 0.5GB.

Maybe related to https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121815

What can I say? Hashes take more memory in blead?

Liz

The RT System itself - Status changed from 'new' to 'open'

From @rjbs

Yves felt this would likely be fixed by e8c6a47.

I must admit to being unable to figure out how to get the analysis system to tell me whether this problem seems solved on the systems that had been affected.

--
rjbs

From @demerphq

On 8 May 2014 09​:32, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

On 08 May 2014, at 08​:41, slaven@​rezic.de (via RT) <
perlbug-followup@​perl.org> wrote​:

# New Ticket Created by slaven@​rezic.de
# Please include the string​: [perl #121826]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121826 >

The test t/06words.t in the String-Lookup distribution exits with an Out
of memory error.

See http​://matrix.cpantesters.org/?dist=String-Lookup-0.12
and the only fail report so far is
http​://www.cpantesters.org/cpan/report/eea073a6-d54b-11e3-a891-875aeea79835(on a FreeBSD system), but there will come more error reports for FreeBSD
and Linux.

To reproduce the problem on a Debian Linux system a large wordlist needs
to be installed​:

sudo aptitude install wamerican-huge
sudo select-default-wordlist
(and select the huge list)

The maximum memory has to be limited to 2GB or so.

The same test works fine with perl 5.18.2 and memory limited to 0.5GB.

Maybe related to https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121815

What can I say? Hashes take more memory in blead?

If you are reading your wordlist with something like

while (<>) {
  chomp;
  $lookup{$_}++;
}

then the memory bloat comes from a bizarre and unfortunate interplay
between sv_gets() and COW, which hopefully is fixed by e8c6a47.

If you could try to replicate this with a blead that includes that patch I
would be very grateful.

BTW, there is reason to believe that e8c6a47 fixes all the strange
out-of-memory errors that are related to COW, but we have not had any
feedback yet if that is actually the case.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

From @andk

"Ricardo SIGNES via RT" <perlbug-followup@​perl.org> writes​:

Yves felt this would likely be fixed by e8c6a47.

I must admit to being unable to figure out how to get the analysis
system to tell me whether this problem seems solved on the systems
that had been affected.

Possible answers​:

http​://matrix.cpantesters.org/?dist=String-Lookup%200.12
http​://analysis.cpantesters.org/reports_by_field?distv=String-Lookup-0.12;field=meta%3Aperl

At the moment, there is only one result for 5.20.0-RC1 which is a PASS.
My smoker just sent out 4 more PASSes.

--
andreas

From @andk

demerphq <demerphq@​gmail.com> writes​:

BTW, there is reason to believe that e8c6a47 fixes all the strange
out-of-memory errors that are related to COW, but we have not had any feedback
yet if that is actually the case.

Unfortunately, it is not the case. Counter example fail reports​:

http​://www.cpantesters.org/cpan/report/0c8948dc-ddc3-11e3-b672-b88fe0bfc7aa
http​://www.cpantesters.org/cpan/report/8364bb1c-dfe9-11e3-adf3-e536e0bfc7aa

I just reproduced one with AAR/Boost-Geometry-Utils-0.15.tar.gz myself
on 5.20.0-RC1.

--
andreas

From @iabyn

On Wed, May 21, 2014 at 10​:21​:14PM +0200, Andreas Koenig wrote​:

demerphq <demerphq@​gmail.com> writes​:

BTW, there is reason to believe that e8c6a47 fixes all the strange
out-of-memory errors that are related to COW, but we have not had any feedback
yet if that is actually the case.

Unfortunately, it is not the case. Counter example fail reports​:

http​://www.cpantesters.org/cpan/report/0c8948dc-ddc3-11e3-b672-b88fe0bfc7aa
http​://www.cpantesters.org/cpan/report/8364bb1c-dfe9-11e3-adf3-e536e0bfc7aa

I just reproduced one with AAR/Boost-Geometry-Utils-0.15.tar.gz myself
on 5.20.0-RC1.

The specific issue with Boost-Geometry-Utils is a change in the API of
av_extend(), where the size to extend to is now a SSize_t rather than I32,

  commit fc16c39
  Author​: Father Chrysostomos <sprout@​cpan.org>
  CommitDate​: Sun Aug 25 06​:39​:28 2013 -0700

  Use SSize_t when extending the stack

The debugger shows av_extend() being called from multi_linestring2perl()
with an arg of 0xffffffff, so I guess something started off as -1 and
failed to get sign extended; so it's trying to extend the array to 4G-1
elements.

For some reason the C++ code in B​::G​::U isn't showing symbols for me in
gdb, but the stack backtrace shows​:

#5 0x0000000000578f04 in Perl_av_extend (av=0xbe3218, key=0xffffffff) at av.c​:82
#6 0x00007ffff166112f in multi_linestring2perl(boost​::geometry​::model​::multi_linestring<boost​::geometry​::model​::linestring<boost​::geometry​::model​::d2​::point_xy<double, boost​::geometry​::cs​::cartesian>, std​::vector, std​::allocator>, std​::vector, std​::allocator> const&) () from /home/davem/tmp/x/b1/blib/arch/auto/Boost/Geometry/Utils/Utils.so

and /src/mline2av.h has this​:

  SV*
  multi_linestring2perl(pTHX_ const multi_linestring& mls)
  {
  AV* av = newAV();
  const unsigned int size = mls.size();
  av_extend(av, size-1);
  ...

Since its using an *unsigned* int, I think the lack of sign extension and
thus memory exhaustion is B​::G​::U's fault.

Are there any other modules which are still showing memory problems in
RC1?

--
The Enterprise is captured by a vastly superior alien intelligence which
does not put them on trial.
  -- Things That Never Happen in "Star Trek" #10

From @eserte

Dana Sri 07. Svibanj 2014, 23​:41​:53, slaven@​rezic.de reče​:

The test t/06words.t in the String-Lookup distribution exits with an
Out of memory error.

See http​://matrix.cpantesters.org/?dist=String-Lookup-0.12
and the only fail report so far is
http​://www.cpantesters.org/cpan/report/eea073a6-d54b-11e3-a891-
875aeea79835 (on a FreeBSD system), but there will come more error
reports for FreeBSD and Linux.

To reproduce the problem on a Debian Linux system a large wordlist
needs to be installed​:

sudo aptitude install wamerican-huge
sudo select-default-wordlist
(and select the huge list)

The maximum memory has to be limited to 2GB or so.

The same test works fine with perl 5.18.2 and memory limited to 0.5GB.

Maybe related to https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121815

The original issue looks like being resolved with 5.20.0-RC1​: on Debian wheezy, FreeBSD 9.2, and FreeBSD 10.0 the distribution tests fine now (as usual on my smokers, with a 0.5GB memory limit). On the same systems the distribution fails with 5.19.11.

This doesn't say anything about the other distributions mentioned in this thread.

Regards,
  Slaven

From @rjbs

I believe String-Lookup is now okay, and B-G-U is covered by https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121815

--
rjbs

@rjbs - Status changed from 'open' to 'resolved'

From @csjewell

Uh, 121815 claims that B-G-U is covered by 121826, and you now say in
this one (121826) that it's covered by 121815. I'm getting a personal
out-of-memory error recursing through that loop!

On Mon, Jun 9, 2014, at 16​:15, Ricardo SIGNES via RT wrote​:

I believe String-Lookup is now okay, and B-G-U is covered by
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121815

--
Curtis Jewell
csjewell@​cpan.org http​://csjewell.dreamwidth.org/
perl@​curtisjewell.name http​://www.curtisjewell.name/
"Your random numbers are not that random" -- perl-5.10.1.tar.gz/util.c

From @rjbs

* Curtis Jewell <perl@​csjewell.fastmail.us> [2014-06-09T20​:45​:36]

Uh, 121815 claims that B-G-U is covered by 121826, and you now say in
this one (121826) that it's covered by 121815. I'm getting a personal
out-of-memory error recursing through that loop!

There's nothing weird going on. 121815 has a link to a *specific comment* on
121826. It doesn't say it's "covered by," but "diagnosed" at. So the ticket
about String-Lookup has been closed, because String-Lookup is fine. B-G-U is
discussed on both tickets, and one of them has been left open, with a
cross-reference to the diagnosis found elsewhere.

--
rjbs