From powerman@powerman.asdfGroup.com

Created by powerman@powerman.asdfgroup.com

Looks like when script doing heavy I/O use all 1024 file descriptors,
fail on opening next one and try to report this error using croak(),
the Carp try to load (require) Carp​::Heavy, and fail too (because there no
available file descriptors). This result in die() with message​:
  Can't locate Carp/Heavy.pm in @​INC (@​INC contains​: ...) at
  /usr/lib/perl5/5.8.8/Carp.pm line 255.
instead of original user's message.

I think Carp should avoid requesting more resources (memory, file
descriptors, etc.) when user call carp/croak/etc. Either Carp​::Heavy
should be loaded together with Carp when script start or in case
Carp​::Heavy can't be loaded Carp should contain some failback code,
which at least output user's message using usual warn/die.

Flags:
    category=library
    severity=low

Site configuration information for perl v5.8.8:

Configured by Gentoo at Sun Apr  1 02:52:52 EEST 2007.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.16-hardened-r11, archname=i686-linux
    uname='linux home 2.6.16-hardened-r11 #24 smp sat mar 24 00:08:46 eet 2007 i686 intel(r) core(tm)2 cpu 6600 @ 2.40ghz genuineintel gnulinux '
    config_args='-des -Darchname=i686-linux -Dcccdlflags=-fPIC -Dccdlflags=-rdynamic -Dcc=i686-pc-linux-gnu-gcc -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr -Dlocincpth=  -Doptimize=-march=pentium-m -msse3 -O2 -pipe -Duselargefiles -Dd_semctl_semun -Dscriptdir=/usr/bin -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dinstallman1dir=/usr/share/man/man1 -Dinstallman3dir=/usr/share/man/man3 -Dman1ext=1 -Dman3ext=3pm -Dinc_version_list=5.8.0 5.8.0/i686-linux 5.8.2 5.8.2/i686-linux 5.8.4 5.8.4/i686-linux 5.8.5 5.8.5/i686-linux 5.8.6 5.8.6/i686-linux 5.8.7 5.8.7/i686-linux  -Dinc_version_list=5.8.0 5.8.0/i686-linux 5.8.2 5.8.2/i686-linux 5.8.4 5.8.4/i686-linux 5.8.5 5.8.5/i686-linux 5.8.6 5.8.6/i686-linux 5.8.7 5.8.7/i686-linux  -Dcf_by=Gentoo -Ud_csh -Dusenm -Di_ndbm -Di_gdbm -Di_db'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='i686-pc-linux-gnu-gcc', ccflags ='-fno-strict-aliasing -pipe -Wdeclaration-after-statement -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-march=pentium-m -msse3 -O2 -pipe',
    cppflags='-fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/include/gdbm'
    ccversion='', gccversion='3.4.6 (Gentoo Hardened 3.4.6-r2, ssp-3.4.6-1.0, pie-8.7.10)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='i686-pc-linux-gnu-gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.3.6.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.3.6'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.8:
    /etc/perl
    /usr/lib/perl5/vendor_perl/5.8.8/i686-linux
    /usr/lib/perl5/vendor_perl/5.8.8
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/site_perl/5.8.8/i686-linux
    /usr/lib/perl5/site_perl/5.8.8
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/5.8.8/i686-linux
    /usr/lib/perl5/5.8.8
    /usr/local/lib/site_perl
    .


Environment for perl v5.8.8:
    HOME=/home/powerman
    LANG=ru_RU.KOI8-R
    LANGUAGE (unset)
    LC_NUMERIC=POSIX
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/powerman/bin:/home/powerman/inferno-os/Linux/386/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/sbin:/usr/sbin:/usr/local/sbin:/usr/games/bin:/usr/i686-pc-linux-gnu/gcc-bin/3.4.6:/opt/sun-jdk-1.4.2.13/bin:/opt/sun-jdk-1.4.2.13/jre/bin:/opt/sun-jdk-1.4.2.13/jre/javaws:/usr/kde/3.5/bin:/usr/qt/3/bin:/usr/games/bin:/opt/vmware/workstation/bin:/var/qmail/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

From @smpeters

On Fri Apr 06 15​:45​:45 2007, powerman <!-- x --> at asdfGroup.com wrote​:

This is a bug report for perl from powerman <!-- x --> at asdfgroup.com,
generated with the help of perlbug 1.35 running under perl v5.8.8.

-----------------------------------------------------------------
[Please enter your report here]

Looks like when script doing heavy I/O use all 1024 file descriptors,
fail on opening next one and try to report this error using croak(),
the Carp try to load (require) Carp​::Heavy, and fail too (because
there no
available file descriptors). This result in die() with message​:
Can't locate Carp/Heavy.pm in @​INC (@​INC contains​: ...) at
/usr/lib/perl5/5.8.8/Carp.pm line 255.
instead of original user's message.

I think Carp should avoid requesting more resources (memory, file
descriptors, etc.) when user call carp/croak/etc. Either Carp​::Heavy
should be loaded together with Carp when script start or in case
Carp​::Heavy can't be loaded Carp should contain some failback code,
which at least output user's message using usual warn/die.

I've pretty much only seen these errors when max open files for a user
is set extremely low (around 20 or so). Can you please send your
results from a "ulimit -n"?

Steve Peters
steve@​fisharerojo.org

The RT System itself - Status changed from 'new' to 'open'

From powerman@powerman.asdfGroup.com

Hi!

On Fri, Jun 29, 2007 at 08​:53​:18AM -0700, Steve Peters via RT wrote​:

Looks like when script doing heavy I/O use all 1024 file descriptors,

I've pretty much only seen these errors when max open files for a user
is set extremely low (around 20 or so). Can you please send your
results from a "ulimit -n"?

1024 :)

As I said, my script doing heavy I/O, and it really used about 1024 FD
(most of them is non-blocking sockets).

--
  WBR, Alex.

From p5p@spam.wizbit.be

Looks like when script doing heavy I/O use all 1024 file descriptors,
fail on opening next one and try to report this error using croak(),
the Carp try to load (require) Carp​::Heavy, and fail too (because
there no
available file descriptors). This result in die() with message​:
Can't locate Carp/Heavy.pm in @​INC (@​INC contains​: ...) at
/usr/lib/perl5/5.8.8/Carp.pm line 255.
instead of original user's message.

I think Carp should avoid requesting more resources (memory, file
descriptors, etc.) when user call carp/croak/etc. Either Carp​::Heavy
should be loaded together with Carp when script start or in case
Carp​::Heavy can't be loaded Carp should contain some failback code,
which at least output user's message using usual warn/die.

-----------------------------------------------------------------

Patch attached that​:
a) shows the error from Carp​::Heavy,
b) the supplied string,
c) a solution/work-around

#!/usr/bin/perl -l

use Carp;
my @​t;
for (1 .. 1024) {
  open my $fh, "<", $0;
  push @​t, $fh;
}
croak "abc";
__END__

Now shows​:

Failed to load Carp​::Heavy. Consider adding use Carp​::Heavy in your
code.
Carp-Error​: Can't locate Carp/Heavy.pm​: Too many open files at lib/
Carp.pm line 44.
Args​: abc

Kind regards,

Bram

From p5p@spam.wizbit.be

patch_Carp.pm

From p5p@perl.wizbit.be

Any comments on this patch?

Kind regards,

Bram

----- Forwarded message from perlbug-followup@​perl.org -----
  Date​: Wed, 30 Apr 2008 03​:52​:55 -0700
  From​: Bram via RT <perlbug-followup@​perl.org>
Reply-To​: perlbug-followup@​perl.org
  Subject​: [perl #42329] [PATCH] Carp​::croak() replace user error
message by own
  To​: "OtherRecipients of perl Ticket #42329"@​
  Cc​: perl5-porters@​perl.org

Looks like when script doing heavy I/O use all 1024 file descriptors,
fail on opening next one and try to report this error using croak(),
the Carp try to load (require) Carp​::Heavy, and fail too (because
there no
available file descriptors). This result in die() with message​:
Can't locate Carp/Heavy.pm in @​INC (@​INC contains​: ...) at
/usr/lib/perl5/5.8.8/Carp.pm line 255.
instead of original user's message.

I think Carp should avoid requesting more resources (memory, file
descriptors, etc.) when user call carp/croak/etc. Either Carp​::Heavy
should be loaded together with Carp when script start or in case
Carp​::Heavy can't be loaded Carp should contain some failback code,
which at least output user's message using usual warn/die.

-----------------------------------------------------------------

Patch attached that​:
a) shows the error from Carp​::Heavy,
b) the supplied string,
c) a solution/work-around

#!/usr/bin/perl -l

use Carp;
my @​t;
for (1 .. 1024) {
  open my $fh, "<", $0;
  push @​t, $fh;
}
croak "abc";
__END__

Now shows​:

Failed to load Carp​::Heavy. Consider adding use Carp​::Heavy in your
code.
Carp-Error​: Can't locate Carp/Heavy.pm​: Too many open files at lib/
Carp.pm line 44.
Args​: abc

Kind regards,

Bram

----- End forwarded message -----

From p5p@perl.wizbit.be

patch_Carp.pm

From p5p@spam.wizbit.be

On Wed Apr 30 03​:52​:52 2008, animator wrote​:

Looks like when script doing heavy I/O use all 1024 file
descriptors,
fail on opening next one and try to report this error using
croak(),
the Carp try to load (require) Carp​::Heavy, and fail too (because
there no
available file descriptors). This result in die() with message​:
Can't locate Carp/Heavy.pm in @​INC (@​INC contains​: ...) at
/usr/lib/perl5/5.8.8/Carp.pm line 255.
instead of original user's message.

I think Carp should avoid requesting more resources (memory, file
descriptors, etc.) when user call carp/croak/etc. Either
Carp​::Heavy
should be loaded together with Carp when script start or in case
Carp​::Heavy can't be loaded Carp should contain some failback code,
which at least output user's message using usual warn/die.

-----------------------------------------------------------------

Patch attached that​:
a) shows the error from Carp​::Heavy,
b) the supplied string,
c) a solution/work-around

#!/usr/bin/perl -l

use Carp;
my @​t;
for (1 .. 1024) {
open my $fh, "<", $0;
push @​t, $fh;
}
croak "abc";
__END__

Now shows​:

Failed to load Carp​::Heavy. Consider adding use Carp​::Heavy in your
code.
Carp-Error​: Can't locate Carp/Heavy.pm​: Too many open files at lib/
Carp.pm line 44.
Args​: abc

Any feedback on this? (patch is in RT)

Best regards,

Bram

From @powerman

Hi!

On Sun, Jun 28, 2009 at 07​:02​:40AM -0700, Bram via RT wrote​:

Now shows​:

Failed to load Carp​::Heavy. Consider adding use Carp​::Heavy in your code.
Carp-Error​: Can't locate Carp/Heavy.pm​: Too many open files at lib/Carp.pm line 44.
Args​: abc

Any feedback on this? (patch is in RT)

I think that's wrong way to solve this issue, but it's better that nothing.

The problem is, right now all my scripts which doing heavy I/O contain line​:

require Carp​::Heavy; # http​://rt.perl.org/rt3/Public/Bug/Display.html?id=42329

and while I love to see this line removed from my scripts, with this patch
it still not possible.

Only I can do is try to explain again​: error-reporting code like Carp MUST NOT
request additional resources - because if some error happens it's very
likely 'not enough resources' error and trying to get more resources just
to show error message just feels wrong.

I think correct solution will be to load Carp​::Heavy together with Carp.
I see no reason to delay loading Carp​::Heavy - it's very small and
shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

--
  WBR, Alex.

From @rgs

2009/6/29 Alex Efros <powerman@​powerman.name>​:

Only I can do is try to explain again​: error-reporting code like Carp MUST NOT
request additional resources - because if some error happens it's very
likely 'not enough resources' error and trying to get more resources just
to show error message just feels wrong.

I think correct solution will be to load Carp​::Heavy together with Carp.
I see no reason to delay loading Carp​::Heavy - it's very small and
shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

I tend to concur. Those days we have more CPU and memory resources, so
a cleanup to merge Carp​::Heavy in Carp is totally feasible.

From @jimc

shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

I tend to concur. Those days we have more CPU and memory resources, so
a cleanup to merge Carp​::Heavy in Carp is totally feasible.

By extension, this also applies to Config ?

From @nwc10

On Mon, Jun 29, 2009 at 08​:31​:06AM -0400, Jim Cromie wrote​:

shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

I tend to concur. Those days we have more CPU and memory resources, so
a cleanup to merge Carp​::Heavy in Carp is totally feasible.

By extension, this also applies to Config ?

Rafael may well disagree with me, but I don't think it should apply to Config.

My reasoning for splitting Config.pm apart was that a monolithic Config.pm
loads (at least) 32K of data that is rarely used, and cloned to each and every
thread, if you're using threads. I've not measured Carp​::Heavy, but I suspect
that it's less than 32K, and as it's mostly ops, there's not going to be
anywhere near as much thread overhead.

Config doesn't fail catastrophically if it fails to load Config_heavy.pl,
whereas as Alex Efros notes, Carp failing will eat the error message.

Additionally, the code needed to split Carp​::Heavy out from Carp looks
complex, and I suspect is fragile, whereas the AUTOLOAD subroutine in
Config.pm is in one place, simple and conventional. (As conventional as
anything using AutoLoader)

Nicholas Clark

From @jimc

On Mon, Jun 29, 2009 at 8​:46 AM, Nicholas Clark<nick@​ccl4.org> wrote​:

On Mon, Jun 29, 2009 at 08​:31​:06AM -0400, Jim Cromie wrote​:

shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

I tend to concur. Those days we have more CPU and memory resources, so
a cleanup to merge Carp​::Heavy in Carp is totally feasible.

By extension, this also applies to Config ?

Rafael may well disagree with me, but I don't think it should apply to Config.

My reasoning for splitting Config.pm apart was that a monolithic Config.pm
loads (at least) 32K of data that is rarely used, and cloned to each and every
thread, if you're using threads. I've not measured Carp​::Heavy, but I suspect
that it's less than 32K, and as it's mostly ops, there's not going to be
anywhere near as much thread overhead.

Config doesn't fail catastrophically if it fails to load Config_heavy.pl,
whereas as Alex Efros notes, Carp failing will eat the error message.

Additionally, the code needed to split Carp​::Heavy out from Carp looks
complex, and I suspect is fragile, whereas the AUTOLOAD subroutine in
Config.pm is in one place, simple and conventional. (As conventional as
anything using AutoLoader)

Nicholas Clark

ack. very different in the details and tradeoffs.

From @iabyn

On Mon, Jun 29, 2009 at 12​:29​:54PM +0200, Rafael Garcia-Suarez wrote​:

2009/6/29 Alex Efros <powerman@​powerman.name>​:

Only I can do is try to explain again​: error-reporting code like Carp MUST NOT
request additional resources - because if some error happens it's very
likely 'not enough resources' error and trying to get more resources just
to show error message just feels wrong.

I think correct solution will be to load Carp​::Heavy together with Carp.
I see no reason to delay loading Carp​::Heavy - it's very small and
shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

I tend to concur. Those days we have more CPU and memory resources, so
a cleanup to merge Carp​::Heavy in Carp is totally feasible.

Just as a data point, Carp​::Heavy seems to use about 250Kbyte, which is,
erm, quite heavy​:

(linux, 4K pagesize, 32-bit, blead, threaded, -O2)​:

$ p -e 'system"ps -lfp $$"'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S davem 9798 3295 0 80 0 - 1474 wait 14​:50 pts/1 00​:00​:00 ./per
$ p -MCarp -e 'system"ps -lfp $$"'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S davem 9804 3295 0 80 0 - 1474 wait 14​:50 pts/1 00​:00​:00 ./per
$ p -MCarp​::Heavy -e 'system"ps -lfp $$"'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S davem 9814 3295 0 80 0 - 1540 wait 14​:50 pts/1 00​:00​:00 ./per

--
Art is anything that has a label (especially if the label is "untitled 1")

From @rgs

2009/6/29 Rafael Garcia-Suarez <rgarciasuarez@​gmail.com>​:

2009/6/29 Alex Efros <powerman@​powerman.name>​:

Only I can do is try to explain again​: error-reporting code like Carp MUST NOT
request additional resources - because if some error happens it's very
likely 'not enough resources' error and trying to get more resources just
to show error message just feels wrong.

I think correct solution will be to load Carp​::Heavy together with Carp.
I see no reason to delay loading Carp​::Heavy - it's very small and
shouldn't affect performance. Moreover, if you wanna speedup Carp loading,
then there a lot of other techniques - like moving POD to the bottom of
file under __END__. Also consider just moving Carp​::Heavy content to Carp,
this shouldn't increase Carp module load time by any noticeable value and
also allow to do some code cleanup.

I tend to concur. Those days we have more CPU and memory resources, so
a cleanup to merge Carp​::Heavy in Carp is totally feasible.

This is now done in blead. There's still a Carp​::Heavy placeholder for
legacy code that wants to force its loading (like File​::Temp.)

@rgs - Status changed from 'open' to 'resolved'