Perl_magic_get reqires ridiculous amounts of stack memory #8244

p5pRT · 2005-12-13T08:59:03Z

Migrated from rt.perl.org#37907 (status was 'resolved')

Searchable as RT37907$

p5pRT · 2005-12-13T08:59:09Z

From perlbug@plan9.de

Created by perlbug@plan9.de

While tracking down why some of my programs segfaulted with mysterious
backtraces under debians perl, I found that Perl_magic_get allocates 256kb
of stack memory.

This is more than what is usually required for that function (<256 bytes),
and exceeds both the stack limit used by Coro as well as the stack limit
used for threads on some platforms (even on DGUX, perl explicitly sets the
stack size to 32k, so a much smaller stack footprint is expected). All
this is just to say that this is a bug: 0.25MB of stack space is too much
:)

Andrew Suffield tracked it down to this code:

On Tue, Dec 13, 2005 at 08:15:38AM +0000, Andrew Suffield <asuffield@debian.org> wrote:

#ifdef HAS_GETGROUPS
{
Groups_t gary[NGROUPS];
i = getgroups(NGROUPS,gary);
while (--i >= 0)
Perl_sv_catpvf(aTHX_ sv, " %"Gid_t_f, gary[i]);
}
#endif

sizeof(gid_t) (4) * NGROUPS (65536) == 256kb

It should be fetching the number of groups with getgroups(0, NULL) and
then allocating dynamically. This happens in several places.

which, aprrently, is used elsewhere. Such a large value of NGROUPS is
probably uncommon today, but as it is what is supported, it should work.

Perl Info


Flags:
    category=core
    severity=low

Site configuration information for perl v5.8.6:

Configured by Marc Lehmann at Tue Nov 30 00:54:44 CET 2004.

Summary of my perl5 (revision 5 version 8 subversion 6) configuration:
  Platform:
    osname=linux, osvers=2.6.10-rc1, archname=amd64-linux
    uname='linux cerebro 2.6.10-rc1 #1 smp mon nov 22 05:47:21 cet 2004 x86_64 gnulinux '
    config_args='-Duselargefiles -Dxuse64bitint -Uxuse64bitall -Dusemymalloc=y -Dcc=gcc-3.4 -Dccflags=-ggdb -Dcppflags=-D_GNU_SOURCE -I/opt/include -Doptimize=-O4 -march=opteron -mtune=opteron -funroll-loops -fno-strict-aliasing -Dcccdlflags=-fPIC -Dldflags=-L/opt/perl/lib -L/opt/lib -Dlibs=-ldl -lm -lcrypt -Darchname=amd64-linux -Dprefix=/opt/perl -Dprivlib=/opt/perl/lib/perl5 -Darchlib=/opt/perl/lib/perl5 -Dvendorprefix=/opt/perl -Dvendorlib=/opt/perl/lib/perl5 -Dvendorarch=/opt/perl/lib/perl5 -Dsiteprefix=/opt/perl -Dsitelib=/opt/perl/lib/perl5 -Dsitearch=/opt/perl/lib/perl5 -Dsitebin=/opt/perl/bin -Dman1dir=/opt/perl/man/man1 -Dman3dir=/opt/perl/man/man3 -Dsiteman1dir=/opt/perl/man/man1 -Dsiteman3dir=/opt/perl/man/man3 -Dman1ext=1 -Dman3ext=3 -Dpager=/usr/bin/less -Uafs -Uusesfio -Uusenm -Uuseshrplib -Dd_dosuid -Dusethreads=undef -Duse5005threads=undef -Duseithreads=undef -Dusemultiplicity=undef -Demail=perl-binary@plan9.de -Dcf_email=perl-binary@plan9.de -Dcf_by=Marc Lehmann -Dlocincpth=/opt/perl/include /opt/include -Dmyhostname=localhost -Dmultiarch=undef -Dbin=/opt/perl/bin -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=define uselongdouble=undef
    usemymalloc=y, bincompat5005=undef
  Compiler:
    cc='gcc-3.4', ccflags ='-ggdb -fno-strict-aliasing -pipe -I/opt/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O4 -march=opteron -mtune=opteron -funroll-loops -fno-strict-aliasing',
    cppflags='-D_GNU_SOURCE -I/opt/include -ggdb -fno-strict-aliasing -pipe -I/opt/include'
    ccversion='', gccversion='3.4.2 (Debian 3.4.2-3)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc-3.4', ldflags ='-L/opt/perl/lib -L/opt/lib -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-ldl -lm -lcrypt
    perllibs=-ldl -lm -lcrypt
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/opt/perl/lib -L/opt/lib -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.6:
    /root/src/sex
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    .


Environment for perl v5.8.6:
    HOME=/root
    LANG (unset)
    LANGUAGE (unset)
    LC_CTYPE=de_DE.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/root/s2:/root/s:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11/bin:/usr/games:/root/src/uunet:.
    PERL5LIB=/root/src/sex
    PERL5_CPANPLUS_CONFIG=/root/.cpanplus/config
    PERLDB_OPTS=ornaments=0
    PERL_BADLANG (unset)
    PERL_UNICODE=SAL
    SHELL=/bin/bash

p5pRT · 2005-12-23T18:45:35Z

From @smpeters

[perlbug@plan9.de - Tue Dec 13 00:59:09 2005]:

This is a bug report for perl from perlbug@plan9.de,
generated with the help of perlbug 1.35 running under perl v5.8.6.

-----------------------------------------------------------------
[Please enter your report here]

While tracking down why some of my programs segfaulted with mysterious
backtraces under debians perl, I found that Perl_magic_get allocates
256kb
of stack memory.

This is more than what is usually required for that function (<256
bytes),
and exceeds both the stack limit used by Coro as well as the stack
limit
used for threads on some platforms (even on DGUX, perl explicitly sets
the
stack size to 32k, so a much smaller stack footprint is expected). All
this is just to say that this is a bug: 0.25MB of stack space is too
much
:)

Andrew Suffield tracked it down to this code:

On Tue, Dec 13, 2005 at 08:15:38AM +0000, Andrew Suffield
<asuffield@debian.org> wrote:

#ifdef HAS_GETGROUPS
{
Groups_t gary[NGROUPS];
i = getgroups(NGROUPS,gary);
while (--i >= 0)
Perl_sv_catpvf(aTHX_ sv, " %"Gid_t_f, gary[i]);
}
#endif

sizeof(gid_t) (4) * NGROUPS (65536) == 256kb

It should be fetching the number of groups with getgroups(0, NULL)
and
then allocating dynamically. This happens in several places.

which, aprrently, is used elsewhere. Such a large value of NGROUPS is
probably uncommon today, but as it is what is supported, it should
work.

OK, comparing Linux to OpenBSD, Linux allows 65536 while OpenBSD allows
16 groups. I'm thinking that this number should probably be lowered.
Maybe 1024 would be safe number that would strike a balance between the
extremes? This would produce an array of 4kb.

p5pRT · 2005-12-23T18:45:37Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2005-12-23T20:23:10Z

From nick@ing-simmons.net

Steve Peters via RT <perlbug-followup@perl.org> writes:

It should be fetching the number of groups with getgroups(0, NULL)
and
then allocating dynamically. This happens in several places.

which, aprrently, is used elsewhere. Such a large value of NGROUPS is
probably uncommon today, but as it is what is supported, it should
work.

OK, comparing Linux to OpenBSD, Linux allows 65536 while OpenBSD allows
16 groups. I'm thinking that this number should probably be lowered.
Maybe 1024 would be safe number that would strike a balance between the
extremes? This would produce an array of 4kb.

What is wrong with the idea of asking OS how many groups user has as
suggested?

p5pRT · 2005-12-23T20:25:13Z

From @jimc

Steve Peters via RT wrote:

[perlbug@plan9.de - Tue Dec 13 00:59:09 2005]:

This is a bug report for perl from perlbug@plan9.de,
generated with the help of perlbug 1.35 running under perl v5.8.6.

-----------------------------------------------------------------
[Please enter your report here]

While tracking down why some of my programs segfaulted with mysterious
backtraces under debians perl, I found that Perl_magic_get allocates
256kb
of stack memory.

This is more than what is usually required for that function (<256
bytes),
and exceeds both the stack limit used by Coro as well as the stack
limit
used for threads on some platforms (even on DGUX, perl explicitly sets
the
stack size to 32k, so a much smaller stack footprint is expected). All
this is just to say that this is a bug: 0.25MB of stack space is too
much
:)

Andrew Suffield tracked it down to this code:

On Tue, Dec 13, 2005 at 08:15:38AM +0000, Andrew Suffield
<asuffield@debian.org> wrote:

#ifdef HAS_GETGROUPS
{
Groups_t gary[NGROUPS];
i = getgroups(NGROUPS,gary);
while (--i >= 0)
Perl_sv_catpvf(aTHX_ sv, " %"Gid_t_f, gary[i]);
}
#endif

sizeof(gid_t) (4) * NGROUPS (65536) == 256kb

It should be fetching the number of groups with getgroups(0, NULL)

and

then allocating dynamically. This happens in several places.

which, aprrently, is used elsewhere. Such a large value of NGROUPS is
probably uncommon today, but as it is what is supported, it should
work.

OK, comparing Linux to OpenBSD, Linux allows 65536 while OpenBSD allows
16 groups. I'm thinking that this number should probably be lowered.
Maybe 1024 would be safe number that would strike a balance between the
extremes? This would produce an array of 4kb.

attached patch calls getgroups 2x, 1st to find number of groups,
2nd to fill the (now) mallocd space with that info.
On normal systems, the malloc will be tiny (my box has 2 grps for me)

[jimc@harpo getgroups]$ ./perl -Ilib t/op/groups.t
# groups = uid=500(jimc) gid=500(jimc) groups=500(jimc),1000(jimc-debian)
# groups=500(jimc),1000(jimc-debian)
# g0 = 500(jimc) 1000(jimc-debian)
# g1 = jimc jimc-debian
1..2
# pwgid = 500, pwgnam = jimc
# gr = jimc-debian
ok 1
ok 2

also attched is 2nd version, which avoids the Newx(), and uses an
automatic var (stack)
with a run-time array dimension. It works for me, but Ive been guilty
of gcc-ism
patches before, so Im just sending it to find out the lazy way ;-)

p5pRT · 2005-12-23T20:25:13Z

From @jimc

Inline Patch

diff -ruNp -X exclude-diffs ../bleadperl/mg.c getgroups/mg.c
--- ../bleadperl/mg.c	2005-12-21 11:36:25.000000000 -0700
+++ getgroups/mg.c	2005-12-23 12:43:48.000000000 -0700
@@ -1006,10 +1006,13 @@ Perl_magic_get(pTHX_ SV *sv, MAGIC *mg)
       add_groups:
 #ifdef HAS_GETGROUPS
 	{
-	    Groups_t gary[NGROUPS];
-	    I32 j = getgroups(NGROUPS,gary);
+	    Groups_t *gary;
+	    I32 j = getgroups(0,gary);
+	    Newx(gary, j, Groups_t);
+	    j = getgroups(j,gary);
 	    while (--j >= 0)
 		Perl_sv_catpvf(aTHX_ sv, " %"Gid_t_f, (long unsigned int)gary[j]);
+	    Safefree(gary);
 	}
 #endif
 	(void)SvIOK_on(sv);	/* what a wonderful hack! */

p5pRT · 2005-12-23T20:25:13Z

From @jimc

Inline Patch

diff -ruNp -X exclude-diffs ../bleadperl/mg.c getgroups/mg.c
--- ../bleadperl/mg.c	2005-12-21 11:36:25.000000000 -0700
+++ getgroups/mg.c	2005-12-23 13:20:00.000000000 -0700
@@ -1006,10 +1006,14 @@ Perl_magic_get(pTHX_ SV *sv, MAGIC *mg)
       add_groups:
 #ifdef HAS_GETGROUPS
 	{
-	    Groups_t gary[NGROUPS];
-	    I32 j = getgroups(NGROUPS,gary);
-	    while (--j >= 0)
-		Perl_sv_catpvf(aTHX_ sv, " %"Gid_t_f, (long unsigned int)gary[j]);
+	    Groups_t *gary;
+	    I32 j = getgroups(0,gary);
+	    {
+		Groups_t gary[j];
+		j = getgroups(j,gary);
+		while (--j >= 0)
+		    Perl_sv_catpvf(aTHX_ sv, " %"Gid_t_f, (long unsigned int)gary[j]);
+	    }
 	}
 #endif
 	(void)SvIOK_on(sv);	/* what a wonderful hack! */

p5pRT · 2005-12-24T14:24:37Z

From @smpeters

On Fri, Dec 23, 2005 at 01:24:29PM -0700, Jim Cromie wrote:

Steve Peters via RT wrote:

OK, comparing Linux to OpenBSD, Linux allows 65536 while OpenBSD allows
16 groups. I'm thinking that this number should probably be lowered.
Maybe 1024 would be safe number that would strike a balance between the
extremes? This would produce an array of 4kb.

attached patch calls getgroups 2x, 1st to find number of groups,
2nd to fill the (now) mallocd space with that info.
On normal systems, the malloc will be tiny (my box has 2 grps for me)

[jimc@harpo getgroups]$ ./perl -Ilib t/op/groups.t
# groups = uid=500(jimc) gid=500(jimc) groups=500(jimc),1000(jimc-debian)
# groups=500(jimc),1000(jimc-debian)
# g0 = 500(jimc) 1000(jimc-debian)
# g1 = jimc jimc-debian
1..2
# pwgid = 500, pwgnam = jimc
# gr = jimc-debian
ok 1
ok 2

Great! I've added the first version as change #26480. I did make one change
to rename the variable "j" to "num_groups".

Thanks!

Steve Peters
steve@fisharerojo.org

p5pRT · 2005-12-24T14:29:05Z

@smpeters - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Dec 24, 2005

p5pRT added Severity Low distro-Linux type-core labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Perl_magic_get reqires ridiculous amounts of stack memory #8244

Perl_magic_get reqires ridiculous amounts of stack memory #8244

p5pRT commented Dec 13, 2005

p5pRT commented Dec 13, 2005

p5pRT commented Dec 23, 2005

p5pRT commented Dec 23, 2005

p5pRT commented Dec 23, 2005

p5pRT commented Dec 23, 2005

p5pRT commented Dec 23, 2005

p5pRT commented Dec 23, 2005

p5pRT commented Dec 24, 2005

p5pRT commented Dec 24, 2005

Perl_magic_get reqires ridiculous amounts of stack memory #8244

Perl_magic_get reqires ridiculous amounts of stack memory #8244

Comments

p5pRT commented Dec 13, 2005

p5pRT commented Dec 13, 2005

From perlbug@plan9.de

Created by perlbug@plan9.de

p5pRT commented Dec 23, 2005

From @smpeters

p5pRT commented Dec 23, 2005

p5pRT commented Dec 23, 2005

From nick@ing-simmons.net

p5pRT commented Dec 23, 2005

From @jimc

p5pRT commented Dec 23, 2005

From @jimc

p5pRT commented Dec 23, 2005

From @jimc

p5pRT commented Dec 24, 2005

From @smpeters

p5pRT commented Dec 24, 2005