From perlbug@schmorp.de

Created by perlbug@schmorp.de

Hi!

I am running into regular segfaults when using modules that use pthreads
and a perl use usemymalloc=y.

The problems come up when perl uses its own malloc and other threads
concurrently use the glibc malloc. It seems that both mallocs can then
return overlapping areas, trashing vital structures and causing spurious
crashes (i.e. segfaults in either perls malloc or glibcs malloc because
their data structures are broken).

These problems go away in a perl with the same configuration but usemyalloc=n.

Investigating the problem further, this could be caused by perl and glibc
both calling sbrk to get more memory, and sbrk is not thread-safe, so
concurrent calls will return the same value to both callers.

I tried this on four perls​:

- debians x86 perl​: usemyalloc=y, threads enabled​: crashes
- my own x86 perl​: usemyalloc=y, threads disabled​: crashes
- my own amd64 perl​: usemyalloc=y, threads disabled​: crashes
- debians amd64 perl​: usemyalloc=n, threads enabled​: works
- my own x86 perl​: usemyalloc=n, threads disabled​: works
- my own amd64 perl​: usemyalloc=n, threads disabled​: works

Summary​:

- facts​:
  * perls malloc and glibc malloc return overlapping
  areas when threads are being used, causing data corruption.
  * sbrk is not thread-safe on gnu/linux.

- conjectures​:
  * using glibc malloc (for perls malloc) instead of sbrk fixes this,
  because glibc malloc is thread-safe.

I can probably test this out if desired. The reason I didn't so far was
because I didn't know how to properly enable the perl sbrk() workaround
when calling Configure.

Flags:
    category=core
    severity=medium

Site configuration information for perl v5.8.8:

Configured by Marc Lehmann at Thu Oct 26 14:18:28 CEST 2006.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=linux, osvers=2.6.17.6, archname=amd64-linux
    uname='linux cerebro 2.6.17.6 #1 smp fri oct 20 19:28:13 cest 2006 x86_64 gnulinux '
    config_args='-Duselargefiles -Dxxxxuse64bitint -Uuse64bitall -Dusemymalloc=n -Dcc=gcc -Dccflags=-DPERL_DONT_CREATE_GVSV -ggdb -Dcppflags=-DPERL_DONT_CREATE_GVSV -D_GNU_SOURCE -I/opt/include -Doptimize=-O4 -march=opteron -mtune=opteron -funroll-loops -fno-strict-aliasing -Dcccdlflags=-fPIC -Dldflags=-L/opt/perl/lib -L/opt/lib -Dlibs=-ldl -lm -lcrypt -Darchname=amd64-linux -Dprefix=/opt/perl -Dprivlib=/opt/perl/lib/perl5 -Darchlib=/opt/perl/lib/perl5 -Dvendorprefix=/opt/perl -Dvendorlib=/opt/perl/lib/perl5 -Dvendorarch=/opt/perl/lib/perl5 -Dsiteprefix=/opt/perl -Dsitelib=/opt/perl/lib/perl5 -Dsitearch=/opt/perl/lib/perl5 -Dsitebin=/opt/perl/bin -Dman1dir=/opt/perl/man/man1 -Dman3dir=/opt/perl/man/man3 -Dsiteman1dir=/opt/perl/man/man1 -Dsiteman3dir=/opt/perl/man/man3 -Dman1ext=1 -Dman3ext=3 -Dpager=/usr/bin/less -Uafs -Uusesfio -Uusenm -Uuseshrplib -Dd_dosuid -Dusethreads=undef -Duse5005threads=undef -Duseithreads=undef -Dusemultiplicity=undef -Demail=perl-binary@plan9.de -Dcf_email=perl-binary@plan9.de -Dcf_by=Marc Lehmann -Dlocincpth=/opt/perl/include /opt/include -Dmyhostname=localhost -Dmultiarch=undef -Dbin=/opt/perl/bin -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-DPERL_DONT_CREATE_GVSV -ggdb -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/opt/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O4 -march=opteron -mtune=opteron -funroll-loops -fno-strict-aliasing',
    cppflags='-DPERL_DONT_CREATE_GVSV -D_GNU_SOURCE -I/opt/include -DPERL_DONT_CREATE_GVSV -ggdb -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/opt/include'
    ccversion='', gccversion='4.1.2 20060901 (prerelease) (Debian 4.1.1-13)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-L/opt/perl/lib -L/opt/lib -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-ldl -lm -lcrypt
    perllibs=-ldl -lm -lcrypt
    libc=/lib/libc-2.3.6.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.3.6'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/opt/perl/lib -L/opt/lib -L/usr/local/lib'

Locally applied patches:
    MAINT28213


@INC for perl v5.8.8:
    /root/src/sex
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    .


Environment for perl v5.8.8:
    HOME=/root
    LANG (unset)
    LANGUAGE (unset)
    LC_CTYPE=de_DE.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/root/s2:/root/s:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11/bin:/usr/games:/root/src/uunet:.
    PERL5LIB=/root/src/sex
    PERL5_CPANPLUS_CONFIG=/root/.cpanplus/config
    PERLDB_OPTS=ornaments=0
    PERL_BADLANG (unset)
    PERL_UNICODE=EAL
    SHELL=/bin/bash

From @doughera88

On Thu, 26 Oct 2006, Marc Lehmann wrote​:

I am running into regular segfaults when using modules that use pthreads
and a perl use usemymalloc=y.

Thanks for the careful investigation. Is it perhaps most sensible to
simply change the hints file to make it very hard to get that combination?
That is, are there significant advantages to usemymalloc='y' on
Linux/glibc so that we should put in some effort to make it possible?

--
  Andy Dougherty doughera@​lafayette.edu

The RT System itself - Status changed from 'new' to 'open'

From @nwc10

On Fri, Oct 27, 2006 at 11​:45​:17AM -0400, Andy Dougherty wrote​:

On Thu, 26 Oct 2006, Marc Lehmann wrote​:

I am running into regular segfaults when using modules that use pthreads
and a perl use usemymalloc=y.

Thanks for the careful investigation. Is it perhaps most sensible to
simply change the hints file to make it very hard to get that combination?
That is, are there significant advantages to usemymalloc='y' on
Linux/glibc so that we should put in some effort to make it possible?

There are advantages to usemymalloc on FreeBSD, and people were seeing
similar problems there with threads. So it may be that Marc has hit the
nail on the head for them too.

If so, I think that various people owe him several $appropriate_beverages

Nicholas Clark

From @doughera88

On Fri, 27 Oct 2006, Nicholas Clark wrote​:

On Fri, Oct 27, 2006 at 11​:45​:17AM -0400, Andy Dougherty wrote​:

On Thu, 26 Oct 2006, Marc Lehmann wrote​:

I am running into regular segfaults when using modules that use pthreads
and a perl use usemymalloc=y.

Thanks for the careful investigation. Is it perhaps most sensible to
simply change the hints file to make it very hard to get that combination?
That is, are there significant advantages to usemymalloc='y' on
Linux/glibc so that we should put in some effort to make it possible?

There are advantages to usemymalloc on FreeBSD, and people were seeing
similar problems there with threads. So it may be that Marc has hit the
nail on the head for them too.

If so, I think that various people owe him several $appropriate_beverages

Ok, that warrants broader investigation. Looking at Solaris, sbrk(2) is
supposed to be threadsafe there, but, the man page has a warning​:

  The behavior of brk() and sbrk() is unspecified if an application
  also uses any other memory functions (such as malloc(3C), mmap(2),
  free(3C)). The brk() and sbrk() functions have been used in
  specialized cases where no other memory allocation function provided
  the same capability. The use of mmap(2) is now preferred because it
  can be used portably with all other memory allocation functions and
  with any function that uses other allocation functions.

and a quick search for 'sbrk' at sun.com shows a couple of bug reports
related to this.

While not exactly related to threads, it certainly is becoming more and
more prevelant to have modules that pull in additional libraries that
might themselves use malloc(). In short, I'm beginning to wonder if
there's *ever* a case where using sbrk() should be the default.

On the other hand, if perl's malloc(size) ends calling the system malloc
for (size - perl's overhead), then I do worry whether we're prone to
head into the doubled-doubling malloc() "optimization" issue.
It may gobble up a lot of memory; only actual testing will tell us for
sure.

--
  Andy Dougherty doughera@​lafayette.edu

From schmorp@schmorp.de

On Fri, Oct 27, 2006 at 09​:10​:10AM -0700, Andy Dougherty via RT <perlbug-followup@​perl.org> wrote​:

Thanks for the careful investigation. Is it perhaps most sensible to
simply change the hints file to make it very hard to get that combination?
That is, are there significant advantages to usemymalloc='y' on
Linux/glibc so that we should put in some effort to make it possible?

Thats the question I ask myself for a decade now... "is the perl malloc more
efficient than the glibc one" (or libc5, before :).

I do not have the slightest. I'd probably go for a modified slice
allocator (as e.g. glib uses) myself if I wanted high perfromance (its
faster than glibc malloc).

In my experience, the different between perl malloc and glibc malloc is,
if it even exists, much much less than the difference between perls slow
process emulation (ithreads) and perls without it.

Then there is the question of memory efficiency.

Accoriding to what I read in documentation and code, glibc malloc should
be more memory efficient and about as fast (and it cares about stuff like
cache coloring). But without hard timings I know nothing for sure.

I guess either somebody who knows could speak up, or somebody who wants to
know enough makes comprehensive benchmarks, or nobody cares and it gets
disabled, or perl uses malloc.

The latter is probably fine if perl allocates "large enough" chunks to not
make this a performance problem.

One could also use mmap, but thats likely much more work for little gain.

On Fri, Oct 27, 2006 at 09​:55​:37AM -0700, Andy Dougherty via RT <perlbug-followup@​perl.org> wrote​:

There are advantages to usemymalloc on FreeBSD, and people were seeing

Thats interesting. Is freebsd malloc just slow or is there something in the
way perl allocates data that favours its own malloc?

(I am just curious, you don't have to answer if you don't know either :)

Ok, that warrants broader investigation. Looking at Solaris, sbrk(2) is
supposed to be threadsafe there, but, the man page has a warning​:

Interesting. In any case, nothing mandates that sbrk is thread-safe, and
glibc usually opts for not being unless SUS mandates it, which would explain
its non-threadsafe sbrk.

While not exactly related to threads, it certainly is becoming more and
more prevelant to have modules that pull in additional libraries that
might themselves use malloc().

Well, in my case, I had to use the system malloc simply because I doubt
very much the perl malloc is threadsafe (when threads are disabled).

there's *ever* a case where using sbrk() should be the default.

It is probably better to make it an exception, and possibly increase the
amount of memory allocated by perls sbrk-on-malloc emulation to somethign
like 64k+.

On the other hand, if perl's malloc(size) ends calling the system malloc
for (size - perl's overhead), then I do worry whether we're prone to
head into the doubled-doubling malloc() "optimization" issue.
It may gobble up a lot of memory; only actual testing will tell us for
sure.

Uhu.

Keep in mind that I didn't test the sbrk theory, I made it up purely in my
mind​: it could be something else.

--
  The choice of a
  -----==- _GNU_
  ----==-- _ generation Marc Lehmann
  ---==---(_)__ __ ____ __ pcg@​goof.com
  --==---/ / _ \/ // /\ \/ / http​://schmorp.de/
  -=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE