Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re: Regular rarely segmentation fault in libperl.so #11841

Open
p5pRT opened this issue Jan 2, 2012 · 12 comments
Open

Re: Regular rarely segmentation fault in libperl.so #11841

p5pRT opened this issue Jan 2, 2012 · 12 comments

Comments

@p5pRT
Copy link

p5pRT commented Jan 2, 2012

Migrated from rt.perl.org#107392 (status was 'open')

Searchable as RT107392$

@p5pRT
Copy link
Author

p5pRT commented Jan 2, 2012

From perlover@perlover.com

Sorry, previous letter had a bad formatted lines
Here normal letter​:

This is a bug report for perl from perlover@​perlover.com,
generated with the help of perlbug 1.39 running under perl 5.10.1.


Hello, developers of perl!

I regulary every day get a core files and segmentation fault in libperl.so
I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64
bit) again after problems
but this bug remained. Other programs work find in my server so i
think this is problem of perl, not memory or bad binaries (i
recompiled it).

Here some backtraces of perl without debug info (i cannot add debug
info - server should work under big loading, sorry)

1st 'bt' from gdb​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newBINOP (my_perl=0x139c010, type=136, flags=0,
first=0x21ab840, last=0x21ab800) at op.c​:3074
3074 op.c​: No such file or directory.
  in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-123.fc13.x86_64
(gdb) bt
#0 Perl_newBINOP (my_perl=0x139c010, type=136, flags=0,
first=0x21ab840, last=0x21ab800) at op.c​:3074
#1 0x00007f78f433167d in Perl_yyparse (my_perl=0x139c010) at perly.y​:819
#2 0x00007f78f439a0ba in S_doeval (my_perl=0x139c010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#3 0x00007f78f439bb65 in Perl_pp_require (my_perl=0x139c010) at pp_ctl.c​:3573
#4 0x00007f78f43657d6 in Perl_runops_standard (my_perl=0x139c010) at run.c​:40
#5 0x00007f78f430af28 in S_run_body (my_perl=0x139c010) at perl.c​:2435
#6 perl_run (my_perl=0x139c010) at perl.c​:2353
#7 0x0000000000400b9c in main ()


Second core file​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newOP (my_perl=0x1c88010, type=12, flags=0) at op.c​:3024
3024 op.c​: No such file or directory.
  in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-123.fc13.x86_64
(gdb) bt
#0 Perl_newOP (my_perl=0x1c88010, type=12, flags=0) at op.c​:3024
#1 0x00007f40fe0cda42 in S_pending_ident (my_perl=0x1c88010) at toke.c​:7106
#2 Perl_yylex (my_perl=0x1c88010) at toke.c​:3337
#3 0x00007f40fe0d9cf6 in Perl_yyparse (my_perl=0x1c88010) at perly.c​:409
#4 0x00007f40fe1440ba in S_doeval (my_perl=0x1c88010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#5 0x00007f40fe145b65 in Perl_pp_require (my_perl=0x1c88010) at pp_ctl.c​:3573
#6 0x00007f40fe10f7d6 in Perl_runops_standard (my_perl=0x1c88010) at run.c​:40
#7 0x00007f40fe0b4f28 in S_run_body (my_perl=0x1c88010) at perl.c​:2435
#8 perl_run (my_perl=0x1c88010) at perl.c​:2353
#9 0x0000000000400b9c in main ()


3rd core file​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newBINOP (my_perl=0xa4f010, type=136, flags=0,
first=0x175d7f0, last=0x175d950) at op.c​:3074
3074 op.c​: No such file or directory.
  in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-123.fc13.x86_64
(gdb) bt
#0 Perl_newBINOP (my_perl=0xa4f010, type=136, flags=0,
first=0x175d7f0, last=0x175d950) at op.c​:3074
#1 0x00007fca68d146dd in Perl_yyparse (my_perl=0xa4f010) at perly.y​:809
#2 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#3 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c​:3573
#4 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c​:40
#5 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010,
sv=0x1711840, flags=6) at perl.c​:2721
#6 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010,
oldscope=21, paramList=0x1711780) at perl.c​:5275
#7 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010,
fullname=<value optimized out>, gv=0x17118e8, cv=0x1711840) at
op.c​:5864
#8 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010,
floor=363, o=<value optimized out>, proto=<value optimized out>,
attrs=0x0, block=0x1686250) at op.c​:5835
#9 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010, aver=1,
floor=363, version=<value optimized out>, idop=0x16edb40, arg=<value
optimized out>) at op.c​:3878
#10 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at perly.y​:659
#11 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#12 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c​:3573
#13 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c​:40
#14 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010,
sv=0x16a6970, flags=6) at perl.c​:2721
#15 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010,
oldscope=15, paramList=0x1390850) at perl.c​:5275
#16 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010,
fullname=<value optimized out>, gv=0xbeb8e0, cv=0x16a6970) at
op.c​:5864
#17 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010,
floor=224, o=<value optimized out>, proto=<value optimized out>,
attrs=0x0, block=0x1369240) at op.c​:5835
#18 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010, aver=1,
floor=224, version=<value optimized out>, idop=0x1392090, arg=<value
optimized out>) at op.c​:3878
#19 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at perly.y​:659
#20 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#21 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at pp_ctl.c​:3573
#22 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at run.c​:40
#23 0x00007fca68cedf28 in S_run_body (my_perl=0xa4f010) at perl.c​:2435
#24 perl_run (my_perl=0xa4f010) at perl.c​:2353
#25 0x0000000000400b9c in main ()


4th core file​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newSTATEOP (my_perl=0xf6d010, flags=0, label=0x0,
o=0x1bb72d0) at op.c​:4350
4350 op.c​: No such file or directory.
  in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-123.fc13.x86_64
(gdb) bt
#0 Perl_newSTATEOP (my_perl=0xf6d010, flags=0, label=0x0,
o=0x1bb72d0) at op.c​:4350
#1 0x00007f09571c61c8 in Perl_yyparse (my_perl=0xf6d010) at perly.y​:230
#2 0x00007f09572300ba in S_doeval (my_perl=0xf6d010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#3 0x00007f0957231b65 in Perl_pp_require (my_perl=0xf6d010) at pp_ctl.c​:3573
#4 0x00007f09571fb7d6 in Perl_runops_standard (my_perl=0xf6d010) at run.c​:40
#5 0x00007f09571a0f28 in S_run_body (my_perl=0xf6d010) at perl.c​:2435
#6 perl_run (my_perl=0xf6d010) at perl.c​:2353
#7 0x0000000000400b9c in main ()

And so on...
Always core files will happen in one place​: op.c file

I don't know fixed this bug or no but i didn't find in Google some
bugs like this.

Best regards,
Perlover



Flags​:
  category=core
  severity=low


This perlbug was built using Perl 5.10.1 in the Fedora build system.
It is being executed now by Perl 5.10.1 - Wed Dec 28 17​:51​:47 YEKT 2011.

Site configuration information for perl 5.10.1​:

Configured by Red Hat, Inc. at Wed Dec 28 17​:51​:47 YEKT 2011.

Summary of my perl5 (revision 5 version 10 subversion 1) configuration​:

  Platform​:
  osname=linux, osvers=2.6.34.9-69.fc13.x86_64,
archname=x86_64-linux-thread-multi
  uname='linux 213.174.142.19 2.6.34.9-69.fc13.x86_64 #1 smp tue may
3 09​:23​:03 utc 2011 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Doptimize=-O2 -g
-Dccdlflags=-Wl,--enable-new-dtags -DDEBUGGING=-g -Dversion=5.10.1
-Dmyhostname=localhost -Dperladmin=root@​localhost -Dcc=gcc -Dcf_by=Red
Hat, Inc. -Dprefix=/usr -Dvend
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.4.5 20101112 (Red Hat 4.4.5-2)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -fstack-protector'
  libpth=/usr/local/lib64 /lib64 /usr/lib64
  libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
  perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.12.2'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64/perl5/CORE'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -fstack-protector'

Locally applied patches​:


@​INC for perl 5.10.1​:
  /home/cjperl/local/lib/perl5/x86_64-linux-thread-multi
  /home/cjperl/local/lib/perl5
  /usr/local/lib64/perl5
  /usr/local/share/perl5
  /usr/lib64/perl5
  /usr/share/perl5
  /usr/lib64/perl5
  /usr/share/perl5
  /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
  /usr/local/lib/perl5/site_perl/5.10.0
  /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
  /usr/lib/perl5/vendor_perl
  /usr/lib/perl5/site_perl
  .


Environment for perl 5.10.1​:
  HOME=/home/cjperl
  LANG=ru_RU.utf8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/home/cjperl/local/bin​:/usr/kerberos/sbin​:/usr/kerberos/bin​:/usr/local/bin​:/bin​:/usr/bin​:/usr/local/sbin​:/usr/sbin​:/sbin​:/home/cjperl/bin
  PERL5LIB=/home/cjperl/local/lib/perl5
  PERL_BADLANG (unset)
  SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Jan 2, 2012

From @cpansprout

On Mon Jan 02 03​:25​:46 2012, Perlover wrote​:

Sorry, previous letter had a bad formatted lines
Here normal letter​:

This is a bug report for perl from perlover@​perlover.com,
generated with the help of perlbug 1.39 running under perl 5.10.1.

-----------------------------------------------------------------

Hello, developers of perl!

I regulary every day get a core files and segmentation fault in
libperl.so
I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64
bit) again after problems
but this bug remained. Other programs work find in my server so i
think this is problem of perl, not memory or bad binaries (i
recompiled it).

Here some backtraces of perl without debug info (i cannot add debug
info - server should work under big loading, sorry)

Are you using any XS modules that create their own ops or fiddle with
PL_check?

Do you know what Perl code is being compiled?

Typing

  call Perl_warn(my_perl, "here")

in the debugger (omit my_perl for non-threaded builds) should result in
output like this​:

  here at lib/Foo/Bar.pm line 3123.

Maybe you could send some code snippets surrounding the offending lines.

I see you are using 5.10.1. There have been a ton of bug fixes since
then, so this problem may already be gone in a later perl. Also, please
be aware that 5.10 is not supported any more. It’s not likely to get
any new bug fixes.

1st 'bt' from gdb​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newBINOP (my_perl=0x139c010, type=136, flags=0,
first=0x21ab840, last=0x21ab800) at op.c​:3074
3074 op.c​: No such file or directory.
in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-
123.fc13.x86_64
(gdb) bt
#0 Perl_newBINOP (my_perl=0x139c010, type=136, flags=0,
first=0x21ab840, last=0x21ab800) at op.c​:3074
#1 0x00007f78f433167d in Perl_yyparse (my_perl=0x139c010) at
perly.y​:819
#2 0x00007f78f439a0ba in S_doeval (my_perl=0x139c010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#3 0x00007f78f439bb65 in Perl_pp_require (my_perl=0x139c010) at
pp_ctl.c​:3573
#4 0x00007f78f43657d6 in Perl_runops_standard (my_perl=0x139c010) at
run.c​:40
#5 0x00007f78f430af28 in S_run_body (my_perl=0x139c010) at
perl.c​:2435
#6 perl_run (my_perl=0x139c010) at perl.c​:2353
#7 0x0000000000400b9c in main ()

---

Second core file​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newOP (my_perl=0x1c88010, type=12, flags=0) at op.c​:3024
3024 op.c​: No such file or directory.
in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-
123.fc13.x86_64
(gdb) bt
#0 Perl_newOP (my_perl=0x1c88010, type=12, flags=0) at op.c​:3024
#1 0x00007f40fe0cda42 in S_pending_ident (my_perl=0x1c88010) at
toke.c​:7106
#2 Perl_yylex (my_perl=0x1c88010) at toke.c​:3337
#3 0x00007f40fe0d9cf6 in Perl_yyparse (my_perl=0x1c88010) at
perly.c​:409
#4 0x00007f40fe1440ba in S_doeval (my_perl=0x1c88010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#5 0x00007f40fe145b65 in Perl_pp_require (my_perl=0x1c88010) at
pp_ctl.c​:3573
#6 0x00007f40fe10f7d6 in Perl_runops_standard (my_perl=0x1c88010) at
run.c​:40
#7 0x00007f40fe0b4f28 in S_run_body (my_perl=0x1c88010) at
perl.c​:2435
#8 perl_run (my_perl=0x1c88010) at perl.c​:2353
#9 0x0000000000400b9c in main ()

---

3rd core file​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newBINOP (my_perl=0xa4f010, type=136, flags=0,
first=0x175d7f0, last=0x175d950) at op.c​:3074
3074 op.c​: No such file or directory.
in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-
123.fc13.x86_64
(gdb) bt
#0 Perl_newBINOP (my_perl=0xa4f010, type=136, flags=0,
first=0x175d7f0, last=0x175d950) at op.c​:3074
#1 0x00007fca68d146dd in Perl_yyparse (my_perl=0xa4f010) at
perly.y​:809
#2 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#3 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at
pp_ctl.c​:3573
#4 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at
run.c​:40
#5 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010,
sv=0x1711840, flags=6) at perl.c​:2721
#6 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010,
oldscope=21, paramList=0x1711780) at perl.c​:5275
#7 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010,
fullname=<value optimized out>, gv=0x17118e8, cv=0x1711840) at
op.c​:5864
#8 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010,
floor=363, o=<value optimized out>, proto=<value optimized out>,
attrs=0x0, block=0x1686250) at op.c​:5835
#9 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010, aver=1,
floor=363, version=<value optimized out>, idop=0x16edb40, arg=<value
optimized out>) at op.c​:3878
#10 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at
perly.y​:659
#11 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#12 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at
pp_ctl.c​:3573
#13 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at
run.c​:40
#14 0x00007fca68ced42f in Perl_call_sv (my_perl=0xa4f010,
sv=0x16a6970, flags=6) at perl.c​:2721
#15 0x00007fca68ced96d in Perl_call_list (my_perl=0xa4f010,
oldscope=15, paramList=0x1390850) at perl.c​:5275
#16 0x00007fca68cd9c39 in S_process_special_blocks (my_perl=0xa4f010,
fullname=<value optimized out>, gv=0xbeb8e0, cv=0x16a6970) at
op.c​:5864
#17 0x00007fca68ce7612 in Perl_newATTRSUB (my_perl=0xa4f010,
floor=224, o=<value optimized out>, proto=<value optimized out>,
attrs=0x0, block=0x1369240) at op.c​:5835
#18 0x00007fca68ce6593 in Perl_utilize (my_perl=0xa4f010, aver=1,
floor=224, version=<value optimized out>, idop=0x1392090, arg=<value
optimized out>) at op.c​:3878
#19 0x00007fca68d14c87 in Perl_yyparse (my_perl=0xa4f010) at
perly.y​:659
#20 0x00007fca68d7d0ba in S_doeval (my_perl=0xa4f010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#21 0x00007fca68d7eb65 in Perl_pp_require (my_perl=0xa4f010) at
pp_ctl.c​:3573
#22 0x00007fca68d487d6 in Perl_runops_standard (my_perl=0xa4f010) at
run.c​:40
#23 0x00007fca68cedf28 in S_run_body (my_perl=0xa4f010) at perl.c​:2435
#24 perl_run (my_perl=0xa4f010) at perl.c​:2353
#25 0x0000000000400b9c in main ()

---

4th core file​:

Program terminated with signal 11, Segmentation fault.
#0 Perl_newSTATEOP (my_perl=0xf6d010, flags=0, label=0x0,
o=0x1bb72d0) at op.c​:4350
4350 op.c​: No such file or directory.
in op.c
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-
123.fc13.x86_64
(gdb) bt
#0 Perl_newSTATEOP (my_perl=0xf6d010, flags=0, label=0x0,
o=0x1bb72d0) at op.c​:4350
#1 0x00007f09571c61c8 in Perl_yyparse (my_perl=0xf6d010) at
perly.y​:230
#2 0x00007f09572300ba in S_doeval (my_perl=0xf6d010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#3 0x00007f0957231b65 in Perl_pp_require (my_perl=0xf6d010) at
pp_ctl.c​:3573
#4 0x00007f09571fb7d6 in Perl_runops_standard (my_perl=0xf6d010) at
run.c​:40
#5 0x00007f09571a0f28 in S_run_body (my_perl=0xf6d010) at perl.c​:2435
#6 perl_run (my_perl=0xf6d010) at perl.c​:2353
#7 0x0000000000400b9c in main ()

And so on...
Always core files will happen in one place​: op.c file

I don't know fixed this bug or no but i didn't find in Google some
bugs like this.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jan 2, 2012

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jan 3, 2012

From perlover@perlover.com

I think this but may be related with bug # 72406 (
https://rt-archive.perl.org/perl5/Public/Bug/Display.html?id=72406 )
I got same segmentation fault if i do​: perl -le 'do{print("foobar");}until(1)}'

@p5pRT
Copy link
Author

p5pRT commented Jan 4, 2012

From @iabyn

On Mon, Jan 02, 2012 at 03​:25​:47AM -0800, Perlover wrote​:

I regulary every day get a core files and segmentation fault in libperl.so
I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64
bit) again after problems
but this bug remained. Other programs work find in my server so i
think this is problem of perl, not memory or bad binaries (i
recompiled it).

Here some backtraces of perl without debug info (i cannot add debug
info - server should work under big loading, sorry)

[snip]

All the segfaults occur where the code is doing the first dereference of a
newly-malloced OP. Interestingly, perl doesn't check the return value of
malloc for a new OP, so it won't detect malloc errors (i.e. out of memory)
in those situations.

So I reckon that the most likely thing is that your perl process is
running out of memory (either because your server doesn't have enough, or
because your perl processes are run within a ulimit-restricted
environment). Which perl then fails to report cleanly.

--
In the 70's we wore flares because we didn't know any better.
What possible excuse does the current generation have?

@p5pRT
Copy link
Author

p5pRT commented Jan 4, 2012

From mark@exonetric.com

On 4 Jan 2012, at 13​:08, Dave Mitchell wrote​:

On Mon, Jan 02, 2012 at 03​:25​:47AM -0800, Perlover wrote​:

I regulary every day get a core files and segmentation fault in libperl.so
I already recompiled perl from sources (SRPMs Linux Fedora Core 13 64
bit) again after problems
but this bug remained. Other programs work find in my server so i
think this is problem of perl, not memory or bad binaries (i
recompiled it).

Here some backtraces of perl without debug info (i cannot add debug
info - server should work under big loading, sorry)

[snip]

All the segfaults occur where the code is doing the first dereference of a
newly-malloced OP. Interestingly, perl doesn't check the return value of
malloc for a new OP, so it won't detect malloc errors (i.e. out of memory)
in those situations.

So I reckon that the most likely thing is that your perl process is
running out of memory (either because your server doesn't have enough, or
because your perl processes are run within a ulimit-restricted
environment). Which perl then fails to report cleanly.

Now, this is an interesting report for us as we have recently implemented
address-space limits (as data segment limits are no good with the
system malloc). We routinely see segfaults as well and we *know*
it's do with running out of address space after a malloc.

I was a little surprised that such an easy to check condition was resulting
in a segfault. This is under perl5.10.0 on 64-bit amd64 linux 2.6.26-something.

- Mark

@p5pRT
Copy link
Author

p5pRT commented Jan 4, 2012

From perlover@perlover.com

Good day!

[snip]

All the segfaults occur where the code is doing the first dereference of a
newly-malloced OP. Interestingly, perl doesn't check the return value of
malloc for a new OP, so it won't detect malloc errors (i.e. out of memory)
in those situations.

Yes, it may be
These scripts are run from not root
This user has a soft limits as​:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128522
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Here "max memory size" is unlimited
But "stack size" is 10Mb
Can this (stack size) affect to new OP or malloc? I think malloc
limited by "max memory size" not stack size

So I reckon that the most likely thing is that your perl process is
running out of memory (either because your server doesn't have enough, or
because your perl processes are run within a ulimit-restricted
environment). Which perl then fails to report cleanly.

ulimits i dumped above

Best regards, Perlover

@p5pRT
Copy link
Author

p5pRT commented Jan 8, 2012

From @iabyn

On Wed, Jan 04, 2012 at 02​:59​:46PM +0100, Perlover wrote​:

All the segfaults occur where the code is doing the first dereference of a
newly-malloced OP. Interestingly, perl doesn't check the return value of
malloc for a new OP, so it won't detect malloc errors (i.e. out of memory)
in those situations.

Yes, it may be
These scripts are run from not root
This user has a soft limits as​:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128522
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Here "max memory size" is unlimited
But "stack size" is 10Mb
Can this (stack size) affect to new OP or malloc? I think malloc
limited by "max memory size" not stack size

Yeah, it shouldn't be affected by stack size (that would just trigger a
SEGV at random places where it tries to call a function).

Could you apply this crude patch to your 5.10.1 perl? It checks for
a zero return from malloc(), outputs a message to stderr, and aborts. With
it we might at least see whether this is where the problem lies.

Inline Patch
--- op.h-	2012-01-08 15:54:03.944453254 +0000
+++ op.h	2012-01-08 16:33:07.592199391 +0000
@@ -640,9 +640,14 @@
 	(var = (OP *) Perl_Slab_Alloc(aTHX_ size))
 #define FreeOp(p) Perl_Slab_Free(aTHX_ p)
 #else
-#define NewOp(m, var, c, type)	\
+#define NewOp(m, var, c, type)	STMT_START { \
 	(var = (MEM_WRAP_CHECK_(c,type) \
-	 (type*)PerlMemShared_calloc(c, sizeof(type))))
+	 (type*)PerlMemShared_calloc(c, sizeof(type)))); \
+	 if (!var) { \
+	    write(2, STR_WITH_LEN("Zero malloc in NewOp\n")); \
+	    abort(); \
+	} \
+	} STMT_END;
 #define NewOpSz(m, var, size)	\
 	(var = (OP*)PerlMemShared_calloc(1, size))
 #define FreeOp(p) PerlMemShared_free(p)

-- 

A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

@p5pRT
Copy link
Author

p5pRT commented Jan 17, 2012

From @iabyn

On Tue, Jan 17, 2012 at 11​:14​:26AM +0100, Perlover wrote​:

I think it is good that happened because it signals to us that your
abort() was executed and so problem in out of memory of malloc

What do you think?

Yeah, I think it's definitely a lack of memory problem at your end, so
there's not much we can do to fix that. What we do need to do at this end
though, is to fix that bit of perl code so that it detects and reports out
of memory errors.

--
The warp engines start playing up a bit, but seem to sort themselves out
after a while without any intervention from boy genius Wesley Crusher.
  -- Things That Never Happen in "Star Trek" #17

@p5pRT
Copy link
Author

p5pRT commented Jan 18, 2012

From perlover@perlover.com

Good day, Dave and everybody in perl bug list!

I have two core files after your patches
My STDERR is empty after segfaults and it looks like segfault from your abort()

Here backtrace from gdb​:

#0 0x00007f07e7db28f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-123.fc13.x86_64
(gdb) bt
#0 0x00007f07e7db28f5 in raise () from /lib64/libc.so.6
#1 0x00007f07e7db40d5 in abort () from /lib64/libc.so.6
#2 0x00007f07e904a363 in Perl_newSTATEOP (my_perl=0x239e010, flags=0,
label=0x0, o=0x32b5080) at op.c​:4344
#3 0x00007f07e907d458 in Perl_yyparse (my_perl=0x239e010) at perly.y​:230
#4 0x00007f07e90e734a in S_doeval (my_perl=0x239e010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#5 0x00007f07e90e8df5 in Perl_pp_require (my_perl=0x239e010) at pp_ctl.c​:3573
#6 0x00007f07e90b2a66 in Perl_runops_standard (my_perl=0x239e010) at run.c​:40
#7 0x00007f07e90576bf in Perl_call_sv (my_perl=0x239e010,
sv=0x2cd1288, flags=6) at perl.c​:2721
#8 0x00007f07e9057bfd in Perl_call_list (my_perl=0x239e010,
oldscope=14, paramList=0x25a1238) at perl.c​:5275
#9 0x00007f07e9043c39 in S_process_special_blocks (my_perl=0x239e010,
fullname=<value optimized out>, gv=0x2cc50a8, cv=0x2cd1288) at
op.c​:5864
#10 0x00007f07e9051872 in Perl_newATTRSUB (my_perl=0x239e010,
floor=217, o=<value optimized out>, proto=<value optimized out>,
attrs=0x0, block=0x2d6f890) at op.c​:5835
#11 0x00007f07e90507f3 in Perl_utilize (my_perl=0x239e010, aver=1,
floor=217, version=<value optimized out>, idop=0x2c74d60, arg=<value
optimized out>) at op.c​:3878
#12 0x00007f07e907ef17 in Perl_yyparse (my_perl=0x239e010) at perly.y​:659
#13 0x00007f07e90e734a in S_doeval (my_perl=0x239e010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#14 0x00007f07e90e8df5 in Perl_pp_require (my_perl=0x239e010) at pp_ctl.c​:3573
#15 0x00007f07e90b2a66 in Perl_runops_standard (my_perl=0x239e010) at run.c​:40
#16 0x00007f07e90581b8 in S_run_body (my_perl=0x239e010) at perl.c​:2435
#17 perl_run (my_perl=0x239e010) at perl.c​:2353
#18 0x0000000000400b9c in main ()

And second corefile​:
#0 0x00007f0f7645a8f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use​: debuginfo-install perl-5.10.1-123.fc13.x86_64
(gdb) bt
#0 0x00007f0f7645a8f5 in raise () from /lib64/libc.so.6
#1 0x00007f0f7645c0d5 in abort () from /lib64/libc.so.6
#2 0x00007f0f776f1b6a in Perl_newOP (my_perl=0x17c5010, type=12,
flags=0) at op.c​:3023
#3 0x00007f0f77718cd2 in S_pending_ident (my_perl=0x17c5010) at toke.c​:7106
#4 Perl_yylex (my_perl=0x17c5010) at toke.c​:3337
#5 0x00007f0f77724f86 in Perl_yyparse (my_perl=0x17c5010) at perly.c​:409
#6 0x00007f0f7778f34a in S_doeval (my_perl=0x17c5010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#7 0x00007f0f77790df5 in Perl_pp_require (my_perl=0x17c5010) at pp_ctl.c​:3573
#8 0x00007f0f7775aa66 in Perl_runops_standard (my_perl=0x17c5010) at run.c​:40
#9 0x00007f0f776ff6bf in Perl_call_sv (my_perl=0x17c5010,
sv=0x1961880, flags=6) at perl.c​:2721
#10 0x00007f0f776ffbfd in Perl_call_list (my_perl=0x17c5010,
oldscope=15, paramList=0x20f7fe0) at perl.c​:5275
#11 0x00007f0f776ebc39 in S_process_special_blocks (my_perl=0x17c5010,
fullname=<value optimized out>, gv=0x19618e0, cv=0x1961880) at
op.c​:5864
#12 0x00007f0f776f9872 in Perl_newATTRSUB (my_perl=0x17c5010,
floor=224, o=<value optimized out>, proto=<value optimized out>,
attrs=0x0, block=0x21982a0) at op.c​:5835
#13 0x00007f0f776f87f3 in Perl_utilize (my_perl=0x17c5010, aver=1,
floor=224, version=<value optimized out>, idop=0x22888e0, arg=<value
optimized out>) at op.c​:3878
#14 0x00007f0f77726f17 in Perl_yyparse (my_perl=0x17c5010) at perly.y​:659
#15 0x00007f0f7778f34a in S_doeval (my_perl=0x17c5010, gimme=0,
startop=0x0, outside=<value optimized out>, seq=<value optimized out>)
at pp_ctl.c​:2981
#16 0x00007f0f77790df5 in Perl_pp_require (my_perl=0x17c5010) at pp_ctl.c​:3573
#17 0x00007f0f7775aa66 in Perl_runops_standard (my_perl=0x17c5010) at run.c​:40
#18 0x00007f0f777001b8 in S_run_body (my_perl=0x17c5010) at perl.c​:2435
#19 perl_run (my_perl=0x17c5010) at perl.c​:2353
#20 0x0000000000400b9c in main ()

I think it is good that happened because it signals to us that your
abort() was executed and so problem in out of memory of malloc

What do you think?

Perlover

2012/1/9 Perlover <perlover@​perlover.com>​:

I patched and installed perl
I will let know to you about "Zero malloc in NewOp" messages in STDERR
if its will be

Best regards :)

2012/1/8 Dave Mitchell <davem@​iabyn.com>​:

On Wed, Jan 04, 2012 at 02​:59​:46PM +0100, Perlover wrote​:

All the segfaults occur where the code is doing the first dereference of a
newly-malloced OP. Interestingly, perl doesn't check the return value of
malloc for a new OP, so it won't detect malloc errors (i.e. out of memory)
in those situations.

Yes, it may be
These scripts are run from not root
This user has a soft limits as​:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128522
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Here "max memory size" is unlimited
But "stack size" is 10Mb
Can this (stack size) affect to new OP or malloc? I think malloc
limited by "max memory size" not stack size

Yeah, it shouldn't be affected by stack size (that would just trigger a
SEGV at random places where it tries to call a function).

Could you apply this crude patch to your 5.10.1 perl? It checks for
a zero return from malloc(), outputs a message to stderr, and aborts. With
it we might at least see whether this is where the problem lies.

--- op.h-       2012-01-08 15​:54​:03.944453254 +0000
+++ op.h        2012-01-08 16​:33​:07.592199391 +0000
@​@​ -640,9 +640,14 @​@​
       (var = (OP *) Perl_Slab_Alloc(aTHX_ size))
 #define FreeOp(p) Perl_Slab_Free(aTHX_ p)
 #else
-#define NewOp(m, var, c, type) \
+#define NewOp(m, var, c, type) STMT_START { \
       (var = (MEM_WRAP_CHECK_(c,type) \
-        (type*)PerlMemShared_calloc(c, sizeof(type))))
+        (type*)PerlMemShared_calloc(c, sizeof(type)))); \
+        if (!var) { \
+           write(2, STR_WITH_LEN("Zero malloc in NewOp\n")); \
+           abort(); \
+       } \
+       } STMT_END;
 #define NewOpSz(m, var, size)  \
       (var = (OP*)PerlMemShared_calloc(1, size))
 #define FreeOp(p) PerlMemShared_free(p)

--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

@p5pRT
Copy link
Author

p5pRT commented Feb 3, 2013

From @jkeenan

On Tue Jan 17 03​:48​:20 2012, davem wrote​:

On Tue, Jan 17, 2012 at 11​:14​:26AM +0100, Perlover wrote​:

I think it is good that happened because it signals to us that your
abort() was executed and so problem in out of memory of malloc

What do you think?

Yeah, I think it's definitely a lack of memory problem at your end, so
there's not much we can do to fix that. What we do need to do at this end
though, is to fix that bit of perl code so that it detects and reports out
of memory errors.

Dave, do you have any suggestions as to how we could fix this failure to
detect out-of-memory problem?

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Feb 4, 2013

From @iabyn

On Sun, Feb 03, 2013 at 10​:48​:41AM -0800, James E Keenan via RT wrote​:

On Tue Jan 17 03​:48​:20 2012, davem wrote​:

On Tue, Jan 17, 2012 at 11​:14​:26AM +0100, Perlover wrote​:

I think it is good that happened because it signals to us that your
abort() was executed and so problem in out of memory of malloc

What do you think?

Yeah, I think it's definitely a lack of memory problem at your end, so
there's not much we can do to fix that. What we do need to do at this end
though, is to fix that bit of perl code so that it detects and reports out
of memory errors.

Dave, do you have any suggestions as to how we could fix this failure to
detect out-of-memory problem?

Not without looking at the code (and I'd look at the code if/when I was in
the process of fixing the issue)

--
The Enterprise successfully ferries an alien VIP from one place to another
without serious incident.
  -- Things That Never Happen in "Star Trek" #7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants