Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash in Perl_hv_common / S_share_hek_flags #14123

Closed
p5pRT opened this issue Sep 30, 2014 · 22 comments
Closed

crash in Perl_hv_common / S_share_hek_flags #14123

p5pRT opened this issue Sep 30, 2014 · 22 comments

Comments

@p5pRT
Copy link

p5pRT commented Sep 30, 2014

Migrated from rt.perl.org#122868 (status was 'open')

Searchable as RT122868$

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2014

From @marcin-gryszkalis

Created by @marcin-gryszkalis

This is a bug report for perl from mg@​fork.pl,
generated with the help of perlbug 1.40 running under perl 5.20.0.

-----------------------------------------------------------------
We have perl based web application. It crashes with segv several times a day.
So it's not easily repeatable (as the creash happens 1 in 100K http requests served).
I have perl 5.20 build with symbols and can provide any gdb stuff needed - but
I cannot provide you with coredump as it contains sensitive data (that's production
environment).

Stacktrace looks like this​:

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00, str=0x815d59660 "COUNT", len=5, hash=1160607206, flags=1024) at hv.c​:2872
2872 if (HeHASH(entry) != hash) /* strings can't be equal */
[New Thread 801c06400 (LWP 100627/perl)]
(gdb) bt
#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00, str=0x815d59660 "COUNT", len=5, hash=1160607206, flags=1024) at hv.c​:2872
#1 0x00000008008df1e6 in Perl_hv_common (my_perl=0x801c19e00, hv=0x816814390, keysv=<value optimized out>, key=0x815d59660 "COUNT", klen=5, flags=<value optimized out>,
  action=<value optimized out>, val=0x8167ea0f0, hash=<value optimized out>) at hv.c​:782
#2 0x00000008009146f9 in Perl_pp_hslice (my_perl=0x801c19e00) at pp.c​:4770
#3 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at run.c​:42
#4 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at perl.c​:2456
#5 0x0000000000400eda in main ()

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00, str=0x816727f30 "cdr", len=3, hash=2934883949, flags=258) at hv.c​:2872
2872 if (HeHASH(entry) != hash) /* strings can't be equal */
[New Thread 801c06400 (LWP 102604/perl)]
(gdb) bt
#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00, str=0x816727f30 "cdr", len=3, hash=2934883949, flags=258) at hv.c​:2872
#1 0x00000008008df1e6 in Perl_hv_common (my_perl=0x801c19e00, hv=0x816afc9d8, keysv=<value optimized out>, key=0x816727f30 "cdr", klen=3, flags=<value optimized out>,
  action=<value optimized out>, val=0x816add810, hash=<value optimized out>) at hv.c​:782
#2 0x00000008008e9bd5 in Perl_pp_helem (my_perl=0x801c19e00) at pp_hot.c​:1769
#3 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at run.c​:42
#4 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at perl.c​:2456
#5 0x0000000000400eda in main ()

entry seems to be broken

(gdb) p *entry
$10 = {hent_next = 0x67007100726463, hent_hek = 0x594152002f, he_valu = {hent_val = 0x480038312e30, hent_refcount = 79165779947056}}

but I don't know much about hash entry internals...

Perl Info

Flags:
    category=core
    severity=high

Site configuration information for perl 5.20.0:

Configured by mg at Mon Sep 22 11:05:52 CEST 2014.

Summary of my perl5 (revision 5 version 20 subversion 0) configuration:
   
  Platform:
    osname=freebsd, osvers=10.0-release-p7, archname=amd64-freebsd-thread-multi
    uname='freebsd piglet.obligo.pl 10.0-release-p7 freebsd 10.0-release-p7 #2 r268534: sat jul 12 00:27:11 cest 2014 root@piglet.obligo.pl:usrobjusrsrcsyspiglet amd64 '
    config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.20/mach -Dprivlib=/usr/local/lib/perl5/5.20 -Dman3dir=/usr/local/lib/perl5/5.20/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.20/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.20 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.20/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.20/BSDPAN" -Doptimize=-O2 -pipe -O2 -pipe -g -march=native -fno-strict-aliasing -Ui_gdbm -Duse64bitint -Dusethreads=y -Dusemymalloc=n'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.20/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
    optimize='-O2 -pipe -O2 -pipe -g -march=native -fno-strict-aliasing',
    cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.20/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-pthread -Wl,-E  -fstack-protector -L/usr/local/lib'
    libpth=/usr/lib /usr/local/lib /usr/include/clang/3.3 /usr/lib
    libs=-lgdbm -lm -lcrypt -lutil
    perllibs=-lm -lcrypt -lutil
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -Wl,-R/usr/local/lib/perl5/5.20/mach/CORE'
    cccdlflags='-DPIC -fPIC', lddlflags='-shared  -L/usr/local/lib -fstack-protector'



@INC for perl 5.20.0:
    /usr/local/lib/perl5/5.20/BSDPAN
    /usr/local/lib/perl5/site_perl/5.20/mach
    /usr/local/lib/perl5/site_perl/5.20
    /usr/local/lib/perl5/5.20/mach
    /usr/local/lib/perl5/5.20
    .


Environment for perl 5.20.0:
    HOME=/root
    LANG (unset)
    LANGUAGE (unset)
    LC_COLLATE=pl_PL.UTF-8
    LC_CTYPE=pl_PL.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/usr/games:/usr/kde/4/bin:/usr/lib/portage/bin:/root/bin
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/zsh

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

From @cpansprout

On Mon Sep 29 17​:18​:16 2014, mg@​fork.pl wrote​:

This is a bug report for perl from mg@​fork.pl,
generated with the help of perlbug 1.40 running under perl 5.20.0.

-----------------------------------------------------------------
We have perl based web application. It crashes with segv several times
a day.
So it's not easily repeatable (as the creash happens 1 in 100K http
requests served).
I have perl 5.20 build with symbols and can provide any gdb stuff
needed - but
I cannot provide you with coredump as it contains sensitive data
(that's production
environment).

Stacktrace looks like this​:

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x815d59660 "COUNT", len=5, hash=1160607206, flags=1024) at
hv.c​:2872
2872 if (HeHASH(entry) != hash) /* strings
can't be equal */
[New Thread 801c06400 (LWP 100627/perl)]
(gdb) bt
#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x815d59660 "COUNT", len=5, hash=1160607206, flags=1024) at
hv.c​:2872
#1 0x00000008008df1e6 in Perl_hv_common (my_perl=0x801c19e00,
hv=0x816814390, keysv=<value optimized out>, key=0x815d59660 "COUNT",
klen=5, flags=<value optimized out>,
action=<value optimized out>, val=0x8167ea0f0, hash=<value
optimized out>) at hv.c​:782
#2 0x00000008009146f9 in Perl_pp_hslice (my_perl=0x801c19e00) at
pp.c​:4770
#3 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00)
at run.c​:42
#4 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at
perl.c​:2456
#5 0x0000000000400eda in main ()

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x816727f30 "cdr", len=3, hash=2934883949, flags=258) at hv.c​:2872
2872 if (HeHASH(entry) != hash) /* strings
can't be equal */
[New Thread 801c06400 (LWP 102604/perl)]
(gdb) bt
#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x816727f30 "cdr", len=3, hash=2934883949, flags=258) at hv.c​:2872
#1 0x00000008008df1e6 in Perl_hv_common (my_perl=0x801c19e00,
hv=0x816afc9d8, keysv=<value optimized out>, key=0x816727f30 "cdr",
klen=3, flags=<value optimized out>,
action=<value optimized out>, val=0x816add810, hash=<value
optimized out>) at hv.c​:782
#2 0x00000008008e9bd5 in Perl_pp_helem (my_perl=0x801c19e00) at
pp_hot.c​:1769
#3 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00)
at run.c​:42
#4 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at
perl.c​:2456
#5 0x0000000000400eda in main ()

entry seems to be broken

(gdb) p *entry
$10 = {hent_next = 0x67007100726463, hent_hek = 0x594152002f, he_valu
= {hent_val = 0x480038312e30, hent_refcount = 79165779947056}}

but I don't know much about hash entry internals...

That’s not much to go by. Could you at least show us the line of Perl code on which it’s crashing, or maybe even the containing function? This should give you the file and line number​:

(gdb) p Perl_warn(my_perl, "")

Also, it may help you to reduce the test case if you get a Perl backtrace​:

(gdb) p Perl_eval_pv(my_perl,"use Carp; Carp​::cluck 'foo'",0)

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

From @demerphq

FC just posted the following advice. I think these kind of tricks should be
gathered up into a perldoc which we can use. I know I could have saved some
time if I knew these tricks. (I know other tricks which I could add.) If
people like the idea how about perlgdbtricks.pod or something like that?

From FC​:

This should give you the file and line number​:

(gdb) p Perl_warn(my_perl, "")

Also, it may help you to reduce the test case if you get a Perl backtrace​:

(gdb) p Perl_eval_pv(my_perl,"use Carp; Carp​::cluck 'foo'",0)

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

From PeterCMartini@GMail.com

On Oct 1, 2014, at 7​:27, demerphq <demerphq@​gmail.com> wrote​:

FC just posted the following advice. I think these kind of tricks should be gathered up into a perldoc which we can use. I know I could have saved some time if I knew these tricks. (I know other tricks which I could add.) If people like the idea how about perlgdbtricks.pod or something like that?

From FC​:

This should give you the file and line number​:

(gdb) p Perl_warn(my_perl, "")

Also, it may help you to reduce the test case if you get a Perl backtrace​:

(gdb) p Perl_eval_pv(my_perl,"use Carp; Carp​::cluck 'foo'",0)

+1

Those two are useful if you have a reproducible crash; I have a couple written down somewhere to inspect those kinds of variables from core dumps, which I'll happily share when I find them :-)

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

From @iabyn

On Tue, Sep 30, 2014 at 05​:12​:54PM -0700, Father Chrysostomos via RT wrote​:

Stacktrace looks like this​:

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x815d59660 "COUNT", len=5, hash=1160607206, flags=1024) at
hv.c​:2872
2872 if (HeHASH(entry) != hash) /* strings
[snip]
entry seems to be broken

(gdb) p *entry
$10 = {hent_next = 0x67007100726463, hent_hek = 0x594152002f, he_valu
= {hent_val = 0x480038312e30, hent_refcount = 79165779947056}}

but I don't know much about hash entry internals...

Perl shares hash keys between different hashes​: this is to make objects,
typically implemented using hashes, more lightweight. It does this by
having a special shared "master" hash, called PL_strtab, that contains a
shared copy of every key used by every hash. When a new key is added to a
hash, that key is first looked for in the shared hash; if not found, a new
key is created, and added to both the hash in question and the shared
hash.

The code is crashing while trying to add a new key ("COUNT") to the shared
hash. In some fashion the shared hash is getting corrupted, so that one of
its entries contains garbage (or, more likely, the pointer to the entry is
garbage).

Thus, its likely that an unrelated piece of code has earlier corrupted
it, and so is going to be hard to diagnose without code to reproduce it.

It may also be that the crash is intermittent based on hash randomisation.
If you run the program with the environment variable
PERL_HASH_SEED_DEBUG=1, then perl will print out the hash seed as it
starts up​:

  $ PERL_HASH_SEED_DEBUG=1 perl -le'print "hello"'
  HASH_FUNCTION = ONE_AT_A_TIME_HARD HASH_SEED = 0xf71b975f74369627 PERTURB_KEYS = 1 (RANDOM)
  hello

It's then possible to run re-perl using the same seed​:

  $ PERL_HASH_SEED=0xf71b975f74369627 perl ....

--
No man treats a motor car as foolishly as he treats another human being.
When the car will not go, he does not attribute its annoying behaviour to
sin, he does not say, You are a wicked motorcar, and I shall not give you
any more petrol until you go. He attempts to find out what is wrong and
set it right.
  -- Bertrand Russell,
  Has Religion Made Useful Contributions to Civilization?

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

From perl5-porters@perl.org

Peter Martini wrote​:

I think these kind of tricks should be
gathered up into a perldoc which we can use. I know I could have saved some
time if I knew these tricks. (I know other tricks which I could add.) If
people like the idea how about perlgdbtricks.pod or something like that?

Such tricks are quite dependent on Perl internals and build options.
And some of them only work sometimes. The Carp​::cluck example I came
up with on the spot. While such a document would be useful, will it
be maintained, or will it end up out of date?

(If you are volunteering to maintain it, I'd say go ahead.)

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2014

From @marcin-gryszkalis

I looks like reply-to-RT-by-email doesn't work or get some hiccups - so I'm pasting here answer I already sent - sorry if it finally gets duplicated.

On 2014-10-01 02​:12, Father Chrysostomos via RT wrote​:

That’s not much to go by. Could you at least show us the line of Perl code on which it’s crashing, or maybe even the containing function? This should give you the file and line number​:
(gdb) p Perl_warn(my_perl, "")
Also, it may help you to reduce the test case if you get a Perl backtrace​:
(gdb) p Perl_eval_pv(my_perl,"use Carp; Carp​::cluck 'foo'",0)

Sure, but I don't have live gdb session - only coredump, ie.

(gdb) p Perl_warn(my_perl, "")
You can't do that without a process to debug.

As I mentioned It's web application with many perl processes serving
requests. They are even created and destroyed to serve required load -
so I'm not sure how I could get live gdb session.

Other problem is that the crashes happen randomly (eg. I didn't have
single one during last 24hrs)

On 2014-10-01 14​:04, Dave Mitchell via RT wrote​:

Thus, its likely that an unrelated piece of code has earlier corrupted
it, and so is going to be hard to diagnose without code to reproduce it.

I checked older cores and they seem to confirm what you wrote. perl
crashes in the same place but the calls come from different places​:

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x8177c9c30 "(eval 11979)", len=12, hash=939841298, flags=0) at
hv.c​:2872
#1 0x00000008008e2445 in Perl_share_hek (my_perl=0x801c19e00,
str=<value optimized out>, len=<value optimized out>, hash=<value
optimized out>) at hv.c​:2846
#2 0x0000000800872854 in Perl_newGP (my_perl=0x801c19e00, gv=<value
optimized out>) at gv.c​:208
#3 0x00000008008725a3 in Perl_gv_init_pvn (my_perl=0x801c19e00,
gv=0x8165a63d8, stash=0x801c3e2e8, name=0x7fffffffcc90 "_<(eval 11979)",
len=14, flags=0) at gv.c​:379
#4 0x00000008008722ee in Perl_gv_fetchfile_flags (my_perl=0x801c19e00,
name=0x8177c9c30 "(eval 11979)", namelen=12, flags=<value optimized
out>) at gv.c​:127
#5 0x000000080087b3f2 in S_incline (my_perl=0x801c19e00, s=<value
optimized out>) at toke.c​:1777
#6 0x000000080087e5db in Perl_yylex (my_perl=0x801c19e00) at toke.c​:5610
#7 0x000000080089cc1d in Perl_yyparse (my_perl=0x801c19e00,
gramtype=<value optimized out>) at perly.c​:343
#8 0x0000000800927928 in S_doeval (my_perl=0x801c19e00, gimme=2,
outside=<value optimized out>, seq=<value optimized out>, hh=0x0) at
pp_ctl.c​:3486
#9 0x00000008009285a4 in Perl_pp_entereval (my_perl=0x801c19e00) at
pp_ctl.c​:4275
#10 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at
run.c​:42
#11 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at perl.c​:2456
#12 0x0000000000400eda in main ()

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x809c90660 "cp874", len=5, hash=4131272831, flags=0) at hv.c​:2872
#1 0x00000008008df1e6 in Perl_hv_common (my_perl=0x801c19e00,
hv=0x809cdf3a8, keysv=<value optimized out>, key=0x809c90660 "cp874",
klen=5, flags=<value optimized out>, action=<value optimized out>,
  val=0x8169ccc60, hash=<value optimized out>) at hv.c​:782
#2 0x00000008008e9bd5 in Perl_pp_helem (my_perl=0x801c19e00) at
pp_hot.c​:1769
#3 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at
run.c​:42
#4 0x000000080086f610 in Perl_call_sv (my_perl=0x801c19e00, sv=<value
optimized out>, flags=6) at perl.c​:2756
#5 0x0000000817043ee6 in Encode_XSEncoding () from
/usr/local/lib/perl5/site_perl/5.20/mach/auto/Encode/Byte/Byte.so
#6 0x0000000817043ad5 in boot_Encode__Byte () from
/usr/local/lib/perl5/site_perl/5.20/mach/auto/Encode/Byte/Byte.so
#7 0x00000008008ec3c7 in Perl_pp_entersub (my_perl=0x801c19e00) at
pp_hot.c​:2795
#8 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at
run.c​:42
#9 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at perl.c​:2456
#10 0x0000000000400eda in main ()

It may also be that the crash is intermittent based on hash randomisation.
If you run the program with the environment variable
PERL_HASH_SEED_DEBUG=1, then perl will print out the hash seed as it
starts up​:

$ PERL\_HASH\_SEED\_DEBUG=1 perl \-le'print "hello"'
HASH\_FUNCTION = ONE\_AT\_A\_TIME\_HARD HASH\_SEED = 0xf71b975f74369627

PERTURB_KEYS = 1 (RANDOM)

hello

It's then possible to run re-perl using the same seed​:

$ PERL\_HASH\_SEED=0xf71b975f74369627 perl \.\.\.\.

I guess I could do this but not sure what could be achieved with this...

--
Marcin Gryszkalis
PGP​: 0xA5DBEEC7 http​://fork.pl/gpg.txt
jabber​: mg@​fork.pl

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @cpansprout

On Wed Oct 01 13​:03​:31 2014, mg@​fork.pl wrote​:

I looks like reply-to-RT-by-email doesn't work or get some hiccups -
so I'm pasting here answer I already sent - sorry if it finally gets
duplicated.

On 2014-10-01 02​:12, Father Chrysostomos via RT wrote​:

That’s not much to go by. Could you at least show us the line of
Perl code on which it’s crashing, or maybe even the containing
function? This should give you the file and line number​:
(gdb) p Perl_warn(my_perl, "")
Also, it may help you to reduce the test case if you get a Perl
backtrace​:
(gdb) p Perl_eval_pv(my_perl,"use Carp; Carp​::cluck 'foo'",0)

Sure, but I don't have live gdb session - only coredump, ie.

(gdb) p Perl_warn(my_perl, "")
You can't do that without a process to debug.

As I mentioned It's web application with many perl processes serving
requests. They are even created and destroyed to serve required load -
so I'm not sure how I could get live gdb session.

I’ve never debugged a core file before. Can you access the entire process’s memory from the moment it crashed? In that case you can get a Perl line number with​:

(gdb) p *my_perl->Icurcop

And look at cop_line and cop_file.

Other problem is that the crashes happen randomly (eg. I didn't have
single one during last 24hrs)

On 2014-10-01 14​:04, Dave Mitchell via RT wrote​:

It may also be that the crash is intermittent based on hash
randomisation.
If you run the program with the environment variable
PERL_HASH_SEED_DEBUG=1, then perl will print out the hash seed as it
starts up​:

$ PERL_HASH_SEED_DEBUG=1 perl -le'print "hello"'
HASH_FUNCTION = ONE_AT_A_TIME_HARD HASH_SEED = 0xf71b975f74369627
PERTURB_KEYS = 1 (RANDOM)
hello

It's then possible to run re-perl using the same seed​:

$ PERL_HASH_SEED=0xf71b975f74369627 perl ....

I guess I could do this but not sure what could be achieved with
this...

If you could run a copy of the script repeatedly outside the context of a web server and capture the hash seed when it crashes, that might help.

That said, it should be possible to get the hash seed from a core file, but I don’t know how offhand.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @cpansprout

On Wed Oct 01 21​:53​:28 2014, sprout wrote​:

That said, it should be possible to get the hash seed from a core
file, but I don’t know how offhand.

I’ve just had a look​:

(gdb) p PL_hash_seed
$2 = "t?8?\003??)"

Getting that into a hexadecimal string is a little more complicated.... If you can capture the output and feed it to unpack("H*",$output), you can then use the hexadecimal string like this​:

$ PERL_HASH_SEED=42baff1edc0ffee5 your_script.pl

and try to reproduce the crash.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @tonycoz

On Mon Sep 29 17​:18​:16 2014, mg@​fork.pl wrote​:

We have perl based web application. It crashes with segv several times
a day.
So it's not easily repeatable (as the creash happens 1 in 100K http
requests served).
I have perl 5.20 build with symbols and can provide any gdb stuff
needed - but
I cannot provide you with coredump as it contains sensitive data
(that's production
environment).

This may be same bug as in https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122873

Could you try reinstalling Encode with the patch I've attached here? (This is the same patch I included in https://rt.cpan.org/Ticket/Display.html?id=99264 )

Tony

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @tonycoz

0001-SPAGAIN-after-call_pv-which-can-reallocate-the-perl-.patch
From 5f0737ab20b7bd46cddaccdfff5a6f78b73f19e1 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Thu, 2 Oct 2014 11:05:03 +1000
Subject: [PATCH] SPAGAIN after call_pv(), which can reallocate the perl stack

---
 Encode.xs |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/Encode.xs b/Encode.xs
index 5ee4539..5e20c3d 100644
--- a/Encode.xs
+++ b/Encode.xs
@@ -686,6 +686,7 @@ CODE:
     /* require_pv(PERLIO_FILENAME); */
 
     eval_pv("require PerlIO::encoding", 0);
+    SPAGAIN;
 
     if (SvTRUE(get_sv("@", 0))) {
     ST(0) = &PL_sv_no;
@@ -703,6 +704,7 @@ CODE:
     encode_t *enc = INT2PTR(encode_t *, SvIV(SvRV(obj)));
     SV *retval;
     eval_pv("require Encode::MIME::Name", 0);
+    SPAGAIN;
 
     if (SvTRUE(get_sv("@", 0))) {
 	ST(0) = &PL_sv_undef;
-- 
1.7.10.4

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @timbunce

On Wed, Oct 01, 2014 at 07​:44​:59PM -0000, Father Chrysostomos wrote​:

Peter Martini wrote​:

I think these kind of tricks should be
gathered up into a perldoc which we can use. I know I could have saved some
time if I knew these tricks. (I know other tricks which I could add.) If
people like the idea how about perlgdbtricks.pod or something like that?

Such tricks are quite dependent on Perl internals and build options.
And some of them only work sometimes. The Carp​::cluck example I came
up with on the spot. While such a document would be useful, will it
be maintained, or will it end up out of date?

Even when out of date, those kinds of docs are still helpful.
The kinds of people reading the doc are likely to be able to use the
tips as raw material to improvise whever they might need in the
circumstances.

(If you are volunteering to maintain it, I'd say go ahead.)

And I'd be grateful.

Tim.

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @demerphq

On 2 October 2014 11​:44, Tim Bunce <Tim.Bunce@​pobox.com> wrote​:

On Wed, Oct 01, 2014 at 07​:44​:59PM -0000, Father Chrysostomos wrote​:

Peter Martini wrote​:

I think these kind of tricks should be
gathered up into a perldoc which we can use. I know I could have saved
some
time if I knew these tricks. (I know other tricks which I could add.)
If
people like the idea how about perlgdbtricks.pod or something like
that?

Such tricks are quite dependent on Perl internals and build options.
And some of them only work sometimes. The Carp​::cluck example I came
up with on the spot. While such a document would be useful, will it
be maintained, or will it end up out of date?

Even when out of date, those kinds of docs are still helpful.
The kinds of people reading the doc are likely to be able to use the
tips as raw material to improvise whever they might need in the
circumstances.

(If you are volunteering to maintain it, I'd say go ahead.)

And I'd be grateful.

Ok, Then I will try to find time to do this.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @timbunce

On Wed, Oct 01, 2014 at 07​:43​:30AM -0400, Peter Martini wrote​:

On Oct 1, 2014, at 7​:27, demerphq <[1]demerphq@​gmail.com> wrote​:

 FC just posted the following advice\. I think these kind of tricks should be gathered up into a perldoc
 which we can use\. I know I could have saved some time if I knew these tricks\. \(I know other tricks
 which I could add\.\) If people like the idea how about perlgdbtricks\.pod or something like that?
 From FC&#8203;:
 This should give you the file and line number&#8203;:

 \(gdb\) p Perl\_warn\(my\_perl\, ""\)

 Also\, it may help you to reduce the test case if you get a Perl backtrace&#8203;:

 \(gdb\) p Perl\_eval\_pv\(my\_perl\,"use Carp; Carp&#8203;::cluck 'foo'"\,0\)

+1
Those two are useful if you have a reproducible crash; I have a couple written down somewhere to inspect
those kinds of variables from core dumps, which I'll happily share when I find them :-)

This reminded me of https://metacpan.org/source/GOZER/mod_perl-1.31/.gdbinit

I'm sure that's out of date but it shows a nice way to package the tips
into gdb functions.

Then I wondered if any other distros contained .gdbinit files, and
remembered that metacpan could tell me​:

curl -XPOST 'api.metacpan.org/v0/file' -d '{"query"​:{"match_all"​:{}},"filter"​:{"and"​:[{"term"​:{"path"​:".gdbinit"}},{"term"​:{"status"​:"latest"}}]},"fields"​:["release"],"size"​:200}' | grep release

  "release" : "Devel-SizeMe-0.19"
  "release" : "B-C-1.52"
  "release" : "Devel-NYTProf-5.06"
  "release" : "Marpa-R2-2.094000"

The Devel-SizeMe and Devel-NYTProf files are mostly random cargo-cult stuff.
Marpa-R2's is minimal. B-C's is more interesting​:
https://metacpan.org/source/RURBAN/B-C-1.52/.gdbinit

Tim.

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @marcin-gryszkalis

On 2014-10-01 02​:12, Father Chrysostomos via RT wrote​:

That’s not much to go by. Could you at least show us the line of Perl code on which it’s crashing, or maybe even the containing function? This should give you the file and line number​:
(gdb) p Perl_warn(my_perl, "")
Also, it may help you to reduce the test case if you get a Perl backtrace​:
(gdb) p Perl_eval_pv(my_perl,"use Carp; Carp​::cluck 'foo'",0)

Sure, but I don't have live gdb session - only coredump, ie.

(gdb) p Perl_warn(my_perl, "")
You can't do that without a process to debug.

As I mentioned It's web application with many perl processes serving
requests. They are even created and destroyed to serve required load -
so I'm not sure how I could get live gdb session.

Other problem is that the crashes happen randomly (eg. I didn't have
single one during last 24hrs)

--
Marcin Gryszkalis
PGP​: 0xA5DBEEC7 http​://fork.pl/gpg.txt
jabber​: mg@​fork.pl

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @marcin-gryszkalis

On 2014-10-01 14​:04, Dave Mitchell via RT wrote​:

Thus, its likely that an unrelated piece of code has earlier corrupted
it, and so is going to be hard to diagnose without code to reproduce it.

I checked older cores and they seem to confirm what you wrote. perl
crashes in the same place but the calls come from different places​:

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x8177c9c30 "(eval 11979)", len=12, hash=939841298, flags=0) at
hv.c​:2872
#1 0x00000008008e2445 in Perl_share_hek (my_perl=0x801c19e00,
str=<value optimized out>, len=<value optimized out>, hash=<value
optimized out>) at hv.c​:2846
#2 0x0000000800872854 in Perl_newGP (my_perl=0x801c19e00, gv=<value
optimized out>) at gv.c​:208
#3 0x00000008008725a3 in Perl_gv_init_pvn (my_perl=0x801c19e00,
gv=0x8165a63d8, stash=0x801c3e2e8, name=0x7fffffffcc90 "_<(eval 11979)",
len=14, flags=0) at gv.c​:379
#4 0x00000008008722ee in Perl_gv_fetchfile_flags (my_perl=0x801c19e00,
name=0x8177c9c30 "(eval 11979)", namelen=12, flags=<value optimized
out>) at gv.c​:127
#5 0x000000080087b3f2 in S_incline (my_perl=0x801c19e00, s=<value
optimized out>) at toke.c​:1777
#6 0x000000080087e5db in Perl_yylex (my_perl=0x801c19e00) at toke.c​:5610
#7 0x000000080089cc1d in Perl_yyparse (my_perl=0x801c19e00,
gramtype=<value optimized out>) at perly.c​:343
#8 0x0000000800927928 in S_doeval (my_perl=0x801c19e00, gimme=2,
outside=<value optimized out>, seq=<value optimized out>, hh=0x0) at
pp_ctl.c​:3486
#9 0x00000008009285a4 in Perl_pp_entereval (my_perl=0x801c19e00) at
pp_ctl.c​:4275
#10 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at
run.c​:42
#11 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at perl.c​:2456
#12 0x0000000000400eda in main ()

#0 0x00000008008de137 in S_share_hek_flags (my_perl=0x801c19e00,
str=0x809c90660 "cp874", len=5, hash=4131272831, flags=0) at hv.c​:2872
#1 0x00000008008df1e6 in Perl_hv_common (my_perl=0x801c19e00,
hv=0x809cdf3a8, keysv=<value optimized out>, key=0x809c90660 "cp874",
klen=5, flags=<value optimized out>, action=<value optimized out>,
  val=0x8169ccc60, hash=<value optimized out>) at hv.c​:782
#2 0x00000008008e9bd5 in Perl_pp_helem (my_perl=0x801c19e00) at
pp_hot.c​:1769
#3 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at
run.c​:42
#4 0x000000080086f610 in Perl_call_sv (my_perl=0x801c19e00, sv=<value
optimized out>, flags=6) at perl.c​:2756
#5 0x0000000817043ee6 in Encode_XSEncoding () from
/usr/local/lib/perl5/site_perl/5.20/mach/auto/Encode/Byte/Byte.so
#6 0x0000000817043ad5 in boot_Encode__Byte () from
/usr/local/lib/perl5/site_perl/5.20/mach/auto/Encode/Byte/Byte.so
#7 0x00000008008ec3c7 in Perl_pp_entersub (my_perl=0x801c19e00) at
pp_hot.c​:2795
#8 0x00000008008e5456 in Perl_runops_standard (my_perl=0x801c19e00) at
run.c​:42
#9 0x000000080086ef49 in perl_run (my_perl=0x801c19e00) at perl.c​:2456
#10 0x0000000000400eda in main ()

It may also be that the crash is intermittent based on hash randomisation.
If you run the program with the environment variable
PERL_HASH_SEED_DEBUG=1, then perl will print out the hash seed as it
starts up​:

$ PERL\_HASH\_SEED\_DEBUG=1 perl \-le'print "hello"'
HASH\_FUNCTION = ONE\_AT\_A\_TIME\_HARD HASH\_SEED = 0xf71b975f74369627

PERTURB_KEYS = 1 (RANDOM)

hello

It's then possible to run re-perl using the same seed​:

$ PERL\_HASH\_SEED=0xf71b975f74369627 perl \.\.\.\.

I guess I could do this but not sure what could be achieved with this...

--
Marcin Gryszkalis
PGP​: 0xA5DBEEC7 http​://fork.pl/gpg.txt
jabber​: mg@​fork.pl

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @marcin-gryszkalis

On Wed Oct 01 22​:07​:39 2014, tonyc wrote​:

Could you try reinstalling Encode with the patch I've attached here?
(This is the same patch I included in
https://rt.cpan.org/Ticket/Display.html?id=99264 )

I did, no crashes for last few hours but we'll have to wait a bit longer...

@p5pRT
Copy link
Author

p5pRT commented Oct 2, 2014

From @marcin-gryszkalis

On Wed Oct 01 21​:53​:28 2014, sprout wrote​:

On Wed Oct 01 13​:03​:31 2014, mg@​fork.pl wrote​:
I’ve never debugged a core file before. Can you access the entire
process’s memory from the moment it crashed? In that case you can get
a Perl line number with​:

(gdb) p *my_perl->Icurcop

And look at cop_line and cop_file.

I did for few coredumps I have and it's pretty random​:

$1 = 0x80cc50680 "/usr/local/lib/perl5/site_perl/5.20/MIME/Types.pm"
$2 = 59

$1 = 0x8177c9c30 "(eval 11979)"
$2 = 9

$1 = 0x80cc4a700 "/usr/local/lib/perl5/site_perl/5.20/MIME/Types.pm"
$2 = 59

$1 = 0x80689fd80 "/usr/local/lib/perl5/site_perl/5.20/mach/Data/Dumper.pm"
$2 = 509

$1 = 0x815d5bd40 "/usr/local/lib/perl5/site_perl/5.20/mach/Template/Iterator.pm"
$2 = 100

$1 = 0x8047caf40 "/usr/local/lib/perl5/site_perl/5.20/CGI/Simple/Util.pm"
$2 = 26

$1 = 0x816de1310 "(eval 12310)"
$2 = 9

$1 = 0x809cdc2c0 "/usr/local/lib/perl5/site_perl/5.20/mach/Encode.pm"
$2 = 90

@p5pRT
Copy link
Author

p5pRT commented Oct 6, 2014

From @marcin-gryszkalis

On Wed Oct 01 22​:07​:39 2014, tonyc wrote​:

This may be same bug as in
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122873

Could you try reinstalling Encode with the patch I've attached here?
(This is the same patch I included in
https://rt.cpan.org/Ticket/Display.html?id=99264 )

Hi, I tried your patch on 5.20 (still segv) and then I downgraded to 5.18.4 and tested with and without the patch - in both cases I get segv :(

I guess I didn't mention that application worked without problems on 5.14

@p5pRT
Copy link
Author

p5pRT commented Dec 1, 2014

From @marcin-gryszkalis

After upgrading to FreeBSD 10.1 with new clang compiler
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
crashes are gone, forever I hope :)

Note that other confirmed similar problems
http​://www.perlmonks.org/?node_id=1091458 - it mentions RT#122199 but I have no rights to see it.

@marcin-gryszkalis
Copy link

As mentioned in #13962 (RT#122199) bug was specific to freebsd 10.0 (and clang included) - now it's long gone.
The bug can be closed.

@jkeenan jkeenan closed this as completed Jan 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants