Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5.22.0 documentation request: How to debug "handshake key" errors #14714

Open
p5pRT opened this issue May 22, 2015 · 7 comments
Open

5.22.0 documentation request: How to debug "handshake key" errors #14714

p5pRT opened this issue May 22, 2015 · 7 comments

Comments

@p5pRT
Copy link

p5pRT commented May 22, 2015

Migrated from rt.perl.org#125236 (status was 'open')

Searchable as RT125236$

@p5pRT
Copy link
Author

p5pRT commented May 22, 2015

From @andk

From smoker report
http​://www.cpantesters.org/cpan/report/3133bbca-ba7e-11e4-92d3-0eaa8971dd2f :

  pegex_parser_t_c4f8477.c​: loadable library and perl binaries are mismatched (got handshake key 0xd800000, needed 0xd700000)

In total my smokers produced 13 fail reports with this error between
v5.21.9-18-g0f9d53b and v5.22.0-RC1 on eight different distros. It's
very likely that there were my own mistakes involved but at least I
could reproduce one case and would like to find out which mistake it
was.

It is not a showstopper but a documentation request for 5.22.1.

The following description in perldiag is not deemed helpful enough,
especially in the case when reinstalling the XS module does not fix the
error​:

  %s​: loadable library and perl binaries are mismatched (got handshake key
  %p, needed %p)
  (P) A dynamic loading library ".so" or ".dll" was being loaded into
  the process that was built against a different build of perl than the
  said library was compiled against. Reinstalling the XS module will
  likely fix this error.

Can some guidance be added how to proceed if it happens?
--
andreas

P.S. the above perldiag comes from

Summary of my perl5 (revision 5 version 22 subversion 0) configuration​:
  Commit id​: 6378109
  Platform​:
  osname=linux, osvers=4.0.0-1-amd64, archname=x86_64-linux
  uname='linux k83 4.0.0-1-amd64 #1 smp debian 4.0.2-1 (2015-05-11) x86_64 gnulinux '
  config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/perl/v5.22.0-RC2/165a -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Uuseithreads -Uuselongdouble -DDEBUGGING=-g'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
  optimize='-O2 -g',
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='4.9.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Locally applied patches​:
  RC2
  Built under linux
  Compiled at May 22 2015 01​:09​:33

@p5pRT
Copy link
Author

p5pRT commented May 22, 2015

From @bulk88

On Fri May 22 08​:45​:38 2015, andreas.koenig.7os6VVqR@​franz.ak.mind.de wrote​:

From smoker report
http​://www.cpantesters.org/cpan/report/3133bbca-ba7e-11e4-92d3-
0eaa8971dd2f :

pegex_parser_t_c4f8477.c​: loadable library and perl binaries are
mismatched (got handshake key 0xd800000, needed 0xd700000)

In total my smokers produced 13 fail reports with this error between
v5.21.9-18-g0f9d53b and v5.22.0-RC1 on eight different distros. It's
very likely that there were my own mistakes involved but at least I
could reproduce one case and would like to find out which mistake it
was.

It is not a showstopper but a documentation request for 5.22.1.

The following description in perldiag is not deemed helpful enough,
especially in the case when reinstalling the XS module does not fix
the
error​:

%s​: loadable library and perl binaries are mismatched (got handshake
key
%p, needed %p)
(P) A dynamic loading library ".so" or ".dll" was being loaded
into
the process that was built against a different build of perl than
the
said library was compiled against. Reinstalling the XS module will
likely fix this error.

Can some guidance be added how to proceed if it happens?

The guide
|
|
\|/
v

The XS module reports the interp struct (or a fake unallocated one if it is no threads perl) is 0xd80 bytes long, the perl core says the interp struct is 0xd70 bytes long. The low 16 bits on the handshake are a bitfield of misc things, there is no problem here since both sides are 0x0000.

The -D'es on the cmd line to the CC dont match the -D'es that the perl core was compiled with, or somehow EUMM/MB is using the wrong CORE dir and the wrong config.h file, or your @​INC patch or arch dirs are messed up. A wild guess is you have -DDEBUGGING on the cmd line to the CC in your module. DEBUGGING can only be set at perl interp build time, not randomly, per module, afterwards. If you want C asserts for a single XS module, eghh, I forgot how to turn on C asserts without turning on DEBUGGING. Perl, assert.h, and -DNDEBUG have a complicated sketchy interaction.

The current code added in 5.21.7 in perl.h is

#ifdef I_ASSERT
# if !defined(DEBUGGING) && !defined(NDEBUG)
# define NDEBUG 1
# endif
# include <assert.h>
#endif

http​://perl5.git.perl.org/perl.git/commitdiff/6d59e610a3f269be73ffea56a90d1cd7dc8bf2fd

Before 5.21.7 is was

#if defined(DEBUGGING) && defined(I_ASSERT)
# include <assert.h>
#endif

Worst case scenario for debugging interp size differences is producing TWO .i files. One from a working XS module, one from a failing (wrong interp size) module. Send both .i files through gnu indent to normalize whitespace. Then copy paste the interp structs alone into separate files, then run diff on the 2 files. Now you know what members are different. If the interp structs are text identical, then probably the PL_sys_intern and struct interp_intern are different. If that doesn't reveal the different, produce the 2 .i files again, do a diff of the full size .i files and look for the difference. Stripping all ^#.+$ lines from both .i files and gnu indent may or may not help reduce the noise in the diff.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented May 22, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented May 22, 2015

From @bulk88

On Fri May 22 10​:20​:25 2015, bulk88 wrote​:

On Fri May 22 08​:45​:38 2015, andreas.koenig.7os6VVqR@​franz.ak.mind.de
wrote​:

From smoker report
http​://www.cpantesters.org/cpan/report/3133bbca-ba7e-11e4-92d3-
0eaa8971dd2f :

pegex_parser_t_c4f8477.c​: loadable library and perl binaries are
mismatched (got handshake key 0xd800000, needed 0xd700000)

In total my smokers produced 13 fail reports with this error between
v5.21.9-18-g0f9d53b and v5.22.0-RC1 on eight different distros. It's
very likely that there were my own mistakes involved but at least I
could reproduce one case and would like to find out which mistake it
was.

It is not a showstopper but a documentation request for 5.22.1.

The following description in perldiag is not deemed helpful enough,
especially in the case when reinstalling the XS module does not fix
the
error​:

%s​: loadable library and perl binaries are mismatched (got handshake
key
%p, needed %p)
(P) A dynamic loading library ".so" or ".dll" was being loaded
into
the process that was built against a different build of perl than
the
said library was compiled against. Reinstalling the XS module
will
likely fix this error.

Can some guidance be added how to proceed if it happens?

About that warning message. I think "reinstall the XS module" is the limit of what advice we can give, other than contact the author. You can't seriously tell a [pure] Perl users to debug XS/C/.i files/headers/drink boiling macro soup/rebuild with symbols then use gdb stacktraces and C variable watching.

The handshake feature stops people from copy-paste installing XS modules across different perl interp versions/builds. It also stops segvs from upgrading blead perl to blead, but not reinstalling any /site XS modules (im lazy). If the blead perls are ABI compatible (enough, I guess, handshake api wont catch changing the integers behind perl SV flags for example, but will catch threads, DEBUGGING, DEBUG_LEAKING_SCALARS, 64 bit IVs on 32 bit), you dont need to recompile, if they are ABI not compatible, you get the handshake message. Also some .t files do "push @​INC, 'lib', '../lib','../../lib' and pick up another Perl's /lib on my system, and previously, trying to do the god damn croak() to tell the user that module was compiled with 5.12, but I am running on 5.19 (or any other version other than 5.12), would SEGV inside 5.12 interp, when passed a my_perl ptr from 5.19. None of that is possible now (assuming the XS module was compiled against a handshake enabled perl version, hence the call stack below will still crash to this day) See the below call stack for hilarity.

  perl512.dll!Perl_safesysmalloc(unsigned int size=12) Line 94 + 0xd bytes C
  perl512.dll!Perl_sv_grow(interpreter * my_perl=0x01c45c34, sv * const sv=0x01ee8600, unsigned int newlen=1) Line 1562 + 0x6 bytes C
  perl512.dll!Perl_sv_setpvn(interpreter * my_perl=0x01c45c34, sv * const sv=0x01ee8600, const char * const ptr=0x02031748, const unsigned int len=0) Line 4343 + 0x14 bytes C
  perl512.dll!Perl_newSVpvn_flags(interpreter * my_perl=0x01c45c34, const char * const s=0x02031748, const unsigned int len=0, const unsigned long flags=524288) Line 7711 C
  perl512.dll!S_mess_alloc(interpreter * my_perl=0x01c45c34) Line 1055 + 0x12 bytes C
  perl512.dll!Perl_form(interpreter * my_perl=0x01c45c34, const char * pat=0x01f75f18, ...) Line 1113 + 0xf bytes C
  Win32.dll!boot_Win32(interpreter * my_perl=0x01ee046c, cv * cv=0x01e9bff4) Line 2495 + 0x7f bytes C
  perl519.dll!Perl_pp_entersub(interpreter * my_perl=0x01c466e8) Line 2763 C
  perl519.dll!Perl_runops_standard(interpreter * my_perl=0x01c45c34) Line 42 + 0x4 bytes C
  perl519.dll!Perl_call_sv(interpreter * my_perl=0x01c45c34, sv * sv=0x01e79b8c, volatile long flags=13) Line 2747 + 0xc bytes C
  perl519.dll!Perl_call_list(interpreter * my_perl=0x01c45c34, long oldscope=8, av * paramList=0x01c6ec34) Line 4843 C
  perl519.dll!S_process_special_blocks(interpreter * my_perl=0x00000000, long floor=0, const char * const fullname=0x01c795e0, gv * const gv=0x01dfc564, cv * const cv=0x01e79b8c) Line 7932 C
  perl519.dll!Perl_newATTRSUB_flags(interpreter * my_perl=0x00000000, long floor=43, op * o=0x01e6a9d0, op * proto=0x00000000, void * attrs=0x2802c589, op * block=0x01e6a9f0, unsigned long flags=0) Line 7894 + 0x16 bytes C
  perl519.dll!Perl_newATTRSUB(interpreter * my_perl=0x01c45c34, long floor=43, op * o=0x01e6a9d0, op * proto=0x00000000, op * attrs=0x00000000, op * block=0x01e6a9f0) Line 7569 + 0x19 bytes C
  perl519.dll!Perl_utilize(interpreter * my_perl=0x00000000, int aver=1, long floor=43, op * version=0x01e83744, op * idop=0x01e92f54, op * arg=0x01e6f864) Line 5317 + 0x7e bytes C
  perl519.dll!Perl_vload_module(interpreter * my_perl=0x01e92f54, unsigned long flags=2, sv * name=0x01caf56c, sv * ver=0x01e83744, char * * args=0x002df984) Line 5463 + 0x23 bytes C
  perl519.dll!Perl_load_module(interpreter * my_perl=0x01c45c34, unsigned long flags=2, sv * name=0x01caf56c, sv * ver=0x01e70154, ...) Line 5405 + 0x16 bytes C
  perl519.dll!w32_CORE_all(interpreter * my_perl=0x01c45c34, cv * cv=0x01c4bb84) Line 33 + 0x3c bytes C
  perl519.dll!Perl_pp_entersub(interpreter * my_perl=0x01c4bb74) Line 2763 C
  perl519.dll!Perl_runops_standard(interpreter * my_perl=0x01c45c34) Line 42 + 0x4 bytes C
  perl519.dll!Perl_call_sv(interpreter * my_perl=0x01c45c34, sv * sv=0x01e099b4, volatile long flags=13) Line 2747 + 0xc bytes C
  perl519.dll!Perl_call_list(interpreter * my_perl=0x01c45c34, long oldscope=1, av * paramList=0x01d39f8c) Line 4843 C
  perl519.dll!perl_run(interpreter * my_perl=0x01c45c34) Line 2359 + 0x12 bytes C
  perl519.dll!RunPerl(int argc=3, char * * argv=0x01c45be0, char * * env=0x01c440e0) Line 270 + 0x6 bytes C++
  perl.exe!__tmainCRTStartup() Line 582 + 0x17 bytes C
  kernel32.dll!_BaseProcessStart@​4() + 0x28 bytes

Finally, handshake stops with a user message, a subtle case, lets say 64 bit IVs on 32, where the BOOT XSUB registers all the XSUBs correctly, and basic XSUBs work, since PL_stack_sp and PL_stack_base and PL_markstack are at the start of the interp struct and their positions never change, but if you use MY_CXT, where PL_my_cxt_list is at the bottom of the list/struct, chances are very high you will SEGV since the offsets are different by the end of the interp struct. Then to a pure perl user, or a novice XS programing who set CCFLAGS to an empty string or set to his own randomly created constant string, some XSUBs work, some crash, "IDK it crashes and IDK how to use C debuggers".

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented May 22, 2015

From @bulk88

catpic.jpg

@p5pRT
Copy link
Author

p5pRT commented May 23, 2015

From @andk

On Fri, 22 May 2015 10​:20​:26 -0700, "bulk88 via RT" <perlbug-followup@​perl.org> said​:

Can some guidance be added how to proceed if it happens?

  > The guide

Thanks, this helped a lot in finding the culprit.

  > The XS module reports the interp struct (or a fake unallocated one
  > if it is no threads perl) is 0xd80 bytes long, the perl core says
  > the interp struct is 0xd70 bytes long. The low 16 bits on the
  > handshake are a bitfield of misc things, there is no problem here
  > since both sides are 0x0000.

  > The -D'es on the cmd line to the CC dont match the -D'es that the
  > perl core was compiled with, or somehow EUMM/MB is using the wrong
  > CORE dir and the wrong config.h file, or your @​INC patch or arch
  > dirs are messed up.

This sentence alone contained all I needed (as I realized a bit later).
The differences of the -D'es were glaring. The fastest workaround was to
run

perl Makefile.PL DEFINE='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -DDOUBLE_SLASHES_SPECIAL=0'

MakeMaker added those in one project (Cwd) but not in the other
(DestructAssign). The reason was that CCFLAGS was set in the
Makefile.PL. But with the above incantation I had all tests pass. I'll
rt-cpan this to the author.

  > [...] Worst case scenario for debugging interp size differences is
  > producing TWO .i files. One from a working XS module, one from a
  > failing (wrong interp size) module. Send both .i files through gnu
  > indent to normalize whitespace. Then copy paste the interp structs

I assume you mean PerlHandShakeInterpreter struct.

  > alone into separate files, then run diff on the 2 files. Now you
  > know what members are different. If the interp structs are text
  > identical,

They were identical.

  > then probably the PL_sys_intern and struct interp_intern are
  > different.

They did not exist.

  > If that doesn't reveal the different, produce the 2 .i files again,
  > do a diff of the full size .i files and look for the difference.
  > Stripping all ^#.+$ lines from both .i files and gnu indent may or
  > may not help reduce the noise in the diff.

The diff was in things like

+typedef __ino_t ino_t;
-typedef __ino64_t ino_t;

and that was where I decided I had to go back to your sentence about the
-D'es.

Thanks again,
--
andreas

@p5pRT
Copy link
Author

p5pRT commented Jun 19, 2015

From @bulk88

On Sat May 23 02​:48​:43 2015, andreas.koenig.7os6VVqR@​franz.ak.mind.de wrote​:

MakeMaker added those in one project (Cwd) but not in the other
(DestructAssign). The reason was that CCFLAGS was set in the
Makefile.PL. But with the above incantation I had all tests pass. I'll
rt-cpan this to the author.

Unconditional assignment, instead of catting onto CCFLAGS, in a Makefile.PL will strip most or all of the -Ds cause breakage. Remember, adding your own -Ds to CCFLAGS or the cmd line to the CC is fine, but never remove Perl's -Ds or the -Ds that were there by default (unless you were the author of that define in the perl core).

[...] Worst case scenario for debugging interp size differences is
producing TWO .i files. One from a working XS module, one from a
failing (wrong interp size) module. Send both .i files through gnu
indent to normalize whitespace. Then copy paste the interp structs

I assume you mean PerlHandShakeInterpreter struct.

PerlHandShakeInterpreter is the fake never allocated interp struct for no threads perl. On threaded perl it will be PerlInterpreter.

alone into separate files, then run diff on the 2 files. Now you
know what members are different. If the interp structs are text
identical,

They were identical.

then probably the PL_sys_intern and struct interp_intern are
different.

They did not exist.

They do on Windows. They are different sizes depending on ithread without psuedofork and ithreads with psuedofork. Since its unix, that struct not existing is fine.

If that doesn't reveal the different, produce the 2 .i files again,
do a diff of the full size .i files and look for the difference.
Stripping all ^#.+$ lines from both .i files and gnu indent may or
may not help reduce the noise in the diff.

The diff was in things like

+typedef __ino_t ino_t;
-typedef __ino64_t ino_t;

and that was where I decided I had to go back to your sentence about
the
-D'es.

off_t would be a similar case. Is your "int" 32 or 64 (a performance question if 64 isnt native to the CPU)? is your size_t/pointer size 32 or 64 (what is your virtual memory space? what pointer do you use?)? is your maximum file on your os size 32 or 64 bits (a property of OS FS driver API and libc)?

All those numbers can be different. Plain C, unlike C++, offers no protection against declaring a function with a 32 bit arg vs a 64 bit arg between declaration and definition, and the segv/uninit data crash that results when you mix the 2 up.

Late response in this ticket is because I wrote the response then forgot about this tab for a month.

--
bulk88 ~ bulk88 at hotmail.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants