Skip Menu |

To: perlbug [...] perl.org
From: frederik [...] ofb.net
Subject: binaries mismatched again
Date: Fri, 10 Aug 2018 10:25:39 -0700
Download (untitled) / with headers
text/plain 6.2k
This is a bug report for perl from frederik@ofb.net, generated with the help of perlbug 1.41 running under perl 5.28.0. ----------------------------------------------------------------- [Please describe your issue here] I was able to find that I already reported this bug, sorry: https://rt.perl.org/Public/Bug/Display.html?id=130718 Maybe I have a bad memory, but it came up again and I wrote a new report before I remembered the old one. Here it is: Every time I upgrade my system, I run into this problem. But I don't upgrade often enough to remember what the solution is. The problem is that when running Perl code I see error messages containing the text "loadable library and perl binaries are mismatched". These messages don't point to a remedy, or even a specific module; instead they give the name of a C source file in a module. Googling is not very helpful. Looking in my shell history, I see that when I upgrade I should do something like "cpan-outdated --exclude-core -p | cpanm". However, now this command gives such an error: $ cpan-outdated --exclude-core -p Zlib.c: loadable library and perl binaries are mismatched (got handshake key 0xde00080, needed 0xce00080) After using 'locate' to search my filesystem for Zlib.c, I was able to figure out the offending module from the path name, and remove it. $ cpanm -U Compress::Raw::Zlib This made cpan-outdated work again. However, I think this is all a bit too complicated for casual users to understand. Where is the "discoverability"...? Am I an atypical user for having basic stuff like Compress::Raw::Zlib installed "locally" (i.e. in my home directory)? (I'm not sure how it got there) Or maybe everyone who encounters this problem knows what to do with a C file name? Or maybe cpan/cpanm are for experts-only? Or maybe Perl itself is experts-only these days? I should note that I'm using Arch Linux; is there a better situation on Debian-based distros? I'm not trying to be facetious, I just have a confusing situation and I don't know why other people are not also finding it as confusing as I do. Maybe there is a simple answer. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=medium --- Site configuration information for perl 5.28.0: Configured by builduser at Wed Aug 1 10:43:08 CEST 2018. Summary of my perl5 (revision 5 version 28 subversion 0) configuration: Platform: osname=linux osvers=4.17.11-arch1 archname=x86_64-linux-thread-multi uname='linux flo-64s 4.17.11-arch1 #1 smp preempt sun jul 29 10:11:16 utc 2018 x86_64 gnulinux ' config_args='-des -Dusethreads -Duseshrplib -Doptimize=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -Dprefix=/usr -Dvendorprefix=/usr -Dprivlib=/usr/share/perl5/core_perl -Darchlib=/usr/lib/perl5/5.28/core_perl -Dsitelib=/usr/share/perl5/site_perl -Dsitearch=/usr/lib/perl5/5.28/site_perl -Dvendorlib=/usr/share/perl5/vendor_perl -Dvendorarch=/usr/lib/perl5/5.28/vendor_perl -Dscriptdir=/usr/bin/core_perl -Dsitescript=/usr/bin/site_perl -Dvendorscript=/usr/bin/vendor_perl -Dinc_version_list=none -Dman1ext=1perl -Dman3ext=3perl -Dcccdlflags='-fPIC' -Dlddlflags=-shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -Dldflags=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='cc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' optimize='-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt' cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='8.1.1 20180531' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='cc' ldflags ='-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-pc-linux-gnu/8.1.1/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64 libs=-lpthread -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -ldl -lm -lcrypt -lutil -lc libc=libc-2.27.so so=so useshrplib=true libperl=libperl.so gnulibc_version='2.27' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.28/core_perl/CORE' cccdlflags='-fPIC' lddlflags='-shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -L/usr/local/lib -fstack-protector-strong' --- @INC for perl 5.28.0: /home/frederik/scripts-misc/perl /home/frederik/.local/lib/perl5/x86_64-linux-thread-multi /home/frederik/.local/lib/perl5 /usr/lib/perl5/5.28/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/5.28/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/5.28/core_perl /usr/share/perl5/core_perl --- Environment for perl 5.28.0: HOME=/home/frederik LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH=/home/frederik/.local/arch/x86_64/lib:/home/frederik/.local/lib:/usr/local/lib LOGDIR (unset) PATH=/home/frederik/.local/bin:/home/frederik/projects/mailproc:/home/frederik/scripts-misc:/home/frederik/.local/arch/x86_64/bin:/usr/bin/core_perl:/usr/bin/vendor_perl:/usr/bin/site_perl:/usr/local/bin:/usr/local/sbin:/usr/bin PERL5LIB=/home/frederik/scripts-misc/perl:/home/frederik/.local/lib/perl5: PERL_BADLANG (unset) PERL_LOCAL_LIB_ROOT=/home/frederik/.local/:/home/frederik/.local/:/home/frederik/.local/:/home/frederik/.local/ PERL_MB_OPT=--install_base "/home/frederik/.local/" PERL_MM_OPT=INSTALL_BASE=/home/frederik/.local/ SHELL=/bin/zsh
From: Andy Dougherty <doughera [...] lafayette.edu>
To: perl5-porters [...] perl.org
Date: Fri, 10 Aug 2018 16:18:32 -0400
Subject: Re: [perl #133440] binaries mismatched again
Download (untitled) / with headers
text/plain 2.1k
On Fri, Aug 10, 2018 at 11:24:20AM -0700, (via RT) wrote: Show quoted text
> # New Ticket Created by > # Please include the string: [perl #133440] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=133440 > > > > This is a bug report for perl from frederik@ofb.net, > generated with the help of perlbug 1.41 running under perl 5.28.0. > > ----------------------------------------------------------------- > [Please describe your issue here] > > I was able to find that I already reported this bug, sorry: > > https://rt.perl.org/Public/Bug/Display.html?id=130718 > > The problem is that when running Perl code I see error messages containing the text "loadable library and perl binaries are mismatched".
Show quoted text
> Am I an atypical user for having basic stuff like Compress::Raw::Zlib installed "locally" (i.e. in my home directory)? (I'm not sure how it got there)
Show quoted text
> I'm not trying to be facetious, I just have a confusing situation and I don't know why other people are not also finding it as confusing as I do. Maybe there is a simple answer.
The immediate problem is that you have version-specific stuff stored in a non-version-specific directory. That is, looking at your @INC, you have "locally" installed modules in Show quoted text
> /home/frederik/.local/lib/perl5/x86_64-linux-thread-multi
Perl's configuration defaults are to store version-specific stuff in version-specific directories. Lots more details are in the INSTALL file, under the section "Coexistence with earlier versions of perl 5". I'm not sure why you have Compress::Raw::Zlib installed locally, since it has been bundled with perl since version 5.9.4. Of course you are certainly welcome to install different versions than those bundled with perl, but if you are going to do so, and if you're going to store them in a non-version-specific directory, you'll need to recompile them when you upgrade to a new major version. Upgrade time is also a good time to re-visit the question of whether you want to continue overriding the standard version with your own local version. Hope that helps a little, -- Andy Dougherty doughera@lafayette.edu
Date: Fri, 10 Aug 2018 21:23:06 -0700
Subject: Re: [perl #133440] binaries mismatched again
To: Andy Dougherty via RT <perlbug-followup [...] perl.org>
From: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 3.1k
Show quoted text
> The immediate problem is that you have version-specific stuff stored > in a non-version-specific directory. That is, looking at your @INC, > you have "locally" installed modules in >
> > /home/frederik/.local/lib/perl5/x86_64-linux-thread-multi
> > Perl's configuration defaults are to store version-specific stuff in > version-specific directories. Lots more details are in the INSTALL file, > under the section "Coexistence with earlier versions of perl 5". > > I'm not sure why you have Compress::Raw::Zlib installed locally, since > it has been bundled with perl since version 5.9.4. Of course you are > certainly welcome to install different versions than those bundled with > perl, but if you are going to do so, and if you're going to store them in > a non-version-specific directory, you'll need to recompile them when > you upgrade to a new major version. Upgrade time is also a good time > to re-visit the question of whether you want to continue overriding the > standard version with your own local version. > > Hope that helps a little,
Thank you, it does. I already know a bit about the technical reason for the problem. Perhaps I installed something that depended on Compress::Raw::Zlib over 10 years ago, then that module ended up in a list of modules that I think I need to run various personal scripts and this is why I have it installed locally. What's this about INSTALL and version-specific directories? Did I configure things wrong when I set up cpanm? I guess I'm wondering why more people aren't coming to you with questions about this. And why not change the error message to point to a manual page which explains what to do. By "discoverability" I meant, a way for users to understand what they're seeing by reading the documentation that logically presents itself, for instance, if a module author says "you can install my module using the cpan tool", then two years later the user gets this error message, what is he expected to do next, given lack of omniscience - i.e. only knowing what he had to know to install cpan, and knowing the text of the error message. Reading what you wrote, I downloaded 'perl' and looked at INSTALL, at all the lines containing 'version.*directory' (I didn't have time to read all 3000 lines yet), and it didn't really answer my question. But this is not something I think most users would be expected to do; what happens is you get Perl from your distro and then you see there is a module in CPAN but not in the distro, so you install it locally using cpanm, etc. Then things break. I would like you to say, "Frederick, here is where you went wrong, you made a mistake that other users are not making when you did X". Because that would explain why you haven't been getting other bug reports about this. Lacking such an explanation, I'm kind of fearful that the answer is that Perl has just become a tool of an older generation of power users, many young programmers these days haven't even heard of it, they use bland corporate languages with fewer "gotchas"... (I'm only guessing here, because I don't use them myself...) But maybe that's off-topic. By the way I think I set up cpanm entirely by adding lines from `perl -Mlocal::lib=.local` to my shell profile. Best wishes, Frederick
RT-Send-CC: perl5-porters [...] perl.org
On Fri, 10 Aug 2018 11:24:20 -0700, frederik@ofb.net wrote: Show quoted text
> Maybe I have a bad memory, but it came up again and I wrote a new > report before I remembered the old one. Here it is: > > Every time I upgrade my system, I run into this problem. But I don't > upgrade often enough to remember what the solution is. > > The problem is that when running Perl code I see error messages > containing the text "loadable library and perl binaries are > mismatched". > > These messages don't point to a remedy, or even a specific module; > instead they give the name of a C source file in a module. Googling is > not very helpful.
https://perldoc.perl.org/perldiag.html#List-form-of-piped-open-not-implemented %s: loadable library and perl binaries are mismatched (got handshake key %p, needed %p) (P) A dynamic loading library .so or .dll was being loaded into the process that was built against a different build of perl than the said library was compiled against. Reinstalling the XS module will likely fix this error. -- bulk88 ~ bulk88 at hotmail.com
To: frederik [...] ofb.net
Date: Mon, 13 Aug 2018 23:56:26 +0200
Subject: Re: [perl #133440] binaries mismatched again
CC: perlbug <perlbug-followup [...] perl.org>
From: Leon Timmermans <fawaka [...] gmail.com>
On Sat, Aug 11, 2018 at 6:23 AM, <frederik@ofb.net> wrote: Show quoted text
> I guess I'm wondering why more people aren't coming to you with > questions about this. And why not change the error message to point to > a manual page which explains what to do. By "discoverability" I meant, > a way for users to understand what they're seeing by reading the > documentation that logically presents itself, for instance, if a > module author says "you can install my module using the cpan tool", > then two years later the user gets this error message, what is he > expected to do next, given lack of omniscience - i.e. only knowing > what he had to know to install cpan, and knowing the text of the error > message. > > Reading what you wrote, I downloaded 'perl' and looked at INSTALL, at > all the lines containing 'version.*directory' (I didn't have time to > read all 3000 lines yet), and it didn't really answer my question. But > this is not something I think most users would be expected to do; what > happens is you get Perl from your distro and then you see there is a > module in CPAN but not in the distro, so you install it locally using > cpanm, etc. Then things break. > > I would like you to say, "Frederick, here is where you went wrong, you > made a mistake that other users are not making when you did X". > Because that would explain why you haven't been getting other bug > reports about this. Lacking such an explanation, I'm kind of fearful > that the answer is that Perl has just become a tool of an older > generation of power users, many young programmers these days haven't > even heard of it, they use bland corporate languages with fewer > "gotchas"... (I'm only guessing here, because I don't use them > myself...) But maybe that's off-topic. > > By the way I think I set up cpanm entirely by adding lines from `perl > -Mlocal::lib=.local` to my shell profile.
I think most people avoid configurations where incompatible perl builds share the same arch directories. Judging by your @INC my first guess is that you're reusing your old perl's local::lib directories, which would indeed blow up in your face. Leon
To: Leon Timmermans <fawaka [...] gmail.com>
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 14 Aug 2018 09:41:42 +0100
CC: frederik [...] ofb.net, perlbug <perlbug-followup [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 580b
On Mon, Aug 13, 2018 at 11:56:26PM +0200, Leon Timmermans wrote: Show quoted text
> I think most people avoid configurations where incompatible perl > builds share the same arch directories. Judging by your @INC my first > guess is that you're reusing your old perl's local::lib directories, > which would indeed blow up in your face.
From a quick perusal of local::lib's documentation, it appears that it doesn't use version-specific paths under the local directory, which on the face of it seems like a design flaw. -- If life gives you lemons, you'll probably develop a citric acid allergy.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 871b
Dana Tue, 14 Aug 2018 01:42:09 -0700, davem reče: Show quoted text
> On Mon, Aug 13, 2018 at 11:56:26PM +0200, Leon Timmermans wrote:
> > I think most people avoid configurations where incompatible perl > > builds share the same arch directories. Judging by your @INC my first > > guess is that you're reusing your old perl's local::lib directories, > > which would indeed blow up in your face.
> > From a quick perusal of local::lib's documentation, it appears that > it doesn't use version-specific paths under the local directory, which > on the face of it seems like a design flaw.
Interestingly, local::lib creates version-specific paths on first-time initialization. But I think the problem is that by using ExtUtils::MakeMaker's INSTALL_BASE no installation to version-specific paths happens, so the design flaw is probably there. (Did not check Module::Build's --install_base)
CC: Frederick Eaton <frederik [...] ofb.net>, perlbug <perlbug-followup [...] perl.org>, Graham Knop <haarg [...] haarg.org>
From: Leon Timmermans <fawaka [...] gmail.com>
Date: Wed, 15 Aug 2018 01:19:16 +0200
Subject: Re: [perl #133440] binaries mismatched again
To: Dave Mitchell <davem [...] iabyn.com>
On Tue, Aug 14, 2018 at 10:41 AM, Dave Mitchell <davem@iabyn.com> wrote: Show quoted text
> On Mon, Aug 13, 2018 at 11:56:26PM +0200, Leon Timmermans wrote:
>> I think most people avoid configurations where incompatible perl >> builds share the same arch directories. Judging by your @INC my first >> guess is that you're reusing your old perl's local::lib directories, >> which would indeed blow up in your face.
> > From a quick perusal of local::lib's documentation, it appears that > it doesn't use version-specific paths under the local directory, which > on the face of it seems like a design flaw.
Perl doesn't quite provide enough rope to tie that knot. Or at least it doesn't for the common scenario where one has multiple (incompatible) perls and at login time one doesn't know yet which to use when. The only primitive we provide (PERL5LIB) does "prepend these dirs to @INC". I can sort of imagine a tool that does what you suggest, but the additional complexity and fragility sound uncomfortable. It's a real problem, but I don't think it can be solved on the local::lib side alone. Leon
From: frederik [...] ofb.net
CC: Dave Mitchell <davem [...] iabyn.com>, perlbug <perlbug-followup [...] perl.org>, Graham Knop <haarg [...] haarg.org>
To: Leon Timmermans <fawaka [...] gmail.com>
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 14 Aug 2018 17:14:41 -0700
Download (untitled) / with headers
text/plain 2.9k
On Wed, Aug 15, 2018 at 01:19:16AM +0200, Leon Timmermans wrote: Show quoted text
> On Tue, Aug 14, 2018 at 10:41 AM, Dave Mitchell <davem@iabyn.com> wrote:
> > On Mon, Aug 13, 2018 at 11:56:26PM +0200, Leon Timmermans wrote:
> >> I think most people avoid configurations where incompatible perl > >> builds share the same arch directories. Judging by your @INC my first > >> guess is that you're reusing your old perl's local::lib directories, > >> which would indeed blow up in your face.
> > > > From a quick perusal of local::lib's documentation, it appears that > > it doesn't use version-specific paths under the local directory, which > > on the face of it seems like a design flaw.
> > Perl doesn't quite provide enough rope to tie that knot. Or at least > it doesn't for the common scenario where one has multiple > (incompatible) perls and at login time one doesn't know yet which to > use when. The only primitive we provide (PERL5LIB) does "prepend these > dirs to @INC". I can sort of imagine a tool that does what you > suggest, but the additional complexity and fragility sound > uncomfortable. > > It's a real problem, but I don't think it can be solved on the > local::lib side alone.
Thanks for the discussion. I don't have two different versions of Perl installed, just one version from my distribution Arch, which gets updated occasionally. I'm not sure where this is headed but I just wanted us to keep in mind what it looks like for a new user who is doing the standard (?) thing of updating Perl via his distribution packaging system, while having CPAN modules installed locally (e.g. in his home directory, is this not typical?). There seems to be no discussion of changing the mismatched binary error message to point to documentation about e.g. updating CPAN modules. If the default setup were fixed so that version-specific paths were used for local modules, then what would our user experience consist of - user installs CPAN modules to make his scripts work, then a year later he upgrades his distribution to a new version of Perl and his scripts stop working with "Can't locate XXX.pm in @INC"? I guess that is better than the error message I saw, because then the user at least has a module name to start with, but I think it would be preferable to have an error that somehow ultimately leads (e.g. via a reference to a man page) to instructions on upgrading locally-installed CPAN modules. To make this concrete, what if you could change the error: Zlib.c: loadable library and perl binaries are mismatched (got handshake key 0xde00080, needed 0xce00080) to say: In Perl module Compress::Raw::Zlib (Zlib.c): loadable library and perl binaries are mismatched (got handshake key 0xde00080, needed 0xce00080). Do you need to recompile this module for a new Perl version? See `man perlmodupgrade`. (Actually, I'm noticing that the "perlmodinstall" manual page doesn't mention cpanm or cpan tools, I wonder if those should be there too) Hope these suggestions aren't too annoying, Frederick
From: Dan Book <grinnz [...] gmail.com>
CC: Leon Timmermans <fawaka [...] gmail.com>, Dave Mitchell <davem [...] iabyn.com>, Dave Mitchell via RT <perlbug-followup [...] perl.org>, haarg [...] haarg.org
To: Frederick Eaton <frederik [...] ofb.net>
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 14 Aug 2018 20:27:23 -0400
Download (untitled) / with headers
text/plain 2.2k
On Tue, Aug 14, 2018 at 8:16 PM <frederik@ofb.net> wrote:
Show quoted text

Thanks for the discussion. I don't have two different versions of Perl
installed, just one version from my distribution Arch, which gets
updated occasionally.

I'm not sure where this is headed but I just wanted us to keep in mind
what it looks like for a new user who is doing the standard (?) thing
of updating Perl via his distribution packaging system, while having
CPAN modules installed locally (e.g. in his home directory, is this
not typical?).

There seems to be no discussion of changing the mismatched binary
error message to point to documentation about e.g. updating CPAN
modules.

If the default setup were fixed so that version-specific paths were
used for local modules, then what would our user experience consist of
- user installs CPAN modules to make his scripts work, then a year
later he upgrades his distribution to a new version of Perl and his
scripts stop working with "Can't locate XXX.pm in @INC"? I guess that
is better than the error message I saw, because then the user at least
has a module name to start with, but I think it would be preferable to
have an error that somehow ultimately leads (e.g. via a reference to a
man page) to instructions on upgrading locally-installed CPAN modules.

To make this concrete, what if you could change the error:

    Zlib.c: loadable library and perl binaries are mismatched (got handshake key
0xde00080, needed 0xce00080)

to say:

    In Perl module Compress::Raw::Zlib (Zlib.c): loadable library and perl binaries are mismatched (got handshake key 0xde00080, needed 0xce00080). Do you need to recompile this module for a new Perl version? See `man perlmodupgrade`.

(Actually, I'm noticing that the "perlmodinstall" manual page doesn't
mention cpanm or cpan tools, I wonder if those should be there too)

Hope these suggestions aren't too annoying,

Frederick

These are good ideas. The fundamental issue though is that modules you installed for one major version of Perl must always be installed again for another, and users don't tend to expect this whether they use local::lib or not, it will always be a problem if you install modules without using your package manager in a Perl managed by your package manager.

-Dan 
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.5k
Dana Tue, 14 Aug 2018 16:19:54 -0700, LeonT reče: Show quoted text
> On Tue, Aug 14, 2018 at 10:41 AM, Dave Mitchell <davem@iabyn.com> wrote:
> > On Mon, Aug 13, 2018 at 11:56:26PM +0200, Leon Timmermans wrote:
> >> I think most people avoid configurations where incompatible perl > >> builds share the same arch directories. Judging by your @INC my first > >> guess is that you're reusing your old perl's local::lib directories, > >> which would indeed blow up in your face.
> > > > From a quick perusal of local::lib's documentation, it appears that > > it doesn't use version-specific paths under the local directory, which > > on the face of it seems like a design flaw.
> > Perl doesn't quite provide enough rope to tie that knot. Or at least > it doesn't for the common scenario where one has multiple > (incompatible) perls and at login time one doesn't know yet which to > use when. The only primitive we provide (PERL5LIB) does "prepend these > dirs to @INC". I can sort of imagine a tool that does what you > suggest, but the additional complexity and fragility sound > uncomfortable.
The PERL5LIB mechanism is sufficient here. If set user sets something like PERL5LIB=/root/perl5/lib/perl5 then @INC contains automatically /root/perl5/lib/perl5/5.28.0/x86_64-linux-thread-multi /root/perl5/lib/perl5/5.28.0 /root/perl5/lib/perl5/x86_64-linux-thread-multi /root/perl5/lib/perl5 so architecture and perl api (version) specific files have their place. The problem is, as I said, that INSTALL_BASE does not use these version-specific files.
To: Dan Book via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 14 Aug 2018 22:58:52 -0700
From: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 3.3k
On Tue, Aug 14, 2018 at 05:28:03PM -0700, Dan Book via RT wrote: Show quoted text
> On Tue, Aug 14, 2018 at 8:16 PM <frederik@ofb.net> wrote: >
> > > > Thanks for the discussion. I don't have two different versions of Perl > > installed, just one version from my distribution Arch, which gets > > updated occasionally. > > > > I'm not sure where this is headed but I just wanted us to keep in mind > > what it looks like for a new user who is doing the standard (?) thing > > of updating Perl via his distribution packaging system, while having > > CPAN modules installed locally (e.g. in his home directory, is this > > not typical?). > > > > There seems to be no discussion of changing the mismatched binary > > error message to point to documentation about e.g. updating CPAN > > modules. > > > > If the default setup were fixed so that version-specific paths were > > used for local modules, then what would our user experience consist of > > - user installs CPAN modules to make his scripts work, then a year > > later he upgrades his distribution to a new version of Perl and his > > scripts stop working with "Can't locate XXX.pm in @INC"? I guess that > > is better than the error message I saw, because then the user at least > > has a module name to start with, but I think it would be preferable to > > have an error that somehow ultimately leads (e.g. via a reference to a > > man page) to instructions on upgrading locally-installed CPAN modules. > > > > To make this concrete, what if you could change the error: > > > > Zlib.c: loadable library and perl binaries are mismatched (got > > handshake key > > 0xde00080, needed 0xce00080) > > > > to say: > > > > In Perl module Compress::Raw::Zlib (Zlib.c): loadable library and perl > > binaries are mismatched (got handshake key 0xde00080, needed 0xce00080). Do > > you need to recompile this module for a new Perl version? See `man > > perlmodupgrade`. > > > > (Actually, I'm noticing that the "perlmodinstall" manual page doesn't > > mention cpanm or cpan tools, I wonder if those should be there too) > > > > Hope these suggestions aren't too annoying, > > > > Frederick > >
> > These are good ideas. The fundamental issue though is that modules you > installed for one major version of Perl must always be installed again for > another, and users don't tend to expect this whether they use local::lib or > not, it will always be a problem if you install modules without using your > package manager in a Perl managed by your package manager.
Thanks. So people who use cpanm tend to have Perl compiled locally? I'm still trying to figure out the common use case. I didn't realize that you can do "cpanm Perl" but it looks like it fails: Perl.xs: At top level: Perl.xs:33:27: error: ‘PL_tokenbuf’ undeclared here (not in a function); did you mean ‘PL_top_env’? char ptokenbuf[sizeof(PL_tokenbuf)]; ^~~~~~~~~~~ PL_top_env I wonder how much space this ends up taking up were it to succeed. I think the idea with having modules installed in my home directory is that other scripts in my home directory can then depend on them, and I can sync the scripts between different hosts along with my shell config and so on. But that would argue for version-specific paths (because the different hosts might have different Perl versions). Thank you, Frederick
Date: Wed, 15 Aug 2018 08:47:08 +0200
Subject: Re: [perl #133440] binaries mismatched again
From: Andreas Koenig <andreas.koenig.7os6VVqR [...] franz.ak.mind.de>
CC: Dan Book via RT <perlbug-followup [...] perl.org>
To: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 608b
Show quoted text
>>>>> On Tue, 14 Aug 2018 22:58:52 -0700, frederik@ofb.net said:
Show quoted text
> But that would argue for version-specific paths (because the different > hosts might have different Perl versions).
You can sync your .local directory to other machines only if the two machines are perfectly in sync with regard to all involved libraries. As I read the whole thread here, your problem is recompilation after some library upgrade. The shell that comes with CPAN.pm has a commend for that, it is called `recompile`. Maye you try that out. Disclaimer: I wrote it. Let me know if you have questions about it. -- andreas
From: frederik [...] ofb.net
Date: Wed, 15 Aug 2018 00:55:44 -0700
Subject: Re: [perl #133440] binaries mismatched again
To: "(Andreas J. Koenig) via RT" <perlbug-followup [...] perl.org>
Download (untitled) / with headers
text/plain 1.5k
On Tue, Aug 14, 2018 at 11:47:54PM -0700, (Andreas J. Koenig) via RT wrote: Show quoted text
> >>>>> On Tue, 14 Aug 2018 22:58:52 -0700, frederik@ofb.net said:
>
> > But that would argue for version-specific paths (because the different > > hosts might have different Perl versions).
> > You can sync your .local directory to other machines only if the two > machines are perfectly in sync with regard to all involved libraries. > > As I read the whole thread here, your problem is recompilation after > some library upgrade. The shell that comes with CPAN.pm has a commend > for that, it is called `recompile`. Maye you try that out. Disclaimer: I > wrote it. Let me know if you have questions about it.
That's good advice, although I would say that the main problem for me is users being faced with an error message that doesn't point to a clear solution. But maybe that's what Stack Exchange or IRC is for? And maybe I'm the only one that's confused? Nevertheless one would hope that the local documentation would be sufficient for something this basic. FWIW I tried your 'recompile' and I got a bunch of errors. I'm attaching the output. I'm rerunning it after "cpanm -U Digest::SHA" and it seems to be doing something productive. The approach I tried earlier, which I think I mentioned, was to run cpan-outdated, but I guess this is something different - its man page doesn't say anything about helping you recompile binary modules which may not have actually changed versions. Your suggestion seems more promising. Thank you, Frederick
Download cpan.out
text/plain 65.5k

Message body is not shown because sender requested not to inline it.

CC: Leon Timmermans <fawaka [...] gmail.com>, perlbug <perlbug-followup [...] perl.org>, Graham Knop <haarg [...] haarg.org>
From: Dave Mitchell <davem [...] iabyn.com>
Date: Wed, 15 Aug 2018 11:35:36 +0100
Subject: Re: [perl #133440] binaries mismatched again
To: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 3.9k
On Tue, Aug 14, 2018 at 05:14:41PM -0700, frederik@ofb.net wrote: Show quoted text
> Thanks for the discussion. I don't have two different versions of Perl > installed, just one version from my distribution Arch, which gets > updated occasionally. > > I'm not sure where this is headed but I just wanted us to keep in mind > what it looks like for a new user who is doing the standard (?) thing > of updating Perl via his distribution packaging system, while having > CPAN modules installed locally (e.g. in his home directory, is this > not typical?).
The more typical approach is to have a personal perl installation separate from the system's one: this would contain both the perl binary plus any additional modules you have installed. That way you have complete control over what gets upgraded and when. Using the system perl in combination with privately built/installed modules puts you at the mercy of whatever your OS's update/upgrade procedures do to the system perl. Show quoted text
> If the default setup were fixed so that version-specific paths were > used for local modules,
Note that the default behaviour for perl *is* always to install in version-specific directories. The behaviour you are seeing is that of a third-party module, local::lib, which is not bundled with perl, and which we (p5p) have no control over. (Although elsewhere in this thread it appears that MakeMaker etc may be to blame too.) Show quoted text
> then what would our user experience consist of > - user installs CPAN modules to make his scripts work, then a year > later he upgrades his distribution to a new version of Perl and his > scripts stop working with "Can't locate XXX.pm in @INC"? I guess that > is better than the error message I saw, because then the user at least > has a module name to start with, but I think it would be preferable to > have an error that somehow ultimately leads (e.g. via a reference to a > man page) to instructions on upgrading locally-installed CPAN modules. > > To make this concrete, what if you could change the error: > > Zlib.c: loadable library and perl binaries are mismatched (got handshake key > 0xde00080, needed 0xce00080) > > to say: > > In Perl module Compress::Raw::Zlib (Zlib.c): loadable library and perl binaries are mismatched (got handshake key 0xde00080, needed 0xce00080). Do you need to recompile this module for a new Perl version? See `man perlmodupgrade`. > > (Actually, I'm noticing that the "perlmodinstall" manual page doesn't > mention cpanm or cpan tools, I wonder if those should be there too)
Some observations. As I pointed out earlier, I'm not sure that it will be possible (or at least easy) to display the module name as part of the error message. I'm very much in favour of removing the 'handshake' part of the message, and replacing it with something informative and useful - i.e. to actually decode the hex values and explain what was mismatched. I'm also not opposed to the error message referring to a man page; although note that if your perl installation is borked due to version mismatches, then it's possible that the tool you use for viewing man pages (such as perldoc) also wont work! It may be overkill to have a complete new man page - perhaps just an extra section in an existing document instead? It will be quite difficult to write such a perlmodupgrade man page. Bear in mind that the error message you got refers to any sort of mismatch between the perl binary you are executing, and an XS module which that perl is trying to load. Your particular scenario is just one of many, and reinstalling the offending module isn't always the correct solution. Such a manual page would have to explain many possible scenarios, including interactions with third-party solutions such as OS packages, perlbrew, local::lib etc. Personally I don't feel I have the expertise to write all of such a document. -- Any [programming] language that doesn't occasionally surprise the novice will pay for it by continually surprising the expert. -- Larry Wall
Date: Wed, 15 Aug 2018 12:14:10 -0400
Subject: Re: [perl #133440] binaries mismatched again
To: Dave Mitchell <davem [...] iabyn.com>
From: Dan Book <grinnz [...] gmail.com>
CC: Frederick Eaton <frederik [...] ofb.net>, Leon Timmermans <fawaka [...] gmail.com>, Dave Mitchell via RT <perlbug-followup [...] perl.org>, Graham Knop <haarg [...] haarg.org>
Download (untitled) / with headers
text/plain 716b
Download (untitled) / with headers
text/html 1017b
On Wed, Aug 15, 2018 at 6:36 AM Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text

The more typical approach is to have a personal perl installation separate
from the system's one: this would contain both the perl binary plus any
additional modules you have installed. That way you have complete control
over what gets upgraded and when. Using the system perl in combination with
privately built/installed modules puts you at the mercy of whatever your
OS's update/upgrade procedures do to the system perl.

To add to this: typical methods of doing this are Perl::Build, perlbrew, and plenv. The first simply installs a Perl somewhere, the other two are Perl-based and shell-based multiple-Perl managers.

-Dan 
To: "(Andreas J. Koenig) via RT" <perlbug-followup [...] perl.org>
From: frederik [...] ofb.net
Date: Thu, 16 Aug 2018 21:51:39 -0700
Subject: Re: [perl #133440] binaries mismatched again
Just wanted to briefly follow this up... Andreas: "cpan[1]> recompile" worked fine on both my machines. I also noticed that the second time I run it, it notices nothing needs to be done, which is great. Dave: Thanks for the observations. I'm happy to look at the code and send a patch for a better error message when I have time, which is probably not going to be within the next month or so. Also I think it should be worthwhile to investigate a way to get the full module name in the message, presumably whatever compiles modules (MakeMaker?) can be changed to provide a CPP macro with the module name, and I think this could be done in a backwards-compatible manner - whatever that means. Dan: I can look at perlbrew et al (it was also recommended off-list), I'd of course like to see how far I can go with the binaries provided by my distro. One problem with my argument that the error message isn't user-friendly, is that anyone can easily submit a bug for it and get friendly and helpful replies from the developers, as I have done. Still, I'm interested in improving things and I hope that I myself or someone else can find time to work on this in the not-too-distant future. Thanks, Frederick On Tue, Aug 14, 2018 at 11:47:54PM -0700, (Andreas J. Koenig) via RT wrote: Show quoted text
> >>>>> On Tue, 14 Aug 2018 22:58:52 -0700, frederik@ofb.net said:
>
> > But that would argue for version-specific paths (because the different > > hosts might have different Perl versions).
> > You can sync your .local directory to other machines only if the two > machines are perfectly in sync with regard to all involved libraries. > > As I read the whole thread here, your problem is recompilation after > some library upgrade. The shell that comes with CPAN.pm has a commend > for that, it is called `recompile`. Maye you try that out. Disclaimer: I > wrote it. Let me know if you have questions about it. > > -- > andreas >
On Wed, Aug 15, 2018 at 03:35:57AM -0700, Dave Mitchell via RT wrote: Show quoted text
> On Tue, Aug 14, 2018 at 05:14:41PM -0700, frederik@ofb.net wrote:
> > Thanks for the discussion. I don't have two different versions of Perl > > installed, just one version from my distribution Arch, which gets > > updated occasionally. > > > > I'm not sure where this is headed but I just wanted us to keep in mind > > what it looks like for a new user who is doing the standard (?) thing > > of updating Perl via his distribution packaging system, while having > > CPAN modules installed locally (e.g. in his home directory, is this > > not typical?).
> > The more typical approach is to have a personal perl installation separate > from the system's one: this would contain both the perl binary plus any > additional modules you have installed. That way you have complete control > over what gets upgraded and when. Using the system perl in combination with > privately built/installed modules puts you at the mercy of whatever your > OS's update/upgrade procedures do to the system perl. >
> > If the default setup were fixed so that version-specific paths were > > used for local modules,
> > Note that the default behaviour for perl *is* always to install in > version-specific directories. The behaviour you are seeing is that of a > third-party module, local::lib, which is not bundled with perl, and which > we (p5p) have no control over. (Although elsewhere in this thread it > appears that MakeMaker etc may be to blame too.) >
> > then what would our user experience consist of > > - user installs CPAN modules to make his scripts work, then a year > > later he upgrades his distribution to a new version of Perl and his > > scripts stop working with "Can't locate XXX.pm in @INC"? I guess that > > is better than the error message I saw, because then the user at least > > has a module name to start with, but I think it would be preferable to > > have an error that somehow ultimately leads (e.g. via a reference to a > > man page) to instructions on upgrading locally-installed CPAN modules. > > > > To make this concrete, what if you could change the error: > > > > Zlib.c: loadable library and perl binaries are mismatched (got handshake key > > 0xde00080, needed 0xce00080) > > > > to say: > > > > In Perl module Compress::Raw::Zlib (Zlib.c): loadable library and perl binaries are mismatched (got handshake key 0xde00080, needed 0xce00080). Do you need to recompile this module for a new Perl version? See `man perlmodupgrade`. > > > > (Actually, I'm noticing that the "perlmodinstall" manual page doesn't > > mention cpanm or cpan tools, I wonder if those should be there too)
> > Some observations. > > As I pointed out earlier, I'm not sure that it will be possible (or at > least easy) to display the module name as part of the error message. > > I'm very much in favour of removing the 'handshake' part of the message, > and replacing it with something informative and useful - i.e. to actually > decode the hex values and explain what was mismatched. > > I'm also not opposed to the error message referring to a man page; > although note that if your perl installation is borked due to version > mismatches, then it's possible that the tool you use for viewing man pages > (such as perldoc) also wont work! > > It may be overkill to have a complete new man page - perhaps just an extra > section in an existing document instead? > > It will be quite difficult to write such a perlmodupgrade man page. Bear > in mind that the error message you got refers to any sort of mismatch > between the perl binary you are executing, and an XS module which that perl > is trying to load. Your particular scenario is just one of many, and > reinstalling the offending module isn't always the correct solution. Such > a manual page would have to explain many possible scenarios, including > interactions with third-party solutions such as OS packages, perlbrew, > local::lib etc. > > Personally I don't feel I have the expertise to write all of such a > document. > > -- > Any [programming] language that doesn't occasionally surprise the > novice will pay for it by continually surprising the expert. > -- Larry Wall >
Date: Fri, 17 Aug 2018 22:40:07 +0200
Subject: Re: [perl #133440] binaries mismatched again
From: Andreas Koenig <andreas.koenig.7os6VVqR [...] franz.ak.mind.de>
CC: "\(Andreas J. Koenig\) via RT" <perlbug-followup [...] perl.org>
To: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 765b
Show quoted text
>>>>> On Thu, 16 Aug 2018 21:51:39 -0700, frederik@ofb.net said:
Show quoted text
> Just wanted to briefly follow this up... > Andreas: "cpan[1]> recompile" worked fine on both my machines. I also > noticed that the second time I run it, it notices nothing needs to be > done, which is great.
Thanks for letting me know and for sending the report how noisy it was during its performance. I'm sorry for this bug, I've since fixed it in the repository so it will appear in version 2.21. Besides that I have also found the original ticket in which we received the first guide that explained the "got handshake key" message. It was https://rt.perl.org/Public/Bug/Display.html?id=125236. Maybe the two tickets would be suited to be folded into one, I'm not sure. -- andreas
CC: frederik [...] ofb.net, Leon Timmermans <fawaka [...] gmail.com>, Graham Knop <haarg [...] haarg.org>, Dave Mitchell <davem [...] iabyn.com>
From: Karen Etheridge <perl [...] froods.org>
To: perlbug <perlbug-followup [...] perl.org>
Date: Sat, 18 Aug 2018 19:55:14 -0700
Subject: Re: [perl #133440] binaries mismatched again
Download (untitled) / with headers
text/plain 1.5k


On Wed, Aug 15, 2018 at 3:35 AM, Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text
Note that the default behaviour for perl *is* always to install in
version-specific directories. The behaviour you are seeing is that of a
third-party module, local::lib, which is not bundled with perl, and which
we (p5p) have no control over. (Although elsewhere in this thread it
appears that MakeMaker etc may be to blame too.)

But the toolchain gang does, and there's a large overlap between that group and p5p. We should not go "oh well, not in core, not our problem" and leave it to the original bug reporter to have to pursue the issue in another bug queue (and attempt to explain the problem again, which leads to a game of telephone). We can do better than that.

All local::lib does is set some environment variables, e.g.:

$ cd ~; perl -Mlocal::lib=local
Attempting to create directory /Users/ether/local
PATH="/Users/ether/local/bin${PATH:+:${PATH}}"; export PATH;
PERL5LIB="/Users/ether/local/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export PERL5LIB;
PERL_LOCAL_LIB_ROOT="/Users/ether/local${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; export PERL_LOCAL_LIB_ROOT;
PERL_MB_OPT="--install_base \"/Users/ether/local\""; export PERL_MB_OPT;
PERL_MM_OPT="INSTALL_BASE=/Users/ether/local"; export PERL_MM_OPT;
Perhaps those environment variables should always contain the perl version.  There would be some backcompat issues, but we could possibly handle that by putting the old non-versioned directory into PERL5LIB after the new versioned one (or something clever with symlinks, or.. well, we'd have to discuss it.)




To: "(Andreas J. Koenig) via RT" <perlbug-followup [...] perl.org>
Subject: Re: [perl #133440] binaries mismatched again
Date: Sat, 18 Aug 2018 20:24:41 -0700
From: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 2.8k
Thanks for your reply. While trying to downgrade my system (debugging another issue) I discovered a possible reason why I have problematic binary modules in my home directory which are also in Perl core (causing cpan to break when I upgrade Perl). The reason appears to be that the "recompile" cpan command reinstalls modules which I had removed earlier using 'cpanm'. $ PERL5LIB= cpanm -U Digest::SHA ... $ PERL5LIB= cpanm -U Time::HiRes ... $ perldoc -l Time::HiRes /usr/lib/perl5/5.28/core_perl/Time/HiRes.pm $ perldoc -l Digest::SHA /usr/lib/perl5/5.28/core_perl/Digest/SHA.pm (after cpan recompile) $ perldoc -l Time::HiRes /home/frederik/.local/lib/perl5/x86_64-linux-thread-multi/Time/HiRes.pm $ perldoc -l Digest::SHA /home/frederik/.local/lib/perl5/x86_64-linux-thread-multi/Digest/SHA.pm I don't know if this is a bug or just an annoyance, but it may go a little way towards solving the mystery of why I have these modules around, since I think I recall removing them earlier. Obviously, I don't look too carefully at the ouput of 'cpan -r', because I have better things to do, I just assume it's doing the "right thing". By the time I realize that I have duplicate modules installed, it's probably a year later and I don't remember having removed them. Also, I notice that if I ask 'cpanm' to install a module which is already in core, then it doesn't warn me: $ perldoc -l DB_File /usr/lib/perl5/5.28/core_perl/DB_File.pm $ cpanm DB_File --> Working on DB_File Fetching http://www.cpan.org/authors/id/P/PM/PMQS/DB_File-1.842.tar.gz ... OK Configuring DB_File-1.842 ... ^C I haven't checked what happens if I use cpanm to install a module which depends on modules in core, but if you give me an example of a CPAN module with dependencies in core then I can try this out. I couldn't figure out how to use the 'cpan' tool to remove modules, but 'cpanm' has a nice simple interface for doing that so I thought it would be compatible with 'cpan'. Also, I see 'cpan -r -T' runs lots of tests, I thought '-T' was to supposed to skip the tests. Hope this helps, Frederick On Fri, Aug 17, 2018 at 01:40:52PM -0700, (Andreas J. Koenig) via RT wrote: Show quoted text
> >>>>> On Thu, 16 Aug 2018 21:51:39 -0700, frederik@ofb.net said:
>
> > Just wanted to briefly follow this up... > > Andreas: "cpan[1]> recompile" worked fine on both my machines. I also > > noticed that the second time I run it, it notices nothing needs to be > > done, which is great.
> > Thanks for letting me know and for sending the report how noisy it was > during its performance. I'm sorry for this bug, I've since fixed it in > the repository so it will appear in version 2.21. > > Besides that I have also found the original ticket in which we received > the first guide that explained the "got handshake key" message. It was > https://rt.perl.org/Public/Bug/Display.html?id=125236. Maybe the two > tickets would be suited to be folded into one, I'm not sure.
From: Dan Book <grinnz [...] gmail.com>
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
To: Frederick Eaton <frederik [...] ofb.net>
Date: Sun, 19 Aug 2018 02:27:03 -0400
Subject: Re: [perl #133440] binaries mismatched again
Download (untitled) / with headers
text/plain 2.5k
On Sat, Aug 18, 2018 at 11:26 PM <frederik@ofb.net> wrote:
Show quoted text
Thanks for your reply. While trying to downgrade my system (debugging
another issue) I discovered a possible reason why I have problematic
binary modules in my home directory which are also in Perl core
(causing cpan to break when I upgrade Perl). The reason appears to be
that the "recompile" cpan command reinstalls modules which I had
removed earlier using 'cpanm'.

$ PERL5LIB= cpanm -U Digest::SHA
...
$ PERL5LIB= cpanm -U Time::HiRes
...
$ perldoc -l Time::HiRes
/usr/lib/perl5/5.28/core_perl/Time/HiRes.pm
$ perldoc -l Digest::SHA
/usr/lib/perl5/5.28/core_perl/Digest/SHA.pm

(after cpan recompile)

$ perldoc -l Time::HiRes
/home/frederik/.local/lib/perl5/x86_64-linux-thread-multi/Time/HiRes.pm
$ perldoc -l Digest::SHA
/home/frederik/.local/lib/perl5/x86_64-linux-thread-multi/Digest/SHA.pm

I don't know if this is a bug or just an annoyance, but it may go a
little way towards solving the mystery of why I have these modules
around, since I think I recall removing them earlier. Obviously, I
don't look too carefully at the ouput of 'cpan -r', because I have
better things to do, I just assume it's doing the "right thing". By
the time I realize that I have duplicate modules installed, it's
probably a year later and I don't remember having removed them.

Also, I notice that if I ask 'cpanm' to install a module which is
already in core, then it doesn't warn me:

$ perldoc -l DB_File
/usr/lib/perl5/5.28/core_perl/DB_File.pm
$ cpanm DB_File
--> Working on DB_File
Fetching http://www.cpan.org/authors/id/P/PM/PMQS/DB_File-1.842.tar.gz ... OK
Configuring DB_File-1.842 ... ^C

I haven't checked what happens if I use cpanm to install a module
which depends on modules in core, but if you give me an example of a
CPAN module with dependencies in core then I can try this out.

I couldn't figure out how to use the 'cpan' tool to remove modules,
but 'cpanm' has a nice simple interface for doing that so I thought it
would be compatible with 'cpan'.

 These are dual-life modules; it is expected and often desired to install them a second time in site or vendor libs to upgrade them from the core module version. Since Perl 5.12, site and vendor libs take precedence in @INC for that reason. local::lib of course takes precedence over all standard lib directories. cpanm's uninstall command relies on packlists, and thus can only uninstall modules installed by a cpan client (whether cpanm or cpan).

-Dan
From: frederik [...] ofb.net
Date: Tue, 21 Aug 2018 16:43:58 -0700
Subject: Re: [perl #133440] binaries mismatched again
To: Dan Book via RT <perlbug-followup [...] perl.org>
Show quoted text
> These are dual-life modules; it is expected and often desired to install > them a second time in site or vendor libs to upgrade them from the core > module version. Since Perl 5.12, site and vendor libs take precedence > in @INC for that reason. local::lib of course takes precedence over all > standard lib directories. cpanm's uninstall command relies on packlists, > and thus can only uninstall modules installed by a cpan client (whether > cpanm or cpan).
You're saying that these modules are purposefully installed by 'cpan' even when compatible versions already exist in 'core'? I guess not everyone expects this, because when I first submitted this bug report I got a response from Andy Dougherty saying "I'm not sure why you have Compress::Raw::Zlib installed locally, since it has been bundled with perl since version 5.9.4." And I have in my shell history that I uninstalled it, while now it is back in ~/.local. Although when I removed it again and re-ran 'cpan -r', it didn't reappear, so I'm not entirely sure what's happening. Thanks, Frederick
To: Frederick Eaton <frederik [...] ofb.net>
Date: Tue, 21 Aug 2018 19:59:42 -0400
Subject: Re: [perl #133440] binaries mismatched again
From: Dan Book <grinnz [...] gmail.com>
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
Download (untitled) / with headers
text/plain 1.1k
On Tue, Aug 21, 2018 at 7:45 PM <frederik@ofb.net> wrote:
Show quoted text

You're saying that these modules are purposefully installed by 'cpan'
even when compatible versions already exist in 'core'?

I guess not everyone expects this, because when I first submitted this
bug report I got a response from Andy Dougherty saying "I'm not sure
why you have Compress::Raw::Zlib installed locally, since it has been
bundled with perl since version 5.9.4." And I have in my shell history
that I uninstalled it, while now it is back in ~/.local. Although when
I removed it again and re-ran 'cpan -r', it didn't reappear, so I'm
not entirely sure what's happening.

That is correct. His response seemed to account for this possibility but was asking why you had installed a newer version in a local lib without the use of version specific directories, which could be considered a deficiency in the toolchain. Regardless, upgrades to dual-life modules are installed in sitelib or local libs very commonly, this is the reason they are available from CPAN. I can't comment on how CPAN.pm's recompile function interacts with dual-life modules or local libs.

-Dan
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 21 Aug 2018 17:35:31 -0700
To: Karen Etheridge via RT <perlbug-followup [...] perl.org>
From: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 1.8k
By the way I never took the time to thank you for pointing these things out and trying to up the responsibility level a bit here. Thanks! Frederick On Sat, Aug 18, 2018 at 07:55:29PM -0700, Karen Etheridge via RT wrote: Show quoted text
> On Wed, Aug 15, 2018 at 3:35 AM, Dave Mitchell <davem@iabyn.com> wrote: >
> > Note that the default behaviour for perl *is* always to install in > > version-specific directories. The behaviour you are seeing is that of a > > third-party module, local::lib, which is not bundled with perl, and which > > we (p5p) have no control over. (Although elsewhere in this thread it > > appears that MakeMaker etc may be to blame too.) > >
> > But the toolchain gang does, and there's a large overlap between that group > and p5p. We should not go "oh well, not in core, not our problem" and leave > it to the original bug reporter to have to pursue the issue in another bug > queue (and attempt to explain the problem again, which leads to a game of > telephone). We can do better than that. > > All local::lib does is set some environment variables, e.g.: > > $ cd ~; perl -Mlocal::lib=local > Attempting to create directory /Users/ether/local > PATH="/Users/ether/local/bin${PATH:+:${PATH}}"; export PATH; > PERL5LIB="/Users/ether/local/lib/perl5${PERL5LIB:+:${PERL5LIB}}"; export > PERL5LIB; > PERL_LOCAL_LIB_ROOT="/Users/ether/local${PERL_LOCAL_LIB_ROOT:+:${PERL_LOCAL_LIB_ROOT}}"; > export PERL_LOCAL_LIB_ROOT; > PERL_MB_OPT="--install_base \"/Users/ether/local\""; export PERL_MB_OPT; > PERL_MM_OPT="INSTALL_BASE=/Users/ether/local"; export PERL_MM_OPT; > Perhaps those environment variables should always contain the perl > version. There would be some backcompat issues, but we could possibly > handle that by putting the old non-versioned directory into PERL5LIB after > the new versioned one (or something clever with symlinks, or.. well, we'd > have to discuss it.) > > > -ether@cpan.org >
To: Karen Etheridge <perl [...] froods.org>
Date: Wed, 22 Aug 2018 09:50:18 +0100
Subject: Re: [perl #133440] binaries mismatched again
From: Dave Mitchell <davem [...] iabyn.com>
CC: perlbug <perlbug-followup [...] perl.org>, frederik [...] ofb.net, Leon Timmermans <fawaka [...] gmail.com>, Graham Knop <haarg [...] haarg.org>
Download (untitled) / with headers
text/plain 1.8k
On Sat, Aug 18, 2018 at 07:55:14PM -0700, Karen Etheridge wrote: Show quoted text
> On Wed, Aug 15, 2018 at 3:35 AM, Dave Mitchell <davem@iabyn.com> wrote: >
> > Note that the default behaviour for perl *is* always to install in > > version-specific directories. The behaviour you are seeing is that of a > > third-party module, local::lib, which is not bundled with perl, and which > > we (p5p) have no control over. (Although elsewhere in this thread it > > appears that MakeMaker etc may be to blame too.)
> > But the toolchain gang does, and there's a large overlap between that group > and p5p. We should not go "oh well, not in core, not our problem" and leave > it to the original bug reporter to have to pursue the issue in another bug > queue (and attempt to explain the problem again, which leads to a game of > telephone). We can do better than that.
You quoted one paragraph of mine from a long email, where I had clearly spent some some and thought on the issue, and was attempting to narrow the issues, and trying to find out which bits need fixing and by who etc. At no point did I say or imply "not a perl issue, go bug the owner of local::lib instead; I'm closing this ticket". What I said in that paragraph is also completely true - whatever discussions we might have here, and what grand solutions we might propose to fix local::lib, ultimately the owner of the cpan module is free to have their own ideas as to what what their module should do and be irritated that p5p is drying to dictate solutions to them. So I'm mildly peeved by your response, and the OP's follow up of "oh at last someone's taking some responsibility" - after about 7 porters have already taken time and effort to try and diagnose the issue or suggest workarounds and/or long-term solutions. -- "There's something wrong with our bloody ships today, Chatfield." -- Admiral Beatty at the Battle of Jutland, 31st May 1916.
To: Dave Mitchell via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133440] binaries mismatched again
Date: Wed, 22 Aug 2018 09:41:36 -0700
From: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 1.2k
Show quoted text
> So I'm mildly peeved by your response, and the OP's follow up of "oh at > last someone's taking some responsibility" - after about 7 porters have > already taken time and effort to try and diagnose the issue or suggest > workarounds and/or long-term solutions.
I'm sorry Dave, my reply to Karen was meant to be off-list, but I failed to note the "via RT" in the email address. (Apologies to Karen as well) In any case I wasn't thanking her for taking responsibility, but for encouraging P5P to do so. I'm also very familiar with the annoyance of having to "pursue the issue in another bug queue", although I'm not sure where I would send what suggestions, as I think we haven't really discussed that yet. I'm attaching some proposed patches for P5P, maybe I have enough information to take some responsibility as well at this point. However, these are not meant to be a "finished product", I'm sure they will need some additional work before they can be applied. I would hope that eventually the full module name can be reported by the util.c error message, but rather than dig around to figure out how to do that (I am badly unfamiliar with the software) I decided that my documentation patch would just explain how to search for the module using the information supplied by the current message. Thank you, Frederick
Download util.c.diff
text/plain 540b

Message body is not shown because sender requested not to inline it.

Download perldiag.diff
text/plain 2.8k

Message body is not shown because sender requested not to inline it.

To: Dave Mitchell via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133440] binaries mismatched again
Date: Mon, 27 Aug 2018 09:26:35 -0700
From: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 791b
Here is an updated version of the documentation patch. I made some clarifications and also added a paragraph suggesting that users prefer distribution packages of perl modules. After I learned that 'cpan -r' and other installation commands cause core modules to get reinstalled locally, I'm interested in learning how to prevent that from happening. The 'cpanm' man page has some promising options for excluding core and vendor modules but when I try these options with such a module, then the module still gets installed: $ cpanm --self-contained --exclude-vendor HTTP::Date --> Working on HTTP::Date Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTTP-Date-6.02.tar.gz ... OK If anyone can comment on this... I also welcome comments on the patches. Thanks, Frederick
From: Dave Mitchell <davem [...] iabyn.com>
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
Date: Tue, 28 Aug 2018 08:47:54 +0100
Subject: Re: [perl #133440] binaries mismatched again
To: frederik [...] ofb.net
Download (untitled) / with headers
text/plain 264b
On Mon, Aug 27, 2018 at 09:26:35AM -0700, frederik@ofb.net wrote: Show quoted text
> Here is an updated version of the documentation patch.
I'm not seeing anything attached. -- But Pity stayed his hand. "It's a pity I've run out of bullets", he thought. -- "Bored of the Rings"
To: Dave Mitchell via RT <perlbug-followup [...] perl.org>
From: frederik [...] ofb.net
Date: Tue, 28 Aug 2018 08:23:57 -0700
Subject: Re: [perl #133440] binaries mismatched again
Download (untitled) / with headers
text/plain 386b
Apologies, trying that again... On Tue, Aug 28, 2018 at 12:48:29AM -0700, Dave Mitchell via RT wrote: Show quoted text
> On Mon, Aug 27, 2018 at 09:26:35AM -0700, frederik@ofb.net wrote:
> > Here is an updated version of the documentation patch.
> > I'm not seeing anything attached. > > -- > But Pity stayed his hand. "It's a pity I've run out of bullets", > he thought. -- "Bored of the Rings" >
Download perldiag.2.diff
text/plain 3k

Message body is not shown because sender requested not to inline it.

From: frederik [...] ofb.net
To: Dave Mitchell via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133440] binaries mismatched again
Date: Fri, 14 Sep 2018 15:44:22 -0700
Any comments on the patch?
Subject: Re: [perl #133440] binaries mismatched again
Date: Mon, 17 Sep 2018 12:24:31 +0100
To: frederik [...] ofb.net
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 676b
On Tue, Aug 28, 2018 at 08:23:57AM -0700, frederik@ofb.net wrote: Show quoted text
> +Typically this error occurs after an OS distribution upgrade, in which > +a new Perl version has been installed system-wide, but modules > +installed locally (e.g. in the user's home directory) have not been > +recompiled yet.
I dislike this text (and by extension the whole patch). You are mentioning one particular scenario, which happened to occur to you, and are assuming that this is the common case. There are many reasons why a user might get this error, and your case was just one of them. -- Indomitable in retreat, invincible in advance, insufferable in victory -- Churchill on Montgomery
From: frederik [...] ofb.net
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
To: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133440] binaries mismatched again
Date: Mon, 17 Sep 2018 07:15:55 -0700
Download (untitled) / with headers
text/plain 2.4k
On Mon, Sep 17, 2018 at 12:24:31PM +0100, Dave Mitchell wrote: Show quoted text
> On Tue, Aug 28, 2018 at 08:23:57AM -0700, frederik@ofb.net wrote:
> > +Typically this error occurs after an OS distribution upgrade, in which > > +a new Perl version has been installed system-wide, but modules > > +installed locally (e.g. in the user's home directory) have not been > > +recompiled yet.
> > I dislike this text (and by extension the whole patch). You are mentioning > one particular scenario, which happened to occur to you, and are assuming > that this is the common case. > > There are many reasons why a user might get this error, and your case was > just one of them.
Thanks for the feedback. Do you use Linux From Scratich or something? I guess my patch makes a lot of assumptions about how people would get this particular error. I can try to change the wording if you give me more guidance. The underlying goal is to make it so that people can figure out how to fix their systems and get them into a working state again without using Google, i.e. just by using the manual pages and other documentation that comes with Perl. I guess another assumption is that a user doesn't have to have read and remembered all the documentation that comes with all the software he installed, because otherwise you could argue that this is already the case. Another thing which is confusing is that the software, while presumably knowing the name of the offending module which it was attempting to load at the time of the error, did not report this name to the user. That's confusing because it makes it seem like the problem is not with a module. The documentation and diagnostics changes I suggested were also partly meant to compensate for this information gap. If it is just that one sentence that's a problem, then it would probably be easy to rephrase it in a way that preserves the intent while broadening the scope: "Typically this error occurs after a new version of Perl has been installed, for example during an operating system upgrade. Any non-core modules with XS code will need to be recompiled at the same time, or they will trigger this error when loaded. In the common case where such modules have been installed using the 'cpan' or 'cpanm' tool [...]" Also I don't know the "pod" format, I'm assuming that it would be easier for whoever applies the patch to fix any formatting problems himself rather than engage in further back-and-forth, but let me know if that is not the case. Thanks, Frederick
CC: Dave Mitchell <davem [...] iabyn.com>
From: Tomasz Konojacki <me [...] xenu.pl>
To: perl5-porters [...] perl.org
Date: Mon, 17 Sep 2018 16:34:06 +0200
Subject: Re: [perl #133440] binaries mismatched again
Download (untitled) / with headers
text/plain 873b
On Mon, 17 Sep 2018, at 13:24, Dave Mitchell wrote: Show quoted text
> On Tue, Aug 28, 2018 at 08:23:57AM -0700, frederik@ofb.net wrote:
> > +Typically this error occurs after an OS distribution upgrade, in which > > +a new Perl version has been installed system-wide, but modules > > +installed locally (e.g. in the user's home directory) have not been > > +recompiled yet.
> > I dislike this text (and by extension the whole patch). You are mentioning > one particular scenario, which happened to occur to you, and are assuming > that this is the common case. > > There are many reasons why a user might get this error, and your case was > just one of them.
As someone who spends a lot of time on perl support irc channels, I'd say it's the most common case *by far*. It happens all the time to users of rolling-release linux distributions like arch, gentoo, opensuse tumbleweed etc.
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
To: frederik [...] ofb.net
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 18 Sep 2018 10:43:27 +0100
On Mon, Sep 17, 2018 at 07:15:55AM -0700, frederik@ofb.net wrote: Show quoted text
> If it is just that one sentence that's a problem, then it would > probably be easy to rephrase it in a way that preserves the intent > while broadening the scope:
No, I think the general thrust of the patch is misguided. As I said a lot earlier in this thread, I would find it hard to write such a patch. The particular scenario you encountered seems to me to be relatively rare. It required the following combination of circumstances: 1) You were using the OS's perl installation. 2) You didn't use the OS's packing system to add extra CPAN modules: for example on Fedora to install the Time::ParseDate module, I would install the perl-Time-ParseDate RPM provided by Fedora. 3) You didn't use cpan or similar in a way that installs the extra packages as part of the perl installation which the cpan tool is a part of. 4) You used the third-party local::lib tool install the extra CPAN modules under your home directory, which doesn't install them in version-specific paths. Change any of those conditions and you likely wouldn't have seen that error message (most likely you would instead have seen an error message about not being able to find the module). Some other ways of getting that error message include: * having both a system perl and another perl installed, and an incorrect setting of PERL5LIB or similar causes one perl to pick up modules from paths intended for the other perl. * Or rebuilding the same version of perl, but with different build options. * Or someone thinking they can "install" a module by just copying the *.pm and *.so files from one system to another. All you can really say about that error message, is that a perl interpreter is trying to load an XS module which was built using a different perl interpreter (but not necessarily a different version). -- Spock (or Data) is fired from his high-ranking position for not being able to understand the most basic nuances of about one in three sentences that anyone says to him. -- Things That Never Happen in "Star Trek" #19
To: Dave Mitchell <davem [...] iabyn.com>
Date: Tue, 18 Sep 2018 05:39:15 -0700
Subject: Re: [perl #133440] binaries mismatched again
From: frederik [...] ofb.net
CC: Dave Mitchell via RT <perlbug-followup [...] perl.org>
Download (untitled) / with headers
text/plain 2.7k
Hi Dave, I'm looking over the messages in this thread and I see that you said you would find it difficult to write a manual page about upgrading perl modules, but you also said that you were not opposed to having the error message refer to a manual page, and you suggested adding an extra section in an existing page. Then I submitted a patch along the lines of your suggestions - modifying the error message to refer to a manual page and adding additional material to a section in an existing manual page. Are you still interested in brainstorming about ways to fix this particular issue? If so, then I'm happy to continue modifying what I've done to satisfy the requirements of the situation. Just let me know in what general direction you would like to see this go. Thanks, Frederick On Tue, Sep 18, 2018 at 10:43:27AM +0100, Dave Mitchell wrote: Show quoted text
> On Mon, Sep 17, 2018 at 07:15:55AM -0700, frederik@ofb.net wrote:
> > If it is just that one sentence that's a problem, then it would > > probably be easy to rephrase it in a way that preserves the intent > > while broadening the scope:
> > No, I think the general thrust of the patch is misguided. As I said a lot > earlier in this thread, I would find it hard to write such a patch. > > The particular scenario you encountered seems to me to be relatively rare. > It required the following combination of circumstances: > > 1) You were using the OS's perl installation. > 2) You didn't use the OS's packing system to add extra CPAN modules: > for example on Fedora to install the Time::ParseDate module, > I would install the perl-Time-ParseDate RPM provided by Fedora. > 3) You didn't use cpan or similar in a way that installs the extra packages > as part of the perl installation which the cpan tool is a part of. > 4) You used the third-party local::lib tool install the extra CPAN modules > under your home directory, which doesn't install them in > version-specific paths. > > Change any of those conditions and you likely wouldn't have seen that > error message (most likely you would instead have seen an error message > about not being able to find the module). > > Some other ways of getting that error message include: > * having both a system perl and another perl installed, and an incorrect > setting of PERL5LIB or similar causes one perl to pick up modules from > paths intended for the other perl. > * Or rebuilding the same version of perl, but with different build options. > * Or someone thinking they can "install" a module by just copying the *.pm > and *.so files from one system to another. > > All you can really say about that error message, is that a perl > interpreter is trying to load an XS module which was built using a > different perl interpreter (but not necessarily a different version).
To: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133440] binaries mismatched again
Date: Tue, 18 Sep 2018 18:56:42 +0200
CC: Frederick Eaton <frederik [...] ofb.net>, perlbug <perlbug-followup [...] perl.org>
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 2.3k
On Tue, Sep 18, 2018 at 11:44 AM Dave Mitchell <davem@iabyn.com> wrote: Show quoted text
> On Mon, Sep 17, 2018 at 07:15:55AM -0700, frederik@ofb.net wrote:
> > If it is just that one sentence that's a problem, then it would > > probably be easy to rephrase it in a way that preserves the intent > > while broadening the scope:
> > No, I think the general thrust of the patch is misguided. As I said a lot > earlier in this thread, I would find it hard to write such a patch. > > The particular scenario you encountered seems to me to be relatively rare. > It required the following combination of circumstances: > > 1) You were using the OS's perl installation. > 2) You didn't use the OS's packing system to add extra CPAN modules: > for example on Fedora to install the Time::ParseDate module, > I would install the perl-Time-ParseDate RPM provided by Fedora. > 3) You didn't use cpan or similar in a way that installs the extra packages > as part of the perl installation which the cpan tool is a part of. > 4) You used the third-party local::lib tool install the extra CPAN modules > under your home directory, which doesn't install them in > version-specific paths. > > Change any of those conditions and you likely wouldn't have seen that > error message (most likely you would instead have seen an error message > about not being able to find the module). > > Some other ways of getting that error message include: > * having both a system perl and another perl installed, and an incorrect > setting of PERL5LIB or similar causes one perl to pick up modules from > paths intended for the other perl. > * Or rebuilding the same version of perl, but with different build options. > * Or someone thinking they can "install" a module by just copying the *.pm > and *.so files from one system to another.
Yeah, especially the first option is quite easily triggered in my experience (and the reason I don't mix perlbrew and local::lib). I think it's useful to list various reasons why this can happen. That said, there is one thing about the case Frederik describes that sets it apart from the others: it's a "but it worked yesterday" scenario. One can trigger it without the end-user doing anything perl related (unlike compiling a new and incompatible perl). That probably explains Tomasz' observation that questions on IRC tend to be about this scenario. Leon


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org