Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perl5.18.0 build-config seems to ignore some 64-bit types #13216

Closed
p5pRT opened this issue Aug 30, 2013 · 23 comments
Closed

perl5.18.0 build-config seems to ignore some 64-bit types #13216

p5pRT opened this issue Aug 30, 2013 · 23 comments

Comments

@p5pRT
Copy link

p5pRT commented Aug 30, 2013

Migrated from rt.perl.org#119537 (status was 'resolved')

Searchable as RT119537$

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From perl-diddler@tlinx.org

Created by perl-diddler@tlinx.org

I know that some people manage to get this to build on SuSE, but
there seem to be some fundamental issues that I'm not sure how or why
perl would be configured to use ...

I.e​:

Ishtar​:packages/build/perl-5.18> make |& tee ../../logs/perl518-a.log
`sh cflags "optimize='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -Wall -pipe'" perlmini.o` -fPIC -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
  CCCMD = cc -DPERL_CORE -c -D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -Wall -pipe -Wall
  In file included from perl.c​:33​:0​:
  perl.h​:739​:14​: error​: conflicting types for ‘syscall’
  EXTERN_C int syscall(int, ...);
  ^
  In file included from perl.h​:726​:0,
  from perl.c​:33​:
  /usr/include/unistd.h​:1080​:17​: note​: previous declaration of ‘syscall’ was here
  extern long int syscall (long int __sysno, ...) __THROW;

There were other compile warnings based on unused return values but the above error ends the build. It's *as if* (and this makes it look related, maybe? to previous bug regarding mmap?) perl is using a 32-bit int to hold what should be
a 64-bit quantity on a 64-bit machine.

The build doesn't get any farther than this when run single-job.

The config line I used coming into this was​:

Ishtar​:packages/build/perl-5.18> ./configure.gnu --prefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Dlibswanted="gdbm gdbm_compat" -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=\'true\' $options |& tee perlconf.log

# for options =

-Doptimize='-g -ggdb -O2 -m64 -fasynchronous-unwind-tables -fbranch-target-load-optimize -fdelete-null-pointer-checks -fgcse-after-reload -fgcse-las -fgcse-sm -fgraphite-identity -fipa-pta -fivopts -floop-block -floop-flatten -floop-interchange -floop-strip-mine -flto -fmessage-length=0 -fpredictive-commoning -frename-registers -freorder-blocks-and-partition -ftree-loop-linear -ftree-loop-distribution -ftree-loop-distribute-patterns -ftree-loop-im -ftree-loop-ivcanon -ftree-vectorize -ftree-slp-vectorize -funswitch-loops -funwind-tables -fvariable-expansion-in-unroller -fvect-cost-model -fweb -march=native -Wno-unused-result -pipe' -Accflags='-DPERL_USE_SAFE_PUTENV' -Dotherlibdirs=/usr/lib/perl5/site_perl

Is it just coincidence that this another place (previous (see #119475)) where perl isn't using the libc compat definition of a type in a system call?

Perl Info

Flags:
    category=core
    severity=medium

This perlbug was built using Perl 5.16.2 - Fri Feb 15 01:17:37 UTC 2013
It is being executed now by  Perl 5.16.2 - Fri Feb 15 01:12:05 UTC 2013.

Site configuration information for perl 5.16.2:

Configured by abuild at Fri Feb 15 01:12:05 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=3.4.6-2.10-default, archname=x86_64-linux-thread-multi
    uname='linux build34 3.4.6-2.10-default #1 smp thu jul 26 09:36:26 utc 2012 (641c197) x86_64 x86_64 x86_64 gnulinux '
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
    ccversion='', gccversion='4.7.2 20130108 [gcc-4_7-branch revision 195012]', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
    libpth=/lib64 /usr/lib64 /usr/local/lib64
    libs=-lm -ldl -lcrypt -lpthread
    perllibs=-lm -ldl -lcrypt -lpthread
    libc=/lib64/libc-2.17.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.17'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.16.2/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches:
    


@INC for perl 5.16.2:
    /home/law/bin/lib
    /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.2
    /usr/lib/perl5/vendor_perl/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.16.2
    /usr/lib/perl5/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/5.16.2
    /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.2
    /usr/lib/perl5/site_perl
    .


Environment for perl 5.16.2:
    HOME=/home/law
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH=/usr/lib64/mpi/gcc/openmpi/lib64
    LOGDIR (unset)
    PATH=/home/law/bin/lib:/sbin:/usr/local/sbin:/usr/lib64/mpi/gcc/openmpi/bin:/home/law/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/usr/lib/qt3/bin:/opt/dell/srvadmin/bin:/usr/sbin:/etc/local/func_lib:/home/law/lib
    PERL5OPT=-Mutf8 -CSA -I/home/law/bin/lib
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From @Tux

On Fri, 30 Aug 2013 12​:41​:50 -0700, Linda Walsh (via RT)
<perlbug-followup@​perl.org> wrote​:

I know that some people manage to get this to build on SuSE, but
there seem to be some fundamental issues that I'm not sure how or why
perl would be configured to use ...

Which SUSE? I might want to try to reproduce. I have (devel) acces to
these​:

Linux 2.6.31.14-0.6-desktop [openSUSE 11.2 (Emerald)]
Linux 2.6.34.7-0.7-desktop [openSUSE 11.3 (Teal)]
Linux 2.6.34.10-0.6-desktop [openSUSE 11.3 (Teal)]
Linux 3.1.10-1.29-default [openSUSE 12.1 (Asparagus)]
Linux 3.4.28-2.20-desktop [openSUSE 12.2 (Mantis)]
Linux 3.4.47-2.38-desktop [openSUSE 12.2 (Mantis)]
Linux 3.7.10-1.11-desktop [openSUSE 12.3 (Dartmouth)]
Linux 3.7.10-1.16-desktop [openSUSE 12.3 (Dartmouth)]

13.1 will be released in November, and it already has been tagged
"Evergreen"

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From perl-diddler@tlinx.org

On Fri Aug 30 12​:58​:23 2013, hmbrand wrote​:

Linux 3.7.10-1.11-desktop [openSUSE 12.3 (Dartmouth)]
Linux 3.7.10-1.16-desktop [openSUSE 12.3 (Dartmouth)]


I have factory packages from as late as June 18, installed on top of
my last working 12.3 where factory had newer packages.

kernel info​:

uname -a
Linux Ishtar 3.9.8-Isht-Van #3 SMP PREEMPT Sun Jul 7 20​:43​:00 PDT 2013
x86_64 x86_64 x86_64 GNU/Linux


I usually try to keep up on a more recent vanilla kernel build than suse
provides.

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From @Tux

On Fri, 30 Aug 2013 13​:10​:21 -0700, "Linda Walsh via RT"
<perlbug-followup@​perl.org> wrote​:

On Fri Aug 30 12​:58​:23 2013, hmbrand wrote​:

Linux 3.7.10-1.11-desktop [openSUSE 12.3 (Dartmouth)]
Linux 3.7.10-1.16-desktop [openSUSE 12.3 (Dartmouth)]
----
I have factory packages from as late as June 18, installed on top of
my last working 12.3 where factory had newer packages.

kernel info​:

uname -a
Linux Ishtar 3.9.8-Isht-Van #3 SMP PREEMPT Sun Jul 7 20​:43​:00 PDT 2013
x86_64 x86_64 x86_64 GNU/Linux

---
I usually try to keep up on a more recent vanilla kernel build than suse
provides.

Right, so my Linux 3.7.10-1.11-desktop [openSUSE 12.3 (Dartmouth)]
comes closest. Both are 64bit machines, meaning 64bit builds are default

From scratch, without using my Policy.sh …

$ wget http​://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-5.18.0.tar.bz2
$ tbz -sx perl-5.18.0.tar.bz2
$ cd perl-5.18.0
$ ./Configure -ds -e -Dprefix=/usr -Dvendorprefix=/usr
-Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize="-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe" -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl
:
Now you must run 'make'.

If you compile perl5 on a different machine or from a different object
directory, copy the Policy.sh file from this object directory to the
new one before you run Configure -- this will help you with most of
the policy defaults.
$ make -j6
:
make[1]​: Leaving directory `/pro/3gl/CPAN/perl-5.18.0/x2p'

  Everything is up to date. Type 'make test' to run test suite.
$ env TEST_JOBS=5 make test_harness
:
All tests successful.
Files=2392, Tests=685711, 265 wallclock secs (62.51 usr 14.47 sys + 497.08 cusr 59.64 csys = 633.70 CPU)
Result​: PASS

You had this​:
--8<---
`sh cflags "optimize='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -Wall -pipe'" perlmini.o` -fPIC -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
  CCCMD = cc -DPERL_CORE -c -D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -Wall -pipe -Wall
-->8---

Where is that -m64 coming from? I see in the ticket that you set
$options, but it does not show in config_args from perl -V

And, if you want a 64bit perl, why not do it the way installation
instructions tell you to​: -Duse64bitall
(I personally also add -Duselongdouble)

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From perl-diddler@tlinx.org

On Fri Aug 30 13​:43​:04 2013, hmbrand wrote​:

Right, so my Linux 3.7.10-1.11-desktop [openSUSE 12.3 (Dartmouth)]
comes closest. Both are 64bit machines, meaning 64bit builds are
default

From scratch, without using my Policy.sh …

$ wget http​://cpan.metacpan.org/authors/id/R/RJ/RJBS/perl-
5.18.0.tar.bz2
$ tbz -sx perl-5.18.0.tar.bz2
$ cd perl-5.18.0
$ ./Configure -ds -e -Dprefix=/usr -Dvendorprefix=/usr
-Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm
-Dd_dbm_open -Duseshrplib=true -Doptimize="-fmessage-length=0 -O2
-Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables
-fasynchronous-unwind-tables -g -Wall -pipe"
-Accflags=-DPERL_USE_SAFE_PUTENV
-Dotherlibdirs=/usr/lib/perl5/site_perl
:
Now you must run 'make'.


I know if I don't bend over backwards to get the gdbm and gdbm_compat
libs included, it won't have a chance of compiling in dbm functionality.

If you compile perl5 on a different machine or from a different object
directory, copy the Policy.sh file from this object directory to the
new one before you run Configure -- this will help you with most of
the policy defaults.
$ make -j6
:
make[1]​: Leaving directory `/pro/3gl/CPAN/perl-5.18.0/x2p'


Unfortunately, the perl make (like most "config"-based make processes)
alters itself based on the configuration of your compile machine (most
importantly, what packages and features you have installed).

  Theoretically, if you don't have the pertinent "devel" packages loaded
you shouldn't even be getting as far as you are. So I'm assuming you
have some set of 'devel' rpms loaded with some headers&libs -- but the
presence/absence of many different configurations is dependent on
"which" set of 'devel' packages (headers&libs) you have installed.

Starting in about SuSE 11.4 or 12.0, suse switched over to using
disposable build environments that only include the necessary packages
needed to build the specific package being built. And by necessary
packages ('needs', meaning for 1 running configuration with unspecified
functionality). However, some things, like a running kernel are
assumed, but kernel versions, for example, usually are not.

Same goes for many other "assumed to be present" utils, though the list
of assumed packages has been shrinking as more dependencies are added to
the .spec file.

You had this​:
--8<---
`sh cflags "optimize='-O2 -g -m64 -fmessage-length=0
-D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables
-fasynchronous-unwind-tables -Wall -pipe'" perlmini.o` -fPIC
-DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
CCCMD = cc -DPERL_CORE -c -D_REENTRANT -D_GNU_SOURCE
-DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV
-DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV
-DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV
-DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV
-DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -O2 -g -m64
-fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector
-funwind-tables -fasynchronous-unwind-tables -Wall -pipe -Wall
-->8---

Where is that -m64 coming from? I see in the ticket that you set
$options, but it does not show in config_args from perl -V


The config_args from perl-V is from the perl installed on my system that
is being used to send the report -- which has next to nil to do
with the perl I'm generating... Funny, as I was submitting this I
thought about the incongruity as I submitted this bug -- thinking what
does this config information have to do with the config I'm building with?

I don't think the m64 is necessary and don't remember why it was there
-- a historical carryover from some early 64-bit release when there was
uncertainty about whether the 32 or 64 bit model was begin used as a target.

It was one of the ones I copied from the existing line that was in use
in my /etc/rmprc -- to which I add optimization options above and
beyond the default of O2, but < O3 to try to add in all options that may
take more compile time, but generally don't add codesize.

... ah , the m64 option comes, originally, from
the default rpmrc configuration in /usr/lib/rpm/rpmrc
Where the default x86_64 optflags =
optflags​: x86_64 -O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2
-fstack-protector -funwind-tables -fasynchronous-unwind-tables

I have added to that over the years, and my opt flags looks like​:

optflags​: -g -ggdb -O2 -m64 -fasynchronous-unwind-tables \
  -fbranch-target-load-optimize -fdelete-null-pointer-checks \
  -fgcse-after-reload -fgcse-las -fgcse-sm -fgraphite-identity \
  -fipa-pta -fivopts -floop-block -floop-flatten \
  -floop-interchange -floop-strip-mine -flto -fmessage-length=0 \
  -fpredictive-commoning -frename-registers \
  -freorder-blocks-and-partition -ftree-loop-linear \
  -ftree-loop-distribution -ftree-loop-distribute-patterns \
  -ftree-loop-im -ftree-loop-ivcanon -ftree-vectorize \
  -ftree-slp-vectorize -funswitch-loops -funwind-tables \
  -fvariable-expansion-in-unroller -fvect-cost-model \
  -fweb -march=native

And, if you want a 64bit perl, why not do it the way installation
instructions tell you to​: -Duse64bitall
(I personally also add -Duselongdouble)


I would too!
I'm trying to use a build based on the suse .spec file, since I know
that not using the spec file will fail, in at least the DB area.

So I am trying to do as "SuSE suggests -- and trying to build based on
the .spec file"....

That said, when I build only using the options and invocation method as
listed by you in this bug report,
  *it builds* (gets farther than with suse spec file)
but does not past the make test phase.

The tests fail in the dbm related tests
*AND*
fail in some CPAN related tests in the same way that
"invalid closed"[sic] bug 119493 described in trying to access cpan.

I am attaching the suse perl spec file I was using (done an rpmbuild
-bp) to unpack it into a dir and was doing the build by-hand to more
closely watch each step. Also enclosing the test output.

Can provide the output from configure as well if that is of usefulness...

Am going to look at why there is a difference in the 'make' for the suse
perl spec file and options I used and see if that uncovers anything.

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From perl-diddler@tlinx.org

perl.spec

@p5pRT
Copy link
Author

p5pRT commented Aug 30, 2013

From perl-diddler@tlinx.org

p18test.log

@p5pRT
Copy link
Author

p5pRT commented Aug 31, 2013

From @doughera88

On Fri, Aug 30, 2013 at 12​:41​:50PM -0700, Linda Walsh wrote​:

# New Ticket Created by Linda Walsh
# Please include the string​: [perl #119537]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=119537 >

This is a bug report for perl from perl-diddler@​tlinx.org,
generated with the help of perlbug 1.39 running under perl 5.16.2.

-----------------------------------------------------------------
[Please describe your issue here]

I know that some people manage to get this to build on SuSE, but
there seem to be some fundamental issues that I'm not sure how or why
perl would be configured to use ...

I.e​:

Ishtar​:packages/build/perl-5.18> make |& tee ../../logs/perl518-a.log
`sh cflags "optimize='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -Wall -pipe'" perlmini.o` -fPIC -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
CCCMD = cc -DPERL_CORE -c -D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -DPERL_USE_SAFE_PUTENV -O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -Wall -pipe -Wall
In file included from perl.c​:33​:0​:
perl.h​:739​:14​: error​: conflicting types for ‘syscall’
EXTERN_C int syscall(int, ...);
^
In file included from perl.h​:726​:0,
from perl.c​:33​:
/usr/include/unistd.h​:1080​:17​: note​: previous declaration of ‘syscall’ was here
extern long int syscall (long int __sysno, ...) __THROW;

This particular problem likely arose because Configure couldn't figure
out that your system had a valid syscall prototype. I don't know why
Configure had that problem, and I have been unable to reproduce it.

The fact that -DPERL_USE_SAFE_PUTENV appears 10 times suggests that
this might not have been a clean build from a fresh source tree.
The fact that a variety of '-floop-*' options appear in your configure
command below, but not in the command above suggest to me that the
command above was not generated from the configure command below.

In short, I can't reproduce this and really don't understand how you got
there either. It would not surprise me if there were a problem with
some of the Configure prototype tests under some unusual conditions,
but I can't guess precisely what those conditions might be, and you
haven't given us enough information for us to determine that either.

I recommend the following​:

1. Start with a clean source tree.

2. Figure out the *minimal* Configure command line necessary to
reproduce the problem. Remove config.sh and Policy.sh between each
trial.

3. Post the *minimal* Configure command line, the *output* of the
./myconfig script, and the output of 'grep syscall config.sh'. That
might help us reproduce or diagnose the problem.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented Sep 1, 2013

From perl-diddler@tlinx.org

On Fri Aug 30 18​:45​:24 2013, doughera wrote​:

This particular problem likely arose because Configure couldn't figure
out that your system had a valid syscall prototype. I don't know why
Configure had that problem, and I have been unable to reproduce it.

The fact that -DPERL_USE_SAFE_PUTENV appears 10 times suggests that
this might not have been a clean build from a fresh source tree.


That might have had to do with some excess merging of ENV var that
contained that with the ENV-options string, though never have seen that
many repetitions.

The fact that a variety of '-floop-*' options appear in your configure
command below, but not in the command above suggest to me that the
command above was not generated from the configure command below.

In short, I can't reproduce this and really don't understand how you
got
there either. It would not surprise me if there were a problem with
some of the Configure prototype tests under some unusual conditions,
but I can't guess precisely what those conditions might be, and you
haven't given us enough information for us to determine that either.

I recommend the following​:

1. Start with a clean source tree.


Everything after the 1st build was from a fresh tree.

I.e when Merijin showed their sequence of untaring and building,
I just went to my work directory and untar'ed the tar image there.
Vs. trying to get it to work in the "package" directory where I'm using
a SuSE rpm .spec file that they would be using, but likely in a freshly
formatted build-container.

2. Figure out the *minimal* Configure command line necessary to
reproduce the problem. Remove config.sh and Policy.sh between each
trial.


  I wasn't sure of what the minimal files to remove were, and
untarring the files take about 2 seconds, so was easier to go that
route. Have tried about 5 different ways and seem to get further
in my tools dir (building from tar) vs. package(bld from spec).

I also tried to build from the spec file by removing various patches
that are included, but I think I'm doing as much harm as good in doing
that (temporary harm, in that I'm commenting out patch lines that I
can comment back in).

Of the ones built from tar, as I mentioned in response to merijin, I was
able to get those to build but not test -- with the test results
included in this report. The failure of the DBtests has been a
persistent failure in perl tests for at least a year or more at this
point (suse's answer build​: from rpm on our build machines; dont' expect
to use an installed suse machine for making builds...we don't support it).

The CPAN test failures seen in the test output are relatively new and
first appeared in "invalid" bug 119493 where I mentioned it with other
problems in 5.18.

3. Post the *minimal* Configure command line, the *output* of the
./myconfig script, and the output of 'grep syscall config.sh'. That
might help us reproduce or diagnose the problem.


My last built with the rpm fails here​:

/var/tmp/rpm-tmp.RUgHm8#50> read f
/var/tmp/rpm-tmp.RUgHm8#51>
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/bin/perl
-I/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/5.18.0
-I/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/5.18.0/x86_64-linux-thread-multi
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/bin/h2ph -d
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/vendor_perl/5.18.0/x86_64-linux-thread-multi
linux/map_to_7segment.h
linux/map_to_7segment.h -> linux/map_to_7segment.ph
/var/tmp/rpm-tmp.RUgHm8#50> read f
/var/tmp/rpm-tmp.RUgHm8#53> popd
/usr/src/packages/BUILD/perl-5.18

/var/tmp/rpm-tmp.RUgHm8#54> gcc -print-file-name=include
/var/tmp/rpm-tmp.RUgHm8#54> d=/usr/lib64/gcc/x86_64-suse-linux/4.8/include
/var/tmp/rpm-tmp.RUgHm8#55> test -f
/usr/lib64/gcc/x86_64-suse-linux/4.8/include/stdarg.h
/var/tmp/rpm-tmp.RUgHm8#55> cd
/usr/lib64/gcc/x86_64-suse-linux/4.8/include
/var/tmp/rpm-tmp.RUgHm8#55>
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/bin/perl
-I/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/5.18.0
-I/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/5.18.0/x86_64-linux-thread-multi
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/bin/h2ph -d
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/vendor_perl/5.18.0/x86_64-linux-thread-multi
stdarg.h stddef.h float.h
stdarg.h -> stdarg.ph
stddef.h -> stddef.ph
float.h -> float.ph
/var/tmp/rpm-tmp.RUgHm8#57> rm
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/lib/perl5/5.18.0/Pod/Perldoc/ToTk.pm
/var/tmp/rpm-tmp.RUgHm8#63>
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/bin/perl -e
'$r=chr(128)."\\x{100}";/$r/'
/var/tmp/rpm-tmp.RUgHm8#65>
/usr/src/packages/BUILDROOT/perl-5.18-0.0.x86_64/usr/bin/perl -e
'/\6666666666/'
/var/tmp/rpm-tmp.RUgHm8​: line 65​: 17540 Segmentation fault (core
dumped) $RPM_BUILD_ROOT/usr/bin/perl -e '/\6666666666/'
error​: Bad exit status from /var/tmp/rpm-tmp.RUgHm8 (%install)

RPM build errors​:
  Bad exit status from /var/tmp/rpm-tmp.RUgHm8 (%install)


more myconfig
#!/bin/sh

# This script is designed to provide a handy summary of the configuration
# information being used to build perl. This is especially useful if you
# are requesting help from comp.lang.perl.misc on usenet or via mail.

# Note that the text lines /^Summary of/ .. /^\s*$/ are copied into
Config.pm.
cat <<'!NO!SUBS!'
Summary of my perl5 (revision 5 version 18 subversion 0) configuration​:
 
  Platform​:
  osname=linux, osvers=3.9.11-isht-van, archname=x86_64-linux-thread-multi
  uname='linux ishtar 3.9.11-isht-van #2 smp preempt sat aug 31
15​:25​:04 pdt 2013 x86_64 x86_64 x86_64 gnulinux '
  config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr
-Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm
-Dd_dbm_open -Doptimize=-O2 -g -m64 -fmessage-length=0
-D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables
-fasynchronous-unwind-tables -pipe -Accflags=-DPERL_USE_SAFE_PUTENV
-Dotherlibdirs=/usr/lib/perl5/site_perl'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV
-fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2
-fstack-protector -funwind-tables -fasynchronous-unwind-tables -pipe',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV
-fno-strict-aliasing -pipe -fstack-protector'
  ccversion='', gccversion='4.8.0', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
  libpth=/lib64 /usr/lib64 /usr/local/lib64
  libs=-lm -ldl -lcrypt -lpthread
  perllibs=-lm -ldl -lcrypt -lpthread
  libc=/lib64/libc-2.17.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.17'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64
-fstack-protector'

!NO!SUBS!


I don't know if it is worth adding in the grep for syscall stuff at this
point, as this is failing in a different way -- perhaps something was
getting re-used in the old build dir that wasn't compat with current
headers -- but the above build and a few others w/rpm and about 5-6 from
the tar, were all done in clean directories (freshly untar'ed)...

I can forget about the ".rpm" not building -- I used to build from tar
images -- only after problems starting with perl5.16 under suse 12.0 and
later, that I first tried to address through the suse email lists,
did I start trying to make it work with rpm-- as the suse folks
recommended starting with their rpm's (double edged sword).

The build w/o the rpm -- fresh untar gets the test results already
attached (fails in db & CPAN)...

FWIW, I find it hard to get a reliable, reproducible, controlled build,
as I'm used to specifying all of the configure options in a shell script
and calling it from there, but most of the options aren't documented as
to how to call them from a shell script.

i.e. for squid, for example (with majority lines elided).

#!/bin/bash
pkg=squid
. /home/tools/config.include
...
CFLAGS="-g -m64 -fPIC -fmessage-length=0 -O2 -march=native"
...
  CFLAGS="$CFLAGS" CCFLAGS="$CCFLAGS" \
  LDFLAGS="$LDFLAGS" configure \
...
  --with-logdir=/var/log/$pkg \
...
  --enable-kill-parent-hack \
  --enable-linux-netfilter \
  --enable-ltdl-install \
...
  --enable-ssl


With perl, I wasn't sure where, but I noticed it kept state info, so I
wrote a script to rebuild it from the tar image, fresh each time --
that automatically unpacks the latest tar image in my tools/perl
dir...


I.e. usually I start with a clean perl-dir each build, because it
doesn't run the same way each time unless I start afresh (due to
the various state files it keeps around).

----Just so you know, I'm not kidding ---
Ishtar​:tools/perl> more remake_perl_from_tar
#!/bin/bash
shopt -s extglob
if (($#<1)); then
  declare -a s=( $(echo perl-5.??.[0-9].t*@​(gz|bz2|xz|ar) ) )
  declare -i numsrc=${#s[*]}
  if ((numsrc>1)); then
  echo "Found more than one perl source; must specify which to use
(5.16*, etc)"
  exit 1
  elif ((numsrc==0)); then
  echo "Found no source to build (perl-5.<xx>.x.t*<something>)"
  exit 2
  fi
  src="${s[0]}"
else
  src="$1";q=1
  if [[ ! -f $src ]]; then
  tf="$(echo perl-$src.t* )"
  if [[ -f $tf ]]; then
  q=0
  src=$tf
  else #try liberal
  tf="$(echo perl*$src*)"
  src="$tf"; q=1 #question
  fi
  fi
fi
if [[ ! -f $src || ! -r $src ]]; then
  echo "Can't read source from $src"
  exit 3
fi
if [[ -f $src ]]; then
  type="$(file -z --mime-type "$src"|cut -d\ -f2)"
  type=${type#application/}
  if [[ $type != x-tar ]]; then
  echo "File $src doesn't seem to be right type."
  echo "Expecting x-tar, found \"$type\""
  exit 4
  fi
fi
if ((q)); then
  echo "Had to be a little liberal w/your version#..."
  echo "Using $src as the source file? Is that ok? (y/n)"
  ans=""
  while [[ $ans != "y" ]]; do
  read -p "Using $src as the source file? Is that ok? (y/n) " ans
  if [[ $ans == "n" ]]; then exit 5; fi
  done
fi
dir=${src%%.t*}
if [[ -d "$dir" ]]; then
  echo "rm -fr $dir"
  sleep .5
  rm -fr "$dir"
fi
tar xaf "$src" || {
  echo "tar exited with status $?; stopping"
}

dir=${src%%.t*}
if [[ ! -d "$dir" ]]; then
  echo "don't know what directory to start in"
  echo "I thought it was \"$dir\""
fi

log=/tmp/makeperl-$$.log

{
  cd "$dir" && {
  sh Configure -de &&
  make &&
  make test
  status=$?
  }

  if (($status==0)) ; then echo "perl make & tested ok"; exit 0; fi

  echo "Bad status \"$status\" on exit"
  echo "Make NOT OKAY!"
} >& "$log" &

echo "Make is running. To monitor the log do​:"
echo "tail -f \"$log\""


So how should I move forward -- the DB problem has been a thorn for
some time. The CPAN test fails in the test phase are new w/5.18.

Thanks! ;-)

 

@p5pRT
Copy link
Author

p5pRT commented Sep 1, 2013

From @Tux

On Sat, 31 Aug 2013 21​:50​:44 -0700, "Linda Walsh via RT"
<perlbug-followup@​perl.org> wrote​:

Of the ones built from tar, as I mentioned in response to merijin, I was
able to get those to build but not test -- with the test results
included in this report. The failure of the DBtests has been a
persistent failure in perl tests for at least a year or more at this
point (suse's answer build​: from rpm on our build machines; dont' expect
to use an installed suse machine for making builds...we don't support it).

Shall I call you Lindia?

Does this help?

--8<--- INSTALL
=item error​: too few arguments to function 'dbmclose'

Building ODBM_File on some (Open)SUSE distributions might run into this
error, as the header file is broken. There are two ways to deal with
this

1. Disable the use of ODBM_FILE

  Configure ... -Dnoextensions=ODBM_File

2. Fix the header file, somewhat like this​:

  --- a/usr/include/dbm.h 2010-03-24 08​:54​:59.000000000 +0100
  +++ b/usr/include/dbm.h 2010-03-24 08​:55​:15.000000000 +0100
  @​@​ -59,4 +59,4 @​@​ extern datum firstkey __P((void));

  extern datum nextkey __P((datum key));

  -extern int dbmclose __P((DBM *));
  +extern int dbmclose __P((void));
-->8---

FWIW my 12.3 has

$ grep dbmclose /usr/include/dbm.h
extern int dbmclose (void);
$ rpm -qf /usr/include/dbm.h
gdbm-devel-1.10-2.2.1.i586

I can forget about the ".rpm" not building -- I used to build from tar
images -- only after problems starting with perl5.16 under suse 12.0 and
later, that I first tried to address through the suse email lists,
did I start trying to make it work with rpm-- as the suse folks
recommended starting with their rpm's (double edged sword).

The build w/o the rpm -- fresh untar gets the test results already
attached (fails in db & CPAN)...

FWIW, I find it hard to get a reliable, reproducible, controlled build,
as I'm used to specifying all of the configure options in a shell script
and calling it from there, but most of the options aren't documented as
to how to call them from a shell script.

I read the mail trice and still did not see if you tried the "minimal"
Configure as Andy and I suggested.

unpack from tar, then e.g.

$ ./Configure -Duse64bitall -des

The general top-level script you are looking for to make "reliable"
builds in *your* environment is called Policy.sh, and it is described
in INSTALL

--8<--- INSTALL
=head2 Site-wide Policy settings

After Configure runs, it stores a number of common site-wide "policy"
answers (such as installation directories) in the Policy.sh file.
If you want to build perl on another system using the same policy
defaults, simply copy the Policy.sh file to the new system's perl build
directory, and Configure will use it. This will work even if Policy.sh was
generated for another version of Perl, or on a system with a
different architecture and/or operating system. However, in such cases,
you should review the contents of the file before using it​: for
example, your new target may not keep its man pages in the same place
as the system on which the file was generated.

Alternatively, if you wish to change some or all of those policy
answers, you should

  rm -f Policy.sh

to ensure that Configure doesn't re-use them.

Further information is in the Policy_sh.SH file itself.

If the generated Policy.sh file is unsuitable, you may freely edit it
to contain any valid shell commands. It will be run just after the
platform-specific hints files.
-->8---

As an example, I'll share my stripped down Policy.sh which I use
across OS's. I use it like this​:

untar source
$ cd perl-5.18.0
$ cp ../Policy.sh .
$ ./Configure ...

My install location is /pro

--8<--- Policy.sh
#!/usr/bin/sh

# put this file in the perl source tree and run 'Configure -ds'

man1dir=/pro/local/man/man1
man3dir=/pro/local/man/man3

case "$versiononly" in
  '') versiononly=$undef
  if [ -x /pro/bin/perl ]; then
  if [ `/pro/bin/perl -e '$v="5";while(<>){m/#define\s+PERL_(?​:SUB)?VERSION\s+(\d+)/and$v.=".$1"}$v=~s/\.(.)\./.00\1/;print STDERR "$v <=> $]\n";$]<$v?1​:0' $src/patchlevel.h` ]; then
  versiononly=$define
  installusrbinperl='undef'
  fi
  fi
  ;;
  esac
prefix=/pro
case "$cc" in
  '') cc=cc ;;
  esac
cc="ccache $cc"
perladmin='my-email-address'
cf_email='my-email-address'

[ "X$OSTYPE" = "X" ] && OSTYPE="$osname"
echo "===== PROCURA Policy for $OSTYPE/$cc ========================">&2

# The -DDEBUGGING part will be stripped out optionally later
# It is easier to strip than to add
ccflags='-fPIC -DDEBUGGING'

extras='none'
inc_version_list='none'
noextensions=ODBM_File

libsdirs=' /lib /pro/local/lib'
ldflags='-L/pro/local/lib'

locincpth='/pro/local/include'
loclibpth='/pro/local/lib'

awk='gawk'
csh='tcsh'

case "$OSTYPE" in
  aix)
  case "$cc" in
  *gcc*) if [ `uname -r` = 2 ]; then
  : AIX 4.2 does not support these options
  else
  if [ "X$use64bitall" = "X" ]; then
  ccflags="-maix32 $ccflags"
  else
  ccflags="-maix64 $ccflags"
  fi
  fi
  ;;
  *) if [ "$useithreads" = "define" ]; then
  cc=xlc_r
  else
  cc=xlc
  fi
  #optimize='-O2'
  ;;
  esac
  ;;

  hpux)
  case "$cc" in
  *gcc*) if [ "X$use64bitint" = "X" -a "X$use64bitall" = "X" ]; then
  true
  else
  cc=gcc64
  ldflags=''
  fi
  case `uname -r` in
  B.10.20) ccflags="-mpa-risc-1-1 $ccflags" ;;
  *) ccflags="-mpa-risc-2-0 $ccflags" ;;
  esac
  echo " Using" `which $cc` >&4
  echo 'int main(){long l;printf("%d\\n",sizeof(l));}'>try.c
  $cc -o try $ccflags $ldflags try.c
  if [ "`try`" = "8" ]; then
  echo "Your C compiler ($cc) is 64 bit native!" >&4
  PATH=/usr/local/pa20_64/bin​:$PATH
  locincpth=' /usr/local/pa20_64/include'
  libsdirs=' /usr/local/pa20_64/lib /usr/lib/pa20_64'
  libdirs=' /usr/local/pa20_64/lib /usr/lib/pa20_64'
  loclibpth='/usr/local/pa20_64/lib'
  ldflags=''
  fi
  rm -f try try.c
  ;;
  *) case `uname -r` in
  B.10.20) ccflags="$ccflags +DAportable" ;;
  esac
  # -fast = +O3 +Onolooptransform +Olibcalls +FPD +Oentrysched +Ofastaccess'
  # optimize='+O2 +Onolooptransform +Olibcalls +FPD +Onolimit'

  # For Oracle, will fail without perlio in threads
  # 5.8.0 and up have useperlio in default
  ;;
  esac
  # For Oracle
  libswanted="cl pthread $libswanted"
  ;;

  osf1)
  ccflags="$ccflags -std -D_INTRINSICS -D_INLINE_INTRINSICS"
  optimize='-O2'
  useshrplib='false'
  ;;

  linux)
  awk='awk'
  _awk='/usr/bin/awk'
  full_awk='/usr/bin/awk'
  csh='tcsh'
  _csh='/usr/bin/tcsh'
  full_csh='/usr/bin/tcsh'
  ;;
  esac
-->8---

Add as many OS's as you support this way (I stripped a few).

A Policy.sh should be written to be ably to create reproducible builds.
ENV vars and options often do not mix very well with existing leftovers
from previous builds.

So, the way to go for you I guess is to create a Policy.sh which holds
all the things you do want to deviate from the default Configure
settings, getting to a point where it will always fail. Then post that
Policy.sh here and get us to see where things go wrong

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.19 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Sep 1, 2013

From @doughera88

On Sat, Aug 31, 2013 at 09​:50​:44PM -0700, "Linda Walsh via RT" wrote​:

On Fri Aug 30 18​:45​:24 2013, doughera wrote​:

I recommend the following​:

1. Start with a clean source tree.
2. Figure out the *minimal* Configure command line necessary to
reproduce the problem.

Have tried about 5 different ways and seem to get further
in my tools dir (building from tar) vs. package(bld from spec).

I don't need to hear about 5 ways. I need to see a single way, from a
clean perl-5.18.0 distribution, that I can try to reproduce. I don't
want to see the failure of an rpm build. I don't use rpm.

[lots of irrelevant lines deleted.]

So how should I move forward -- the DB problem has been a thorn for
some time. The CPAN test fails in the test phase are new w/5.18.

You move forward exactly as I said in my previous email.

First, you provide us with the *minimal* Configure command line
necessary to reproduce the problem in a clean perl directory.

Second, you provide the output of the ./myconfig script from that
directory.

Third, you provide the exact error messages or test failures that
result.

Only then can someone else hope to understand your problem, try to
reproduce it, and possibly even suggest ways to fix it.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented Sep 2, 2013

From perl-diddler@tlinx.org

On Sun Sep 01 15​:16​:33 2013, doughera wrote​:

First, you provide us with the *minimal* Configure command line
necessary to reproduce the problem in a clean perl directory.


Going from tar and Configure, the problem I see now has to do with the
DB feature; the dir where the source is built is recreated at the start
of my make.

I had been using Configure -da, but Merijn suggested​:
Configure -Duse64bitall -des, so altered my script to call that.

In answer to Merijn, RE, the suse-specific info provided​:
I checked the ".h" file indicated and it was already
prototyped as having no params(void)​:

grep dbmclose /usr/include/dbm.h
extern int dbmclose (void);

which, unfortunately indicates that the suse-specific problem mentioned,
is not likely a contributing cause to anything I'm seeing.

Second, you provide the output of the ./myconfig script from that
directory.


myconfig script attached.

Third, you provide the exact error messages or test failures that
result.


Among others​:

lib/AnyDBM_File ............................................... #
Failed test
'Tie'
# at ../lib/AnyDBM_File.t line 25.
FAILED at test 1
....
Failed 10 tests out of 2254, 99.56% okay.
  ../cpan/Memoize/t/errors.t
  ../cpan/Memoize/t/tie_gdbm.t
  ../cpan/Memoize/t/tie_ndbm.t
  ../ext/GDBM_File/t/fatal.t
  ../ext/GDBM_File/t/gdbm.t
  ../ext/NDBM_File/t/ndbm.t
  ../ext/ODBM_File/t/odbm.t
  ../lib/AnyDBM_File.t
  op/coreamp.t
  op/dbm.t
....
u=12.15 s=6.23 cu=402.92 cs=75.09 scripts=2254 tests=674295
make​: *** [test] Error 1
Bad status "2" on exit
Make NOT OKAY!


Complete log attached in makeperl-26754.log

@p5pRT
Copy link
Author

p5pRT commented Sep 2, 2013

From perl-diddler@tlinx.org

myconfig

@p5pRT
Copy link
Author

p5pRT commented Sep 2, 2013

@p5pRT
Copy link
Author

p5pRT commented Sep 3, 2013

From @doughera88

On Sun Sep 01 17​:41​:19 2013, LAWalsh wrote​:

On Sun Sep 01 15​:16​:33 2013, doughera wrote​:

First, you provide us with the *minimal* Configure command line
necessary to reproduce the problem in a clean perl directory.
----
Going from tar and Configure, the problem I see now has to do with the
DB feature; the dir where the source is built is recreated at the start
of my make.

Ok. This is certainly different from the problem originally reported
(prototype of syscall). Let's go forward from here.

I had been using Configure -da, but Merijn suggested​:
Configure -Duse64bitall -des, so altered my script to call that.

Good. (On amd64, -Duse64bitall doesn't end up doing anything, but it's
harmless.)

Second, you provide the output of the ./myconfig script from that
directory.
----
myconfig script attached.

Ok. That looks pretty normal.

Third, you provide the exact error messages or test failures that
result.
---

Among others​:

lib/AnyDBM_File ............................................... #
Failed test
'Tie'
# at ../lib/AnyDBM_File.t line 25.
FAILED at test 1
....
Failed 10 tests out of 2254, 99.56% okay.
../cpan/Memoize/t/errors.t
../cpan/Memoize/t/tie_gdbm.t
../cpan/Memoize/t/tie_ndbm.t
../ext/GDBM_File/t/fatal.t
../ext/GDBM_File/t/gdbm.t
../ext/NDBM_File/t/ndbm.t
../ext/ODBM_File/t/odbm.t
../lib/AnyDBM_File.t
op/coreamp.t
op/dbm.t
....
u=12.15 s=6.23 cu=402.92 cs=75.09 scripts=2254 tests=674295
make​: *** [test] Error 1
Bad status "2" on exit
Make NOT OKAY!
---------------
Complete log attached in makeperl-26754.log

Thank you. Now let's focus on the gdbm-related errors. It looks like
there might be an issue with your gdbm library. Here are a few things
to test​:

1. in the perl-5.18.0 directory, try simply running

  ./perl -Ilib -MGDBM_File -e 1;

  to make sure you can load the module.

2. If that works, try a simple test of the module, e.g. something
  like (note -- I don't use GDBM, so these examples are all just
  written from the man pages. Nevertheless, they seem to work
  for me.)

#!./perl
use warnings;
use strict;
use lib 'lib';
use GDBM_File;
my (%hash, %check);
my ($key, $val);
my $filename = "tied.gdbm";
tie %hash, 'GDBM_File', $filename, &GDBM_WRCREAT, 0644;
$hash{'firstkey'} = 'firstvalue';
$hash{'lastkey'} = 'lastvalue';
untie %hash;

tie %check, 'GDBM_File', $filename, &GDBM_READER, 0644;
while (($key, $val) = each(%check)) {
  print "key = $key, val = $val\n";
}
untie %check;
unlink $filename;

3. If either #1 or #2 fails, you can also check the integrity of your
gdbm library from a pure C test, something like this​:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <gdbm.h>
#include <errno.h>

int main (int argc, char **argv)
{
  GDBM_FILE dbf;
  datum key = { "somekey", 8 };
  datum value = { "somevalue", 8 };
  int ret;
  datum content;
  errno = 0;
  printf("gdbm_version = %s\n", gdbm_version);
  dbf = gdbm_open ("trial.gdbm", 0, GDBM_NEWDB, 0644, 0);
  if (!dbf) {
  printf("%s, errno = %d, gdbm_errno = %d\n",
  "Fatal error opening trial.gdbm", errno, gdbm_errno);
  exit(1);
  }
  ret = gdbm_store (dbf, key, value, GDBM_INSERT);
  if (ret != -1) {
  printf("Successfully stored key.\n");
  }
  else {
  printf("%s ret = %d, errno = %d, gdbm_errno = %d\n",
  "Failed to store key.", ret, errno, gdbm_errno);
  }
  content = gdbm_fetch(dbf, key);
  if (content.dptr && strncmp(content.dptr, value.dptr, 7) == 0) {
  printf("Successfully fetched key.\n");
  }
  else {
  printf("%s errno = %d, gdbm_errno = %d\n",
  "Failed to retrieve key.", errno, gdbm_errno);
  }
  gdbm_close(dbf);
  unlink("trial.gdbm");
  printf ("done.\n");
  return 0;
}

My suspicion is that there's something wrong with your gdbm
installation. If the C test works, but the perl tests fail, and they
are both using the same libgdbm.so installed shared library, then we
have a real puzzle on our hands. I'm hoping it's something simple.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From perl-diddler@tlinx.org

On Tue Sep 03 06​:44​:36 2013, doughera wrote​:

Ok. This is certainly different from the problem originally reported
(prototype of syscall). Let's go forward from here.


  I think that one might be a case where perl doesn't check or
clean ENV vars that might strongly affect how perl generates itself.

  Ideally, at some point, it would be useful, in order to be able
to know all of the factors that go into a perl build, to make sure
and ENV vars that could be used on a platform are set to standard or
default values while perl is building. Otherwise you end up with
something that is difficult to reproduce depending on how you login and
where.

But that isn't what's going on here, at this point.

... shortcutting to the problem that the "C" program you provided
very *wonderfully* reproduces. You left the block size @​ "0"
in the gdbm_open routing​:
  dbf = gdbm_open ("trial.gdbm", 0, GDBM_NEWDB, 0644, 0);

  *IF* gdbm was working correctly then this would be a case of using a
documented feature. However, as some of you may remember, I also tried
to get the wording in the perlrun(stat) man page changed to reflect that
the blksize returned *wasn't* the size of the blocks returned by the
same call.
1st problem is that the gdbm lib, in the absence of the actual blksize
struct member, uses 1024 as a default for the number of blocks,
which is documented in the manpage as :
  blkcnt_t st_blocks; /* number of 512B blocks allocated */

2nd problem) As the code assumes that blksize the size of a block on
disk, (vs. returning suggested I/O size to use when reading/writing to
that file on that device).

2a) Somewhere it also makes the assumption that the optimal I/O size
will always be a power of 2. For a RAID, the optimal is the strip-size
* the strip-width. On the disk I am doing most of my dev work on,
that's 64k * (12-data groups) or 768K bytes which is NOT a power of 2.
That causes failure.

==========================
The source of the bug is in gdbm, last modified in 2011. A workaround
for this situation is not leave the block-size field as '0' (asking
it to be filled in), but set it to a value that will allow the test
suite to pass. Values that I've tested in the "C" program you include​:
4K, 64K, 512K, 1M -- what doesn't work​: 768K (the ideal I/O size for the
device).

To protect against buggy gdbm (1.10 - 1.8.3 have the problem)
implementations, you might want to specify a record size where you can
control that it is a power of 2. Yeah, we can hope the gdbm maintainers
will fix this soon, and wait for that to percolate out to the distros,
but meanwhile, IMO, perl should protect itself so it's build/config
process isn't affected by transient bugs (or ENV vars).

My suspicion is that there's something wrong with your gdbm
installation.


Yup -- and likely nearly 100% of others, as well, from 1.8.3-1.10,
which won't trigger until you try to create a DB on a disk subsystem
that comes back with a non-power-of-2 strip size. Not common, but
certainly far from impossible.

While I can probably, minimally, put in a hack on my machine, that won't
protect anyone else using the feature on a machine that has an ideal
'ioblocksize' that isn't divisible by 2.

This likely explains various problems I've had with Mail​::Spamassassin
-- most likely after an upgrade, where it complains about the data base
format and claims it cannot open it.

It's made worse by the rpm that suse installs tries to put spamassassin
in some dir under /var and I have to move it to a larger/faster disk
(=>/home/spamassassin), where that "ioblksize" changes. I didn't know
gdbm was sensitive to that optimal IO size, nor that it might become
upset if it was changed on a given DB (due to being relocated to a
different disk).

So need to figure out where gdbmopen is called in perl and made sure any
values used there are a power of 2, and in my case, >=64k (as 1 strip
unit = that and that's the minimum size that can be updated on this RAID50.

Hopefully that will let it build & pass testing. But maybe such a
workaround belongs in perl as well, given how long the bug has been in
gdbm and no one has run into it?

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From @doughera88

On Wed, Sep 04, 2013 at 09​:57​:46PM -0700, "Linda Walsh via RT" wrote​:

On Tue Sep 03 06​:44​:36 2013, doughera wrote​:

My suspicion is that there's something wrong with your gdbm
installation.

The source of the bug is in gdbm, last modified in 2011. A workaround
for this situation is not leave the block-size field as '0' (asking
it to be filled in), but set it to a value that will allow the test
suite to pass. Values that I've tested in the "C" program you include​:
4K, 64K, 512K, 1M -- what doesn't work​: 768K (the ideal I/O size for the
device).

Good detective work. Thank you.

To protect against buggy gdbm (1.10 - 1.8.3 have the problem)
implementations, you might want to specify a record size where you can
control that it is a power of 2. Yeah, we can hope the gdbm maintainers
will fix this soon, and wait for that to percolate out to the distros,
but meanwhile, IMO, perl should protect itself so it's build/config
process isn't affected by transient bugs (or ENV vars).

So need to figure out where gdbmopen is called in perl and made sure any
values used there are a power of 2, and in my case, >=64k (as 1 strip
unit = that and that's the minimum size that can be updated on this RAID50.

Hopefully that will let it build & pass testing. But maybe such a
workaround belongs in perl as well, given how long the bug has been in
gdbm and no one has run into it?

The quick workaround for you is to edit line 26 in
ext/GDBM_FILE/GDBM_FILE.xs which currently reads

#define GDBM_BLOCKSIZE 0 /* gdbm defaults to stat blocksize */

and change the value of '0' to something that will work well for you.

I agree that although the bug is in gdbm, perl ought to try to work around
it, or at least make it easier for you to work around it. Unfortunately,
we can't just stat() the file, since it doesn't exist yet. The workaround
will have to be a little more involved. I will open up a separate bug
in RT for this, but I don't exepct to get to it anytime very soon.

Meanwhile, would you be willing to report it to the gdbm maintainers?

Thank you for the clear diagnosis.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From @doughera88

On Wed, Sep 04, 2013 at 09​:57​:46PM -0700, "Linda Walsh via RT" wrote​:

On Tue Sep 03 06​:44​:36 2013, doughera wrote​:

Ok. This is certainly different from the problem originally reported
(prototype of syscall). Let's go forward from here.
----
I think that one might be a case where perl doesn't check or
clean ENV vars that might strongly affect how perl generates itself.

Ideally\, at some point\, it would be useful\, in order to be able

to know all of the factors that go into a perl build, to make sure
and ENV vars that could be used on a platform are set to standard or
default values while perl is building. Otherwise you end up with
something that is difficult to reproduce depending on how you login and
where.

I have filed a bug for the underlying gdbm issue as​:
[perl #119623] GDBM_FILE​: gdbm_open requires a blocksize to be a power of two

As for the other build problems, I don't know what, specifically, you are
recommending. I don't have any idea which variables you think Configure
ought to be setting. Apart from variables related to dynamic loading,
Configure generally doesn't use environment variables to communicate
build information.

More broadly, my general bias is that Configure ought to respect what ever
environment variables you set up, in order to allow you to set them to
whatever you want for whatever reason you want. (Configure does alter
a few environment variables that have been historically problematic,
but generally we prefer not to.)

I am going to close this ticket. If you have a specific build problem
to address, please open a new ticket with a minimal recipe that
someone can use to reproduce the problem.

Thank you,

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

@doughera88 - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Sep 5, 2013
@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From perl-diddler@tlinx.org

On Thu Sep 05 07​:30​:11 2013, doughera wrote​:

I have filed a bug for the underlying gdbm issue as​:
[perl #119623] GDBM_FILE​: gdbm_open requires a blocksize to be a power
of two


Thanks. Unfortunately, that only gets around part of the problem.

The other part involves "NDBM/ODBM" -- they both end up calling
gdbm_open, and both end up creating their own value for the blocksize
and guess what they use! :-(.... "0"...

On the bright side, patching that 0 in the gdbm(compat) source to (I
used 4k), worked -- 5.18 built & tested. Without that part, the gdbm
tests past, but the ndbm/odbm ones failed.

As for the other build problems, I don't know what, specifically, you
are recommending. I don't have any idea which variables you
think Configure ought to be setting. Apart from variables
related to dynamic loading, Configure generally doesn't use
environment variables to communicate build information.


Ok, let me rephrase -- Suppose someone thinks they want to build
perl as utf8 source. I.e. someone makes one or more changes
to include utf8 in perl source used in build & test, so they
add "-Mutf8" to PERL5OPS. Will perl build and test with that
set in the PERL5OPTS?

Compare that to to making the linux kernel. If I go into the
".config" and tell it to build the kernel with values that wouldn't
work -- the kernel make detects that .config was changed and
runs it's "lint-config" equiv on the config file to verify you
didn't specify an illegal configuration. If so, it will
autocorrect it usually and then continue -- but the 'illegal
option' will be removed from the "build-env" (in this case a config
file) before the build starts.

Similarly, if values in PERL5OPTS (like -Mutf8​::all, or -Mutf8 or
whatever) will cause the build to fail in some unclear way, then what I
was suggesting was that perl "check" it's input parameters and attempt
to coerce them into validity. If an ENV var is including modules that
shouldn't be included, then it would remove the offending directives.

It may be the case that some values (-M<special>) are
valid options at build time. Ideally those would be placed on a
"white list" -- at least that's one way of addressing the issue. I
don't know how large the white list would be -- (and this gets a bit
squirrelly) or if that is practical.

To augment the default white-list, or, to skip extra validation of
inputs from ENV & config files, perhaps an "--as-is" (or whatever!)
switch would be a way to re-allow the current abilities to include
anything -- with the difference from the current situation,
being that a perl-maker/builder would have to explicitly
set "--as-is", to use what might otherwise be "bogus" values in their
env or input configuration.

More broadly, my general bias is that Configure ought to respect what
ever
environment variables you set up, in order to allow you to set them to
whatever you want for whatever reason you want. (Configure does alter
a few environment variables that have been historically problematic,
but generally we prefer not to.)


  Generally, I agree -- I'm just thinking of a sanity check to make
sure that options that are known to cause problems with the build are
filtered. If you think something like -Mutf8 or -Mutf8​::all in your
PERL5OPTS is harmless and should be allowed, might I suggest trying a
build with those set? I.e. does it make sense for any reason allow
options that might cause a build to fail in some obscure way?


As for the gdbm bug submission​:... I sent the below, yesterday, but have
gotten no response. It may be I'll have to figure out a way to
patch the source, since the default value of '0' used in ndbm/odbm is
still a problem even if the gdbm values are fixed. ;-(


-------- Original Message --------
Subject​: bug in 1.10​: "optimal i/o size is *assumed* to be power of 2.
This is not always true.
Date​: Wed, 04 Sep 2013 23​:09​:35 -0700
From​: Linda A. Walsh <gnu...tlinx.org>
To​: bug-gdbm@​gnu.org

In gdbmopen.c, ~ lines 235-242 we can see the dir size starting off
with 8 'datums' of size (off_t), which it says take 3 bits to store.
Fine. Then it loops on dir_size < block_size and uses
a leftshift on size, and +1 on bits until dir_size >= blocksize.

Using a 12-data disk RAID of 64KB/segment, => 768K = 1 fullwidth
stripe on the RAID -- which is exactly what is returned from "stat" when
asked
for the blocksize.

When dirsize becomes > 768K, it will have jumped from 512K->1M.

Following that at line 244, is a check​:

/* Check for correct block_size. */
if (dbf->header->dir_size != dbf->header->block_size)
  {
  gdbm_close (dbf);
  gdbm_errno = GDBM_BLOCK_SIZE_ERROR;
  return NULL;
  }


But dir block size Cannot be equal to the desired block size,
to the way it is calculated by powers of 2.

Either the prog needs to use dir_size/(sizeof(off_t)) to get dir
entries, or if power of 2 is needed for other reasons, then 256K
needs to be "allocated" to padding after the dir_block_size gets
to 512K -- thus causing further DB writes to be aligned.

The above was detected using the SuSE factory source rpm for what
will be "13.1" of opensuse. Note -- it is also the case that because
of this bug, perl won't build and pass it's DB tests on
some machines (like mine) because of this error.


Also, a bit of oddness -- I know that several places use
a hard-coded '31' as top end for number of bits.. Might this
cause problems on 64-bit machines? (bucket.c being the
main offender)

@p5pRT
Copy link
Author

p5pRT commented Sep 6, 2013

From @doughera88

On Thu, Sep 05, 2013 at 04​:56​:34PM -0700, "Linda Walsh via RT" wrote​:

On Thu Sep 05 07​:30​:11 2013, doughera wrote​:

I have filed a bug for the underlying gdbm issue as​:
[perl #119623] GDBM_FILE​: gdbm_open requires a blocksize to be a power
of two
----
Thanks. Unfortunately, that only gets around part of the problem.

The other part involves "NDBM/ODBM" -- they both end up calling
gdbm_open, and both end up creating their own value for the blocksize
and guess what they use! :-(.... "0"...

Oh, right. You're using the gdbm emulations.

On the bright side, patching that 0 in the gdbm(compat) source to (I
used 4k), worked -- 5.18 built & tested. Without that part, the gdbm
tests past, but the ndbm/odbm ones failed.

Fine. Or, you could just not include ndbm and odbm, unless you
really want them.

As for the other build problems, I don't know what, specifically, you
are recommending. I don't have any idea which variables you
think Configure ought to be setting. Apart from variables
related to dynamic loading, Configure generally doesn't use
environment variables to communicate build information.
----
Ok, let me rephrase -- Suppose someone thinks they want to build
perl as utf8 source. I.e. someone makes one or more changes
to include utf8 in perl source used in build & test, so they
add "-Mutf8" to PERL5OPS. Will perl build and test with that
set in the PERL5OPTS?

Oh, ok. That's a different question. Your original problem report
never built perl at all, so all the PERL5* environment variables were
completely irrelevant.

During the build, yes, some PERL5OPT settings can cause trouble. See
[perl #5844] for more details. But again, that wasn't what was
happening in the case reported here.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented Sep 6, 2013

From perl-diddler@tlinx.org

On Thu Sep 05 18​:44​:19 2013, doughera wrote​:

On Thu, Sep 05, 2013 at 04​:56​:34PM -0700, "Linda Walsh via RT" wrote​:

On Thu Sep 05 07​:30​:11 2013, doughera wrote​:

I have filed a bug for the underlying gdbm issue as​:
[perl #119623] GDBM_FILE​: gdbm_open requires a blocksize to be a power
of two
----
Thanks. Unfortunately, that only gets around part of the problem.

The other part involves "NDBM/ODBM" -- they both end up calling
gdbm_open, and both end up creating their own value for the blocksize
and guess what they use! :-(.... "0"...

Oh, right. You're using the gdbm emulations.

On the bright side, patching that 0 in the gdbm(compat) source to (I
used 4k), worked -- 5.18 built & tested. Without that part, the gdbm
tests past, but the ndbm/odbm ones failed.

Fine. Or, you could just not include ndbm and odbm, unless you
really want them.

As for the other build problems, I don't know what, specifically, you
are recommending. I don't have any idea which variables you
think Configure ought to be setting. Apart from variables
related to dynamic loading, Configure generally doesn't use
environment variables to communicate build information.
----
Ok, let me rephrase -- Suppose someone thinks they want to build
perl as utf8 source. I.e. someone makes one or more changes
to include utf8 in perl source used in build & test, so they
add "-Mutf8" to PERL5OPS. Will perl build and test with that
set in the PERL5OPTS?

Oh, ok. That's a different question. Your original problem report
never built perl at all, so all the PERL5* environment variables were
completely irrelevant.


The initial build failed -- and I think that might have been due to
some ENV var, like PERL5OPT being set for 'normal' usage, and not being
"unset" for a perl build.

During the build, yes, some PERL5OPT settings can cause trouble. See
[perl #5844] for more details. But again, that wasn't what was
happening in the case reported here.


I'm saying that some of the "variability of symptoms" may have been
because of ENV vars -- though that's not the problem with the DB issues.

Recently, I noted that the settings in the "GREP_OPTIONS" var can
contaminate a kernel build tree, such that when you do subsequent builds
(even with GREP_OPTIONS cleared), it won't build correctly.
You need to do a "make distclean" between builds to get rid of
"Environmental contamination"...

Perl is small enough that removing the old tree and and untarring a
fresh copy take <1 second, so that seems to be the easier option, so
after the first attempt in a pre-existing tree in my rpm-build dir, I
switched to fresh-trees on each attempt -- that cleared up many problems
right there (I'd last tried to build 5.18-rpms about 3-4 months ago).

Thanks for the help... now have to figure out what to do with ndbm/odbm...

Me, personally -- I don't care so much, but wanting it to be compat
with the perl released by my distro, as I usually drop new perl's in
place of older distro perls (with recompiles)...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant