Skip Menu |
Report information
Id: 119623
Status: resolved
Priority: 0/
Queue: perl5

Owner: doughera <doughera [at] lafayette.edu>
Requestors: doughera <doughera [at] lafayette.edu>
Cc:
AdminCc:

Operating System: Linux
PatchStatus: (no value)
Severity: low
Type: library
Perl Version: 5.18.0
Fixed In: 5.27.7



Subject: GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Thu, 5 Sep 2013 09:46:20 -0400
To: perlbug [...] perl.org
From: Andy Dougherty <doughera [...] lafayette.edu>
Download (untitled) / with headers
text/plain 3.6k
This is a bug report for perl from doughera@lafayette.edu, generated with the help of perlbug 1.39 running under perl 5.18.0. ----------------------------------------------------------------- [Please describe your issue here] ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0. This instructs gdbm to use the stat block size for the file system. Unfortunately, gdbm will fail if that stat block size is not a power of two. See [perl #119537] for some more details. Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we should retry with a 'default' blocksize, something simple like 1024. Or, perhaps GDBM_File.xs ought to do something like stat(dirname(name)) and then work out the "closest" power-of-two. (Note, though,that dirname() may alter "name", so we'd have to first make a copy and then free it.) [config info below is irrelvant] [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=library severity=low module=GDBM_File --- Site configuration information for perl 5.18.0: Configured by doughera at Mon May 13 07:10:40 EDT 2013. Summary of my perl5 (revision 5 version 18 subversion 0) configuration: Platform: osname=linux, osvers=3.2.0-4-amd64, archname=x86_64-linux uname='linux xxx 3.2.0-4-amd64 #1 smp debian 3.2.41-2 x86_64 gnulinux ' config_args='-Dprefix=/opt/perl -Dscriptdir=/opt/perl/bin -des' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.7.2', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.13' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Locally applied patches: RC2 --- @INC for perl 5.18.0: /opt/perl/lib/site_perl/5.18.0/x86_64-linux /opt/perl/lib/site_perl/5.18.0 /opt/perl/lib/5.18.0/x86_64-linux /opt/perl/lib/5.18.0 /opt/perl/lib/site_perl/5.16.2 /opt/perl/lib/site_perl/5.16.0 /opt/perl/lib/site_perl/5.14.2 /opt/perl/lib/site_perl/5.12.3 /opt/perl/lib/site_perl/5.10.1 /opt/perl/lib/site_perl/5.8.9 /opt/perl/lib/site_perl . --- Environment for perl 5.18.0: HOME=/home/doughera LANG (unset) LANGUAGE (unset) LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/usr/local/lib LOGDIR (unset) PATH=/home/doughera/script:/home/doughera/bin:/opt/perl/bin:/opt/dist/bin:/opt/gnuplot/bin:/opt/groff/bin:/usr/local/script:/usr/local/bin:/opt/dist/bin:/bin:/usr/bin:/usr/X11R6/bin:. PERL_BADLANG (unset) SHELL=/bin/bash
RT-Send-CC: perl5-porters [...] perl.org
On Thu Sep 05 06:46:45 2013, doughera wrote: Show quoted text
> > This is a bug report for perl from doughera@lafayette.edu, > generated with the help of perlbug 1.39 running under perl 5.18.0. > > > ----------------------------------------------------------------- > [Please describe your issue here] > > ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0. > This instructs gdbm to use the stat block size for the file system. > Unfortunately, gdbm will fail if that stat block size is not a power > of two. See [perl #119537] for some more details. > > Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we > should retry with a 'default' blocksize, something simple like 1024. > Or, > perhaps GDBM_File.xs ought to do something like stat(dirname(name)) > and > then work out the "closest" power-of-two. (Note, though,that > dirname() > may alter "name", so we'd have to first make a copy and then free it.) >
Andy, Do you have any suggestions as to how we would test for the presence of this problem? I think that would be a necessary first step toward a solution. Thank you very much. Jim Keenan
Subject: Re: [perl #119623] GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Wed, 11 Dec 2013 21:56:13 -0500
To: James E Keenan via RT <perlbug-followup [...] perl.org>
From: Andy Dougherty <doughera [...] lafayette.edu>
Download (untitled) / with headers
text/plain 1.5k
On Wed, Dec 11, 2013 at 04:06:08PM -0800, James E Keenan via RT wrote: Show quoted text
> On Thu Sep 05 06:46:45 2013, doughera wrote:
> > > > This is a bug report for perl from doughera@lafayette.edu, > > generated with the help of perlbug 1.39 running under perl 5.18.0. > > > > > > ----------------------------------------------------------------- > > [Please describe your issue here] > > > > ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0. > > This instructs gdbm to use the stat block size for the file system. > > Unfortunately, gdbm will fail if that stat block size is not a power > > of two. See [perl #119537] for some more details. > > > > Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we > > should retry with a 'default' blocksize, something simple like 1024. > > Or, > > perhaps GDBM_File.xs ought to do something like stat(dirname(name)) > > and > > then work out the "closest" power-of-two. (Note, though,that > > dirname() > > may alter "name", so we'd have to first make a copy and then free it.) > >
> > Andy, > > Do you have any suggestions as to how we would test for the presence of this problem? > > I think that would be a necessary first step toward a solution.
This problem only manifests itself on certain filesystems, which might not match the filesystem on which perl is built and tested. I don't see an easy and reliable way to test for it. I think that checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then retrying with 1024 would probably be good enough. -- Andy Dougherty doughera@lafayette.edu
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.3k
On Wed Dec 11 18:56:53 2013, doughera wrote: Show quoted text
> On Wed, Dec 11, 2013 at 04:06:08PM -0800, James E Keenan via RT wrote:
> > Andy, > > > > Do you have any suggestions as to how we would test for the presence > > of this problem? > > > > I think that would be a necessary first step toward a solution.
> > This problem only manifests itself on certain filesystems, which might > not match the filesystem on which perl is built and tested. > I don't see an easy and reliable way to test for it. I think that > checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then > retrying with 1024 would probably be good enough.
----------- It may be doable with other file sytems, but xfs has parameters to specify the strip size and width. I have stripes of 64K and had a RAID 50 with 3 RAID5's. Each member of the RAID5 had a 64K stripe size, with 4 data disks -- total width 256k. The striping of those in a RAID0 with 3 units gave me the odd "optimal i/o size of 3x256 => 756K. To work around the problem I gave up on exposing the larger optimal I/O and went with a 256K. The linux "mkfs.xfs" Could be used to construct a virtual drive on a ram disk or in a file. Wrote a quick bash script to create such a disk on linux (assumes you are running as root). It's a quick and simple script but should be easily modified or used as a ref to build something more complex...(attached)
Subject: testdisk
Download testdisk
application/octet-stream 1.4k

Message body not shown because it is not plain text.

From: Andy Dougherty <doughera [...] lafayette.edu>
To: Linda Walsh via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #119623] GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Fri, 31 Jan 2014 08:19:20 -0500
Download (untitled) / with headers
text/plain 1.8k
On Thu, Jan 30, 2014 at 10:20:24PM -0800, Linda Walsh via RT wrote: Show quoted text
> On Wed Dec 11 18:56:53 2013, doughera wrote:
> > On Wed, Dec 11, 2013 at 04:06:08PM -0800, James E Keenan via RT wrote:
> > > Andy, > > > > > > Do you have any suggestions as to how we would test for the presence > > > of this problem? > > > > > > I think that would be a necessary first step toward a solution.
> > > > This problem only manifests itself on certain filesystems, which might > > not match the filesystem on which perl is built and tested. > > I don't see an easy and reliable way to test for it. I think that > > checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then > > retrying with 1024 would probably be good enough.
> ----------- > > It may be doable with other file sytems, but xfs has parameters to > specify the strip size and width. I have stripes of 64K and had > a RAID 50 with 3 RAID5's. Each member of the RAID5 had a 64K stripe > size, with 4 data disks -- total width 256k. The striping of > those in a RAID0 with 3 units gave me the odd "optimal i/o size > of 3x256 => 756K. > > To work around the problem I gave up on exposing the larger > optimal I/O and went with a 256K. > > The linux "mkfs.xfs" Could be used to construct a virtual > drive on a ram disk or in a file. Wrote a quick bash script > to create such a disk on linux (assumes you are running as > root). It's a quick and simple script but should be > easily modified or used as a ref to build something more > complex...(attached)
Thank you, but I don't know what you imagine we should do with this script. I can't imagine including it in the perl test suite, and a developer exploring a fix can reproduce the problem by simply redefining GDBM_BLOCKSIZE in ext/GDBM_File.xs to something other than a power of 2. -- Andy Dougherty doughera@lafayette.edu
From: Zefram <zefram [...] fysh.org>
Subject: Re: [perl #119623] GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Sat, 16 Dec 2017 02:31:10 +0000
To: perl5-porters [...] perl.org
Fixed in commit d1500e115d7f380f58559a7db59d099856652b5b. -zefram
Download (untitled) / with headers
text/plain 317b
Thank you for filing this report. You have helped make Perl better. With the release yesterday of Perl 5.28.0, this and 185 other issues have been resolved. Perl 5.28.0 may be downloaded via: https://metacpan.org/release/XSAWYERX/perl-5.28.0 If you find that the problem persists, feel free to reopen this ticket.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org