Skip Menu |
Report information
Id: 119623
Status: pending release
Priority: 0/
Queue: perl5

Owner: doughera <doughera [at] lafayette.edu>
Requestors: doughera <doughera [at] lafayette.edu>
Cc:
AdminCc:

Operating System: Linux
PatchStatus: (no value)
Severity: low
Type: library
Perl Version: 5.18.0
Fixed In: 5.27.7



Subject: GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Thu, 5 Sep 2013 09:46:20 -0400
To: perlbug [...] perl.org
From: Andy Dougherty <doughera [...] lafayette.edu>
Download (untitled) / with headers
text/plain 3.6k
This is a bug report for perl from doughera@lafayette.edu, generated with the help of perlbug 1.39 running under perl 5.18.0. ----------------------------------------------------------------- [Please describe your issue here] ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0. This instructs gdbm to use the stat block size for the file system. Unfortunately, gdbm will fail if that stat block size is not a power of two. See [perl #119537] for some more details. Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we should retry with a 'default' blocksize, something simple like 1024. Or, perhaps GDBM_File.xs ought to do something like stat(dirname(name)) and then work out the "closest" power-of-two. (Note, though,that dirname() may alter "name", so we'd have to first make a copy and then free it.) [config info below is irrelvant] [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=library severity=low module=GDBM_File --- Site configuration information for perl 5.18.0: Configured by doughera at Mon May 13 07:10:40 EDT 2013. Summary of my perl5 (revision 5 version 18 subversion 0) configuration: Platform: osname=linux, osvers=3.2.0-4-amd64, archname=x86_64-linux uname='linux xxx 3.2.0-4-amd64 #1 smp debian 3.2.41-2 x86_64 gnulinux ' config_args='-Dprefix=/opt/perl -Dscriptdir=/opt/perl/bin -des' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.7.2', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.13' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Locally applied patches: RC2 --- @INC for perl 5.18.0: /opt/perl/lib/site_perl/5.18.0/x86_64-linux /opt/perl/lib/site_perl/5.18.0 /opt/perl/lib/5.18.0/x86_64-linux /opt/perl/lib/5.18.0 /opt/perl/lib/site_perl/5.16.2 /opt/perl/lib/site_perl/5.16.0 /opt/perl/lib/site_perl/5.14.2 /opt/perl/lib/site_perl/5.12.3 /opt/perl/lib/site_perl/5.10.1 /opt/perl/lib/site_perl/5.8.9 /opt/perl/lib/site_perl . --- Environment for perl 5.18.0: HOME=/home/doughera LANG (unset) LANGUAGE (unset) LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH=/usr/local/lib LOGDIR (unset) PATH=/home/doughera/script:/home/doughera/bin:/opt/perl/bin:/opt/dist/bin:/opt/gnuplot/bin:/opt/groff/bin:/usr/local/script:/usr/local/bin:/opt/dist/bin:/bin:/usr/bin:/usr/X11R6/bin:. PERL_BADLANG (unset) SHELL=/bin/bash
RT-Send-CC: perl5-porters [...] perl.org
On Thu Sep 05 06:46:45 2013, doughera wrote: Show quoted text
> > This is a bug report for perl from doughera@lafayette.edu, > generated with the help of perlbug 1.39 running under perl 5.18.0. > > > ----------------------------------------------------------------- > [Please describe your issue here] > > ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0. > This instructs gdbm to use the stat block size for the file system. > Unfortunately, gdbm will fail if that stat block size is not a power > of two. See [perl #119537] for some more details. > > Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we > should retry with a 'default' blocksize, something simple like 1024. > Or, > perhaps GDBM_File.xs ought to do something like stat(dirname(name)) > and > then work out the "closest" power-of-two. (Note, though,that > dirname() > may alter "name", so we'd have to first make a copy and then free it.) >
Andy, Do you have any suggestions as to how we would test for the presence of this problem? I think that would be a necessary first step toward a solution. Thank you very much. Jim Keenan
Subject: Re: [perl #119623] GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Wed, 11 Dec 2013 21:56:13 -0500
To: James E Keenan via RT <perlbug-followup [...] perl.org>
From: Andy Dougherty <doughera [...] lafayette.edu>
Download (untitled) / with headers
text/plain 1.5k
On Wed, Dec 11, 2013 at 04:06:08PM -0800, James E Keenan via RT wrote: Show quoted text
> On Thu Sep 05 06:46:45 2013, doughera wrote:
> > > > This is a bug report for perl from doughera@lafayette.edu, > > generated with the help of perlbug 1.39 running under perl 5.18.0. > > > > > > ----------------------------------------------------------------- > > [Please describe your issue here] > > > > ext/GDBM_File/GDBM_File.xs calls gdbm_open with GDBM_BLOCKSIZE=0. > > This instructs gdbm to use the stat block size for the file system. > > Unfortunately, gdbm will fail if that stat block size is not a power > > of two. See [perl #119537] for some more details. > > > > Maybe if gdbm_open fails with gdbm_error == GDBM_BLOCK_SIZE_ERROR, we > > should retry with a 'default' blocksize, something simple like 1024. > > Or, > > perhaps GDBM_File.xs ought to do something like stat(dirname(name)) > > and > > then work out the "closest" power-of-two. (Note, though,that > > dirname() > > may alter "name", so we'd have to first make a copy and then free it.) > >
> > Andy, > > Do you have any suggestions as to how we would test for the presence of this problem? > > I think that would be a necessary first step toward a solution.
This problem only manifests itself on certain filesystems, which might not match the filesystem on which perl is built and tested. I don't see an easy and reliable way to test for it. I think that checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then retrying with 1024 would probably be good enough. -- Andy Dougherty doughera@lafayette.edu
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.3k
On Wed Dec 11 18:56:53 2013, doughera wrote: Show quoted text
> On Wed, Dec 11, 2013 at 04:06:08PM -0800, James E Keenan via RT wrote:
> > Andy, > > > > Do you have any suggestions as to how we would test for the presence > > of this problem? > > > > I think that would be a necessary first step toward a solution.
> > This problem only manifests itself on certain filesystems, which might > not match the filesystem on which perl is built and tested. > I don't see an easy and reliable way to test for it. I think that > checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then > retrying with 1024 would probably be good enough.
----------- It may be doable with other file sytems, but xfs has parameters to specify the strip size and width. I have stripes of 64K and had a RAID 50 with 3 RAID5's. Each member of the RAID5 had a 64K stripe size, with 4 data disks -- total width 256k. The striping of those in a RAID0 with 3 units gave me the odd "optimal i/o size of 3x256 => 756K. To work around the problem I gave up on exposing the larger optimal I/O and went with a 256K. The linux "mkfs.xfs" Could be used to construct a virtual drive on a ram disk or in a file. Wrote a quick bash script to create such a disk on linux (assumes you are running as root). It's a quick and simple script but should be easily modified or used as a ref to build something more complex...(attached)
Subject: testdisk
Download testdisk
application/octet-stream 1.4k

Message body not shown because it is not plain text.

From: Andy Dougherty <doughera [...] lafayette.edu>
To: Linda Walsh via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #119623] GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Fri, 31 Jan 2014 08:19:20 -0500
Download (untitled) / with headers
text/plain 1.8k
On Thu, Jan 30, 2014 at 10:20:24PM -0800, Linda Walsh via RT wrote: Show quoted text
> On Wed Dec 11 18:56:53 2013, doughera wrote:
> > On Wed, Dec 11, 2013 at 04:06:08PM -0800, James E Keenan via RT wrote:
> > > Andy, > > > > > > Do you have any suggestions as to how we would test for the presence > > > of this problem? > > > > > > I think that would be a necessary first step toward a solution.
> > > > This problem only manifests itself on certain filesystems, which might > > not match the filesystem on which perl is built and tested. > > I don't see an easy and reliable way to test for it. I think that > > checking for gdbm_open failing with the GDBM_BLOCK_SIZE_ERROR and then > > retrying with 1024 would probably be good enough.
> ----------- > > It may be doable with other file sytems, but xfs has parameters to > specify the strip size and width. I have stripes of 64K and had > a RAID 50 with 3 RAID5's. Each member of the RAID5 had a 64K stripe > size, with 4 data disks -- total width 256k. The striping of > those in a RAID0 with 3 units gave me the odd "optimal i/o size > of 3x256 => 756K. > > To work around the problem I gave up on exposing the larger > optimal I/O and went with a 256K. > > The linux "mkfs.xfs" Could be used to construct a virtual > drive on a ram disk or in a file. Wrote a quick bash script > to create such a disk on linux (assumes you are running as > root). It's a quick and simple script but should be > easily modified or used as a ref to build something more > complex...(attached)
Thank you, but I don't know what you imagine we should do with this script. I can't imagine including it in the perl test suite, and a developer exploring a fix can reproduce the problem by simply redefining GDBM_BLOCKSIZE in ext/GDBM_File.xs to something other than a power of 2. -- Andy Dougherty doughera@lafayette.edu
From: Zefram <zefram [...] fysh.org>
Subject: Re: [perl #119623] GDBM_FILE: gdbm_open requires a blocksize to be a power of two
Date: Sat, 16 Dec 2017 02:31:10 +0000
To: perl5-porters [...] perl.org
Fixed in commit d1500e115d7f380f58559a7db59d099856652b5b. -zefram


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org