Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Module Loading on Case-Insensitive File Systems #6480

Open
p5pRT opened this issue May 1, 2003 · 11 comments
Open

Module Loading on Case-Insensitive File Systems #6480

p5pRT opened this issue May 1, 2003 · 11 comments

Comments

@p5pRT
Copy link

p5pRT commented May 1, 2003

Migrated from rt.perl.org#22077 (status was 'open')

Searchable as RT22077$

@p5pRT
Copy link
Author

p5pRT commented May 1, 2003

From david@kineticode.com

This is a bug report for perl from david@​kineticode.com,
generated with the help of perlbug 1.34 running under perl v5.8.0.


An issue has just come up with Stas Bekman's release of Apache​::Test. It turns
out that on case-insensitive file systems, such as HFS+ on Mac OS X, and
whatever Win32 runs these days, Perl can load the wrong module. For example,
Stas' new module is called "Apache​::Test". Scripts that load it, such as
Makefile.PL in libapreq 1.2 (http​://httpd.apache.org/~joes/), do so like this​:

use constant HAS_APACHE_TEST => eval {require Apache​::Test};

The problem, however, is that those of us who have already installed mod_perl
1.x have a copy of "Apache​::test" installed already. Note the different case​:
"Apache​::Test" vs. "Apache​::test". It turns out that, because the mod_perl
installer installs Apache​::test in the platform directory (e.g.,
site_perl/5.8.0/darwin/Apache/test.pm on my system), while Apache​::Test gets
installed in the usual Perl library directory (e.g., on my system,
site_perl/5.8.0/Apache/Test.pm), because of the order of @​INC (see below),
Apache​::test actually gets loaded instead of Apache​::Test! Even worse, because
Apache​::test has no version number CPANPLUS seems to think that Apache​::Test
is up-to-date, even though, technically, it's not installed at all!

This appears to be because although HFS+ is a case-insensitive file system, it
is also a case-preserving file system, and Perl doesn't necessarily know
that. What would be great is if, on case-insensitive but case-preserving file
systems, Perl was smart enough to check the case of any files it finds on the
file system before actually loading them. FWIW, the `find` utility that comes
with Mac OS X seems to be able to tell the difference​:

  mercury# find . -name "Test.pm"
  ./5.8.0/Test.pm
  ./site_perl/5.8.0/Apache/Test.pm
  ./site_perl/5.8.0/Devel/Profiler/Test.pm
  ./site_perl/5.8.0/HTTP/WebTest/Test.pm
  ./site_perl/5.8.0/SOAP/Test.pm
  ./site_perl/5.8.0/XMLRPC/Test.pm
  mercury# find . -name "test.pm"
  ./site_perl/5.8.0/darwin/Apache/test.pm
  ./site_perl/5.8.0/Mail/Mailer/test.pm

Note that when I told `find` to search for "test.pm" it didn't find any
"Test.pm" files, and when I told it to search for "Test.pm" it didn't find any
"test.pm" files. So it pays attention to the case you tell it to look for when
it searches the file system. Theoretically, Perl can do this, too, at least on
case-insensitive file systems that are also case-preserving. Or so I would
think.

Stas and I are discussing other strategies for overcoming this problem. But if
there was some way to tell Perl to do a case-sensitive require, that would be
preferable. There isn't some kind of pragma or something to correct this
behavior, is there?

Thanks!

David



Flags​:
  category=core
  severity=low


Site configuration information for perl v5.8.0​:

Configured by david at Sat Dec 21 18​:17​:50 PST 2002.

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration​:
  Platform​:
  osname=darwin, osvers=6.3, archname=darwin
  uname='darwin mercury.local. 6.3 darwin kernel version 6.3​: sat dec 14 03​:11​:25 pst 2002; root​:xnuxnu-344.23.obj~4release_ppc power macintosh powerpc '
  config_args='-des -Uinstallusrbinperl -Dprefix=/usr/local -Dperladmin=david@​kineticode.com -Dcf_email=david@​kineticode.com'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-pipe -fno-common -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include',
  optimize='-O3',
  cppflags='-no-cpp-precomp -pipe -fno-common -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include'
  ccversion='', gccversion='3.1 20020420 (prerelease)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -flat_namespace -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib
  libs=-lm -lc
  perllibs=-lm -lc
  libc=/usr/lib/libc.dylib, so=dylib, useshrplib=true, libperl=libperl.dylib
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -flat_namespace -bundle -undefined suppress -L/usr/local/lib'

Locally applied patches​:
 


@​INC for perl v5.8.0​:
  /usr/local/lib/perl5/5.8.0/darwin
  /usr/local/lib/perl5/5.8.0
  /usr/local/lib/perl5/site_perl/5.8.0/darwin
  /usr/local/lib/perl5/site_perl/5.8.0
  /usr/local/lib/perl5/site_perl
  .


Environment for perl v5.8.0​:
  DYLD_LIBRARY_PATH (unset)
  HOME=/Users/david
  LANG=en_US
  LANGUAGE (unset)
  LC_ALL=C
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/usr/local/bin​:/usr/bin​:/bin​:/usr/sbin​:/sbin​:/Users/david​:/usr/local/bin​:/sbin​:/usr/sbin​:/Developer/Tools//Users/david/bin​:/sbin​:/usr/sbin​:/usr/local/sbin​:/Developer/Tools​:/Users/david/bin​::/usr/local/pgsql/bin​:/usr/local/jakarta-ant-1.4.1/bin
  PERL5LIB=​:/usr/local/bricolage/lib​:/Users/david/dev/perl/myco/classes
  PERL_BADLANG (unset)
  SHELL=/bin/zsh

@p5pRT
Copy link
Author

p5pRT commented May 1, 2003

From @eserte

David Wheeler (via RT) <perlbug-followup@​perl.org> writes​:

# New Ticket Created by David Wheeler
# Please include the string​: [perl #22077]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt2/Ticket/Display.html?id=22077 >

This is a bug report for perl from david@​kineticode.com,
generated with the help of perlbug 1.34 running under perl v5.8.0.

-----------------------------------------------------------------
An issue has just come up with Stas Bekman's release of Apache​::Test. It turns
out that on case-insensitive file systems, such as HFS+ on Mac OS X, and
whatever Win32 runs these days, Perl can load the wrong module. For example,
Stas' new module is called "Apache​::Test". Scripts that load it, such as
Makefile.PL in libapreq 1.2 (http​://httpd.apache.org/~joes/), do so like this​:

use constant HAS_APACHE_TEST => eval {require Apache​::Test};

The problem, however, is that those of us who have already installed mod_perl
1.x have a copy of "Apache​::test" installed already. Note the different case​:
"Apache​::Test" vs. "Apache​::test". It turns out that, because the mod_perl
installer installs Apache​::test in the platform directory (e.g.,
site_perl/5.8.0/darwin/Apache/test.pm on my system), while Apache​::Test gets
installed in the usual Perl library directory (e.g., on my system,
site_perl/5.8.0/Apache/Test.pm), because of the order of @​INC (see below),
Apache​::test actually gets loaded instead of Apache​::Test! Even worse, because
Apache​::test has no version number CPANPLUS seems to think that Apache​::Test
is up-to-date, even though, technically, it's not installed at all!

This appears to be because although HFS+ is a case-insensitive file system, it
is also a case-preserving file system, and Perl doesn't necessarily know
that. What would be great is if, on case-insensitive but case-preserving file
systems, Perl was smart enough to check the case of any files it finds on the
file system before actually loading them. FWIW, the `find` utility that comes
with Mac OS X seems to be able to tell the difference​:

mercury# find . -name "Test.pm"
./5.8.0/Test.pm
./site_perl/5.8.0/Apache/Test.pm
./site_perl/5.8.0/Devel/Profiler/Test.pm
./site_perl/5.8.0/HTTP/WebTest/Test.pm
./site_perl/5.8.0/SOAP/Test.pm
./site_perl/5.8.0/XMLRPC/Test.pm
mercury# find . -name "test.pm"
./site_perl/5.8.0/darwin/Apache/test.pm
./site_perl/5.8.0/Mail/Mailer/test.pm

Note that when I told `find` to search for "test.pm" it didn't find any
"Test.pm" files, and when I told it to search for "Test.pm" it didn't find any
"test.pm" files. So it pays attention to the case you tell it to look for when
it searches the file system. Theoretically, Perl can do this, too, at least on
case-insensitive file systems that are also case-preserving. Or so I would
think.

Stas and I are discussing other strategies for overcoming this problem. But if
there was some way to tell Perl to do a case-sensitive require, that would be
preferable. There isn't some kind of pragma or something to correct this
behavior, is there?

I don't know how things are actually implemented, but my guess is that
find uses opendir/readdir/closedir while loading a module is done
through an open(2) call. In the first case, you get a list of all file
names with the correct case, while open succeeds regardless of the
case.

For a workaround one could do something like this (in pseudo code)​:

  open modulefile
  opendir dirname(modulefile)
  if (! grep { $_ eq basename(modulefile) } readdir)
  fail, try next directory in @​INC

Also, I am not sure how to determine whether a file system is
case-insensitive and case-preserving. I think that asking for $^O does
not suffice, because Mac OS X may also use case-sensitive file
systems, right?

Regards,
  Slaven

Thanks!

David

-----------------------------------------------------------------
---
Flags​:
category=core
severity=low
---
Site configuration information for perl v5.8.0​:

Configured by david at Sat Dec 21 18​:17​:50 PST 2002.

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration​:
Platform​:
osname=darwin, osvers=6.3, archname=darwin
uname='darwin mercury.local. 6.3 darwin kernel version 6.3​: sat dec 14 03​:11​:25 pst 2002; root​:xnuxnu-344.23.obj~4release_ppc power macintosh powerpc '
config_args='-des -Uinstallusrbinperl -Dprefix=/usr/local -Dperladmin=david@​kineticode.com -Dcf_email=david@​kineticode.com'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='cc', ccflags ='-pipe -fno-common -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include',
optimize='-O3',
cppflags='-no-cpp-precomp -pipe -fno-common -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include'
ccversion='', gccversion='3.1 20020420 (prerelease)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries​:
ld='cc', ldflags =' -flat_namespace -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib
libs=-lm -lc
perllibs=-lm -lc
libc=/usr/lib/libc.dylib, so=dylib, useshrplib=true, libperl=libperl.dylib
gnulibc_version=''
Dynamic Linking​:
dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' -flat_namespace -bundle -undefined suppress -L/usr/local/lib'

Locally applied patches​:

---
@​INC for perl v5.8.0​:
/usr/local/lib/perl5/5.8.0/darwin
/usr/local/lib/perl5/5.8.0
/usr/local/lib/perl5/site_perl/5.8.0/darwin
/usr/local/lib/perl5/site_perl/5.8.0
/usr/local/lib/perl5/site_perl
.

---
Environment for perl v5.8.0​:
DYLD_LIBRARY_PATH (unset)
HOME=/Users/david
LANG=en_US
LANGUAGE (unset)
LC_ALL=C
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/usr/local/bin​:/usr/bin​:/bin​:/usr/sbin​:/sbin​:/Users/david​:/usr/local/bin​:/sbin​:/usr/sbin​:/Developer/Tools//Users/david/bin​:/sbin​:/usr/sbin​:/usr/local/sbin​:/Developer/Tools​:/Users/david/bin​::/usr/local/pgsql/bin​:/usr/local/jakarta-ant-1.4.1/bin
PERL5LIB=​:/usr/local/bricolage/lib​:/Users/david/dev/perl/myco/classes
PERL_BADLANG (unset)
SHELL=/bin/zsh

--
Slaven Rezic - slaven@​rezic.de
  babybike - routeplanner for cyclists in Berlin
  handheld (e.g. Compaq iPAQ with Linux) version of bbbike
  http​://bbbike.sourceforge.net

@p5pRT
Copy link
Author

p5pRT commented May 6, 2003

From david@kineticode.com

On Thursday, May 1, 2003, at 11​:45 AM, slaven@​rezic.de (via RT) wrote​:

I don't know how things are actually implemented, but my guess is that
find uses opendir/readdir/closedir while loading a module is done
through an open(2) call. In the first case, you get a list of all file
names with the correct case, while open succeeds regardless of the
case.

Yes, that makes sense. We're using Perl's readdir in Apache​::Test to
get 'round the problem.

For a workaround one could do something like this (in pseudo code)​:

open modulefile
opendir dirname\(modulefile\)
if \(\! grep \{ $\_ eq basename\(modulefile\) \} readdir\)
    fail\, try next directory in @&#8203;INC

Also, I am not sure how to determine whether a file system is
case-insensitive and case-preserving. I think that asking for $^O does
not suffice, because Mac OS X may also use case-sensitive file
systems, right?

Correct, though in practice few use it. Does Config.pm know? I know
that Perl knows at compile time.

Regards,

David

@p5pRT
Copy link
Author

p5pRT commented May 6, 2003

From @doughera88

On Mon, 5 May 2003, David Wheeler wrote​:

On Thursday, May 1, 2003, at 11​:45 AM, slaven@​rezic.de (via RT) wrote​:

Also, I am not sure how to determine whether a file system is
case-insensitive and case-preserving. I think that asking for $^O does
not suffice, because Mac OS X may also use case-sensitive file
systems, right?

Correct, though in practice few use it. Does Config.pm know? I know
that Perl knows at compile time.

Alas, we can't assume that the filesystem where perl was built is also the
file system where it will be running. Nor can we assume that all file
systems mounted under operating system $^O even have the same behavior
with respect to case. Yes, it's a mess.

--
  Andy Dougherty doughera@​lafayette.edu

@p5pRT
Copy link
Author

p5pRT commented May 6, 2003

From david@kineticode.com

On Tuesday, May 6, 2003, at 07​:39 AM, Andy Dougherty (via RT) wrote​:

Alas, we can't assume that the filesystem where perl was built is also
the
file system where it will be running. Nor can we assume that all file
systems mounted under operating system $^O even have the same behavior
with respect to case. Yes, it's a mess.

Yes, I knew that. It just seemed the safest assumption. The only other
way to know is to try writing test files to disk and see what happens.
But that'd be a stupid thing to do every time one started a Perl
program. It'd be much nicer if Perl could just somehow know the
difference before calling open(). But I expect that this is a rare
enough problem that it may not be worth the overheard.

Regards,

David

@p5pRT
Copy link
Author

p5pRT commented Aug 31, 2013

From @jkeenan

On Tue May 06 10​:11​:19 2003, david@​kineticode.com wrote​:

On Tuesday, May 6, 2003, at 07​:39 AM, Andy Dougherty (via RT) wrote​:

Alas, we can't assume that the filesystem where perl was built is also
the
file system where it will be running. Nor can we assume that all file
systems mounted under operating system $^O even have the same behavior
with respect to case. Yes, it's a mess.

Yes, I knew that. It just seemed the safest assumption. The only other
way to know is to try writing test files to disk and see what happens.
But that'd be a stupid thing to do every time one started a Perl
program. It'd be much nicer if Perl could just somehow know the
difference before calling open(). But I expect that this is a rare
enough problem that it may not be worth the overheard.

Regards,

David

There has been no further discussion in this ticket in over 10 years.
My sense of the discussion is that the original poster had workarounds
and that the problem was not sufficiently severe or widespread to be
worth the effort of finding a solution.

So I'm taking this ticket for the purpose of closing it in 7 days unless
someone wants to pick up the discussion and move it forward.

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Aug 31, 2013

From @Hugmeir

On Fri, Aug 30, 2013 at 10​:37 PM, James E Keenan via RT <
perlbug-followup@​perl.org> wrote​:

On Tue May 06 10​:11​:19 2003, david@​kineticode.com wrote​:

On Tuesday, May 6, 2003, at 07​:39 AM, Andy Dougherty (via RT) wrote​:

Alas, we can't assume that the filesystem where perl was built is also
the
file system where it will be running. Nor can we assume that all file
systems mounted under operating system $^O even have the same behavior
with respect to case. Yes, it's a mess.

Yes, I knew that. It just seemed the safest assumption. The only other
way to know is to try writing test files to disk and see what happens.
But that'd be a stupid thing to do every time one started a Perl
program. It'd be much nicer if Perl could just somehow know the
difference before calling open(). But I expect that this is a rare
enough problem that it may not be worth the overheard.

Regards,

David

There has been no further discussion in this ticket in over 10 years.
My sense of the discussion is that the original poster had workarounds
and that the problem was not sufficiently severe or widespread to be
worth the effort of finding a solution.

So I'm taking this ticket for the purpose of closing it in 7 days unless
someone wants to pick up the discussion and move it forward.

I believe this should stay open. The bug is still present and leads to
unintuitive behavior​:
'use Strict;' might end up loading strict.pm or Strict.pm, depending on the
order of @​INC.

It strikes me that this and non-ascii modules are in the same bag; Perl has
no way of knowing
how a module's filename will be represented by the filesystem, so it just
charges forward in
hope that things will work out.

Perhaps, if we could settle on some way to unambiguously represent module
names as filenames
in some subset of ascii, we could teach perl & the toolchain to use that?

@p5pRT
Copy link
Author

p5pRT commented Aug 31, 2013

From @xdg

On Fri, Aug 30, 2013 at 10​:39 PM, Brian Fraser <fraserbn@​gmail.com> wrote​:

It strikes me that this and non-ascii modules are in the same bag; Perl has
no way of knowing
how a module's filename will be represented by the filesystem, so it just
charges forward in
hope that things will work out.

Acme​::require​::case "solves" this by asking the parent directory for a
list of files and trying to match it up. This, of course, has a fair
degree of extra overhead. I'm not sure if a similar change should be
made in core or not. If so, it should be optional, I would think.

I'm fine with the ticket staying open. Just because we don't have a
good answer doesn't mean we should ignore the problem.

David

--
David Golden <xdg@​xdg.me>
Take back your inbox! → http​://www.bunchmail.com/
Twitter/IRC​: @​xdg

@p5pRT
Copy link
Author

p5pRT commented Aug 31, 2013

From victor@vsespb.ru

2013/8/31 David Golden <xdg@​xdg.me>

On Fri, Aug 30, 2013 at 10​:39 PM, Brian Fraser <fraserbn@​gmail.com> wrote​:

It strikes me that this and non-ascii modules are in the same bag; Perl
has

Non-ascii modules can suffer also from Unicode normalization (NFC/NFD) on
MacOSX and some ZFS filesystems.

Acme​::require​::case "solves" this by asking the parent directory for a
list of files and trying to match it up. This, of course, has a fair
degree of extra overhead.

Possible optimization would be to determine case-sensitiveness of
filesystem by "testing" it (i.e. if one require 'Strict' -
also try to load 'strict' and 'stRICT' and if those are found and node_id
matches, that means current directory is case-insensitive).
Next, get the device_id for directory, and remember that whole device is
case-insensitive.

Then use Acme​::require​::case algorithm for directories on case-insensitive
devices only.

David

--
David Golden <xdg@​xdg.me>
Take back your inbox! → http​://www.bunchmail.com/
Twitter/IRC​: @​xdg

@p5pRT
Copy link
Author

p5pRT commented Sep 2, 2013

From david@kineticode.com

On Aug 30, 2013, at 7​:40 PM, Brian Fraser via RT <perlbug-followup@​perl.org> wrote​:

On Fri, Aug 30, 2013 at 10​:37 PM, James E Keenan via RT <perlbug-followup@​perl.org> wrote​:

There has been no further discussion in this ticket in over 10 years.

Man, I’m *old*.

I believe this should stay open. The bug is still present and leads to unintuitive behavior​:
'use Strict;' might end up loading strict.pm or Strict.pm, depending on the order of @​INC.

I agree it is still present, but it could perhaps be closed by documenting how all this stuff is supposed to work.

Best,

David

@theory
Copy link
Contributor

theory commented Jan 31, 2020

Almost seventeen years! Damn. I'm impressed the community elects to keep something like this open all that time, despite not much complaint, always with the thought to properly resolve it at some point. Thanks Perl for such a resilient process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants