Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filetests sometimes do not set _ #9069

Closed
p5pRT opened this issue Oct 16, 2007 · 30 comments
Closed

filetests sometimes do not set _ #9069

p5pRT opened this issue Oct 16, 2007 · 30 comments

Comments

@p5pRT
Copy link

p5pRT commented Oct 16, 2007

Migrated from rt.perl.org#46463 (status was 'resolved')

Searchable as RT46463$

@p5pRT
Copy link
Author

p5pRT commented Oct 16, 2007

From Mark@Overmeer.net

Created by mark@overmeer.net

On the Perl5.10.0 of today (patch 32115), I see filetest errors which
are probably in bleadperl for quite some time... as the cpan-testers
show me.

Somewhere in a large program, I do something like this​:

sub ... {
  foreach ...
  { -w "/etc/passwd";
  warn (-f _ ? "file\n" : "directory\n");
  }
}

And the result is "directory"!!
After -r w x R W X and t, the password file is reported as directory.

with -o e z s f d u g k b c l T B M A C, it is reported as file.

I have tried, but was not successful in reproducing the error in a
small code example. But maybe above is already sufficiently clear
to someone. Or, maybe someone knows a way to investigate this further.

  Greetings, MarkOv

Perl Info

Flags:
    category=core
    severity=medium

This perlbug was built using Perl 5.10.0 - Tue Oct 16 15:34:41 CEST 2007

Summary of my perl5 (revision 5 version 10 subversion 0 patch 32115) configuration:
  Platform:
    osname=linux, osvers=2.6.13-15.16-bigsmp, archname=i686-linux
    uname='linux earth 2.6.13-15.16-bigsmp #1 smp fri jun 8 15:35:39 utc 2007 i686 i686 i386 gnulinux '
    config_args='-de'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.0.2 20050901 (prerelease) (SUSE Linux)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.3.5.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.3.5'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib'


Characteristics of this binary (from libperl): 
  Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP
                        USE_LARGE_FILES USE_PERLIO
  Locally applied patches:
	DEVEL
  Built under linux
  Compiled at Oct 16 2007 15:36:41
  @INC:
    /usr/local/lib/perl5/5.10.0/i686-linux
    /usr/local/lib/perl5/5.10.0
    /usr/local/lib/perl5/site_perl/5.10.0/i686-linux
    /usr/local/lib/perl5/site_perl/5.10.0
    .

Environment for perl 5.8.7:
    HOME=/home/markov
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/markov/shared/bin:~/shared/bin:/home/markov/shared/bin:~/shared/bin:/home/markov/shared/bin:~/shared/bin:/home/markov/shared/bin:/home/markov/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/games:/opt/gnome/bin:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:./bin:./bin:./bin:./bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Oct 17, 2007

From Mark@Overmeer.net

* Mark Overmeer (perlbug-followup@​perl.org) [071016 14​:04]​:

# Please include the string​: [perl #46463]
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=46463 >

On the Perl5.10.0 of today (patch 32115), I see filetest errors which
are probably in bleadperl for quite some time... as the cpan-testers
show me.

sub ... {
foreach ...
{ -w "/etc/passwd";
warn (-f _ ? "file\n" : "directory\n");
}
}

Hum... the error report was even more right than I expected it to be...
After a long bug-hunt, I figured out that it is not '-w' itself which
is misbehaving (the file test flags), but it is the module 'filetest'.

The following script reports 'directory' twice​:

  #!/usr/local/bin/perl5.10.0
  use filetest 'access';

  -d '/etc';
  -w '/etc/passwd';
  warn -f _ ? "file\n" : "directory\n";
  warn -d _ ? "directory\n" : "file\n";

Apparently, -r/-w/-x/-R/-W/-X implemented in filetest to not
set '_' anymore, where they did so in perl5.8.8
--
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Oct 17, 2007

From @rgs

On 17/10/2007, Mark Overmeer <mark@​overmeer.net> wrote​:

Hum... the error report was even more right than I expected it to be...
After a long bug-hunt, I figured out that it is not '-w' itself which
is misbehaving (the file test flags), but it is the module 'filetest'.

The following script reports 'directory' twice​:

#!/usr/local/bin/perl5.10.0
use filetest 'access';

-d '/etc';
-w '/etc/passwd';
warn -f _ ? "file\n" : "directory\n";
warn -d _ ? "directory\n" : "file\n";

Apparently, -r/-w/-x/-R/-W/-X implemented in filetest to not
set '_' anymore, where they did so in perl5.8.8

Looking at the code, I think this bug was already present in 5.8.x.
But maybe the access code wasn't compiled in your version.

What does
perl -V​:.*access.*
show for each of your perls ?

@p5pRT
Copy link
Author

p5pRT commented Oct 17, 2007

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Oct 17, 2007

From Mark@Overmeer.net

* Rafael Garcia-Suarez (rgarciasuarez@​gmail.com) [071017 08​:49]​:

On 17/10/2007, Mark Overmeer <mark@​overmeer.net> wrote​:

Apparently, -r/-w/-x/-R/-W/-X implemented in filetest to not
set '_' anymore, where they did so in perl5.8.8

Looking at the code, I think this bug was already present in 5.8.x.
But maybe the access code wasn't compiled in your version.

What does
perl -V​:.*access.*
show for each of your perls ?

markov@​earth​:&perl> perl5.10.0 -V​:.*access.*
d_access='define';
d_accessx='undef';
d_eaccess='undef';
i_sysaccess='undef';

markov@​earth​:&perl> perl5.8.7 -V​:.*access.*
d_access='define';
d_accessx='undef';
d_eaccess='undef';
i_sysaccess='undef';

The code which now fails was ran by cpan-testers on many many
systems, without problems in 5.6 and 5.8, but fails on all 5.10/5.9.5

According to the cpantesters results
(http​://cpantesters.perl.org/show/Mail-Box.html), it looks like something
broke between 5.9.2 and 5.9.5.

By the way​: quite some 5.9.5 tests reported successes, but actually
were failures.
--
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Oct 18, 2007

From @andk

On Wed, 17 Oct 2007 11​:02​:18 +0200, Mark Overmeer <mark@​overmeer.net> said​:

  > The code which now fails was ran by cpan-testers on many many
  > systems, without problems in 5.6 and 5.8, but fails on all 5.10/5.9.5

  > According to the cpantesters results
  > (http​://cpantesters.perl.org/show/Mail-Box.html), it looks like something
  > broke between 5.9.2 and 5.9.5.

Binary search reveals the test fails since 25986

Change 25986 by nicholas@​nicholas-saigo on 2005/11/04 13​:02​:42

  ftrwrite, ftrexec, fteread, ftewrite and fteexec can all be merged
  with Perl_pp_ftrread().

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Oct 18, 2007

From @nwc10

On Thu, Oct 18, 2007 at 05​:29​:36AM +0200, Andreas J. Koenig wrote​:

On Wed, 17 Oct 2007 11​:02​:18 +0200, Mark Overmeer <mark@​overmeer.net> said​:

The code which now fails was ran by cpan-testers on many many
systems, without problems in 5.6 and 5.8, but fails on all 5.10/5.9.5

According to the cpantesters results
(http​://cpantesters.perl.org/show/Mail-Box.html), it looks like something
broke between 5.9.2 and 5.9.5.

Binary search reveals the test fails since 25986

Change 25986 by nicholas@​nicholas-saigo on 2005/11/04 13​:02​:42

ftrwrite\, ftrexec\, fteread\, ftewrite and fteexec can all be merged
with Perl\_pp\_ftrread\(\)\.

Mmm, I'm not sure about this. The script starts failing for me on 5.8.1

$ cat 46463
#!/usr/local/bin/perl5.10.0
use filetest 'access';

-d '/etc';
-w '/etc/passwd';
warn -f _ ? "file\n" : "directory\n";
warn -d _ ? "directory\n" : "file\n";
$ ~/Reference/5.8.0/bin/perl5.8.0 46463
file
file
$ ~/Reference/5.8.1/bin/perl5.8.1 46463
directory
directory
$ ~/Reference/5.8.2/bin/perl5.8.2 46463
directory
directory
$ ~/Reference/5.8.3/bin/perl5.8.3 46463
directory
directory
$ ~/Reference/5.8.4/bin/perl5.8.4 46463
directory
directory
$ ~/Reference/5.8.5/bin/perl5.8.5 46463
directory
directory
$ ~/Reference/5.8.6/bin/perl5.8.6 46463
directory
directory
$ ~/Reference/5.8.7/bin/perl5.8.7 46463
directory
directory
$ ~/Reference/5.8.8/bin/perl5.8.8 46463
directory
directory

It's also failing on 5.6.2​:

$ /usr/local/perl/5.6.2/bin/perl ~/p4perl/perl/46463
directory
directory

Nicholas Clark

@p5pRT
Copy link
Author

p5pRT commented Oct 18, 2007

From @andk

On Thu, 18 Oct 2007 16​:07​:18 +0100, Nicholas Clark <nick@​ccl4.org> said​:

  > On Thu, Oct 18, 2007 at 05​:29​:36AM +0200, Andreas J. Koenig wrote​:

On Wed, 17 Oct 2007 11​:02​:18 +0200, Mark Overmeer <mark@​overmeer.net> said​:

The code which now fails was ran by cpan-testers on many many
systems, without problems in 5.6 and 5.8, but fails on all 5.10/5.9.5

According to the cpantesters results
(http​://cpantesters.perl.org/show/Mail-Box.html), it looks like something
broke between 5.9.2 and 5.9.5.

Binary search reveals the test fails since 25986

Change 25986 by nicholas@​nicholas-saigo on 2005/11/04 13​:02​:42

ftrwrite, ftrexec, fteread, ftewrite and fteexec can all be merged
with Perl_pp_ftrread().

  > Mmm, I'm not sure about this. The script starts failing for me on 5.8.1

  > $ cat 46463
  > #!/usr/local/bin/perl5.10.0
  > use filetest 'access';

  > -d '/etc';
  > -w '/etc/passwd';
  > warn -f _ ? "file\n" : "directory\n";
  > warn -d _ ? "directory\n" : "file\n";

Ahh, this is a different bug then. Mark told us that he could not
reproduce it in a small test case. What I tested was the test suite of
Mail​::Box. My binary search appears solid to me. Both perls were
compiled yesterday and all parameters were equal.

----Program----
eval q{use Mail​::Box 2.075};
print $@​ ? "N/A" : "OK";
print "\n";

----Output of .../plC1Mru/perl-5.8.0@​25985/bin/perl----
OK

----EOF ($?='0')----
----Output of .../pGlFMnW/perl-5.8.0@​25986/bin/perl----
N/A

----EOF ($?='0')----

The N/A comes from the fact that the test suite failed.

Whereas your program says "directory" on both 2598[56].

A binary search for your program leads us to the interval between
19367 and 19449. I'm running the binary search for this one over night...

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Oct 18, 2007

From Mark@Overmeer.net

* Andreas J. Koenig (andreas.koenig.7os6VVqR@​franz.ak.mind.de) [071018 19​:46]​:

$ cat 46463
#!/usr/local/bin/perl5.10.0
use filetest 'access';

-d '/etc';
-w '/etc/passwd';
warn -f _ ? "file\n" : "directory\n";
warn -d _ ? "directory\n" : "file\n";

Ahh, this is a different bug then. Mark told us that he could not
reproduce it in a small test case. What I tested was the test suite of
Mail​::Box.

You may be very right. This script does shows the same problem and
solution, but I suspect that some other reason hid the bug for a long
time.

Remove the 'use filetest', or use f.i. -s i.s.o -w does solve the
problems in my program and in this example.

In the program the situation is far more complex​: the '-d' is run
in some other method. There must something what was hiding the
bug.

I have tried and tried, but no luck on a small example which does
work on 5.8.7 and not on 5.10.0... only my Mail​::Box test. For the
people who want to have a look at it​:

  unpack Mail​::Box-*gz
  cd Mail-Box-*
  perl5.8.7 ./test.pl -v 40mbox/50 --> success
  perl5.10.0 ./test.pl -v 40mbox/50 --> failure

tests 3 and 4 fail. They call Mail​::Box​::Mbox​::listSubFolders().
The problem appears in "-f _", line 146 of lib/Mail/Box/Mbox.pm

My hopes are that a fix for above bug will fix the second.
--
Thanks for investigating,

  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Oct 19, 2007

From @andk

On Thu, 18 Oct 2007 21​:46​:50 +0200, andreas.koenig.7os6VVqR@​franz.ak.mind.de (Andreas J. Koenig) said​:

  > A binary search for your program leads us to the interval between
  > 19367 and 19449. I'm running the binary search for this one over night...

---Program----
use filetest 'access';

-d '/etc';
-w '/etc/passwd';
warn -f _ ? "file\n" : "directory\n";
warn -d _ ? "directory\n" : "file\n";

----Output of .../pwzF3Lg/perl-5.8.0@​19391/bin/perl----
file
file

----EOF ($?='0')----
----Output of .../p3rjkvF/perl-5.8.0@​19392/bin/perl----
directory
directory

----EOF ($?='0')----

  Change 19392 by jhi@​kosh on 2003/05/03 05​:33​:04

  Salvage 'use filetest "access"' from compiletime to runtime.

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Oct 21, 2007

From @nwc10

On Wed, Oct 17, 2007 at 10​:22​:58AM +0200, Mark Overmeer wrote​:

Hum... the error report was even more right than I expected it to be...
After a long bug-hunt, I figured out that it is not '-w' itself which
is misbehaving (the file test flags), but it is the module 'filetest'.

The following script reports 'directory' twice​:

#!/usr/local/bin/perl5.10.0
use filetest 'access';

-d '/etc';
-w '/etc/passwd';
warn -f _ ? "file\n" : "directory\n";
warn -d _ ? "directory\n" : "file\n";

Apparently, -r/-w/-x/-R/-W/-X implemented in filetest to not
set '_' anymore, where they did so in perl5.8.8

On Fri, Oct 19, 2007 at 02​:20​:26AM +0200, Andreas J. Koenig wrote​:

Change 19392 by jhi@&#8203;kosh on 2003/05/03 05&#8203;:33&#8203;:04

    Salvage 'use filetest "access"' from compiletime to runtime\.

It seems that Jarkko actually fixed a bug here - I'd infer that access() wasn't
being called when it was expected that it would be.

I don't believe that the current behaviour is a behavioural bug.
The documentation could be clearer, however.

It is documented that _ is set to the last stat.
It is documented that under filetest, -r/-w/-x/-R/-W/-X use access() rather
than the mode bits from the file.

Hence one does not need to call stat(), so perl does not. *That*
implementation efficiency is not documented.

So I think that the current behaviour is preferable. If we changed it so that
-r/-w/-x/-R/-W/-X also call stat() as well as access(), then we give everyone
inefficiency that they can't avoid. Whereas the current behaviour is merely
caveat emptor on anyone wishing to take advantage of the _ filehandle.

I'm curious - on a modern Unix system with decent OS caching, how much does
not calling stat again really save?

Nicholas Clark

@p5pRT
Copy link
Author

p5pRT commented Oct 21, 2007

From Mark@Overmeer.net

* Nicholas Clark (nick@​ccl4.org) [071021 20​:53]​:

On Wed, Oct 17, 2007 at 10​:22​:58AM +0200, Mark Overmeer wrote​:

Apparently, -r/-w/-x/-R/-W/-X implemented in filetest to not
set '_' anymore, where they did so in perl5.8.8

On Fri, Oct 19, 2007 at 02​:20​:26AM +0200, Andreas J. Koenig wrote​:

Change 19392 by jhi@&#8203;kosh on 2003/05/03 05&#8203;:33&#8203;:04
    Salvage 'use filetest "access"' from compiletime to runtime\.

It seems that Jarkko actually fixed a bug here - I'd infer that access()
wasn't being called when it was expected that it would be.

Oh, that's a good reason for that patch, and does explain things.

I don't believe that the current behaviour is a behavioural bug.
The documentation could be clearer, however.

It is documented that _ is set to the last stat.
It is documented that under filetest, -r/-w/-x/-R/-W/-X use access() rather
than the mode bits from the file.

Well, perldoc -f stat says​:

  if "stat" is passed the special filehandle consisting of an
  underline, no stat is done, but the current contents of the
  stat structure from the last "stat", "lstat", or filetest are
  returned.

I do read "filetest" here... (perl5.10.0)

Hence one does not need to call stat(), so perl does not. *That*
implementation efficiency is not documented.

On many places, Perl prefers qualilty over performance. I would even
say that 'use filetest' should be the default on all platforms which
support it (which support ACLs).

That file test behavioral change when 'use filetest' is given somewhere
else in the file is at least very unexpected and error-prone.

On the other hand, the whole use of -w / -r /-x should be avoided in
general​: open to write and handle the failure otherwise you get into
race conditions. So, the actual number of access()/stat() doubles in
good code should be minimal... I have such a rare case where I need to
use -r (I think), because I am checking the filesystem structure.

So I think that the current behaviour is preferable. If we changed it so that
-r/-w/-x/-R/-W/-X also call stat() as well as access(), then we give everyone
inefficiency that they can't avoid. Whereas the current behaviour is merely
caveat emptor on anyone wishing to take advantage of the _ filehandle.

I'm curious - on a modern Unix system with decent OS caching, how much does
not calling stat again really save?

Probably only an extra context switch. (Do you known how many a simple
'use' makes?)
--
Regards,
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Nov 2, 2007

From Mark@Overmeer.net

Guys,

What will happen with the reported bug? I don't like the silence
over the subject, after Nicks findings.

IMO, we should decide between​:

  (1) change the docs, reflecting the special behavior of
  '_'/stat when filetest is active.

  (2) produce an error when _ is used when filetest is enabled.

  (3) change the internal implementation of -r/-w/-x to use
  access() when available on the platform. The performance
  benefit for "_" will be lost on these flags.

IMO, (3) [filetest is always used] is strongly preferred​: if you test
for readibility, you have to take ACLs into account when available.
Imperfection is worse than performance loss, and -r should be used
very sparsely in the first place.

But this probably shouldn't go into 5.10... At least the first
option must go in a.s.a.p, I think.

On Wed, Oct 17, 2007 at 10​:22​:58AM +0200, Mark Overmeer wrote​:

Apparently, -r/-w/-x/-R/-W/-X implemented in filetest to not
set '_' anymore, where they did so in perl5.8.8

* Nicholas Clark (nick@​ccl4.org) [071021 20​:53]​:

I don't believe that the current behaviour is a behavioural bug.
The documentation could be clearer, however.

It is documented that _ is set to the last stat.
It is documented that under filetest, -r/-w/-x/-R/-W/-X use access()
rather than the mode bits from the file.

* Mark Overmeer (mark@​overmeer.net) [071022 00​:47]​:

Well, perldoc -f stat says​:

if "stat" is passed the special filehandle consisting of an
underline, no stat is done, but the current contents of the
stat structure from the last "stat", "lstat", or filetest are
returned.

I do read "filetest" here... (perl5.10.0)

On many places, Perl prefers qualilty over performance. I would even
say that 'use filetest' should be the default on all platforms which
support it (which support ACLs).
--
Regards,

  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Nov 2, 2007

From @rgs

On 02/11/2007, Mark Overmeer <mark@​overmeer.net> wrote​:

Guys,

What will happen with the reported bug? I don't like the silence
over the subject, after Nicks findings.

IMO, we should decide between​:

(1) change the docs, reflecting the special behavior of
'_'/stat when filetest is active.

That's the best we can do for 5.10.0 I think. A good place to patch
the docs would be filetest.pm (and a note in perfunc/-X)

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

From Mark@Overmeer.net

* Rafael Garcia-Suarez (rgarciasuarez@​gmail.com) [071102 15​:27]​:

On 02/11/2007, Mark Overmeer <mark@​overmeer.net> wrote​:

(1) change the docs, reflecting the special behavior of
'_'/stat when filetest is active.

That's the best we can do for 5.10.0 I think. A good place to patch
the docs would be filetest.pm (and a note in perfunc/-X)

Just at the last moment, two proposed updates to perlfunc/-X and
filetest.pm about the discussed issues.
--
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

From Mark@Overmeer.net

Inline Patch
--- pod/perlfunc.pod.old	2007-11-30 11:19:32.000000000 +0100
+++ pod/perlfunc.pod	2007-11-30 11:33:46.000000000 +0100
@@ -341,13 +341,16 @@
 	#...
     }
 
-The interpretation of the file permission operators C<-r>, C<-R>,
-C<-w>, C<-W>, C<-x>, and C<-X> is by default based solely on the mode
-of the file and the uids and gids of the user.  There may be other
-reasons you can't actually read, write, or execute the file.  Such
-reasons may be for example network filesystem access controls, ACLs
-(access control lists), read-only filesystems, and unrecognized
-executable formats.
+The interpretation of the file permission operators C<-r>, C<-R>, C<-w>,
+C<-W>, C<-x>, and C<-X> is by default based solely on the mode of the
+file and the uids and gids of the user.  There may be other reasons you
+can't actually read, write, or execute the file: for example network
+filesystem access controls, ACLs (access control lists), read-only
+filesystems, and unrecognized executable formats.  The use of these six
+specific operators is usually a mistake in the first place, because it
+may be open to race conditions.  See the filetest pragma manual L<filetest>
+for a better approach and more details, but be warned that the filetest
+pragma may cause C<_> not to be set! 
 
 Also note that, for the superuser on the local filesystems, the C<-r>,
 C<-R>, C<-w>, and C<-W> tests always return 1, and C<-x> and C<-X> return 1
@@ -355,16 +358,6 @@
 may thus need to do a stat() to determine the actual mode of the file,
 or temporarily set their effective uid to something else.
 
-If you are using ACLs, there is a pragma called C<filetest> that may
-produce more accurate results than the bare stat() mode bits.
-When under the C<use filetest 'access'> the above-mentioned filetests
-will test whether the permission can (not) be granted using the
-access() family of system calls.  Also note that the C<-x> and C<-X> may
-under this pragma return true even if there are no execute permission
-bits set (nor any extra execute permission ACLs).  This strangeness is
-due to the underlying system calls' definitions.  Read the
-documentation for the C<filetest> pragma for more information.
-
 Note that C<-s/a/b/> does not do a negated substitution.  Saying
 C<-exp($foo)> still works as expected, however--only single letters
 following a minus are interpreted as file tests.

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

From @jhi

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-11/msg01085.html

In principle I approve clarifications but it seems to me that you are
losing some information in this edit, and I do not think EXCLAMATION
POINTS to perlfunc!!! :-)

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

From Mark@Overmeer.net

* Jarkko Hietaniemi (jhi@​iki.fi) [071130 12​:55]​:

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-11/msg01085.html

In principle I approve clarifications but it seems to me that you are
losing some information in this edit, and I do not think EXCLAMATION
POINTS to perlfunc!!! :-)

Please be specific
--
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

From @jhi

What exactly are you trying to add / change? Something about the
semantics of the -X vs _, but what?

On Nov 30, 2007 8​:11 AM, Mark Overmeer <mark@​overmeer.net> wrote​:

* Jarkko Hietaniemi (jhi@​iki.fi) [071130 12​:55]​:

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2007-11/msg01085.html

In principle I approve clarifications but it seems to me that you are
losing some information in this edit, and I do not think EXCLAMATION
POINTS to perlfunc!!! :-)

Please be specific
--
MarkOv

------------------------------------------------------------------------
Mark Overmeer MSc MARKOV Solutions
Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

--
There is this special biologist word we use for 'stable'. It is
'dead'. -- Jack Cohen

@p5pRT
Copy link
Author

p5pRT commented Nov 30, 2007

From Mark@Overmeer.net

On Nov 30, 2007 8​:11 AM, Mark Overmeer <mark@​overmeer.net> wrote​:

* Jarkko Hietaniemi (jhi@​iki.fi) [071130 12​:55]​:

In principle I approve clarifications but it seems to me that you are
losing some information in this edit, and I do not think EXCLAMATION
POINTS to perlfunc!!! :-)

* Jarkko Hietaniemi (jhi@​iki.fi) [071130 14​:06]​:

What exactly are you trying to add / change? Something about the
semantics of the -X vs _, but what?

The changed text should reflect the debates on the porters lists, which
we had a month or so ago. 5.10.0 had a changed behavior with my tests,
and this has something to do with filetest. Probably, some bug was
removed which has hidden a problem with filetest for me.

in short

  -r $filename
  should be used sparsely​: chance on race conditions in case
  of combinations with other functions on that file.

  no filetest 'access' # default
  -r $filename
  tests the read-bit in mode field of stat(), it does not
  promise readability (on most platforms)

  use filetest 'access'
  -r $fn
  does check for readability correctly, but doesn't use _

Under the filetest pragma, -r/-w/-x do not set '_'. There was no
indication of this "feature" in the manuals. And, yes, this behavior
is very dangerous, because the pragma is typically added later, when
porting to a platform which requires it.

IMO, filetest 'access' should be default, and we should try to trick
'_' by adding an extra stat. But that's 5.10.1 or 5.12 material.

The patches I provide include two major changes.
- The filetest pragma is described in a bit more detail, and reorganized.
  Nothing thrown away, afaik.
- the perlfunc/-X has a warning and reference to the filetest pragma. A
  paragraph which contained a condensed version of the filetest text
  was removed, because it is too short to explain such complex concept.

Changes to the text are (of course) possible. At least, this suggestion
attempts to describe the situation better then the current text.
--
Regards,

  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Dec 3, 2007

From @nwc10

On Fri, Nov 30, 2007 at 05​:05​:31PM +0100, Mark Overmeer wrote​:

Under the filetest pragma, -r/-w/-x do not set '_'. There was no
indication of this "feature" in the manuals. And, yes, this behavior
is very dangerous, because the pragma is typically added later, when
porting to a platform which requires it.

Changing code always has the possibility of adding a bug. This should be
documented with the access pragma.

IMO, filetest 'access' should be default, and we should try to trick
'_' by adding an extra stat. But that's 5.10.1 or 5.12 material.

My opinion is that forcing everyone to take a speed hit to protect against
sloppy coding/testing by maintenance programmers is worse.

I assume that it would be possible for the implementation of the filetest
pragma to flag that _ is now stale, and so make anything that accesses it
(at least) warn.

It, presumably, would also be possible to have the filetest pragma be able
to do either, in which case we then get down to splitting hairs about which
is the default. That's possibly the best solution.

Nicholas Clark

@p5pRT
Copy link
Author

p5pRT commented Dec 3, 2007

From Mark@Overmeer.net

* Nicholas Clark (nick@​ccl4.org) [071203 14​:52]​:

On Fri, Nov 30, 2007 at 05​:05​:31PM +0100, Mark Overmeer wrote​:

Under the filetest pragma, -r/-w/-x do not set '_'. There was no
indication of this "feature" in the manuals. And, yes, this behavior
is very dangerous, because the pragma is typically added later, when
porting to a platform which requires it.

Changing code always has the possibility of adding a bug. This should be
documented with the access pragma.

It is documented in with the access pragma.

IMO, filetest 'access' should be default, and we should try to trick
'_' by adding an extra stat. But that's 5.10.1 or 5.12 material.

My opinion is that forcing everyone to take a speed hit to protect against
sloppy coding/testing by maintenance programmers is worse.

Depends on the hit. Doing a stat() is mainly a wait-for-disk. A stat()
after an access() does problably not need to wait for the disk​: I cannot
imagine an access() implementation which does not require the stat() info,
so it will be in cash.

A speed it of a few percent (extra context switch maybe) is not bad.
certainly if people use -r/-w/-x as they should​: very sparsely.

I assume that it would be possible for the implementation of the filetest
pragma to flag that _ is now stale, and so make anything that accesses it
(at least) warn.

Of course, possible. Not so easy to get right.
By giving a run-time error, you reduce the power of _ in some situations,
which is harder to explain then simply make it DWIM always.

It, presumably, would also be possible to have the filetest pragma be able
to do either, in which case we then get down to splitting hairs about which
is the default. That's possibly the best solution.

I am always proud on Perl, that it does not bother people with string
length limitiations, and all those horrors. And, these great benefits
all have a penalty. No joy without some pain. IMO, we should take the
pain (in some form) here as well.
--
Regards,

  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Dec 4, 2007

From @hvds

Nicholas Clark <nick@​ccl4.org> wrote​:
:On Fri, Nov 30, 2007 at 05​:05​:31PM +0100, Mark Overmeer wrote​:
:> IMO, filetest 'access' should be default, and we should try to trick
:> '_' by adding an extra stat. But that's 5.10.1 or 5.12 material.
:
:My opinion is that forcing everyone to take a speed hit to protect against
:sloppy coding/testing by maintenance programmers is worse.

It need not be a stat()-sized speed hit if after an access we set a flag
near the stored C<_> stat buffer along with details of the relevant file,
and then only do the actual stat() call if we need to.

That reduces the universal hit to checking a flag on access of C<_>, and
clearing it (and maybe freeing some stuff) for anything that would normally
invalidate C<_>, but it is also a rather more complex approach, and clearly
not for 5.10.0.

Hugo

@p5pRT
Copy link
Author

p5pRT commented Dec 5, 2007

From Mark@Overmeer.net

* hv@​crypt.org (hv@​crypt.org) [071204 13​:27]​:

Nicholas Clark <nick@​ccl4.org> wrote​:
:On Fri, Nov 30, 2007 at 05​:05​:31PM +0100, Mark Overmeer wrote​:
:> IMO, filetest 'access' should be default, and we should try to trick
:> '_' by adding an extra stat. But that's 5.10.1 or 5.12 material.
:
:My opinion is that forcing everyone to take a speed hit to protect against
:sloppy coding/testing by maintenance programmers is worse.

It need not be a stat()-sized speed hit if after an access we set a flag
near the stored C<_> stat buffer along with details of the relevant file,
and then only do the actual stat() call if we need to.

With all these solutions, you introduce a danger

  -r $filename;
  unlink $filename;
  if(-f _) { ... }

The more I think about it, the more convinced I am that -r/-w/-x in any
combination with '_' is a very bad idea. Readability is (in general)
not defined by the mode bits, but by all kinds of environmental things
in which one component is the mode bits.

Maybe, we should move into the direction where we add the provide
core functions (in File​::Spec?)
  is_readable
  is_writable
  is_executable
which always use 'access' when available, and kill filetest.pm

The -r/-w/-x should than be documented as
  test the read/write/execute bits in the inode. To test a file
  for readablity, use File​::Spec​::is_readable(). Most applications
  which test these bit are flawed, see ...
--
Regards,
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Dec 5, 2007

From @rgs

On 05/12/2007, Mark Overmeer wrote​:

It need not be a stat()-sized speed hit if after an access we set a flag
near the stored C<_> stat buffer along with details of the relevant file,
and then only do the actual stat() call if we need to.

With all these solutions, you introduce a danger

-r $filename;
unlink $filename;
if(-f _) { ... }

The more I think about it, the more convinced I am that -r/-w/-x in any
combination with '_' is a very bad idea. Readability is (in general)
not defined by the mode bits, but by all kinds of environmental things
in which one component is the mode bits.

One of the goals (in my mind) of introducing stacked filetest operators
in 5.10 was to discourage explicit usage of _.

Maybe, we should move into the direction where we add the provide
core functions (in File​::Spec?)

Oh, no, not File​::Spec for that !

is_readable
is_writable
is_executable
which always use 'access' when available, and kill filetest.pm

The -r/-w/-x should than be documented as
test the read/write/execute bits in the inode. To test a file
for readablity, use File​::Spec​::is_readable(). Most applications
which test these bit are flawed, see ...

That wouldn't solve the race condition problem. I think the only sure
way to know whether a file is readable at time T, is to read it...

@p5pRT
Copy link
Author

p5pRT commented Dec 5, 2007

From Mark@Overmeer.net

* Rafael Garcia-Suarez (rgarciasuarez@​gmail.com) [071205 10​:33]​:

Maybe, we should move into the direction where we add the provide
core functions (in File​::Spec?)

Oh, no, not File​::Spec for that !

Well, I think that the module is outdated as well... but it looks
like the logical location.

is_readable
is_writable
is_executable
which always use 'access' when available, and kill filetest.pm

The -r/-w/-x should than be documented as
test the read/write/execute bits in the inode. To test a file
for readablity, use File​::Spec​::is_readable(). Most applications
which test these bit are flawed, see ...

That wouldn't solve the race condition problem. I think the only sure
way to know whether a file is readable at time T, is to read it...

This is very true... although there are situations that the readability
test is the only thing you need. These very rare situations have something
to do with filesystem administration. Of course, in these situations, you
could do

  if(open F, '<', $fn)
  { close F;
  ... do something
  }

but it is nicer to be able to do

  if(is_readable $fn)
  { ... do something
  }

For instance, the auto-completion of bash only suggests files which can
be read... probably access() is faster than open()/close().
--
Regards,

  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Dec 6, 2007

From @rgs

On 30/11/2007, Mark Overmeer <mark@​overmeer.net> wrote​:

Just at the last moment, two proposed updates to perlfunc/-X and
filetest.pm about the discussed issues.

Thanks, I've applied modified/augmented/rephrased versions of those to
blead as #32587.

@p5pRT
Copy link
Author

p5pRT commented Nov 7, 2008

@smpeters - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Nov 7, 2008
@p5pRT
Copy link
Author

p5pRT commented Nov 8, 2008

From solutions@overmeer.net

* Steve Peters via RT (perlbug-followup@​perl.org) [081107 20​:39]​:

According to our records, your request regarding
"filetests sometimes do not set _"
has been resolved.

Well, if you declare that "documenting a problem" is equivalent to
resolving the problem, then yes. But the core of the problem is
still present.

It could be solved by calling stat() also when in "use filetest 'access';"
is enabled. The performance penalty is minimal, because people should
not use -r/-w/-x anyway​: in general that is a bad idea.
--
Regards,
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant