From david@davidfavor.com

Created by david@davidfavor.com

The -w operator is broken for GSA directories.

Here is the output of a test script which shows the problem.
The script follows. Regular and AFS directories work correctly.
GSA directories may return high order bits which appears to
break both perl and bash.

You may contact me directly with a patch and I'll try it and
return the results.

Thanks.

======= test script results =======

reg​: okay​: 200​: .
reg​: owr : 0x000000200
reg​: mode​: 0x200042751
dir​: ksh​: writable
dir​: ksh93​: writable
dir​: zsh​: writable
reg​: bash​: not writable
reg​: perl​: not writable

afs​: okay​: 200​: /afs/apd.pok.ibm.com/u/dfavor
afs​: owr : 0x000000200
afs​: mode​: 0x000040777
dir​: ksh​: writable
dir​: ksh93​: writable
dir​: zsh​: writable
dir​: bash​: writable
afs​: perl​: writable

gsa​: okay​: 200​: /gsa/ausgsa/home/d/f/dfavor/favor-tools/archive/aix510
gsa​: owr : 0x000000200
gsa​: mode​: 0x200042751 <<<<<<======= note high order bit set
dir​: ksh​: writable
dir​: ksh93​: writable
dir​: zsh​: writable
gsa​: bash​: not writable
gsa​: perl​: not writable

======= test script =======

#!/usr/local/bin/perl -w

use strict;
use Switch;

use File​::stat;

my @​dirs = qw(. /afs/apd.pok.ibm.com/u/dfavor /gsa/ausgsa/home/d/f/dfavor/favor-tools/archive/aix510);

# owner writable bit from /usr/include/sys/mode.h
my $S_IWUSR = 0200;

foreach my $dir (@​dirs) {

  my $type;

  switch ($dir) {
  case /gsa/ { $type = 'gsa'; }
  case /afs/ { $type = 'afs'; }
  else { $type = 'reg'; }
  }

  my $stats = stat($dir);

  # is stat() returning correct bits?
  if ($stats->mode && $S_IWUSR != 0) {
  printf("$type​: okay​: %lo​: %s\n", ($stats->mode && $S_IWUSR), $dir);
  }

  printf("$type​: owr : 0x%09O\n",$S_IWUSR);
  printf("$type​: mode​: 0x%09O\n",$stats->mode);

  foreach my $shell ('ksh','ksh93','zsh','bash') {

  my $res = system("$shell -c '[[ -w $dir ]] && echo \"dir​: $shell​: writable\"'");
  $res /= 256;
  if ($res != 0) {
  print "$type​: $shell​: not writable\n"
  }

  }

  if (-w $dir) { print "$type​: perl​: writable\n"; }
  else { print "$type​: perl​: not writable\n"; }

  print "\n";

}

Flags:
    category=core
    severity=critical

Site configuration information for perl v5.8.5:

Configured by dfavor at Tue Jul 20 12:15:22 CDT 2004.

Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
  Platform:
    osname=aix, osvers=5.1.0.0, archname=aix-thread-multi
    uname='aix aasc24 1 5 000f2b4c4c00 '
    config_args='-Dprefix=/usr/local/perl-5.8.5-RC2 -ders -Dusedevel -Uuse64bitall -Doptimize=-O2 -Dusenm -Dusethreads -Duseshrplib -Dd_dosuid -Dd_socket -Dd_msg -Dd_sem -Dd_shm -Dbison -Dyacc=bison -y -Dcf_email=david@davidfavor.com'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc_r', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_INIT -I/usr/local/include -q32 -D_LARGE_FILES -qlonglong',
    optimize='-O2',
    cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE -D_POSIX_SOURCE -qmaxmem=-1 -qnoansialias -DUSE_NATIVE_DLOPEN -DNEED_PTHREAD_INIT -I/usr/local/include'
    ccversion='6.0.0.7', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='ld', ldflags =' -brtl -L/usr/local/lib -b32'
    libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
    libs=-lbind -lnsl -lgdbm -ldbm -ldb -ldl -lld -lm -lcrypt -lpthreads -lc -lbsd
    perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lpthreads -lc -lbsd
    libc=, so=a, useshrplib=true, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -bE:/usr/local/perl-5.8.5-RC2/lib/5.8.5/aix-thread-multi/CORE/perl.exp'
    cccdlflags=' ', lddlflags='-bhalt:4 -bM:SRE -bI:$(PERL_INC)/perl.exp -bE:$(BASEEXT).exp -bnoentry -lpthreads -lc -L/usr/local/lib'

Locally applied patches:
    RC2


@INC for perl v5.8.5:
    /usr/local/perl-5.8.5-RC2/lib/5.8.5/aix-thread-multi
    /usr/local/perl-5.8.5-RC2/lib/5.8.5
    /usr/local/perl-5.8.5-RC2/lib/site_perl/5.8.5/aix-thread-multi
    /usr/local/perl-5.8.5-RC2/lib/site_perl/5.8.5
    /usr/local/perl-5.8.5-RC2/lib/site_perl
    .


Environment for perl v5.8.5:
    HOME=/home/dfavor
    LANG=en_US
    LANGUAGE (unset)
    LC__FASTMSG=true
    LD_LIBRARY_PATH (unset)
    LIBPATH=/usr/local/lib:/opt/freeware/lib
    LOGDIR (unset)
    PATH=.:/home/dfavor/bin:/home/dfavor/binibm:/usr/local/perl-5.8.5-RC2/bin:/gsa/ausgsa/home/d/f/dfavor/favor-tools/scripts/stable:/gsa/ausgsa/home/d/f/dfavor/favor-tools/scripts/dev:/gsa/ausgsa/home/d/f/dfavor/favor-tools/build/bin:/usr/local/bin:/usr/local/Acrobat5/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/dt/bin:/usr/bin/X11:/sbin:/usr/afs/bin:/usr/java130/jre/bin:/usr/java130/bin:/usr/vac/bin:/usr/vacpp/bin:/gsa/ausgsa/home/d/f/dfavor/favor-tools/stable/aix510/gcc/bin:/home/dfavor/binibm:/usr/gsa/bin:/usr/contrib/bin:/usr/prod/bin:/afs/austin/local/bin:/.../austin.ibm.com/fs/projects/service/usr/bin/retain:/usr/tctools:/home/dfavor/local-retain:/.../austin.ibm.com/fs/projects/aix_tools/bin:/home/dfavor/ha:/opt/freeware/bin:/.../austin.ibm.com/fs/projects/hacmp/support/bin:/.../austin.ibm.com/fs/projects/hacmp/support/meetings/bin
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zsh

From @rgs

david@​davidfavor.com (via RT) wrote​:

The -w operator is broken for GSA directories.

Here is the output of a test script which shows the problem.
The script follows. Regular and AFS directories work correctly.
GSA directories may return high order bits which appears to
break both perl and bash.

What's a GSA filesystem ? where can we find docs about the behaviour of
the stat system calls on AIX for this kind of filesystem ? (google
wasn't very helpful this time, it seems)

The RT System itself - Status changed from 'new' to 'open'

From david@davidfavor.com

Rafael Garcia-Suarez via RT wrote​:

david@​davidfavor.com (via RT) wrote​:

The -w operator is broken for GSA directories.

Here is the output of a test script which shows the problem.
The script follows. Regular and AFS directories work correctly.
GSA directories may return high order bits which appears to
break both perl and bash.

What's a GSA filesystem ? where can we find docs about the behaviour of
the stat system calls on AIX for this kind of filesystem ? (google
wasn't very helpful this time, it seems)

GSA is the filesystem IBM is promoting to replace AFS and DFS. My initial
message shows an example which seems to indicate the problem is when high
order bits get set, perl seems to incorrectly to mask them off. On AIX there
are many high order bits defined in /usr/include/sys/mode.h which currently
include the following​:

  /* version information */
  #define S_INMOD 0xC0000000 /* normal mode */
  #define S_IXMOD 0x40000000 /* extended mode */

  #define S_IJRNL 0x04000000 /* journalled */
  #define S_IXACL 0x02000000 /* extended ACL */
  #define S_ITCB 0x01000000 /* Trusted Computing Base */
  #define S_ITP 0x00800000 /* Trusted Process */

GSA supports extended ACLS, so 0x02000000 is set automatically for
GSA directories.

AIX now supports extended ACLs on regular directories also, so when
extended ACLs on regular directories are defined, these will probably
fail also.

This is a can of worms. There are several ways to fix/bandage this​:

1) attempt to use the AIX extended ACL API everywhere permissions
  are checked

2) mask off high order bits and only use low order 3 octets

#1 is the most correct and complex. There is an abundance of info about
how to get and set ACLs. I've yet to find any description of how to test
them to see if an operation is permitted, before it's executed.

#2 is probably the best approach for now. This does mean if extended ACLs
deny an operation, errors will get generated.

Probably best to do #2 now and put #1 on the future todo list.

If someone can provide a patch I can test it or if you prefer, point me
to where the '-w foo' syntax gets converted to the stat lookup and I'll
try to figure out the correct way to mask off all but the lower three
octets 0x777 for permission tests.

Thanks for your assistance.

- David

--
Simple ways to look and feel your best ever!
http​://RadicalHealth.com - join@​radicalhealth.com

From @rgs

David Favor wrote​:

AIX now supports extended ACLs on regular directories also, so when
extended ACLs on regular directories are defined, these will probably
fail also.

This is a can of worms. There are several ways to fix/bandage this​:

1) attempt to use the AIX extended ACL API everywhere permissions
are checked

I think that's the correct way for the filetest pragma.

2) mask off high order bits and only use low order 3 octets

I think that's the correct way for the core.

#1 is the most correct and complex. There is an abundance of info about
how to get and set ACLs. I've yet to find any description of how to test
them to see if an operation is permitted, before it's executed.

#2 is probably the best approach for now. This does mean if extended ACLs
deny an operation, errors will get generated.

Probably best to do #2 now and put #1 on the future todo list.

If someone can provide a patch I can test it or if you prefer, point me
to where the '-w foo' syntax gets converted to the stat lookup and I'll
try to figure out the correct way to mask off all but the lower three
octets 0x777 for permission tests.

The precise internal function that implements -w is pp_ftewrite() in pp_sys.c.
It defers to Perl_cando() in doio.c the test for the S_IWUSR flag on the
stat buffer returned by a stat(2) system call. As it tests for "st_mode & S_IWUSR"
I don't see what's wrong at a first glance; but there is some AIX-specific
test mode (looking at the PERL_EFF_ACCESS_W_OK macro defined in pp_sys.c.)
This test mode is only used when the filetest pragma is in effect.

From david@davidfavor.com

For every pp_*() routine in pp_sys.c change​:

  if (cando(S_IXUSR, 1, &PL_statcache))
  RETPUSHYES;
  RETPUSHNO;

to​:

  #ifdef _AIX
  RETPUSHYES;
  #else
  if (cando(S_IWUSR, 1, &PL_statcache))
  RETPUSHYES;
  #endif

This seems to be correct as access()/accessx() has already determined
the operation's validity based on whatever credentials are currently
active, so there's no reason to call Perl_cando().

Also hard coding _AIX is a bad idea, as many operating systems use
extended ACLs. Better to say​:

  #ifdef (OSLIST)

and define OSLIST as all operating systems implementing access()/accessx().

An even simplier approach might be to simply say​:

  #ifdef PERL_EFF_ACCESS_W_OK
  RETPUSHYES;
  #else
  if (cando(S_IWUSR, 1, &PL_statcache))
  RETPUSHYES;
  #endif

I'll try to work this all out for AIX and send you a copy of the patch.
Let me know the correct syntax to create a patch file and I'll send it
along when I'm done.

Thanks for your assistance. Easy to see the problem, once you told me
where to look.

--
Simple ways to look and feel your best ever!
http​://RadicalHealth.com - join@​radicalhealth.com

From david@davidfavor.com

The precise internal function that implements -w is pp_ftewrite() in pp_sys.c.
It defers to Perl_cando() in doio.c the test for the S_IWUSR flag on the
stat buffer returned by a stat(2) system call. As it tests for "st_mode & S_IWUSR"
I don't see what's wrong at a first glance; but there is some AIX-specific
test mode (looking at the PERL_EFF_ACCESS_W_OK macro defined in pp_sys.c.)
This test mode is only used when the filetest pragma is in effect.

Here's what I found.

1) Code for pp_ftewrite() in pp_sys.c is correct. Depending on availability
  of access() or accessx(), the correct function is called.

2) Code for Perl_cando() in doio.c is broken. After all the hoops gone through
  in pp_sys.c using access/accessx, then Perl_cando() does a brute force check
  against uid and gid for permission checks.

This breaks for any file system with any type of extended ACLs like DFS, AFS
and GSA. The correct way to do this is to duplicate the code in pp_sys first
trying accessx(), then access() then brute force if neither of these exist.

The problem in all these cases is the uid and gid may or may not match the
local user's id. In cases where there is a mismatch, then accessx()/access()
should be called to check current credentials (DFS/GSA tickets, AFS tokens)
to see if the operation is permitted.

I've never been under the hood of perl, so am unsure if I should try and fix
this or if this should defer to someone with knowledge of the correct way to
put in this fix.

Let me know.

--
Simple ways to look and feel your best ever!
http​://RadicalHealth.com - join@​radicalhealth.com

From @rgs

David Favor wrote​:

Also hard coding _AIX is a bad idea, as many operating systems use
extended ACLs. Better to say​:

  \#ifdef \(OSLIST\)

and define OSLIST as all operating systems implementing access()/accessx().

An even simplier approach might be to simply say​:

  \#ifdef PERL\_EFF\_ACCESS\_W\_OK
     RETPUSHYES;
  \#else
     if \(cando\(S\_IWUSR\, 1\, &PL\_statcache\)\)
        RETPUSHYES;
  \#endif

That sounds like a good approach.

I'll try to work this all out for AIX and send you a copy of the patch.
Let me know the correct syntax to create a patch file and I'll send it
along when I'm done.

The output of diff -u is preferred (diff -c if your diff doesn't support
unified diffs).

From @rgs

I'm putting P5P in the mail loop again...

David Favor wrote​:

Attached is a patch to fix this problem on AIX. Both
filetest.t and stat.t pass with this fix. Also the
operators -[rwxRWX] seem to work correctly now.

How to get -[oO] is anyone's guess and I'm unsure if
there's even a reason to do this. One guess might be
to do something along the lines of​:

if \(stringify\($gsauid\) eq stringify\($nativeuid\)\)
   return MATCH;

and how to do this for each different file system type
is a can of worms. The other can of worms is when local
user 'foo' and GSA user 'foo' are really two different
people. Many worms...

I put liberal comments as to the reasoning behind this
patch in the hopes other perl code gurus can take my
contribution and refine it.

Many thanks for your assistance.

Thanks for your work :)

I've several questions :
- against which version of perl did you patch ? I assume 5.8.5
- "use filetest" is with your patch a no-op on AIX, since you
  provide an alternate implementation of pp_ft* for AIX.
- Your patch works only for -w "string", not for -w FILEHANDLE.
  -w FILEHANDLE (et alii) is implemented in Perl_my_stat;
  that's what the SvPOK(TOPs) check is for.
- why is the difference between ACC_SELF and ACC_INVOKER ?
  can't we just retrofit your change into the
  PERL_EFF_ACCESS_?_OK macros ?

From david@davidfavor.com

Rafael Garcia-Suarez wrote​:

I'm putting P5P in the mail loop again...

David Favor wrote​:

Attached is a patch to fix this problem on AIX. Both
filetest.t and stat.t pass with this fix. Also the
operators -[rwxRWX] seem to work correctly now.

How to get -[oO] is anyone's guess and I'm unsure if
there's even a reason to do this. One guess might be
to do something along the lines of​:

if (stringify($gsauid) eq stringify($nativeuid))
return MATCH;

and how to do this for each different file system type
is a can of worms. The other can of worms is when local
user 'foo' and GSA user 'foo' are really two different
people. Many worms...

I put liberal comments as to the reasoning behind this
patch in the hopes other perl code gurus can take my
contribution and refine it.

Many thanks for your assistance.

Thanks for your work :)

You're welcome... and... thanks may be premature. Several other
tests in the test suite failed last night when I did a full 'make test'
run. I'm investigating now and will let you know what I find.

I've several questions :
- against which version of perl did you patch ? I assume 5.8.5

5.8.5-RC2.

- "use filetest" is with your patch a no-op on AIX, since you
provide an alternate implementation of pp_ft* for AIX.
- Your patch works only for -w "string", not for -w FILEHANDLE.
-w FILEHANDLE (et alii) is implemented in Perl_my_stat;
that's what the SvPOK(TOPs) check is for.

Ah... Perhaps a perl guru should consider how best to arrange
for calls to accessx() on platforms which have accessx().

- why is the difference between ACC_SELF and ACC_INVOKER ?

ACC_SELF checks effective user and ACC_INVOKER checks real
user, so the ft[er]* entry points should use the appropriate
constant for the appropriate test.

can't we just retrofit your change into the
PERL_EFF_ACCESS_?_OK macros ?

The SvPOK(TOPs) check explains one thing I couldn't figure out.
In the filetest.t check the code path always fell through to​:

  result = my_stat()

so my fix turned off this test altogether, which is why filetest.t
works and other things fail. The real fix on AIX then is to use​:

  faccessx() - with file descriptors
  accessx() - with paths

Along the same lines there is another consideration. When stat()
and lstat() are called the rwx bits returned are meaningless for
extended ACLs. A line item ought to be put on the "todo list" for
a perl guts guru to consider how to modify these bits depending on
the return values of [f]accessx(). This is way off track with what
we're working on now and is best noted now so it's remembered.

_____

This patch appears to be more complicated then I first imagined as
there are two code paths​:

  # path #1 for file descriptors
  if ((PL_op->op_private & OPpFT_ACCESS) && SvPOK(TOPs)) {
  result = PERL_EFF_ACCESS_W_OK(POPpx);
  if (result == 0)
  RETPUSHYES;
  if (result < 0)
  RETPUSHUNDEF;
  RETPUSHNO;
  }

  # path #2 for file paths
  else
  result = my_stat();

This means all the PERL_EFF_ACCESS_[RWX]_OK() macros appear to be
incorrect. They should reference faccessx() for file desriptors.

Then my_stat() in doio.c should be rewritten to use accessx().

Along the lines of the the future patch I talked about concerning
stat() and lstat(), there are many places in doio.c where bits are
directly tested from these calls. These are meaningless in the case
of extended ACLs and will probably cause other subtle, hard to find
bugs in the future.

It might be worthwhile to consider some sort of wrapper for determining
path and file descriptor permissions rather than operating directly on
stat() and lstat() returns. This way changes could be made in one area
rather than scattered throughout the code.

_______

If someone has an idea of how to patch the current problem by modifying the
ft[re]* routines and my_stat(), they can send me the patch and I'll try it
with AFS, DFS, GSA and Native Extended ACLs and return the results.

Thanks for your assistance.

--
Simple ways to look and feel your best ever!
http​://RadicalHealth.com - join@​radicalhealth.com

From tammer@tammer.net

Hello,
there is no activity on this report since over 5 years. I think that
this ticket should be closed.

Bye
  Rainer

From david@davidfavor.com

Rainer Tammer via RT wrote​:

Hello,
there is no activity on this report since over 5 years. I think that
this ticket should be closed.

Bye
Rainer

Geez... yes. Do it!

--
Love feeling your best ever, all day, every day?
Click http​://RadicalHealth.com for the easy way!

@cpansprout - Status changed from 'open' to 'stalled'