From SignWriting@users.sourceforge.net

Created by signwriting@sourceforge.sf.net

Perl 5.6.1, Sun OS 5.9, both x86 and SPARC.

Hi,

I'm using perl on the SourceForge Compile Farm for portable access to
various facilities. For some reason, the "flock" operation does not
work properly (you can't say that it fails, because the problem is
that it always succeeds). It works properly on all the other
machines in the Compile Farm, even the old ones, so it must be
something specific to the implementation on Solaris.

I'm trying to use flock to provide exclusive access to resources. If
the lock fails (some other process has the resource), this process
should exit as the process holding the lock will eventually take care
of things. As a result, the lock flags are LOCK_EX|LOCK_NB and I've
verified that they have the values given in the man page (2 and 4,
respectively).

I can see that Solaris does not have an flock primitive system call,
so perl is presumably using fcntl internally to simulate flock. As
far as I can tell, the fcntl system call is working correctly, so the
problem seems to be internal to perl itself.

Despite what is reported below, 'perl --version' claims it is 5.6.1,
not 5.8. There's a 5.8 version installed, but it's not in the
default PATH. However, even forcing the 5.8.0 binary to be used does
not change the outcome; the flock always succeeds, so I can't get
excusionary access.

I'd appreciate any insight you might have as to what the actual
problem is and how I can fix it. Note that 'upgrade' is not a
solution (the systems are not under my control) nor is any soution
that involves installing my own version of perl (I don't have the
space).

I'd appreciate your prompt consideration to this report; I'm
completely stalled until I can get exclusionary access to my
resources. Thank you.

Flags:
     category=library
     severity=critical

Site configuration information for perl v5.8.0:

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
   Platform:
     osname=solaris, osvers=2.9, archname=sun4-solaris
     uname='sunos solaris 5.9 generic sun4u sparc sunw,ultra-5_10 '
     config_args='-Dcc=gcc -B/usr/ccs/bin/'
     hint=recommended, useposix=true, d_sigaction=define
     usethreads=undef use5005threads=undef useithreads=undef  
usemultiplicity=undef
     useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
     use64bitint=undef use64bitall=undef uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='gcc -B/usr/ccs/bin/', ccflags ='-fno-strict-aliasing - 
D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
     optimize='-O',
     cppflags='-fno-strict-aliasing'
     ccversion='', gccversion='3.1', gccosandvers='solaris2.9'
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
     ivtype='long', ivsize=4, nvtype='double', nvsize=8,  
Off_t='off_t', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='gcc -B/usr/ccs/bin/', ldflags =' -L/usr/local/lib '
     libpth=/usr/local/lib /usr/lib /usr/ccs/lib
     libs=-lsocket -lnsl -ldb -ldl -lm -lc
     perllibs=-lsocket -lnsl -ldl -lm -lc
     libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
     cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'

Locally applied patches:



@INC for perl v5.8.0:
     /usr/local/lib/perl5/5.8.0/sun4-solaris
     /usr/local/lib/perl5/5.8.0
     /usr/local/lib/perl5/site_perl/5.8.0/sun4-solaris
     /usr/local/lib/perl5/site_perl/5.8.0
     /usr/local/lib/perl5/site_perl
     .


Environment for perl v5.8.0:
     HOME=/home/users/s/si/signwriting
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=/home/users/s/si/signwriting/bin:/usr/bin:/bin:/usr/sbin:/ 
sbin:/usr/local/bin:/usr/local/sbin
     PERL_BADLANG (unset)
     SHELL=/bin/bash

From abe@ztreet.demon.nl

Op een mooie winterdag (Thursday 19 January 2006 20​:29),schreef SignWriting
(via RT)​:

# New Ticket Created by SignWriting
# Please include the string​: [perl #38282]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=38282 >

This is a bug report for perl from signwriting@​sourceforge.sf.net,
generated with the help of perlbug 1.34 running under perl v5.8.0.

-----------------------------------------------------------------
[Please enter your report here]

Perl 5.6.1, Sun OS 5.9, both x86 and SPARC.

Hi,

I'm using perl on the SourceForge Compile Farm for portable access to
various facilities. For some reason, the "flock" operation does not
work properly (you can't say that it fails, because the problem is
that it always succeeds). It works properly on all the other
machines in the Compile Farm, even the old ones, so it must be
something specific to the implementation on Solaris.

Hmmm... I checked on my Solaris 10 x86 and a Solaris 9 sparc, and found it
always fails to aquire the lock (with filehandles opened for read).

I'm trying to use flock to provide exclusive access to resources. If
the lock fails (some other process has the resource), this process
should exit as the process holding the lock will eventually take care
of things. As a result, the lock flags are LOCK_EX|LOCK_NB and I've
verified that they have the values given in the man page (2 and 4,
respectively).

This bit is from perldoc -f flock​:

  Note that the fcntl(2) emulation of flock(3) requires that
  FILEHANDLE be open with read intent to use LOCK_SH and requires
  that it be open with write intent to use LOCK_EX.

And this is from man flock on Solaris​:

  int flock( fd, operation);
  int fd, operation;

DESCRIPTION
  flock() applies or removes an advisory lock on the file
  associated with the file descriptor fd. The compatibility
  version of flock() has been implemented on top of fcntl(2)
  locking.
  It does not provide complete binary compatibility.
...
  Read permission is required on a file to obtain a shared
  lock, and write permission is required to obtain an
  exclusive lock. Locking a segment that is already locked by
  the calling process causes the old lock type to be removed
  and the new lock type to take effect.

So you'll need to open the filehandle with write permission ('+<' or '>>' will
do).
This was my testprogram​:

~/klad$ cat try.pl
#! /usr/bin/perl
use strict;
use warnings;

use Fcntl qw( :flock );

my $fn = $0;

my $out = '';
open my $fh, "+< $fn" or die "cannot open($fn)​: $!";
if ( flock $fh, LOCK_EX | LOCK_NB ) {
  print "Got lock, fire '$0' off again\n";
  $out = qx/$^X $0/;
} else {
  print "Could not get a lock ($fh/$fn)​:-(\n";
}
close $fh;
print $out =~ /Could not get a lock/ ? "ok" : "not ok";
print " # second lock failed\n";

~/klad$ perl try.pl
Got lock, fire 'try.pl' off again
ok # second lock failed

HTH +
Good luck,

Abe
--
NI-S> All in all not one of my better bits of code.
EINSUFFICIENTPARANOIA?

Remember, just because you don't believe that everyone is out to get

you, it doesn't actually mean that they are not.
  -- Nicholas Clark on p5p @​ 20020522

The RT System itself - Status changed from 'new' to 'open'

From SignWriting@users.sourceforge.net

On Jan 21, 2006, at 3​:03 AM, Abe Timmerman via RT wrote​:

... you'll need to open the filehandle with write permission ('+<'
or '>>' will do).

I do open the file for write; the particular fragment to acquire the
lock is this​:

sysopen(LOG, $file, O_WRONLY|O_CREAT, 0644) || die "Can't open $file​:
$!";
flock(LOG, LOCK_EX|LOCK_NB) || exit;

The first invocation of the script should acquire the lock; the
second one should exit without comment. Instead, both invocations
proceed as if they owned the lock.

Could it be a bug specific to SourceForge? Did you try it on their
Solaris machines?

Thanks for looking at this.

From abe@ztreet.demon.nl

Op een mooie winterdag (Saturday 21 January 2006 13​:13),schreef SignWriting​:

On Jan 21, 2006, at 3​:03 AM, Abe Timmerman via RT wrote​:

... you'll need to open the filehandle with write permission ('+<'
or '>>' will do).

I do open the file for write; the particular fragment to acquire the
lock is this​:

sysopen(LOG, $file, O_WRONLY|O_CREAT, 0644) || die "Can't open $file​:
$!";
flock(LOG, LOCK_EX|LOCK_NB) || exit;

The first invocation of the script should acquire the lock; the
second one should exit without comment. Instead, both invocations
proceed as if they owned the lock.

This (from perldoc -f flock) might also be a clue​:

  Note also that some versions of "flock" cannot lock
  things over the network; you would need to use the
  more system-specific "fcntl" for that. If you like

My test program with a file on an NFS mount hangs on Solaris 9 sparc (perl
5.6.1) althoug my Solaris 10 x86 (perl 5.6.1 && 5.8.4) doesn't have a problem
with that.

Can you try this? It passes on my Solaris boxes​:

#! /usr/bin/perl
use strict;
use warnings;

use Test​::More tests => 3;
use Fcntl qw( :DEFAULT :flock );

my $fn = 'test.lock';
my $label = shift || 'ORG';
my $out = 'No CPY run';

local *LOG;
my $res = sysopen LOG, $fn, O_WRONLY | O_CREAT, 0644;
ok $res, "sysopen succeded";
SKIP​: {
  !$res and skip "Cannot continue", 2;
  if ( flock LOG, LOCK_EX | LOCK_NB ) {
  if ( $label eq 'CPY' ) {
  fail "$label did get first lock";
  fail "$label shouls fail";
  } else {
  pass "Got the (first) lock";
  $out = qx/$^X $0 CPY/;
  like $out, qr/^ok \d - CPY did not get first lock/m,
  "Concurrent lock faild";
  }
  } else {
  if ( $label eq 'CPY' ) {
  pass "$label did not get first lock";
  pass "$label did not get second lock";
  } else {
  fail "$label did not get first lock";
  fail "$label did not get second lock";
  }
  }
}
print LOG $out;
close LOG;

__END__

Could it be a bug specific to SourceForge? Did you try it on their
Solaris machines?

Dunno, never been there (I already have access to too many boxes ;)

HTH +
Good luck,

Abe
--
[About a VMS specific path]
G.Berry> Can I point at you if anything other than this goes wrong? ;-)

Everyone else will, so I don't see why you shouldn't.
  -- Nicholas Clark on p5p @​ 2004-07-20

From SignWriting@users.sourceforge.net

On Jan 22, 2006, at 3​:37 AM, Abe Timmerman via RT wrote​:

Op een mooie winterdag ...

"On a beautiful? winter day"? My Dutch is almost non-existent, but
if I got the spirit of that translation, it's gorgeous in Southern
California, too; it's expected to be 22 degrees with clear skies.

My test program with a file on an NFS mount hangs on Solaris 9
sparc (perl 5.6.1) althoug my Solaris 10 x86 (perl 5.6.1 && 5.8.4)
doesn't have a problem with that.

I'm using local disk, so I don't have the hang because the remote
daemon isn't answering. In particular, I'm using files in /tmp
(hmmm, which is a memory partition, but I just checked, and it fails
with /var/tmp, a real disk partition, as well).

Late news​: a Python script with the same logic works on Solaris, even
in /tmp. If Python was installed on all the compile farm machines, I
could use that instead of perl (but it isn't, so I can't---but maybe
a mixed solution would work...).

Can you try this? It passes on my Solaris boxes​:

I changed 'test.lock' to '/tmp/test.lock' to simulate my actual
environment more closely. The script works on a Linux box and a BSD
box, but gets this on Solaris​:

$ perl abe.pl
Can't locate Test/More.pm in @​INC (@​INC contains​: /usr/perl5/5.6.1/
lib/sun4-solaris-64int /usr/perl5/5.6.1/lib /usr/perl5/site_perl/
5.6.1/sun4-solaris-64int /usr/perl5/site_perl/5.6.1 /usr/perl5/
site_perl /usr/perl5/vendor_perl/5.6.1/sun4-solaris-64int /usr/perl5/
vendor_perl/5.6.1 /usr/perl5/vendor_perl .) at abe.pl line 5.
BEGIN failed--compilation aborted at abe.pl line 5.

Obviously, the Test module isn't installed. (Unsurprising; they're
pretty poor about installing secondary modules.)

My script is so short, I'll just send it to you, plus a shell script
that demonstrates the problem. Maybe it will help you in your
testing. (Yeah, it's horrible, but I'm not much of a perl hacker.
Advice to improve it would be appreciated.)

-------- Begin LockLog.pl
if (@​ARGV < 2) {
  print "Too few args; usage​: perl LockLog.pl file cmd
[arg ...]\n";
  exit
}
# first arg is lock file (also used as log file)
$file = shift @​ARGV;
# open the file and lock it; if we fail, just go away
use Fcntl; use Fcntl '​:flock';
sysopen(LOG, $file, O_WRONLY|O_CREAT, 0644) || die "Can't open
$file​: $!";
flock(LOG, LOCK_EX|LOCK_NB) || exit;
# we got the file; rewind and truncate it so output will be at the
begining
seek(LOG, 0, 0) || die "Can't rewind $file​: $!";
truncate(LOG, 0) || die "Can't trunc $file​: $!";
# redirect stdout and stderr to log file
open STDOUT, ">&LOG" || die "Can't redirect stdout to
$file​: $!";
open STDERR, ">&LOG" || die "Can't redirect stderr to
$file​: $!";
# exec remainder of parameters as a command
exec @​ARGV;
-------- Begin lock.test.sh
TMP=/tmp
#TMP=/var/tmp
echo short test to demonstrate that a command is executed
rm -f "$TMP"/foo "$TMP"/foo.bar
perl LockLog.pl "$TMP"/foo touch "$TMP"/foo.bar
if test -f "$TMP"/foo.bar
then echo "yup, it worked"
else echo "OOPS, didn't execute command"; exit 1
fi
echo real test to see if we have an exclusionary lock
perl LockLog.pl "$TMP"/foo sleep 10 &
sleep 3
rm -f "$TMP"/foo.bar
perl LockLog.pl "$TMP"/foo touch "$TMP"/foo.bar
wait
if test -f "$TMP"/foo.bar
then echo "OOPS; this one didn't work"
else echo "yup, it worked"
fi
rm -f "$TMP"/foo "$TMP"/foo.bar
-------- End scripts

From abe@ztreet.demon.nl

Op een mooie winterdag (Sunday 22 January 2006 15​:13),schreef SignWriting​:

On Jan 22, 2006, at 3​:37 AM, Abe Timmerman via RT wrote​:

Op een mooie winterdag ...

"On a beautiful? winter day"?

Yup.

                           My Dutch is almost non\-existent\, but

if I got the spirit of that translation, it's gorgeous in Southern
California, too; it's expected to be 22 degrees with clear skies.

Knowing Dutch weather, that is intended with a wink ;-) And it was rather grim
today.

My test program with a file on an NFS mount hangs on Solaris 9
sparc (perl 5.6.1) althoug my Solaris 10 x86 (perl 5.6.1 && 5.8.4)
doesn't have a problem with that.

I'm using local disk, so I don't have the hang because the remote
daemon isn't answering. In particular, I'm using files in /tmp
(hmmm, which is a memory partition, but I just checked, and it fails
with /var/tmp, a real disk partition, as well).

Late news​: a Python script with the same logic works on Solaris, even
in /tmp. If Python was installed on all the compile farm machines, I
could use that instead of perl (but it isn't, so I can't---but maybe
a mixed solution would work...).

Can you try this? It passes on my Solaris boxes​:

I changed 'test.lock' to '/tmp/test.lock' to simulate my actual
environment more closely. The script works on a Linux box and a BSD
box, but gets this on Solaris​:

$ perl abe.pl
Can't locate Test/More.pm in @​INC (@​INC contains​: /usr/perl5/5.6.1/
lib/sun4-solaris-64int /usr/perl5/5.6.1/lib /usr/perl5/site_perl/
5.6.1/sun4-solaris-64int /usr/perl5/site_perl/5.6.1 /usr/perl5/
site_perl /usr/perl5/vendor_perl/5.6.1/sun4-solaris-64int /usr/perl5/
vendor_perl/5.6.1 /usr/perl5/vendor_perl .) at abe.pl line 5.
BEGIN failed--compilation aborted at abe.pl line 5.

Sorry about that; 5.6.1 didn't ship whith Test​::More (but it should be with
the 5.8.0 you mentioned)

Obviously, the Test module isn't installed. (Unsurprising; they're
pretty poor about installing secondary modules.)

My script is so short, I'll just send it to you, plus a shell script
that demonstrates the problem. Maybe it will help you in your
testing. (Yeah, it's horrible, but I'm not much of a perl hacker.
Advice to improve it would be appreciated.)

-------- Begin LockLog.pl
if (@​ARGV < 2) {
print "Too few args; usage​: perl LockLog.pl file cmd
[arg ...]\n";
exit
}
# first arg is lock file (also used as log file)
$file = shift @​ARGV;
# open the file and lock it; if we fail, just go away
use Fcntl; use Fcntl '​:flock';

That could be a single use statement​:

  use Fcntl qw( :DEFAULT :flock );

sysopen(LOG, $file, O_WRONLY|O_CREAT, 0644) || die "Can't open
$file​: $!";
flock(LOG, LOCK_EX|LOCK_NB) || exit;
# we got the file; rewind and truncate it so output will be at the
begining
seek(LOG, 0, 0) || die "Can't rewind $file​: $!";
truncate(LOG, 0) || die "Can't trunc $file​: $!";
# redirect stdout and stderr to log file
open STDOUT, ">&LOG" || die "Can't redirect stdout to
$file​: $!";

You might need 'or' and not '||' there when there are no parenthesis aroud
open's arguments...

open STDERR, ">&LOG" || die "Can't redirect stderr to
$file​: $!";

same here.

More important, the lock seems go after the dup; so they should go before the
flock() call.

# exec remainder of parameters as a command
exec @​ARGV;

This is where it goes wrong, but I don't know why. It will work with system()
or qx// though.

perldoc -f exec​:
  If there is more than one argument in LIST, or if
  LIST is an array with more than one value, calls
  execvp(3) with the arguments in LIST. If there is

man 3 execvp​:
  File descriptors open in the calling process image remain
  open in the new process image, except for those whose
  close-on-exec flag FD_CLOEXEC is set; (see fcntl(2)). For
  those file descriptors that remain open, all attributes of
  the open file description, including file locks, remain
  unchanged.

If Python manages to do that, I guess there is a perl bug.

HTH +
Good luck,

Abe
--
Well, dereferencing that (as CvGV() would do) leads nowhere.  Or, as
the Ten Commandments for C Programmers quoth, "Thou shalt not follow
the NULL pointer, for chaos and madness await thee at its end."
  -- Jarkko Hietaniemi on p5p @​ 2002-05-14

From SignWriting@users.sourceforge.net

On Jan 22, 2006, at 10​:16 AM, Abe Timmerman via RT wrote​:

Knowing Dutch weather, that is intended with a wink ;-) And it was
rather grim today.

It was beautiful when we were there last year, and Amsterdam was very
enjoyable---except that the canal boat ran aground and it took us two
hours to get free, making us late back to the bus.

Sorry about that; 5.6.1 didn't ship whith Test​::More (but it should
be with the 5.8.0 you mentioned)

Indeed, and forcing 5.8.0, the run succeeds and gets the same result
you got. So it's not testing what's failing for me.

This is where it goes wrong, but I don't know why. It will work
with system() or qx// though.

I replaced exec with system and it still failed. Watching it with top
(1) verified that the parent process stayed around, so if the parent
process still had the lock, it should have still worked.

More important, the lock seems go after the dup; so they should go
before the flock() call.

This turns out to be the insight needed. Duping the files first
didn't work, but after some fiddling, I was able to make it work by
creating the lock on STDOUT. That eventually led me to the man page
on dup(2), which clearly says what is shared when duping a
filehandle; locks aren't mentioned. (The equivalent man pages on
Linux and BSD specifically include locks.) So it's a system
incompatibility; Solaris does it differently from the rest of the world.

I don't know why perl drops the lock when duping the filehandle, but
it appears to be related to duping it to something like STDOUT or
STDERR. Perl tries to preserve these if the operation fails, but if
it's explicitly closed before the dup, it works. That's really
strange, but that's what appears to be what's happening. It might be
worth looking at the code to see why it occurs.

In any event, once I was armed with this information, I was able to
create a Python script with the same symptoms, which verifies the
conclusion.

This problem might be worth a note in perldoc when it's talking about
duping a filehandle in the description of the open function. Some
part of it is discussed a little later, but the focus of that
paragraph is being parsimonious of filehandles (and, in fact, flock
(A) and flock(B) _will_ work, since the lock is on the underlying
filesystem object; it's trying to make the point that the lock sets
could be independent, but it's muddled).

I think this bug report as it currently stands can be retired. There
presumably needs to be a bug report to get the documentation updated,
but that doesn't need me. I'm now in a good position with both perl
and python scripts to do the job, so I can proceed with what I was
doing. Many thanks for your help.

From @jkeenan

On Mon Jan 23 11​:37​:21 2006, SignWriting@​users.sourceforge.net wrote​:

On Jan 22, 2006, at 10​:16 AM, Abe Timmerman via RT wrote​:

Sorry about that; 5.6.1 didn't ship whith Test​::More (but it should
be with the 5.8.0 you mentioned)

Indeed, and forcing 5.8.0, the run succeeds and gets the same result
you got. So it's not testing what's failing for me.

This is where it goes wrong, but I don't know why. It will work
with system() or qx// though.

I replaced exec with system and it still failed. Watching it with top
(1) verified that the parent process stayed around, so if the parent
process still had the lock, it should have still worked.

More important, the lock seems go after the dup; so they should go
before the flock() call.

This turns out to be the insight needed. Duping the files first
didn't work, but after some fiddling, I was able to make it work by
creating the lock on STDOUT. That eventually led me to the man page
on dup(2), which clearly says what is shared when duping a
filehandle; locks aren't mentioned. (The equivalent man pages on
Linux and BSD specifically include locks.) So it's a system
incompatibility; Solaris does it differently from the rest of the world.

I don't know why perl drops the lock when duping the filehandle, but
it appears to be related to duping it to something like STDOUT or
STDERR. Perl tries to preserve these if the operation fails, but if
it's explicitly closed before the dup, it works. That's really
strange, but that's what appears to be what's happening. It might be
worth looking at the code to see why it occurs.

In any event, once I was armed with this information, I was able to
create a Python script with the same symptoms, which verifies the
conclusion.

This problem might be worth a note in perldoc when it's talking about
duping a filehandle in the description of the open function. Some
part of it is discussed a little later, but the focus of that
paragraph is being parsimonious of filehandles (and, in fact, flock
(A) and flock(B) _will_ work, since the lock is on the underlying
filesystem object; it's trying to make the point that the lock sets
could be independent, but it's muddled).

I think this bug report as it currently stands can be retired. There
presumably needs to be a bug report to get the documentation updated,
but that doesn't need me. I'm now in a good position with both perl
and python scripts to do the job, so I can proceed with what I was
doing. Many thanks for your help.

This RT was filed more than six years ago. The original poster
recommended that the ticket be closed. Is there anyone -- particularly
anyone conversant with Perl on recent Solarises -- who feels there is
some issue for which it should be kept open?

Thank you very much.
Jim Keenan

From @jkeenan

On Sun Oct 21 16​:10​:45 2012, jkeenan wrote​:

This RT was filed more than six years ago. The original poster
recommended that the ticket be closed. Is there anyone -- particularly
anyone conversant with Perl on recent Solarises -- who feels there is
some issue for which it should be kept open?

Thank you very much.
Jim Keenan

No further comments made in nearly three weeks. Closing ticket.

Thank you very much.
Jim Keenan

@jkeenan - Status changed from 'open' to 'resolved'