Skip Menu |
Report information
Id: 132733
Status: open
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: zefram [at] fysh.org
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: library
Perl Version: 5.27.7
Fixed In: (no value)



CC: zefram [...] fysh.org
To: perlbug [...] perl.org
Subject: getcwd() fails to fail on Cygwin
Date: Wed, 17 Jan 2018 20:58:40 +0000
From: zefram [...] fysh.org
Download (untitled) / with headers
text/plain 4.6k
This is a bug report for perl from zefram@fysh.org, generated with the help of perlbug 1.41 running under perl 5.27.7. ----------------------------------------------------------------- [Please describe your issue here] I'm spinning this off from [perl #132648] because it's got too big. On Cygwin (not reflected in the perl -V below), it has been found that multiple implementations of the getcwd() functionality in Cwd return a non-error result in a situation where the current directory actually has no resolvable name. Specifically, if the current directory is rmdir()ed, then these subs return its former name, but stat() on that name yields ENOENT. getcwd() functions should also yield ENOENT in that case. This situation is exercised by the test script dist/PathTools/t/cwd_enoent.t which was added for [perl #132648], so adding that uncovered this preexisting bug. Both the XS implementation Cwd::getcwd() and _backtick_pwd() exhibit the faulty behaviour. The XS one uses the core's getcwd_sv(), which therefore also has the same bug. Fixing getcwd_sv() seems necessary, but due to dual-lifing Cwd::getcwd() can't rely on the core fix for it to achieve correct behaviour. The probable fix, which will be needed in multiple places, is to stat the pathname returned by pwd(1)/getcwd(3)/getcwd_sv() to check that it refers to the same file as ".". This extra logic should only be used on Cygwin. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=library severity=low module=Cwd --- Site configuration information for perl 5.27.7: Configured by zefram at Thu Dec 21 04:41:17 GMT 2017. Summary of my perl5 (revision 5 version 27 subversion 7) configuration: Platform: osname=linux osvers=3.16.0-4-amd64 archname=x86_64-linux-thread-multi uname='linux barba.rous.org 3.16.0-4-amd64 #1 smp debian 3.16.43-2+deb8u2 (2017-06-26) x86_64 gnulinux ' config_args='-des -Dprefix=/home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52 -Duselargefiles -Dusethreads -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dusedevel -Uversiononly -Ui_db' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='cc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2' optimize='-O2' cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='4.9.2' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='cc' ldflags =' -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.19.so so=so useshrplib=true libperl=libperl.so gnulibc_version='2.19' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E -Wl,-rpath,/home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52/lib/5.27.7/x86_64-linux-thread-multi/CORE' cccdlflags='-fPIC' lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong' --- @INC for perl 5.27.7: /home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52/lib/site_perl/5.27.7/x86_64-linux-thread-multi /home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52/lib/site_perl/5.27.7 /home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52/lib/5.27.7/x86_64-linux-thread-multi /home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52/lib/5.27.7 --- Environment for perl 5.27.7: HOME=/home/zefram LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/zefram/usr/perl/perl_install/perl-5.27.7-i64-f52/bin:/home/zefram/usr/perl/util:/home/zefram/pub/x86_64-unknown-linux-gnu/bin:/home/zefram/pub/common/bin:/usr/bin:/bin:/usr/local/bin:/usr/games PERL_BADLANG (unset) SHELL=/usr/bin/zsh
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
Date: Thu, 18 Jan 2018 19:12:28 +0100
From: Achim Gratz <Stromeko [...] nexgo.de>
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.1k
Zefram writes: Show quoted text
> On Cygwin (not reflected in the perl -V below), it has been found that > multiple implementations of the getcwd() functionality in Cwd return a > non-error result in a situation where the current directory actually > has no resolvable name. Specifically, if the current directory > is rmdir()ed, then these subs return its former name, but stat() > on that name yields ENOENT.
This exposes behaviour mandated by the underlying OS or rather the file system. The directory can't really be deleted until the last process having it as the cwd exits. Show quoted text
> getcwd() functions should also yield ENOENT in that case.
It's perfectly possible even on a fully conformant POSIX system that the two calls yield diverging results, just not as likely. Also on Linux you can easily get a string that is not reachable from the same process (somewhere in Linux 2.x this string will now be prepended with "(unreachable)", but you must be able to deal with ENOENT anyway). If you rely on racy behaviour you need to be prepared to get the wrong result once in a while. Show quoted text
> This situation is exercised by the test script > dist/PathTools/t/cwd_enoent.t which was added for [perl #132648], so > adding that uncovered this preexisting bug.
I'd say the bug here is in the assumption made by the test. Show quoted text
> Both the XS implementation Cwd::getcwd() and _backtick_pwd() exhibit > the faulty behaviour. The XS one uses the core's getcwd_sv(), which > therefore also has the same bug. Fixing getcwd_sv() seems necessary, > but due to dual-lifing Cwd::getcwd() can't rely on the core fix for it > to achieve correct behaviour. > > The probable fix, which will be needed in multiple places, is to stat the > pathname returned by pwd(1)/getcwd(3)/getcwd_sv() to check that it refers > to the same file as ".". This extra logic should only be used on Cygwin.
As the maintainer of the official Cygwin Perl I'd be inclined to remove such a fix since it would likely introduce considerable slowdown for no real gain. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
Date: Thu, 18 Jan 2018 22:01:53 +0000
From: Zefram <zefram [...] fysh.org>
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.7k
Achim Gratz wrote: Show quoted text
>This exposes behaviour mandated by the underlying OS or rather the file >system. The directory can't really be deleted until the last process >having it as the cwd exits.
Earlier in the discussion on [perl #132648] I thought that might be what's happening, because of some old knowledge about Windows. But the test output that Tony provided showed that it's not. As described in my paragraph that you quoted, stat() yields ENOENT for the directory's former name. rmdir() is not leaving the directory in place; it *is* removing it, at least from that location. It doesn't matter whether it's temporarily getting some other name, because no other name is arising. getcwd() is objectively erroneous in returning a name that cannot be used to refer to the directory. Show quoted text
>I'd say the bug here is in the assumption made by the test.
Which assumption exactly? The assumption that rmdir(), having indicated success, would have succeeded in making the directory unreachable through its original name? That assumption turns out to be true. Perhaps the assumption that a successful return from getcwd() would mean it's providing a usable name? That's not so much an assumption as the feature that's being tested. If it's not warranted to test for that, then we'd better add some caution to the getcwd() documentation, something like "the return value of this function doesn't actually mean anything". Show quoted text
>As the maintainer of the official Cygwin Perl I'd be inclined to remove >such a fix since it would likely introduce considerable slowdown for no >real gain.
If the decision is that getcwd() remain buggy on Cygwin, then I suppose we keep the test skip for Cygwin, and document "on Cygwin the return value of this function doesn't actually mean anything". -zefram
To: perl5-porters [...] perl.org
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
Date: Fri, 19 Jan 2018 20:57:21 +0100
From: Achim Gratz <Stromeko [...] nexgo.de>
Download (untitled) / with headers
text/plain 4.5k
Zefram writes: Show quoted text
> Achim Gratz wrote:
>>This exposes behaviour mandated by the underlying OS or rather the file >>system. The directory can't really be deleted until the last process >>having it as the cwd exits.
> > Earlier in the discussion on [perl #132648] I thought that might be > what's happening, because of some old knowledge about Windows. But the > test output that Tony provided showed that it's not. As described in > my paragraph that you quoted, stat() yields ENOENT for the directory's > former name. rmdir() is not leaving the directory in place; it *is* > removing it, at least from that location.
As always on Windows, things are a bit more complicated than that. I've discussed this with the Cygwin developers and I don't pretend to understand all of the details myself, but in a nutshell once you mark the directory for deletion the handle and the directory name can not be stat'ed anymore, but the directory is still enlisted in it's former parent (maybe in order to make it impossible to create a new directory with the same name), going away only after the last handle to it closes. You'd actually have to stat that entry (respectively ask Windows for its file information) to learn that it is no longer valid for almost all file operations, but that syscall is one of those that are quite expensive on Windows. Show quoted text
> It doesn't matter whether it's temporarily getting some other name, > because no other name is arising. getcwd() is objectively erroneous > in returning a name that cannot be used to refer to the directory.
From my reading of the corresponding man page POSIX doesn't mandate that the string returned is actually accessible by the time you use it and makes no explicit guarantee that it was when you made the call. Show quoted text
>>I'd say the bug here is in the assumption made by the test.
> > Which assumption exactly? The assumption that rmdir(), having indicated > success, would have succeeded in making the directory unreachable > through its original name? That assumption turns out to be true.
It seems to me you are making the assumption that the getcwd function creates an error condition and thus returns a NULL pointer rather than a string immediately when that directory becomes unreachable (which in the test code you know from the fact that rmdir has successfully returned). Your further expectation that this error condition ought to be ENOENT is explicitly mentioned as a GNU/Linux (glibc) extension and there actually is still "POSIX" fallback code in place in glibc that simply doesn't do that. POSIX just mandates that you can't make use of the name of a deleted directory should you happen to still hold onto it, so for instance you can't create any files there anymore or list its entries. Show quoted text
> Perhaps the assumption that a successful return from getcwd() would mean > it's providing a usable name? That's not so much an assumption as the > feature that's being tested.
Yet that feature is a property of the Linux syscall API if even that, not POSIX. I'm pretty sure you'd also run into trouble on a few network file systems with that assumption on GNU/Linux. Show quoted text
> If it's not warranted to test for that, then we'd better add some > caution to the getcwd() documentation, something like "the return > value of this function doesn't actually mean anything".
That may not be the worst outcome. It actually does mean something, but just as with most non-atomic, non-transactional file operations, you have to expect that once in a while you can't actually use the result as you thought you might and get an error at some later operation that you thought was guarded by an earlier one. Show quoted text
>>As the maintainer of the official Cygwin Perl I'd be inclined to remove >>such a fix since it would likely introduce considerable slowdown for no >>real gain.
> > If the decision is that getcwd() remain buggy on Cygwin, then I suppose > we keep the test skip for Cygwin, and document "on Cygwin the return > value of this function doesn't actually mean anything".
Well, the question here is if the gain outweighs the cost, given that Cygwin would try to more closely emulate GNU/Linux behaviour that is strictly valid only for local filesystems while no races are going on, so even GNU/Linux applications would be well advised to not rely on it. Note that for a complete fix it's not enough to add more checks in getcwd, the real fix would be in readdir and that would entail an extra syscall for each directory entry. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
To: perl5-porters [...] perl.org
Date: Sat, 20 Jan 2018 00:46:54 +0000
From: Zefram <zefram [...] fysh.org>
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
Download (untitled) / with headers
text/plain 694b
Achim Gratz wrote: Show quoted text
>From my reading of the corresponding man page POSIX doesn't mandate that >the string returned is actually accessible by the time you use it
That's a red herring. Obviously concurrent activity can in general render the result invalid between its generation and its use, but that's not the issue here. There are no race conditions in this case. Show quoted text
>makes no explicit guarantee that it was when you made the call.
"The getcwd() function shall place an absolute pathname of the current working directory in the array pointed to by buf". If the string doesn't actually resolve to the directory, then it is not "an absolute pathname of the current working directory". -zefram
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
Date: Sat, 20 Jan 2018 10:36:35 +0100
From: Achim Gratz <Stromeko [...] nexgo.de>
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.1k
Zefram writes: Show quoted text
> That's a red herring. Obviously concurrent activity can in general > render the result invalid between its generation and its use, but that's > not the issue here. There are no race conditions in this case.
Yes there is and that's the reason your test fails. The entry for cwd in its parent directory doesn't go away until the last file handle to it closes, which is why you can still access the name through the file handle to the cwd (that is what getcwd does). That is just how Windows' file system works and not some odd choice on the part of Cygwin that can be changed. In fact I've learned it's not the first time that it has been tried to get papered over in one way or the other. But these attempts have not been successful, either for not fixing the problem completely or for creating larger problems someplace else. Show quoted text
> "The getcwd() function shall place an absolute pathname of the current > working directory in the array pointed to by buf".
Which it dutifully does. Note the complete absence of _any_ discussion of what should happen when the cwd gets deleted even in the informative section. Now, I think it's preposterous to assume the POSIX folks didn't think about that case, so that omission is quite likely intentional to allow for system-dependent behaviour. Show quoted text
> If the string doesn't actually resolve to the directory, then it is > not "an absolute pathname of the current working directory".
It still does resolve to it and the directory entry is in fact still there if you go and look it up in the parent directory, you just can't use that name to do a successful stat or readdir anymore. Again, just skip that test on Cygwin (or mark it as expected to fail) since you are making assumptions that are generally expected to be true (assuming no other process works in the same directory where you do the testing), but are known not to be in this specific instance. There's nothing to do about it that wouldn't create a comparatively greater loss someplace else. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
To: perl5-porters [...] perl.org
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
From: Zefram <zefram [...] fysh.org>
Date: Sat, 20 Jan 2018 23:10:12 +0000
Download (untitled) / with headers
text/plain 1.4k
Achim Gratz wrote: Show quoted text
>Zefram writes:
>> not the issue here. There are no race conditions in this case.
> >Yes there is and that's the reason your test fails. The entry for cwd >in its parent directory doesn't go away until the last file handle to it >closes,
That's not a race condition. It's not dependent on any concurrent activity, and adding pauses to the code doesn't affect it. Show quoted text
> Note the complete absence of _any_ discussion >of what should happen when the cwd gets deleted even in the informative >section.
That is an unfortunate omission for getcwd(3), but the situation is discussed a bit for rmdir(2). There is explicit permission for rmdir() to yield EBUSY for a directory that is the root or cwd of any process. I can't say whether it would be feasible for Cygwin to do that, but it would certainly be more Unix-like and more POSIX conformant for it to have rmdir() fail in that situation than for rmdir() to partially succeed while blocking reuse of the name. Note that cwd_enoent.t skips if rmdir() reports failure. Show quoted text
>It still does resolve to it and the directory entry is in fact still >there if you go and look it up in the parent directory, you just can't >use that name to do a successful stat or readdir anymore.
That doesn't sound like "resolving" to me. That the name still exists in the parent directory is certainly interesting, but if the name can't be used in file operations then it's just blocking a directory slot. -zefram
To: perl5-porters [...] perl.org
From: Achim Gratz <Stromeko [...] nexgo.de>
Date: Sun, 21 Jan 2018 10:55:47 +0100
Subject: Re: [perl #132733] getcwd() fails to fail on Cygwin
Download (untitled) / with headers
text/plain 3.3k
Zefram writes: Show quoted text
>>Yes there is and that's the reason your test fails. The entry for cwd >>in its parent directory doesn't go away until the last file handle to it >>closes,
> > That's not a race condition. It's not dependent on any concurrent > activity, and adding pauses to the code doesn't affect it.
Yes it is, just not one you've created and so you don't expect it to be observable. POSIX mandates that rmdir does a few things that individually are supposed to be atomic, but makes no offer for atomicity of the whole thing: If the directory's link count becomes 0 and no process has the directory open, the space occupied by the directory shall be freed and the directory shall no longer be accessible. If one or more processes have the directory open when the last link is removed, the dot and dot-dot entries, if present, shall be removed before rmdir() returns and no new entries may be created in the directory, but the directory shall not be removed until all references to the directory are closed. On Windows you can observe the point where the . and .. entries have been removed, but the entry in the parent directory is still there (because the system call removing the directory returns as soon as it has marked the directory as deleted and hence made the access to it impossible). Show quoted text
> That is an unfortunate omission for getcwd(3), but the situation is > discussed a bit for rmdir(2). There is explicit permission for rmdir() > to yield EBUSY for a directory that is the root or cwd of any process.
Permission, not mandate. Whether or not "the implementation considers this an error" is not something that you control and POSIX very wisely left open. Show quoted text
> I can't say whether it would be feasible for Cygwin to do that, but it > would certainly be more Unix-like and more POSIX conformant for it to > have rmdir() fail in that situation than for rmdir() to partially succeed > while blocking reuse of the name. Note that cwd_enoent.t skips if rmdir() > reports failure.
Cygwin is a user-mode layer on top of Windows' Win32 subsytem. There are things that only the kernel can do correctly and it has a slightly different idea on this very topic than you do. You do in general not know whether a directory about to be removed is the cwd of some random process, even if you'd look it up in the process table. In addition for Cygwin it's not enough to look for Cygwin processes either, so you'd need to traverse the Windows process list as well (which you might not even have the right to do). Show quoted text
> That doesn't sound like "resolving" to me. That the name still exists > in the parent directory is certainly interesting, but if the name can't > be used in file operations then it's just blocking a directory slot.
Which might entirely be the point of the exercise, so that you can't go and create a new directory with the same name and then have an old process use that as cwd (which on Windows has security implications when you can plant DLL there and create new processes from the running application). In fact that is a situation that is discussed as a possibility in the POSIX getcwd documentation as one of the differences between implementations using fchdir and chdir. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org