Skip Menu |
Report information
Id: 72526
Status: rejected
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: kissg [at] ssg.ki.iif.hu
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: PerlIO
Perl Version: (no value)
Fixed In: (no value)



Subject: __DATA__ conflicts with Proc::Daemon. A "half closed" filehandle.
Date: Thu, 4 Feb 2010 10:28:24 +0100 (CET)
To: perlbug [...] perl.org
From: kissg [...] ssg.ki.iif.hu (Kiss Gabor)
Download (untitled) / with headers
text/plain 8.1k
This is a bug report for perl from kissg@ssg.ki.iif.hu, generated with the help of perlbug 1.36 running under perl 5.10.0. ----------------------------------------------------------------- Please run this shell script that calls the PERL program below: --------------8<---------------8<---------------- #/bin/sh echo "No fork" ./perlbug cat ./perlbug.out echo echo "Daemonize process before opening files" ./perlbug -fork sleep 2 # allow child process to finish cat ./perlbug.out --------------8<---------------8<---------------- File perlbug: --------------8<---------------8<---------------- #!/usr/bin/perl -s use Proc::Daemon; use IO::File; our $fork; system("(echo before open pid=$$: ; ls -l /proc/$$/fd) > /tmp/perlbug.out"); Proc::Daemon::Init if $fork; { my $fh1 = IO::File->new('/etc/passwd'); # file is open my $fh2 = IO::File->new('/etc/fstab'); # file is open my $fh3 = IO::File->new('/etc/inittab');# file is open system("(echo after open pid=$$:;ls -l /proc/$$/fd)>>/tmp/perlbug.out"); } # $fh1, $fh2, $fh3 objects are destroyed. Files must be closed. # Let's check it: system("(echo after destroy pid=$$: ; ls -l /proc/$$/fd) >> /tmp/perlbug.out"); # comment out this line: __DATA__ --------------8<---------------8<---------------- The result is this: --------------8<---------------8<---------------- kissg@bakacsin:/tmp$ ./perlbug.sh No fork before open pid=31200: total 0 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 0 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 1 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 2 -> /dev/pts/9 lr-x------ 1 kissg kissg 64 2010-02-04 09:25 3 -> /tmp/perlbug lr-x------ 1 kissg kissg 64 2010-02-04 09:25 4 -> pipe:[6127332] after open pid=31200: total 0 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 0 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 1 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 2 -> /dev/pts/9 lr-x------ 1 kissg kissg 64 2010-02-04 09:25 3 -> /tmp/perlbug lr-x------ 1 kissg kissg 64 2010-02-04 09:25 4 -> /etc/passwd lr-x------ 1 kissg kissg 64 2010-02-04 09:25 5 -> /etc/fstab lr-x------ 1 kissg kissg 64 2010-02-04 09:25 6 -> /etc/inittab lr-x------ 1 kissg kissg 64 2010-02-04 09:25 7 -> pipe:[6127345] after destroy pid=31200: total 0 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 0 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 1 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 2 -> /dev/pts/9 lr-x------ 1 kissg kissg 64 2010-02-04 09:25 3 -> /tmp/perlbug lr-x------ 1 kissg kissg 64 2010-02-04 09:25 4 -> pipe:[6127355] Daemonize process before opening files before open pid=31211: total 0 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 0 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 1 -> /dev/pts/9 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 2 -> /dev/pts/9 lr-x------ 1 kissg kissg 64 2010-02-04 09:25 3 -> /tmp/perlbug lr-x------ 1 kissg kissg 64 2010-02-04 09:25 4 -> pipe:[6127368] after open pid=31216: total 0 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 0 -> /dev/null lrwx------ 1 kissg kissg 64 2010-02-04 09:25 1 -> /dev/null lrwx------ 1 kissg kissg 64 2010-02-04 09:25 2 -> /dev/null lr-x------ 1 kissg kissg 64 2010-02-04 09:25 3 -> /etc/passwd lr-x------ 1 kissg kissg 64 2010-02-04 09:25 4 -> /etc/fstab lr-x------ 1 kissg kissg 64 2010-02-04 09:25 5 -> /etc/inittab lr-x------ 1 kissg kissg 64 2010-02-04 09:25 6 -> pipe:[6127381] after destroy pid=31216: total 0 lrwx------ 1 kissg kissg 64 2010-02-04 09:25 0 -> /dev/null lrwx------ 1 kissg kissg 64 2010-02-04 09:25 1 -> /dev/null lrwx------ 1 kissg kissg 64 2010-02-04 09:25 2 -> /dev/null lr-x------ 1 kissg kissg 64 2010-02-04 09:25 3 -> /etc/passwd lr-x------ 1 kissg kissg 64 2010-02-04 09:25 4 -> pipe:[6127397] /tmp$ --------------8<---------------8<---------------- Note that /etc/passwd (filehandle 3) remained open at the second run even after the associated IO::Handle object $fh1 was destroyed. However if you comment out the __DATA__ line everything works well. The problem is not limited to ordinary files. Network sockets are also affected. Actually this is the way I found the problem. A server process that daemonize itself with Proc::Daemon::Init() does not close the TCP connection coming from the _first_ client. The daemon uses Device::USB module that contains __DATA__ and inline C code. At compile time the interpreter reopens USB.pm as 5 and never closes it. Then Proc::Daemon::Init() forks and closes all handles including 5. The first accept() returns 5 as the first free number but when all event handlers finish their jobs and all of them unreferences the IO::Handle object, "somebody" keeps it open. Subsequent clients are connected via 6,7,8 etc and they are closed well when object's reference reaches zero. (Yes, I could find a workaround to handle this situation.) As far as I can see the real problem is that upgrade of a "third party" module can break well working old programs if the author of the module starts to use __DATA__ for any reason. My Linux system is a Debian lenny with kernel 2.6.26-2-686. The bug is also reproduced on a 5 year old Red Hat machine with 2.4.21-32.ELsmp kernel and PERL 5.8.0. Gabor ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl 5.10.0: Configured by Debian Project at Fri Aug 28 22:30:10 UTC 2009. Summary of my perl5 (revision 5 version 10 subversion 0) configuration: Platform: osname=linux, osvers=2.6.26-2-amd64, archname=i486-linux-gnu-thread-multi uname='linux puccini 2.6.26-2-amd64 #1 smp fri aug 14 07:12:04 utc 2009 i686 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='4.3.2', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /usr/lib64 libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0 gnulibc_version='2.7' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib' Locally applied patches: --- @INC for perl 5.10.0: /etc/perl /usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl . --- Environment for perl 5.10.0: HOME=/home/kissg LANG=en_US LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/kissg/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games PERL_BADLANG (unset) SHELL=/bin/bash
Subject: Re: [perl #72526] __DATA__ conflicts with Proc::Daemon. A "half closed" filehandle.
Date: Fri, 5 Feb 2010 23:52:08 +0000
To: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 1.3k
On Thu, Feb 04, 2010 at 01:29:18AM -0800, kissg@ssg.ki.iif.hu wrote: Show quoted text
> Note that /etc/passwd (filehandle 3) remained open at the second run > even after the associated IO::Handle object $fh1 was destroyed.
This isn't a problem with perl. Proc::Daemon is "secretly" closing file descriptor 3 (the one perl has associated with DATA) without telling perl. When perl is asked to open a new file, the OS returns fd 3, so there are now two perl handles associated with fd 3: DATA and $fh1. You ask perl to close one of the handles, but there's now two perl file handles associated with fd 3, so it just decrements the refcount rather than closing fh3. This leaves DATA associated with fh 3, which is still opened as /etc/passwd. You can see a similar effect withe these two programs that don't require Proc::Daemon: use POSIX (); POSIX::close(3); open my $fh, '<', '/etc/passwd' or die "open: $!\n"; close $fh; system "ls -l /proc/$$/fd"; __DATA__ and use POSIX (); open my $data, $0, or die; POSIX::close(3); open my $fh, '<', '/etc/passwd' or die "open: $!\n"; close $fh; system "ls -l /proc/$$/fd"; -- The warp engines start playing up a bit, but seem to sort themselves out after a while without any intervention from boy genius Wesley Crusher. -- Things That Never Happen in "Star Trek" #17
CC: perl5-porters [...] perl.org
Subject: Re: [perl #72526] __DATA__ conflicts with Proc::Daemon. A "half closed" filehandle.
Date: Tue, 9 Feb 2010 14:23:34 +0100 (CET)
To: Dave Mitchell <davem [...] iabyn.com>
From: "Kiss Gabor (Bitman)" <kissg [...] ssg.ki.iif.hu>
Download (untitled) / with headers
text/plain 478b
I think if you are right, there is no way to daemonize a process in a correct way without exec()-ing another Perl interpreter and replaying all the initialization phase. (Corollary: Proc::Daemon has to be forgotten.) An application any time can use a third party module with a hidden __DATA__ segment. So programmer has no way to figure out which are the decriptors that must not be closed. In general case (s)he cannot get a list of open files. Did I miss something? Gabor
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 888b
On Fr. 05. Feb. 2010, 15:52:56, davem wrote: Show quoted text
> This isn't a problem with perl. Proc::Daemon is "secretly" closing
file Show quoted text
> descriptor 3 (the one perl has associated with DATA) without telling
perl. 1) Not a perl problem? Isn't POSIX a part of perl nowadays? 2) Why isn't it documented (http://perldoc.perl.org/POSIX.html) that POSIX::close( $FD ) does not close the real file descriptor? 3) Does POSIX::close( $FD ) have any function at all if it doesn't tell perl that the file must be closed? 4) How can I close a file knowing only the file descriptor, not knowing the file handle? Or, how can I get the file handle to <CORE::close ( $FH )> if I only know the file descriptor number. For example if I want to close all file descriptors from 0 .. POSIX::sysconf ( POSIX::_SC_OPEN_MAX ) as Proc::Daemon does to detach from the parent process. Thanks and regards Detlef Pilzecker
CC: perl5-porters [...] perl.org
Subject: Re: [perl #72526] __DATA__ conflicts with Proc::Daemon. A "half closed" filehandle.
Date: Thu, 9 Jun 2011 14:44:36 +0100
To: Detlef Pilzecker via RT <perlbug-followup [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 2.7k
On Sat, May 21, 2011 at 11:32:29AM -0700, Detlef Pilzecker via RT wrote: Show quoted text
> On Fr. 05. Feb. 2010, 15:52:56, davem wrote:
> > This isn't a problem with perl. Proc::Daemon is "secretly" closing
> file
> > descriptor 3 (the one perl has associated with DATA) without telling
> perl. > > 1) Not a perl problem? Isn't POSIX a part of perl nowadays? > 2) Why isn't it documented (http://perldoc.perl.org/POSIX.html) that > POSIX::close( $FD ) does not close the real file descriptor?
It's documented to do what it does: close the OS-level file descriptor; i.e. it's a thin wrapper over the close(2) system call. Show quoted text
> 3) Does POSIX::close( $FD ) have any function at all if it doesn't tell > perl that the file must be closed?
It's useful if you just want to close a UNIX file descriptor. Show quoted text
> 4) How can I close a file knowing only the file descriptor, not knowing > the file handle? Or, how can I get the file handle to <CORE::close > ( $FH )> if I only know the file descriptor number. For example if I > want to close all file descriptors from 0 .. POSIX::sysconf > ( POSIX::_SC_OPEN_MAX ) as Proc::Daemon does to detach from the parent > process.
There isn't an easy way. Bear in mind that several different Perl-level file handles, spread over multiple threads, may all map to the same OS file descriptor. Some approaches to making this available at the perl level would be: 1) at the bottom: add an extra level of indirection between the PerlIO stack and the file descriptor table; this would allow file descriptors to be "invalidated" rather than being accidentally reallocated. Forcible closing at this level would mean that all buffer flushing, tied CLOSE() calling etc would be skipped, similar to what's happening now with POSIX::close(). 2) At the middle: call the close method on every file stack in PL_perlio; this ensures that all buffers are flushed etc, but has the insurmountable drawback that only handles in the current thread are closed. 3) At the top: scan the SV arenas for PVIOs, and close each one. This has the same advantages and disadvantages as (2), except that in addition it will call CLOSE() on tied handles, but there's no guarantee that all PerlIO objects are linked form PVIOs, so some handles might be missed. Note that the standard C library suffers from exactly the same problem: main (int argc, char **argv, char **env) { FILE *f, *g; char s[1000];; f = fopen("/etc/passwd", "r"); close(3); g = fopen("/etc/rpc", "r"); fgets(s, 100, f); printf("s=[%s]\n", s); /* prints a line from rpc, not passwd */ } In short, there isn't a practical way to do what you want (close all files without side-effects). -- "There's something wrong with our bloody ships today, Chatfield." -- Admiral Beatty at the Battle of Jutland, 31st May 1916.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org