Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

__DATA__ conflicts with Proc::Daemon. A "half closed" filehandle. #10130

Closed
p5pRT opened this issue Feb 4, 2010 · 7 comments
Closed

__DATA__ conflicts with Proc::Daemon. A "half closed" filehandle. #10130

p5pRT opened this issue Feb 4, 2010 · 7 comments

Comments

@p5pRT
Copy link

p5pRT commented Feb 4, 2010

Migrated from rt.perl.org#72526 (status was 'rejected')

Searchable as RT72526$

@p5pRT
Copy link
Author

p5pRT commented Feb 4, 2010

From kissg@ssg.ki.iif.hu

This is a bug report for perl from kissg@​ssg.ki.iif.hu,
generated with the help of perlbug 1.36 running under perl 5.10.0.


Please run this shell script that calls the PERL program below​:

--------------8<---------------8<----------------
#/bin/sh

echo "No fork"
./perlbug
cat ./perlbug.out

echo
echo "Daemonize process before opening files"
./perlbug -fork
sleep 2 # allow child process to finish
cat ./perlbug.out
--------------8<---------------8<----------------

File perlbug​:
--------------8<---------------8<----------------
#!/usr/bin/perl -s

use Proc​::Daemon;
use IO​::File;
our $fork;

system("(echo before open pid=$$​: ; ls -l /proc/$$/fd) > /tmp/perlbug.out");
Proc​::Daemon​::Init if $fork;

{
  my $fh1 = IO​::File->new('/etc/passwd'); # file is open
  my $fh2 = IO​::File->new('/etc/fstab'); # file is open
  my $fh3 = IO​::File->new('/etc/inittab');# file is open
  system("(echo after open pid=$$​:;ls -l /proc/$$/fd)>>/tmp/perlbug.out");
}
# $fh1, $fh2, $fh3 objects are destroyed. Files must be closed.
# Let's check it​:
system("(echo after destroy pid=$$​: ; ls -l /proc/$$/fd) >> /tmp/perlbug.out");

# comment out this line​:
__DATA__
--------------8<---------------8<----------------

The result is this​:
--------------8<---------------8<----------------
kissg@​bakacsin​:/tmp$ ./perlbug.sh
No fork
before open pid=31200​:
total 0
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 0 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 1 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 2 -> /dev/pts/9
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 3 -> /tmp/perlbug
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 4 -> pipe​:[6127332]
after open pid=31200​:
total 0
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 0 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 1 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 2 -> /dev/pts/9
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 3 -> /tmp/perlbug
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 4 -> /etc/passwd
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 5 -> /etc/fstab
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 6 -> /etc/inittab
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 7 -> pipe​:[6127345]
after destroy pid=31200​:
total 0
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 0 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 1 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 2 -> /dev/pts/9
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 3 -> /tmp/perlbug
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 4 -> pipe​:[6127355]

Daemonize process before opening files
before open pid=31211​:
total 0
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 0 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 1 -> /dev/pts/9
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 2 -> /dev/pts/9
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 3 -> /tmp/perlbug
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 4 -> pipe​:[6127368]
after open pid=31216​:
total 0
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 0 -> /dev/null
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 1 -> /dev/null
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 2 -> /dev/null
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 3 -> /etc/passwd
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 4 -> /etc/fstab
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 5 -> /etc/inittab
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 6 -> pipe​:[6127381]
after destroy pid=31216​:
total 0
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 0 -> /dev/null
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 1 -> /dev/null
lrwx------ 1 kissg kissg 64 2010-02-04 09​:25 2 -> /dev/null
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 3 -> /etc/passwd
lr-x------ 1 kissg kissg 64 2010-02-04 09​:25 4 -> pipe​:[6127397]
/tmp$
--------------8<---------------8<----------------

Note that /etc/passwd (filehandle 3) remained open at the second run
even after the associated IO​::Handle object $fh1 was destroyed.

However if you comment out the __DATA__ line everything works well.

The problem is not limited to ordinary files.
Network sockets are also affected.
Actually this is the way I found the problem.

A server process that daemonize itself with Proc​::Daemon​::Init()
does not close the TCP connection coming from the _first_ client.
The daemon uses Device​::USB module that contains __DATA__ and
inline C code. At compile time the interpreter reopens USB.pm as 5
and never closes it. Then Proc​::Daemon​::Init() forks and closes
all handles including 5. The first accept() returns 5 as the first
free number but when all event handlers finish their jobs and all of
them unreferences the IO​::Handle object, "somebody" keeps it open.
Subsequent clients are connected via 6,7,8 etc and they
are closed well when object's reference reaches zero.

(Yes, I could find a workaround to handle this situation.)
As far as I can see the real problem is that upgrade of
a "third party" module can break well working old programs
if the author of the module starts to use __DATA__ for any reason.

My Linux system is a Debian lenny with kernel 2.6.26-2-686.
The bug is also reproduced on a 5 year old Red Hat machine with
2.4.21-32.ELsmp kernel and PERL 5.8.0.

Gabor



Flags​:
  category=core
  severity=low


Site configuration information for perl 5.10.0​:

Configured by Debian Project at Fri Aug 28 22​:30​:10 UTC 2009.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration​:
  Platform​:
  osname=linux, osvers=2.6.26-2-amd64, archname=i486-linux-gnu-thread-multi
  uname='linux puccini 2.6.26-2-amd64 #1 smp fri aug 14 07​:12​:04 utc 2009 i686 gnulinux '
  config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
  ccversion='', gccversion='4.3.2', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib /usr/lib64
  libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
  perllibs=-ldl -lm -lpthread -lc -lcrypt
  libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0
  gnulibc_version='2.7'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'

Locally applied patches​:
 


@​INC for perl 5.10.0​:
  /etc/perl
  /usr/local/lib/perl/5.10.0
  /usr/local/share/perl/5.10.0
  /usr/lib/perl5
  /usr/share/perl5
  /usr/lib/perl/5.10
  /usr/share/perl/5.10
  /usr/local/lib/site_perl
  .


Environment for perl 5.10.0​:
  HOME=/home/kissg
  LANG=en_US
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/home/kissg/bin​:/usr/local/bin​:/usr/bin​:/bin​:/usr/bin/X11​:/usr/games
  PERL_BADLANG (unset)
  SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Feb 5, 2010

From @iabyn

On Thu, Feb 04, 2010 at 01​:29​:18AM -0800, kissg@​ssg.ki.iif.hu wrote​:

Note that /etc/passwd (filehandle 3) remained open at the second run
even after the associated IO​::Handle object $fh1 was destroyed.

This isn't a problem with perl. Proc​::Daemon is "secretly" closing file
descriptor 3 (the one perl has associated with DATA) without telling perl.
When perl is asked to open a new file, the OS returns fd 3, so there are
now two perl handles associated with fd 3​: DATA and $fh1. You ask perl to
close one of the handles, but there's now two perl file handles associated
with fd 3, so it just decrements the refcount rather than closing fh3.
This leaves DATA associated with fh 3, which is still opened as
/etc/passwd.

You can see a similar effect withe these two programs that don't require
Proc​::Daemon​:

  use POSIX ();
  POSIX​::close(3);
  open my $fh, '&lt;', '/etc/passwd' or die "open​: $!\n";
  close $fh;
  system "ls -l /proc/$$/fd";
  __DATA__

and

  use POSIX ();
  open my $data, $0, or die;
  POSIX​::close(3);
  open my $fh, '&lt;', '/etc/passwd' or die "open​: $!\n";
  close $fh;
  system "ls -l /proc/$$/fd";

--
The warp engines start playing up a bit, but seem to sort themselves out
after a while without any intervention from boy genius Wesley Crusher.
  -- Things That Never Happen in "Star Trek" #17

@p5pRT
Copy link
Author

p5pRT commented Feb 5, 2010

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Feb 10, 2010

From kissg@ssg.ki.iif.hu

I think if you are right, there is no way to daemonize a process
in a correct way without exec()-ing another Perl interpreter and
replaying all the initialization phase.
(Corollary​: Proc​::Daemon has to be forgotten.)

An application any time can use a third party module with a
hidden __DATA__ segment. So programmer has no way to figure
out which are the decriptors that must not be closed.
In general case (s)he cannot get a list of open files.

Did I miss something?

Gabor

@p5pRT
Copy link
Author

p5pRT commented May 21, 2011

From deti@cpan.org

On Fr. 05. Feb. 2010, 15​:52​:56, davem wrote​:

This isn't a problem with perl. Proc​::Daemon is "secretly" closing
file
descriptor 3 (the one perl has associated with DATA) without telling
perl.

1) Not a perl problem? Isn't POSIX a part of perl nowadays?
2) Why isn't it documented (http​://perldoc.perl.org/POSIX.html) that
POSIX​::close( $FD ) does not close the real file descriptor?
3) Does POSIX​::close( $FD ) have any function at all if it doesn't tell
perl that the file must be closed?
4) How can I close a file knowing only the file descriptor, not knowing
the file handle? Or, how can I get the file handle to <CORE​::close
( $FH )> if I only know the file descriptor number. For example if I
want to close all file descriptors from 0 .. POSIX​::sysconf
( POSIX​::_SC_OPEN_MAX ) as Proc​::Daemon does to detach from the parent
process.

Thanks and regards

Detlef Pilzecker

@p5pRT
Copy link
Author

p5pRT commented Jun 9, 2011

From @iabyn

On Sat, May 21, 2011 at 11​:32​:29AM -0700, Detlef Pilzecker via RT wrote​:

On Fr. 05. Feb. 2010, 15​:52​:56, davem wrote​:

This isn't a problem with perl. Proc​::Daemon is "secretly" closing
file
descriptor 3 (the one perl has associated with DATA) without telling
perl.

1) Not a perl problem? Isn't POSIX a part of perl nowadays?
2) Why isn't it documented (http​://perldoc.perl.org/POSIX.html) that
POSIX​::close( $FD ) does not close the real file descriptor?

It's documented to do what it does​: close the OS-level file descriptor;
i.e. it's a thin wrapper over the close(2) system call.

3) Does POSIX​::close( $FD ) have any function at all if it doesn't tell
perl that the file must be closed?

It's useful if you just want to close a UNIX file descriptor.

4) How can I close a file knowing only the file descriptor, not knowing
the file handle? Or, how can I get the file handle to <CORE​::close
( $FH )> if I only know the file descriptor number. For example if I
want to close all file descriptors from 0 .. POSIX​::sysconf
( POSIX​::_SC_OPEN_MAX ) as Proc​::Daemon does to detach from the parent
process.

There isn't an easy way. Bear in mind that several different Perl-level
file handles, spread over multiple threads, may all map to the same OS
file descriptor.

Some approaches to making this available at the perl level would be​:

1) at the bottom​: add an extra level of indirection between the PerlIO
stack and the file descriptor table; this would allow file descriptors to
be "invalidated" rather than being accidentally reallocated. Forcible
closing at this level would mean that all buffer flushing, tied CLOSE()
calling etc would be skipped, similar to what's happening now with
POSIX​::close().

2) At the middle​: call the close method on every file stack in PL_perlio;
this ensures that all buffers are flushed etc, but has the insurmountable
drawback that only handles in the current thread are closed.

3) At the top​: scan the SV arenas for PVIOs, and close each one. This has
the same advantages and disadvantages as (2), except that in addition it
will call CLOSE() on tied handles, but there's no guarantee that all
PerlIO objects are linked form PVIOs, so some handles might be missed.

Note that the standard C library suffers from exactly the same problem​:

  main (int argc, char **argv, char **env)
  {
  FILE *f, *g;
  char s[1000];;
  f = fopen("/etc/passwd", "r");
  close(3);
  g = fopen("/etc/rpc", "r");
  fgets(s, 100, f);
  printf("s=[%s]\n", s); /* prints a line from rpc, not passwd */
  }

In short, there isn't a practical way to do what you want (close all files
without side-effects).

--
"There's something wrong with our bloody ships today, Chatfield."
  -- Admiral Beatty at the Battle of Jutland, 31st May 1916.

@p5pRT
Copy link
Author

p5pRT commented Jun 12, 2011

@iabyn - Status changed from 'open' to 'rejected'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant