From dbaca@atc-nycorp.com

To​: perlbug@​perl.org
Subject​: windows fork on Perl 5.8
Cc​: mfp_info@​atc-nycorp.com
Reply-To​: dbaca@​atc-nycorp.com
Message-Id​: <5.8.0_4412_1057001926@​STAR_ORA>

This is a bug report for perl from dbaca@​atc-nycorp.com,
generated with the help of perlbug 1.34 running under perl v5.8.0.

The bug I am about to describe happens on the version 5.8. On Perl 5.6, the
behavior is correct.
On Windows, fork is emulated by ithreads. Each ithread is supposed to have
its own copy of file handles. I found out that it is not true in the version
5.8. Redirecting a file handle in one ithread also redirects the same file
handle in the second ithread. The following code demonstrates the problem.

my $fn = 'fork_test.txt';
my $pid;
if ($pid = fork)
{
  print "Hello from ithread 1\n";
  open(STDOUT,'>',$fn);
  system('ipconfig');
  close STDOUT;
  waitpid($pid,0);
}
else
{
  die 'Couldn\'t fork!' unless defined $pid;
  open(READ,'<',$fn);
  print "Hello from ithread 2\n";
  my $l;
  do
  {
  undef $l; my $i = 0;
  while(!defined($l))
  {
  my $slp = int(($i++ + 9) / 10);
  sleep($slp);
  $l = <READ>;
  };
  print $l;
  } until($l =~ /Gateway/);
  close READ;
}

When this program is run with the Perl 5.6, it prints to the screen
(STDOUT)​:
Hello from ithread 1
Hello from ithread 2
ipconfig output

The ipconfig output is however printed to the file first and the child
ithread reads it and prints it to the screen which is the correct behavior.

In the contrary, when the program is run with the Perl 5.8, usually only the
first Hello (Hello from ithread 1) is printed to the screen. Occasionally,
the second Hello appears there, but it's rare and it happens because the
redirection in the parent thread occurs after the Hello gets printed in the
child ithread. Nevertheless, the ipconfig output never gets to the screen
and as it's read from the file is being printed back to the file, so it ends
up twice in the file. The reason is that the redirection of STDOUT in the
parent ithread redirects also STDOUT in the child ithread which is a bug.

Flags​:
  category=core
  severity=high

Site configuration information for perl v5.8.0​:

Configured by rob at Tue Jun 24 16​:00​:44 2003.

Summary of my perl5 (revision 5 version 8 subversion 0) configuration​:
  Platform​:
  osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread
  uname=''
  config_args='undef'
  hint=recommended, useposix=true, d_sigaction=undef
  usethreads=undef use5005threads=undef useithreads=define
usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=undef usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cl', ccflags ='-nologo -Gf -W3 -MD -DNDEBUG -O1 -DWIN32 -D_CONSOLE
-DNO_STRICT -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO
-DPERL_MSVCRT_READFIX',
  optimize='-MD -DNDEBUG -O1',
  cppflags='-DWIN32'
  ccversion='', gccversion='', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=4
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='link', ldflags ='-nologo -nodefaultlib -release
-libpath​:"c​:\perl\5.8.0\lib\MSWin32-x86-multi-thread\CORE" -machine​:x86'
  libpth=C​:\PROGRA1\MICROS3\VC98\lib
  libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib
uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib
msvcrt.lib
  perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib
uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib
msvcrt.lib
  libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib
  gnulibc_version='undef'
  Dynamic Linking​:
  dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -release
-libpath​:"c​:\perl\5.8.0\lib\MSWin32-x86-multi-thread\CORE" -machine​:x86'

Locally applied patches​:
 

@​INC for perl v5.8.0​:
  C​:/perl/5.8.0/lib/MSWin32-x86-multi-thread
  C​:/perl/5.8.0/lib
  C​:/perl/site/5.8.0/lib/MSWin32-x86-multi-thread
  C​:/perl/site/5.8.0/lib
  .

Environment for perl v5.8.0​:
  HOME (unset)
  LANG (unset)
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=C​:\Perl\bin\;C​:\perl\5.8.0\bin\MSWin32-x86-multi-thread;C​:\Program
Files\MiKTeX\miktex\bin;C​:\WINDOWS\system32;C​:\WINDOWS;C​:\WINDOWS\System32\W
bem;C​:\Program Files\ATI Technologies\ATI Control Panel;C​:\Program
Files\Common Files\Adaptec Shared\System
  PERL_BADLANG (unset)
  SHELL (unset)

From @TJC

Hi,
could you please re-test this bug on the current version of Perl, 5.10.1,
and report back if the behaviour has been fixed.

Thanks.

The RT System itself - Status changed from 'new' to 'open'

From myfwhite@gmail.com

On Wed Nov 11 02​:42​:31 2009, tjc wrote​:

Hi,
could you please re-test this bug on the current version of Perl, 5.10.1,
and report back if the behaviour has been fixed.

Thanks.

I don't have 5.10.1, but the behaviour still occurs in 5.10.0.

From @bulk88

On Wed Nov 11 02​:42​:31 2009, tjc wrote​:

Hi,
could you please re-test this bug on the current version of Perl, 5.10.1,
and report back if the behaviour has been fixed.

Thanks.

Using Perl 5.17.6, this is still a problem, I guess, I didn't run 5.6 to
see what happens. Anyways

" open(STDOUT,'>',$fn) or die "cannot open parent : $!";"

makes a call to CRT's dup2, fh 3 is the handle to fork_test.txt. fh 1 is
stdout.

The line that calls PerlLIODup2 is

http​://perl5.git.perl.org/perl.git/blob/93a641ae382638ffd1​:/doio.c#l617

____________________________________________________________
  msvcr71.dll!_dup2(int fh1=3, int fh2=1) Line 57 C
  perl517.dll!PerlLIODup2(IPerlLIO * piPerl=0x00344364, int handle1=3,
int handle2=1) Line 959 + 0xd C

perl517.dll!Perl_do_openn(interpreter * my_perl=0x00345ecc, gv *
gv=0x008fd034, const char * oname=0x00b41414, long len=1, int as_raw=0,
int rawmode=0, int rawperm=0, _PerlIO * * supplied_fp=0x00000000, sv * *
svp=0x00a7b020, long num_svs=1) Line 617 + 0x1e C
  perl517.dll!Perl_pp_open(interpreter * my_perl=0x00345ecc) Line 638
+ 0x2e C
  perl517.dll!Perl_runops_standard(interpreter * my_perl=0x00345ecc)
Line 42 + 0xa C
  perl517.dll!S_run_body(interpreter * my_perl=0x00345ecc, long
oldscope=1) Line 2430 + 0xd C
  perl517.dll!perl_run(interpreter * my_perl=0x00345ecc) Line 2349 C
  perl517.dll!RunPerl(int argc=2, char * * argv=0x00342478, char * *
env=0x003451f0) Line 270 + 0x9 C
  perl.exe!mainCRTStartup() Line 398 + 0xe C
  kernel32.dll!_BaseProcessStart@​4() + 0x23
___________________________________________________________________
Then CRT dup2 winds up calling
" SetStdHandle( STD_OUTPUT_HANDLE, (HANDLE)value );",
which is per Win32 process, with callstack,
______________________________________________________________________
kernel32.dll!_SetStdHandle@​8()
  msvcr71.dll!_dup2(int fh1=3, int fh2=1) Line 98 + 0x7 C
  perl517.dll!win32_dup2(int fd1=3, int fd2=1) Line 3293 + 0xe C
  perl517.dll!PerlLIODup2(IPerlLIO * piPerl=0x00344364, int handle1=3,
int handle2=1) Line 959 + 0xd C
  perl517.dll!Perl_do_openn(interpreter * my_perl=0x00345ecc, gv *
gv=0x008fd034, const char * oname=0x00becc34, long len=1, int as_raw=0,
int rawmode=0, int rawperm=0, _PerlIO * * supplied_fp=0x00000000, sv * *
svp=0x00a7aff8, long num_svs=1) Line 617 + 0x1e C
  perl517.dll!Perl_pp_open(interpreter * my_perl=0x00345ecc) Line 638
+ 0x2e C
  perl517.dll!Perl_runops_standard(interpreter * my_perl=0x00345ecc)
Line 42 + 0xa C
  perl517.dll!S_run_body(interpreter * my_perl=0x00345ecc, long
oldscope=1) Line 2430 + 0xd C
  perl517.dll!perl_run(interpreter * my_perl=0x00345ecc) Line 2349 C
  perl517.dll!RunPerl(int argc=2, char * * argv=0x00342478, char * *
env=0x003451f0) Line 270 + 0x9 C
  perl.exe!mainCRTStartup() Line 398 + 0xe C
  kernel32.dll!_BaseProcessStart@​4() + 0x23
______________________________________________________________________

The test script I am using is
___________________________________________________________________

#!/usr/bin/perl -w
use Data​::Dumper;
use warnings;
use strict;
use Win32API​::File 'FdGetOsFHandle';

`del fork_test.txt`;
my $fn = 'fork_test.txt';
my $pid;
warn "b4 fork STDOUT ".sprintf("%x", FdGetOsFHandle(fileno(STDOUT)))."\n";
if ($pid = fork)
{
  print "Hello from ithread 1\n";
  #sleep 1;
  warn "p b4 STDOUT ".sprintf("%x", FdGetOsFHandle(fileno(STDOUT)))."\n";
  sleep 5;
  open(STDOUT,'>',$fn) or die "cannot open parent : $!";
  warn "p aftr STDOUT ".sprintf("%x",
FdGetOsFHandle(fileno(STDOUT)))."\n";
  system('ipconfig');
  close STDOUT;
  waitpid($pid,0);
}
else
{
  die 'Couldn\'t fork!' unless defined $pid;
  #sleep 2;
  warn "c1 STDOUT ".sprintf("%x", FdGetOsFHandle(fileno(STDOUT)))."\n";
  warn "c1 READ ".sprintf("%x", FdGetOsFHandle(fileno(READ)))."\n";
  open(READ,'<',$fn) or die "cannot open parent : $!";
  warn "c2 READ ".sprintf("%x", FdGetOsFHandle(fileno(READ)))."\n";
  print "Hello from ithread 2\n";
  my $l;
  do
  {
  undef $l; my $i = 0;
  while(!defined($l))
  {
  my $slp = int(($i++ + 9) / 10);
  sleep($slp);
  $l = <READ>;
  };
  print $l;
  } until($l =~ /Gateway/);
  close READ;
  sleep 10000;
}
____________________________________________________________
How to fix this, I guess all fileno calls need to be hooked, with a per
perl interp set of Perl level FDs that are then converted to CRT level
FDs. On a clone/fork, all the FDs of the parent interp are duped to new
CRT level FDs, but the Perl level FDs stay the same. Not sure where and
how the hooking should happen. No idea if this will work, but its beyond
my skill set. Comments?

I wonder if this bug can be done with ithreads.
--
bulk88 ~ bulk88 at hotmail.com

From @Leont

On Sun Jul 13 22​:35​:09 2003, dbaca@​atc-nycorp.com wrote​:

The bug I am about to describe happens on the version 5.8. On Perl
5.6, the
behavior is correct.
On Windows, fork is emulated by ithreads. Each ithread is supposed to have
its own copy of file handles. I found out that it is not true in the
version
5.8. Redirecting a file handle in one ithread also redirects the same file
handle in the second ithread. The following code demonstrates the problem.

my $fn = 'fork_test.txt';
my $pid;
if ($pid = fork)
{
print "Hello from ithread 1\n";
open(STDOUT,'>',$fn);
system('ipconfig');
close STDOUT;
waitpid($pid,0);
}
else
{
die 'Couldn\'t fork!' unless defined $pid;
open(READ,'<',$fn);
print "Hello from ithread 2\n";
my $l;
do
{
undef $l; my $i = 0;
while(!defined($l))
{
my $slp = int(($i++ + 9) / 10);
sleep($slp);
$l = <READ>;
};
print $l;
} until($l =~ /Gateway/);
close READ;
}

When this program is run with the Perl 5.6, it prints to the screen
(STDOUT)​:
Hello from ithread 1
Hello from ithread 2
ipconfig output

The ipconfig output is however printed to the file first and the child
ithread reads it and prints it to the screen which is the correct
behavior.

In the contrary, when the program is run with the Perl 5.8, usually
only the
first Hello (Hello from ithread 1) is printed to the screen. Occasionally,
the second Hello appears there, but it's rare and it happens because the
redirection in the parent thread occurs after the Hello gets printed
in the
child ithread. Nevertheless, the ipconfig output never gets to the screen
and as it's read from the file is being printed back to the file, so
it ends
up twice in the file. The reason is that the redirection of STDOUT in the
parent ithread redirects also STDOUT in the child ithread which is a bug.

I think this is a clear case of "every abstraction leaks". File
descriptors are shared between threads, and STDIN/STDOUT/STDERR are
hard-coded to fd 0, 1, 2. I really don't think that there isn't any way
to handle this situation that isn't surprising in some important way.
Either​:
* Redirecting STDIN/STDOUT/STDERR is thread-global.
* STDIN/STDOUT/STDERR doesn't necessarily match fileno 0/1/2

Leon

@Leont does your last comment suggest we close this as unfixable?