From jimc@math.ucla.edu

This is a bug report for perl from jimc@​math.ucla.edu,
generated with the help of perlbug 1.34 running under perl v5.8.3.

#!/usr/bin/perl
# If the script is not seekable, and if the DATA filehandle is duped, an
# initial segment of the DATA is lost when reading from the duplicate
# handle. With this script the lossage is 4216 bytes from start of
# script or 2959 bytes from start of DATA. In the production
# script where this was discovered, the loss from start of DATA is in
# the range 3500 to 4000 bytes.
# The production context is, we concatenate a "tripwire" type script
# and the filenames and checksums it should verify (customized per
# host), and do ssh remotehost "perl - -f '<&DATA'" where -f tells the
# script where to find the checksum list. That way the hackers have
# little to attack -- only by munging perl itself can they defend
# themselves.
print <<EOF;
Compare these three command lines​:
  perl datatrunc.pl # Seekable script, runs perfectly
  perl - < datatrunc.pl # Seekable script, runs perfectly
  cat datatrunc.pl | perl - # Not seekable, loses initial DATA
EOF
open(DUPL, "<&DATA") or die "Failed to dup DATA filehandle​: $!\n";
while (<DUPL>) { # Each line is exactly 80 bytes including \n
  print
}
__DATA__
Line 01 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 02 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 03 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 04 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 05 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 06 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 07 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 08 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 09 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 10 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 11 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 12 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 13 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 14 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 15 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 16 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 17 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 18 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 19 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 20 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 21 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 22 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 23 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 24 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 25 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 26 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 27 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 28 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 29 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 30 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 31 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 32 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 33 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 34 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 35 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 36 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 37 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 38 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 39 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 40 b123456789c123456789d123456789e123456789f123456789g123456789h12345678

Flags​:
  category=core
  severity=medium

This perlbug was built using Perl v5.8.3 - Sat Apr 3 00​:50​:30 UTC 2004
It is being executed now by Perl v5.8.3 - Sat Apr 3 00​:37​:21 UTC 2004.

Site configuration information for perl v5.8.3​:

Configured by abuild at Sat Apr 3 00​:37​:21 UTC 2004.

Summary of my perl5 (revision 5.0 version 8 subversion 3) configuration​:
  Platform​:
  osname=linux, osvers=2.6.4, archname=i586-linux-thread-multi
  uname='linux d209 2.6.4 #1 smp thu mar 11 17​:56​:49 utc 2004 i686 i686 i386 gnulinux '
  config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=true -Doptimize=-O2 -march=i586 -mcpu=i686 -fmessage-length=0 -Wall -Wall -pipe'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2 -march=i586 -mcpu=i686 -fmessage-length=0 -Wall -Wall -pipe',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing'
  ccversion='', gccversion='3.3.3 (SuSE Linux)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =''
  libpth=/lib /usr/lib /usr/local/lib
  libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.3.3'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.3/i586-linux-thread-multi/CORE'
  cccdlflags='-fPIC', lddlflags='-shared'

Locally applied patches​:
 

@​INC for perl v5.8.3​:
  /usr/lib/perl5/5.8.3/i586-linux-thread-multi
  /usr/lib/perl5/5.8.3
  /usr/lib/perl5/site_perl/5.8.3/i586-linux-thread-multi
  /usr/lib/perl5/site_perl/5.8.3
  /usr/lib/perl5/site_perl
  /usr/lib/perl5/vendor_perl/5.8.3/i586-linux-thread-multi
  /usr/lib/perl5/vendor_perl/5.8.3
  /usr/lib/perl5/vendor_perl
  .

Environment for perl v5.8.3​:
  HOME=/home/jimc
  LANG (unset)
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=.​:/home/jimc/bin​:/usr/diklo/bin​:/usr/local/sbin​:/usr/local/bin​:/usr/sbin​:/usr/bin​:/usr/X11R6/bin​:/sbin​:/bin​:/usr/games​:/opt/gnome/bin​:/opt/kde3/bin​:/sbin​:/usr/sbin
  PERL_BADLANG (unset)
  SHELL=/bin/tcsh

From nick@ing-simmons.net

Jim Carter <perl5-porters@​perl.org> writes​:

# New Ticket Created by Jim Carter
# Please include the string​: [perl #31455]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org​:80/rt3/Ticket/Display.html?id=31455 >

Can you try this with
PERLIO=perlio perl ...
and
PERLIO=stdio perl ...
and see if it occurs in both cases?

Note that a dup'ed file handle shares the file pointer.
For DATA which perl is reading using buffered IO the
dup tries to do a seek to get fd level file pointer
to match buffered point. Do you _need_ to do a dup ?

This is a bug report for perl from jimc@​math.ucla.edu,
generated with the help of perlbug 1.34 running under perl v5.8.3.

#!/usr/bin/perl
# If the script is not seekable, and if the DATA filehandle is duped, an
# initial segment of the DATA is lost when reading from the duplicate
# handle. With this script the lossage is 4216 bytes from start of
# script or 2959 bytes from start of DATA. In the production
# script where this was discovered, the loss from start of DATA is in
# the range 3500 to 4000 bytes.
# The production context is, we concatenate a "tripwire" type script
# and the filenames and checksums it should verify (customized per
# host), and do ssh remotehost "perl - -f '<&DATA'" where -f tells the
# script where to find the checksum list. That way the hackers have
# little to attack -- only by munging perl itself can they defend
# themselves.
print <<EOF;
Compare these three command lines​:
perl datatrunc.pl # Seekable script, runs perfectly
perl - < datatrunc.pl # Seekable script, runs perfectly
cat datatrunc.pl | perl - # Not seekable, loses initial DATA
EOF
open(DUPL, "<&DATA") or die "Failed to dup DATA filehandle​: $!\n";
while (<DUPL>) { # Each line is exactly 80 bytes including \n
print
}
__DATA__
Line 01 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 02 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 03 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 04 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 05 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 06 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 07 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 08 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 09 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 10 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 11 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 12 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 13 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 14 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 15 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 16 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 17 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 18 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 19 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 20 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 21 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 22 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 23 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 24 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 25 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 26 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 27 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 28 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 29 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 30 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 31 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 32 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 33 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 34 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 35 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 36 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 37 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 38 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 39 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
Line 40 b123456789c123456789d123456789e123456789f123456789g123456789h12345678
---
Flags​:
category=core
severity=medium
---
This perlbug was built using Perl v5.8.3 - Sat Apr 3 00​:50​:30 UTC 2004
It is being executed now by Perl v5.8.3 - Sat Apr 3 00​:37​:21 UTC 2004.

Site configuration information for perl v5.8.3​:

Configured by abuild at Sat Apr 3 00​:37​:21 UTC 2004.

Summary of my perl5 (revision 5.0 version 8 subversion 3) configuration​:
Platform​:
osname=linux, osvers=2.6.4, archname=i586-linux-thread-multi
uname='linux d209 2.6.4 #1 smp thu mar 11 17​:56​:49 utc 2004 i686 i686 i386 gnulinux '
config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Duseshrplib=true -Doptimize=-O2 -march=i586 -mcpu=i686 -fmessage-length=0 -Wall -Wall -pipe'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -march=i586 -mcpu=i686 -fmessage-length=0 -Wall -Wall -pipe',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing'
ccversion='', gccversion='3.3.3 (SuSE Linux)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries​:
ld='cc', ldflags =''
libpth=/lib /usr/lib /usr/local/lib
libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.3.3'
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.3/i586-linux-thread-multi/CORE'
cccdlflags='-fPIC', lddlflags='-shared'

Locally applied patches​:

---
@​INC for perl v5.8.3​:
/usr/lib/perl5/5.8.3/i586-linux-thread-multi
/usr/lib/perl5/5.8.3
/usr/lib/perl5/site_perl/5.8.3/i586-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.3
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.3/i586-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.3
/usr/lib/perl5/vendor_perl
.

---
Environment for perl v5.8.3​:
HOME=/home/jimc
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=.​:/home/jimc/bin​:/usr/diklo/bin​:/usr/local/sbin​:/usr/local/bin​:/usr/sbin​:/usr/bin​:/usr/X11R6/bin​:/sbin​:/bin​:/usr/games​:/opt/gnome/bin​:/opt/kde3/bin​:/sbin​:/usr/sbin
PERL_BADLANG (unset)
SHELL=/bin/tcsh

The RT System itself - Status changed from 'new' to 'open'

From jimc@math.ucla.edu

On Mon, 6 Sep 2004, Nick Ing-Simmons via RT wrote​:

Can you try this with
PERLIO=perlio perl ...
and
PERLIO=stdio perl ...

Yes, it fails equally with perlio and stdio.

Do I _need_ to do a dup? My original motivation was to use standard
features to encompass this unusual case​: normally the table is a plain file
named on the command line, but for ultra paranoia I concatenate the script
and the table, and then specify '<&DATA'. As a workaround, what I'm doing
now is to test for the string '<&DATA', and in that case return \*DATA
rather than opening the so-called filename.

I kind of figured that when a stream was duped, the remainder of the
original filehandle's buffer would be pre-stuffed into the new stream
object, so the duplication would be exact. But of course that takes work,
and the cross-platform issues are probably a disaster. If a partially used
stream object were copied either by memcpy or by its copy constructor, I
wonder if the result would be useable on most platforms?

Your point is well taken that both instances share the same file pointer,
so if you do buffered I/O from both instances interleaved, the data would
be totally trashed.

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email​: jimc@​math.ucla.edu http​://www.math.ucla.edu/~jimc (q.v. for PGP key)

From nick@ing-simmons.net

Jim Carter <jimc@​math.ucla.edu> writes​:

On Mon, 6 Sep 2004, Nick Ing-Simmons via RT wrote​:

Can you try this with
PERLIO=perlio perl ...
and
PERLIO=stdio perl ...

Yes, it fails equally with perlio and stdio.

Good - principle of least surprise seems to be working for our perlio
clone of stdio behaviour ;-)

Do I _need_ to do a dup? My original motivation was to use standard
features to encompass this unusual case​: normally the table is a plain file
named on the command line, but for ultra paranoia I concatenate the script
and the table, and then specify '<&DATA'. As a workaround, what I'm doing
now is to test for the string '<&DATA', and in that case return \*DATA
rather than opening the so-called filename.

I kind of figured that when a stream was duped, the remainder of the
original filehandle's buffer would be pre-stuffed into the new stream
object, so the duplication would be exact.

It does not work like that (as you have discovered). Pre-stuffing
buffer is no use if (as is common) dup is for benefit of a sub-process.
Instead it tries to achieve same effect by doing seek().

But of course that takes work,

We could probably pre-stuff buffers most places (at least with :perlio),
but I am unconvinced it would be a big win.

and the cross-platform issues are probably a disaster. If a partially used
stream object were copied either by memcpy

Wouldn't work - two streams would try and use same buffer memory.

or by its copy constructor,

Most IO libraries are not C++ but C - so no such thing as a "copy constructor".

As I say for :perlio we could make ->Dup() call pre-stuff the buffer.
Perhaps we could do that if seek fails, but that won't help
if dup'ed handle is handed off to a sub-process or accessed
with unbuffered IO.
So I am not sure it is enough "better" to be worth the
complexity speed hit etc.

I
wonder if the result would be useable on most platforms?

Your point is well taken that both instances share the same file pointer,
so if you do buffered I/O from both instances interleaved, the data would
be totally trashed.

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email​: jimc@​math.ucla.edu http​://www.math.ucla.edu/~jimc (q.v. for PGP key)

From @jkeenan

On Thu Sep 16 02​:18​:01 2004, ni-s wrote​:

Jim Carter <jimc@​math.ucla.edu> writes​:

On Mon, 6 Sep 2004, Nick Ing-Simmons via RT wrote​:

Can you try this with
PERLIO=perlio perl ...
and
PERLIO=stdio perl ...

Yes, it fails equally with perlio and stdio.

Good - principle of least surprise seems to be working for our perlio
clone of stdio behaviour ;-)

Do I _need_ to do a dup? My original motivation was to use standard
features to encompass this unusual case​: normally the table is a
plain file
named on the command line, but for ultra paranoia I concatenate the
script
and the table, and then specify '<&DATA'. As a workaround, what I'm
doing
now is to test for the string '<&DATA', and in that case return
\*DATA
rather than opening the so-called filename.

I kind of figured that when a stream was duped, the remainder of the
original filehandle's buffer would be pre-stuffed into the new stream
object, so the duplication would be exact.

It does not work like that (as you have discovered). Pre-stuffing
buffer is no use if (as is common) dup is for benefit of a sub-
process.
Instead it tries to achieve same effect by doing seek().

But of course that takes work,

We could probably pre-stuff buffers most places (at least with
:perlio),
but I am unconvinced it would be a big win.

and the cross-platform issues are probably a disaster. If a
partially used
stream object were copied either by memcpy

Wouldn't work - two streams would try and use same buffer memory.

or by its copy constructor,

Most IO libraries are not C++ but C - so no such thing as a "copy
constructor".

As I say for :perlio we could make ->Dup() call pre-stuff the buffer.
Perhaps we could do that if seek fails, but that won't help
if dup'ed handle is handed off to a sub-process or accessed
with unbuffered IO.
So I am not sure it is enough "better" to be worth the
complexity speed hit etc.

I
wonder if the result would be useable on most platforms?

Your point is well taken that both instances share the same file
pointer,
so if you do buffered I/O from both instances interleaved, the data
would
be totally trashed.

James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA
90095-1555
Email​: jimc@​math.ucla.edu http​://www.math.ucla.edu/~jimc (q.v. for
PGP key)

This is another RT where the person who was responding to the OP's
problem is no longer with us.

Is there anyone who could review the issues raised in this ticket and
make a recommendation?

Thank you very much.
Jim Keenan