From @wchristian

Created by @wchristian

Reproduction Steps​:

1. run windows system
2. create a file 'newline' with a \n inside it
3. open shell
4. > set PERLIO=raw
5. > perl -p -e "" -- newline > newline2

Actual Result​:

The file newline2 contains \r\n.

Expected Result​:

The file newline2 should contain a single \n.

Comments​:

Besides -p ignoring the PERLIO env variable neither Tony Cook nor
i could find a way to make -p not mangle the newline.

Flags:
     category=core
     severity=medium

Site configuration information for perl 5.12.4:

Configured by gecko at Mon Jun 20 18:32:45 2011.

Summary of my perl5 (revision 5 version 12 subversion 4) configuration:

   Platform:
     osname=MSWin32, osvers=5.2, archname=MSWin32-x86-multi-thread
     uname=''
     config_args='undef'
     hint=recommended, useposix=true, d_sigaction=undef
     useithreads=define, usemultiplicity=define
     useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
     use64bitint=undef, use64bitall=undef, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='C:/Perl/site/bin/gcc.exe', ccflags ='-DNDEBUG -DWIN32 -D_CONSOLE -DNO_STRICT -DHAVE_DES_FCRYPT -DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T -DPERL_MSVCRT_READFIX -DHASATTRIBUTE -fno-strict-aliasing -mms-bitfields',
     optimize='-O2',
     cppflags='-DWIN32'
     ccversion='', gccversion='3.4.5 (mingw-vista special r3)', gccosandvers=''
     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
     d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8
     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='C:\Perl\site\bin\g++.exe', ldflags ='-L"C:\Perl\lib\CORE"'
     libpth=\lib
     libs=-lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 -lmsvcrt
     perllibs=-lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 -lmsvcrt
     libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl512.lib
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
     cccdlflags=' ', lddlflags='-mdll -L"C:\Perl\lib\CORE"'

Locally applied patches:
     ACTIVEPERL_LOCAL_PATCHES_ENTRY
     c6fbf28 [perl #71806] perldb does not setup %dbline with the shebang option -d
     1fd8fa4 Add Wolfram Humann to AUTHORS
     f120055 make string-append on win32 100 times faster
     a2a8d15 Define _USE_32BIT_TIME_T for VC6 and VC7
     007cfe1 Don't pretend to support really old VC++ compilers
     6d8f7c9 Get rid of obsolete PerlCRT.dll support
     d956618 Make Term::ReadLine::findConsole fall back to STDIN if /dev/tty can't be opened
     321e50c Escape patch strings before embedding them in patchlevel.h


@INC for perl 5.12.4:
     C:/Perl/site/lib
     C:/Perl/lib
     .


Environment for perl 5.12.4:
     CYGWIN=nodosfilewarning
     HOME=c:\Dokumente und Einstellungen\Administrator
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=C:\Perl\site\bin;C:\Perl\bin;C:\Perl14\site\bin;C:\Perl14\bin;C:\Programme\PC Connectivity Solution\;C:\Programme\ActiveState Komodo IDE 6;C:\Programme\NVIDIA Corporation\PhysX\Common;C:\wamp\bin\php\php5.3.0\;C:\Programme\ImageMagick-6.6.7-Q16;C:\Perl\bin;C:\Perl\site\bin;C:\Programme\CollabNet Subversion Server\;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\Programme\Gemeinsame Dateien\Lenovo;C:\Programme\Microsoft SQL Server\90\Tools\binn\;C:\Programme\WinSCP\;C:\Programme\VDMSound;C:\Programme\Subversion;C:\Programme\QuickTime\QTSystem\;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Programme\Intel\WiFi\bin\;c:\Programme\PostgreSQL\8.4\bin\;C:\Programme\TortoiseSVN\bin;c:\MinGW\dmake\;C:\MinGW\bin\;C:\blast\bin;C:\Programme\OpenVPN\bin;c:\Sysinternals;c:\Programme\MySQL\MySQL Server 5.1\bin\;C:\cygwin\bin;C:\Programme\Graphviz2.26.3\bin;C:\Programme\Git\cmd;C:\Programme\TortoiseGit\bin
     PERLIO=raw
     PERL_BADLANG (unset)
     SHELL (unset)

From @wchristian

Related and much more detailed, tchrist stumbled over this as well​:

http​://groups.google.com/group/perl.perl5.porters/browse_thread/
thread/59be3b60706f339b?pli=1

The RT System itself - Status changed from 'new' to 'open'

From @ikegami

On Sun, Oct 2, 2011 at 7​:17 AM, Christian Walde
<perlbug-followup@​perl.org>wrote​:

# New Ticket Created by "Christian Walde"
# Please include the string​: [perl #100496]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=100496 >

This is a bug report for perl from walde.christian@​googlemail.com,
generated with the help of perlbug 1.39 running under perl 5.12.4.

-----------------------------------------------------------------
[Please describe your issue here]

Reproduction Steps​:

1. run windows system
2. create a file 'newline' with a \n inside it
3. open shell
4. > set PERLIO=raw
5. > perl -p -e "" -- newline > newline2

-p seems o be a red herring.

set PERLIO=raw

perl -E"say 'foo';" > foo

perl -E"open my $fh, '&gt;', 'bar' or die $!; say $fh 'bar';"

dir
...
2011-10-02 03​:06 PM 4 bar
2011-10-02 03​:04 PM 5 foo
...

(4 = "f","o","o",LF)
(5 = "f","o","o",CR,LF)

So the problem is that PERLIO doesn't affect STDOUT. There may be similar
issues regarding ARGV and STDIN.

(I used 5.14.0)

From @Leont

On Sun, Oct 2, 2011 at 9​:08 PM, Eric Brine <ikegami@​adaelis.com> wrote​:

So the problem is that PERLIO doesn't affect STDOUT. There may be similar
issues regarding ARGV and STDIN.

No, it does affect STDOUT and friends, it just does the wrong thing​:
it knocks out the buffering layer (which on windows moonlights as crlf
layer). Check the output of «PERLIO=raw perl -E 'say STDERR join "\n",
PerlIO​::get_layers(\*STDOUT)'». This bug doesn't even depend on -p, or
on OS (though ).

I have a hunch what's happening, the PERLIO variables triggers a code
path that doesn't handle pseudo layers properly (​:bytes and :utff are
similarly affected). PerlIO is such a mess :-(

Leon T

From @craigberry

On Sun, Oct 2, 2011 at 5​:24 PM, Leon Timmermans <fawaka@​gmail.com> wrote​:

On Sun, Oct 2, 2011 at 9​:08 PM, Eric Brine <ikegami@​adaelis.com> wrote​:

So the problem is that PERLIO doesn't affect STDOUT. There may be similar
issues regarding ARGV and STDIN.

No, it does affect STDOUT and friends, it just does the wrong thing​:
it knocks out the buffering layer

Yes, that looks like exactly what it's doing​:

1191 if (s) {
1192 PerlIO_parse_layers(aTHX_ PL_def_layerlist, s);
1193 }
1194 else {
1195 PerlIO_default_buffer(aTHX_ PL_def_layerlist);
1196 }

from

http​://perl5.git.perl.org/perl.git/blob/HEAD​:/perlio.c#l1191

where s is the string containing what PERLIO translated to.

From @ikegami

On Sun, Oct 2, 2011 at 6​:24 PM, Leon Timmermans <fawaka@​gmail.com> wrote​:

On Sun, Oct 2, 2011 at 9​:08 PM, Eric Brine <ikegami@​adaelis.com> wrote​:

So the problem is that PERLIO doesn't affect STDOUT. There may be similar
issues regarding ARGV and STDIN.

No, it does affect STDOUT and friends, it just does the wrong thing​:
it knocks out the buffering layer

raw has long done more than it's documented as doing. The following didn't
work before 5.14 because raw would pop the scalar var handler​:

perl -wE"{ open my $fh, '&gt;​:raw', \$_ or die $!; say $fh 'foo' } print;"

I guess the fix wasn't complete.

(which on windows moonlights as crlf

layer). Check the output of «PERLIO=raw perl -E 'say STDERR join "\n",

PerlIO​::get_layers(\*STDOUT)'»

On Windows, that shows the :crlf gone.

Thanks for helping me refine the problem.

- Eric

From @Leont

On Mon, Oct 3, 2011 at 1​:04 AM, Craig A. Berry <craig.a.berry@​gmail.com> wrote​:

Yes, that looks like exactly what it's doing​:

1191         if (s) {
1192             PerlIO_parse_layers(aTHX_ PL_def_layerlist, s);
1193         }
1194         else {
1195             PerlIO_default_buffer(aTHX_ PL_def_layerlist);
1196         }

from

http​://perl5.git.perl.org/perl.git/blob/HEAD​:/perlio.c#l1191

where s is the string containing what PERLIO translated to.

When PerlIO is initialized, the standard in/out/err are all explicitly
opened as textstreams. It seems that, lacking the crlf layer because
of :raw the newline translation is actually done natively; this is
rather surprising because it's kind of hard to trigger this edge case,
I'm not even sure why we'd want this to be possible in the first
place.

Leon