Skip Menu |
Report information
Id: 126881
Status: resolved
Priority: 0/
Queue: perl6

Owner: Nobody
Requestors: nanis [at] cpan.org
Cc:
AdminCc:

Severity: (no value)
Tag: (no value)
Platform: (no value)
Patch Status: (no value)
VM: (no value)



Date: Fri, 11 Dec 2015 12:31:18 -0500
Subject: Newline handling is broken on Windows
From: "A. Sinan Unur" <nanis [...] cpan.org>
To: rakudobug [...] perl.org
Download (untitled) / with headers
text/plain 14.2k
64-bit Windows 10.0.10586. 64-bit perl6 on MoarVM built using MS Visual Studio 2013 tools. See below for more details. I noticed[1] that several tests fail if the test script has Unix line endings instead of Windows. These tests succeed if the test script is converted using unix2dos. This is problematic behavior. Tests should pass or fail independently of the EOL convention used in the script file. E.g. see test 20 in t\spec\S02-literals\heredocs.t: my $e = q:to<END>; END is no-r($e), "\n", 'Heredoc one empty line'; This fails with: # Failed test 'Heredoc one empty line' # at t\spec\S02-literals\heredocs.t line 171 # expected: "\r\n" # got: "\n" That is, when the script source uses Unix line endings, $e contains "\n", but, because the test is being run on Windows, the comparison string, "\n" becomes "\r\n". Converting the file with unix2dos fixes the failures, but that is not a solution. "\n" should not be converted to "\r\n" behind the scenes in a program. EOL conversion should only happen at input-output boundaries. What happens in this and other tests that fail depending on whether the test script is saved using Unix or Windows line endings is contrary to decades of convention of doing EOL translation only at input-output boundaries. For more exploration, see [1]: https://www.nu42.com/2015/12/perl6-newline-translation-botched.html moar::syslibs[6]=iphlpapi moar::dcbrule=@: moar::ar=lib moar::mastdir=C:\opt\perl6\share\nqp\lib\MAST moar::tomrule=$(AR) $(ARFLAGS) /out:$@ 3rdparty\libtommath\*.obj moar::syslibs[2]=mswsock moar::shaobjects=3rdparty\sha1\sha1.obj moar::ccshared= moar::be=0 moar::dynasmlua=.\3rdparty\dynasm\dynasm.lua moar::ld=link moar::mtlib=3rdparty\tinymt\tinymt.lib moar::mtobjects=3rdparty\tinymt\tinymt64.obj moar::noreturnspecifier=__declspec(noreturn) moar::perl=c:\opt\perl\5.23.5\bin\perl.exe moar::shaclean=$(RM) 3rdparty\sha1\sha1.lib 3rdparty\sha1\*.obj moar::static_inline=static __inline moar::ccdebugflags=/Zi moar::auxclean=$(RM) moar.ilk moar.pdb moar.dll.lib moar.dll.exp vc100.pdb moar::thirdpartylibs=3rdparty\dyncall\dyncall\libdyncall_s.lib 3rdparty\dyncall\dyncallback\libdyncallback_s.lib 3rdparty\dyncall\dynload\libdynload_s.lib 3rdparty\tinymt\tinymt.lib 3rdparty\sha1\sha1.lib 3rdparty\libtommath\tommath.lib 3rdparty\libuv\uv.lib moar::config=--optimize --prefix=C:\opt\perl6 --make-install moar::ccinc=/I moar::ldout=/out: moar::moarlib=moar.lib moar::mtrule=$(AR) $(ARFLAGS) /out:$@ 3rdparty\tinymt\*.obj moar::syslibs[0]=shell32 moar::dcrule=cd 3rdparty\dyncall && configure.bat /target-x64 && $(MAKE) -f Nmakefile moar::dcbobjects= moar::cflags=/nologo /MT /Ox /GL /DNDEBUG /DWIN32 /DAO_ASSUME_WINDOWS98 moar::laolib=__lao__ moar::libdir=C:\opt\perl6\bin moar::nul=NUL moar::osvers=10.0 moar::dlrule=@: moar::cat=type moar::can_unaligned_int32=1 moar::ldimp=%s.dll.lib moar::win32_compiler_toolchain=win32 moar::ccdef=/D moar::lib=%s.lib moar::tomclean=$(RM) 3rdparty\libtommath\tommath.lib 3rdparty\libtommath\*.obj moar::dlllocal= moar::dcbclean=$(RM) 3rdparty\dyncall\dyncallback\libdyncallback_s.lib moar::noreturnattribute= moar::ptr_size=8 moar::syslibs[5]=psapi moar::dllimport=__declspec(dllimport) moar::ccout=/Fo moar::can_unaligned_num64=1 moar::has_pthread_yield=0 moar::ldrpath= moar::ldusr=%s.lib moar::mainflags=/DMVM_SHARED moar::name=moar moar::dll=%s.dll moar::cincludes= /I3rdparty\libuv\include /I3rdparty\libuv\src /I3rdparty\libatomic_ops\src /I3rdparty\libtommath /I3rdparty\dynasm /I3rdparty\dyncall\dynload /I3rdparty\dyncall\dyncall /I3rdparty\dyncall\dyncallback moar::uvrule=$(AR) $(ARFLAGS) /out:$@ 3rdparty\libuv\src\*.obj 3rdparty\libuv\src\win\*.obj moar::tomlib=3rdparty\libtommath\tommath.lib moar::syslibs[3]=rpcrt4 moar::staticlib= moar::dlclean=$(RM) 3rdparty\dyncall\dynload\libdynload_s.lib moar::make=nmake moar::ccinstflags= moar::ldflags=/nologo /LTCG moar::crossconf= moar::obj=.obj moar::dlobjects= moar::defs[0]=WIN32 moar::cppout=/Fi moar::ccoptiflags=/Ox /GL /DNDEBUG moar::asmswitch=/c /FAs moar::ldoptiflags=/LTCG moar::mkflags=/nologo moar::moar=moar.dll moar::version=2015.11 moar::dcclean=$(RM) 3rdparty\dyncall\ConfigVars 3rdparty\dyncall\dyncall\libdyncall_s.lib 3rdparty\dyncall\dyncallback\libdyncallback_s.lib 3rdparty\dyncall\dynload\libdynload_s.lib 3rdparty\dyncall\dyncall\*.obj 3rdparty\dyncall\dyncallback\*.obj 3rdparty\dyncall\dynload\*.obj moar::moarshared=/implib:moar.dll.lib moar::sharule=$(AR) $(ARFLAGS) /out:$@ 3rdparty\sha1\*.obj moar::ccwarnflags= moar::jit=$(JIT_WIN32_X64) moar::moardll=moar.dll moar::asm=.asm moar::mknoisy=!IF $(NOISY) != 1MSG = @echoCMD = @NOOUT = > NULNOERR = Show quoted text
2> NUL!ENDIF
moar::dclib=3rdparty\dyncall\dyncall\libdyncall_s.lib moar::ccdefflags=/DWIN32 /DAO_ASSUME_WINDOWS98 moar::cc=cl moar::exe=.exe moar::lddir=/libpath: moar::uvlib=3rdparty\libuv\uv.lib moar::dllib=3rdparty\dyncall\dynload\libdynload_s.lib moar::shaincludedir=3rdparty\sha1 moar::versionmajor=2015 moar::cppswitch=/P moar::can_unaligned_int64=1 moar::lua=.\3rdparty\dynasm\minilua.exe moar::dcobjects= moar::cancgoto=0 moar::ldlibs=shell32.lib ws2_32.lib mswsock.lib rpcrt4.lib advapi32.lib psapi.lib iphlpapi.lib moar::mainlibs=/libpath:. moar.dll.lib moar::nativecall_backend=dyncall moar::objflags=/DMVM_BUILD_SHARED moar::versionpatch=0 moar::laoclean=@: moar::versionminor=11 moar::ccmiscflags=/nologo /MT moar::install= $(MKPATH) $(DESTDIR)$(PREFIX)\include\libuv $(CP) 3rdparty\libuv\include\*.h $(DESTDIR)$(PREFIX)\include\libuv $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\armcc $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\gcc $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\hpc $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\ibmc $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\icc $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\loadstore $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\msftc $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\sunc $(CP) 3rdparty\libatomic_ops\src\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops $(CP) 3rdparty\libatomic_ops\src\atomic_ops\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\armcc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\armcc $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\gcc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\gcc $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\hpc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\hpc $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\ibmc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\ibmc $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\icc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\icc $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\loadstore\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\loadstore $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\msftc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\msftc $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\sunc\*.h $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\sunc $(CP) 3rdparty\libtommath\*.h $(DESTDIR)$(PREFIX)\include\libtommath $(MKPATH) $(DESTDIR)$(PREFIX)\include\dyncall $(CP) 3rdparty\dyncall\dynload\*.h $(DESTDIR)$(PREFIX)\include\dyncall $(CP) 3rdparty\dyncall\dyncall\*.h $(DESTDIR)$(PREFIX)\include\dyncall $(CP) 3rdparty\dyncall\dyncallback\*.h $(DESTDIR)$(PREFIX)\include\dyncall moar::laoobjects= moar::mtclean=$(RM) 3rdparty\tinymt\tinymt.lib 3rdparty\tinymt\*.obj moar::rm=del moar::uvclean=$(RM) 3rdparty\libuv\uv.lib 3rdparty\libuv\src\*.obj 3rdparty\libuv\src\win\*.obj moar::sharedlib=moar.dll.lib moar::dllexport=__declspec(dllexport) moar::shalib=3rdparty\sha1\sha1.lib moar::arout=/out: moar::lddebugflags=/debug /pdb:$@.pdb moar::osname=MSWin32 moar::prefix=C:\opt\perl6 moar::uvobjects=3rdparty\libuv\src\fs-poll.obj 3rdparty\libuv\src\inet.obj 3rdparty\libuv\src\threadpool.obj 3rdparty\libuv\src\uv-common.obj 3rdparty\libuv\src\version.obj 3rdparty\libuv\src\win\async.obj 3rdparty\libuv\src\win\core.obj 3rdparty\libuv\src\win\dl.obj 3rdparty\libuv\src\win\error.obj 3rdparty\libuv\src\win\fs-event.obj 3rdparty\libuv\src\win\fs.obj 3rdparty\libuv\src\win\getaddrinfo.obj 3rdparty\libuv\src\win\getnameinfo.obj 3rdparty\libuv\src\win\handle.obj 3rdparty\libuv\src\win\loop-watcher.obj 3rdparty\libuv\src\win\pipe.obj 3rdparty\libuv\src\win\poll.obj 3rdparty\libuv\src\win\process-stdio.obj 3rdparty\libuv\src\win\process.obj 3rdparty\libuv\src\win\req.obj 3rdparty\libuv\src\win\signal.obj 3rdparty\libuv\src\win\stream.obj 3rdparty\libuv\src\win\tcp.obj 3rdparty\libuv\src\win\thread.obj 3rdparty\libuv\src\win\timer.obj 3rdparty\libuv\src\win\tty.obj 3rdparty\libuv\src\win\udp.obj 3rdparty\libuv\src\win\util.obj 3rdparty\libuv\src\win\winapi.obj 3rdparty\libuv\src\win\winsock.obj moar::dcblib=3rdparty\dyncall\dyncallback\libdyncallback_s.lib moar::bindir=C:\opt\perl6\bin moar::asmout=/Fa moar::laorule=@: moar::os=win32 moar::sh=cmd moar::arflags=/nologo moar::ldshared=/dll moar::defs[1]=AO_ASSUME_WINDOWS98 moar::impinst=moar.dll.lib moar::ldinstflags=/Profile moar::ldmiscflags=/nologo moar::ldsys=%s.lib moar::platform=$(PLATFORM_WIN32) moar::ccswitch=/c moar::formatattribute= moar::tomobjects=3rdparty\libtommath\bn_error.obj 3rdparty\libtommath\bn_fast_mp_invmod.obj 3rdparty\libtommath\bn_fast_mp_montgomery_reduce.obj 3rdparty\libtommath\bn_fast_s_mp_mul_digs.obj 3rdparty\libtommath\bn_fast_s_mp_mul_high_digs.obj 3rdparty\libtommath\bn_fast_s_mp_sqr.obj 3rdparty\libtommath\bn_mp_2expt.obj 3rdparty\libtommath\bn_mp_abs.obj 3rdparty\libtommath\bn_mp_add.obj 3rdparty\libtommath\bn_mp_add_d.obj 3rdparty\libtommath\bn_mp_addmod.obj 3rdparty\libtommath\bn_mp_and.obj 3rdparty\libtommath\bn_mp_clamp.obj 3rdparty\libtommath\bn_mp_clear.obj 3rdparty\libtommath\bn_mp_clear_multi.obj 3rdparty\libtommath\bn_mp_cmp.obj 3rdparty\libtommath\bn_mp_cmp_d.obj 3rdparty\libtommath\bn_mp_cmp_mag.obj 3rdparty\libtommath\bn_mp_cnt_lsb.obj 3rdparty\libtommath\bn_mp_copy.obj 3rdparty\libtommath\bn_mp_count_bits.obj 3rdparty\libtommath\bn_mp_div.obj 3rdparty\libtommath\bn_mp_div_2.obj 3rdparty\libtommath\bn_mp_div_2d.obj 3rdparty\libtommath\bn_mp_div_3.obj 3rdparty\libtommath\bn_mp_div_d.obj 3rdparty\libtommath\bn_mp_dr_is_modulus.obj 3rdparty\libtommath\bn_mp_dr_reduce.obj 3rdparty\libtommath\bn_mp_dr_setup.obj 3rdparty\libtommath\bn_mp_exch.obj 3rdparty\libtommath\bn_mp_expt_d.obj 3rdparty\libtommath\bn_mp_exptmod.obj 3rdparty\libtommath\bn_mp_exptmod_fast.obj 3rdparty\libtommath\bn_mp_exteuclid.obj 3rdparty\libtommath\bn_mp_fread.obj 3rdparty\libtommath\bn_mp_fwrite.obj 3rdparty\libtommath\bn_mp_gcd.obj 3rdparty\libtommath\bn_mp_get_int.obj 3rdparty\libtommath\bn_mp_get_long.obj 3rdparty\libtommath\bn_mp_grow.obj 3rdparty\libtommath\bn_mp_init.obj 3rdparty\libtommath\bn_mp_init_copy.obj 3rdparty\libtommath\bn_mp_init_multi.obj 3rdparty\libtommath\bn_mp_init_set.obj 3rdparty\libtommath\bn_mp_init_set_int.obj 3rdparty\libtommath\bn_mp_init_size.obj 3rdparty\libtommath\bn_mp_invmod.obj 3rdparty\libtommath\bn_mp_invmod_slow.obj 3rdparty\libtommath\bn_mp_is_square.obj 3rdparty\libtommath\bn_mp_jacobi.obj 3rdparty\libtommath\bn_mp_karatsuba_mul.obj 3rdparty\libtommath\bn_mp_karatsuba_sqr.obj 3rdparty\libtommath\bn_mp_lcm.obj 3rdparty\libtommath\bn_mp_lshd.obj 3rdparty\libtommath\bn_mp_mod.obj 3rdparty\libtommath\bn_mp_mod_2d.obj 3rdparty\libtommath\bn_mp_mod_d.obj 3rdparty\libtommath\bn_mp_montgomery_calc_normalization.obj 3rdparty\libtommath\bn_mp_montgomery_reduce.obj 3rdparty\libtommath\bn_mp_montgomery_setup.obj 3rdparty\libtommath\bn_mp_mul.obj 3rdparty\libtommath\bn_mp_mul_2.obj 3rdparty\libtommath\bn_mp_mul_2d.obj 3rdparty\libtommath\bn_mp_mul_d.obj 3rdparty\libtommath\bn_mp_mulmod.obj 3rdparty\libtommath\bn_mp_n_root.obj 3rdparty\libtommath\bn_mp_neg.obj 3rdparty\libtommath\bn_mp_or.obj 3rdparty\libtommath\bn_mp_prime_fermat.obj 3rdparty\libtommath\bn_mp_prime_is_divisible.obj 3rdparty\libtommath\bn_mp_prime_is_prime.obj 3rdparty\libtommath\bn_mp_prime_miller_rabin.obj 3rdparty\libtommath\bn_mp_prime_next_prime.obj 3rdparty\libtommath\bn_mp_prime_rabin_miller_trials.obj 3rdparty\libtommath\bn_mp_prime_random_ex.obj 3rdparty\libtommath\bn_mp_radix_size.obj 3rdparty\libtommath\bn_mp_radix_smap.obj 3rdparty\libtommath\bn_mp_rand.obj 3rdparty\libtommath\bn_mp_read_radix.obj 3rdparty\libtommath\bn_mp_read_signed_bin.obj 3rdparty\libtommath\bn_mp_read_unsigned_bin.obj 3rdparty\libtommath\bn_mp_reduce.obj 3rdparty\libtommath\bn_mp_reduce_2k.obj 3rdparty\libtommath\bn_mp_reduce_2k_l.obj 3rdparty\libtommath\bn_mp_reduce_2k_setup.obj 3rdparty\libtommath\bn_mp_reduce_2k_setup_l.obj 3rdparty\libtommath\bn_mp_reduce_is_2k.obj 3rdparty\libtommath\bn_mp_reduce_is_2k_l.obj 3rdparty\libtommath\bn_mp_reduce_setup.obj 3rdparty\libtommath\bn_mp_rshd.obj 3rdparty\libtommath\bn_mp_set.obj 3rdparty\libtommath\bn_mp_set_int.obj 3rdparty\libtommath\bn_mp_set_long.obj 3rdparty\libtommath\bn_mp_shrink.obj 3rdparty\libtommath\bn_mp_signed_bin_size.obj 3rdparty\libtommath\bn_mp_sqr.obj 3rdparty\libtommath\bn_mp_sqrmod.obj 3rdparty\libtommath\bn_mp_sqrt.obj 3rdparty\libtommath\bn_mp_sub.obj 3rdparty\libtommath\bn_mp_sub_d.obj 3rdparty\libtommath\bn_mp_submod.obj 3rdparty\libtommath\bn_mp_to_signed_bin.obj 3rdparty\libtommath\bn_mp_to_signed_bin_n.obj 3rdparty\libtommath\bn_mp_to_unsigned_bin.obj 3rdparty\libtommath\bn_mp_to_unsigned_bin_n.obj 3rdparty\libtommath\bn_mp_toom_mul.obj 3rdparty\libtommath\bn_mp_toom_sqr.obj 3rdparty\libtommath\bn_mp_toradix.obj 3rdparty\libtommath\bn_mp_toradix_n.obj 3rdparty\libtommath\bn_mp_unsigned_bin_size.obj 3rdparty\libtommath\bn_mp_xor.obj 3rdparty\libtommath\bn_mp_zero.obj 3rdparty\libtommath\bn_prime_tab.obj 3rdparty\libtommath\bn_reverse.obj 3rdparty\libtommath\bn_s_mp_add.obj 3rdparty\libtommath\bn_s_mp_exptmod.obj 3rdparty\libtommath\bn_s_mp_mul_digs.obj 3rdparty\libtommath\bn_s_mp_mul_high_digs.obj 3rdparty\libtommath\bn_s_mp_sqr.obj 3rdparty\libtommath\bn_s_mp_sub.obj 3rdparty\libtommath\bncore.obj moar::syslibs[4]=advapi32 moar::syslibs[1]=ws2_32 perl6::name=rakudo perl6::codename= perl6::release-number= perl6::build-date=2015-12-10T20:31:31Z perl6::version=2015.11
RT-Send-CC: perl6-compiler [...] perl.org
Download (untitled) / with headers
text/plain 2.1k
On Fri Dec 11 09:31:44 2015, nanis@cpan.org wrote: Show quoted text
> 64-bit Windows 10.0.10586. 64-bit perl6 on MoarVM built using MS > Visual Studio 2013 tools. See below for more details. > > I noticed[1] that several tests fail if the test script has Unix line > endings instead of Windows. These tests succeed if the test script is > converted using unix2dos. >
Also was fine if you have autoclrf set in Git, fwiw. Show quoted text
> This is problematic behavior. Tests should pass or fail independently > of the EOL convention used in the script file. > > E.g. see test 20 in t\spec\S02-literals\heredocs.t: > > my $e = q:to<END>; > > END > is no-r($e), "\n", 'Heredoc one empty line'; > > This fails with: > > # Failed test 'Heredoc one empty line' > # at t\spec\S02-literals\heredocs.t line 171 > # expected: "\r\n" > # got: "\n" > > That is, when the script source uses Unix line endings, $e contains > "\n", but, because the test is being run on Windows, the comparison > string, "\n" becomes "\r\n". Converting the file with unix2dos fixes > the failures, but that is not a solution. "\n" should not be converted > to "\r\n" behind the scenes in a program. EOL conversion should only > happen at input-output boundaries. > > What happens in this and other tests that fail depending on whether > the test script is saved using Unix or Windows line endings is > contrary to decades of convention of doing EOL translation only at > input-output boundaries. > > For more exploration, see > > [1]: https://www.nu42.com/2015/12/perl6-newline-translation- > botched.html >
The logical \n in strings (mirroring logical \n in regexes) didn't work out so well in reality as it did in theory. Now: * We do translation of \r\n graphemes to \n on all input read as text except sockets, independent of platform * We do translation of all \n graphemes to \r\n on text output to handles except sockets, on Windows only * \n is now, unless `use newline` is in force, always \x0A * We don't do any such translation when using .encode/.decode, and of course when reading/writing Bufs to files, providing an escape hatch from translation if needed This hopefully addresses the various issues. Various tests have been updated accordingly.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org