From andrew@pimlott.net

Created by andrew@pimlott.net

I found that the PERL_UNICODE setting affects the "main" script as expected,
but does not affect any file included with 'do'. This is terribly
unintuitive and not suggested by perlrun(1). I hope it is a mistake that
can be fixed.

To demo, I put basically the same code in t1.pl​:

  open( OUT, ">+out" ) or die $!;
  print OUT chr(1234), "\n";
  close OUT;
  do 't2.pl';

and t2.pl​:

  open( OUT, ">out" ) or die $!;
  print OUT chr(1234), "\n";
  close OUT;

and run

  % PERL_UNICODE=31 perl -w t1.pl
  Wide character in print at t2.pl line 2.

Note the warning about t2.pl but not t1.pl.

I stumbled across this while trying to figure out why the perl debugger
didn't have it's handles in utf8 mode when I set PERL_UNICODE. Actually,
I'd really like the debugger, being an interactive application, to set its
handles based on my locale by default. However, at very least, it should
respect my PERL_UNICODE setting, or at very very least, give me some way to
set its handles to utf8. It seems like an obvious need, so maybe I've
missed something?

This transcript demonstrates that my stdout is utf8, but the debugger's OUT
handle is not​:

  % PERL_UNICODE=31 perl -debug

  Loading DB routines from perl5db.pl version 1.3
  Editor support available.

  Enter h or `h h' for help, or `man perldebug' for more help.

  main​::(-e​:1)​: bug
  DB<1> print chr(1234)
  Ӓ
  DB<2> p chr(1234)
  Wide character in print at (eval 6)[/usr/share/perl/5.10/perl5db.pl​:638] line 2.
  at (eval 6)[/usr/share/perl/5.10/perl5db.pl​:638] line 2
  eval '($@​, $!, $^E, $,, $/, $\\, $^W) = @​saved;package main; $^D = $^D | $DB​::db_stop;
  print {$DB​::OUT} chr(1234);

  ;' called at /usr/share/perl/5.10/perl5db.pl line 638
  DB​::eval called at /usr/share/perl/5.10/perl5db.pl line 3434
  DB​::DB called at -e line 1
  Ӓ

Andrew

Flags:
    category=core
    severity=low

Site configuration information for perl 5.10.0:

Configured by Debian Project at Sun Aug 16 22:37:28 UTC 2009.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.26-2-amd64, archname=i486-linux-gnu-thread-multi
    uname='linux puccini 2.6.26-2-amd64 #1 smp fri aug 14 07:12:04 utc 2009 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.3.4', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /usr/lib64
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.9.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0
    gnulibc_version='2.9'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.10.0:
    /home/andrew/lib/perl
    /home/andrew/local/lib/perl
    /etc/perl
    /usr/local/lib/perl/5.10.0
    /usr/local/share/perl/5.10.0
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.10
    /usr/share/perl/5.10
    /usr/local/lib/site_perl
    .


Environment for perl 5.10.0:
    HOME=/home/andrew
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/andrew/bin:/usr/sbin:/sbin:/home/andrew/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
    PERL5LIB=/home/andrew/lib/perl:/home/andrew/local/lib/perl
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zsh

From @rgs

2009/9/10 andrew@​pimlott.net (via RT) <perlbug-followup@​perl.org>​:

I found that the PERL_UNICODE setting affects the "main" script as expected,
but does not affect any file included with 'do'.  This is terribly
unintuitive and not suggested by perlrun(1).  I hope it is a mistake that
can be fixed.

PERL_UNICODE is a red herring; if you print ${^UNICODE}, you'll see
that it's correctly (and globally) set everywhere.

However, at startup, PERL_UNICODE (or -C) also trigger the setting of
the ${^OPEN} variable if relevant, to store what the default layers
should be for non-stdin/out/err filehandles. (The "open" pragma also
uses this magic variable.) But, ${^OPEN} is lexically scoped, like all
other internal variables that reflect pragma states ($^H, %^H, $[). So
its value is lost in the file required by do().

This is arguably a bug and PERL_UNICODE should rule in all scopes.
The potential of breakage is impressive, here; but the current behaviour
is not documented and is probably not the one that was intended.

Opinions ?

--
Anaximander of Miletus considered that from warmed up water and earth
emerged either fish or entirely fishlike animals. Inside these animals,
men took form and embryos were held prisoners until puberty.
  -- Censorinus

The RT System itself - Status changed from 'new' to 'open'

From @cowens

On Thu, Sep 10, 2009 at 18​:34, Rafael Garcia-Suarez
<rgarciasuarez@​gmail.com> wrote​:

2009/9/10 andrew@​pimlott.net (via RT) <perlbug-followup@​perl.org>​:

I found that the PERL_UNICODE setting affects the "main" script as expected,
but does not affect any file included with 'do'.  This is terribly
unintuitive and not suggested by perlrun(1).  I hope it is a mistake that
can be fixed.

PERL_UNICODE is a red herring; if you print ${^UNICODE}, you'll see
that it's correctly (and globally) set everywhere.

However, at startup, PERL_UNICODE (or -C) also trigger the setting of
the ${^OPEN} variable if relevant, to store what the default layers
should be for non-stdin/out/err filehandles. (The "open" pragma also
uses this magic variable.) But, ${^OPEN} is lexically scoped, like all
other internal variables that reflect pragma states ($^H, %^H, $[). So
its value is lost in the file required by do().

This is arguably a bug and PERL_UNICODE should rule in all scopes.
The potential of breakage is impressive, here; but the current behaviour
is not documented and is probably not the one that was intended.

Opinions ?

--
Anaximander of Miletus considered that from warmed up water and earth
emerged either fish or entirely fishlike animals. Inside these animals,
men took form and embryos were held prisoners until puberty.
 -- Censorinus

I use PERL_UNICODE=SDL and had expected it to hold true for everything
my program did.

--
Chas. Owens
wonkden.net
The most important skill a programmer can have is the ability to read.