embedded perl always calls setlocale(LC_ALL,"") #8274

p5pRT · 2006-01-09T23:15:52Z

Migrated from rt.perl.org#38193 (status was 'resolved')

Searchable as RT38193$

p5pRT · 2006-01-09T23:15:53Z

From andrew@dunslane.net

Created by andrew@dunslane.net

An embedded perl interpreter always starts by calling setlocale(LC_ALL,"").
See perl.c::perl_construct() and locale.c::Perl_init_i18nl10n().

This is not very friendly - embedded interpreters should operate in the locale
that is set up for them by the containing program. In our case (PLPerl inside
PostgreSQL) it can potentially cause data corruption by monkeying with the
collation order. On Unix systems we can mitigate this effect by setting up
the environment as needed, but on Windows setlocale(LC_*,"") does not get
its values from the environment.

There should be a way create an embedded interpreter that avoids calling
Perl_init_i18nl10n() or at least a way of either inhibiting the calls to
setlocale() or forcing the second argument to be NULL (thus causing no change).

Perl Info


Flags:
    category=core
    severity=medium

This perlbug was built using Perl v5.8.5 in the Red Hat build system.
It is being executed now by Perl v5.8.5 - Fri Dec 16 14:48:17 EST 2005.

Site configuration information for perl v5.8.5:

Configured by Red Hat, Inc. at Fri Dec 16 14:48:17 EST 2005.

Summary of my perl5 (revision 5 version 8 subversion 5) configuration:
  Platform:
    osname=linux, osvers=2.6.9-1.906_elsmp, archname=i386-linux-thread-multi
    uname='linux tweety.build.redhat.com 2.6.9-1.906_elsmp #1 smp sun dec 12 22:58:08 est 2004 i686 i686 i386 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -m32 -march=i386 -mtune=pentium4 -Dversion=5.8.5 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dinc_version_list=5.8.4 5.8.3 5.8.2 5.8.1 5.8.0'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -g -pipe -m32 -march=i386 -mtune=pentium4',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.4.4 20050721 (Red Hat 3.4.4-2)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.3.6.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.3.6'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.5:
    /usr/lib/perl5/5.8.5/i386-linux-thread-multi
    /usr/lib/perl5/5.8.5
    /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.5
    /usr/lib/perl5/site_perl/5.8.4
    /usr/lib/perl5/site_perl/5.8.3
    /usr/lib/perl5/site_perl/5.8.2
    /usr/lib/perl5/site_perl/5.8.1
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.5
    /usr/lib/perl5/vendor_perl/5.8.4
    /usr/lib/perl5/vendor_perl/5.8.3
    /usr/lib/perl5/vendor_perl/5.8.2
    /usr/lib/perl5/vendor_perl/5.8.1
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    .


Environment for perl v5.8.5:
    HOME=/home/andrew
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/tmp/.dgtaX:/usr/kerberos/bin:/usr/lib/ccache/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/andrew/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2013-02-01T18:45:19Z

From @khwilliamson

Could this be a Configure option, or an environment variable option?

--
Karl Williamson

p5pRT · 2013-02-01T18:45:19Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2013-02-01T22:07:17Z

From andrew@dunslane.net

On 02/01/2013 01:45 PM, Karl Williamson via RT wrote:

Could this be a Configure option, or an environment variable option?

I'm not sure why are you asking me.

cheers

andrew

p5pRT · 2013-02-02T16:48:06Z

From @khwilliamson

On 02/01/2013 12:03 PM, Andrew Dunstan wrote:

On 02/01/2013 01:45 PM, Karl Williamson via RT wrote:

Could this be a Configure option, or an environment variable option?

I'm not sure why are you asking me.

cheers

andrew

Someone with your email address filed a bug report
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=38193
against the Perl programming language 7 years ago. The report has sat
unresponded to until I looked at it and sent the email you got.

If it was you who filed the original ticket, is this a problem that you
still think should be addressed? even if you've found a work around, or
abandoned Perl, etc.

An easy implementation that occurred to me was to have an environment
variable, say PERL_LOCALE_ALREADY_SET, which if defined would tell Perl
not to do a setlocale(LC_ALL,""). I hope that that would meet the
original requester's concern.

If you are not the original requestor, please let us know so that we can
remove your email address from this bug report; if you have an opinion
on this, please respond with it.

Perl is an essentially all volunteer organization, and apparently no one
with enough time and appropriate skill set looked at this bug in all the
intervening years. I'm sorry about that.

I think that something like this fix has merit, even if the original
requester is long gone.

p5pRT · 2013-02-04T00:19:14Z

From andrew@dunslane.net

On 02/02/2013 11:47 AM, Karl Williamson wrote:

On 02/01/2013 12:03 PM, Andrew Dunstan wrote:

On 02/01/2013 01:45 PM, Karl Williamson via RT wrote:

Could this be a Configure option, or an environment variable option?

I'm not sure why are you asking me.

cheers

andrew

Someone with your email address filed a bug report
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=38193
against the Perl programming language 7 years ago. The report has sat
unresponded to until I looked at it and sent the email you got.

If it was you who filed the original ticket, is this a problem that
you still think should be addressed? even if you've found a work
around, or abandoned Perl, etc.

An easy implementation that occurred to me was to have an environment
variable, say PERL_LOCALE_ALREADY_SET, which if defined would tell
Perl not to do a setlocale(LC_ALL,""). I hope that that would meet
the original requester's concern.

If you are not the original requestor, please let us know so that we
can remove your email address from this bug report; if you have an
opinion on this, please respond with it.

Perl is an essentially all volunteer organization, and apparently no
one with enough time and appropriate skill set looked at this bug in
all the intervening years. I'm sorry about that.

I think that something like this fix has merit, even if the original
requester is long gone.

No, it was probably me. 7 years is a long time :-) I'm glad the Perl
community is cleaning up the old bugs. PERL_LOCALE_ALREADY_SET would
work. But if we're to use it we need to know if the feature is
supported, and that test needs to be available before we call
perl_alloc() - I'm not sure how easy that would be to do - maybe a
static test would do. See
<https://github.com/postgres/postgres/blob/master/src/pl/plperl/plperl.c> at
about line 695 for an example of the current PostgreSQL workaround.

cheers

andrew

p5pRT · 2013-06-24T20:07:08Z

From @khwilliamson

On 02/03/2013 08:13 AM, Andrew Dunstan wrote:

On 02/02/2013 11:47 AM, Karl Williamson wrote:

On 02/01/2013 12:03 PM, Andrew Dunstan wrote:

On 02/01/2013 01:45 PM, Karl Williamson via RT wrote:

Could this be a Configure option, or an environment variable option?

I'm not sure why are you asking me.

cheers

andrew

Someone with your email address filed a bug report
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=38193
against the Perl programming language 7 years ago. The report has sat
unresponded to until I looked at it and sent the email you got.

If it was you who filed the original ticket, is this a problem that
you still think should be addressed? even if you've found a work
around, or abandoned Perl, etc.

An easy implementation that occurred to me was to have an environment
variable, say PERL_LOCALE_ALREADY_SET, which if defined would tell
Perl not to do a setlocale(LC_ALL,""). I hope that that would meet
the original requester's concern.

If you are not the original requestor, please let us know so that we
can remove your email address from this bug report; if you have an
opinion on this, please respond with it.

Perl is an essentially all volunteer organization, and apparently no
one with enough time and appropriate skill set looked at this bug in
all the intervening years. I'm sorry about that.

I think that something like this fix has merit, even if the original
requester is long gone.

No, it was probably me. 7 years is a long time :-) I'm glad the Perl
community is cleaning up the old bugs. PERL_LOCALE_ALREADY_SET would
work. But if we're to use it we need to know if the feature is
supported, and that test needs to be available before we call
perl_alloc() - I'm not sure how easy that would be to do - maybe a
static test would do. See
<https://github.com/postgres/postgres/blob/master/src/pl/plperl/plperl.c> at
about line 695 for an example of the current PostgreSQL workaround.

I am actually starting work on this now, and am gathering more
information. The windows setlocale documentation
http://msdn.microsoft.com/en-us/library/hzz3tw78

says "If locale points to an empty string, the locale is the
implementation-defined native environment."

I presume the problem is that you have set things up to override the
native environment of the implementation, and Perl is screwing that up.
If so, it would seem that the issue isn't just LC_ALL, but all the
categories that Perl calls setlocal(foo, "") on. If it is just the
LC_ALL call, please give me more detail about your situation.

p5pRT · 2013-07-10T04:27:18Z

From @khwilliamson

This is fixed by commit ccd65d5

This commit causes the locale initialization to skip calling
setlocal(foo, "") if the environment variable PERL_SKIP_LOCALE_INIT is
set. Instead, the setup code calls setlocale(LC_ALL, NULL) (plus other
similar calls for the subcategories) in order to find out what the
current locale is.

The original poster for this ticket has a workaround for it which
involves using a modified copy of Perl core code. This patch defines
the C preprocessor variable HAS_SKIP_LOCALE_INIT that can be used by XS
writers to discover if the current Perl version needs the workaround or
not.

--
Karl Williamson

p5pRT · 2013-07-10T04:27:19Z

@khwilliamson - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Jul 10, 2013

p5pRT added Severity Medium XS distro-Linux type-Unicode type-core labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

embedded perl always calls setlocale(LC_ALL,"") #8274

embedded perl always calls setlocale(LC_ALL,"") #8274

p5pRT commented Jan 9, 2006

p5pRT commented Jan 9, 2006

p5pRT commented Feb 1, 2013

p5pRT commented Feb 1, 2013

p5pRT commented Feb 1, 2013

p5pRT commented Feb 2, 2013

p5pRT commented Feb 4, 2013

p5pRT commented Jun 24, 2013

p5pRT commented Jul 10, 2013

p5pRT commented Jul 10, 2013

embedded perl always calls setlocale(LC_ALL,"") #8274

embedded perl always calls setlocale(LC_ALL,"") #8274

Comments

p5pRT commented Jan 9, 2006

p5pRT commented Jan 9, 2006

From andrew@dunslane.net

Created by andrew@dunslane.net

p5pRT commented Feb 1, 2013

From @khwilliamson

p5pRT commented Feb 1, 2013

p5pRT commented Feb 1, 2013

From andrew@dunslane.net

p5pRT commented Feb 2, 2013

From @khwilliamson

p5pRT commented Feb 4, 2013

From andrew@dunslane.net

p5pRT commented Jun 24, 2013

From @khwilliamson

p5pRT commented Jul 10, 2013

From @khwilliamson

p5pRT commented Jul 10, 2013