Skip Menu |
Report information
Id: 133157
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: carlos [at] carlosguevara.com
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: unknown
Perl Version: (no value)
Fixed In: (no value)



To: perlbug [...] perl.org
Date: Fri, 27 Apr 2018 00:06:07 -0400
Subject: Solaris Failing Some Locale Tests
From: Carlos Guevara <carlos [...] carlosguevara.com>
Download (untitled) / with headers
text/plain 2.9k
This is a bug report for perl from "Carlos Guevara" <carlos@carlosguevara.com>, generated with the help of perlbug 1.41 running under perl 5.28.0. ----------------------------------------------------------------- Solaris is failing some locale tests: http://perl5.test-smoke.org/report/65421 ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl 5.28.0: Configured by cpan at Thu Apr 26 22:26:15 CDT 2018. Summary of my perl5 (revision 5 version 28 subversion 0) configuration: Snapshot of: 5dbe8f0a915c25666dd9c760775f619c34a51538 Platform: osname=solaris osvers=2.11 archname=i86pc-solaris-64 uname='sunos cjg-hipster 5.11 illumos-094e47e980 i86pc i386 i86pc ' config_args='-des -Dprefix=~/bin/perl-blead -Dscriptdir=~/bin/perl-blead/bin -Dusedevel -Duse64bitall -Dcc=gcc' hint=recommended useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags ='-m64 -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/gnu/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -DPERL_USE_SAFE_PUTENV' optimize='-O' cppflags='-m64 -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/gnu/include' ccversion='' gccversion='6.4.0' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='gcc' ldflags =' -m64 -fstack-protector-strong -L/usr/gnu/lib ' libpth=/usr/gcc/6/lib /usr/lib /usr/gnu/lib /usr/ccs/lib libs=-lpthread -lsocket -lnsl -lgdbm -ldb -ldl -lm -lc perllibs=-lpthread -lsocket -lnsl -ldl -lm -lc libc=/lib/libc.so so=so useshrplib=true libperl=libperl.so gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags=' -R /home/cpan/bin/perl-blead/lib/5.28.0/i86pc-solaris-64/CORE' cccdlflags='-fPIC' lddlflags=' -shared -m64 -L/usr/gnu/lib -fstack-protector-strong' --- @INC for perl 5.28.0: /home/cpan/bin/perl-blead/lib/site_perl/5.28.0/i86pc-solaris-64 /home/cpan/bin/perl-blead/lib/site_perl/5.28.0 /home/cpan/bin/perl-blead/lib/5.28.0/i86pc-solaris-64 /home/cpan/bin/perl-blead/lib/5.28.0 --- Environment for perl 5.28.0: HOME=/home/cpan LANG=en_US LANGUAGE (unset) LC_ALL=C LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/cpan/bin/perl-blead/bin:/home/cpan/bin:/usr/bin:/usr/sbin:/sbin:/usr/gnu/bin PERL_BADLANG (unset) SHELL=/usr/bin/bash
To: perl5-porters [...] perl.org
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
Date: Fri, 27 Apr 2018 13:07:26 -0600
Download (untitled) / with headers
text/plain 4.2k
On 04/26/2018 10:07 PM, Carlos Guevara (via RT) wrote: Show quoted text
> # New Ticket Created by Carlos Guevara > # Please include the string: [perl #133157] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=133157 > > > > This is a bug report for perl from "Carlos Guevara" <carlos@carlosguevara.com>, > generated with the help of perlbug 1.41 running under perl 5.28.0. > > > ----------------------------------------------------------------- > Solaris is failing some locale tests: > http://perl5.test-smoke.org/report/65421
The problem is that these locales have a UTF-8 decimal radix character, and it appears that the OS doesn't properly handle this case. A very similar issue was present in cygwin until we reported it to them, and they have since fixed it. Attached is a short C program to verify that it's an OS problem. In the meantime, solaris smokes are failing. I've made this a 5.28 blocker. I have patches that skip or todo the failing tests. But I'll wait until Carlos runs the program. Also, this is openindiana solaris. I have no idea if Oracle solaris has this issue. The bug tracker is not open to the public, which I find astonishing and disconcerting. I did not find this issue in the openindiana list----------------------------------------------------------------- Show quoted text
> --- > Flags: > category=core > severity=low > --- > Site configuration information for perl 5.28.0: > > Configured by cpan at Thu Apr 26 22:26:15 CDT 2018. > > Summary of my perl5 (revision 5 version 28 subversion 0) configuration: > Snapshot of: 5dbe8f0a915c25666dd9c760775f619c34a51538 > Platform: > osname=solaris > osvers=2.11 > archname=i86pc-solaris-64 > uname='sunos cjg-hipster 5.11 illumos-094e47e980 i86pc i386 i86pc ' > config_args='-des -Dprefix=~/bin/perl-blead > -Dscriptdir=~/bin/perl-blead/bin -Dusedevel -Duse64bitall -Dcc=gcc' > hint=recommended > useposix=true > d_sigaction=define > useithreads=undef > usemultiplicity=undef > use64bitint=define > use64bitall=define > uselongdouble=undef > usemymalloc=n > default_inc_excludes_dot=define > bincompat5005=undef > Compiler: > cc='gcc' > ccflags ='-m64 -fwrapv -fno-strict-aliasing -pipe > -fstack-protector-strong -I/usr/gnu/include -D_LARGEFILE_SOURCE > -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -DPERL_USE_SAFE_PUTENV' > optimize='-O' > cppflags='-m64 -fwrapv -fno-strict-aliasing -pipe > -fstack-protector-strong -I/usr/gnu/include' > ccversion='' > gccversion='6.4.0' > gccosandvers='' > intsize=4 > longsize=8 > ptrsize=8 > doublesize=8 > byteorder=12345678 > doublekind=3 > d_longlong=define > longlongsize=8 > d_longdbl=define > longdblsize=16 > longdblkind=3 > ivtype='long' > ivsize=8 > nvtype='double' > nvsize=8 > Off_t='off_t' > lseeksize=8 > alignbytes=8 > prototype=define > Linker and Libraries: > ld='gcc' > ldflags =' -m64 -fstack-protector-strong -L/usr/gnu/lib ' > libpth=/usr/gcc/6/lib /usr/lib /usr/gnu/lib /usr/ccs/lib > libs=-lpthread -lsocket -lnsl -lgdbm -ldb -ldl -lm -lc > perllibs=-lpthread -lsocket -lnsl -ldl -lm -lc > libc=/lib/libc.so > so=so > useshrplib=true > libperl=libperl.so > gnulibc_version='' > Dynamic Linking: > dlsrc=dl_dlopen.xs > dlext=so > d_dlsymun=undef > ccdlflags=' -R /home/cpan/bin/perl-blead/lib/5.28.0/i86pc-solaris-64/CORE' > cccdlflags='-fPIC' > lddlflags=' -shared -m64 -L/usr/gnu/lib -fstack-protector-strong' > > > --- > @INC for perl 5.28.0: > /home/cpan/bin/perl-blead/lib/site_perl/5.28.0/i86pc-solaris-64 > /home/cpan/bin/perl-blead/lib/site_perl/5.28.0 > /home/cpan/bin/perl-blead/lib/5.28.0/i86pc-solaris-64 > /home/cpan/bin/perl-blead/lib/5.28.0 > > --- > Environment for perl 5.28.0: > HOME=/home/cpan > LANG=en_US > LANGUAGE (unset) > LC_ALL=C > LD_LIBRARY_PATH (unset) > LOGDIR (unset) > PATH=/home/cpan/bin/perl-blead/bin:/home/cpan/bin:/usr/bin:/usr/sbin:/sbin:/usr/gnu/bin > PERL_BADLANG (unset) > SHELL=/usr/bin/bash > >
Download radix.c
text/x-csrc 344b

Message body is not shown because sender requested not to inline it.

Date: Fri, 27 Apr 2018 23:56:59 -0400
To: perlbug-followup [...] perl.org
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
From: Carlos Guevara <carlos [...] carlosguevara.com>
Download (untitled) / with headers
text/plain 426b
Revised radix.c: ##### #include <stdio.h> #include <locale.h> int main(int argc, char ** argv) { unsigned char buf[100]; unsigned int i; printf("%s\n", setlocale(LC_ALL, "ar_AE.UTF-8")); snprintf(buf, sizeof(buf), "%g", 3.2); for (i = 0; i < sizeof(buf); i++) { if (buf[i] == '\0') break; printf(" %x", buf[i]); } printf("\n"); } ##### Output: ##### ar_AE.UTF-8 33 d9 32 #####
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
From: Karl Williamson <public [...] khwilliamson.com>
To: Carlos Guevara <carlos [...] carlosguevara.com>, perlbug-followup [...] perl.org
Date: Fri, 27 Apr 2018 22:24:08 -0600
Download (untitled) / with headers
text/plain 890b
On 04/27/2018 09:56 PM, Carlos Guevara wrote: Show quoted text
> Revised radix.c: > ##### > #include <stdio.h> > #include <locale.h> > > int > main(int argc, char ** argv) > { > unsigned char buf[100]; > unsigned int i; > > printf("%s\n", setlocale(LC_ALL, "ar_AE.UTF-8")); > snprintf(buf, sizeof(buf), "%g", 3.2); > > for (i = 0; i < sizeof(buf); i++) { > if (buf[i] == '\0') break; > printf(" %x", buf[i]); > } > printf("\n"); > } > ##### > > Output: > ##### > ar_AE.UTF-8 > 33 d9 32 > ##### >
That should instead have been 33 d9 ab 32. And that indicates that the problem is indeed with the OS. My guess is that it doesn't consider the possibility of a multi-byte radix character, so it uses just the first byte, but \xd9 is a start byte of a two byte sequence, so this is leading to malformed UTF-8. I'll submit a trouble ticket for them.
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
From: Karl Williamson <public [...] khwilliamson.com>
To: Carlos Guevara <carlos [...] carlosguevara.com>, perlbug-followup [...] perl.org
Date: Sat, 28 Apr 2018 10:02:59 -0600
On 04/27/2018 10:24 PM, Karl Williamson wrote: Show quoted text
> On 04/27/2018 09:56 PM, Carlos Guevara wrote:
>> Revised radix.c: >> ##### >> #include <stdio.h> >> #include <locale.h> >> >> int >> main(int argc, char ** argv) >> { >>      unsigned char buf[100]; >>      unsigned int i; >> >>      printf("%s\n", setlocale(LC_ALL, "ar_AE.UTF-8")); >>      snprintf(buf, sizeof(buf), "%g", 3.2); >> >>      for (i = 0; i < sizeof(buf); i++) { >>          if (buf[i] == '\0') break; >>          printf(" %x", buf[i]); >>      } >>      printf("\n"); >> } >> ##### >> >> Output: >> ##### >> ar_AE.UTF-8 >>   33 d9 32 >> ##### >>
> > That should instead have been 33 d9 ab 32. > And that indicates that the problem is indeed with the OS. > My guess is that it doesn't consider the possibility of a multi-byte > radix character, so it uses just the first byte, but \xd9 is a start > byte of a two byte sequence, so this is leading to malformed UTF-8. > > I'll submit a trouble ticket for them. >
Now done as https://www.illumos.org/issues/9511
To: Carlos Guevara <carlos [...] carlosguevara.com>, perlbug-followup [...] perl.org
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
Date: Sat, 28 Apr 2018 10:24:59 -0600
Download (untitled) / with headers
text/plain 1.7k
On 04/28/2018 10:02 AM, Karl Williamson wrote: Show quoted text
> On 04/27/2018 10:24 PM, Karl Williamson wrote:
>> On 04/27/2018 09:56 PM, Carlos Guevara wrote:
>>> Revised radix.c: >>> ##### >>> #include <stdio.h> >>> #include <locale.h> >>> >>> int >>> main(int argc, char ** argv) >>> { >>>      unsigned char buf[100]; >>>      unsigned int i; >>> >>>      printf("%s\n", setlocale(LC_ALL, "ar_AE.UTF-8")); >>>      snprintf(buf, sizeof(buf), "%g", 3.2); >>> >>>      for (i = 0; i < sizeof(buf); i++) { >>>          if (buf[i] == '\0') break; >>>          printf(" %x", buf[i]); >>>      } >>>      printf("\n"); >>> } >>> ##### >>> >>> Output: >>> ##### >>> ar_AE.UTF-8 >>>   33 d9 32 >>> ##### >>>
>> >> That should instead have been 33 d9 ab 32. >> And that indicates that the problem is indeed with the OS. >> My guess is that it doesn't consider the possibility of a multi-byte >> radix character, so it uses just the first byte, but \xd9 is a start >> byte of a two byte sequence, so this is leading to malformed UTF-8. >> >> I'll submit a trouble ticket for them. >>
> > Now done as https://www.illumos.org/issues/9511 >
Attached are three patches that cause these tests to pass on solaris. A version specification should probably be added to the one for t/run/locale.t. But there are complications that I don't know how to deal with. I don't know the version spec to use for openindiana which I understand has a different kind of release deal. And I don't know if this is a bug in the Oracle solaris, which has a very different version number. I think these patches, after the versioning is ironed out, should go in 5.28, so that this platform passes the test suite. These affect only two .t files.

Message body is not shown because sender requested not to inline it.

Message body is not shown because sender requested not to inline it.

Message body is not shown because sender requested not to inline it.

Date: Mon, 30 Apr 2018 23:50:45 +0300
To: Karl Williamson <public [...] khwilliamson.com>, Carlos Guevara <carlos [...] carlosguevara.com>, perlbug-followup [...] perl.org
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
From: Sawyer X <xsawyerx [...] gmail.com>
Download (untitled) / with headers
text/plain 1.9k
On 04/28/2018 07:24 PM, Karl Williamson wrote: Show quoted text
> On 04/28/2018 10:02 AM, Karl Williamson wrote:
>> On 04/27/2018 10:24 PM, Karl Williamson wrote:
>>> On 04/27/2018 09:56 PM, Carlos Guevara wrote:
>>>> Revised radix.c: >>>> ##### >>>> #include <stdio.h> >>>> #include <locale.h> >>>> >>>> int >>>> main(int argc, char ** argv) >>>> { >>>>      unsigned char buf[100]; >>>>      unsigned int i; >>>> >>>>      printf("%s\n", setlocale(LC_ALL, "ar_AE.UTF-8")); >>>>      snprintf(buf, sizeof(buf), "%g", 3.2); >>>> >>>>      for (i = 0; i < sizeof(buf); i++) { >>>>          if (buf[i] == '\0') break; >>>>          printf(" %x", buf[i]); >>>>      } >>>>      printf("\n"); >>>> } >>>> ##### >>>> >>>> Output: >>>> ##### >>>> ar_AE.UTF-8 >>>>   33 d9 32 >>>> ##### >>>>
>>> >>> That should instead have been 33 d9 ab 32. >>> And that indicates that the problem is indeed with the OS. >>> My guess is that it doesn't consider the possibility of a multi-byte >>> radix character, so it uses just the first byte, but \xd9 is a start >>> byte of a two byte sequence, so this is leading to malformed UTF-8. >>> >>> I'll submit a trouble ticket for them. >>>
>> >> Now done as https://www.illumos.org/issues/9511 >>
> > > Attached are three patches that cause these tests to pass on solaris.  > A version specification should probably be added to the one for > t/run/locale.t.  But there are complications that I don't know how to > deal with.  I don't know the version spec to use for openindiana which > I understand has a different kind of release deal.  And I don't know > if this is a bug in the Oracle solaris, which has a very different > version number. > > I think these patches, after the versioning is ironed out, should go > in 5.28, so that this platform passes the test suite.  These affect > only two .t files.
I'd like one of the committers to approve this before it is merged to blead. Dave, Tony, Yves, Zefram, etc.?
To: Sawyer X <xsawyerx [...] gmail.com>
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133157] Solaris Failing Some Locale Tests
CC: Karl Williamson <public [...] khwilliamson.com>, Carlos Guevara <carlos [...] carlosguevara.com>, perlbug-followup [...] perl.org
Date: Tue, 1 May 2018 09:22:08 +0100
Download (untitled) / with headers
text/plain 1.1k
On Mon, Apr 30, 2018 at 11:50:45PM +0300, Sawyer X wrote: Show quoted text
> > > On 04/28/2018 07:24 PM, Karl Williamson wrote:
> > Attached are three patches that cause these tests to pass on solaris.  > > A version specification should probably be added to the one for > > t/run/locale.t.  But there are complications that I don't know how to > > deal with.  I don't know the version spec to use for openindiana which > > I understand has a different kind of release deal.  And I don't know > > if this is a bug in the Oracle solaris, which has a very different > > version number. > > > > I think these patches, after the versioning is ironed out, should go > > in 5.28, so that this platform passes the test suite.  These affect > > only two .t files.
> > I'd like one of the committers to approve this before it is merged to > blead. Dave, Tony, Yves, Zefram, etc.?
I approve, and and just merged them, as v5.27.11-26-ge3e8c0d65c v5.27.11-27-ga6bc52d6f4 v5.27.11-28-gb974d2c0b3 As regards the specifics of openindiana id and versions, that can always be added later if we obtain that info. -- Modern art: "That's easy, I could have done that!" "Ah, but you didn't!"


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org