Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib/locale.t: Test failures in threaded builds on FreeBSD 10.3 #15500

Closed
p5pRT opened this issue Aug 7, 2016 · 17 comments
Closed

lib/locale.t: Test failures in threaded builds on FreeBSD 10.3 #15500

p5pRT opened this issue Aug 7, 2016 · 17 comments

Comments

@p5pRT
Copy link

p5pRT commented Aug 7, 2016

Migrated from rt.perl.org#128867 (status was 'resolved')

Searchable as RT128867$

@p5pRT
Copy link
Author

p5pRT commented Aug 7, 2016

From @jkeenan

Yesterday, for the first time I was able to smoke test Perl 5 blead on FreeBSD 10.3. This OS is a virtual machine hosted on a Linux x86_64 machine. All tests PASSed on non-threaded builds, but there were failures in lib/locale.t on threaded builds. The relevant smoke test report may be found at​: http​://perl5.test-smoke.org/report/48882.

Note that the "FAILED 439" in that report apparently means that the first failure occurred at test 439 -- not that there were 439 failures overall. There were 7 failures in each of 4 different variations of threaded builds.

Per previous coaching from Karl Williamson, I re-ran the failing test file with PERL_DEBUG_FULL_TEST=2. The output, as expected, is enormous, so I'll only send that to khw and to anyone else who requests it. I'm attaching a reduced version of that output as well as the 'perl -V' output for that build.

Thank you very much.
Jim Keenan
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Aug 7, 2016

From @jkeenan

$ cd t;./perl harness -v ../lib/locale.t; cd -

# lower=FF; uc=178; ; fc=FF; FF=/178/i=1; 178=/FF/i=1
not ok 381 Verify that /[[​:upper​:]]/ matches all alpha X for which uc(X) == X and lc(X) != X
not ok 382 Verify that /[[​:lower​:]]/i matches all alpha X for which uc(X) == X and lc(X) != X
ok 383 Verify that /[[​:lower​:]]/ matches all alpha X for which lc(X) == X and uc(X) != X
--
ok 389 Verify that [​:lower​:] contains at least a-z
not ok 390 Verify that [​:lower​:] is a subset of [​:alpha​:]
ok 391 Verify that [​:upper​:] contains at least A-Z
ok 392 Verify that [​:upper​:] is a subset of [​:alpha​:]
not ok 393 Verify that /[[​:lower​:]]/i is a subset of [​:alpha​:]
ok 394 Verify that [​:alpha​:] is a subset of [​:alnum​:]
--
ok 438 Verify that non-ASCII UTF-8 error messages are in UTF-8
not ok 439 ... and are ASCII outside "use locale"
ok 440 Verify that a number with a UTF-8 radix has a UTF-8 stringification
--
ok 445 Verify that strftime returns date with UTF-8 flag appropriately set
not ok 446 "$!" is ASCII only outside of locale scope
ok 447 Verify "lc(foo) cmp lc(bar)" is the same as using intermediaries for the cmp
not ok 448 Verify case insensitive matching works
ok 449 Verify atof with locale radix and negative exponent

Failed 7/682 subtests

Test Summary Report


../lib/locale.t (Wstat​: 0 Tests​: 682 Failed​: 7)
  Failed tests​: 381-382, 390, 393, 439, 446, 448
Files=1, Tests=682, 18 wallclock secs ( 0.56 usr 0.80 sys + 4.64 cusr 1.27 csys = 7.27 CPU)
Result​: FAIL
/usr/home/jkeenan/p5smoke/perl-current

@p5pRT
Copy link
Author

p5pRT commented Aug 7, 2016

From @jkeenan

Summary of my perl5 (revision 5 version 25 subversion 4) configuration​:
  Snapshot of​: 6a29646
  Platform​:
  osname=freebsd
  osvers=10.3-release
  archname=amd64-freebsd-thread-multi
  uname='freebsd localhost 10.3-release freebsd 10.3-release #0 r297264​: fri mar 25 02​:10​:02 utc 2016 root@​releng1.nyi.freebsd.org​:usrobjusrsrcsysgeneric amd64 '
  config_args='-des -Dusedevel -Duseithreads -DDEBUGGING'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=define
  usemultiplicity=define
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  bincompat5005=undef
  Compiler​:
  cc='cc'
  ccflags ='-DDEBUGGING -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_FORTIFY_SOURCE=2'
  optimize='-O -g'
  cppflags='-DDEBUGGING -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion=''
  gccversion='4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='cc'
  ldflags ='-pthread -Wl,-E -fstack-protector -L/usr/local/lib'
  libpth=/usr/lib /usr/local/lib /usr/include/clang/3.4.1 /usr/lib
  libs=-lpthread -lgdbm -lm -lcrypt -lutil
  perllibs=-lpthread -lm -lcrypt -lutil
  libc=
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags=' '
  cccdlflags='-DPIC -fPIC'
  lddlflags='-shared -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  DEBUGGING
  HAS_TIMES
  MULTIPLICITY
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
  PERL_IMPLICIT_CONTEXT
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_TRACK_MEMPOOL
  PERL_USE_DEVEL
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_ITHREADS
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  USE_REENTRANT_API
  Locally applied patches​:
  SMOKE6a29646efd2b54a6feec66ee83d2852418bfb748
  Built under freebsd
  Compiled at Aug 6 2016 22​:51​:52
  %ENV​:
  PERL_WORKDIR="/home/jkeenan/gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.25.4/amd64-freebsd-thread-multi
  /usr/local/lib/perl5/site_perl/5.25.4
  /usr/local/lib/perl5/5.25.4/amd64-freebsd-thread-multi
  /usr/local/lib/perl5/5.25.4
  .

@p5pRT
Copy link
Author

p5pRT commented Aug 10, 2016

From @khwilliamson

A bug report has been file against freebsd
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211743
It includes the attached file that demonstrates a problem in freebsd without involving perl
--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Aug 10, 2016

From @khwilliamson

#include <stdio.h>
#include <locale.h>
#include <string.h>

int
main(int argc, char **argv)
{

    int i;
    locale_t C_locale_object = newlocale(LC_ALL_MASK, "C", NULL);

    setlocale(LC_MESSAGES, "be_BY.UTF-8");

    uselocale(C_locale_object);

    for (i=1; i < 10; i++) {
        /* This should print using the C locale, hence in ASCII */
        printf("%s\n", strerror(i));
    }
}

@p5pRT
Copy link
Author

p5pRT commented Aug 10, 2016

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @cpansprout

On Wed Aug 10 16​:57​:45 2016, khw wrote​:

A bug report has been file against freebsd
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211743
It includes the attached file that demonstrates a problem in freebsd
without involving perl

In what way does the attached program misbehave (for the sake of future readers)?

Is there any way to work around the issue in perl, so that it passes tests (without skipping them) on current FreeBSDs?

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @cpansprout

On Wed Aug 10 19​:52​:33 2016, sprout wrote​:

On Wed Aug 10 16​:57​:45 2016, khw wrote​:

A bug report has been file against freebsd
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211743
It includes the attached file that demonstrates a problem in freebsd
without involving perl

In what way does the attached program misbehave (for the sake of
future readers)?

Is there any way to work around the issue in perl, so that it passes
tests (without skipping them) on current FreeBSDs?

I see you have committed 7536d89.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @jkeenan

On Wed Aug 10 19​:52​:33 2016, sprout wrote​:

On Wed Aug 10 16​:57​:45 2016, khw wrote​:

A bug report has been file against freebsd
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211743
It includes the attached file that demonstrates a problem in freebsd
without involving perl

In what way does the attached program misbehave (for the sake of
future readers)?

1. Let me, as the original poster, take a stab at answering this. (khw can offer more details.) Disclaimer​: I am not a C expert; I'm working from an initial reading of the relevant man pages (both Linux and FreeBSD).

a. The program creates a new locale object in the "C" locale.

b. The program then requests that localizable natural-language messages be rendered in Byelorussian UTF-8.

c. The program then sets the current locale for the calling thread to that in the locale object created in 'a' above. Presumably this means that error messages will be rendered in the "C" locale's way, not in Byelorussian.

d. We see what actually happens by looping through 9 error messages.

On Linux (Ubuntu 16.04 LTS), running the compiled program produces​:

#####
Operation not permitted
No such file or directory
No such process
Interrupted system call
Input/output error
No such device or address
Argument list too long
Exec format error
Bad file descriptor
#####

On a Vagrant box containing FreeBSD 10.3, hosted on that Linux machine, I get​:

#####
Аперацыя не дазволена
Няма такога файлу ці каталогу
Няма такога працэсу
Перарваны сістэмны выклік
Памылка ўводу/вываду
Прыстасаванне не сканфігуравана
Надта доўгі спіс аргументаў
Памылка фармату файлу, які выконваецца
Некарэктны дэскрыптар файлу
#####

On the same version of FreeBSD 10.3, run inside a VirtualBox 5.1 VM *not* accessed by Vagrant, I get gobbledygook which I'll simply post here​:
http​://thenceforward.net/perl/misc/non-vagrant-freebsd-vm-2016-08-10_17-17-17.png

So it appears that the 'uselocale' call on FreeBSD is not DWIMming. Until we can get clarification from FreeBSD folks about this, we'll use this hint provided by khw​:

#####
+# As of 10.3-RELEASE FreeBSD. See [perl #128867]
+d_uselocale='undef'
#####

I should add that, because we were not getting smoke tests about FreeBSD until I started providing them this month, and because the only VMs I have of FreeBSD are 10.3, we can't exactly say when this problem first appeared.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @khwilliamson

On 08/11/2016 07​:28 AM, James E Keenan via RT wrote​:

On Wed Aug 10 19​:52​:33 2016, sprout wrote​:

On Wed Aug 10 16​:57​:45 2016, khw wrote​:

A bug report has been file against freebsd
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211743
It includes the attached file that demonstrates a problem in freebsd
without involving perl

In what way does the attached program misbehave (for the sake of
future readers)?

1. Let me, as the original poster, take a stab at answering this. (khw can offer more details.) Disclaimer​: I am not a C expert; I'm working from an initial reading of the relevant man pages (both Linux and FreeBSD).

a. The program creates a new locale object in the "C" locale.

b. The program then requests that localizable natural-language messages be rendered in Byelorussian UTF-8.

c. The program then sets the current locale for the calling thread to that in the locale object created in 'a' above. Presumably this means that error messages will be rendered in the "C" locale's way, not in Byelorussian.

d. We see what actually happens by looping through 9 error messages.

On Linux (Ubuntu 16.04 LTS), running the compiled program produces​:

#####
Operation not permitted
No such file or directory
No such process
Interrupted system call
Input/output error
No such device or address
Argument list too long
Exec format error
Bad file descriptor
#####

On a Vagrant box containing FreeBSD 10.3, hosted on that Linux machine, I get​:

#####
Аперацыя не дазволена
Няма такога файлу ці каталогу
Няма такога працэсу
Перарваны сістэмны выклік
Памылка ўводу/вываду
Прыстасаванне не сканфігуравана
Надта доўгі спіс аргументаў
Памылка фармату файлу, які выконваецца
Некарэктны дэскрыптар файлу
#####

On the same version of FreeBSD 10.3, run inside a VirtualBox 5.1 VM *not* accessed by Vagrant, I get gobbledygook which I'll simply post here​:
http​://thenceforward.net/perl/misc/non-vagrant-freebsd-vm-2016-08-10_17-17-17.png

So it appears that the 'uselocale' call on FreeBSD is not DWIMming. Until we can get clarification from FreeBSD folks about this, we'll use this hint provided by khw​:

#####
+# As of 10.3-RELEASE FreeBSD. See [perl #128867]
+d_uselocale='undef'
#####

I should add that, because we were not getting smoke tests about FreeBSD until I started providing them this month, and because the only VMs I have of FreeBSD are 10.3, we can't exactly say when this problem first appeared.

Thank you very much.

A complete description of the problem and the program are at the link I
posted. But basically, it's what Jim said.

If it helps, you can kind of think of it this way​:

A locale walks into a bar and orders some C, straight up.
Later, for variety, he asks for a little Belorussian.
He didn't like that so orders C again.
"Blecch. You gave me Belorussian, and I asked for C"

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @mauke

Am 11.08.2016 um 15​:28 schrieb James E Keenan via RT​:

On a Vagrant box containing FreeBSD 10.3, hosted on that Linux machine, I get​:

#####
Аперацыя не дазволена
Няма такога файлу ці каталогу
Няма такога працэсу
Перарваны сістэмны выклік
Памылка ўводу/вываду
Прыстасаванне не сканфігуравана
Надта доўгі спіс аргументаў
Памылка фармату файлу, які выконваецца
Некарэктны дэскрыптар файлу
#####

On the same version of FreeBSD 10.3, run inside a VirtualBox 5.1 VM *not* accessed by Vagrant, I get gobbledygook which I'll simply post here​:
http​://thenceforward.net/perl/misc/non-vagrant-freebsd-vm-2016-08-10_17-17-17.png

That gobbledygook is the same Byelorussian strings you showed above,
encoded as UTF-8, then reinterpreted as CP-437
<https://en.wikipedia.org/wiki/Code_page_437>.

I can reproduce it using
perl -CS -wE 'use utf8; use Encode; say decode "CP-437", encode "UTF-8",
"Аперацыя не дазволена"'

--
Lukas Mai <plokinom@​gmail.com>

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @khwilliamson

FreeBSD now passes, based on the workaround I added in 7536d89.
As stated earlier in the commentary on this ticket, the problem is in FreeBSD, and a ticket has been filed. When they fix it, we can modify the hints file to not do the workaround for fixed releases.
--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

@khwilliamson - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Aug 11, 2016
@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @jkeenan

On 08/11/2016 01​:20 PM, Karl Williamson via RT wrote​:

FreeBSD now passes, based on the workaround I added in 7536d89.
As stated earlier in the commentary on this ticket, the problem is in FreeBSD, and a ticket has been filed. When they fix it, we can modify the hints file to not do the workaround for fixed releases.

Here is the smoke report supporting the above​:
http​://perl5.test-smoke.org/report/48986

@p5pRT
Copy link
Author

p5pRT commented Aug 11, 2016

From @khwilliamson

On 08/11/2016 11​:43 AM, James E Keenan wrote​:

On 08/11/2016 01​:20 PM, Karl Williamson via RT wrote​:

FreeBSD now passes, based on the workaround I added in
7536d89.
As stated earlier in the commentary on this ticket, the problem is in
FreeBSD, and a ticket has been filed. When they fix it, we can modify
the hints file to not do the workaround for fixed releases.

Here is the smoke report supporting the above​:
http​://perl5.test-smoke.org/report/48986

I had already seen it, which is why I closed the ticket. I look at
http​://perl.develop-help.com/?b=blead

@p5pRT
Copy link
Author

p5pRT commented Nov 29, 2017

From @khwilliamson

This was fixed in freebsd :

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211743

It also turned up in dragonfly, and was fixed

https://gitweb.dragonflybsd.org/dragonfly.git/commit/21a0e97de9138e49c0f26ecb49e6cbba2478f410

Attached is a test file that reproduces the problem, so we don't have to write a new one for any other buggy systems out there

@p5pRT
Copy link
Author

p5pRT commented Nov 29, 2017

From @khwilliamson

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <locale.h>

int main (int argc, char **argv) {

    if (argc != 2) {
            fprintf(stderr, "Usage: $0 errno-value\n");
            exit(2);
    }

    printf("%s: %d: Setting global locale to %s\n", __FILE__, __LINE__, setlocale(LC_ALL, "C"));

    locale_t PL_C_locale_obj = newlocale(LC_ALL_MASK, "C", (locale_t) NULL);
    printf("%s: %d: Created a locale object for the C locale: %p\n", __FILE__, __LINE__, PL_C_locale_obj);

    printf("%s: %d: Setting global locale to %s\n", __FILE__, __LINE__, setlocale(LC_ALL, "be_BY.UTF-8"));

    printf("%s: %d: This should print in Belarussion: %s\n", __FILE__, __LINE__, strerror(atoi(argv[1])));

    printf("%s: %d: Changing thread locale to C, previous one should be -1 for global: %p\n", __FILE__, __LINE__, uselocale(PL_C_locale_obj));

    printf("%s: %d: Confirming that the thread locale was changed: %p\n", __FILE__, __LINE__, uselocale((locale_t) 0));

    printf("%s: %d: This should print in ASCII: %s\n", __FILE__, __LINE__, strerror(atoi(argv[1])));
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant