Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

locale.t on Win32 failing "Verify that strings with embedded NUL collate" and "Verify that strings with embedded NUL and extra trailing NUL collate" #15444

Closed
p5pRT opened this issue Jul 15, 2016 · 26 comments

Comments

@p5pRT
Copy link

p5pRT commented Jul 15, 2016

Migrated from rt.perl.org#128629 (status was 'resolved')

Searchable as RT128629$

@p5pRT
Copy link
Author

p5pRT commented Jul 15, 2016

From @bulk88

Created by @bulk88

------------------------------------------------------
ok 410 Verify that isn't both [​:cntrl​:] and [​:print​:]
ok 411 Verify that isn't both [​:alpha​:] and [​:digit​:]
ok 412 Verify that isn't both [​:alnum​:] and [​:punct​:]
ok 413 Verify that isn't both [​:xdigit​:] and [​:punct​:]
ok 414 Verify that isn't both [​:graph​:] and [​:space​:]
ok 415 Verify that cmp works with a read-only scalar; no- vs locale
ok 416 Verify that "le", "ne", etc work
ok 417 Skip in locales where \001 has primary sorting weight; otherwise
verify that \0 doesn't have primary sorting weight
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 418'
not ok 418 Verify that strings with embedded NUL collate
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 419'
not ok 419 Verify that strings with embedded NUL and extra trailing NUL
collate
ok 420 Verify that empty strings collate
ok 421 Skip in non-UTF-8 locales; otherwise verify that UTF8ness doesn't
matter with collation
ok 422 Skip in UTF-8 locales; otherwise verify that single byte collates
before 0x100 and above
ok 423 Skip in UTF-8 locales; otherwise verify that 0x100 and above
collate in code point order
ok 424 Verify that an intervening printf doesn't change assignment results
ok 425 Verify that an intervening sprintf doesn't change assignment results
ok 426 Verify that a different locale radix works when doing "==" with a
constant
ok 427 Verify that a different locale radix works whe
---------------------------------------------------------------
ok 659 verify that is tainted
ok 660 fc("Ā") in C locale (use locale; encoded in utf8) should be "ā",
got "ā"
ok 661 verify that is tainted
ok 662 fc("") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "", got ""
ok 663 verify that isn't tainted
ok 664 fc("") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "", got ""
ok 665 verify that isn't tainted
ok 666 fc("A") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "a", got "a"
ok 667 verify that isn't tainted
ok 668 fc("A") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "a", got "a"
ok 669 verify that isn't tainted
ok 670 fc("À") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "à", got "à"
ok 671 verify that isn't tainted
ok 672 fc("À") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "à", got "à"
ok 673 verify that isn't tainted
ok 674 fc("ſ") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "s", got "s"
ok 675 verify that isn't tainted
ok 676 fc("ſ") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "s", got "s"
ok 677 verify that isn't tainted
ok 678 fc("Ā") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "ā", got "ā"
ok 679 verify that isn't tainted
ok 680 fc("Ā") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "ā", got "ā"
ok 681 verify that isn't tainted
#
# The locale definitions
#
# Danish Norwegian da no no\@​nynorsk
#
# on your system may have errors because the locale test 418
# "Verify that strings with embedded NUL collate"
# failed in those locales.
#
# If your users are not using these locales you are safe for the moment,
# but please report this failure first to perlbug@​perl.com using the
# perlbug script (as described in the INSTALL file) so that the exact
# details of the failures can be sorted out first and then your operating
# system supplier can be alerted about these anomalies.
#
#
# The locale definitions
#
# Danish Norwegian da no no\@​nynorsk
#
# on your system may have errors because the locale test 419
# "Verify that strings with embedded NUL and extra trailing NUL collate"
# failed in those locales.
#
# If your users are not using these locales you are safe for the moment,
# but please report this failure first to perlbug@​perl.com using the
# perlbug script (as described in the INSTALL file) so that the exact
# details of the failures can be sorted out first and then your operating
# system supplier can be alerted about these anomalies.
#
# The following locales
#
# Afrikaans Albanian American Basque Bulgarian C Catalan
# Català Croatian Czech Dutch English Estonian Faroese Finnish
# French Føroyskt Galician German Greek Gáidhlig Hungarian
# Icelandic Indonesian Italian Latvian Lithuanian Polish
# Portuguese Português Russian Serbian Slovak Slovenian
# Spanish Swedish Sámi Turkish Welsh af ca en es fi fr fy ga
# hu it la mo po ro ru sl
#
# tested okay.
#
# The following locales
#
# Danish Norwegian da no no\@​nynorsk
#
# had problems.
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
1..681
Failed 2/681 subtests

Test Summary Report
-------------------
../lib/locale.t (Wstat​: 0 Tests​: 681 Failed​: 2)
  Failed tests​: 418-419
Files=1, Tests=681, 9 wallclock secs ( 0.13 usr + 0.00 sys = 0.13 CPU)
Result​: FAIL
--------------------------------------------------------

PERL_DEBUG_FULL_TEST run attached.

Perl Info

Flags:
     category=library
     severity=medium
     module=locale

Site configuration information for perl 5.25.3:

Configured by Owner at Wed Jul 13 21:11:05 2016.

Summary of my perl5 (revision 5 version 25 subversion 3) configuration:

   Platform:
     osname=MSWin32
     osvers=5.1
     archname=MSWin32-x86-multi-thread
     uname=''
     config_args='undef'
     hint=recommended
     useposix=true
     d_sigaction=undef
     useithreads=define
     usemultiplicity=define
     use64bitint=undef
     use64bitall=undef
     uselongdouble=undef
     usemymalloc=n
     bincompat5005=undef
   Compiler:
     cc='cl'
     ccflags ='-nologo -GF -W3 -Od -MD -Zi -DDEBUGGING -DWIN32 -D_CONSOLE
-DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DWIN32_NO_REGISTRY -DUSE_PERLIO'
     optimize='-Od -MD -Zi -DDEBUGGING'
     cppflags='-DWIN32'
     ccversion='13.10.6030'
     gccversion=''
     gccosandvers=''
     intsize=4
     longsize=4
     ptrsize=4
     doublesize=8
     byteorder=1234
     doublekind=3
     d_longlong=undef
     longlongsize=8
     d_longdbl=define
     longdblsize=8
     longdblkind=0
     ivtype='long'
     ivsize=4
     nvtype='double'
     nvsize=8
     Off_t='__int64'
     lseeksize=8
     alignbytes=8
     prototype=define
   Linker and Libraries:
     ld='link'
     ldflags ='-nologo -nodefaultlib -debug -libpath:"c:\perl\lib\CORE"
-machine:x86'
     libpth=\lib
     libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
     perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
     libc=msvcrt.lib
     so=dll
     useshrplib=true
     libperl=perl525.lib
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_win32.xs
     dlext=dll
     d_dlsymun=undef
     ccdlflags=' '
     cccdlflags=' '
     lddlflags='-dll -nologo -nodefaultlib -debug
-libpath:"c:\perl\lib\CORE" -machine:x86'



@INC for perl 5.25.3:
     lib
     C:/perl521/src/lib
     .


Environment for perl 5.25.3:
     HOME (unset)
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=C:\sperl\c\bin;C:\WINDOWS\system32;C:\Program Files\Microsoft
Visual Studio .NET 2003\Vc7\bin;C:\Program Files\Microsoft Visual Studio
.NET 2003\Common7\IDE;C:\WINDOWS;C:\Program Files\Git\cmd;C:\Program
Files\Microsoft Visual Studio .NET 2003\Common7\Tools\bin;C:\perl\bin
     PERL_BADLANG (unset)
     PERL_JSON_BACKEND=Cpanel::JSON::XS
     PERL_YAML_BACKEND=YAML
     SHELL (unset)



@p5pRT
Copy link
Author

p5pRT commented Jul 15, 2016

From @bulk88

locale_t_full.txt

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2016

From @khwilliamson

On 07/15/2016 04​:18 PM, bulk88 (via RT) wrote​:

# New Ticket Created by bulk88
# Please include the string​: [perl #128629]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128629 >

This is a bug report for perl from bulk88@​hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.25.3.

-----------------------------------------------------------------
[Please describe your issue here]

------------------------------------------------------
ok 410 Verify that isn't both [​:cntrl​:] and [​:print​:]
ok 411 Verify that isn't both [​:alpha​:] and [​:digit​:]
ok 412 Verify that isn't both [​:alnum​:] and [​:punct​:]
ok 413 Verify that isn't both [​:xdigit​:] and [​:punct​:]
ok 414 Verify that isn't both [​:graph​:] and [​:space​:]
ok 415 Verify that cmp works with a read-only scalar; no- vs locale
ok 416 Verify that "le", "ne", etc work
ok 417 Skip in locales where \001 has primary sorting weight; otherwise
verify that \0 doesn't have primary sorting weight
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 418'
not ok 418 Verify that strings with embedded NUL collate
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 419'
not ok 419 Verify that strings with embedded NUL and extra trailing NUL
collate
ok 420 Verify that empty strings collate

620f73f may have fixed this. Please
let me know.

@p5pRT
Copy link
Author

p5pRT commented Jul 19, 2016

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jul 25, 2016

From @bulk88

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 25, 2016

From @bulk88

[log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt](https://rt-archive.perl.org/perl5/Ticket/Attachment/1412499/763073/log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt)

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @khwilliamson

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @bulk88

On Mon Jul 25 17​:06​:42 2016, public@​khwilliamson.com wrote​:

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

Recompiled perl with debugging and did -DLv see attached.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @bulk88

[log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt](https://rt-archive.perl.org/perl5/Ticket/Attachment/1412529/763098/log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt)

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @bulk88

On Mon Jul 25 18​:01​:39 2016, bulk88 wrote​:

On Mon Jul 25 17​:06​:42 2016, public@​khwilliamson.com wrote​:

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

Recompiled perl with debugging and did -DLv see attached.

7.8 MB of text in my attachment, yeah, that post was rejected from the ML...

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @khwilliamson

On 07/25/2016 07​:22 PM, bulk88 via RT wrote​:

On Mon Jul 25 18​:01​:39 2016, bulk88 wrote​:

On Mon Jul 25 17​:06​:42 2016, public@​khwilliamson.com wrote​:

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

Recompiled perl with debugging and did -DLv see attached.

7.8 MB of text in my attachment, yeah, that post was rejected from the ML...

Thanks. This showed up two problems. One is that there was unintended
recalculations going on if something failed. That just slows things
down and isn't relevant to the bug at hand.

The other is that your platform's strxfrm() is returning a length that
includes a trailing NUL, and the code is not expecting that. The win32
VM on dromedary doesn't do that. So I've added code to check for and
compensate for platforms like yours. This is smoking at
http​://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/khw-locale

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that the
output will be much smaller.

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @khwilliamson

locale.t

@p5pRT
Copy link
Author

p5pRT commented Jul 26, 2016

From @bulk88

On Tue Jul 26 15​:32​:16 2016, public@​khwilliamson.com wrote​:

The other is that your platform's strxfrm() is returning a length that
includes a trailing NUL, and the code is not expecting that. The
win32
VM on dromedary doesn't do that. So I've added code to check for and
compensate for platforms like yours. This is smoking at
http​://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/khw-
locale

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that
the
output will be much smaller.

I'll build it later, for right now, i googled up this https://github.com/apache/stdcxx/blob/4.1.3/src/collate.cpp#L477 but I can't find the ticket associated with "29935" for that project. MSVC 2003 calls itself 1310, so based on "#if defined _MSC_VER && _MSC_VER < 1400" the bug was fixed in VC 2005 and newer.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 27, 2016

From @bulk88

On Tue Jul 26 15​:32​:16 2016, public@​khwilliamson.com wrote​:

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that
the
output will be much smaller.

Your khw-locale branch fixes the test fail.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 27, 2016

From @khwilliamson

On Tue Jul 26 19​:41​:54 2016, bulk88 wrote​:

On Tue Jul 26 15​:32​:16 2016, public@​khwilliamson.com wrote​:

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that
the
output will be much smaller.

Your khw-locale branch fixes the test fail.

Ok, but I'll need to look at the log, now that the other issues (should) have been cleaned up
--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Jul 27, 2016

From @bulk88

On Tue Jul 26 21​:37​:02 2016, khw wrote​:

Ok, but I'll need to look at the log, now that the other issues
(should) have been cleaned up

The one already in this ticket or do I have to make a new one?

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 28, 2016

From @khwilliamson

On 07/27/2016 02​:42 PM, bulk88 via RT wrote​:

On Tue Jul 26 21​:37​:02 2016, khw wrote​:

Ok, but I'll need to look at the log, now that the other issues
(should) have been cleaned up

The one already in this ticket or do I have to make a new one?

Make a new one, and to shorten its length, you can use the modified
locale.t that was attached a couple of messages ago

@p5pRT
Copy link
Author

p5pRT commented Aug 3, 2016

From @khwilliamson

Blead has several changes in it that have fixed similar failures on other platforms. Let me know if that fixes your example. If not please rerun the tests

cd t
PERL_DEBUG_FULL_TEST=2 myperl -DLv -T ../lib/locale.t > khw.log 2>&1

and attach the output to this ticket.

--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Aug 3, 2016

From @cpansprout

On Tue Aug 02 18​:46​:18 2016, khw wrote​:

Blead has several changes in it that have fixed similar failures on
other platforms. Let me know if that fixes your example. If not
please rerun the tests

cd t
PERL_DEBUG_FULL_TEST=2 myperl -DLv -T ../lib/locale.t > khw.log 2>&1

Surely you mean ../perl, not myperl? (Or just ./perl after ‘make test’.)

and attach the output to this ticket.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Aug 3, 2016

From @khwilliamson

On 08/02/2016 08​:56 PM, Father Chrysostomos via RT wrote​:

On Tue Aug 02 18​:46​:18 2016, khw wrote​:

Blead has several changes in it that have fixed similar failures on
other platforms. Let me know if that fixes your example. If not
please rerun the tests

cd t
PERL_DEBUG_FULL_TEST=2 myperl -DLv -T ../lib/locale.t > khw.log 2>&1

Surely you mean ../perl, not myperl? (Or just ./perl after ‘make test’.)

Darn. This is the 3rd time in 2 days I made this mistake.

s!myperl!./perl -I../lib!

and attach the output to this ticket.

@p5pRT
Copy link
Author

p5pRT commented Aug 4, 2016

From @bulk88

On Tue Aug 02 18​:46​:18 2016, khw wrote​:

Blead has several changes in it that have fixed similar failures on
other platforms. Let me know if that fixes your example. If not
please rerun the tests

cd t
PERL_DEBUG_FULL_TEST=2 myperl -DLv -T ../lib/locale.t > khw.log 2>&1

and attach the output to this ticket.

Blead at


Revision​: c7202de
Author​: Karl Williamson <khw@​cpan.org>
Date​: 7/31/2016 5​:34​:47 AM
Message​:
locale.c​: Add some DEBUG statements

This also involves moving some complicated debugging statements to a
separate function so that it can be called from more than one place


passes for me in locale.t

Attaching the file you wanted.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Aug 4, 2016

From @bulk88

khw.log

@p5pRT
Copy link
Author

p5pRT commented Sep 22, 2016

From @khwilliamson

The OP says this is now fixed
--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Sep 22, 2016

@khwilliamson - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

p5pRT commented May 30, 2017

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT p5pRT closed this as completed May 30, 2017
@p5pRT
Copy link
Author

p5pRT commented May 30, 2017

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant