Skip Menu |
Report information
Id: 131646
Status: pending release
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: randir <sergey.aleynikov [at] gmail.com>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: medium
Type: core
Perl Version: 5.27.1
Fixed In: (no value)

Attachments


Date: Sat, 24 Jun 2017 16:01:17 +0300
From: Sergey Aleynikov <sergey.aleynikov [...] gmail.com>
Subject: utf8.c:832: S_unexpected_non_continuation_text: Assertion `expect_len == UTF8SKIP(s)' failed.
To: perlbug [...] perl.org
Download (untitled) / with headers
text/plain 5.7k
This is a bug report for perl from sergey.aleynikov@gmail.com, generated with the help of perlbug 1.40 running under perl 5.27.1. ----------------------------------------------------------------- [Please describe your issue here] While fuzzing perl v5.27.1-37-g4c95ee9f29 built with afl and run under libdislocator, I found the following program for(uc 0..t){0~~pack"UXp>",exp} to cause an assertion failure. This is a regression in 5.26, bisect points to: commit 7cf8d05d1e856f3bd3a392b3ccea008f1c1eb743 Author: Karl Williamson <khw@cpan.org> Date: Wed Sep 28 15:05:17 2016 -0600 Add details to UTF-8 malformation error messages GDB info about the crash location is: (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00007ffff6cf63fa in __GI_abort () at abort.c:89 #2 0x00007ffff6cede37 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x5555559a83ad "expect_len == UTF8SKIP(s)", file=file@entry=0x5555559a7b65 "utf8.c", line=line@entry=832, function=function@entry=0x5555559b2280 <__PRETTY_FUNCTION__.15262> "S_unexpected_non_continuation_text") at assert.c:92 #3 0x00007ffff6cedee2 in __GI___assert_fail (assertion=0x5555559a83ad "expect_len == UTF8SKIP(s)", file=0x5555559a7b65 "utf8.c", line=832, function=0x5555559b2280 <__PRETTY_FUNCTION__.15262> "S_unexpected_non_continuation_text") at assert.c:101 #4 0x00005555558857da in S_unexpected_non_continuation_text (s=0x555555c062d1 "", print_len=2, non_cont_byte_pos=1, expect_len=2) at utf8.c:832 #5 0x00005555558871db in Perl_bytes_cmp_utf8 (b=0x555555c05260 "0", blen=1, u=0x555555c062d2 "", ulen=9) at utf8.c:1876 #6 0x0000555555780e0a in Perl_sv_eq_flags (sv1=0x555555c0b3b0, sv2=0x555555c0b428, flags=0) at sv.c:7887 #7 0x00005555557b63da in Perl_pp_seq () at pp.c:2383 #8 0x000055555580a669 in S_do_smartmatch (seen_this=0x0, seen_other=0x0, copied=false) at pp_ctl.c:5152 #9 0x0000555555803a27 in Perl_pp_smartmatch () at pp_ctl.c:4670 #10 0x00005555556d5a7d in Perl_runops_debug () at dump.c:2451 #11 0x00005555555cbb3d in S_run_body (oldscope=1) at perl.c:2548 #12 0x00005555555cb0bb in perl_run (my_perl=0x555555bed010) at perl.c:2471 #13 0x0000555555583f3e in main (argc=2, argv=0x7fffffffe1e8, env=0x7fffffffe200) at perlmain.c:123 (gdb) p expect_len $1 = 2 (gdb) p s $2 = (const U8 * const) 0x555555c062d1 "" (gdb) p UTF8SKIP(s) $3 = 1 '\001' [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=medium --- Site configuration information for perl 5.27.1: Configured by root at Sun May 28 01:44:41 MSK 2017. Summary of my perl5 (revision 5 version 26 subversion 0) configuration: Derived from: 4c95ee9f298c2edfc1382d540ff89288790e78b6 Platform: osname=linux osvers=4.9.0-3-amd64 archname=x86_64-linux uname='linux dorothy 4.9.0-3-amd64 #1 smp debian 4.9.25-1 (2017-05-02) x86_64 gnulinux ' config_args='-des -Dusedevel -DDEBUGGING -Dcc=afl-clang-fast -Doptimize=-O0 -g -ggdb3 -fno-omit-frame-pointer' hint=previous useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='afl-clang-fast' ccflags ='-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2' optimize='-O0 -g -ggdb3 -fno-omit-frame-pointer' cppflags='-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/rc2)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='afl-clang-fast' ldflags =' -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/local/lib /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib /usr/include/x86_64-linux-gnu /usr/lib libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.24.so so=so useshrplib=false libperl=libperl.a gnulibc_version='2.24' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E' cccdlflags='-fPIC' lddlflags='-shared -O0 -g -ggdb3 -fno-omit-frame-pointer -L/usr/local/lib -fstack-protector-strong' Locally applied patches: uncommitted-changes --- @INC for perl 5.27.1: lib /usr/local/lib/perl5/site_perl/5.26.0/x86_64-linux /usr/local/lib/perl5/site_perl/5.26.0 /usr/local/lib/perl5/5.26.0/x86_64-linux /usr/local/lib/perl5/5.26.0 --- Environment for perl 5.27.1: HOME=/home/afl LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_CTYPE=en_US.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.24.1-dbg/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games PERLBREW_BASHRC_VERSION=0.78 PERLBREW_HOME=/home/afl/.perlbrew PERLBREW_MANPATH=/home/afl/perlbrew/perls/perl-5.24.1-dbg/man PERLBREW_PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.24.1-dbg/bin PERLBREW_PERL=perl-5.24.1-dbg PERLBREW_ROOT=/home/afl/perlbrew PERLBREW_VERSION=0.78 PERL_BADLANG (unset) SHELL=/usr/bin/zshpe
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 125b
Thanks, fixed by 1d5030e143202c1e963e1fc91eb6f3afaa2df83e I'm adding my vote for this to go into 5.26.1 -- Karl Williamson
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 784b
On Sat, 24 Jun 2017 11:09:30 -0700, khw wrote: Show quoted text
> Thanks, fixed by > 1d5030e143202c1e963e1fc91eb6f3afaa2df83e > > I'm adding my vote for this to go into 5.26.1
The test case: for(uc 0..t){0~~pack"UXp>",exp} here is fragile. The pack pattern here is: - pack as unicode (the error case triggers on exp(5)) - back up one byte - pack a pointer to the PV of the supplied SV, in big-endian (none is supplied, so the PV of PL_sv_no is used) So the test case depends on the high-byte of the PV in PL_sv_no being zero, which may not be the case, especially on 32-bit systems. Changing the pattern to: "UXc" should be less fragile (a zero byte will be packed instead of the high-byte of a pointer.) I discovered this while testing some other pack changes. Patch attached. Tony
Subject: 0001-perl-131646-make-the-test-less-fragile.patch
From 8e61f24b2b152a5e6a8bf22ddbab304ccdb413aa Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@debian9-x32.tony.develop-help.com> Date: Tue, 8 Aug 2017 11:09:02 +1000 Subject: (perl #131646) make the test less fragile The original pattern "UXp>" with the $_ that causes the failure, 5, so we end up packing exp(5) or 148.... with U packs: - U (148), producing C2 94, with the UTF8 flag set - X - back up a byte, - p> - write the address of PL_sv_no's PV in big-ending The final p> will typically overwrite the 94 with a zero on 64-bit systems, but with the smaller address space of 32-bit systems that high-byte is much less likely to be 0, causing the comparison to fail. Instead just pack a zero byte. --- t/lib/warnings/utf8 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/lib/warnings/utf8 b/t/lib/warnings/utf8 index dfc58c12db..a9a6388d31 100644 --- a/t/lib/warnings/utf8 +++ b/t/lib/warnings/utf8 @@ -779,7 +779,7 @@ BEGIN{ } no warnings; use warnings 'utf8'; -for(uc 0..t){0~~pack"UXp>",exp} +for(uc 0..t){0~~pack"UXc",exp} EXPECT OPTIONS regex Malformed UTF-8 character: \\x([[:xdigit:]]{2})\\x([[:xdigit:]]{2}) \(unexpected non-continuation byte 0x\2, immediately after start byte 0x\1; need 2 bytes, got 1\) in smart match at - line 9. -- 2.11.0
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 351b
On Mon, 07 Aug 2017 18:13:07 -0700, tonyc wrote: Show quoted text
> So the test case depends on the high-byte of the PV in PL_sv_no being > zero, which may not be the case, especially on 32-bit systems.
Actually, it doesn't depend on the high byte being zero, but it does depend on the high-byte not being a valid continuation byte, which might not be the case. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 518b
On Mon, 07 Aug 2017 18:23:00 -0700, tonyc wrote: Show quoted text
> On Mon, 07 Aug 2017 18:13:07 -0700, tonyc wrote:
> > So the test case depends on the high-byte of the PV in PL_sv_no being > > zero, which may not be the case, especially on 32-bit systems.
> > Actually, it doesn't depend on the high byte being zero, but it does > depend on the high-byte not being a valid continuation byte, which > might not be the case.
Patch applied as 9c6b56dc65cdd9256fbe04a7baf4f085db1c04dd with a changed comment to reflect the above. Tony


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org