From @andk

Slaven digged it out, the second on this day, delightfully hard to track.

bisect

commit 13f4dd3
Author​: Abigail <abigail@​abigail.be>
Date​: Tue Jun 6 18​:51​:37 2017 +0200

  Forbid out of range Unicode code points.

diagnostics

Note​: no evidence available for threaded perls; you need not try because
of https://rt.cpan.org/Ticket/Display.html?id=109841

: t/m4.t ................. 73/?
:
: # Failed test 'regexp - GNU emacs'
: # at t/m4.t line 96.
: # got​: '
: #
: #
: #
: # '
: # expected​: '5
: # -1
: # *** Unix *** nix ***
: #
: # '
:
: # Failed test 'patsubst'
: # at t/m4.t line 96.
: # got​: 'OBS​: GNUs not Unix
: # OBS​: GNUs OBS​: not OBS​: Unix
: #
: #
: # GN not
: # GNUs NOT Unix
: # '
: # expected​: 'OBS​: GNUs not Unix
: # OBS​: GNUs OBS​: not OBS​: Unix
: # (GNUs)() (not)() (Unix)()
: # (GNUs) (not) (Unix)
: # GN not
: # GNUs NOT Unix
: # '
:
: # Failed test 'patsubst - composites upcase/downcase/capitalize'
: # at t/m4.t line 96.
: # got​: '
: # GNUS NOT UNIX
: # gnus not unix
: #
: # '
: # expected​: '
: # GNUS NOT UNIX
: # gnus not unix
: # Gnus Not Unix
: # '
: t/m4.t ................. 120/?
: # Looks like you failed 3 tests of 141.
: t/m4.t ................. Dubious, test returned 3 (wstat 768, 0x300)
: Failed 3/141 subtests

perl -V

Summary of my perl5 (revision 5 version 27 subversion 1) configuration​:
  Commit id​: 13f4dd3
  Platform​:
  osname=linux
  osvers=4.9.0-2-amd64
  archname=x86_64-linux
  uname='linux k93msid 4.9.0-2-amd64 #1 smp debian 4.9.18-1 (2017-03-30) x86_64 gnulinux '
  config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.0-132-g13f4dd346e/89ad -Dmyhostname=k93msid -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Dlibswanted=cl pthread socket inet nsl gdbm dbm malloc dl ld sun m crypt sec util c cposix posix ucb BSD gdbm_compat -Uuseithreads -Uuselongdouble -DDEBUGGING=-g'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=undef
  usemultiplicity=undef
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='cc'
  ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
  optimize='-O2 -g'
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion=''
  gccversion='6.3.0 20170406'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='cc'
  ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.24.so
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version='2.24'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,-E'
  cccdlflags='-fPIC'
  lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  HAS_TIMES
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_USE_DEVEL
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  Built under linux
  Compiled at Jun 27 2017 05​:08​:23
  @​INC​:
  /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.0-132-g13f4dd346e/89ad/lib/site_perl/5.27.1/x86_64-linux
  /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.0-132-g13f4dd346e/89ad/lib/site_perl/5.27.1
  /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.0-132-g13f4dd346e/89ad/lib/5.27.1/x86_64-linux
  /home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.0-132-g13f4dd346e/89ad/lib/5.27.1

--
andreas

From @khwilliamson

I'll take a look at this.
--
Karl Williamson

The RT System itself - Status changed from 'new' to 'open'

From @khwilliamson

On Tue, 27 Jun 2017 12​:27​:03 -0700, khw wrote​:

I'll take a look at this.

I have traced this down to these lines in the distro​:

  use re​::engine​::GNU 0.019;
  $regexp = qr/$regexpString/sm;
  no re​::engine​::GNU;

And, the GNU engine as interpreted by re​::engine​::GNU is calling back to Perl's utf8_to_uvchr() with code points that are no longer legal because they are above IV_MAX, in particular 0xFF...FF80. (I lost track of how many F's, and its not important enough to figure it out.)

If I use Perl's engine by commenting out the 'use re...', this error doesn't happen, though other failures occur, due presumably to the differences in semantics in GNU's.

A backtrace yields​:

(gdb) bt
#0 Perl_utf8n_to_uvchr_error (s=0x7fffffffc307 "", curlen=13, retlen=0x0, flags=0, errors=0x7fffffffc244) at utf8.c​:1630
#1 0x000055555587f9e8 in Perl__force_out_malformed_utf8_message (p=0x7fffffffc2fa "\377\200\217\277\277\277\277\277\277\277\277\276\200", e=0x7fffffffc308 "", flags=0, die_here=true)
  at utf8.c​:90
#2 0x000055555588476b in S_is_utf8_common_with_len (p=0x7fffffffc2fa "\377\200\217\277\277\277\277\277\277\277\277\276\200", e=0x7fffffffc308 "",
  swash=0x555555be0860 <PL_utf8_swash_ptrs>, swashname=0x5555559994a0 "XPosixWord", invlist=0x555555be2f50) at utf8.c​:2816
#3 0x00005555558852dc in Perl__is_utf8_FOO_with_len (classnum=0 '\000', p=0x7fffffffc2fa "\377\200\217\277\277\277\277\277\277\277\277\276\200", e=0x7fffffffc308 "") at utf8.c​:2975
#4 0x000055555588333d in Perl__is_uni_FOO (classnum=0 '\000', c=18446744073709551488) at utf8.c​:2460
#5 0x00007fffeec0d10a in rpl_Perl_isalnum (c=18446744073709551488) at regex_internal.h​:1101
#6 0x00007fffeec19fd5 in build_charclass (trans=0x0, sbcset=0x55555add9250, mbcset=0x55555aeb4fe0, char_class_alloc=0x7fffffffc3f8, class_name=0x7fffeec2aa84 "alnum", syntax=0)
  at regcomp.c​:3576
#7 0x00007fffeec1a903 in build_charclass_op (dfa=0x55555aebe080, trans=0x0, class_name=0x7fffeec2aa84 "alnum", extra=0x7fffeec2b0c4 "_", non_match=false, err=0x7fffffffc624)
  at regcomp.c​:3645
#8 0x00007fffeec183a8 in parse_expression (regexp=0x7fffffffc630, preg=0x55555ae5a278, token=0x7fffffffc5c0, syntax=64, nest=0, err=0x7fffffffc624) at regcomp.c​:2428
#9 0x00007fffeec17be0 in parse_branch (regexp=0x7fffffffc630, preg=0x55555ae5a278, token=0x7fffffffc5c0, syntax=64, nest=0, err=0x7fffffffc624) at regcomp.c​:2237
#10 0x00007fffeec179fc in parse_reg_exp (regexp=0x7fffffffc630, preg=0x55555ae5a278, token=0x7fffffffc5c0, syntax=64, nest=0, err=0x7fffffffc624) at regcomp.c​:2189
#11 0x00007fffeec178da in parse (regexp=0x7fffffffc630, preg=0x55555ae5a278, syntax=64, err=0x7fffffffc624) at regcomp.c​:2158
#12 0x00007fffeec14563 in re_compile_internal (preg=0x55555ae5a278, pattern=0x55555aea1e80 "\\<[a-z]\\w+", length=10, syntax=64, is_utf8=true) at regcomp.c​:813
#13 0x00007fffeec2645f in GNU_comp (pattern=0x555558cf3000, flags=259) at GNU.xs​:439
#14 0x0000555555674dd8 in Perl_re_op_compile (patternp=0x55555a24db30, pat_count=1, expr=0x0, eng=0x7fffeee2f340 <engine_GNU>, old_re=0x55555ac02138, is_bare_re=0x7fffffffd02a,
  orig_rx_flags=259, pm_flags=1073742083) at regcomp.c​:7016
#15 0x00005555557de49e in Perl_pp_regcomp () at pp_ctl.c​:108
#16 0x00005555556cdbdf in Perl_runops_debug () at dump.c​:2465
#17 0x00005555555c8e45 in S_run_body (oldscope=1) at perl.c​:2543
#18 0x00005555555c83fd in perl_run (my_perl=0x555555be1010) at perl.c​:2471
#19 0x0000555555580a9e in main (argc=3, argv=0x7fffffffd418, env=0x7fffffffd438) at perlmain.c​:123

Stack position

#5 0x00007fffeec0d10a in rpl_Perl_isalnum (c=18446744073709551488)

indicates that the re​::engine function rpl_Perl_isalnum() is being called with a code point (c=) that is now illegal in perl. It calls the perl function Perl__is_uni_FOO, which no longer accepts this input value.

So this is the problem. Perl calls the substitute engine in frame #13, and it calls functions within itself until the call at #4. (It has its own regcomp.c, so that isn't ours in those frames.)

At this point, I'm uncertain what to do. I'm pretty sure this isn't Perl's bug. We deprecated use of these code points for 2 cycles, and now they're illegal.

--
Karl Williamson

From @jkeenan

On Wed, 26 Jul 2017 16​:31​:02 GMT, khw wrote​:

At this point, I'm uncertain what to do. I'm pretty sure this isn't
Perl's bug. We deprecated use of these code points for 2 cycles, and
now they're illegal.

If that's the case, then can you open a bug ticket at https://rt.cpan.org/Dist/Display.html?Name=MarpaX-Languages-M4?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

From @khwilliamson

On 07/26/2017 04​:53 PM, James E Keenan via RT wrote​:

On Wed, 26 Jul 2017 16​:31​:02 GMT, khw wrote​:

At this point, I'm uncertain what to do. I'm pretty sure this isn't
Perl's bug. We deprecated use of these code points for 2 cycles, and
now they're illegal.

If that's the case, then can you open a bug ticket at https://rt.cpan.org/Dist/Display.html?Name=MarpaX-Languages-M4?

Thank you very much.

Now done

https://rt.cpan.org/Ticket/Display.html?id=122641

From @khwilliamson

The owner has already issued a fix for this, in re​::engine​::GNU 0.022.

I have tested it, and the test no longer fails.

--
Karl Williamson

@khwilliamson - Status changed from 'open' to 'resolved'