From @eserte

This is a bug report for perl from slaven@​rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log​:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
  Slaven

Flags​:
  category=core
  severity=low

Site configuration information for perl 5.29.8​:

Configured by eserte at Thu Feb 21 06​:51​:23 UTC 2019.

Summary of my perl5 (revision 5 version 29 subversion 8) configuration​:
 
  Platform​:
  osname=linux
  osvers=4.15.0-45-generic
  archname=x86_64-linux
  uname='linux ubuntu18 4.15.0-45-generic #48-ubuntu smp tue jan 29 16​:28​:13 utc 2019 x86_64 x86_64 x86_64 gnulinux '
  config_args='-ds -e -Dprefix=/opt/perl-5.29.8 -Dusedevel -Dusemallocwrap=no -Dcf_email=srezic@​cpan.org'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=undef
  usemultiplicity=undef
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='cc'
  ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
  optimize='-O2'
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion=''
  gccversion='7.3.0'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='cc'
  ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.27.so
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version='2.27'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,-E'
  cccdlflags='-fPIC'
  lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

@​INC for perl 5.29.8​:
  /opt/perl-5.29.8/lib/site_perl/5.29.8/x86_64-linux
  /opt/perl-5.29.8/lib/site_perl/5.29.8
  /opt/perl-5.29.8/lib/5.29.8/x86_64-linux
  /opt/perl-5.29.8/lib/5.29.8

Environment for perl 5.29.8​:
  HOME=/home/eserte
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LC_ALL=de_DE.UTF-8
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/usr/local/bin​:/usr/bin​:/bin​:/usr/local/sbin​:/usr/sbin​:/sbin​:/home/eserte/bin/linux-gnu​:/home/eserte/bin/sh​:/home/eserte/bin​:/home/eserte/bin/pistachio-perl/bin​:/usr/games​:/home/eserte/devel
  PERLDOC=-MPod​::Perldoc​::ToTextOverstrike
  PERL_BADLANG (unset)
  SHELL=/usr/bin/zsh

From @jkeenan

On Mon, 18 Mar 2019 20​:11​:06 GMT, slaven@​rezic.de wrote​:

This is a bug report for perl from slaven@​rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

-----------------------------------------------------------------
The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log​:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of
InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at
t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
Slaven

Bisecting with the following invocation ...

#####
perl Porting/bisect.pl --start=ca152fd8207cf53816b1407d5f54f6ea160a3ef8 --end=ca8b93afd02ddde55c1aa9e6fbff9acdad31593d --module=Lingua​::JA​::Moji
#####

... pointed to this commit​:

#####
HEAD is now at 73b9584 Move \p{user-defined} to core from utf8_heavy.pl
bad - non-zero exit from /tmp/BGZpCa7exV/bin/perl -I /home/jkeenan/.cpan -MCPAN​::MyConfig -MCPAN -e $CPAN​::Config->{build_dir}=q{/tmp/vrCuaE5dQx}; -e install('Lingua​::JA​::Moji'); die unless CPAN​::Shell->expand(Module => 'Lingua​::JA​::Moji')->uptodate;
73b9584 is the first bad commit
commit 73b9584
Author​: Karl Williamson <khw@​cpan.org>
Date​: Mon Aug 20 18​:31​:04 2018 -0600

Move \p{user-defined} to core from utf8_heavy.pl

This large commit moves the handling of user-defined
properties to C code. This should speed it up, but the main
reason to do this is to stop using swashes in this case,
leaving only tr/// using them. Once that too is converted,
all swash handling can be ripped out of perl.

Doing this in perl has caused some nasty interactions that
will now be fixed automatically.

The change is not entirely transparent, however (besides
speed and the possibility of removing these interactions).
perldelta in this commit details these.
#####

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

The RT System itself - Status changed from 'new' to 'open'

From @khwilliamson

On 3/18/19 8​:24 PM, James E Keenan via RT wrote​:

On Mon, 18 Mar 2019 20​:11​:06 GMT, slaven@​rezic.de wrote​:

This is a bug report for perl from slaven@​rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

-----------------------------------------------------------------
The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log​:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of
InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at
t/InKana.t line 22.
t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
...

Regards,
Slaven

Bisecting with the following invocation ...

#####
perl Porting/bisect.pl --start=ca152fd8207cf53816b1407d5f54f6ea160a3ef8 --end=ca8b93afd02ddde55c1aa9e6fbff9acdad31593d --module=Lingua​::JA​::Moji
#####

... pointed to this commit​:

#####
HEAD is now at 73b9584 Move \p{user-defined} to core from utf8_heavy.pl
bad - non-zero exit from /tmp/BGZpCa7exV/bin/perl -I /home/jkeenan/.cpan -MCPAN​::MyConfig -MCPAN -e $CPAN​::Config->{build_dir}=q{/tmp/vrCuaE5dQx}; -e install('Lingua​::JA​::Moji'); die unless CPAN​::Shell->expand(Module => 'Lingua​::JA​::Moji')->uptodate;
73b9584 is the first bad commit
commit 73b9584
Author​: Karl Williamson <khw@​cpan.org>
Date​: Mon Aug 20 18​:31​:04 2018 -0600

Move \p{user-defined} to core from utf8_heavy.pl

This large commit moves the handling of user-defined
properties to C code. This should speed it up, but the main
reason to do this is to stop using swashes in this case,
leaving only tr/// using them. Once that too is converted,
all swash handling can be ripped out of perl.

Doing this in perl has caused some nasty interactions that
will now be fixed automatically.

The change is not entirely transparent, however (besides
speed and the possibility of removing these interactions).
perldelta in this commit details these.
#####

Thank you very much.

My guess is that it's the same problem as in
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133860

which is the result of the module unsuspectingly relying on a bug in the
former implementation. I may have to come up with a workaround.

From @khwilliamson

On 3/18/19 8​:38 PM, Karl Williamson wrote​:

On 3/18/19 8​:24 PM, James E Keenan via RT wrote​:

On Mon, 18 Mar 2019 20​:11​:06 GMT, slaven@​rezic.de wrote​:

This is a bug report for perl from slaven@​rezic.de,
generated with the help of perlbug 1.41 running under perl 5.29.8.

-----------------------------------------------------------------
The test suite of BKB/Lingua-JA-Moji-0.55.tar.gz fails
with perl 5.29.8. 5.29.7 and earlier passes on all systems, it
seems.

Excerpt of test log​:

...
Can't find Unicode property definition "FF9E\tFF9F" in expansion of
InKana in regex; marked by <-- HERE in m/^\p{InKana} <-- HERE +$/ at
t/InKana.t line 22.
  t/InKana.t .....................
Dubious, test returned 255 (wstat 65280, 0xff00)
  No subtests run
...

Regards,
     Slaven

Bisecting with the following invocation ...

#####
perl Porting/bisect.pl
--start=ca152fd8207cf53816b1407d5f54f6ea160a3ef8
--end=ca8b93afd02ddde55c1aa9e6fbff9acdad31593d --module=Lingua​::JA​::Moji
#####

... pointed to this commit​:

#####
HEAD is now at 73b9584 Move \p{user-defined} to core from
utf8_heavy.pl
bad - non-zero exit from /tmp/BGZpCa7exV/bin/perl -I
/home/jkeenan/.cpan -MCPAN​::MyConfig -MCPAN -e
$CPAN​::Config->{build_dir}=q{/tmp/vrCuaE5dQx}; -e
install('Lingua​::JA​::Moji'); die unless CPAN​::Shell->expand(Module =>
'Lingua​::JA​::Moji')->uptodate;
73b9584 is the first bad commit
commit 73b9584
Author​: Karl Williamson <khw@​cpan.org>
Date​:   Mon Aug 20 18​:31​:04 2018 -0600

Move \p{user-defined} to core from utf8_heavy.pl

This large commit moves the handling of user-defined
properties to C code. This should speed it up, but the main
reason to do this is to stop using swashes in this case,
leaving only tr/// using them. Once that too is converted,
all swash handling can be ripped out of perl.

Doing this in perl has caused some nasty interactions that
will now be fixed automatically.

The change is not entirely transparent, however (besides
speed and the possibility of removing these interactions).
perldelta in this commit details these.
#####

Thank you very much.

My guess is that it's the same problem as in
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133860

which is the result of the module unsuspectingly relying on a bug in the
former implementation.  I may have to come up with a workaround.

I investigated further and my above guess is wrong.
It turns out that it is a bug in the distro, and I have created a pull
request benkasminbullock/Lingua-JA-Moji#23

The problem is that it had single-quoted '\t' instead of double-quoted
"\t", which created not the intended tab, but backslash-t. The improved
error checking in the new implementation caught it. I did not
investigate to see what the old implementation did; perhaps it compiled
as DWIM, but perhaps it compiled garbage.

But there is a bug in the new implementation which didn't report the
error properly. I will investigate and fix that.

From @khwilliamson

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged
--
Karl Williamson

@khwilliamson - Status changed from 'open' to 'pending release'

From @eserte

Dana Mon, 25 Mar 2019 09​:46​:37 -0700, khw reče​:

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged

Unfortunately it's not yet completely fixed. The test suite of the new version still fails with 5.29.9​:

# Failed test 'decomposition of katakana into syllables'
# at t/katakana2syllable.t line 19.
# Structures begin differing at​:
# $got->[0] = 'ック'
# $expected->[0] = 'ソー'

# Failed test 'ya plus chouon'
# at t/katakana2syllable.t line 27.
# Structures begin differing at​:
# $got->[0] = Does not exist
# $expected->[0] = 'ソー'

# Failed test at t/katakana2syllable.t line 36.
# Structures begin differing at​:
# $got->[0] = Does not exist
# $expected->[0] = 'ノー'
# Looks like you failed 3 tests of 4.
t/katakana2syllable.t ..........
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/4 subtests

From @khwilliamson

On 3/26/19 2​:50 PM, slaven@​rezic.de via RT wrote​:

Dana Mon, 25 Mar 2019 09​:46​:37 -0700, khw reče​:

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged

Unfortunately it's not yet completely fixed. The test suite of the new version still fails with 5.29.9​:

# Failed test 'decomposition of katakana into syllables'
# at t/katakana2syllable.t line 19.
# Structures begin differing at​:
# $got->[0] = 'ック'
# $expected->[0] = 'ソー'

# Failed test 'ya plus chouon'
# at t/katakana2syllable.t line 27.
# Structures begin differing at​:
# $got->[0] = Does not exist
# $expected->[0] = 'ソー'

# Failed test at t/katakana2syllable.t line 36.
# Structures begin differing at​:
# $got->[0] = Does not exist
# $expected->[0] = 'ノー'
# Looks like you failed 3 tests of 4.
t/katakana2syllable.t ..........
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/4 subtests

---
This should be fixed by

commit 766d6d3
  Author​: Karl Williamson <khw@​cpan.org>
  Date​: Thu Mar 28 12​:26​:43 2019 -0600

  PATCH​: [perl #133942] BBC​: BKB/Lingua-JA-Moji

  This ticket was originally fixed by a PR being issued on the
  distribution, which was merged and a new version placed on CPAN.

  But before Slaven had a chance to test it, commit
  765e6ec came along, and happened to
  break it again from a totally independent cause.

  So we have two separate bugs on the same ticket, only one of which was
  perl's fault. And this commit fixes that. The problematic commit was
  still looking in the regnode FLAGS field in one function, whereas that
  field has been repurposed for ANYOFH nodes, so should be ignored for
  them.

From @jkeenan

On Thu, 28 Mar 2019 19​:31​:35 GMT, public@​khwilliamson.com wrote​:

On 3/26/19 2​:50 PM, slaven@​rezic.de via RT wrote​:

Dana Mon, 25 Mar 2019 09​:46​:37 -0700, khw reče​:

Commit 6256cf2
fixes the failure mentioned above.

Otherwise, the patch to the distro has now been merged

Unfortunately it's not yet completely fixed. The test suite of the
new version still fails with 5.29.9​:

# Failed test 'decomposition of katakana into syllables'
# at t/katakana2syllable.t line 19.
# Structures begin differing at​:
# $got->[0] = 'ック'
# $expected->[0] = 'ソー'

# Failed test 'ya plus chouon'
# at t/katakana2syllable.t line 27.
# Structures begin differing at​:
# $got->[0] = Does not exist
# $expected->[0] = 'ソー'

# Failed test at t/katakana2syllable.t line 36.
# Structures begin differing at​:
# $got->[0] = Does not exist
# $expected->[0] = 'ノー'
# Looks like you failed 3 tests of 4.
t/katakana2syllable.t ..........
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/4 subtests

---
This should be fixed by

commit 766d6d3
Author​: Karl Williamson <khw@​cpan.org>
Date​: Thu Mar 28 12​:26​:43 2019 -0600

PATCH​: [perl #133942] BBC​: BKB/Lingua-JA-Moji

This ticket was originally fixed by a PR being issued on the
distribution, which was merged and a new version placed on CPAN.

But before Slaven had a chance to test it, commit
765e6ec came along, and happened to
break it again from a totally independent cause.

So we have two separate bugs on the same ticket, only one of which was
perl's fault. And this commit fixes that. The problematic commit was
still looking in the regnode FLAGS field in one function, whereas that
field has been repurposed for ANYOFH nodes, so should be ignored for
them.

Lingua​::JA​::Moji now PASS for me at v5.29.9-38-gca21d46d3d in the FreeBSD environment cited earlier.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.30.0, this and 160 other issues have been
resolved.

Perl 5.30.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.30.0

If you find that the problem persists, feel free to reopen this ticket.

@khwilliamson - Status changed from 'pending release' to 'resolved'