Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adjust macros dealing with UTF-8 to require extra parameter #17024

Closed
p5pRT opened this issue May 25, 2019 · 11 comments
Closed

Adjust macros dealing with UTF-8 to require extra parameter #17024

p5pRT opened this issue May 25, 2019 · 11 comments
Labels
BBC Blead Breaks CPAN - changes in blead broke a cpan module(s)
Milestone
5.32.0

Comments

@p5pRT
Copy link

p5pRT commented May 25, 2019

Migrated from rt.perl.org#134142 (status was 'new')

Searchable as RT134142$
@p5pRT
Copy link
Author

p5pRT commented May 25, 2019

From @jkeenan

pod/perldeprecation.pod contains the following entry for perl-5.32​:

#####
In XS code, use of various macros dealing with UTF-8.

These macros will require an extra parameter in Perl 5.32​:
"isALPHANUMERIC_utf8", "isASCII_utf8", "isBLANK_utf8",
"isCNTRL_utf8", "isDIGIT_utf8", "isIDFIRST_utf8",
"isPSXSPC_utf8", "isSPACE_utf8", "isVERTWS_utf8",
"isWORDCHAR_utf8", "isXDIGIT_utf8",
"isALPHANUMERIC_LC_utf8", "isALPHA_LC_utf8",
"isASCII_LC_utf8", "isBLANK_LC_utf8", "isCNTRL_LC_utf8",
"isDIGIT_LC_utf8", "isGRAPH_LC_utf8", "isIDCONT_LC_utf8",
"isIDFIRST_LC_utf8", "isLOWER_LC_utf8", "isPRINT_LC_utf8",
"isPSXSPC_LC_utf8", "isPUNCT_LC_utf8", "isSPACE_LC_utf8",
"isUPPER_LC_utf8", "isWORDCHAR_LC_utf8",
"isXDIGIT_LC_utf8", "toFOLD_utf8", "toLOWER_utf8",
"toTITLE_utf8", and "toUPPER_utf8".

There is now a macro that corresponds to each one of these,
simply by appending "_safe" to the name. It takes the extra
parameter. For example, "isDIGIT_utf8_safe" corresponds to
"isDIGIT_utf8", but takes the extra parameter, and its use
doesn't generate a deprecation warning. All are documented
in "Character case changing" in perlapi and "Character
classification" in perlapi.

You can change to use these versions at any time, or, if
you can live with the deprecation messages, wait until 5.32
and add the parameter to the existing calls, without
changing the names.

This change was originally scheduled for 5.30, but was
delayed.
#####

This entry was recorded in the following commit​:

#####
commit 5203d63
Author​: Karl Williamson <khw@​cpan.org>
Date​: Tue Jan 8 21​:04​:36 2019 -0700

Delay fatalization of deprecation of unsafe UTF-8 macros
from XS

Trying to earlier make this fatal led to some issues in
primarily Tk, which I haven't had a chance to address, and
it's getting to late in the development cycle to do it now.
#####

Make it so.

@p5pRT
Copy link
Author

p5pRT commented May 25, 2019 •
edited by toddr

From @jkeenan

Summary of my perl5 (revision 5 version 31 subversion 0) configuration​:
  Commit id​: 58f4626
  Platform​:
  osname=linux
  osvers=4.15.0-50-generic
  archname=x86_64-linux
  uname='linux zareason 4.15.0-50-generic #54-ubuntu smp mon may 6 18​:46​:08 utc 2019 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dusedevel -Dusemymalloc=y'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=undef
  usemultiplicity=undef
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=y
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='cc'
  ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
  optimize='-O2'
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion=''
  gccversion='7.4.0'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='cc'
  ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.27.so
  so=so
  useshrplib=false
  libperl=libperl.a
  gnulibc_version='2.27'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,-E'
  cccdlflags='-fPIC'
  lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  HAS_TIMES
  MYMALLOC
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_USE_DEVEL
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  Built under linux
  Compiled at May 22 2019 14​:58​:04
  %ENV​:
  PERL2DIR="/home/jkeenan/gitwork/perl2"
  PERLBREW_HOME="/home/jkeenan/.perlbrew"
  PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.28.0/man"
  PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin​:/home/jkeenan/perl5/perlbrew/perls/perl 5.28.0/bin"
  PERLBREW_PERL="perl-5.28.0"
  PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew"
  PERLBREW_SHELLRC_VERSION="0.84"
  PERLBREW_VERSION="0.84"
  PERL_WORKDIR="/home/jkeenan/gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.31.0/x86_64-linux
  /usr/local/lib/perl5/site_perl/5.31.0
  /usr/local/lib/perl5/5.31.0/x86_64-linux
  /usr/local/lib/perl5/5.31.0

@toddr
Copy link
Member

toddr commented Oct 21, 2019

@khwilliamson

@khwilliamson
Copy link
Contributor

I don't understand why I'm being pinged on this. The macros have now been changed, but doing so led to BBC tickets, as expected, and those have to be fixed before we can close this.

@p5pRT p5pRT added the khw label Oct 25, 2019
@toddr toddr removed the khw label Oct 25, 2019
@p5pRT p5pRT added the 5.32.0 label Oct 25, 2019
@toddr toddr added this to the 5.32.0 milestone Oct 25, 2019
@toddr toddr removed the 5.32.0 label Oct 25, 2019
@xsawyerx
Copy link
Member

xsawyerx commented Nov 2, 2019

I can't find the distributions this affects.

@khwilliamson
Copy link
Contributor

Tk and Devel::Cover.

After the next release of Devel::PPPort, I will submit a PR for Tk. I haven't looked at Devel::Cover. It's using "stolen_chunk_of_toke", which should be fixed. Hopefully its the same chunk as another distro was using and we can do the same thing as we did for that. I'd have to research which distro.

@xsawyerx
Copy link
Member

xsawyerx commented Nov 3, 2019

Looping in @pjcj on this. Is this the same breakage as this issue?

@pjcj
Copy link
Contributor

pjcj commented Nov 3, 2019 •
edited

At a first glance, I suspect it is not. The Devel::Cover breakage in that ticket, I think, comes from 4df8577

khwilliamson added a commit to khwilliamson/perl-tk that referenced this issue Nov 12, 2019
@khwilliamson
Use supported case changing macros
da280c9 
The code as-is no longer compiles due to changes in the API required
because of security issues.  However this patch causes it to compile
back to 5.6.0 as long as ppport.h version 3.55 or later is included.

This fix is part of Perl/perl5#17024
@khwilliamson
Copy link
Contributor

eserte/perl-tk#61 has been issued for this

@toddr toddr added the BBC Blead Breaks CPAN - changes in blead broke a cpan module(s) label Feb 17, 2020
@toddr
Copy link
Member

toddr commented Feb 17, 2020

The PR for eserte/perl-tk#61 seems to be ready to merge. I assume we can close this.

@toddr toddr added the Closable? We might be able to close this ticket, but we need to check with the reporter label Feb 17, 2020
@jkeenan
Copy link
Contributor

jkeenan commented Feb 17, 2020

The PR for eserte/perl-tk#61 seems to be ready to merge. I assume we can close this.

Let's give the maintainer the breathing room he has requested to get a new Tk release out.

eserte pushed a commit to eserte/perl-tk that referenced this issue Feb 23, 2020
@khwilliamson @eserte
Use supported case changing macros
934e354 
The code as-is no longer compiles due to changes in the API required
because of security issues.  However this patch causes it to compile
back to 5.6.0 as long as ppport.h version 3.55 or later is included.

This fix is part of Perl/perl5#17024
@toddr toddr closed this as completed Feb 23, 2020
@toddr toddr removed the Closable? We might be able to close this ticket, but we need to check with the reporter label Feb 23, 2020
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Sep 7, 2020
p5-Tk: update to 804.035.
5a9b7d3 
Tk-804.035 release (2020-03-28)
------------------

This is basically the same like Tk-804.034_501.

Tk-804.034_501 release (2020-03-19)
----------------------

Fixes
 Add dummy rules for newer EUMM (RT #124910)

 Try harder to find libpng on FreeBSD

Tk-804.034_500 release (2020-02-23)
----------------------

Fixes
 Compiles again with perl 5.31.x (see also
 github pull request #61 and Perl/perl5#17024
 by Karl Williamson). Now ppport.h is included in the
 distribution.

 Switch to X's locale handling if needed (Karl Williamson)

 Don't use buttons 4 and 5 on non-X11 platforms
 (Christopher Chavez)

 Fix corner case in XPM handling (/* in colors section)

 Spelling/grammar fixes in comments and code (Christopher Chavez)

 Improved demo script photo1.pl (Christopher Chavez)

Documentation
 Smaller Pod fixes (Slaven Rezic, Christopher Chavez)

Tests
 travis-ci fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BBC Blead Breaks CPAN - changes in blead broke a cpan module(s)
Projects
None yet
Milestone
5.32.0
Development

No branches or pull requests
6 participants
@xsawyerx @jkeenan @toddr @khwilliamson @pjcj @p5pRT