From dcd@fluke.com

Created by dcd@fluke.com

I have .xs and typemap code that I've been using for 10-plus years
that has recently broken.

Fragments of my typemap include

TYPEMAP
o_max T_O_MAX

and

INPUT
T_O_MAX
  /* @​{[ $func_args =~ s/\b$var\b/$var,sizeof $var/ ]} */
  *$var = 0

this .xs code take an input fragment like​:

MODULE = DPC3 PACKAGE = DPC3
PROTOTYPES​: ENABLE

void
DPC3version(dllVersionBuf)
  o_max dllVersionBuf
  OUTPUT​:
  dllVersionBuf

and process it in a way that the function

DPCVOID DPC3version(
  char * dllVersionBuf, /* pointer to the buffer */
  int dllVersionBufSize); /* size of the buffer */

could be called.
( I've used this feature to allow 54 functions with buffer sized
to be called using this feature )

It's been a while since I dove into perl internals and packages,
so I could use some hints as to how to begin to diagnose and fix
this issue. ( I don't think the .xs file has been touched for ages either )

--
Here's a sample output of the generated .c file

#line 68 "DPC3.c"

XS(XS_DPC3_DPC3version); /* prototype to pass -Wmissing-prototypes */
XS(XS_DPC3_DPC3version)
{
#ifdef dVAR
  dVAR; dXSARGS;
#else
  dXSARGS;
#endif
  if (items != 1)
  Perl_croak(aTHX_ "Usage​: %s(%s)", "DPC3​::DPC3version", "dllVersionBuf");
  PERL_UNUSED_VAR(cv); /* -W */
  {
  o_max dllVersionBuf;

  /* 1 */
  *dllVersionBuf = 0;

  DPC3version(dllVersionBuf,sizeof dllVersionBuf);
  sv_setpv(ST(0), dllVersionBuf );
  SvSETMAGIC(ST(0));
  }
  XSRETURN_EMPTY;
}

---
The new generated code looks like this​:

XS_EUPXS(XS_DPC3_DPC3version); /* prototype to pass -Wmissing-prototypes */
XS_EUPXS(XS_DPC3_DPC3version)
{
  dVAR; dXSARGS;
  if (items != 1)
  croak_xs_usage(cv, "dllVersionBuf");
  {
  o_max dllVersionBuf;

  DPC3version(dllVersionBuf);
  sv_setpv(ST(0), dllVersionBuf );
  SvSETMAGIC(ST(0));
  }
  XSRETURN_EMPTY;
}

Flags:
    category=library
    severity=high
    module=ExtUtils::ParseXS

This perlbug was built using Perl 5.16.0 - Mon Mar 11 10:57:50 UTC 2013
It is being executed now by  Perl 5.16.0 - Mon Mar 11 10:54:42 UTC 2013.

Site configuration information for perl 5.16.0:

Configured by abuild at Mon Mar 11 10:54:42 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
   
  Platform:
    osname=linux, osvers=3.4.6-2.10-default, archname=x86_64-linux-thread-multi
    uname='linux build30 3.4.6-2.10-default #1 smp thu jul 26 09:36:26 utc 2012 (641c197) x86_64 x86_64 x86_64 gnulinux '
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
    ccversion='', gccversion='4.7.1 20120723 [gcc-4_7-branch revision 189773]', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
    libpth=/lib64 /usr/lib64 /usr/local/lib64
    libs=-lm -ldl -lcrypt -lpthread
    perllibs=-lm -ldl -lcrypt -lpthread
    libc=/lib64/libc-2.15.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.15'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.16.0/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches:
    


@INC for perl 5.16.0:
    /usr0/dcd/perl5/lib/perl5/x86_64-linux-thread-multi
    /usr0/dcd/perl5/lib/perl5
    /usr/lib/perl5/site_perl/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.0
    /usr/lib/perl5/vendor_perl/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.16.0
    /usr/lib/perl5/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.16.0
    /usr/lib/perl5/site_perl/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.0
    /usr/lib/perl5/site_perl
    .


Environment for perl 5.16.0:
    HOME=/evtfs/home/dcd
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=POSIX
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr0/dcd/perl5/bin:/usr0/dcd/bin:/usr0/dcd/sys/x86_64.linux:/usr0/hobbes/tools/scripts:/usr0/hobbes/tools/x86_64.linux:/usr0/hobbes/tools/scripts:/usr0/igdev/grponly/sw/tools/scripts:/opt/ecos/gnutools/arm-elf/bin:/evtfs/home/dcd/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/qt3/bin:/usr/sbin:/usr/ucb
    PERL5LIB=/usr0/dcd/perl5/lib/perl5
    PERL_BADLANG (unset)
    SHELL=/bin/ksh

From @iabyn

On Thu, Jul 24, 2014 at 04​:03​:37PM -0700, David Dyck wrote​:

I have .xs and typemap code that I've been using for 10-plus years
that has recently broken.

Fragments of my typemap include

TYPEMAP
o_max T_O_MAX

and

INPUT
T_O_MAX
/* @​{[ $func_args =~ s/\b$var\b/$var,sizeof $var/ ]} */
*$var = 0

I'm not a typemap expert - so if anyone wants to chip in to contradict me,
feel free - but it appears that the $func_args variable is an undocumented
internal implementation detail that not intended to be manipulated within
a type map (unlike $var, $arg etc), and which has been removed in newer
releases.

--
Little fly, thy summer's play my thoughtless hand
has terminated with extreme prejudice.
  (with apologies to William Blake)

The RT System itself - Status changed from 'new' to 'open'

From @bulk88

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to contradict me,
feel free - but it appears that the $func_args variable is an undocumented
internal implementation detail that not intended to be manipulated within
a type map (unlike $var, $arg etc), and which has been removed in newer
releases.

There are many things that are undocumented but should be documented. Here are my favorites.

T_BUF
  ".(unshift($ExtUtils​::ParseXS​::VERSION >= 3.01 ? $self->{line} : @​{eval('\@​line')}, (
  'PREINIT​:',
  ' SV * sv'.$var.';',
  'INPUT​:')),'').
  "{
  sv${\$var} = $arg;
  $var= ($type) ArgIn(aTHX_ sv${\$var});};

and older version of that which very convenient once you try using it (never updated to deal with parsexs changes) is

T_STR_LEN
  ".(unshift(@​line, (
  'PREINIT​:',
  ' STRLEN '.$var.'Len;',
  'INPUT​:')),'').
  "{
  SV * TmpSV = $arg;
  $var = ($type) SvPV( TmpSV , ${\$var}Len);
  };

Here we get the sub name, and full sub name, usually as C literals, or optimizeawayable pointer derefs.

void
Next(self)
ALIAS​:
  First = 1
  Last = 2
INPUT​:
  HV * self
  const char * longName = ${ \(eval {my $str = qq|ix == 0 ? "$pname" : |; foreach my $funcName (sort {$XsubAliases{$a} <=> $XsubAliases{$b} } keys %XsubAliases) { $str .= qq| ix == $XsubAliases{$funcName} ? "$funcName" : |;} $str .= qq| "$pname" |; return $str;})};
  const char * shortName = ${$ALIAS?\q[GvNAME(CvGV(cv))]​:\qq["$clean_func_name"]};

--
bulk88 ~ bulk88 at hotmail.coml

From @jkeenan

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

On Thu, Jul 24, 2014 at 04​:03​:37PM -0700, David Dyck wrote​:

I have .xs and typemap code that I've been using for 10-plus years
that has recently broken.

[snip]

I'm not a typemap expert - so if anyone wants to chip in to contradict me,
feel free - but it appears that the $func_args variable is an undocumented
internal implementation detail that not intended to be manipulated within
a type map (unlike $var, $arg etc), and which has been removed in newer
releases.

Mostly correct. It was an internal implementation variable. When I tried to make ExtUtils​::ParseXS into an object-oriented module as part of the "corehackers" project in 2009-2010. It was one of those entities that, IIRC, got wrapped into the EU​::PXS object. If you grep EU-PXS 3.24 -- the latest version released with Perl 5.20.0 -- you will find the string 'func_args', but not $func_args.

In any event, ExtUtils​::ParseXS​::Utilities has a function called 'assign_func_args()' but explicitly advises about its functions​:

#####
The following functions are not considered to be part of the public interface. They are documented here for the benefit of future maintainers of this module.
#####

So feel free to explore more recent versions of EU-PXS, but understand that YMMV.

Thank you very much.
Jim Keenan

From david.dyck@fluke.com

So how do we get it func_args documented ? :-)
  Early on, very little was documented, and the source code was the best documentation - I don't remember
  how I found it in the early xsubpp sources.
  I had asked some questions about it back in '98 - (http​://markmail.org/message/swontazet7pqtq2f )

I was able to patch around the problem (using $self->{func_args} instead of $func_args, but now my code is not backward compatible :-(
I could create two typemap files, and choose the correct one or use some typemap run-time test - if $self is a ref eval "x" vs eval "y"...

Index​: typemap

--- typemap (revision 19222)
+++ typemap (working copy)
@​@​ -23,7 +23,7 @​@​

INPUT
T_O_MAX
- /* @​{[ $func_args =~ s/\b$var\b/$var,sizeof $var/ ]} */
+ /* @​{[ $self->{func_args} =~ s/\b$var\b/$var,sizeof $var/ ]} */
  *$var = 0
T_FLOAT
  $var = (float)SvNV($arg)

-----Original Message-----
From​: Dave Mitchell via RT [mailto​:perlbug-followup@​perl.org]
Sent​: Friday, July 25, 2014 5​:03 AM
To​: Dyck, David
Subject​: Re​: [perl #122392] xsubpp breaks typemap $func_args processing

On Thu, Jul 24, 2014 at 04​:03​:37PM -0700, David Dyck wrote​:

I have .xs and typemap code that I've been using for 10-plus years
that has recently broken.

Fragments of my typemap include

TYPEMAP
o_max T_O_MAX

and

INPUT
T_O_MAX
/* @​{[ $func_args =~ s/\b$var\b/$var,sizeof $var/ ]} */
*$var = 0

I'm not a typemap expert - so if anyone wants to chip in to contradict me,
feel free - but it appears that the $func_args variable is an undocumented
internal implementation detail that not intended to be manipulated within
a type map (unlike $var, $arg etc), and which has been removed in newer
releases.

--
Little fly, thy summer's play my thoughtless hand
has terminated with extreme prejudice.
  (with apologies to William Blake)

From @tsee

On 07/25/2014 06​:29 PM, bulk88 via RT wrote​:

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to contradict me,
feel free - but it appears that the $func_args variable is an undocumented
internal implementation detail that not intended to be manipulated within
a type map (unlike $var, $arg etc), and which has been removed in newer
releases.

There are many things that are undocumented but should be documented. Here are my favorites.

Sorry, but there is no way (as long as I maintain it) that these things
will ever be considered EU​::ParseXS API. It's beating it with a stick. :(

--Steffen

From @tonycoz

On Sun Jul 27 01​:30​:53 2014, smueller@​cpan.org wrote​:

On 07/25/2014 06​:29 PM, bulk88 via RT wrote​:

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to
contradict me,
feel free - but it appears that the $func_args variable is an
undocumented
internal implementation detail that not intended to be manipulated
within
a type map (unlike $var, $arg etc), and which has been removed in
newer
releases.

There are many things that are undocumented but should be documented.
Here are my favorites.

Sorry, but there is no way (as long as I maintain it) that these
things
will ever be considered EU​::ParseXS API. It's beating it with a stick.
:(

I wonder if it would be worthwhile provide "official" mechanisms to do what they're doing in the typemap, so maybe​:

T_STR_LEN
  PREINIT​:
  STRLEN ${var}Len;
  EXPAND​:
  $var, ${var}Len
  INPUT​:
  {
  SV * TmpSV = $arg;
  $var = ($type) SvPV( TmpSV , ${var}Len);
  };

I haven't tried to implement this yet.

Tony

From dcd@fluke.com

On Wed, 22 Oct 2014 19​:59​:45 -0700, tonyc wrote​:

On Sun Jul 27 01​:30​:53 2014, smueller@​cpan.org wrote​:

On 07/25/2014 06​:29 PM, bulk88 via RT wrote​:

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to
contradict me,
feel free - but it appears that the $func_args variable is an
undocumented
internal implementation detail that not intended to be manipulated
within
a type map (unlike $var, $arg etc), and which has been removed in
newer
releases.

There are many things that are undocumented but should be
documented.
Here are my favorites.

Sorry, but there is no way (as long as I maintain it) that these
things
will ever be considered EU​::ParseXS API. It's beating it with a
stick.
:(

I wonder if it would be worthwhile provide "official" mechanisms to do
what they're doing in the typemap, so maybe​:

T_STR_LEN
PREINIT​:
STRLEN ${var}Len;
EXPAND​:
$var, ${var}Len
INPUT​:
{
SV * TmpSV = $arg;
$var = ($type) SvPV( TmpSV , ${var}Len);
};

I haven't tried to implement this yet.

Tony

It's been a few years and we are moving from /perl5/5.18.1/ExtUtils/ParseXS.pm $VERSION = '3.18';
to perl/5.22/ExtUtils/ParseXS.pm $VERSION = '3.28';

My earlier workaround is now harder to do because the older (3.18) ParseXS code
shared "$self" with the statement

our $self = bless {} => __PACKAGE__;

right before sub process_file

Now process_file prevents me from accessing self the same way since it protects it with
inside process_file with my $self;

Looks like I have some more work ahead of me :-(

Any hints on how to proceed with this recent breakage would be appreciated

From [Unknown Contact. See original ticket]

On Wed, 22 Oct 2014 19​:59​:45 -0700, tonyc wrote​:

On Sun Jul 27 01​:30​:53 2014, smueller@​cpan.org wrote​:

On 07/25/2014 06​:29 PM, bulk88 via RT wrote​:

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to
contradict me,
feel free - but it appears that the $func_args variable is an
undocumented
internal implementation detail that not intended to be manipulated
within
a type map (unlike $var, $arg etc), and which has been removed in
newer
releases.

There are many things that are undocumented but should be
documented.
Here are my favorites.

Sorry, but there is no way (as long as I maintain it) that these
things
will ever be considered EU​::ParseXS API. It's beating it with a
stick.
:(

I wonder if it would be worthwhile provide "official" mechanisms to do
what they're doing in the typemap, so maybe​:

T_STR_LEN
PREINIT​:
STRLEN ${var}Len;
EXPAND​:
$var, ${var}Len
INPUT​:
{
SV * TmpSV = $arg;
$var = ($type) SvPV( TmpSV , ${var}Len);
};

I haven't tried to implement this yet.

Tony

It's been a few years and we are moving from /perl5/5.18.1/ExtUtils/ParseXS.pm $VERSION = '3.18';
to perl/5.22/ExtUtils/ParseXS.pm $VERSION = '3.28';

My earlier workaround is now harder to do because the older (3.18) ParseXS code
shared "$self" with the statement

our $self = bless {} => __PACKAGE__;

right before sub process_file

Now process_file prevents me from accessing self the same way since it protects it with
inside process_file with my $self;

Looks like I have some more work ahead of me :-(

Any hints on how to proceed with this recent breakage would be appreciated

From dcd@fluke.com

On Mon, 26 Jun 2017 16​:29​:13 -0700, dcd@​fluke.com wrote​:

On Wed, 22 Oct 2014 19​:59​:45 -0700, tonyc wrote​:

On Sun Jul 27 01​:30​:53 2014, smueller@​cpan.org wrote​:

On 07/25/2014 06​:29 PM, bulk88 via RT wrote​:

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to
contradict me,
feel free - but it appears that the $func_args variable is an
undocumented
internal implementation detail that not intended to be
manipulated
within
a type map (unlike $var, $arg etc), and which has been removed
in
newer
releases.

There are many things that are undocumented but should be
documented.
Here are my favorites.

Sorry, but there is no way (as long as I maintain it) that these
things
will ever be considered EU​::ParseXS API. It's beating it with a
stick.
:(

I wonder if it would be worthwhile provide "official" mechanisms to
do
what they're doing in the typemap, so maybe​:

T_STR_LEN
PREINIT​:
STRLEN ${var}Len;
EXPAND​:
$var, ${var}Len
INPUT​:
{
SV * TmpSV = $arg;
$var = ($type) SvPV( TmpSV , ${var}Len);
};

I haven't tried to implement this yet.

Tony

It's been a few years and we are moving from
/perl5/5.18.1/ExtUtils/ParseXS.pm $VERSION = '3.18';
to perl/5.22/ExtUtils/ParseXS.pm $VERSION = '3.28';

My earlier workaround is now harder to do because the older (3.18)
ParseXS code
shared "$self" with the statement

our $self = bless {} => __PACKAGE__;

right before sub process_file

Now process_file prevents me from accessing self the same way since it
protects it with
inside process_file with my $self;

Looks like I have some more work ahead of me :-(

Any hints on how to proceed with this recent breakage would be
appreciated

Just a background and follow up on some progress I've made so far.
1) This code has been working with perl since 1998 - see​: http​://markmail.org/message/swontazet7pqtq2f

2) I was able to manually build the .xsc file by using the xsubpp, typemap, and modules from the ExtUtils distributed with v5.18.1

It is unfortunate that I can't get the ExtUtils distribute with perl v5.22.1 to work.

I wish I could have been running regression tests when perl or ExtUtils was updated
but we only discover these changes when we upgrade operating systems.

From [Unknown Contact. See original ticket]

On Mon, 26 Jun 2017 16​:29​:13 -0700, dcd@​fluke.com wrote​:

On Wed, 22 Oct 2014 19​:59​:45 -0700, tonyc wrote​:

On Sun Jul 27 01​:30​:53 2014, smueller@​cpan.org wrote​:

On 07/25/2014 06​:29 PM, bulk88 via RT wrote​:

On Fri Jul 25 05​:03​:25 2014, davem wrote​:

I'm not a typemap expert - so if anyone wants to chip in to
contradict me,
feel free - but it appears that the $func_args variable is an
undocumented
internal implementation detail that not intended to be
manipulated
within
a type map (unlike $var, $arg etc), and which has been removed
in
newer
releases.

There are many things that are undocumented but should be
documented.
Here are my favorites.

Sorry, but there is no way (as long as I maintain it) that these
things
will ever be considered EU​::ParseXS API. It's beating it with a
stick.
:(

I wonder if it would be worthwhile provide "official" mechanisms to
do
what they're doing in the typemap, so maybe​:

T_STR_LEN
PREINIT​:
STRLEN ${var}Len;
EXPAND​:
$var, ${var}Len
INPUT​:
{
SV * TmpSV = $arg;
$var = ($type) SvPV( TmpSV , ${var}Len);
};

I haven't tried to implement this yet.

Tony

It's been a few years and we are moving from
/perl5/5.18.1/ExtUtils/ParseXS.pm $VERSION = '3.18';
to perl/5.22/ExtUtils/ParseXS.pm $VERSION = '3.28';

My earlier workaround is now harder to do because the older (3.18)
ParseXS code
shared "$self" with the statement

our $self = bless {} => __PACKAGE__;

right before sub process_file

Now process_file prevents me from accessing self the same way since it
protects it with
inside process_file with my $self;

Looks like I have some more work ahead of me :-(

Any hints on how to proceed with this recent breakage would be
appreciated

Just a background and follow up on some progress I've made so far.
1) This code has been working with perl since 1998 - see​: http​://markmail.org/message/swontazet7pqtq2f

2) I was able to manually build the .xsc file by using the xsubpp, typemap, and modules from the ExtUtils distributed with v5.18.1

It is unfortunate that I can't get the ExtUtils distribute with perl v5.22.1 to work.

I wish I could have been running regression tests when perl or ExtUtils was updated
but we only discover these changes when we upgrade operating systems.