experimental bitwise feature breaks PDL's & operator #16307

p5pRT · 2017-12-15T22:28:25Z

Migrated from rt.perl.org#132589 (status was 'rejected')

Searchable as RT132589$

p5pRT · 2017-12-15T22:28:25Z

From @djerius

Created by @djerius

The experimental bitwise feature breaks PDL code which uses the overloaded '&' operator.

Sample code:

-----------------------------------

use PDL;

print "\n";

print "bitwise off: ", pdl( 1 ) & pdl(2 ), "\n";

use feature 'bitwise';

print "bitwise on: ", pdl( 1 ) & pdl(2 ), "\n";

-----------------------------------

The output:

-----------------------------------

% perl bitwise.pl
The bitwise feature is experimental at bitwise.pl line 9.

bitwise off: 0
Usage: PDL::and2(a,b,c,swap) (you may leave temporaries or output variables out of list) at bitwise.pl line 9.

-----------------------------------

PDL::and2 is an XS function which provides the overloaded behavior. Its arguments are getting changed by
whatever bitwise is doing.

Thanks,
Diab

Perl Info


Flags:
    category=core
    severity=low

Site configuration information for perl 5.22.1:

Configured by dj at Wed Jul 27 15:59:25 EDT 2016.

Summary of my perl5 (revision 5 version 22 subversion 1) configuration:
   
  Platform:
    osname=linux, osvers=3.10.0-327.22.2.el7.x86_64, archname=x86_64-linux_CentOS-7-thread-multi
    uname='linux sparkles.cfa.harvard.edu 3.10.0-327.22.2.el7.x86_64 #1 smp thu jun 23 17:05:11 utc 2016 x86_64 x86_64 x86_64 gnulinux '
    config_args='-de -Dprefix=/proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7 -Uinstallusrbinperl -Dperladmin=none -Dlocincpth=/proj/axaf/ots/include -Darchname=x86_64-linux_CentOS-7 -Dusethreads -Dotherlibdirs=/proj/axaf/simul/lib/perl5:/proj/axaf/simul/x86_64-linux_CentOS-7/lib/perl5 -Dloclibpth=/proj/axaf/ots/x86_64-linux_CentOS-7/lib -Dglibpth=/lib /usr/lib -Dldflags=-L/proj/axaf/ots/x86_64-linux_CentOS-7/lib -Wl,-rpath=/proj/axaf/ots/x86_64-linux_CentOS-7/lib -Duseshrplib'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/proj/axaf/ots/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/proj/axaf/ots/include'
    ccversion='', gccversion='4.8.5 20150623 (Red Hat 4.8.5-4)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-L/proj/axaf/ots/x86_64-linux_CentOS-7/lib -Wl,-rpath=/proj/axaf/ots/x86_64-linux_CentOS-7/lib -fstack-protector-strong'
    libpth=/usr/local/lib /usr/lib /proj/axaf/ots/x86_64-linux_CentOS-7/lib /lib/../lib64 /usr/lib/../lib64 /lib
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.17.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.17'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7/lib/5.22.1/x86_64-linux_CentOS-7-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/proj/axaf/ots/x86_64-linux_CentOS-7/lib -fstack-protector-strong'



@INC for perl 5.22.1:
    /proj/axaf/simul/lib/perl5
    /home/dj/lib/perl5
    /home/dj/research/lib/perl5
    /proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7/lib/site_perl/5.22.1/x86_64-linux_CentOS-7-thread-multi
    /proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7/lib/site_perl/5.22.1
    /proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7/lib/5.22.1/x86_64-linux_CentOS-7-thread-multi
    /proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7/lib/5.22.1
    /proj/axaf/simul/lib/perl5
    /proj/axaf/simul/x86_64-linux_CentOS-7/lib/perl5
    .


Environment for perl 5.22.1:
    HOME=/home/dj
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/soft/SYBASE15.7/DataAccess64/ODBC/lib:/soft/SYBASE15.7/DataAccess/ODBC/lib:/soft/SYBASE15.7/OCS-15_0/lib:/soft/SYBASE15.7/OCS-15_0/lib3p64:/soft/SYBASE15.7/OCS-15_0/lib3p:/soft/SYBASE15.7/DataAccess64/ODBC/lib:/soft/SYBASE15.7/DataAccess/ODBC/lib:/soft/SYBASE15.7/OCS-15_0/lib:/soft/SYBASE15.7/OCS-15_0/lib3p64:/soft/SYBASE15.7/OCS-15_0/lib3p:
    LOGDIR (unset)
    PATH=/home/dj/.plenv/libexec:/home/dj/.plenv/plugins/link-axaf/bin:/home/dj/.plenv/plugins/perl-build/bin:/home/dj/.plenv/plugins/plenv-contrib/bin:/soft/SYBASE15.7/OCS-15_0/bin:/home/dj/.rbenv/shims:/home/dj/.rbenv/bin:/home/dj/.pyenv/plugins/pyenv-virtualenv/shims:/home/dj/.pyenv/shims:/home/dj/.pyenv/bin:/home/dj/.plenv/shims:/home/dj/.plenv/bin:/home/dj/root/bin:/home/dj/root/bin/hosts:/home/dj/research/bin:/proj/axaf/ots/pkgs/perl-5.22/x86_64-linux_CentOS-7/bin:/proj/axaf/bin:/proj/axaf/simul/bin:/proj/axaf/simul/x86_64-linux_CentOS-7/bin:/proj/axaf/ots/x86_64-linux_CentOS-7/bin:/proj/axaf/ots/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/sbin:/bin:/usr/bin:/soft/saord/bin
    PERL5LIB=/proj/axaf/simul/lib/perl5:/home/dj/lib/perl5:/home/dj/research/lib/perl5
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2017-12-17T19:08:38Z

From @cpansprout

On Fri, 15 Dec 2017 14:28:25 -0800, djerius wrote:

This is a bug report for perl from djerius@cpan.org,
generated with the help of perlbug 1.40 running under perl 5.22.1.

-----------------------------------------------------------------
[Please describe your issue here]

The experimental bitwise feature breaks PDL code which uses the
overloaded '&' operator.

Usage: PDL::and2(a,b,c,swap) (you may leave temporaries or output
variables out of list) at bitwise.pl line 9.

Due to the nature of the bitwise feature, it is necessary to pass extra arguments to overload functions. overload.pm says the following:

If the experimental "bitwise" feature is enabled (see feature), a fifth
TRUE argument is passed to subroutines handling "&", "|", "^" and "~".
This indicates that the caller is expecting numeric behaviour. The fourth
argument will be "undef", as that position ($_[3]) is reserved for use by
"nomethod".

I have submitted a PDL patch at <https://rt.cpan.org/Ticket/Display.html?id=123901>.

There is no Perl bug here.

--

Father Chrysostomos

p5pRT · 2017-12-17T19:08:38Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2017-12-17T19:08:39Z

@cpansprout - Status changed from 'open' to 'rejected'

p5pRT · 2017-12-18T11:54:17Z

From @sisyphus

Note that both Math::GMP and Math::GMPz are also similarly broken by
enabling the bitwise feature. For example:

################################
C:\_32>cat bitwise.pl
use warnings;
use Math::GMP;
use feature 'bitwise';

print Math::GMP->new(2) & Math::GMP->new(3);

C:\_32>perl bitwise.pl
The bitwise feature is experimental at bitwise.pl line 5.
Usage: Math::GMP::band(m, n, swap) at bitwise.pl line 5.

C:\_32>
################################

I've submitted a bug report against Math::GMP for this (see
https://rt.cpan.org/Ticket/Display.html?id=123907 )

I'll fix Math::GMPz as soon as I work out how best to deal with this from
inside an XSub.
I need to find out:
1) How does the XSub establish that the caller has enabled the bitwise
feature;
2) How does the XSub silence the warning.

I wonder how many other modules might be broken by this feature.
I've checked both Math::BigInt and Bit::Vector (as they were the first to
come to mind), but they're unaffected.
I haven't checked any other modules.

Cheers,
Rob

p5pRT · 2017-12-18T12:32:40Z

From zefram@fysh.org

Sisyphus via RT wrote:

I need to find out:
1) How does the XSub establish that the caller has enabled the bitwise
feature;

From the fifth argument supplied to the xsub. But Math::GMP has no need
to know that, because its bitwise operators only provide numeric behaviour
in any case. There's no difference to Math::GMP between the old ambiguous
conjunction and the new numeric-only conjunction. Math::GMP::band()
should ignore the fifth argument (and the fourth, and the third).

2) How does the XSub silence the warning.

It doesn't. The warning happens in the calling scope, which is where
the bitwise feature is enabled, and it can be silenced by that scope's
warning flags.

-zefram

p5pRT · 2017-12-18T15:11:02Z

From @djerius

On Sun, Dec 17, 2017 at 2:08 PM, Father Chrysostomos via RT
<perlbug-followup@perl.org> wrote:

On Fri, 15 Dec 2017 14:28:25 -0800, djerius wrote:

This is a bug report for perl from djerius@cpan.org,
generated with the help of perlbug 1.40 running under perl 5.22.1.

-----------------------------------------------------------------
[Please describe your issue here]

The experimental bitwise feature breaks PDL code which uses the
overloaded '&' operator.

Usage: PDL::and2(a,b,c,swap) (you may leave temporaries or output
variables out of list) at bitwise.pl line 9.

Due to the nature of the bitwise feature, it is necessary to pass extra arguments to overload functions. overload.pm says the following:
If the experimental "bitwise" feature is enabled $see feature$\, a fifth
TRUE argument is passed to subroutines handling "&"\, "|"\, "^" and "~"\.
This indicates that the caller is expecting numeric behaviour\. The fourth
argument will be "undef"\, as that position $$\_\[3\]$ is reserved for use by
"nomethod"\.
I have submitted a PDL patch at <https://rt.cpan.org/Ticket/Display.html?id=123901>.

There is no Perl bug here.

Thanks.

PDL doesn't use rt.cpan.org for bug-tracking, so I've submitted a
report at their SourceForge site:

https://sourceforge.net/p/pdl/bugs/442/

p5pRT · 2017-12-19T00:06:00Z

From @sisyphus

On Mon, 18 Dec 2017 04:32:40 -0800, zefram@fysh.org wrote:

Sisyphus via RT wrote:

I need to find out:
1) How does the XSub establish that the caller has enabled the bitwise
feature;

From the fifth argument supplied to the xsub. But Math::GMP has no need
to know that, because its bitwise operators only provide numeric behaviour
in any case. There's no difference to Math::GMP between the old ambiguous
conjunction and the new numeric-only conjunction. Math::GMP::band()
should ignore the fifth argument (and the fourth, and the third).

2) How does the XSub silence the warning.

It doesn't. The warning happens in the calling scope, which is where
the bitwise feature is enabled, and it can be silenced by that scope's
warning flags.

-zefram

Oh, I see.
It's as simple as appending a "..." to the argument list.
At least that's all that Math::GMPz needs, and I presume the same would hold for Math::GMP.

The far more elaborate PDL patch that FC provided threw me a bit. But I can well believe that PDL, being what it is, would require a more elaborate fix.

Cheers,
Rob

p5pRT · 2017-12-20T04:42:32Z

From @cpansprout

On Mon, 18 Dec 2017 16:06:00 -0800, sisyphus wrote:

Oh, I see.
It's as simple as appending a "..." to the argument list.
At least that's all that Math::GMPz needs, and I presume the same
would hold for Math::GMP.

The far more elaborate PDL patch that FC provided threw me a bit. But
I can well believe that PDL, being what it is, would require a more
elaborate fix.

Cheers,
Rob

FYI, I have added an explanation to perlxs.pod in commit daf3bcf.

--

Father Chrysostomos

p5pRT closed this as completed Dec 17, 2017

p5pRT added Severity Low distro-Linux type-core labels Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

experimental bitwise feature breaks PDL's & operator #16307

experimental bitwise feature breaks PDL's & operator #16307

p5pRT commented Dec 15, 2017

p5pRT commented Dec 15, 2017

p5pRT commented Dec 17, 2017

p5pRT commented Dec 17, 2017

p5pRT commented Dec 17, 2017

p5pRT commented Dec 18, 2017

p5pRT commented Dec 18, 2017

p5pRT commented Dec 18, 2017

p5pRT commented Dec 19, 2017

p5pRT commented Dec 20, 2017

experimental bitwise feature breaks PDL's & operator #16307

experimental bitwise feature breaks PDL's & operator #16307

Comments

p5pRT commented Dec 15, 2017

p5pRT commented Dec 15, 2017

From @djerius

Created by @djerius

p5pRT commented Dec 17, 2017

From @cpansprout

p5pRT commented Dec 17, 2017

p5pRT commented Dec 17, 2017

p5pRT commented Dec 18, 2017

From @sisyphus

p5pRT commented Dec 18, 2017

From zefram@fysh.org

p5pRT commented Dec 18, 2017

From @djerius

p5pRT commented Dec 19, 2017

From @sisyphus

p5pRT commented Dec 20, 2017

From @cpansprout