From @mauke

Created by @mauke

% cat anon_attr.t
#!perl
use strict;
use warnings;

use Test​::More tests => 2;

my %last_attr_coderef;

sub MODIFY_CODE_ATTRIBUTES {
  diag "MODIFY_CODE_ATTRIBUTES(@​_)";
  my ($pkg, $code, $attr) = @​_;
  $last_attr_coderef{$attr} = $code;
  return;
}

sub foo : A {}
diag "&foo = " . \&foo;
is $last_attr_coderef{A}, \&foo;

*bar = sub : B {};
diag "&bar = " . \&bar;
is $last_attr_coderef{B}, \&bar;
__END__

% perl anon_attr.t
1..2
# MODIFY_CODE_ATTRIBUTES(main CODE(0x8bee368) A)
# MODIFY_CODE_ATTRIBUTES(main CODE(0x8bee468) B)
# &foo = CODE(0x8bee368)
ok 1
# &bar = CODE(0x8bee468)
ok 2

% PERLDB_OPTS=NonStop perl -d anon_attr.t
1..2
# MODIFY_CODE_ATTRIBUTES(main CODE(0x9640088) A)
# MODIFY_CODE_ATTRIBUTES(main CODE(0x96400e8) B)
# &foo = CODE(0x9640088)
ok 1
# &bar = CODE(0x970b698)
not ok 2
# Failed test at anon_attr.t line 22.
# got​: 'CODE(0x96400e8)'
# expected​: 'CODE(0x970b698)'
# Looks like you failed 1 test of 2.

It seems like attribute handlers receive the wrong coderef when running under
perl -d (but only for anonymous subs).

See https://rt.cpan.org/Public/Bug/Display.html?id=90126 for consequences of
this issue. Short version​: If you use Catalyst and use Method​::Signatures​::Simple
to declare your actions (e.g. via 'use syntax "method"'), everything will work
- until you try to use the debugger or profile your code, at which point
everything blows up because all of your URLs throw 404s.

Is this a core bug or am I overlooking something?

Flags:
    category=core
    severity=medium

This perlbug was built using Perl 5.12.1 - Thu Jun  3 20:09:15 CEST 2010
It is being executed now by  Perl 5.18.1 - Tue Aug 13 07:08:47 CEST 2013.

Site configuration information for perl 5.18.1:

Configured by mauke at Tue Aug 13 07:08:47 CEST 2013.

Summary of my perl5 (revision 5 version 18 subversion 1) configuration:
   
  Platform:
    osname=linux, osvers=3.5.7-gentoo, archname=i686-linux
    uname='linux nora 3.5.7-gentoo #5 preempt sat jan 26 16:46:10 cet 2013 i686 amd athlon(tm) 64 processor 3200+ authenticamd gnulinux '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -flto',
    cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.8.1', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags ='-O2 -flto -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib /usr/lib/../lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.15.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.15'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -flto -L/usr/local/lib -fstack-protector'

Locally applied patches:
    SAVEARGV0 - disable magic open in <ARGV>


@INC for perl 5.18.1:
    /home/mauke/usr/local/lib/perl5/site_perl/5.18.1/i686-linux
    /home/mauke/usr/local/lib/perl5/site_perl/5.18.1
    /home/mauke/usr/local/lib/perl5/5.18.1/i686-linux
    /home/mauke/usr/local/lib/perl5/5.18.1
    .


Environment for perl 5.18.1:
    HOME=/home/mauke
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=POSIX
    LD_LIBRARY_PATH=/home/mauke/usr/local/lib
    LOGDIR (unset)
    PATH=/home/mauke/usr/perlbrew/bin:/home/mauke/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.6.3:/opt/sun-jdk-1.4.2.13/bin:/opt/sun-jdk-1.4.2.13/jre/bin:/opt/sun-jdk-1.4.2.13/jre/javaws:/opt/dmd/bin:/usr/games/bin
    PERLBREW_BASHRC_VERSION=0.43
    PERLBREW_HOME=/home/mauke/.perlbrew
    PERLBREW_PATH=/home/mauke/usr/perlbrew/bin
    PERLBREW_ROOT=/home/mauke/usr/perlbrew
    PERLBREW_VERSION=0.27
    PERL_BADLANG (unset)
    PERL_UNICODE=SAL
    SHELL=/bin/bash

From PeterCMartini@GMail.com

On Sat, Nov 9, 2013 at 5​:35 PM, l.mai@​web.de <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by l.mai@​web.de
# Please include the string​: [perl #120500]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=120500 >

This is a bug report for perl from l.mai@​web.de,
generated with the help of perlbug 1.39 running under perl 5.18.1.

-----------------------------------------------------------------
[Please describe your issue here]

% cat anon_attr.t
#!perl
use strict;
use warnings;

use Test​::More tests => 2;

my %last_attr_coderef;

sub MODIFY_CODE_ATTRIBUTES {
diag "MODIFY_CODE_ATTRIBUTES(@​_)";
my ($pkg, $code, $attr) = @​_;
$last_attr_coderef{$attr} = $code;
return;
}

sub foo : A {}
diag "&foo = " . \&foo;
is $last_attr_coderef{A}, \&foo;

*bar = sub : B {};
diag "&bar = " . \&bar;
is $last_attr_coderef{B}, \&bar;
__END__

% perl anon_attr.t
1..2
# MODIFY_CODE_ATTRIBUTES(main CODE(0x8bee368) A)
# MODIFY_CODE_ATTRIBUTES(main CODE(0x8bee468) B)
# &foo = CODE(0x8bee368)
ok 1
# &bar = CODE(0x8bee468)
ok 2

% PERLDB_OPTS=NonStop perl -d anon_attr.t
1..2
# MODIFY_CODE_ATTRIBUTES(main CODE(0x9640088) A)
# MODIFY_CODE_ATTRIBUTES(main CODE(0x96400e8) B)
# &foo = CODE(0x9640088)
ok 1
# &bar = CODE(0x970b698)
not ok 2
# Failed test at anon_attr.t line 22.
# got​: 'CODE(0x96400e8)'
# expected​: 'CODE(0x970b698)'
# Looks like you failed 1 test of 2.

It seems like attribute handlers receive the wrong coderef when running under
perl -d (but only for anonymous subs).

See https://rt.cpan.org/Public/Bug/Display.html?id=90126 for consequences of
this issue. Short version​: If you use Catalyst and use Method​::Signatures​::Simple
to declare your actions (e.g. via 'use syntax "method"'), everything will work
- until you try to use the debugger or profile your code, at which point
everything blows up because all of your URLs throw 404s.

Is this a core bug or am I overlooking something?

MODIFY_CODE_ATTRIBUTES runs at compile time, so it will always see the
base code reference, while the $sub = sub {} is a runtime operation
and will get the cloned code reference *if a clone needed to be made*.
Under perl -d, all anonymous subs are turned into closures (cloned).

#!./perl

sub MODIFY_CODE_ATTRIBUTES {print "$_[2] $_[1]\n"; return;}
my $i = 1;
my $foo = sub : A {;};
print "A $foo\n";
my $bar = sub : B {eval "";};
print "B $bar\n";
my $baz = sub : C {$i;};
print "C $baz\n";

The two "A"s print out the same code ref, since there's nothing being
cloned over; the two "C"s print out different code refs, since it
closes over $i, and the two "B" print out different code refs, since
there's no way to determine if the code in the eval will reference an
outer lexical until its actually evaluated.

I'm out of time to dig into the cpan bug reports, but they're likely
just missing clone/copy handlers. For example, checkcall magic in the
core has a 'copy' handler explicitly to copy the call handler from the
base CV to the cloned copies.

The RT System itself - Status changed from 'new' to 'open'

From @mauke

On Sun Nov 10 07​:38​:21 2013, pcm wrote​:

MODIFY_CODE_ATTRIBUTES runs at compile time, so it will always see the
base code reference, while the $sub = sub {} is a runtime operation
and will get the cloned code reference *if a clone needed to be made*.
Under perl -d, all anonymous subs are turned into closures (cloned).

Oh, I see. In that case I don't understand why attributes work that way. How is the current behavior useful? If you put an attribute on an anon sub, the handler receives a coderef that can't be called ("Closure prototype called at $file line $line.") or stored for later (because it doesn't match the runtime refaddr of any real sub created from it). Why do we allow attribute handlers to see closure prototypes?

I would have expected to see attributes applied at runtime, to the cloned sub, not the prototype.

I'm out of time to dig into the cpan bug reports, but they're likely
just missing clone/copy handlers.

I'm not sure how that works. In a code attribute handler​: 1) how do I detect that the "sub" I get is actually just a closure prototype? 2) How do I attach a callback that notifies me when a real sub is instantiated? Does this involve XS magic?

[...time passes...]

#1 can be done in pure perl​:
  use B qw(svref_2object CVf_CLONE);
  my $is_proto = svref_2object($sub)->CvFLAGS & CVf_CLONE;

#2 is sort of doable with Variable​::Magic​:

#!perl
use v5.12;
use warnings;

use B qw(svref_2object CVf_CLONE);
use Variable​::Magic qw(wizard cast);

my $wizard; BEGIN {
  $wizard = wizard
  data => sub {
  my $victim = shift;
  [@​_]
  },
  copy => sub {
  my $oldsub = $_[0];
  my ($pkg, @​attrs) = @​{$_[1]};
  my $newsub = \&{$_[3]};
  MODIFY_CODE_ATTRIBUTES($pkg, $newsub, @​attrs);
  # XXX complain about unhandled attributes??
  },
  ;
}

sub MODIFY_CODE_ATTRIBUTES {
  my ($pkg, $sub, @​attrs) = @​_;
  my $cv = svref_2object $sub;
  my $is_proto = $cv->CvFLAGS & CVf_CLONE;
  if ($is_proto) {
  cast &$sub, $wizard, $pkg, @​attrs;
  return;
  }
  say "applying [@​attrs] to $sub";
  return;
}

sub foo : Zomg {}

my @​subs;
for my $i (1 .. 2) {
  push @​subs, sub : Hello(World) {
  print "[$i] @​_\n";
  };
}
__END__

% perl try.pl
applying [Zomg] to CODE(0xa03de98)
applying [Hello(World)] to CODE(0x9f981f8)
applying [Hello(World)] to CODE(0x9fa9d60)

... but this feels like a terrible hack for something I expect perl to do natively.

From @mauke

try.pl

From PeterCMartini@GMail.com

On Sun, Nov 10, 2013 at 12​:36 PM, l.mai@​web.de via RT
<perlbug-followup@​perl.org> wrote​:

I'm not sure how that works. In a code attribute handler​: 1) how do I detect that the "sub" I get is actually just a closure prototype? 2) How do I attach a callback that notifies me when a real sub is instantiated? Does this involve XS magic?

[...time passes...]

#1 can be done in pure perl​:
use B qw(svref_2object CVf_CLONE);
my $is_proto = svref_2object($sub)->CvFLAGS & CVf_CLONE;

#2 is sort of doable with Variable​::Magic​:

#!perl
use v5.12;
use warnings;

use B qw(svref_2object CVf_CLONE);
use Variable​::Magic qw(wizard cast);

my $wizard; BEGIN {
$wizard = wizard
data => sub {
my $victim = shift;
[@​_]
},
copy => sub {
my $oldsub = $_[0];
my ($pkg, @​attrs) = @​{$_[1]};
my $newsub = \&{$_[3]};
MODIFY_CODE_ATTRIBUTES($pkg, $newsub, @​attrs);
# XXX complain about unhandled attributes??
},
;
}

sub MODIFY_CODE_ATTRIBUTES {
my ($pkg, $sub, @​attrs) = @​_;
my $cv = svref_2object $sub;
my $is_proto = $cv->CvFLAGS & CVf_CLONE;
if ($is_proto) {
cast &$sub, $wizard, $pkg, @​attrs;
return;
}
say "applying [@​attrs] to $sub";
return;
}

sub foo : Zomg {}

my @​subs;
for my $i (1 .. 2) {
push @​subs, sub : Hello(World) {
print "[$i] @​_\n";
};
}
__END__

% perl try.pl
applying [Zomg] to CODE(0xa03de98)
applying [Hello(World)] to CODE(0x9f981f8)
applying [Hello(World)] to CODE(0x9fa9d60)

... but this feels like a terrible hack for something I expect perl to do natively.

I don't quite follow why you need to distinguish between the prototype
cv and the individual instances of it. Do you know if those modules
are attaching magic to the CV itself? The call checker API in core
uses magic to attach the call checker, and it specifically use the
'copy' member of the magic table to handle moving prototypes.

Right now, if a CV is cloned, the copy method in the magic vtbl is
called if it exists, and nothing is done otherwise. I wonder if its
too late to change the default (for CVs) to copy/refcnt inc by
default? Anything that already had a handler would be unaffected, the
callchecker would no longer need magic, and this case, if its using
magic but doesn't have a copy method at the moment, would either start
working, or blow up in a way that proves its a bad idea.

---
via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=120500

From @leonerd

On Mon, 11 Nov 2013 08​:20​:11 -0500
Peter Martini <petercmartini@​gmail.com> wrote​:

Right now, if a CV is cloned, the copy method in the magic vtbl is
called if it exists, and nothing is done otherwise. I wonder if its
too late to change the default (for CVs) to copy/refcnt inc by
default? Anything that already had a handler would be unaffected, the
callchecker would no longer need magic, and this case, if its using
magic but doesn't have a copy method at the moment, would either start
working, or blow up in a way that proves its a bad idea.

Without knowing in advance what the attribute handling function wanted
to do, you can't decide whether to run it just on the protosub, or just
on the cloned closures, or both. A more fine-grained interface is
required that can distinguish the two.

Note that there is a lot of scope for inventing such an interface;
perldoc attributes does point out​:

  WARNING​: the mechanisms described here are still experimental.
  Do not rely on the current implementation. In particular, there
  is no provision for applying package attributes to 'cloned'
  copies of subroutines used as closures.

--
Paul "LeoNerd" Evans

leonerd@​leonerd.org.uk
ICQ# 4135350 | Registered Linux# 179460
http​://www.leonerd.org.uk/

From @mauke

Am So 10. Nov 2013, 09​:36​:08, mauke- schrieb​:

On Sun Nov 10 07​:38​:21 2013, pcm wrote​:

MODIFY_CODE_ATTRIBUTES runs at compile time, so it will always see
the
base code reference, while the $sub = sub {} is a runtime operation
and will get the cloned code reference *if a clone needed to be
made*.
Under perl -d, all anonymous subs are turned into closures (cloned).

Oh, I see. In that case I don't understand why attributes work that
way. How is the current behavior useful? If you put an attribute on an
anon sub, the handler receives a coderef that can't be called
("Closure prototype called at $file line $line.") or stored for later
(because it doesn't match the runtime refaddr of any real sub created
from it). Why do we allow attribute handlers to see closure
prototypes?

I would have expected to see attributes applied at runtime, to the
cloned sub, not the prototype.

More elaborate braindump​:

So here's what I think about attributes. Semantically a thing X with an
attribute A is equivalent to a call to attributes->import(\X, A).

Examples​:
  sub foo :Hi {} # attributes->import(\&foo, 'Hi')
  my $bar :Hi; # attributes->import(\$bar, 'Hi')

The crucial question is​: When does the call to attributes->import happen? Turns
out that for 'sub foo' the call happens at compile time (when the sub is
parsed) and for 'my $bar' the call happens at runtime (each time the
declaration is executed). The latter makes sense because each execution creates
a new private variable $bar.

But what happens with anonymous subs? Like any anonymous constructor ([], {}),
'sub {...}' creates a new value each time it is executed (at least
conceptually, more on that later). But as for named subs, the attribute->import
call happens at compile time. So what the heck is passed to attribute->import
if the actual sub is only created at runtime?

Implementation details digression! When perl parses a 'sub { ... }', it stores
the body in a fake subroutine (also known as a "closure prototype" or
"protosub"). At runtime a "pad" (variable environment) is allocated and
combined with the body from the protosub to form an actual subroutine.

So​: What ends up being passed to attribute->import is the protosub (because
that's all we have at compile time). Protosubs are very weird beasts because
they're purely an implementation artifact that isn't exposed anywhere else. A
protosub looks like a normal coderef to Perl code, but it's completely useless
because 1) you can't call it (trying to call a protosub throws an exception)
and 2) you can't store its address for later comparisons because no real sub
will compare equal to it.

So why do anonymous subroutine attributes appear to work some/most of the time?
That's because of an interesting optimization done by perl. The rationale is​:
Because subs have no mutable structure (unlike arrays or hashes), they can
safely be shared. Thus 'sub { ... }' can be constant folded, replacing the
"create a sub" constructor op by a constant coderef.

This optimization doesn't apply to closures (subs that use variables from
surrounding scopes) because they do have mutable structure​:

  my $x = 0;
  my $f = sub { $x };
  say $f->(); # "0"
  $x++;
  say $f->(); # "1"

Similarly, functions that use string eval are treated like closures because
they may access unknown variables at runtime.

The constant folding described above is itself optimized​: It doesn't actually
allocate a new sub (as the runtime op would); instead it promotes the protosub
to a real sub and uses it directly. So in this case everything appears to work​:
attributes->import sees the same ref that runtime code would.

Digression, bad news #1​: The above optimization is actually invalid because all
subs have mutable structure, accessible via 'bless'​:

  % perl -wE 'for my $i (1, 2) { my $f = sub { $i }; bless $f, "hello" if $i == 1; say ref $f }'
  hello
  CODE

This is the normal case. $f is a closure, so everything works as expected ($f
is only blessed in the first iteration).

  % perl -wE 'for my $i (1, 2) { my $f = sub { $_ }; bless $f, "hello" if $i == 1; say ref $f }'
  hello
  hello

This is the broken optimization case. $f isn't a closure and thus remains
blessed because it's shared across all iterations.

More to the point, bad news #2​: Even for anonymous subs that aren't closures
there is a problem with attributes once you use 'perl -d', e.g. to run your
code in the debugger or to profile your code. -d seems to inhibit the "shared
anon coderef" optimization.

My proposed fix​: Make anon sub attributes apply at runtime.

  my $var :Hi;
  # desugars to
  my $var; attributes->import(\$var, 'Hi');

  my $f = sub :Hi {...};
  # should desugar to
  my $f = sub {...}; attributes->import($f, 'Hi');

This would break attributes that want to munge the sub's optree because those
attributes need to modify the protosub. If this is not acceptable, I propose
not passing the protosub in form of a coderef because it really isn't a coderef
and Perl code should probably not even see it.

From zefram@fysh.org

l.mai@​web.de via RT wrote​:

This optimization doesn't apply to closures (subs that use variables from
surrounding scopes) because they do have mutable structure​:

That's a bit mangled. Really, the optimisation doesn't apply to
closures because closures produce functionally different objects for
each evaluation of the sub{...} expression. They're visibly distinct
even without making use of mutability​:

  sub foo {
  my $x = $_[0];
  return sub { $x };
  };
  my $a = foo(0);
  my $b = foo(1);
  say $a->(); # 0
  say $b->(); # 1

Where the `closure' doesn't actually refer to any outer lexical variables,
each closure produced by the sub{...} expression would be functionally
identical. Hence the sub{...} can be constant-folded in that case,
to a non-closure sub.

Digression, bad news #1​: The above optimization is actually invalid because all
subs have mutable structure, accessible via 'bless'​:

Yes. This mutable aspect means that they can be functionally
distinguished even without anything based on the variables they close
over.

My proposed fix​: Make anon sub attributes apply at runtime.

What do you intend this to fix? It doesn't address the problem of the
broken optimisation. The issue with the optimisation is the question of
which objects are distinct, not of when a mutation gets applied. I think
you're imagining an implication, that you haven't actually stated, that
sub{...} would never be constant-folded, and so would always produce a
distinct subroutine object for each evaluation.

Dropping the optimisation would be a potentially significant penalty in
the fairly common case of using a repeatedly-executed sub{...} expression
to refer to a constant non-closure subroutine. It is possible to restore
the optimisation in an explicit fashion by Memoize​::Lift or equivalent
code transformation, but it wouldn't be popular.

This would break attributes that want to munge the sub's optree because those
attributes need to modify the protosub.

More precisely, it would break these attributes because they need to
munge the shared optree exactly once.

                                   If this is not acceptable\, I propose

not passing the protosub in form of a coderef because it really isn't a coderef

We don't have any better form for it. It has most of the attributes of
a subroutine (CvFLAGS et al), not just the ops. It's also established
behaviour that attributes get applied at the protosub stage.

I'd like to see some analysis of the prevalence of code attributes,
and how many uses would favour each possible arrangement.

-zefram

From perl5-porters@perl.org

Zefram wrote​:

I think
you're imagining an implication, that you haven't actually stated, that
sub{...} would never be constant-folded, and so would always produce a
distinct subroutine object for each evaluation.

Dropping the optimisation would be a potentially significant penalty in
the fairly common case of using a repeatedly-executed sub{...} expression
to refer to a constant non-closure subroutine.

The only part of sub cloning that is slower than, say, copying a
string is cloning the pad. If we could get subs to share protopads,
that may speed things up significantly.

This would break attributes that want to munge the sub's optree because those
attributes need to modify the protosub.

More precisely, it would break these attributes because they need to
munge the shared optree exactly once.

Exactly. The question here is whether attributes apply to the CV head
or to the OP body.

Lukas wants the former; you need the latter.

From zefram@fysh.org

Father Chrysostomos wrote​:

Exactly. The question here is whether attributes apply to the CV head
or to the OP body.

Lukas wants the former; you need the latter.

I'm not especially attached to my (one) op-munging attribute (in
Sub​::Filter). The op munging is useful, but I'm not a great fan of the
attribute mode of delivery, and I'm actually in the process of splitting
the involved modules on these lines. I'm OK if that attribute gets
broken in the service of a higher cause.

-zefram

From @leonerd

On Tue, 19 Nov 2013 12​:12​:52 +0000
Zefram <zefram@​fysh.org> wrote​:

I'd like to see some analysis of the prevalence of code attributes,
and how many uses would favour each possible arrangement.

I have a number of modules that do attributes on code, but all of the
uses of those have been applied to named subs, never anons/closures.

I too share Zefram's concern about breaking the use-case of applying
attributes that want to mutate the optree of a protosub. I suspect the
solution to this is to have a potentially two-phase approach with one
phase on the protosub at compiletime, and the other phase on the cloned
CODE ref at runtime.

--
Paul "LeoNerd" Evans

leonerd@​leonerd.org.uk
ICQ# 4135350 | Registered Linux# 179460
http​://www.leonerd.org.uk/