Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lexical subs #12238

Closed
p5pRT opened this issue Jul 1, 2012 · 150 comments
Closed

Lexical subs #12238

p5pRT opened this issue Jul 1, 2012 · 150 comments

Comments

@p5pRT
Copy link

p5pRT commented Jul 1, 2012

Migrated from rt.perl.org#113930 (status was 'resolved')

Searchable as RT113930$

@p5pRT
Copy link
Author

p5pRT commented Jul 1, 2012

From @cpansprout

There doesn’t appear to be a ticket for this already, so I’m making one. There is also another reason​: I think I know how to make it work.

The last time this came up, the two main issues were​:

1)

If ‘my sub’ follows the same scoping rules as existing my and sub declarations, then the lexical sub will not be visible inside itself, unless declared beforehand. But if ‘my’ creates a new pad entry, then this won’t work​:

my sub foo;
my sub foo { ... }

because the second one creates a new pad entry and does not modify the existing one.

2)

Should declarations be compile-time or run-time? If they are compile-time declarations, then how does one redefine a sub (c.f. problem 1); and how does a lexical sub close over variables? If it is a run-time thing, then it is radically different from ‘sub foo {...}’, so shouldn’t it look different?

These questions can be summarized as​: When does a lexical closure close over outer variables? What else has to happen to make it work?


I think the solution to #1 is to make ‘sub foo { ... }’ respect a lexical declaration that precedes it. Just as $x = 3 does not assign to the package variable $x after a my $x declaration, so ‘sub foo { ... }’ should not create a package sub after ‘my sub foo;’.

That in itself pretty much resolves #2, but we have (at least) two variants​:

• Lexical subs close over variables when the name comes into scope (which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since optimisations would make all three variants nearly identical in that regard.

For for loops we have to decide what this will do​:

for my $x (1..10) {
  my sub foo { push @​results, $x }
  foo();
}

The @​results will be 1..10, 1..10 and (1)x10, respectively for the three variants.

And what about this?

my sub foo;
for my $x (1..10) {
  sub foo { push @​results, $x }
  foo();
}

Results​: (undef)x10, 1..10, (1)x10

And this would produce the same​:

my $x;
my sub foo { push @​results, $x }
for $x (1..10) {
  foo();
}

In all these cases, the second variant seems more intuitive, at least to me. In fact, that’s similar to how format closures work. This prints the numbers from 1 to 10​:

my $x;
format =
@​<<<<<<
$x
.
for $x (1..10) {
  write;
}

If we introduce lexical aliases via \my $x = $y, we will need to figure this out for the sake of that, too.


Next issue​: What should state sub do? Should it behave the same way as my sub? Should it close over the variables the first time through the function? Should it be an error (currently it is not) until we decide?

I think it should be just a named package sub (‘Variable will not stay shared’ and all), but visible only in a lexical scope.


I think it is pretty obvious that our sub should create a lexical alias to a package sub, so that this will work​:

sub foo { warn "main" }
package bar;
foo(); # calls main​::foo


‘my format’ and ‘my package’ would be going a bit too far. :-)

@p5pRT
Copy link
Author

p5pRT commented Jul 1, 2012

From @cpansprout

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

Next issue​: What should state sub do? Should it behave the same way
as my sub? Should it close over the variables the first time
through the function? Should it be an error (currently it is not)
until we decide?

I think it should be just a named package sub (‘Variable will not stay
shared’ and all), but visible only in a lexical scope.

s/just/just like/

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 1, 2012

From [Unknown Contact. See original ticket]

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

Next issue​: What should state sub do? Should it behave the same way
as my sub? Should it close over the variables the first time
through the function? Should it be an error (currently it is not)
until we decide?

I think it should be just a named package sub (‘Variable will not stay
shared’ and all), but visible only in a lexical scope.

s/just/just like/

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 1, 2012

@cpansprout - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jul 1, 2012

From @kentfredric

On 2 July 2012 09​:23, Father Chrysostomos <perlbug-followup@​perl.org> wrote​:

for my $x (1..10) {
my sub foo { push @​results, $x }
foo();
}

The @​results will be 1..10, 1..10 and (1)x10, respectively for the three variants.

As far as what I expect would happen, I'd imagine, if I saw that code,
that it would do the same as​:

for my $x (1..10) {
  my $foo = sub { push @​results, $x }
  $foo();
}

And what about this?

my sub foo;
for my $x (1..10) {
sub foo { push @​results, $x }
foo();
}

If I saw that, I'd expect the same as​:

my $foo = sub {};
for my $x (1..10) {
  $foo = sub { push @​results, $x };
  $foo();
}

In both case, my brain sees "my sub foo" and thinks its just sugar for
"my $foo = sub "

However, if you were to use 'state foo' , the expectation I'd have is :

for my $x (1..10) {
  state sub foo { push @​results, $x }
  foo();
}

would be the same as​:

my $foo;
for my $x (1..10) {
  $foo //= sub { push @​results, $x }
  $foo();
}

That being, @​results would be (1)x10 ( Though, slightly unintuitive I feel )

I think people who are doing that though are doing it to be
"optimised" somehow, and would expect the behaviour of​:

my $foo;
my $x;
for $x (1..10) {
  $foo //= sub { push @​results, $x }
  $foo();
}

--
Kent

perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@​tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

@p5pRT
Copy link
Author

p5pRT commented Jul 5, 2012

From @cpansprout

newMYSUB is listed in the API. (It has an A in embed.fnc and shows up
in the undocumented list in perlapi.pod.)

This is its signature​:

#ifdef PERL_MAD
OP *
#else
void
#endif
Perl_newMYSUB(pTHX_ I32 floor, OP *o, OP *proto, OP *attrs, OP *block)

Implementing lexical subs within the constrains of the current
parameters and return value is not impossible, but hardly sane. It
would be easier to make newMYSUB a stub that dies, and create a
completely new function.

Instead, can I just change this? Nothing uses it. All it does
currently is die anyway.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 5, 2012

From [Unknown Contact. See original ticket]

newMYSUB is listed in the API. (It has an A in embed.fnc and shows up
in the undocumented list in perlapi.pod.)

This is its signature​:

#ifdef PERL_MAD
OP *
#else
void
#endif
Perl_newMYSUB(pTHX_ I32 floor, OP *o, OP *proto, OP *attrs, OP *block)

Implementing lexical subs within the constrains of the current
parameters and return value is not impossible, but hardly sane. It
would be easier to make newMYSUB a stub that dies, and create a
completely new function.

Instead, can I just change this? Nothing uses it. All it does
currently is die anyway.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 5, 2012

From @ikegami

On Thu, Jul 5, 2012 at 5​:46 PM, Father Chrysostomos via RT <
perlbug-comment@​perl.org> wrote​:

newMYSUB is listed in the API. (It has an A in embed.fnc and shows up
in the undocumented list in perlapi.pod.)

This is its signature​:

#ifdef PERL_MAD
OP *
#else
void
#endif
Perl_newMYSUB(pTHX_ I32 floor, OP *o, OP *proto, OP *attrs, OP *block)

Implementing lexical subs within the constrains of the current
parameters and return value is not impossible, but hardly sane. It
would be easier to make newMYSUB a stub that dies, and create a
completely new function.

Instead, can I just change this? Nothing uses it. All it does
currently is die anyway.

I'm particularly interested in the ability of "use Module;" to create a
lexical sub in the using scope. I looked at C<< newMYSUB >>, but I ended up
using Lexical​::Sub for Syntax​::Feature​::QwComments and :​:Loop. So if your
new interface makes that easy, I like it :D

- Eric

@p5pRT
Copy link
Author

p5pRT commented Jul 5, 2012

From @cpansprout

On Thu Jul 05 15​:11​:55 2012, ikegami@​adaelis.com wrote​:

On Thu, Jul 5, 2012 at 5​:46 PM, Father Chrysostomos via RT <
perlbug-comment@​perl.org> wrote​:

newMYSUB is listed in the API. (It has an A in embed.fnc and shows up
in the undocumented list in perlapi.pod.)

This is its signature​:

#ifdef PERL_MAD
OP *
#else
void
#endif
Perl_newMYSUB(pTHX_ I32 floor, OP *o, OP *proto, OP *attrs, OP *block)

Implementing lexical subs within the constrains of the current
parameters and return value is not impossible, but hardly sane. It
would be easier to make newMYSUB a stub that dies, and create a
completely new function.

Instead, can I just change this? Nothing uses it. All it does
currently is die anyway.

I'm particularly interested in the ability of "use Module;" to create a
lexical sub in the using scope. I looked at C<< newMYSUB >>, but I
ended up
using Lexical​::Sub for Syntax​::Feature​::QwComments and :​:Loop. So if your
new interface makes that easy, I like it :D

If you already have a compiled CV, it might be possible using
PAD_SETSV(pad_add_name_pv("&foo", 0, 0, 0), cv). Currently in bleadperl
you can do that (untested), but ‘foo’ in the source code won’t look it up.

My changes should make ‘foo’ generate a padcv op (nonexistent in blead)
in that case.

newMYSUB is probably not what you want, as it is analogous to newSUB​:
It takes PL_compcv and stuffs it in the slot indicated by the second
argument, attaching the passed-in op tree, and popping the stack back to
what it was before start_subparse() (assuming you passed the return
value of start_subparse() as the first argument to newSUB).

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 5, 2012

From [Unknown Contact. See original ticket]

On Thu Jul 05 15​:11​:55 2012, ikegami@​adaelis.com wrote​:

On Thu, Jul 5, 2012 at 5​:46 PM, Father Chrysostomos via RT <
perlbug-comment@​perl.org> wrote​:

newMYSUB is listed in the API. (It has an A in embed.fnc and shows up
in the undocumented list in perlapi.pod.)

This is its signature​:

#ifdef PERL_MAD
OP *
#else
void
#endif
Perl_newMYSUB(pTHX_ I32 floor, OP *o, OP *proto, OP *attrs, OP *block)

Implementing lexical subs within the constrains of the current
parameters and return value is not impossible, but hardly sane. It
would be easier to make newMYSUB a stub that dies, and create a
completely new function.

Instead, can I just change this? Nothing uses it. All it does
currently is die anyway.

I'm particularly interested in the ability of "use Module;" to create a
lexical sub in the using scope. I looked at C<< newMYSUB >>, but I
ended up
using Lexical​::Sub for Syntax​::Feature​::QwComments and :​:Loop. So if your
new interface makes that easy, I like it :D

If you already have a compiled CV, it might be possible using
PAD_SETSV(pad_add_name_pv("&foo", 0, 0, 0), cv). Currently in bleadperl
you can do that (untested), but ‘foo’ in the source code won’t look it up.

My changes should make ‘foo’ generate a padcv op (nonexistent in blead)
in that case.

newMYSUB is probably not what you want, as it is analogous to newSUB​:
It takes PL_compcv and stuffs it in the slot indicated by the second
argument, attaching the passed-in op tree, and popping the stack back to
what it was before start_subparse() (assuming you passed the return
value of start_subparse() as the first argument to newSUB).

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 6, 2012

From @doy

On Thu, Jul 05, 2012 at 02​:46​:38PM -0700, Father Chrysostomos via RT wrote​:

newMYSUB is listed in the API. (It has an A in embed.fnc and shows up
in the undocumented list in perlapi.pod.)

This is its signature​:

#ifdef PERL_MAD
OP *
#else
void
#endif
Perl_newMYSUB(pTHX_ I32 floor, OP *o, OP *proto, OP *attrs, OP *block)

Implementing lexical subs within the constrains of the current
parameters and return value is not impossible, but hardly sane. It
would be easier to make newMYSUB a stub that dies, and create a
completely new function.

Instead, can I just change this? Nothing uses it. All it does
currently is die anyway.

If all it does is die, clearly nobody can be using this for any real
purpose. I'd say go ahead and change it.

-doy

@p5pRT
Copy link
Author

p5pRT commented Jul 6, 2012

From @ikegami

On Thu, Jul 5, 2012 at 6​:59 PM, Father Chrysostomos via RT <
perlbug-comment@​perl.org> wrote​:

I'm particularly interested in the ability of "use Module;" to create a
lexical sub in the using scope. I looked at C<< newMYSUB >>, but I
ended up
using Lexical​::Sub for Syntax​::Feature​::QwComments and :​:Loop. So if your
new interface makes that easy, I like it :D

If you already have a compiled CV, it might be possible using
PAD_SETSV(pad_add_name_pv("&foo", 0, 0, 0), cv). Currently in bleadperl
you can do that (untested), but ‘foo’ in the source code won’t look it up.

No, I did not have an already compiled CV. I needed a stub to pass to
cv_set_call_parser and cv_set_call_checker In fact, I found I just needed
to *declare* the sub, so I ended up simply using

CV* const qwcv =
get_cvn_flags("Syntax​::Feature​::QwComments​::replacement_qw", 43, GV_ADD);

If you already have a compiled CV, it might be possible using

PAD_SETSV(pad_add_name_pv("&foo", 0, 0, 0), cv). Currently in bleadperl
you can do that (untested), but ‘foo’ in the source code won’t look it up.

Are you planning on having Perl look there?

- Eric

@p5pRT
Copy link
Author

p5pRT commented Jul 6, 2012

From @cpansprout

On Thu Jul 05 20​:56​:18 2012, ikegami@​adaelis.com wrote​:

On Thu, Jul 5, 2012 at 6​:59 PM, Father Chrysostomos via RT <
perlbug-comment@​perl.org> wrote​:

I'm particularly interested in the ability of "use Module;" to
create a
lexical sub in the using scope. I looked at C<< newMYSUB >>, but I
ended up
using Lexical​::Sub for Syntax​::Feature​::QwComments and :​:Loop. So
if your
new interface makes that easy, I like it :D

If you already have a compiled CV, it might be possible using
PAD_SETSV(pad_add_name_pv("&foo", 0, 0, 0), cv).

No, I did not have an already compiled CV. I needed a stub to pass to
cv_set_call_parser and cv_set_call_checker In fact, I found I just
needed
to *declare* the sub, so I ended up simply using

CV* const qwcv =
get_cvn_flags("Syntax​::Feature​::QwComments​::replacement_qw", 43,
GV_ADD);

I misspoke. I meant ‘your own CV’, as opposed to PL_compcv.

Currently in bleadperl
you can do that (untested), but ‘foo’ in the source code won’t look
it up.

Are you planning on having Perl look there?

Yes. On the sprout/lexsub branch it already does, but only works for
our subs.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 6, 2012

From [Unknown Contact. See original ticket]

On Thu Jul 05 20​:56​:18 2012, ikegami@​adaelis.com wrote​:

On Thu, Jul 5, 2012 at 6​:59 PM, Father Chrysostomos via RT <
perlbug-comment@​perl.org> wrote​:

I'm particularly interested in the ability of "use Module;" to
create a
lexical sub in the using scope. I looked at C<< newMYSUB >>, but I
ended up
using Lexical​::Sub for Syntax​::Feature​::QwComments and :​:Loop. So
if your
new interface makes that easy, I like it :D

If you already have a compiled CV, it might be possible using
PAD_SETSV(pad_add_name_pv("&foo", 0, 0, 0), cv).

No, I did not have an already compiled CV. I needed a stub to pass to
cv_set_call_parser and cv_set_call_checker In fact, I found I just
needed
to *declare* the sub, so I ended up simply using

CV* const qwcv =
get_cvn_flags("Syntax​::Feature​::QwComments​::replacement_qw", 43,
GV_ADD);

I misspoke. I meant ‘your own CV’, as opposed to PL_compcv.

Currently in bleadperl
you can do that (untested), but ‘foo’ in the source code won’t look
it up.

Are you planning on having Perl look there?

Yes. On the sprout/lexsub branch it already does, but only works for
our subs.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From @cpansprout

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

  push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also completely
undocumented.) It means that ‘state $x’ might only create one variable,
or might create more than one, depending on what type of sub it is
defined in.

It also creates yet another set of scoping rules for people to remember.
Flip-flops are shared between clones, so why not state variables?

The problem it raises for lexical subs is that it is not at all clear
when my sub foo { state $x } will create a new $x.

If state variables were intended to replace this sort of thing​:

{
  my $x;
  sub fooer {
  sub { ++$x; foo($x); }
  }
}

then they have failed to do so.

Is this something we can fix, or will it break too many things?

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From [Unknown Contact. See original ticket]

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

  push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also completely
undocumented.) It means that ‘state $x’ might only create one variable,
or might create more than one, depending on what type of sub it is
defined in.

It also creates yet another set of scoping rules for people to remember.
Flip-flops are shared between clones, so why not state variables?

The problem it raises for lexical subs is that it is not at all clear
when my sub foo { state $x } will create a new $x.

If state variables were intended to replace this sort of thing​:

{
  my $x;
  sub fooer {
  sub { ++$x; foo($x); }
  }
}

then they have failed to do so.

Is this something we can fix, or will it break too many things?

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From perl-diddler@tlinx.org

` Father Chrysostomos via RT wrote​:

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

push @​subs, sub { state $x } for 1..10;

That hurts.

I would have thought it would have declared it once like a 'my' and
named sub...

Erg......Honestly, since 'my' already isn't persistent across such, is
it possible to
fix state?

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also completely
undocumented.) It means that ‘state $x’ might only create one variable,
or might create more than one, depending on what type of sub it is
defined in.

It also creates yet another set of scoping rules for people to remember.
Flip-flops are shared between clones, so why not state variables?

The problem it raises for lexical subs is that it is not at all clear
when my sub foo { state $x } will create a new $x.

If state variables were intended to replace this sort of thing​:

{
my $x;
sub fooer {
sub { ++$x; foo($x); }
}
}

then they have failed to do so.

Is this something we can fix, or will it break too many things?

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From @cpansprout

On Sat Jul 07 11​:35​:48 2012, sprout wrote​:

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also completely
undocumented.) It means that ‘state $x’ might only create one variable,
or might create more than one, depending on what type of sub it is
defined in.

It also creates yet another set of scoping rules for people to remember.
Flip-flops are shared between clones, so why not state variables?

The problem it raises for lexical subs is that it is not at all clear
when my sub foo { state $x } will create a new $x.

If state variables were intended to replace this sort of thing​:

{
my $x;
sub fooer {
sub { ++$x; foo($x); }
}
}

then they have failed to do so.

Is this something we can fix, or will it break too many things?

In fact, they have been this way since they were added in commit
952306a, at least for closures. The code that causes state vars to
be created anew was not touched by that patch, so it seems like an
oversight. Other anonymous subs started recreating state vars in commit
a74073a, sixteen months later, which I think was for consistency’s sake.

...

And I just found this message, which shows that it was very deliberate​:

http​://www.nntp.perl.org/group/perl.perl5.porters/;msgid=20070906085723.GA30617@​iabyn.com

So now I’m stuck on how to implement my sub foo { state $x }.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From [Unknown Contact. See original ticket]

On Sat Jul 07 11​:35​:48 2012, sprout wrote​:

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also completely
undocumented.) It means that ‘state $x’ might only create one variable,
or might create more than one, depending on what type of sub it is
defined in.

It also creates yet another set of scoping rules for people to remember.
Flip-flops are shared between clones, so why not state variables?

The problem it raises for lexical subs is that it is not at all clear
when my sub foo { state $x } will create a new $x.

If state variables were intended to replace this sort of thing​:

{
my $x;
sub fooer {
sub { ++$x; foo($x); }
}
}

then they have failed to do so.

Is this something we can fix, or will it break too many things?

In fact, they have been this way since they were added in commit
952306a, at least for closures. The code that causes state vars to
be created anew was not touched by that patch, so it seems like an
oversight. Other anonymous subs started recreating state vars in commit
a74073a, sixteen months later, which I think was for consistency’s sake.

...

And I just found this message, which shows that it was very deliberate​:

http​://www.nntp.perl.org/group/perl.perl5.porters/;msgid=20070906085723.GA30617@​iabyn.com

So now I’m stuck on how to implement my sub foo { state $x }.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From tchrist@perl.com

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
  on Sat, 07 Jul 2012 11​:35​:49 PDT​:

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also completely
undocumented.)

Not so. Camel v4, chapter 7, page 324​:

  Finally, when we say that a state variable is initialized only once, we don’t mean
  to imply that state variables in separate closures are the same variables. They
  aren’t, so each gets its own initialization. This is how state variables differ from
  static variables in other languages.

Larry was quite certain about that part, because he wanted to make sure
people understood how it worked, and that it did so by deliberate intent​:
each closure is meant to get *its own* copy of a state variable.

It means that ‘state $x’ might only create one variable, or might
create more than one, depending on what type of sub it is defined in.

True.

It also creates yet another set of scoping rules for people to remember.
Flip-flops are shared between clones, so why not state variables?

Because it was designed to work this way.

The problem it raises for lexical subs is that it is not at all clear
when my sub foo { state $x } will create a new $x.

If we view C<my sub> as a declaration, than I cannot see why there
would ever be more than one of them, and therefore, more than one $x.

Are you worried about this situation?

  sub outer {
  my @​subs;
  for $i (1 .. 10) {
  my sub inner {
  state $x = rand();
  return [ $i => $x ];
  }
  push @​subs, \&inner;
  }
  return @​subs;
  }

The important question here is whether all those $x variables have
the same random number, or whether they have different ones.

And no, I don't know what the right answer is. I do agree that
is the right question, though. :)

If state variables were intended to replace this sort of thing​:

{
my $x;
sub fooer {
sub { ++$x; foo($x); }
}
}

then they have failed to do so.

I don't believe they were.

Is this something we can fix, or will it break too many things?

I'm not sure "fix" is the operative term here, considering that
it would certainly break existing code.

  my @​subs;
  for my $decade (0 .. 10) {
  push @​subs, sub {
  state $counter = 10 * $decade;
  return $counter++ % 10;
  };
  }

There's also the argument that breaking this changes the
designed intent of state. And the documentation.

Can't see a good argument for all that.

Nonetheless, I can see your conundrum. I don't suppose you
have a current rakudo/perl6 and could check out how/whether
it works there? S04 has this​:

  http​://perlcabal.org/syn/S04.html

  There is a new state declarator that introduces a lexically scoped
  variable like my does, but with a lifetime that persists for the life
  of the closure, so that it keeps its value from the end of one call to
  the beginning of the next. Separate clones of the closure get separate
  state variables. However, recursive calls to the same clone use the
  same state variable.

And later on​:

  The semantics of INIT and START are not equivalent to each other in the
  case of cloned closures. An INIT only runs once for all copies of a cloned
  closure. A START runs separately for each clone, so separate clones can
  keep separate state variables​:

  our $i = 0;
  ...
  $func = { state $x will start { $x = $i++ }; dostuff($i) };

  But state automatically applies "start" semantics to any initializer, so this also works​:

  $func = { state $x = $i++; dostuff($i) }

  Each subsequent clone gets an initial state that is one higher than the
  previous, and each clone maintains its own state of $x, because that's
  what state variables do.

And later on there is this, which is interesting but not completely
revealing, since it deals with a my not a state​:

  Lexical names do not share this problem, since the symbol goes out of
  scope synchronously with its usage. Unlike global subs, they do not need a
  compile-time binding, but like global subs, they perform a binding to the
  lexical symbol at clone time (again, conceptually at the entry to the
  outer lexical scope, but possibly deferred.)

  sub foo {
  # conceptual cloning happens to both blocks below
  my $x = 1;
  my sub bar { print $x } # already conceptually cloned, but can be lazily deferred
  my &baz := { bar(); print $x }; # block is cloned immediately, forcing cloning of bar
  my $code = &bar; # this would also force bar to be cloned
  return &baz;
  }

There may also be applicable points in S06, since you can my subs
in perl6, and in fact, the default is my sub not our sub​:

  http​://perlcabal.org/syn/S06.html#Named_subroutines

Hm, have you thought about our sub? I think it is the same in perl6
as regular perl5 subs are — that is, ones interred in the package symbol
table — but I might be wrong.

--tom

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From @cpansprout

On Sat Jul 07 12​:53​:32 2012, tom christiansen wrote​:

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
on Sat, 07 Jul 2012 11​:35​:49 PDT​:

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or
called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also
completely
undocumented.)

Not so. Camel v4, chapter 7, page 324​:

Finally\, when we say that a state variable is initialized only

once, we don’t mean
to imply that state variables in separate closures are the same
variables. They
aren’t, so each gets its own initialization. This is how state
variables differ from
static variables in other languages.

Larry was quite certain about that part, because he wanted to make
sure
people understood how it worked, and that it did so by deliberate
intent​:
each closure is meant to get *its own* copy of a state variable.

Thank you.

Could you write a patch to perlsub then? :-)

What it currently says is terribly vague. In fact, I would go so far as
to say that it doesn’t even state that ‘state’ creates persistent
variables. It begins by saying that ‘state’ can be used instead of ‘my’
to declare a variable, and then goes straight into examples. Examples
are supposed to supplement, not supplant, descriptions.

It means that ‘state $x’ might only create one variable, or might
create more than one, depending on what type of sub it is defined
in.

True.

It also creates yet another set of scoping rules for people to
remember.
Flip-flops are shared between clones, so why not state variables?

Because it was designed to work this way.

The problem it raises for lexical subs is that it is not at all
clear
when my sub foo { state $x } will create a new $x.

If we view C<my sub> as a declaration, than I cannot see why there
would ever be more than one of them, and therefore, more than one $x.

Are you worried about this situation?

sub outer \{
    my @&#8203;subs;
    for $i \(1 \.\. 10\) \{
        my sub inner \{
            state $x = rand\(\);
            return \[ $i => $x \];
        \}
        push @&#8203;subs\, \\&inner;
    \}
    return @&#8203;subs;
\}

The important question here is whether all those $x variables have
the same random number, or whether they have different ones.

And no, I don't know what the right answer is. I do agree that
is the right question, though. :)

I think I have the right answer now. If we document that only
*anonymous* subroutines get their own copies of state variables when
cloned, then my subs (personal subs? idiotic subs?) share them.

In the example you gave, those $x variables would all have the same
random number.

If whether \&inner will clone the sub or not is supposed to be a matter
of optimisation, that’s the only way it can work.

There's also the argument that breaking this changes the
designed intent of state. And the documentation.

Maybe the Camel, but certainly not perlsub. :-)

And later on there is this, which is interesting but not completely
revealing, since it deals with a my not a state​:

Lexical names do not share this problem\, since the symbol goes out

of
scope synchronously with its usage. Unlike global subs, they do
not need a
compile-time binding, but like global subs, they perform a binding
to the
lexical symbol at clone time (again, conceptually at the entry to
the
outer lexical scope, but possibly deferred.)

    sub foo \{
        \# conceptual cloning happens to both blocks below
        my $x = 1;
        my sub bar \{ print $x \}         \# already conceptually

cloned, but can be lazily deferred
my &baz := { bar(); print $x }; # block is cloned
immediately, forcing cloning of bar
my $code = &bar; # this would also force
bar to be cloned
return &baz;
}

That is interesting, since it is very similar to what I came up with on
my own.

If a sub is conceptually cloned when the block enters, does that mean
that my $code = &bar (\&bar in p5) twice in a row should produce the
same value?

How does Perl 6 deal with my subs in for loops?

There may also be applicable points in S06, since you can my subs
in perl6, and in fact, the default is my sub not our sub​:

http&#8203;://perlcabal\.org/syn/S06\.html\#Named\_subroutines

Hm, have you thought about our sub?

I’ve already implemented it. :-) See the sprout/lexsub branch.

It is analogous to our $var, in that the declaration persists across
package declarations.

$ ./perl -Ilib -e 'our sub foo { warn 42 } package bar; foo()'
42 at -e line 1.

$ ./perl -Ilib -MO=Concise -e 'package a; our sub foo { warn 42 }
package bar; foo()'
6 <@​> leave[1 ref] vKP/REFC ->(end)
1 <0> enter ->2
2 <;> nextstate(bar 3 -e​:1) v​:{ ->3
5 <1> entersub[t2] vKS/TARG ->6
- <1> ex-list K ->5
3 <0> pushmark s ->4
- <1> ex-rv2cv sK ->-
4 <$> gv(*a​::foo) s ->5
-e syntax OK

I think it is the same in perl6
as regular perl5 subs are — that is, ones interred in the package
symbol
table — but I might be wrong.

Whether it is or no, I wot not. I have not followed Perl 6 developement
closely at all. If it is different in Perl 6, I do not advocate
adopting ‘our sub’ from Perl 6, since ‘our’ already works a certain way
in Perl 5, so ‘our sub’ should follow that.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2012

From [Unknown Contact. See original ticket]

On Sat Jul 07 12​:53​:32 2012, tom christiansen wrote​:

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
on Sat, 07 Jul 2012 11​:35​:49 PDT​:

On Sun Jul 01 14​:23​:26 2012, sprout wrote​:

...[W]e have (at least) two
variants​:

• Lexical subs close over variables when the name comes into scope
(which could happen multiple times with loops).
• Lexical subs close over variables when referenced or called.
• Lexical subs close over variables when *first* referenced or
called.

For the most part, there is no observable difference here, until it
comes to referential identity and for loops.

I don’t think we have to worry about referential identity, since
optimisations would make all three variants nearly identical in
that regard.

Actually we do.

State variables are not persistent across anonymous subroutines.

push @​subs, sub { state $x } for 1..10;

will give you ten subroutines, each with a different $x.

I find that to be completely counterintuitive. (It is also
completely
undocumented.)

Not so. Camel v4, chapter 7, page 324​:

Finally\, when we say that a state variable is initialized only

once, we don’t mean
to imply that state variables in separate closures are the same
variables. They
aren’t, so each gets its own initialization. This is how state
variables differ from
static variables in other languages.

Larry was quite certain about that part, because he wanted to make
sure
people understood how it worked, and that it did so by deliberate
intent​:
each closure is meant to get *its own* copy of a state variable.

Thank you.

Could you write a patch to perlsub then? :-)

What it currently says is terribly vague. In fact, I would go so far as
to say that it doesn’t even state that ‘state’ creates persistent
variables. It begins by saying that ‘state’ can be used instead of ‘my’
to declare a variable, and then goes straight into examples. Examples
are supposed to supplement, not supplant, descriptions.

It means that ‘state $x’ might only create one variable, or might
create more than one, depending on what type of sub it is defined
in.

True.

It also creates yet another set of scoping rules for people to
remember.
Flip-flops are shared between clones, so why not state variables?

Because it was designed to work this way.

The problem it raises for lexical subs is that it is not at all
clear
when my sub foo { state $x } will create a new $x.

If we view C<my sub> as a declaration, than I cannot see why there
would ever be more than one of them, and therefore, more than one $x.

Are you worried about this situation?

sub outer \{
    my @&#8203;subs;
    for $i \(1 \.\. 10\) \{
        my sub inner \{
            state $x = rand\(\);
            return \[ $i => $x \];
        \}
        push @&#8203;subs\, \\&inner;
    \}
    return @&#8203;subs;
\}

The important question here is whether all those $x variables have
the same random number, or whether they have different ones.

And no, I don't know what the right answer is. I do agree that
is the right question, though. :)

I think I have the right answer now. If we document that only
*anonymous* subroutines get their own copies of state variables when
cloned, then my subs (personal subs? idiotic subs?) share them.

In the example you gave, those $x variables would all have the same
random number.

If whether \&inner will clone the sub or not is supposed to be a matter
of optimisation, that’s the only way it can work.

There's also the argument that breaking this changes the
designed intent of state. And the documentation.

Maybe the Camel, but certainly not perlsub. :-)

And later on there is this, which is interesting but not completely
revealing, since it deals with a my not a state​:

Lexical names do not share this problem\, since the symbol goes out

of
scope synchronously with its usage. Unlike global subs, they do
not need a
compile-time binding, but like global subs, they perform a binding
to the
lexical symbol at clone time (again, conceptually at the entry to
the
outer lexical scope, but possibly deferred.)

    sub foo \{
        \# conceptual cloning happens to both blocks below
        my $x = 1;
        my sub bar \{ print $x \}         \# already conceptually

cloned, but can be lazily deferred
my &baz := { bar(); print $x }; # block is cloned
immediately, forcing cloning of bar
my $code = &bar; # this would also force
bar to be cloned
return &baz;
}

That is interesting, since it is very similar to what I came up with on
my own.

If a sub is conceptually cloned when the block enters, does that mean
that my $code = &bar (\&bar in p5) twice in a row should produce the
same value?

How does Perl 6 deal with my subs in for loops?

There may also be applicable points in S06, since you can my subs
in perl6, and in fact, the default is my sub not our sub​:

http&#8203;://perlcabal\.org/syn/S06\.html\#Named\_subroutines

Hm, have you thought about our sub?

I’ve already implemented it. :-) See the sprout/lexsub branch.

It is analogous to our $var, in that the declaration persists across
package declarations.

$ ./perl -Ilib -e 'our sub foo { warn 42 } package bar; foo()'
42 at -e line 1.

$ ./perl -Ilib -MO=Concise -e 'package a; our sub foo { warn 42 }
package bar; foo()'
6 <@​> leave[1 ref] vKP/REFC ->(end)
1 <0> enter ->2
2 <;> nextstate(bar 3 -e​:1) v​:{ ->3
5 <1> entersub[t2] vKS/TARG ->6
- <1> ex-list K ->5
3 <0> pushmark s ->4
- <1> ex-rv2cv sK ->-
4 <$> gv(*a​::foo) s ->5
-e syntax OK

I think it is the same in perl6
as regular perl5 subs are — that is, ones interred in the package
symbol
table — but I might be wrong.

Whether it is or no, I wot not. I have not followed Perl 6 developement
closely at all. If it is different in Perl 6, I do not advocate
adopting ‘our sub’ from Perl 6, since ‘our’ already works a certain way
in Perl 5, so ‘our sub’ should follow that.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From @cpansprout

I’m forwarding this to the Perl 6 language list, so see if I can find an
answer there.

[This conversation is about how lexical subs should be implemented in
Perl 5. What Perl 6 does may help in determining how to iron out the
edge cases.]

On Sat Jul 07 13​:23​:17 2012, sprout wrote​:

On Sat Jul 07 12​:53​:32 2012, tom christiansen wrote​:

Are you worried about this situation?

sub outer \{
    my @&#8203;subs;
    for $i \(1 \.\. 10\) \{
        my sub inner \{
            state $x = rand\(\);
            return \[ $i => $x \];
        \}
        push @&#8203;subs\, \\&inner;
    \}
    return @&#8203;subs;
\}

The important question here is whether all those $x variables have
the same random number, or whether they have different ones.

And no, I don't know what the right answer is. I do agree that
is the right question, though. :)

I think I have the right answer now. If we document that only
*anonymous* subroutines get their own copies of state variables when
cloned, then my subs (personal subs? idiotic subs?) share them.

In the example you gave, those $x variables would all have the same
random number.

If whether \&inner will clone the sub or not is supposed to be a matter
of optimisation, that’s the only way it can work.
...

And later on there is this, which is interesting but not completely
revealing, since it deals with a my not a state​:

Lexical names do not share this problem\, since the symbol goes out

of
scope synchronously with its usage. Unlike global subs, they do
not need a
compile-time binding, but like global subs, they perform a binding
to the
lexical symbol at clone time (again, conceptually at the entry to
the
outer lexical scope, but possibly deferred.)

    sub foo \{
        \# conceptual cloning happens to both blocks below
        my $x = 1;
        my sub bar \{ print $x \}         \# already conceptually

cloned, but can be lazily deferred
my &baz := { bar(); print $x }; # block is cloned
immediately, forcing cloning of bar
my $code = &bar; # this would also force
bar to be cloned
return &baz;
}

That is interesting, since it is very similar to what I came up with on
my own.

If a sub is conceptually cloned when the block enters, does that mean
that my $code = &bar (\&bar in p5) twice in a row should produce the
same value?

How does Perl 6 deal with my subs in for loops?

This question might be more appropriate​: In this example, which @​a does
the bar subroutine see (in Perl 6)?

  sub foo {
  my @​a = (1,2,3);
  my sub bar { say @​a };
  @​a := [4,5,6];
  bar();
  }

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From [Unknown Contact. See original ticket]

I’m forwarding this to the Perl 6 language list, so see if I can find an
answer there.

[This conversation is about how lexical subs should be implemented in
Perl 5. What Perl 6 does may help in determining how to iron out the
edge cases.]

On Sat Jul 07 13​:23​:17 2012, sprout wrote​:

On Sat Jul 07 12​:53​:32 2012, tom christiansen wrote​:

Are you worried about this situation?

sub outer \{
    my @&#8203;subs;
    for $i \(1 \.\. 10\) \{
        my sub inner \{
            state $x = rand\(\);
            return \[ $i => $x \];
        \}
        push @&#8203;subs\, \\&inner;
    \}
    return @&#8203;subs;
\}

The important question here is whether all those $x variables have
the same random number, or whether they have different ones.

And no, I don't know what the right answer is. I do agree that
is the right question, though. :)

I think I have the right answer now. If we document that only
*anonymous* subroutines get their own copies of state variables when
cloned, then my subs (personal subs? idiotic subs?) share them.

In the example you gave, those $x variables would all have the same
random number.

If whether \&inner will clone the sub or not is supposed to be a matter
of optimisation, that’s the only way it can work.
...

And later on there is this, which is interesting but not completely
revealing, since it deals with a my not a state​:

Lexical names do not share this problem\, since the symbol goes out

of
scope synchronously with its usage. Unlike global subs, they do
not need a
compile-time binding, but like global subs, they perform a binding
to the
lexical symbol at clone time (again, conceptually at the entry to
the
outer lexical scope, but possibly deferred.)

    sub foo \{
        \# conceptual cloning happens to both blocks below
        my $x = 1;
        my sub bar \{ print $x \}         \# already conceptually

cloned, but can be lazily deferred
my &baz := { bar(); print $x }; # block is cloned
immediately, forcing cloning of bar
my $code = &bar; # this would also force
bar to be cloned
return &baz;
}

That is interesting, since it is very similar to what I came up with on
my own.

If a sub is conceptually cloned when the block enters, does that mean
that my $code = &bar (\&bar in p5) twice in a row should produce the
same value?

How does Perl 6 deal with my subs in for loops?

This question might be more appropriate​: In this example, which @​a does
the bar subroutine see (in Perl 6)?

  sub foo {
  my @​a = (1,2,3);
  my sub bar { say @​a };
  @​a := [4,5,6];
  bar();
  }

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From tchrist@perl.com

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
  on Sat, 07 Jul 2012 17​:44​:46 PDT​:

I’m forwarding this to the Perl 6 language list, so see if I can find
an answer there.

I do have an answer from Damian, which I will enclose below, and a
Rakudo result for you.

[This conversation is about how lexical subs should be implemented in
Perl 5. What Perl 6 does may help in determining how to iron out the
edge cases.]

[...]

This question might be more appropriate​: In this example, which @​a
does the bar subroutine see (in Perl 6)?

sub foo \{
    my @&#8203;a = \(1\,2\,3\);
    my sub bar \{ say @&#8203;a \};
    @&#8203;a := \[4\,5\,6\];
    bar\(\);
\}

The answer to your immediate question is that if you call foo(),
it prints out 456 under Rakudo.

Following is Damian's answer to my question, shared with permission.

--tom

  From​: Damian Conway <damian@​conway.org>
  To​: Tom Christiansen <tchrist@​perl.com>
  CC​: Larry Wall <larry@​wall.org>
  Date​: Sun, 08 Jul 2012 07​:17​:19 +1000
  Delivery-Date​: Sat, 07 Jul 2012 15​:19​:09
  Subject​: Re​: my subs and state vars
  In-Reply-To​: <22255.1341691089@​chthon>

  X-Spam-Status​: No, score=-102.6 required=4.5 tests=BAYES_00,RCVD_IN_DNSWL_LOW,
  USER_IN_WHITELIST autolearn=ham version=3.3.0

  X-Google-Sender-Auth​: UHLwfgo2kyvv2prdl6qJm-RfLF8
  Content-Type​: text/plain; charset=ISO-8859-1

  > It looks like perl5 may be close to having my subs, but a puzzle
  > has emerged about how in some circumstances to treat state
  > variables within those. [I'm pretty sure that perl6 has thought
  > this through thoroughly, but [I] am personally unfamiliar with the
  > outcome of said contemplations.]
  >
  > I bet you aren't, though. Any ideas or clues?

  The right things to do (and what Rakudo actually does) is to treat
  lexical subs as lexically scoped *instances* of the specified sub
  within the current surrounding block.

  That is​: a lexical sub is like a "my" var, in that you get a new one
  each time the surrounding block is executed. Rather than like an "our"
  variable, where you get a new lexically scoped alias to the same package
  scoped variable.

  By that reasoning, state vars inside a my sub must belong to each
  instance of the sub, just as state vars inside anonymous subs belong to
  each instance of the anonymous sub.

  Another way of thinking about what Perl 6 does is that​:

  my sub foo { whatever() }

  is just syntactic sugar for​:

  my &foo := sub { whatever() }

  That is​: create a lexically scoped Code object and alias it at run-time
  to an anonymous subroutine. So the rules for state variables inside
  lexical subs *must* be the same as the rules for state variables inside
  anonymous subs, since they're actually just two ways of creating the
  same thing.

  With this approach, in Perl 6 it's easy to specify exactly what you want​:

  sub recount_from ($n) {

  my sub counter {
  state $count = $n; # Each instance of &counter has its own count
  say $count--;
  die if $count == 0;
  }

  while prompt "recount $n> " {
  counter;
  }
  }

  vs​:

  sub first_count_down_from ($n) {

  state $count = $n; # All instances of &counter share a common count

  my sub counter {
  say $count--;
  die if $count == 0;
  }

  while prompt "first count $n> " {
  counter;
  }
  }

  Feel free to forward the above to anyone who might find it useful.

  Damian

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From @cpansprout

On Sat Jul 07 18​:35​:03 2012, tom christiansen wrote​:

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
on Sat, 07 Jul 2012 17​:44​:46 PDT​:

I’m forwarding this to the Perl 6 language list, so see if I can
find
an answer there.

I do have an answer from Damian, which I will enclose below, and a
Rakudo result for you.

[This conversation is about how lexical subs should be implemented
in
Perl 5. What Perl 6 does may help in determining how to iron out
the
edge cases.]

[...]

This question might be more appropriate​: In this example, which @​a
does the bar subroutine see (in Perl 6)?

sub foo \{
    my @&#8203;a = \(1\,2\,3\);
    my sub bar \{ say @&#8203;a \};
    @&#8203;a := \[4\,5\,6\];
    bar\(\);
\}

The answer to your immediate question is that if you call foo(),
it prints out 456 under Rakudo.

Thank you. So the bar sub seems to be closing over the name @​a (the
container/variable slot/pad entry/whatever), rather than the actual
array itself.

Since I don’t have it installed, could you tell me what this does?

  sub foo {
  my @​a = (1,2,3);
  my sub bar { say @​a };
  bar();
  @​a := [4,5,6];
  bar();
  }
  foo();

And this?

  sub foo {
  my @​a = (1,2,3);
  bar();
  @​a := [4,5,6];
  bar();
  my sub bar { say @​a };
  }
  foo();

And this?

  sub foo {
  my @​a = (1,2,3);
  my sub bar { say @​a };
  my $bar = &bar;
  $bar(); # is this syntax right?
  @​a := [4,5,6];
  $bar();
  }
  foo();

Following is Damian's answer to my question, shared with permission.

--tom

From&#8203;:          Damian Conway \<damian@&#8203;conway\.org>
To&#8203;:            Tom Christiansen \<tchrist@&#8203;perl\.com>
CC&#8203;:            Larry Wall \<larry@&#8203;wall\.org>
Date&#8203;:          Sun\, 08 Jul 2012 07&#8203;:17&#8203;:19 \+1000
Delivery\-Date&#8203;: Sat\, 07 Jul 2012 15&#8203;:19&#8203;:09
Subject&#8203;:       Re&#8203;: my subs and state vars
In\-Reply\-To&#8203;:   \<22255\.1341691089@&#8203;chthon>

X\-Spam\-Status&#8203;: No\, score=\-102\.6 required=4\.5

tests=BAYES_00,RCVD_IN_DNSWL_LOW,
USER_IN_WHITELIST autolearn=ham version=3.3.0

X\-Google\-Sender\-Auth&#8203;: UHLwfgo2kyvv2prdl6qJm\-RfLF8
Content\-Type&#8203;:  text/plain; charset=ISO\-8859\-1

> It looks like perl5 may be close to having my subs\, but a puzzle
> has emerged about how in some circumstances to treat state
> variables  within those\.  \[I'm pretty sure that perl6 has

thought
> this through thoroughly, but [I] am personally unfamiliar with
the
> outcome of said contemplations.]
>
> I bet you aren't, though. Any ideas or clues?

The right things to do \(and what Rakudo actually does\) is to treat
lexical subs as lexically scoped \*instances\* of the specified sub
within the current surrounding block\.

That is&#8203;: a lexical sub is like a "my" var\, in that you get a new

one
each time the surrounding block is executed. Rather than like an
"our"
variable, where you get a new lexically scoped alias to the same
package
scoped variable.

By that reasoning\, state vars inside a my sub must belong to each
instance of the sub\, just as state vars inside anonymous subs

belong to
each instance of the anonymous sub.

Another way of thinking about what Perl 6 does is that&#8203;:

    my sub foo \{ whatever\(\) \}

is just syntactic sugar for&#8203;:

    my &foo := sub \{ whatever\(\) \}

Does that mean I cannot call it before it is declared?

That is&#8203;: create a lexically scoped Code object and alias it at

run-time
to an anonymous subroutine. So the rules for state variables
inside
lexical subs *must* be the same as the rules for state variables
inside
anonymous subs, since they're actually just two ways of creating
the
same thing.

I see.

With this approach\, in Perl 6 it's easy to specify exactly what

you want​:

    sub recount\_from \($n\) \{

        my sub counter \{
            state $count = $n;   \# Each instance of &counter has

its own count
say $count--;
die if $count == 0;
}

        while prompt "recount $n> " \{
            counter;
        \}
    \}

vs&#8203;:

    sub first\_count\_down\_from \($n\) \{

        state $count = $n;       \# All instances of &counter share

a common count

        my sub counter \{
            say $count\-\-;
            die if $count == 0;
        \}

        while prompt "first count $n> " \{
            counter;
        \}
    \}

Feel free to forward the above to anyone who might find it useful\.

What I am really trying to find out is when the subroutine is actually
cloned, and whether there can be multiple clones within a single call of
the enclosing sub.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From [Unknown Contact. See original ticket]

On Sat Jul 07 18​:35​:03 2012, tom christiansen wrote​:

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
on Sat, 07 Jul 2012 17​:44​:46 PDT​:

I’m forwarding this to the Perl 6 language list, so see if I can
find
an answer there.

I do have an answer from Damian, which I will enclose below, and a
Rakudo result for you.

[This conversation is about how lexical subs should be implemented
in
Perl 5. What Perl 6 does may help in determining how to iron out
the
edge cases.]

[...]

This question might be more appropriate​: In this example, which @​a
does the bar subroutine see (in Perl 6)?

sub foo \{
    my @&#8203;a = \(1\,2\,3\);
    my sub bar \{ say @&#8203;a \};
    @&#8203;a := \[4\,5\,6\];
    bar\(\);
\}

The answer to your immediate question is that if you call foo(),
it prints out 456 under Rakudo.

Thank you. So the bar sub seems to be closing over the name @​a (the
container/variable slot/pad entry/whatever), rather than the actual
array itself.

Since I don’t have it installed, could you tell me what this does?

  sub foo {
  my @​a = (1,2,3);
  my sub bar { say @​a };
  bar();
  @​a := [4,5,6];
  bar();
  }
  foo();

And this?

  sub foo {
  my @​a = (1,2,3);
  bar();
  @​a := [4,5,6];
  bar();
  my sub bar { say @​a };
  }
  foo();

And this?

  sub foo {
  my @​a = (1,2,3);
  my sub bar { say @​a };
  my $bar = &bar;
  $bar(); # is this syntax right?
  @​a := [4,5,6];
  $bar();
  }
  foo();

Following is Damian's answer to my question, shared with permission.

--tom

From&#8203;:          Damian Conway \<damian@&#8203;conway\.org>
To&#8203;:            Tom Christiansen \<tchrist@&#8203;perl\.com>
CC&#8203;:            Larry Wall \<larry@&#8203;wall\.org>
Date&#8203;:          Sun\, 08 Jul 2012 07&#8203;:17&#8203;:19 \+1000
Delivery\-Date&#8203;: Sat\, 07 Jul 2012 15&#8203;:19&#8203;:09
Subject&#8203;:       Re&#8203;: my subs and state vars
In\-Reply\-To&#8203;:   \<22255\.1341691089@&#8203;chthon>

X\-Spam\-Status&#8203;: No\, score=\-102\.6 required=4\.5

tests=BAYES_00,RCVD_IN_DNSWL_LOW,
USER_IN_WHITELIST autolearn=ham version=3.3.0

X\-Google\-Sender\-Auth&#8203;: UHLwfgo2kyvv2prdl6qJm\-RfLF8
Content\-Type&#8203;:  text/plain; charset=ISO\-8859\-1

> It looks like perl5 may be close to having my subs\, but a puzzle
> has emerged about how in some circumstances to treat state
> variables  within those\.  \[I'm pretty sure that perl6 has

thought
> this through thoroughly, but [I] am personally unfamiliar with
the
> outcome of said contemplations.]
>
> I bet you aren't, though. Any ideas or clues?

The right things to do \(and what Rakudo actually does\) is to treat
lexical subs as lexically scoped \*instances\* of the specified sub
within the current surrounding block\.

That is&#8203;: a lexical sub is like a "my" var\, in that you get a new

one
each time the surrounding block is executed. Rather than like an
"our"
variable, where you get a new lexically scoped alias to the same
package
scoped variable.

By that reasoning\, state vars inside a my sub must belong to each
instance of the sub\, just as state vars inside anonymous subs

belong to
each instance of the anonymous sub.

Another way of thinking about what Perl 6 does is that&#8203;:

    my sub foo \{ whatever\(\) \}

is just syntactic sugar for&#8203;:

    my &foo := sub \{ whatever\(\) \}

Does that mean I cannot call it before it is declared?

That is&#8203;: create a lexically scoped Code object and alias it at

run-time
to an anonymous subroutine. So the rules for state variables
inside
lexical subs *must* be the same as the rules for state variables
inside
anonymous subs, since they're actually just two ways of creating
the
same thing.

I see.

With this approach\, in Perl 6 it's easy to specify exactly what

you want​:

    sub recount\_from \($n\) \{

        my sub counter \{
            state $count = $n;   \# Each instance of &counter has

its own count
say $count--;
die if $count == 0;
}

        while prompt "recount $n> " \{
            counter;
        \}
    \}

vs&#8203;:

    sub first\_count\_down\_from \($n\) \{

        state $count = $n;       \# All instances of &counter share

a common count

        my sub counter \{
            say $count\-\-;
            die if $count == 0;
        \}

        while prompt "first count $n> " \{
            counter;
        \}
    \}

Feel free to forward the above to anyone who might find it useful\.

What I am really trying to find out is when the subroutine is actually
cloned, and whether there can be multiple clones within a single call of
the enclosing sub.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From tchrist@perl.com

"Father Chrysostomos via RT" <perlbug-comment@​perl.org> wrote
  on Sat, 07 Jul 2012 18​:54​:15 PDT​:

Thank you. So the bar sub seems to be closing over the name @​a (the
container/variable slot/pad entry/whatever), rather than the actual
array itself.

Since I don't have it installed, could you tell me what this does?

All three of those say the same thing​:

  123
  456

--tom

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From damian@conway.org

Father Chrysostomos asked​:

What I am really trying to find out is when the subroutine is actually
cloned,

Yes. It is supposed to be (or at least must *appear* to be),
and currently is (or appears to be) in Rakudo.

and whether there can be multiple clones within a single call of
the enclosing sub.

Yes. For example, a lexical sub might be declared in a loop inside the
enclosing sub, in which case it should produce multiple instances, one
per iteration.

For example, this​:

  sub outer_sub () {
  for (1..3) {
  state $call_num = 1;
  my sub inner_sub {
  state $inner_state = (1..100).pick; # i.e. random number
  say " [call {$call_num++}] \$inner_state = $inner_state";
  }

  say "\nsub id​: ", &inner_sub.id;
  inner_sub();
  inner_sub();
  }
  }

  outer_sub();

produces​:

  sub id​: -4628941774842748435
  [call 1] $inner_state = 89
  [call 2] $inner_state = 89

  sub id​: -4628941774848253711
  [call 3] $inner_state = 16
  [call 4] $inner_state = 16

  sub id​: -4628941774839825925
  [call 5] $inner_state = 26
  [call 6] $inner_state = 26

under Rakudo

BTW, Both the above "yes" answers are consistent with (and can be
inferred from) the previous explanation that​:

  my sub foo { whatever() }

is just a syntactic convenience for​:

  my &foo := sub { whatever() }

HTH,

Damian

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2012

From @cpansprout

On Sat Jul 07 22​:23​:16 2012, thoughtstream wrote​:

Father Chrysostomos asked​:

What I am really trying to find out is when the subroutine is actually
cloned,

Yes. It is supposed to be (or at least must *appear* to be),
and currently is (or appears to be) in Rakudo.

I said when, not whether. :-)

and whether there can be multiple clones within a single call of
the enclosing sub.

Yes. For example, a lexical sub might be declared in a loop inside the
enclosing sub, in which case it should produce multiple instances, one
per iteration.

For example, this​:

sub outer\_sub \(\) \{
    for \(1\.\.3\) \{
        state $call\_num = 1;
        my sub inner\_sub \{
            state $inner\_state = \(1\.\.100\)\.pick; \# i\.e\. random number
            say "    \[call \{$call\_num\+\+\}\] \\$inner\_state =

$inner_state";

        \}

        say "\\nsub id&#8203;: "\, &inner\_sub\.id;
        inner\_sub\(\);
        inner\_sub\(\);
    \}
\}

outer\_sub\(\);

produces​:

sub id&#8203;: \-4628941774842748435
    \[call 1\] $inner\_state = 89
    \[call 2\] $inner\_state = 89

sub id&#8203;: \-4628941774848253711
    \[call 3\] $inner\_state = 16
    \[call 4\] $inner\_state = 16

sub id&#8203;: \-4628941774839825925
    \[call 5\] $inner\_state = 26
    \[call 6\] $inner\_state = 26

under Rakudo

Thank you.

Does Perl 6 have an equivalent to this?

  my $x;
  for $x(1..10) {}

In this case, the loop reuses the $x slot in the pad, making the
existing $x name an alias to a different scalar. But, due to the way
Perl 5 closures work, it is, for all tents and porpoises, the same as this​:

  my $x;
  for my $x(1..10) {}

BTW, Both the above "yes" answers are consistent with (and can be
inferred from) the previous explanation that​:

    my sub foo \{ whatever\(\) \}

is just a syntactic convenience for​:

    my &foo := sub \{ whatever\(\) \}

Except that my sub foo happens upon block entry, right?

HTH,

It does, but I am still trying to wrap my head around the fundamental
difference between 5 and 6 with regard to closures.

In Perl 5, $] in a piece of code is bound to *], not $], so it sees
changes made by local($]) (which actually puts a completely new scalar
in the *]{SCALAR} slot). But ‘my $x; sub { $x }’ is bound, not to the
$x slot in the outer block/sub/file, but to the actual scalar itself.

It seems that Perl 6 closures close over the slot, not the
scalar/array/etc. Is that right?

Anyway, I think I know how to implement this now.

The first time a ‘my’ sub is referenced or called, it is cloned. The
clone is stored for reuse, but that storage is localised to the current
block.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From Eirik-Berg.Hanssen@allverden.no

On Thu, Sep 13, 2012 at 5​:13 PM, Leon Timmermans <fawaka@​gmail.com> wrote​:

I'd strongly prefer experimental​::lexical_subs over
experimental_lexical_subs; the latter doesn't separate the status from
the actual name of the feature.

  As long as we're shedding bikes​: How about "experimental​:lexical_subs"?

  ... it's not as if it's a symbol, after all ...

Eirik, briefly considering "experimantal​: lexical subs" and
"experimental​:lexical-subs" ...

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From @steve-m-hay

Steve Hay wrote on 2012-09-13​:

Father Chrysostomos via RT wrote on 2012-09-13​:

I’m getting black smoke from George Greer’s Windows smoker. Trying to
track it down remotely by tweaking the code and pushing it to a smoke-
me branch multiple times will be very tedious.

Could anyone with access to a functional* Windows machine try and
see which commit on the sprout/lexsub branch caused the first failure?

I got it down to being somewhere between
c2837f2b837d7e5c275a691d8c16fef000cc86b7 (passes all op/*.t tests at
least) and efece7a52991d97ffbb1eacc44736d04c5847ee2 (fails loads of
op/*.t tests). I'll continue bisecting some more later.

Commit 56e168315961440f9bd040fc9a6b32e128fa289b passes all tests.
The next commit, e630ae0c3e7bb47b642a8e758beadd6cc68404f9, fails lots of tests.

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From @ap

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13 17​:55]​:

On Thu Sep 13 07​:25​:52 2012, aristotle wrote​:

Finally I decided to pass on it because try as I might, I could
think of no benefit of :​: over the underscore, and so it seems
merely gratuitous. If anyone can suggest a benefit, please do. In
absence of any, it seems preferable to me to stay within existing
conventions.

Existing conventions are to use only a-z.

Hmm, point. Clearly I should have paid more attention.

And there is certainly no way to do it without underscores and without
a way of delimiting the `experimental` prefix from the feature name. So
any choice will break convention at least a little, so maybe it doesn’t
make an appreciable difference. I guess using just underscore still has
an argument in its favour in that it is the least break possible… but
I dunno if that convinces me.

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From @cpansprout

On Thu Sep 13 09​:59​:45 2012, Steve.Hay@​verosoftware.com wrote​:

Steve Hay wrote on 2012-09-13​:
Commit 56e168315961440f9bd040fc9a6b32e128fa289b passes all tests.
The
next commit, e630ae0c3e7bb47b642a8e758beadd6cc68404f9, fails lots of
tests.

Thank you. Attached is that commit. I cannot see anything in there
that would be platform-specific. Whenever it is convenient for you, do
you think you could try to reduce one of the .t files to a single test?
Maybe it will set off a lightbulb when I see it.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From @cpansprout

commit e630ae0c3e7bb47b642a8e758beadd6cc68404f9
Author​: Father Chrysostomos <sprout@​cpan.org>
Date​: Wed Jul 4 23​:18​:32 2012 -0700

  Let barewords look up our subs
 
  These take precedence over built-in keywords (just as my $AUTOLOAD
  shadows the package var), but not the keyword plugin, as the latter
  takes precedence over labels, and these don’t.

Inline Patch
diff --git a/t/cmd/lexsub.t b/t/cmd/lexsub.t
index cef8070..ef2581e 100644
--- a/t/cmd/lexsub.t
+++ b/t/cmd/lexsub.t
@@ -7,7 +7,7 @@ BEGIN {
     *bar::is = *is;
 }
 no warnings 'deprecated';
-plan 20;
+plan 21;
 
 {
   our sub foo { 42 }
@@ -16,9 +16,7 @@ plan 20;
   is do foo(), 42, 'calling our sub from same package (do)';
   package bar;
   sub bar::foo { 43 }
-  { local $::TODO = ' ';
-    is foo, 42, 'calling our sub from another package';
-  }
+  is foo, 42, 'calling our sub from another package';
   is &foo, 42, 'calling our sub from another package (amper)';
   is do foo(), 42, 'calling our sub from another package (do)';
 }
@@ -44,9 +42,7 @@ package main;
   our sub b {
     if (shift) {
       package bar;
-      { local $::TODO = ' ';
-        is b, 42, 'our sub visible inside itself after decl';
-      }
+      is b, 42, 'our sub visible inside itself after decl';
       is &b, 42, 'our sub visible inside itself after decl (amper)';
       is do b(), 42, 'our sub visible inside itself after decl (do)';
     }
@@ -59,9 +55,7 @@ sub bar::c { 43 }
 {
   our sub c;
   package bar;
-  { local $::TODO = ' ';
-    is c, 42, 'our sub foo; makes lex alias for existing sub';
-  }
+  is c, 42, 'our sub foo; makes lex alias for existing sub';
   is &c, 42, 'our sub foo; makes lex alias for existing sub (amper)';
   is do c(), 42, 'our sub foo; makes lex alias for existing sub (do)';
 }
@@ -76,3 +70,9 @@ sub bar::c { 43 }
   our sub e ($);
   is prototype "::e", '$', 'our sub with proto';
 }
+{
+  # lexical subs (even our) override all keywords
+  our sub if() { 42 }
+  my $x = if if if;
+  is $x, 42;
+}
diff --git a/toke.c b/toke.c
index ff3f86c..8cba215 100644
--- a/toke.c
+++ b/toke.c
@@ -6602,9 +6602,13 @@ Perl_yylex(pTHX)
 
       keylookup: {
 	bool anydelim;
+	bool lex;
 	I32 tmp;
+	SV *sv;
 
+	lex = FALSE;
 	orig_keyword = 0;
+	sv = NULL;
 	gv = NULL;
 	gvp = NULL;
 
@@ -6673,6 +6677,31 @@ Perl_yylex(pTHX)
 	    TOKEN(LABEL);
 	}
 
+	/* Check for lexical sub */
+	if (PL_expect != XOPERATOR) {
+	    char tmpbuf[sizeof PL_tokenbuf + 1];
+	    PADOFFSET off;
+	    *tmpbuf = '&';
+	    Copy(PL_tokenbuf, tmpbuf+1, len, char);
+	    off = pad_findmy_pvn(tmpbuf, len+1, UTF ? SVf_UTF8 : 0);
+	    if (off != NOT_IN_PAD) {
+		if (PAD_COMPNAME_FLAGS_isOUR(off)) {
+		    HV *  const stash = PAD_COMPNAME_OURSTASH(off);
+		    HEK * const stashname = HvNAME_HEK(stash);
+		    sv = newSVhek(stashname);
+                    sv_catpvs(sv, "::");
+                    sv_catpvn_flags(sv, PL_tokenbuf, len,
+				    (UTF ? SV_CATUTF8 : SV_CATBYTES));
+		    gv = gv_fetchsv(sv, GV_NOADD_NOINIT | SvUTF8(sv),
+				    SVt_PVCV);
+		    lex = TRUE;
+		    goto just_a_word;
+		}
+		/* unreachable */
+		else Perl_croak(aTHX_ "\"my sub\" not yet implemented");
+	    }
+	}
+
 	if (tmp < 0) {			/* second-class keyword? */
 	    GV *ogv = NULL;	/* override (winner) */
 	    GV *hgv = NULL;	/* hidden (loser) */
@@ -6737,7 +6766,6 @@ Perl_yylex(pTHX)
 		orig_keyword = 0;
 	    }
 	  just_a_word: {
-		SV *sv;
 		int pkgname = 0;
 		const char lastchar = (PL_bufptr == PL_oldoldbufptr ? 0 : PL_bufptr[-1]);
 		OP *rv2cv_op;
@@ -6805,7 +6833,8 @@ Perl_yylex(pTHX)
 
 		/* if we saw a global override before, get the right name */
 
-		sv = S_newSV_maybe_utf8(aTHX_ PL_tokenbuf,
+		if (!lex)
+		  sv = S_newSV_maybe_utf8(aTHX_ PL_tokenbuf,
 		    len ? len : strlen(PL_tokenbuf));
 		if (gvp) {
 		    SV * const tmp_sv = sv;
@@ -6836,7 +6865,7 @@ Perl_yylex(pTHX)
 		    const_op->op_private = OPpCONST_BARE;
 		    rv2cv_op = newCVREF(0, const_op);
 		}
-		cv = rv2cv_op_cv(rv2cv_op, 0);
+		cv = lex ? GvCV(gv) : rv2cv_op_cv(rv2cv_op, 0);
 
 		/* See if it's the indirect object for a list operator. */
 

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From @cpansprout

On Thu Sep 13 14​:10​:52 2012, aristotle wrote​:

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13
17​:55]​:

On Thu Sep 13 07​:25​:52 2012, aristotle wrote​:

Finally I decided to pass on it because try as I might, I could
think of no benefit of :​: over the underscore, and so it seems
merely gratuitous. If anyone can suggest a benefit, please do. In
absence of any, it seems preferable to me to stay within existing
conventions.

Existing conventions are to use only a-z.

Hmm, point. Clearly I should have paid more attention.

And there is certainly no way to do it without underscores and without
a way of delimiting the `experimental` prefix from the feature name. So
any choice will break convention at least a little, so maybe it doesn’t
make an appreciable difference. I guess using just underscore still has
an argument in its favour in that it is the least break possible… but
I dunno if that convinces me.

Porting/todo.pod says this​:

=head2 enable lexical enabling/disabling of individual warnings

Currently, warnings can only be enabled or disabled by category. There
are times when it would be useful to quash a single warning, not a
whole category.

And Eirik Berg Hanssen says​:

As long as we're shedding bikes​: How about
"experimental​:lexical_subs"?

Now consider that you are proposing a warnings ‘category’ with one
single warning it it.

If you put two and two together, you end up with όλα από πέντε (sorry,
couldn’t resist :-)​:

  use warnings "experimental"; # whole category
  use warnings "experimental​:lexical_subs"; # single warning

  no warnings 'utf8'; # turn them off
  use warnings FATAL => 'utf8​:wide'; # but I still want this one

And that solves the problem of trying to decide what convention to use
when assigning IDs to warnings. Just come up with something alphabetic,
short, and hence memorable.

Now we don’t have to do it all at once, but we can start with the
experimental category.

Experimental warning IDs should match the feature names, too, so they
can be exempt from the no-underscores rule. But generally the perl core
avoids underscores between letters (CLONE_SKIP being one egregious
example of the violation of that principle), so that user code can use
them safely everywhere.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From @ap

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13 23​:44]​:

Now consider that you are proposing a warnings ‘category’ with one
single warning it it.

If you put two and two together, you end up with όλα από πέντε (sorry,
couldn’t resist :-)​:

:-)

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

no warnings 'utf8';   \# turn them off
use warnings FATAL => 'utf8&#8203;:wide'; \# but I still want this one

And that solves the problem of trying to decide what convention to use
when assigning IDs to warnings. Just come up with something alphabetic,
short, and hence memorable.

Now *that* I like.

Now we don’t have to do it all at once, but we can start with the
experimental category.

Yes, it would require going through all of core and naming every single
warning, right? I was going to say it seems a problem for the lexical
subs work to have to block on that but you’re right, it needn’t be done
all or nothing.

Experimental warning IDs should match the feature names, too, so they
can be exempt from the no-underscores rule. But generally the perl
core avoids underscores between letters (CLONE_SKIP being one
egregious example of the violation of that principle), so that user
code can use them safely everywhere.

Yeah. The proposal feels like a unification on the right level that
makes all the answers for the original problems fall out naturally so
I like it rather a lot.

While the approach feels very right though, I feel I can’t ask good
questions about specific designs for it off the cuff/just yet. Is the
colon separating category + ID the right choice? I don’t know. I’d
love to hear if anything occurs to RJBS in particular after letting
it simmer for a while, or anyone.

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From @cpansprout

On Thu Sep 13 15​:11​:14 2012, aristotle wrote​:

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13
23​:44]​:

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

no warnings 'utf8';   \# turn them off
use warnings FATAL => 'utf8&#8203;:wide'; \# but I still want this one

And that solves the problem of trying to decide what convention to use
when assigning IDs to warnings. Just come up with something alphabetic,
short, and hence memorable.

Now *that* I like.

While the approach feels very right though, I feel I can’t ask good
questions about specific designs for it off the cuff/just yet. Is the
colon separating category + ID the right choice? I don’t know. I’d
love to hear if anything occurs to RJBS in particular after letting
it simmer for a while, or anyone.

One problem is that warnings might change categories. One instance I
can think of is ‘vector argument not supported with alpha versions’.
Right now it’s in the internal category, believe it or not. That is
completely wrong, and I have already changed it locally to the printf
category.

I think those will be rare, though. We can make backward-compatibility
aliases if necessary (internal​:valpha maps to printf​:valpha and is
unrelated to the internal category).

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From dcmertens.perl@gmail.com

What's the reason for using one instead of two colons? Perl namespace
encoding suggests two, but Huffman encoding suggests one.

Or should this question be tagged as bike shedding and ignored for the
moment?

David
On Sep 13, 2012 7​:58 PM, "Father Chrysostomos via RT" <
perlbug-followup@​perl.org> wrote​:

On Thu Sep 13 15​:11​:14 2012, aristotle wrote​:

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13
23​:44]​:

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

no warnings 'utf8';   \# turn them off
use warnings FATAL => 'utf8&#8203;:wide'; \# but I still want this one

And that solves the problem of trying to decide what convention to use
when assigning IDs to warnings. Just come up with something
alphabetic,
short, and hence memorable.

Now *that* I like.

While the approach feels very right though, I feel I can’t ask good
questions about specific designs for it off the cuff/just yet. Is the
colon separating category + ID the right choice? I don’t know. I’d
love to hear if anything occurs to RJBS in particular after letting
it simmer for a while, or anyone.

One problem is that warnings might change categories. One instance I
can think of is ‘vector argument not supported with alpha versions’.
Right now it’s in the internal category, believe it or not. That is
completely wrong, and I have already changed it locally to the printf
category.

I think those will be rare, though. We can make backward-compatibility
aliases if necessary (internal​:valpha maps to printf​:valpha and is
unrelated to the internal category).

--

Father Chrysostomos

---
via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=113930

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From @ap

* David Mertens <dcmertens.perl@​gmail.com> [2012-09-14 05​:45]​:

What's the reason for using one instead of two colons? Perl namespace
encoding suggests two, but Huffman encoding suggests one.

Or should this question be tagged as bike shedding and ignored for the
moment?

Since it’s FC who came up with the approach, maybe it should be

  use warnings "internal'valpha";

in his honour. ;-)

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From @cpansprout

On Thu Sep 13 20​:42​:56 2012, dcmertens.perl@​gmail.com wrote​:

What's the reason for using one instead of two colons? Perl namespace
encoding suggests two, but Huffman encoding suggests one.

Warnings categories are not packages. I could equally argue that
"experimental->{lexical_subs}" should be used. :-)

I think the single colon prevents anyone from drawing connections where
there are none.

But we could use anything.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From @cpansprout

On Thu Sep 13 08​:50​:22 2012, sprout wrote​:

On Thu Sep 13 07​:25​:52 2012, aristotle wrote​:

Finally I decided to pass on it because try as I might, I could think of
no benefit of :​: over the underscore, and so it seems merely gratuitous.
If anyone can suggest a benefit, please do. In absence of any, it seems
preferable to me to stay within existing conventions.

Existing conventions are to use only a-z.

I take that back. We already have non_unicode. But why non_unicode and
nonchar have to follow different conventions is beyond me.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From Eirik-Berg.Hanssen@allverden.no

On Fri, Sep 14, 2012 at 7​:14 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

I think the single colon prevents anyone from drawing connections where
there are none.

  That was the idea. Or rather, no Perl-related connections. Colons are
used elsewhere for similar purposes – with URI schemes and MediaWiki
namespaces, to name the two first on my mind. So, while alien to Perl
proper (and thus not misleading), it might still be familiar to users, and
so easier to remember than some arbitrary separator. :)

Eirik

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From @cpansprout

On Thu Sep 13 14​:29​:45 2012, sprout wrote​:

On Thu Sep 13 09​:59​:45 2012, Steve.Hay@​verosoftware.com wrote​:

Steve Hay wrote on 2012-09-13​:
Commit 56e168315961440f9bd040fc9a6b32e128fa289b passes all tests.
The
next commit, e630ae0c3e7bb47b642a8e758beadd6cc68404f9, fails lots of
tests.

Thank you. Attached is that commit. I cannot see anything in there
that would be platform-specific. Whenever it is convenient for you, do
you think you could try to reduce one of the .t files to a single test?
Maybe it will set off a lightbulb when I see it.

Chip sent me a configuration with which I was able to reproduce the
failures on Mac OS X. So I can take over from here. Thank you for your
help so far.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 15, 2012

From @cpansprout

On Fri Sep 14 16​:30​:44 2012, sprout wrote​:

On Thu Sep 13 14​:29​:45 2012, sprout wrote​:

On Thu Sep 13 09​:59​:45 2012, Steve.Hay@​verosoftware.com wrote​:

Steve Hay wrote on 2012-09-13​:
Commit 56e168315961440f9bd040fc9a6b32e128fa289b passes all tests.
The
next commit, e630ae0c3e7bb47b642a8e758beadd6cc68404f9, fails lots of
tests.

Thank you. Attached is that commit. I cannot see anything in there
that would be platform-specific. Whenever it is convenient for you, do
you think you could try to reduce one of the .t files to a single test?
Maybe it will set off a lightbulb when I see it.

Chip sent me a configuration with which I was able to reproduce the
failures on Mac OS X. So I can take over from here. Thank you for your
help so far.

It was an uninitialized C auto. I have squashed this diff with the
offending commit and pushed to perl.git again.

Inline Patch
diff --git a/toke.c b/toke.c
index 47c2b0a..db08ee7 100644
--- a/toke.c
+++ b/toke.c
@@ -6779,6 +6779,7 @@ Perl_yylex(pTHX)
                rv2cv_op = NULL;
                orig_keyword = 0;
                lex = 0;
+               off = 0;
            }
          just_a_word: {
                int pkgname = 0;

-- 

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 16, 2012

From @cpansprout

On Thu Sep 13 15​:11​:14 2012, aristotle wrote​:

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13
23​:44]​:

Now consider that you are proposing a warnings ‘category’ with one
single warning it it.

If you put two and two together, you end up with όλα από πέντε (sorry,
couldn’t resist :-)​:

:-)

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

no warnings 'utf8';   \# turn them off
use warnings FATAL => 'utf8&#8203;:wide'; \# but I still want this one

And that solves the problem of trying to decide what convention to use
when assigning IDs to warnings. Just come up with something alphabetic,
short, and hence memorable.

Now *that* I like.

Now we don’t have to do it all at once, but we can start with the
experimental category.

Yes, it would require going through all of core and naming every single
warning, right? I was going to say it seems a problem for the lexical
subs work to have to block on that but you’re right, it needn’t be done
all or nothing.

Experimental warning IDs should match the feature names, too, so they
can be exempt from the no-underscores rule. But generally the perl
core avoids underscores between letters (CLONE_SKIP being one
egregious example of the violation of that principle), so that user
code can use them safely everywhere.

Yeah. The proposal feels like a unification on the right level that
makes all the answers for the original problems fall out naturally so
I like it rather a lot.

While the approach feels very right though, I feel I can’t ask good
questions about specific designs for it off the cuff/just yet. Is the
colon separating category + ID the right choice? I don’t know. I’d
love to hear if anything occurs to RJBS in particular after letting
it simmer for a while, or anyone.

It’s in now as 74760be. Did I perhaps act too soon? I don’t know.
I’ll see if my commit bit get revoked. :-)

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 16, 2012

@cpansprout - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Sep 16, 2012
@p5pRT
Copy link
Author

p5pRT commented Sep 16, 2012

From @ap

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-16 08​:45]​:

It’s in now as 74760be. Did I perhaps act too soon? I don’t know.
I’ll see if my commit bit get revoked. :-)

Ah, design by fait accompli. :-)

Well, let’s see. I don’t *expect* any surprises from this one. But then
what is a surprise if you expect it?

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

@p5pRT
Copy link
Author

p5pRT commented Sep 17, 2012

From @rjbs

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-16T02​:41​:22]

It’s in now as 74760be. Did I perhaps act too soon? I don’t know.
I’ll see if my commit bit get revoked. :-)

Well, I really don't like the use of : for separation, but that can be sorted
out after simmering completes.

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented Sep 17, 2012

From @nwc10

Sorry, this never got answered at the time, partly because I saw that you
seemed to have answered it yourself within 24 hours

On Fri, Aug 03, 2012 at 11​:21​:05PM -0700, Father Chrysostomos via RT wrote​:

commit adf8f09
Author​: Nicholas Clark <nick@​ccl4.org>
Date​: Fri Feb 26 09​:18​:44 2010 +0000

Set PADSTALE on all lexicals at the end of sub creation\.

The PADSTALEness of lexicals between the 0th and 1st call to a

subroutine is now
consistent with the state between the nth and (n + 1)th call.

This permits a work around in Perl\_padlist\_dup\(\) to avoid leaking

active pad
data into a new thread, whilst still correctly bodging the external
references
needed by the current ?{} implementation. Fix that, and this can be
removed.

I don't know whether this affects my subs yet, but I would like to gain
a better understanding of this commit. How did ?{} blocks work before?
What exactly was the fix intended to do?

I can't remember better than the commit message says.

?{} blocks before and after this commit continued to be a complete
hack. (This was before Dave's fix for them landed). The short description is
"lexicals in ?{} blocks didn't work." Sadly, they *seemed* to work, until
you tried to do anything more exciting than calling a subroutine once.

On Sat, Aug 04, 2012 at 06​:03​:27PM -0700, Father Chrysostomos via RT wrote​:

On Sat Aug 04 06​:43​:00 2012, sprout wrote​:

I can reproduce this bug even further back​:

sub foo {
my $x if @​_;
return if @​_;

$x = 17;
print $x\, "\\n";
print sub \{ $x \}\->\(\)\, "\\n";
return;

}
foo(1); # make $x stale in all perl versions
foo;

This doesn't surprise me. That commit revealed at least one test failure
from the BBC, but I was able to write a different test case for that module
which failed in the same way with perl before the commit you quote.

$ pbpaste|perl5.10.1 -w
17
Variable "$x" is not available at - line 7.
Use of uninitialized value in print at - line 7.

$ pbpaste|perl5.10.0 -w
17
17

I've fixed both those issues with commit cae5dbb.

Good stuff.

Nicholas Clark

@p5pRT
Copy link
Author

p5pRT commented Sep 17, 2012

From @nwc10

On Thu, Aug 23, 2012 at 09​:40​:57AM -0700, Father Chrysostomos via RT wrote​:

Just for future reference​: I have found that I do still need two
separate ops for introcv and clonecv.

[snip detailed explanation]

Very useful. Thanks for taking the time to write a coherent explanation.

Is this in a comment in the code anywhere? I think it's more appropriate to
put it somewhere in the repository, as that's much more easily findable in
the future.

Nicholas Clark

@p5pRT
Copy link
Author

p5pRT commented Sep 17, 2012

From @cpansprout

On Mon Sep 17 03​:39​:21 2012, nicholas wrote​:

On Thu, Aug 23, 2012 at 09​:40​:57AM -0700, Father Chrysostomos via RT
wrote​:

Just for future reference​: I have found that I do still need two
separate ops for introcv and clonecv.

[snip detailed explanation]

Very useful. Thanks for taking the time to write a coherent explanation.

Is this in a comment in the code anywhere? I think it's more
appropriate to
put it somewhere in the repository, as that's much more easily findable in
the future.

Yes, it’s in op.c​:block_end.

I meant to say that, but I apparently never finished writing the message
you cite. But at least it is understandable. :-)

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 21, 2012

From @rjbs

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-13T17​:43​:58]

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

I like the idea, but not enough. I think it will seem pretty good until it
seems bad.

I think we should do two things.

1) these warnings should use :​:, as if we're registering them as part of an
  'experimental' namespace. Introducing "​:" as a separator for something that
  looks like a package/module, and sometimes is (use warnings 'File​::Find') is
  just asking for confusion.

2) we should ensure that the mechanism for identifying core exceptions works on
  core warnings as well; or, in other terms​: we want to solve this problem
  for core exceptions, and a core warning is just an exception with no teeth

I know chromatic, at least, is working on this problem. I'll drop him a line
in the morning, unless he reads this and replies first.

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented Sep 26, 2012

From @cpansprout

On Thu Sep 20 21​:25​:09 2012, perl.p5p@​rjbs.manxome.org wrote​:

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-
13T17​:43​:58]

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

I like the idea, but not enough. I think it will seem pretty good
until it
seems bad.

I think we should do two things.

1) these warnings should use :​:, as if we're registering them as part
of an
'experimental' namespace. Introducing "​:" as a separator for
something that
looks like a package/module, and sometimes is (use warnings
'File​::Find') is
just asking for confusion.

Should I also remove the documentation about extending : to individual
warnings?

2) we should ensure that the mechanism for identifying core exceptions
works on
core warnings as well; or, in other terms​: we want to solve this
problem
for core exceptions, and a core warning is just an exception with
no teeth

I know chromatic, at least, is working on this problem. I'll drop him
a line
in the morning, unless he reads this and replies first.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 26, 2012

From [Unknown Contact. See original ticket]

On Thu Sep 20 21​:25​:09 2012, perl.p5p@​rjbs.manxome.org wrote​:

* Father Chrysostomos via RT <perlbug-followup@​perl.org> [2012-09-
13T17​:43​:58]

use warnings "experimental";  \# whole category
use warnings "experimental&#8203;:lexical\_subs"; \# single warning

I like the idea, but not enough. I think it will seem pretty good
until it
seems bad.

I think we should do two things.

1) these warnings should use :​:, as if we're registering them as part
of an
'experimental' namespace. Introducing "​:" as a separator for
something that
looks like a package/module, and sometimes is (use warnings
'File​::Find') is
just asking for confusion.

Should I also remove the documentation about extending : to individual
warnings?

2) we should ensure that the mechanism for identifying core exceptions
works on
core warnings as well; or, in other terms​: we want to solve this
problem
for core exceptions, and a core warning is just an exception with
no teeth

I know chromatic, at least, is working on this problem. I'll drop him
a line
in the morning, unless he reads this and replies first.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 26, 2012

From @rjbs

* Father Chrysostomos via RT <perlbug-comment@​perl.org> [2012-09-25T21​:05​:13]

Should I also remove the documentation about extending : to individual
warnings?

Please. Sorry for the delay.

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2012

From @cpansprout

On Wed Sep 26 16​:12​:38 2012, perl.p5p@​rjbs.manxome.org wrote​:

* Father Chrysostomos via RT <perlbug-comment@​perl.org> [2012-09-
25T21​:05​:13]

Should I also remove the documentation about extending : to
individual
warnings?

Please. Sorry for the delay.

Now done in commit f1d34ca.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2012

From [Unknown Contact. See original ticket]

On Wed Sep 26 16​:12​:38 2012, perl.p5p@​rjbs.manxome.org wrote​:

* Father Chrysostomos via RT <perlbug-comment@​perl.org> [2012-09-
25T21​:05​:13]

Should I also remove the documentation about extending : to
individual
warnings?

Please. Sorry for the delay.

Now done in commit f1d34ca.

--

Father Chrysostomos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant