Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pair.WHICH mishandles Scalar key #5583

Open
p6rt opened this issue Aug 16, 2016 · 21 comments
Open

Pair.WHICH mishandles Scalar key #5583

p6rt opened this issue Aug 16, 2016 · 21 comments
Labels

Comments

@p6rt
Copy link

p6rt commented Aug 16, 2016

Migrated from rt.perl.org#128965 (status was 'open')

Searchable as RT128965$

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

If I construct a Pair whose key is a Scalar object, then the identity of
that key object is not properly reflected in the .WHICH value of the Pair​:

my $a = 3; my $b = Pair.new($a.VAR, "z"); say $b.WHICH; $a = 5; say $b.WHICH
Pair|Int|3|Str|z
Pair|Int|5|Str|z

As you can see, the Pair.WHICH method is looking at the value inside the
Scalar container. As a result, the .WHICH value of the single Pair object
is changing over time, which should never happen. Pair.WHICH should be
looking only at the identity of the Scalar itself. The correct .WHICH
string would be something like "Pair|Scalar|47948211032816|Str|z".
I believe the fix is for Pair.WHICH to apply .VAR to $!key.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Additional​: the Pair.key method also returns the content of a Scalar key,
rather than the key itself​:

my $a = 3; my $b = Pair.new($a.VAR, "z"); $a = 5; say $b.key.WHICH
Int|5

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 12​:39 PM, Zefram <perl6-bugs-followup@​perl.org>
wrote​:

If I construct a Pair whose key is a Scalar object,

Scalar objects are a leaked internal detail that aren't intended for direct
use (.VAR is an avowed hack, and the likely outcome of your insistence on
treating it as first class is that .VAR will end up being removed or
restricted to keep the Scalar from escaping). You've created a mutable-key
Pair, which probably breaks all the Pair-based containers along with
everything else that knows that Pair is immutable.

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

The RT System itself - Status changed from 'new' to 'open'

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Brandon Allbery via RT wrote​:

                            the likely outcome of your insistence on

treating it as first class is that .VAR will end up being removed or
restricted to keep the Scalar from escaping

I'm treating as first class those things that are presented as first
class. There's a lot of reification in Perl 6, and generally it looks
deliberate, with Perl 6 intended to serve as its own metalanguage.
If containers are not meant to be part of the visible language then by
all means make them invisible.

FWIW, I think Scalar is a good reification, and the language would
be better with it being visible, provided that everything relevant
can handle it. I also think that the latter condition is achievable.
Conversely, with Scalar being so semantically significant, dereifying it
would make things more confusing. This is not a blanket opinion about
reification​: for example, I think reifying Slip was a poor choice.

                                         You've created a mutable\-key

Pair,

As with [perl #​128948], that's not really the Pair being mutable. I can
create two Pairs referencing as their keys the same Scalar, and that's
visibly different from each Pair having its own mutable key storage.
In fact the Pair is immutably referencing a mutable Scalar. The immutable
Pair's .WHICH would be fine if it looked only at the identity of the
immutably-referenced object, and avoided looking inside that mutable
object. It is just an inconvenience for the Pair.WHICH implementor that
looking inside a bound container is the default behaviour.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 3​:30 PM, Zefram <zefram@​fysh.org> wrote​:

FWIW, I think Scalar is a good reification, and the language would
be better with it being visible, provided that everything relevant
can handle it.

It can't handle it. The implementation is precisely that needed for
mutables; your multiple attempts to use them as something else are things
that will not change, because *Scalar is not what you think it is*. It is
*precisely* the implementation of mutable containers, and that is why it
behaves in ways that you believe are "wrong"; making it behave the way you
think it should will break mutable containers, or require reintroduction of
the original Scalar to again be the implementation of mutable containers.

This is not to claim that the thing you are looking for is in any way
wrong; it just is not Scalar. You will indeed be disappointed if you keep
trying to treat Scalar as the thing you want that it is not.

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Brandon Allbery via RT wrote​:

                         \*Scalar is not what you think it is\*\. It is

*precisely* the implementation of mutable containers,

That's what I think it is. I'm mystified as to what you think I think
it is.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 4​:00 PM, Zefram <zefram@​fysh.org> wrote​:

That's what I think it is. I'm mystified as to what you think I think
it is.

Because of all the behaviors central to it being the implementation of a
mutable value, that you have reported as bugs. They're not bugs, they are
proof that Scalar is *not what you want*. It is the implementation of a
mutable value that can be used as a value; they are *not* general mutable
containers which is what you are looking for. They aren't, and maybe if you
look over your recent "bugs" you might recognize a pattern about their
behavior that would tell you that they are not what you want them to be,
they are something different and they must be something different.

Let's try it this way​: earlier I sai that yuou had created a mutable-key
Pair.
I did not mean "You created something equivalent to a mutable-key Pair".
You *precisely* created a Pair that has a mutable component, in something
that is not intended to be mutable. Anything that uses a Pair expects it to
be immutable; your mutable-key Pair is no longer an immutable type.

Arguably this situation should not be allowed to occur, but that's why .VAR
is a hack that is leaking something that should not be visible. You're
breaking language invariants as a result; you can slip mutability into
immutable types at a level the type system can't see --- because Scalar is
precisely the difference between immutable and mutable values.

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @lizmat

On 16 Aug 2016, at 22​:10, Brandon Allbery <allbery.b@​gmail.com> wrote​:

On Tue, Aug 16, 2016 at 4​:00 PM, Zefram <zefram@​fysh.org> wrote​:
That's what I think it is. I'm mystified as to what you think I think
it is.

Because of all the behaviors central to it being the implementation of a mutable value, that you have reported as bugs. They're not bugs, they are proof that Scalar is *not what you want*. It is the implementation of a mutable value that can be used as a value; they are *not* general mutable containers which is what you are looking for. They aren't, and maybe if you look over your recent "bugs" you might recognize a pattern about their behavior that would tell you that they are not what you want them to be, they are something different and they must be something different.

Let's try it this way​: earlier I sai that yuou had created a mutable-key Pair.
I did not mean "You created something equivalent to a mutable-key Pair".
You *precisely* created a Pair that has a mutable component, in something that is not intended to be mutable. Anything that uses a Pair expects it to be immutable; your mutable-key Pair is no longer an immutable type.

Arguably this situation should not be allowed to occur, but that's why .VAR is a hack that is leaking something that should not be visible. You're breaking language invariants as a result; you can slip mutability into immutable types at a level the type system can't see --- because Scalar is precisely the difference between immutable and mutable values.

dd9b760 makes it impossible to create a Pair with a mutable key.

Tests needed.

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Elizabeth Mattijsen via RT wrote​:

dd9b760 makes it impossible to create a Pair with a mutable key.

Doesn't work. It defeats my original test case, but you're not actually
type-constraining the key, you're only removing a container wrapper.
So by adding a second container​:

my $a = 3; my $A = $a.VAR; my $b = Pair.new($A.VAR, "z"); say $b.WHICH; $a = 5; say $b.WHICH
Pair|Int|3|Str|z
Pair|Int|5|Str|z

It would be better to reject the Scalar-as-key case entirely, rather
than unwrap.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 4​:37 PM, Zefram <zefram@​fysh.org> wrote​:

Elizabeth Mattijsen via RT wrote​:

dd9b760 makes it impossible to create a Pair with a mutable key.

Doesn't work. It defeats my original test case, but you're not actually
type-constraining the key, you're only removing a container wrapper.

Right, this kind of fix is a losing proposition.

I'd prefer to see a proxy around the Scalar returned by .VAR that restricts
what can be done with it, so it can't escape into the wild --- because once
a Scalar can be accessed directly, you've broken the type system.

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @lizmat

On 16 Aug 2016, at 22​:40, Brandon Allbery <allbery.b@​gmail.com> wrote​:

On Tue, Aug 16, 2016 at 4​:37 PM, Zefram <zefram@​fysh.org> wrote​:
Elizabeth Mattijsen via RT wrote​:

dd9b760 makes it impossible to create a Pair with a mutable key.

Doesn't work. It defeats my original test case, but you're not actually
type-constraining the key, you're only removing a container wrapper.

Right, this kind of fix is a losing proposition.

I'd prefer to see a proxy around the Scalar returned by .VAR that restricts what can be done with it, so it can't escape into the wild --- because once a Scalar can be accessed directly, you've broken the type system.

But then you still haven’t defeated it, as you can do nqp​::create(Scalar).

But then you also get into a territory where I wonder what sense it makes.

I mean, in Perl 5 you can mark the canonical “undef” as “rw”, and then change it. With hilarious results.

So I’m feeling we’re getting into DIHWIDT territory

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 4​:45 PM, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

But then you still haven’t defeated it, as you can do nqp​::create(Scalar).

But you have to go into NQP to do that; you already took off all the
safeties, and whatever happens is your own fault. (Which is why TimToady
aliased nqp to MONKEY-GUTS.)

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Brandon Allbery via RT wrote​:

But you have to go into NQP to do that; you already took off all the
safeties, and whatever happens is your own fault.

Yes, and to be clear I'm satisfied with that kind of distinction between
language and implementation. If .VAR and the Scalar class are hidden
behind a MONKEY-SEE-NO-CONTAINER pragma then I'll accept that they're
not part of the supported language.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 5​:00 PM, Zefram <zefram@​fysh.org> wrote​:

If .VAR and the Scalar class are hidden
behind a MONKEY-SEE-NO-CONTAINER pragma then I'll accept that they're
not part of the supported language.

I think .VAR can remain visible as long as the Scalar can't escape from it.
(Although I do wonder whether .VAR is actually Perl 6 as opposed to Rakudo.)

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Brandon Allbery via RT wrote​:

I think .VAR can remain visible as long as the Scalar can't escape from it.

What would it yield if not the Scalar? That's the essence of .VAR
expressions. To prevent the Scalar escaping, you can't have that
available as an expression type.

(Although I do wonder whether .VAR is actually Perl 6 as opposed to Rakudo.)

It's mentioned in the specs. S12 specifically advertises using it to
get at Scalar objects. Scalar containers themselves are mentioned in
a few places, but not much elaborated on. As I said, it looks like
intentional reification.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @lizmat

Reverted my Pair.new fix with b00d92d

Suggest we mark this as stalled and/or @​LARRY

On 16 Aug 2016, at 23​:11, Zefram <zefram@​fysh.org> wrote​:

Brandon Allbery via RT wrote​:

I think .VAR can remain visible as long as the Scalar can't escape from it.

What would it yield if not the Scalar? That's the essence of .VAR
expressions. To prevent the Scalar escaping, you can't have that
available as an expression type.

(Although I do wonder whether .VAR is actually Perl 6 as opposed to Rakudo.)

It's mentioned in the specs. S12 specifically advertises using it to
get at Scalar objects. Scalar containers themselves are mentioned in
a few places, but not much elaborated on. As I said, it looks like
intentional reification.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @lizmat

See discussion at http://irclog.perlgeek.de/perl6-dev/2016-08-16#i_13034703

On 16 Aug 2016, at 23​:11, Zefram <zefram@​fysh.org> wrote​:

Brandon Allbery via RT wrote​:

I think .VAR can remain visible as long as the Scalar can't escape from it.

What would it yield if not the Scalar? That's the essence of .VAR
expressions. To prevent the Scalar escaping, you can't have that
available as an expression type.

(Although I do wonder whether .VAR is actually Perl 6 as opposed to Rakudo.)

It's mentioned in the specs. S12 specifically advertises using it to
get at Scalar objects. Scalar containers themselves are mentioned in
a few places, but not much elaborated on. As I said, it looks like
intentional reification.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

On Tue, Aug 16, 2016 at 5​:11 PM, Zefram <zefram@​fysh.org> wrote​:

Brandon Allbery via RT wrote​:

I think .VAR can remain visible as long as the Scalar can't escape from
it.

What would it yield if not the Scalar? That's the essence of .VAR
expressions. To prevent the Scalar escaping, you can't have that
available as an expression type.

If the proxy is itself an immutable type and makes itself visible instead
of trying to hide itself like Scalar does, the result *should* be the kind
of mutable container you were looking for in the first place. Which is the
way mutable containers should be exposed --- not the implementation detail
that breaks things.

That said, we've been tossing this around in IRC, and the Proxy type (which
is not the above hypothetical proxy, but a mechanism for read-write return
values) actually introduces the same issues because it allows you to make
something mutable that normally isn't. So there is still a general problem
here.

(ps lizmat​: I'm geekosaur :)

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From zefram@fysh.org

Brandon Allbery via RT wrote​:

I did not mean "You created something equivalent to a mutable-key Pair".
You *precisely* created a Pair that has a mutable component, in something
that is not intended to be mutable.

That view doesn't work for the situation in [perl #​128948].

It's quite intentional there that it's possible to create a Pair with
a Scalar container in its value slot​: I didn't use .VAR. By the view
you're expressing, the situation there is precisely that the Pair is
mutable in respect of its value. The .WHICH that is presently generated
for that situation, of the "Pair|47804193992464" form, follows that view,
attributing the mutability to the Pair itself.

But I was able to create two Pairs referencing the same Scalar value
container. It's visible that they reference the same mutable storage;
the behaviour upon mutation is different from what would be seen if
each Pair had its own mutable storage. In that ticket the result
was that I had two behaviourally-identical Pairs claiming different
identity. We could paper over that by making the .WHICH value take the
mutable storage identity from the (internal) Scalar, still returning a
"Pair|47804193992464" kind of value, and pretend that the Pairs were
actually the same object. But one can just as well create two Pairs
*with different keys* referencing the same value Scalar, in which case
it's behaviourally visible that the two Pairs are different objects *and*
that they share the value storage.

So it is untenable to hold that it is the Pair itself that is mutable.
The independent identity of a Scalar container is behaviourally
detectable, even if the object is not directly visible to the language.

                                                            Scalar is

precisely the difference between immutable and mutable values.

The rebinding that occurs in Pair.freeze (which I discussed in [perl
#​128955]) shows that the Pair class actually has some mutability of its
own, independent of binding to a Scalar. So Scalar is certainly not
the only way the language implements scalar mutability. There's some
other form of difference between immutable and mutable.

                                     It is the implementation of a

mutable value that can be used as a value; they are *not* general mutable
containers

I'm having difficulty discerning what distinction you're making here.
As best I can tell, your view is that the "implementation of a mutable
value" is an inherently second-class concept. That would make it pretty
much by definition a bad idea to reify it. This reasoning doesn't shed
any light on the question of whether the language's main scalar containers
should be reified.

-zefram

@p6rt
Copy link
Author

p6rt commented Aug 16, 2016

From @geekosaur

I'm trying to think how to put this better.

In effect, Scalar isn't so much a mutable container object, as it is a bit
of magic that changes the fundamental identity of the thing "containing"
it, from immutable to mutable. It's sufficiently magic that, barring .VAR,
the only way to make one is to delve into the NQP layer which lets you poke
and prod (and bend, fold, spindle, and mutilate) the guts of anything. (But
rather than actually rewriting the thing containing it, rakudo's
implementation requires that all code dealing with values call nqp​::decont.
I can see why it was done that way --- it's much cleaner than actually
mutating its parent --- but you can break stuff by forgetting to decont,
and it's not a very scalable way to deal with it. And it means you can
defeat the decont and pass mutable things where only immutable ones should
exist.)

A proper mutable container should have its own identity but expose the
value it contains without an explicit dereference. Scalar goes out of its
way to not expose its own identity, which is why the only way to get one is
.VAR --- and rakudo depends on this.

On Tue, Aug 16, 2016 at 6​:07 PM, Zefram <zefram@​fysh.org> wrote​:

Brandon Allbery via RT wrote​:

I did not mean "You created something equivalent to a mutable-key Pair".
You *precisely* created a Pair that has a mutable component, in something
that is not intended to be mutable.

That view doesn't work for the situation in [perl #​128948].

It's quite intentional there that it's possible to create a Pair with
a Scalar container in its value slot​: I didn't use .VAR. By the view
you're expressing, the situation there is precisely that the Pair is
mutable in respect of its value. The .WHICH that is presently generated
for that situation, of the "Pair|47804193992464" form, follows that view,
attributing the mutability to the Pair itself.

But I was able to create two Pairs referencing the same Scalar value
container. It's visible that they reference the same mutable storage;
the behaviour upon mutation is different from what would be seen if
each Pair had its own mutable storage. In that ticket the result
was that I had two behaviourally-identical Pairs claiming different
identity. We could paper over that by making the .WHICH value take the
mutable storage identity from the (internal) Scalar, still returning a
"Pair|47804193992464" kind of value, and pretend that the Pairs were
actually the same object. But one can just as well create two Pairs
*with different keys* referencing the same value Scalar, in which case
it's behaviourally visible that the two Pairs are different objects *and*
that they share the value storage.

So it is untenable to hold that it is the Pair itself that is mutable.
The independent identity of a Scalar container is behaviourally
detectable, even if the object is not directly visible to the language.

                                                            Scalar is

precisely the difference between immutable and mutable values.

The rebinding that occurs in Pair.freeze (which I discussed in [perl
#​128955]) shows that the Pair class actually has some mutability of its
own, independent of binding to a Scalar. So Scalar is certainly not
the only way the language implements scalar mutability. There's some
other form of difference between immutable and mutable.

                                     It is the implementation of a

mutable value that can be used as a value; they are *not* general mutable
containers

I'm having difficulty discerning what distinction you're making here.
As best I can tell, your view is that the "implementation of a mutable
value" is an inherently second-class concept. That would make it pretty
much by definition a bad idea to reify it. This reasoning doesn't shed
any light on the question of whether the language's main scalar containers
should be reified.

-zefram

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

@p6rt p6rt added the at_larry label Jan 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant