push on scalar forbidden: why? #16506

p5pRT · 2018-04-13T23:31:50Z

Migrated from rt.perl.org#133109 (status was 'open')

Searchable as RT133109$

p5pRT · 2018-04-13T23:31:50Z

From perl-diddler@tlinx.org

Created by perl-diddler@tlinx.org

If a scaler is null or an ARRAY ref, why shouldn't perl be
able to treat it as one in a context where a non-ARRAY ref would generate
an error?

ex:
my $aref;
push $aref, 1;

I'm aware an alternate way to do this was added, but perl, like many
languages often has multiple ways to do something. More so in
perl in that a major feature in perl is autovivification.

This is a perfect example of perl autovivifying an array
ref to use as an array needed by push. So why would it be
"forbidden".

Even more curious is that perl is a permissive language. If you
want to do something it will usually let you -- assuming that
the user is in a position to better know what is wanted/needed
in a specific circumstance. I remember a discussion of
perl's OO design and how perl doesn't enforce class boundaries
or the concept of 'private', by _default_. While extensions
can can add such things, nothing is forbidden. So why would
something that used to work be made "forbidden"?

Can perl be allowed to do the right thing when perl already knows
what it is?

Thanks,

Perl Info


Flags:
    category=core
    severity=high

Site configuration information for perl 5.26.1:

Configured by ASSI at Tue Sep 26 18:50:45 CEST 2017.

Summary of my perl5 (revision 5 version 26 subversion 1) configuration:
   
  Platform:
    osname=cygwin
    osvers=2.9.0(0.31853)
    archname=x86_64-cygwin-threads-multi
    uname='cygwin_nt-6.3 cygwin 2.9.0(0.31853) 2017-09-12 10:18 x86_64 cygwin '
    config_args='-des -Dprefix=/usr -Dmksymlinks -Darchname=x86_64-cygwin-threads -Dlibperl=cygperl5_26.dll -Dcc=gcc -Dld=g++ -Accflags=-ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.26.1-1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.26.1=/usr/src/debug/perl-5.26.1-1 -fwrapv'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.26.1-1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.26.1=/usr/src/debug/perl-5.26.1-1 -fwrapv -fno-strict-aliasing -fstack-protector-strong -D_FORTIFY_SOURCE=2'
    optimize='-O3'
    cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.26.1-1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.26.1=/usr/src/debug/perl-5.26.1-1 -fwrapv -fno-strict-aliasing -fstack-protector-strong'
    ccversion=''
    gccversion='6.4.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags =' -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong'
    libpth=/usr/lib
    libs=-lpthread -lgdbm -ldb -ldl -lcrypt -lgdbm_compat
    perllibs=-lpthread -ldl -lcrypt
    libc=/usr/lib/libcygwin.a
    so=dll
    useshrplib=true
    libperl=cygperl5_26.dll
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' --shared  -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong'

Locally applied patches:
    Cygwin: README
    Cygwin: use auto-image-base instead of fixed DLL base address
    Cygwin: modify hints
    Cygwin: Configure correct libsearch
    Cygwin: Configure correct libpth
    Cygwin: Win32 correct UTF8 handling
    Perl: File-Path-2.14 (fixes CVE2017-6512)


@INC for perl 5.26.1:
    /usr/local/lib/perl5/site_perl/5.26/x86_64-cygwin-threads
    /usr/local/share/perl5/site_perl/5.26
    /usr/lib/perl5/vendor_perl/5.26/x86_64-cygwin-threads
    /usr/share/perl5/vendor_perl/5.26
    /usr/lib/perl5/5.26/x86_64-cygwin-threads
    /usr/share/perl5/5.26


Environment for perl 5.26.1:
    CYGWIN=system nodosfilewarning winsymlinks:native export
    HOME=/Users/l.B
    LANG (unset)
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_US.UTF-8
    LC_MESSAGES=C
    LC_MONETARY=C
    LC_NUMERIC=C
    LC_TIME=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=.:.:/usr/bin:/sbin:/prog64/vim/current:/Prog64/VanDyke Software/SecureCRT:/Prog/Sysinternals/cmd:/Prog/Sysinternals:/Prog/Common Files/DivX Shared:/Prog/QuickTime:/Prog/QuickTime/QTSystem:/Prog/NVIDIA Corporation/PhysX/Common:/Prog64/Razer Chroma SDK/bin:/Prog/Razer Chroma SDK/bin:/Windows/system32:/Windows:/Windows/System32/Wbem:/Prog64/Process Lasso:/usr/sbin:/Users/l.B/bin:/Users/l.B/bin/lib:/usr/local/bin:/bin:/etc/local/func_lib
    PERL_BADLANG (unset)
    SHELL=C:/Bin/Bash.exe

p5pRT · 2018-04-13T23:43:47Z

From @Grinnz

This was an experimental feature added in 5.14: https://metacpan.org/pod/perl5140delta#Array-and-hash-container-functions-accept-references and then removed in 5.24: https://metacpan.org/pod/perl5240delta#The-autoderef-feature-has-been-removed . One of the biggest problems with it is that there's no correct way to handle an object with both hash and array overloads defined.

p5pRT · 2018-04-13T23:43:47Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2018-04-13T23:46:01Z

From @Grinnz

On Fri, 13 Apr 2018 16:43:47 -0700, grinnz@gmail.com wrote:

This was an experimental feature added in 5.14:
https://metacpan.org/pod/perl5140delta#Array-and-hash-container-
functions-accept-references and then removed in 5.24:
https://metacpan.org/pod/perl5240delta#The-autoderef-feature-has-been-
removed . One of the biggest problems with it is that there's no
correct way to handle an object with both hash and array overloads
defined.

Also, adding an @ to dereference it is a very simple workaround, and will autovivify an undefined scalar used in this manner.

p5pRT · 2018-04-14T00:32:21Z

From @philiprbrenan

The initial experimental form was a good idea but too broad because of the
interpretation of:

for($a) {...}

But when it was removed the removal was perhaps too broad as well and
perhaps should not have been removed from pop, push, shift, unshift where
it seems plausible that an extra @ can be safely added. I thought that
perhaps Linda's scheme would fail on:

my $a;
for(keys $a) {...}

but in this case the point is moot so the scheme survives. And in cases
like:

my %a = (0=>undef);
push @{$a{0}}, 1;
say STDERR dump(%a);

it would make code much more readable.

If Linda's scheme can be shown to be safe and unambiguous on all cases of
pop, push, shift, unshift then I would like to see this capability returned
to Perl because it is more concise.

p5pRT · 2018-04-14T00:52:42Z

From @Grinnz

On Fri, 13 Apr 2018 17:32:21 -0700, philiprbrenan@gmail.com wrote:

And in cases
like:

my %a = (0=>undef);
push @{$a{0}}, 1;
say STDERR dump(%a);

it would make code much more readable.

If Linda's scheme can be shown to be safe and unambiguous on all cases of
pop, push, shift, unshift then I would like to see this capability returned
to Perl because it is more concise.

Just for reference, this can also be written as push $a{0}->@* on Perl 5.24+ (or slightly earlier with the experimental feature). That might be more palatable to some when it comes to long dereference chains.

p5pRT · 2018-04-14T02:02:04Z

From perl-diddler@tlinx.org

Dan Book via RT wrote:

One of the biggest problems with it is that there's no correct way to handle an object with both hash and array overloads defined.

I disagree, in so much as if there is ambiguity, it is IMO, inherently an
error (incomplete specification) -- the computer cannot infer proper
function by context.

p5pRT · 2018-04-14T02:31:14Z

From perl-diddler@tlinx.org

Philip R Brenan via RT wrote:

The initial experimental form was a good idea but too broad because of the
interpretation of:

for($a) {...}

??
In an earlier writeup of this, I deleted but should have left
a key phrase:

If there is no way to interpret the expression without an error
being generated, (a '@' is expected), then allow it to be processed
as a reference to an array.

However, the above is legal perl code even if '$a' is
"undef", as it should be processed as "for(undef){...}".

Since this is legal:

for ([1,2,3],[4,5,6]) {...}

(with loop run twice: _=[1,2,3], then _=[4,5,6]).

then this must be treated the same way (and is):

for ([1,2,3]) {...}
(_=[1,2,3]). There can be no treatment of a ref as an
array or hash in that context, because the ref itself is a legal
value.

Are you saying there is some situation where somehow
that for would be treated as:
for(@$a){...}?

If so, I don't see it. I.e. I don't see how there would be
confusion. I.e. in a situation where it is value
to treat the value as a scalar OR an array, then the
scalar would take precedence.

But when it was removed the removal was perhaps too broad as well and
perhaps should not have been removed from pop, push, shift, unshift where
it seems plausible that an extra @ can be safely added. I thought that
perhaps Linda's scheme would fail on:

my $a;
for(keys $a) {...}

If '$a' is a ref to a hash:
my $a={one=>1,two=>2};
for (keys $a) {...}, then I don't see confusion, as keys is
an operator (not a bareword), that requires a hash, right?

In that case, treating it as "%$a" would seem the only valid
interpretation.

but in this case the point is moot so the scheme survives. And in cases
like:

my %a = (0=>undef);
push @{$a{0}}, 1;
say STDERR dump(%a);

it would make code much more readable.

If Linda's scheme can be shown to be safe and unambiguous on all cases of
pop, push, shift, unshift then I would like to see this capability returned
to Perl because it is more concise.

What other cases was it removed from?

I don't see a problem allowing it anywhere where:

1) it would be an error to NOT dereference the ref,
and
2) where there is no language ambiguity.

I'm sure this was thoroughly 'hashed' out at the time, so
maybe there are cases I'm not thinking of where I could
be enlightened? That said, though, I feel there _can't_
be any such cases -- because of the above preconditions.

If there was ambiguity before, and it was allowed,
how was it that it was not flagged as an error?

And if the code was already "legal", then dereferencing
shouldn't be considered. (i.e. "for ($a) {...}" ).

So what am I missing?
Thanks!
Linda

p5pRT · 2018-04-15T01:44:45Z

From perl-diddler@tlinx.org

Philip R Brenan wrote:

So far you do not appear to be missing anything. *for($a)* was
ambiguous under the original implementation

It should never be "ambiguous". According to the language, it
should always resolve to a single value. In shell, using:

prog='P "perl-%s:", $];
my $aref = [ qw(one two three) ];
for ($aref) {P "%s", $_}'

then (in shell & various perl-bin dirs):

(probs building 5.6.2)

perl-5.8.9/usr/bin> PATH=$PWD:$PATH perl -MP -CSA -e "$prog"
perl-5.008009:
["one", "two", "three"]

perl-5.10.1/usr/bin> PATH=$PWD:$PATH perl -MP -CSA -e "$prog"
perl-5.010001:
["one", "two", "three"]

perl-5.12.5/usr/bin> PATH=$PWD:$PATH perl -MP -CSA -e "$prog"
perl-5.012005:
["one", "two", "three"]

(had probs building 5.14)

perl -MP -CSA -e "$prog"
perl-5.016003:
["one", "two", "three"]

(5.18-5.22 not built/tested)

perl-5.24.0/usr/bin> PATH=$PWD:$PATH perl -MP -CSA -e "$prog"
perl-5.024000:
["one", "two", "three"]

/perl-5.26.0/usr/bin> PATH=$PWD:$PATH perl -MP -CSA -e "$prog"
perl-5.026000:
["one", "two", "three"]

(Note: I have 'P' installed from cpan in the versions
tested)

Except for the untested versions, it seems like there is
no ambiguity in "for ($aref) {...}"

Was the problem in one of the untested versions?

p5pRT · 2018-04-18T00:30:48Z

From perl-diddler@tlinx.org

Is perl in a nice version control system where a patch or code-push could
be backed out if one wanted?

Or if a problem developed, could it be bisected like the kernel dev tree?

p5pRT · 2018-04-18T02:57:04Z

From @doughera88

On Tue, Apr 17, 2018 at 05:29:58PM -0700, L A Walsh wrote:

Is perl in a nice version control system where a patch or code-push could
be backed out if one wanted?

Yes, see perldoc pod/perlhack.pod

Or if a problem developed, could it be bisected like the kernel dev tree?

Yes, see perldoc Porting/bisect-runner.pl

--
Andy Dougherty doughera@lafayette.edu

p5pRT · 2018-04-19T16:28:45Z

From @xsawyerx

Hi Linda,

Whether there are situations that are unambiguous for the compiler or
not, there are additional reasons to removing auto deref. This includes
ambiguity for the developers as well - and even if that is arguable, the
confusion for the developer isn't. Providing an auto deref for some
operations but not others is far worse than supporting it in full.

I think this ticket should be respectfully rejected.

On 04/14/2018 01:31 AM, Linda Walsh (via RT) wrote:

# New Ticket Created by Linda Walsh
# Please include the string: [perl #133109]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133109 >

This is a bug report for perl from perl-diddler@tlinx.org,
generated with the help of perlbug 1.40 running under perl 5.26.1.

-----------------------------------------------------------------
[Please describe your issue here]

If a scaler is null or an ARRAY ref, why shouldn't perl be
able to treat it as one in a context where a non-ARRAY ref would generate
an error?

ex:
my $aref;
push $aref, 1;

I'm aware an alternate way to do this was added, but perl, like many
languages often has multiple ways to do something. More so in
perl in that a major feature in perl is autovivification.

This is a perfect example of perl autovivifying an array
ref to use as an array needed by push. So why would it be
"forbidden".

Even more curious is that perl is a permissive language. If you
want to do something it will usually let you -- assuming that
the user is in a position to better know what is wanted/needed
in a specific circumstance. I remember a discussion of
perl's OO design and how perl doesn't enforce class boundaries
or the concept of 'private', by _default_. While extensions
can can add such things, nothing is forbidden. So why would
something that used to work be made "forbidden"?

Can perl be allowed to do the right thing when perl already knows
what it is?

Thanks,

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=high
---
Site configuration information for perl 5.26.1:

Configured by ASSI at Tue Sep 26 18:50:45 CEST 2017.

Summary of my perl5 (revision 5 version 26 subversion 1) configuration:

Platform:
osname=cygwin
osvers=2.9.0(0.31853)
archname=x86_64-cygwin-threads-multi
uname='cygwin_nt-6.3 cygwin 2.9.0(0.31853) 2017-09-12 10:18 x86_64 cygwin '
config_args='-des -Dprefix=/usr -Dmksymlinks -Darchname=x86_64-cygwin-threads -Dlibperl=cygperl5_26.dll -Dcc=gcc -Dld=g++ -Accflags=-ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.26.1-1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.26.1=/usr/src/debug/perl-5.26.1-1 -fwrapv'
hint=recommended
useposix=true
d_sigaction=define
useithreads=define
usemultiplicity=define
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='gcc'
ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.26.1-1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.26.1=/usr/src/debug/perl-5.26.1-1 -fwrapv -fno-strict-aliasing -fstack-protector-strong -D_FORTIFY_SOURCE=2'
optimize='-O3'
cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -ggdb -O2 -pipe -Wimplicit-function-declaration -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/build=/usr/src/debug/perl-5.26.1-1 -fdebug-prefix-map=/mnt/share/maint/perl.x86_64/src/perl-5.26.1=/usr/src/debug/perl-5.26.1-1 -fwrapv -fno-strict-aliasing -fstack-protector-strong'
ccversion=''
gccversion='6.4.0'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='g++'
ldflags =' -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong'
libpth=/usr/lib
libs=-lpthread -lgdbm -ldb -ldl -lcrypt -lgdbm_compat
perllibs=-lpthread -ldl -lcrypt
libc=/usr/lib/libcygwin.a
so=dll
useshrplib=true
libperl=cygperl5_26.dll
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=dll
d_dlsymun=undef
ccdlflags=' '
cccdlflags=' '
lddlflags=' --shared -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,--enable-auto-image-base -fstack-protector-strong'

Locally applied patches:
Cygwin: README
Cygwin: use auto-image-base instead of fixed DLL base address
Cygwin: modify hints
Cygwin: Configure correct libsearch
Cygwin: Configure correct libpth
Cygwin: Win32 correct UTF8 handling
Perl: File-Path-2.14 (fixes CVE2017-6512)

---
@INC for perl 5.26.1:
/usr/local/lib/perl5/site_perl/5.26/x86_64-cygwin-threads
/usr/local/share/perl5/site_perl/5.26
/usr/lib/perl5/vendor_perl/5.26/x86_64-cygwin-threads
/usr/share/perl5/vendor_perl/5.26
/usr/lib/perl5/5.26/x86_64-cygwin-threads
/usr/share/perl5/5.26

---
Environment for perl 5.26.1:
CYGWIN=system nodosfilewarning winsymlinks:native export
HOME=/Users/l.B
LANG (unset)
LANGUAGE (unset)
LC_COLLATE=C
LC_CTYPE=en_US.UTF-8
LC_MESSAGES=C
LC_MONETARY=C
LC_NUMERIC=C
LC_TIME=C
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=.:.:/usr/bin:/sbin:/prog64/vim/current:/Prog64/VanDyke Software/SecureCRT:/Prog/Sysinternals/cmd:/Prog/Sysinternals:/Prog/Common Files/DivX Shared:/Prog/QuickTime:/Prog/QuickTime/QTSystem:/Prog/NVIDIA Corporation/PhysX/Common:/Prog64/Razer Chroma SDK/bin:/Prog/Razer Chroma SDK/bin:/Windows/system32:/Windows:/Windows/System32/Wbem:/Prog64/Process Lasso:/usr/sbin:/Users/l.B/bin:/Users/l.B/bin/lib:/usr/local/bin:/bin:/etc/local/func_lib
PERL_BADLANG (unset)
SHELL=C:/Bin/Bash.exe

p5pRT · 2018-04-19T22:09:39Z

From perl-diddler@tlinx.org

Sawyer X via RT wrote:

Hi Linda,

Whether there are situations that are unambiguous for the compiler or
not, there are additional reasons to removing auto deref. This includes
ambiguity for the developers as well - and even if that is arguable, the
confusion for the developer isn't. Providing an auto deref for some
operations but not others is far worse than supporting it in full.

I think this ticket should be respectfully rejected.

Might I ask, how it is confusing for a developer? Most languages
have no typing indicator -- at least perl will flag an error if the
variable being dereferenced is of the wrong type:

perl -e 'my $a={};
push $a,1;'
Not an ARRAY reference at -e line 2.

I'm not sure what you mean by providing for auto-deref in some
operations but not others. It would be implemented in all operations
where NOT auto-dereferencing would be an error.

How would that be unclear for a perl developer who probably
(right or wrong) knows what would be a syntax error and what would not
be. It's an extension into an area that would previously have been
*unused* -- because all attempts to use invalid syntax, before, would
have been flagged with an error.

But to make it easier, it looks like perfunc holds the answer. It
lists functions that work on ARRAYS and functions that work on HASHes.

I may have missed something, but the dereference feature would only
work where the language would require an ARRAY or a HASH. If the language
requires an ARRAY or a HASH, the user would be able to supply that
with a literal ARRAY or HASH, OR a reference to an ARRAY or HASH.

If you want to complain about it working in some places and not
others -- it's based on the fact that the language only allows
ARRAYs and HASHes in various locations.

The manpage shows:
Functions for real @ARRAYs
"each", "keys", "pop", "push", "shift", "splice", "unshift",
"values"

Functions for real %HASHes
"delete", "each", "exists", "keys", "values"

As far as I can tell, these should be the entirety of functions
that have required a '@' or '%' as part of the syntax. In those cases,
if a SCALAR holds a reference to an ARRAY or a HASH, then the undereferenced
scalar could be used without the need for adding extraneous characters
because
perl knows (and will flag an error) if a scalar does or doesn't contain
a reference to the correct type.

Similarly, the open call can take a scalar with a value of 'undef'
and can store a file reference in that scalar. Whether or not that will
work is entirely dependent on the contents of that var -- if it contains
any other type of reference or value, it is specification says it should
fail.

Similarly, in any location where an ARRAY or HASH is required,
that requirement can be satisfied only with a var of the correct type
whether it be literal or a reference.

The feature was enabled for 2 years, and triggered
warnings for the next 3 years before it was removed for no reason that
I've been able to find out. Even the release notes claim that it was
an experiment that "failed", but give no details about how or why it
failed only that it was replaced with an optional syntax.

When one of the main complaints against perl (as being read by others or
others learning it) has been that it looks like 'line noise'. Adding
postfix sigils in addition to the more concise prefix sigils that
were already there, seems like the wrong direction to go. If anything,
perl should strongly think about going "sigil-optional" where there would
be no ambiguity(no vars of the same name and would require being
pre-defined or referenced), but that's somewhat of a different issue.

I don't see this being some vague nebulous issue the way some
seem to be painting it. In some discussions about this some expressed
what they were told -- that some things were ambiguous. I really
wanted to see an ambiguous case, but so far, no one has been able
to present one (perhaps I didn't understand the extent to which
the original concept had been applied).

Perhaps Sawyer is thinking about something other than what this
is?

p5pRT · 2018-04-19T22:16:12Z

From @Grinnz

On Thu, Apr 19, 2018 at 6:09 PM, L A Walsh <perl-diddler@tlinx.org> wrote:
...

The manpage shows:
Functions for real @ARRAYs
"each", "keys", "pop", "push", "shift", "splice", "unshift",
"values"
  Functions for real %HASHes
      "delete"\, "each"\, "exists"\, "keys"\, "values"

each, keys, and values work on both arrays and hashes. As does exists and
delete, technically.

-Dan

p5pRT · 2018-04-20T01:37:49Z

From perl-diddler@tlinx.org

Dan Book via RT wrote:

On Thu, Apr 19, 2018 at 6:09 PM, L A Walsh <perl-diddler@tlinx.org> wrote:
...
The manpage shows:
Functions for real @ARRAYs
"each", "keys", "pop", "push", "shift", "splice", "unshift",
"values"
  Functions for real %HASHes
      "delete"\, "each"\, "exists"\, "keys"\, "values"
each, keys, and values work on both arrays and hashes. As does exists and
delete, technically.

-Dan

Yes...for dereferencing to work in this case, the scalar would have to
hold either an ARRAY or HASH reference. It really makes no sense to try
to use "each", "keys" nor "values" on a a scalar that has value 'undef'.
It would make no sense and would (should) generate an error as it does
now.

Only functions that require an ARRAY *XOR* HASH could take undef. Off
hand, I
can't think of a function that would autovivify a HASH.

Does that clarify the intent of my proposal? It may be the case that
my proposal is not the same as what was implemented before (I never used
it), but only a subset that is deterministic.

p5pRT · 2018-04-21T15:37:56Z

From @iabyn

On Thu, Apr 19, 2018 at 06:37:26PM -0700, L A Walsh wrote:

Dan Book via RT wrote:
On Thu, Apr 19, 2018 at 6:09 PM, L A Walsh <perl-diddler@tlinx.org> wrote:
...
The manpage shows:
Functions for real @ARRAYs
"each", "keys", "pop", "push", "shift", "splice", "unshift",
"values"
  Functions for real %HASHes
      "delete"\, "each"\, "exists"\, "keys"\, "values"
each, keys, and values work on both arrays and hashes. As does exists and
delete, technically.

-Dan
Yes...for dereferencing to work in this case, the scalar would have to
hold either an ARRAY or HASH reference. It really makes no sense to try
to use "each", "keys" nor "values" on a a scalar that has value 'undef'.
It would make no sense and would (should) generate an error as it does
now.

But when you have a scalar reference which is overloaded, perl can't know
what overload method to call in those cases. For example:

use overload
'@{}' => sub { print "array deref\n" ; return \@real_a },
'%{}' => sub { print "hash deref\n" ; return \%real_h },
;

my $r = bless [];

push $r, 3; # prints "array derdef", modifies @real_a
$r->{foo} = 'bar'; # prints "hash derdef", modifies %real_h
@k = keys $r; # what overload method does this call?

What should perl do in the last case?

--
The optimist believes that he lives in the best of all possible worlds.
As does the pessimist.

p5pRT · 2018-04-21T17:17:44Z

From @philiprbrenan

Presumably it would return () as $r is still a reference to an empty array
despite the ingenious use of side effects implemented via operator
overloading to manipulate other arrays and hashes and thereby give the
impression that $r is both an array and a hash?

If push $r, 3 were always interpreted identically to push @$r, 3 we would
expect to get:

use Data::Dump qw(dump);
my @real_a;
my %real_h;
use overload
'@{}' => sub { print "array deref\n" ; return \@real_a },
'%{}' => sub { print "hash deref\n" ; return \%real_h },
;

my $r = bless [];

push *@$r*, 3;
say STDERR "AAAA ", dump($r); # AAAA bless([3], "main")
say STDERR "BBBB ", dump(\@real_a); # BBBB [3]

I.e. $r is not empty, which illustrates that push $r and push @$r would not
always produce identical results under the proposed scheme in the presence
of operator loading.

I urge that both of these problems can be overcome by Linda's statement of
the the scheme:

*Push gets its array either directly or via a reference. *In this example:

push $r, 3

would affect only $r, not @real_a, because no array dereference would be
implied or performed.

On Sat, Apr 21, 2018 at 4:37 PM, Dave Mitchell <davem@iabyn.com> wrote:

On Thu, Apr 19, 2018 at 06:37:26PM -0700, L A Walsh wrote:
Dan Book via RT wrote:
On Thu, Apr 19, 2018 at 6:09 PM, L A Walsh <perl-diddler@tlinx.org>
wrote:
...
The manpage shows:
Functions for real @ARRAYs
"each", "keys", "pop", "push", "shift", "splice",
"unshift",
"values"
  Functions for real %HASHes
      "delete"\, "each"\, "exists"\, "keys"\, "values"
each, keys, and values work on both arrays and hashes. As does exists
and
delete, technically.

-Dan
Yes...for dereferencing to work in this case, the scalar would have to
hold either an ARRAY or HASH reference. It really makes no sense to try
to use "each", "keys" nor "values" on a a scalar that has value 'undef'.
It would make no sense and would (should) generate an error as it does
now.
But when you have a scalar reference which is overloaded, perl can't know
what overload method to call in those cases. For example:
use overload
    '@&#8203;\{\}' => sub \{ print "array deref\\n" ; return \\@&#8203;real\_a \}\,
    '%\{\}' => sub \{ print "hash  deref\\n" ; return \\%real\_h \}\,
    ;


my $r = bless \[\];

push $r\, 3;        \# prints "array derdef"\, modifies @&#8203;real\_a
$r\->\{foo\} = 'bar'; \# prints "hash  derdef"\, modifies %real\_h
@&#8203;k = keys $r;      \# what overload method does this call?
What should perl do in the last case?

--
The optimist believes that he lives in the best of all possible worlds.
As does the pessimist.

--
Thanks,

Phil <http://www.appaapps.com/howToWriteAnApp.html>

Philip R Brenan <http://www.appaapps.com/howToWriteAnApp.html>

p5pRT · 2018-04-21T19:57:25Z

From @iabyn

On Sat, Apr 21, 2018 at 06:17:09PM +0100, Philip R Brenan wrote:

*Push gets its array either directly or via a reference. *In this example:

push $r, 3

would affect only $r, not @real_a, because no array dereference would be
implied or performed.

Then you've just broken overloading. For example, suppose someone writes
this innocuous method:

sub push_foo {
my ($self, $array_ref) = @_;
push $array_ref, $self->foo;
}

this method works exactly as documented and expected, until the day
someone passes an overloaded scalar as an arg to it, then finds that
the overloading is being quietly ignored.

--
That he said that that that that is is is debatable, is debatable.

p5pRT · 2018-04-21T21:10:50Z

From perl-diddler@tlinx.org

Dave Mitchell wrote:

But when you have a scalar reference which is overloaded, perl can't know
what overload method to call in those cases. For example:

use overload
    '@&#8203;\{\}' => sub \{ print "array deref\\n" ; return \\@&#8203;real\_a \}\,
    '%\{\}' => sub \{ print "hash  deref\\n" ; return \\%real\_h \}\,
    ;


my $r = bless \[\];

push $r\, 3;        \# prints "array derdef"\, modifies @&#8203;real\_a
$r\->\{foo\} = 'bar'; \# prints "hash  derdef"\, modifies %real\_h
@&#8203;k = keys $r;      \# what overload method does this call?

What should perl do in the last case?

Two answers. The first answer: it never gets that far.

PERL5OPT="" perl /tmp/ovld.pl
Not an unblessed ARRAY reference at /tmp/ovld.pl line 9.

2nd answer -- this was brought up by Dan Brook. I answered:

-------- Original Message --------
Date: Fri, 13 Apr 2018 18:47:08 -0700
From: L A Walsh

Dan Book wrote:

One of the biggest problems with it is that there's no correct
way to handle an object with both hash and array overloads defined.

I disagree, in so much as if there is ambiguity, it is IMO, inherently an
error (incomplete specification) -- the computer cannot infer proper
function by context.
---- end Message -----

In any case where there is ambiguity, the compiler must throw an error(like):

Ambiguous dereference @ line 13: Sigil required.

My suggestion was to auto-dereference references where perl knew the type
of the reference. If it reference is ambiguous, it can't know the type.

The 2nd part was to allow a scalar that holds 'undef' to be
auto-vivified where only 1 solution would work. I.e.

my $ref
push $ref, 1;

would autovivify a temporary array pointed to by '$ref', as push requires
an array.

Similarly:
my $ref;
my $val=pop $ref;

would act identically to:

my $val=pop @$ref;

Any case where perl has an ambiguous choice, must be flagged as an
error. I can't see any other option being correct.

p5pRT · 2018-04-21T22:21:58Z

From @iabyn

On Sat, Apr 21, 2018 at 02:10:05PM -0700, L A Walsh wrote:

Two answers. The first answer: it never gets that far.

PERL5OPT="" perl /tmp/ovld.pl
Not an unblessed ARRAY reference at /tmp/ovld.pl line 9.

That's because specifically to get round the problem of the overloading
ambiguity, the perl autodedef ref implementation was changed so that it
croaked on blessed refs. This made things even worse - you could never
use the new syntax in any place where there was a possibility that the
ref you get passed may have been blessed. This was a big motivation for
eliminating the feature.

2nd answer -- this was brought up by Dan Brook. I answered:
In any case where there is ambiguity, the compiler must throw an error(like):

Ambiguous dereference @ line 13: Sigil required.

My suggestion was to auto-dereference references where perl knew the type
of the reference. If it reference is ambiguous, it can't know the type.

Since perl almost never knows the type of a reference at compile-time,
how can this ever work? Can you give a more concrete example?

The 2nd part was to allow a scalar that holds 'undef' to be
auto-vivified where only 1 solution would work. I.e.

my $ref
push $ref, 1;

would autovivify a temporary array pointed to by '$ref', as push requires
an array.

Similarly:
my $ref;
my $val=pop $ref;

would act identically to:

my $val=pop @$ref;

Any case where perl has an ambiguous choice, must be flagged as an
error. I can't see any other option being correct.

Do you mean as a compile-time error or a run-time error?

--
The Enterprise successfully ferries an alien VIP from one place to another
without serious incident.
-- Things That Never Happen in "Star Trek" #7

p5pRT · 2018-04-22T02:57:10Z

From perl-diddler@tlinx.org

Dave Mitchell via RT wrote:

On Sat, Apr 21, 2018 at 02:10:05PM -0700, L A Walsh wrote:

Two answers. The first answer: it never gets that far.

PERL5OPT="" perl /tmp/ovld.pl
Not an unblessed ARRAY reference at /tmp/ovld.pl line 9.

That's because specifically to get round the problem of the overloading
ambiguity, the perl autodedef ref implementation was changed so that it
croaked on blessed refs. This made things even worse -

It would make the new perl incompatible with old code. One of the
rules in the proposed change was that "push $_,3" should be have identically
to push @$_,3 at runtime. That change broke compatibility and doesn't
really protect from the problem it was trying to fix since putting a '@'
in front overrides the restriction in the statement that died.

tperl
P "hello from perl %s", $];
my $aref;
bless $aref=[], "SomeClass";
push @$aref, 3;
P "bye world, aref = %s", $aref;
'
hello from perl 5.016003
bye world, aref = SomeClass[3] # <-- result: blessed array (CLASS[])
# w/3 pushed on

I'm guessing where this feature was really meant to "fix" things was
in this statement:
@k = keys $r;

as I think I understand it, only if the ref is in a class can it
use overloading(?), and the bless-check stopped that in:

Type of argument to keys on reference must be unblessed hashref or
arrayref at /tmp/ovld.pl line 11.

Again, I think checking if there was a bless on the ref was a bit of a
hack, since it "avoided" the problem rather than dealing with it.

In that scenario, blessed or not, the only way it could be resolved is
if perl knew '$r' was one type or the other. Since it can't know, the
programmer needs to resolve that by adding a sigil (either statically,
or at runtime using an eval:

my @k;
if ($wantref eq ARRAY) {
eval {@k = keys @$r};
} else {
eval {@k = keys %$r};
}

or something similar. If you are looking for a compile time
error message, like the primitive syntax check of push/pop @[$]ref
forcing '@', it can't really happen. Perl is a loosely typed language.
It's usually some special case where it can type check at 'compile time'.
But those special cases also become hard-coded limitations that can
prohibit other options.

"

This made things even worse - you could never
use the new syntax in any place where there was a possibility that the
ref you get passed may have been blessed. This was a big motivation for
eliminating the feature.

2nd answer -- this was brought up by Dan Brook. I answered:
In any case where there is ambiguity, the compiler must throw an error(like):

Ambiguous dereference @ line 13: Sigil required.

My suggestion was to auto-dereference references where perl knew the type
of the reference. If it reference is ambiguous, it can't know the type.

Since perl almost never knows the type of a reference at compile-time,
how can this ever work? Can you give a more concrete example?

It works for giving a runtime error just like:

my $r={};
push @$r;

does now. It can't be detected until runtime if you want to use
use runtime references. If forcing the addition of '@' is important,
perhaps that could be toggled via a pragma, but adding '@' won't guarantee
that it will work at runtime as in the example above. The compiler
can only do static/syntax checks, which may give warm fuzzies, but may
mislead some people if they think it prevents runtime errors.

Adding the "overload" module simply allow more options for runtime errors
and in those cases, the requirement for using explicit '@' or '%' can not
go away because it creates a runtime ambiguity that the runtime can't
resolve
and would have to flag as an error at runtime.

Do you mean as a compile-time error or a run-time error?

run-time -- just as the above example generates now.

No algorithm will work for all input. That doesn't mean you throw out
division because you can't divide by 0.

p5pRT · 2018-04-22T08:46:00Z

From @xsawyerx

I'm currently in no position to have a long discussion on this so I'll
just provide the short answer.

We tried autoderef. We removed it. We are not returning it.

On 04/20/2018 12:09 AM, L A Walsh wrote:

Sawyer X via RT wrote:

Hi Linda,

Whether there are situations that are unambiguous for the compiler or
not, there are additional reasons to removing auto deref. This includes
ambiguity for the developers as well - and even if that is arguable, the
confusion for the developer isn't. Providing an auto deref for some
operations but not others is far worse than supporting it in full.
I think this ticket should be respectfully rejected.

---
   Might I ask, how it is confusing for a developer? Most languages
have no typing indicator -- at least perl will flag an error if the
variable being dereferenced is of the wrong type:

perl -e 'my $a={};
push $a,1;'
Not an ARRAY reference at -e line 2.

   I'm not sure what you mean by providing for auto-deref in some
operations but not others. It would be implemented in all operations
where NOT auto-dereferencing would be an error.

   How would that be unclear for a perl developer who probably
(right or wrong) knows what would be a syntax error and what would not
be. It's an extension into an area that would previously have been
*unused* -- because all attempts to use invalid syntax, before, would
have been flagged with an error.

   But to make it easier, it looks like perfunc holds the answer. It
lists functions that work on ARRAYS and functions that work on HASHes.

   I may have missed something, but the dereference feature would only
work where the language would require an ARRAY or a HASH. If the
language
requires an ARRAY or a HASH, the user would be able to supply that
with a literal ARRAY or HASH, OR a reference to an ARRAY or HASH.

   If you want to complain about it working in some places and not
others -- it's based on the fact that the language only allows
ARRAYs and HASHes in various locations.
   The manpage shows:
      Functions for real @ARRAYs
          "each", "keys", "pop", "push", "shift", "splice", "unshift",
          "values"

      Functions for real %HASHes
          "delete", "each", "exists", "keys", "values"

   As far as I can tell, these should be the entirety of functions
that have required a '@' or '%' as part of the syntax. In those cases,
if a SCALAR holds a reference to an ARRAY or a HASH, then the
undereferenced
scalar could be used without the need for adding extraneous characters
because
perl knows (and will flag an error) if a scalar does or doesn't contain
a reference to the correct type.

   Similarly, the open call can take a scalar with a value of 'undef'
and can store a file reference in that scalar. Whether or not that will
work is entirely dependent on the contents of that var -- if it contains
any other type of reference or value, it is specification says it should
fail.

   Similarly, in any location where an ARRAY or HASH is required,
that requirement can be satisfied only with a var of the correct type
whether it be literal or a reference.

   The feature was enabled for 2 years, and triggered
warnings for the next 3 years before it was removed for no reason that
I've been able to find out. Even the release notes claim that it was
an experiment that "failed", but give no details about how or why it
failed only that it was replaced with an optional syntax.
When one of the main complaints against perl (as being read by others or
others learning it) has been that it looks like 'line noise'. Adding
postfix sigils in addition to the more concise prefix sigils that
were already there, seems like the wrong direction to go. If anything,
perl should strongly think about going "sigil-optional" where there would
be no ambiguity(no vars of the same name and would require being
pre-defined or referenced), but that's somewhat of a different issue.

   I don't see this being some vague nebulous issue the way some
seem to be painting it. In some discussions about this some expressed
what they were told -- that some things were ambiguous. I really
wanted to see an ambiguous case, but so far, no one has been able
to present one (perhaps I didn't understand the extent to which
the original concept had been applied).
   Perhaps Sawyer is thinking about something other than what this
is?

p5pRT · 2018-04-22T18:17:02Z

From @iabyn

On Sat, Apr 21, 2018 at 07:56:48PM -0700, L A Walsh wrote:

No algorithm will work for all input. That doesn't mean you throw out
division because you can't divide by 0.

The big difference between the two is that there's (in general) no
practical alternative to using division, while the auto-deref form can be
avoided by the trivial syntactical addition of a '@' or '%' to the source
code, plus possibly {} for precedence issues. Or use the recent posfix
forms.

So there's a very modest gain for allowing push $r, while there are many
drawbacks, requiring documentation for all the special cases, and those
special cases limiting the usefulness of the feature (like breaking
overloading).

This feature, and its removal, has already been extensively discussed;
it's problematical, and its not going to be re-added.

--
Art is anything that has a label (especially if the label is "untitled 1")

p5pRT added Severity High type-core labels Oct 19, 2019

xenu removed affects-5.26 labels Nov 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

push on scalar forbidden: why? #16506

push on scalar forbidden: why? #16506

p5pRT commented Apr 13, 2018

p5pRT commented Apr 13, 2018

p5pRT commented Apr 13, 2018

p5pRT commented Apr 13, 2018

p5pRT commented Apr 13, 2018

p5pRT commented Apr 14, 2018

p5pRT commented Apr 14, 2018

p5pRT commented Apr 14, 2018

p5pRT commented Apr 14, 2018

p5pRT commented Apr 15, 2018

p5pRT commented Apr 18, 2018

p5pRT commented Apr 18, 2018

p5pRT commented Apr 19, 2018

p5pRT commented Apr 19, 2018

p5pRT commented Apr 19, 2018

p5pRT commented Apr 20, 2018

p5pRT commented Apr 21, 2018

p5pRT commented Apr 21, 2018

p5pRT commented Apr 21, 2018

p5pRT commented Apr 21, 2018

p5pRT commented Apr 21, 2018

p5pRT commented Apr 22, 2018

p5pRT commented Apr 22, 2018

p5pRT commented Apr 22, 2018

push on scalar forbidden: why? #16506

push on scalar forbidden: why? #16506

Comments

p5pRT commented Apr 13, 2018

p5pRT commented Apr 13, 2018

From perl-diddler@tlinx.org

Created by perl-diddler@tlinx.org

p5pRT commented Apr 13, 2018

From @Grinnz

p5pRT commented Apr 13, 2018

p5pRT commented Apr 13, 2018

From @Grinnz

p5pRT commented Apr 14, 2018

From @philiprbrenan

p5pRT commented Apr 14, 2018

From @Grinnz

p5pRT commented Apr 14, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 14, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 15, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 18, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 18, 2018

From @doughera88

p5pRT commented Apr 19, 2018

From @xsawyerx

p5pRT commented Apr 19, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 19, 2018

From @Grinnz

p5pRT commented Apr 20, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 21, 2018

From @iabyn

p5pRT commented Apr 21, 2018

From @philiprbrenan

p5pRT commented Apr 21, 2018

From @iabyn

p5pRT commented Apr 21, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 21, 2018

From @iabyn

p5pRT commented Apr 22, 2018

From perl-diddler@tlinx.org

p5pRT commented Apr 22, 2018

From @xsawyerx

p5pRT commented Apr 22, 2018

From @iabyn