pp_each needs its own hash iterator #13341

p5pRT · 2013-10-10T10:59:48Z

Migrated from rt.perl.org#120172 (status was 'open')

Searchable as RT120172$

p5pRT · 2013-10-10T10:59:49Z

From chapter34@yahoo.com

Created by chapter34@yahoo.com

As I have described in my blog article:

http://technicalprose.blogspot.co.uk/2013/10/perl-each-function-is-not-re-entrant.html

the 'each' function is not re-entrant because it relies on hv_iternext()
to get an iterator for a hash. This should be fixed so that 'each' uses
its own local scope iterator.

The following code should work:

#!/usr/bin/perl
use warnings;
use strict;

my %global_hash;
my $max_depth = 3;

sub recurse
{
my ($depth, $hash) = @_;
while (my ($key, $val) = each %$hash) {
print "|" . ("-" x ($depth * 4)) . " $key$val\n";
recurse($depth+1, $hash) if $depth < $max_depth;
}
}

$global_hash{a} = 1;
$global_hash{b} = 2;

recurse(0, \%global_hash);

....and it would work if 'each' had its own iterator instead of using
one tied to the hash. Instead, one has to do this:

my %hashcopy = %$hash;
while (my ($key, $val) = each %hashcopy) {

or:

for my $key (keys %$hash) {
my $val = $hash->{$key};

Perl Info


Flags:
category=core
severity=low

Site configuration information for perl 5.14.2:

Configured by zbanai at Tue Oct  4 11:54:09 BST 2011.

Summary of my perl5 (revision 5 version 14 subversion 2) configuration:

Platform:
osname=solaris, osvers=2.10, archname=i86pc-solaris-thread-multi
uname='sunos oy605c1n6 5.10 generic_141415-08 i86pc i386 i86pc '
config_args='-des -Dusemymalloc=n -Duselargefiles -Ud_flock -Accflags=-DPERL_DISABLE_PMC -Duseithreads -Dcc=cc -Doptimize=-O -Ui_db -Ui_gdbm -Duselargefiles -Dafsroot=/ms -Darchlib=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/lib/perl5 -Darchlibexp=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/lib/perl5 -Dbin=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/bin -Dbinexp=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/bin -Dccflags=-DAPPLLIB_EXP="/ms/dist/perl5/VERS/5.14.2-0-core/lib/perl5:/ms/dist/perl5/VERS/5.14-core/lib/perl5" -Dinstallarchlib=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos..5.10/lib/perl5 -Dinstallbin=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos.5.10/bin -Dinstallman1dir=//ms/dev/perl5/core/5.14.2-0/install/..exec/ia32.sunos.5.10/man/man1 -Dinstallman3dir=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos.5.10/man/man3 -Dinstallprivlib=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos.5.10/lib/perl5 -Dinstallscript=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos.5.10/bin -Dinstallsitearch=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos.5.10/lib/perl5 -Dinstallsitelib=//ms/dev/perl5/core/5.14.2-0/install/.exec/ia32.sunos.5..10/lib/perl5 -Dman1dir=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/man/man1 -Dman1direxp=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/man/man1 -Dman3dir=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/man/man3 -Dman3direxp=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/man/man3 -Dpager=/ms/dist/fsf/bin/less -Dperladmin=perlcore@ms.com -Dperlpath=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/bin/perl -Dprefix=/ms/dist/perl5 -Dprefixexp=/ms/dist/perl5 -Dprivlib=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/lib/perl5 -Dprivlibexp=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5..10/lib/perl5 -Dscriptdir=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/bin -Dscriptdirexp=//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/bin -Dstartperl=#!//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/bin/perl'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -DAPPLLIB_EXP="/ms/dist/perl5/VERS/5.14.2-0-core/lib/perl5:/ms/dist/perl5/VERS/5.14-core/lib/perl5" -DPERL_DISABLE_PMC -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV',
optimize='-O',
cppflags='-D_REENTRANT -DAPPLLIB_EXP="/ms/dist/perl5/VERS/5.14.2-0-core/lib/perl5:/ms/dist/perl5/VERS/5.14-core/lib/perl5" -DPERL_DISABLE_PMC -I/usr/local/include'
ccversion='Sun C 5.8 Patch 121016-07 2007/10/03', gccversion='', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/lib -L/usr/ccs/lib -L/ms/dist/3rd/PROJ/SUNWspro/11.1-0/.exec/@sys/SUNWspro/prod/lib -L/lib -L/usr/local/lib '
libpth=/usr/lib /usr/ccs/lib /ms/dist/3rd/PROJ/SUNWspro/11.1-0/.exec/@sys/SUNWspro/prod/lib /lib /usr/local/lib
libs=-lsocket -lnsl -ldl -lm -lpthread -lc
perllibs=-lsocket -lnsl -ldl -lm -lpthread -lc
libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
cccdlflags='-KPIC', lddlflags='-G -L/usr/lib -L/usr/ccs/lib -L/ms/dist/3rd/PROJ/SUNWspro/11.1-0/.exec/@sys/SUNWspro/prod/lib -L/lib -L/usr/local/lib'

Locally applied patches:



@INC for perl 5.14.2:
/ms/dist/perl5/VERS/5.14.2-0-core/lib/perl5
/ms/dist/perl5/VERS/5.14-core/lib/perl5
//ms/dist/perl5/PROJ/core/5.14.2-0/.exec/ia32.sunos.5.10/lib/perl5
.


Environment for perl 5.14.2:
HOME=/v/global/user/b/ba/bannimar
LANG (unset)
LANGUAGE (unset)
LC_ALL=C
LD_LIBRARY_PATH=/ms/dist/storage/PROJ/widesky/prod/shlib:/opt/VRTSvcs/lib
LOGDIR (unset)
PATH=/v/global/user/b/ba/bannimar/bin:/usr/bin:/sbin:/usr/sbin://ms/dist/sec/PROJ/openssh/prod/bin://ms/dist/fsf/PROJ/make/prod/bin:/ms/dist/fsf/PROJ/gnuemacs/23.3/exec/bin://ms/dist/afs/PROJ/vms/beta/common/bin://ms/dist/afs/PROJ/vms/beta/common/sbin:/ms/dist/unixops/PROJ/ddt/beta/bin://ms/dist/fsf/PROJ/git/prod-1.8/bin:/ms/dist/aquilon/PROJ/aqd/prod/bin:/ms/dist/aquilon/PROJ/aqdw/prod/bin://ms/dist/python/PROJ/core/2.7.3-64/bin://ms/dist/perl5/PROJ/core/5.10/bin://ms/dist/perl5/bin://ms/dist/fsf/PROJ/mutt/incr/bin://ms/dist/fsf/PROJ/mutt/1.5.13/bin:/ms/dist/fsf/PROJ/bash/4.1/bin:/ms/dist/storage/PROJ/sandy/beta/common/sbin:/ms/dist/aurora/PROJ/astro/prod/common/bin:/ms/dist/storage/PROJ/widesky/prod/bin:/ms/dist/storage/PROJ/infra/prod/bin:/ms/dist/storage/sbin:/ms/dist/storage/bin:/v/global/user/b/ba/bannimar/bin:/ms/dist/aurora/bin:/usr/local/bin:/usr/bin:/bin:/usr/ccs/bin:/usr/ucb:/ms/dist/perl5/bin:/ms/dist/fsf/bin:/ms/dist/afs/bin:/usr/X11R6/bin:/usr/openwin/bin:.:/ms/dist/unixops/bin:/ms/dist/aurora/sbin:/ms/dist/fsf/PROJ/xmlstarlet/1.3.0/bin:/ms/dist/perl5/PROJ/Perl-Critic/1.116/bin:/ms/dist/msjava/PROJ/sunjdk/1.6.0_31/exec/bin:/ms/dist/fsf/PROJ/git/prod-1.8/exec/bin:/ms/dist/aquilon/PROJ/tellme/prod/bin:/sbin:/opt/VRTSllt:/opt/VRTSvcs/bin:/ms/dist/ha/PROJ/msvcs/prod/scripts:/ms/dist/ha/PROJ/utils/incr/bin:/ms/dist/ha/PROJ/signoff/prod/bin:/ms/dist/ha/PROJ/aqvcs/prod/bin:/ms/dist/fsf/PROJ/git/master/bin:/ms/dist/elfms/PROJ/panc/prod/bin
PERL_BADLANG (unset)
SHELL=/bin/ksh



--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.=

p5pRT · 2013-10-11T07:09:41Z

From @demerphq

On 10 October 2013 12:59, Mark R. Bannister <perlbug-followup@perl.org> wrote:

# New Ticket Created by Mark R. Bannister
# Please include the string: [perl #120172]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=120172 >

This is a bug report for perl from chapter34@yahoo.com,generated with the help of perlbug 1.39 running under perl 5.14.2.

-----------------------------------------------------------------
[Please describe your issue here]

As I have described in my blog article:

http://technicalprose.blogspot.co.uk/2013/10/perl-each-function-is-not-re-entrant.html

the 'each' function is not re-entrant because it relies on hv_iternext()
to get an iterator for a hash. This should be fixed so that 'each' uses
its own local scope iterator.

The following code should work:

#!/usr/bin/perl
use warnings;
use strict;

my %global_hash;
my $max_depth = 3;

sub recurse
{
my ($depth, $hash) = @_;
while (my ($key, $val) = each %$hash) {
print "|" . ("-" x ($depth * 4)) . " $key$val\n";
recurse($depth+1, $hash) if $depth < $max_depth;
}
}

$global_hash{a} = 1;
$global_hash{b} = 2;

recurse(0, \%global_hash);

....and it would work if 'each' had its own iterator instead of using
one tied to the hash. Instead, one has to do this:

my %hashcopy = %$hash;
while (my ($key, $val) = each %hashcopy) {

or:

for my $key (keys %$hash) {
my $val = $hash->{$key};

We don't normally consider this a bug. More a mis-feature. It *is*
documented behavior.

And, yes, we pretty much all agree that the behavior makes each() less useful.

If anything this is a feature request, that has been made many times,
suggesting it is difficult to do and unlikely to be implemented.

But heck, maybe someone will give it a crack one day.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2013-10-11T07:09:42Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2013-10-11T08:14:09Z

From chapter34@yahoo.com

On Fri Oct 11 00:09:41 2013, demerphq wrote:

We don't normally consider this a bug. More a mis-feature. It *is*
documented behavior.

And, yes, we pretty much all agree that the behavior makes each() less
useful.

If anything this is a feature request, that has been made many times,
suggesting it is difficult to do and unlikely to be implemented.

But heck, maybe someone will give it a crack one day.

Yves

Sorry, so this is a duplicate ticket? I did a search before I opened it
and didn't find a ticket about this issue already.

At a cursory glance it doesn't look difficult to fix, so which specific
part of the solution do you think is difficult to implement and why?

Thanks,
Mark.

p5pRT · 2013-10-11T21:12:12Z

From @mauke

On 11.10.2013 10:14, Mark R. Bannister via RT wrote:

On Fri Oct 11 00:09:41 2013, demerphq wrote:

We don't normally consider this a bug. More a mis-feature. It *is*
documented behavior.

And, yes, we pretty much all agree that the behavior makes each() less
useful.

If anything this is a feature request, that has been made many times,
suggesting it is difficult to do and unlikely to be implemented.

But heck, maybe someone will give it a crack one day.

Yves

Sorry, so this is a duplicate ticket? I did a search before I opened it
and didn't find a ticket about this issue already.

At a cursory glance it doesn't look difficult to fix, so which specific
part of the solution do you think is difficult to implement and why?

Your proposed solution only moves the problem. It would make 'each' work
like '..' (the flip-flop version). What would happen with code like this?

my @hashes = (
{ "a" .. "d" },
{ 1 .. 4 },
{ "Y" => "Z" },
);
while (my ($k, $v) = each %{$hashes[0]}) {
print "$k: $v\n";
push @hashes, shift @hashes;
}

Implicit state is bad in general. But given the choice between state
attached to a variable and state attached to an op, I'd take the variable.

(Also, changing the existing 'each' operator would break all code
everywhere.)

--
Lukas Mai <plokinom@gmail.com>

p5pRT · 2013-10-11T23:15:59Z

From @Leont

On Thu, Oct 10, 2013 at 12:59 PM, Mark R. Bannister <
perlbug-followup@perl.org> wrote:

As I have described in my blog article:

http://technicalprose.blogspot.co.uk/2013/10/perl-each-function-is-not-re-entrant.html

the 'each' function is not re-entrant because it relies on hv_iternext()
to get an iterator for a hash. This should be fixed so that 'each' uses
its own local scope iterator.

The following code should work:

#!/usr/bin/perl
use warnings;
use strict;

my %global_hash;
my $max_depth = 3;

sub recurse
{
my ($depth, $hash) = @_;
while (my ($key, $val) = each %$hash) {
print "|" . ("-" x ($depth * 4)) . " $key$val\n";
recurse($depth+1, $hash) if $depth < $max_depth;
}
}

$global_hash{a} = 1;
$global_hash{b} = 2;

recurse(0, \%global_hash);

....and it would work if 'each' had its own iterator instead of using
one tied to the hash. Instead, one has to do this:

my %hashcopy = %$hash;
while (my ($key, $val) = each %hashcopy) {

or:

for my $key (keys %$hash) {
my $val = $hash->{$key};

Even when ignoring all other issues, this would require to have external
iterators instead of the current internal iterator. However this is
difficult to combine with perl's guarantee that delete on the current
subject of the iterator (so called delayed deletion) is safe and the O(1)
reading semantics.

Not saying it'd be impossible, but I don't see a way out that isn't
compromising on something else. I wouldn't mind being proven wrong though.

Leon

p5pRT · 2013-10-11T23:24:25Z

From @Hugmeir

On Fri, Oct 11, 2013 at 5:14 AM, Mark R. Bannister via RT <
perlbug-followup@perl.org> wrote:

On Fri Oct 11 00:09:41 2013, demerphq wrote:

We don't normally consider this a bug. More a mis-feature. It *is*
documented behavior.

And, yes, we pretty much all agree that the behavior makes each() less
useful.

If anything this is a feature request, that has been made many times,
suggesting it is difficult to do and unlikely to be implemented.

But heck, maybe someone will give it a crack one day.

Yves

Sorry, so this is a duplicate ticket? I did a search before I opened it
and didn't find a ticket about this issue already.

It's documented in perldoc -f each. I don't know if there's a ticket about
it.

At a cursory glance it doesn't look difficult to fix, so which specific
part of the solution do you think is difficult to implement and why?

It's not backwards-compatible. It would slow down every each() call. We
would need to introduce a new way of resetting the iterator, since, the
iterator being bound to the each, 'keys %hash' would no longer do the trick
(this is similar to a problem with the flip-flip operator). Also, I believe
that it would require new rules, or at least documentation, for cases that
modify the hash while iterating through it.

That being said, it's likely possible to prototype the intended behavior on
CPAN, which seems like the best step forward.

p5pRT · 2013-10-11T23:34:02Z

From zefram@fysh.org

Brian Fraser wrote:

That being said, it's likely possible to prototype the intended behavior on
CPAN, which seems like the best step forward.

Start by implementing reified iterators. They'll be useful on their
own merits, as well as as a base on which to build the requested variant
of each().

As with the recent request about the flip-flop operator, the semantics
being requested don't fit with the existing syntax. You need some
new element to determine the scope of the iterator. You'll either end
up with syntax that introduces a new element to explicitly determine
the scope or have some really hairy implicit element. The difference
from flip-flop is that flip-flop already has a hairy implicit aspect,
whereas with each() you'd be adding one.

-zefram

p5pRT · 2013-10-14T15:56:09Z

From @iabyn

On Sat, Oct 12, 2013 at 12:33:30AM +0100, Zefram wrote:

Start by implementing reified iterators. They'll be useful on their
own merits, as well as as a base on which to build the requested variant
of each().

An alternative approach might be to make the hash's iterator localise-able.
This is currently a syntax error:

local each %hash;

--
"But Sidley Park is already a picture, and a most amiable picture too.
The slopes are green and gentle. The trees are companionably grouped at
intervals that show them to advantage. The rill is a serpentine ribbon
unwound from the lake peaceably contained by meadows on which the right
amount of sheep are tastefully arranged." -- Lady Croom, "Arcadia"

p5pRT · 2013-10-14T21:09:59Z

From @Leont

On Sat, Oct 12, 2013 at 1:33 AM, Zefram <zefram@fysh.org> wrote:

Brian Fraser wrote:

That being said, it's likely possible to prototype the intended behavior
on
CPAN, which seems like the best step forward.

Start by implementing reified iterators. They'll be useful on their
own merits, as well as as a base on which to build the requested variant
of each().

Actually, having some kind of generic iterator sounds like a good idea in
general. It could make a construct like this: «for (<>) { ... }» DWIM. It
could enable lazy lists. I could be helpful for quite a few things.
</daydreaming>

Leon

p5pRT · 2013-10-17T03:04:05Z

From @rjbs

* Leon Timmermans <fawaka@gmail.com> [2013-10-14T17:08:50]

Actually, having some kind of generic iterator sounds like a good idea in
general. It could make a construct like this: «for (<>) { ... }» DWIM. It
could enable lazy lists. I could be helpful for quite a few things.
</daydreaming>

Yes, it's hard to imaging programming in any non-Perl P-language and not
longing for more generic iterators when back in Perl.

--
rjbs

p5pRT added the Severity Low label Oct 19, 2019

xenu removed the Severity Low label Dec 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pp_each needs its own hash iterator #13341

pp_each needs its own hash iterator #13341

p5pRT commented Oct 10, 2013

p5pRT commented Oct 10, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

p5pRT commented Oct 14, 2013

p5pRT commented Oct 14, 2013

p5pRT commented Oct 17, 2013

pp_each needs its own hash iterator #13341

pp_each needs its own hash iterator #13341

Comments

p5pRT commented Oct 10, 2013

p5pRT commented Oct 10, 2013

From chapter34@yahoo.com

Created by chapter34@yahoo.com

p5pRT commented Oct 11, 2013

From @demerphq

p5pRT commented Oct 11, 2013

p5pRT commented Oct 11, 2013

From chapter34@yahoo.com

p5pRT commented Oct 11, 2013

From @mauke

p5pRT commented Oct 11, 2013

From @Leont

p5pRT commented Oct 11, 2013

From @Hugmeir

p5pRT commented Oct 11, 2013

From zefram@fysh.org

p5pRT commented Oct 14, 2013

From @iabyn

p5pRT commented Oct 14, 2013

From @Leont

p5pRT commented Oct 17, 2013

From @rjbs