Skip Menu |
 
Report information
Id: 30123
Status: open
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: schwern <schwern [at] pobox.com>
Cc:
AdminCc:

Operating System: All
PatchStatus: (no value)
Severity: Wishlist
Type: core
Perl Version:
  • 5.8.1
  • 5.8.2
  • 5.8.3
  • 5.8.4
  • 5.8.5
  • 5.9.0
  • 5.9.1
  • 5.9.2
Fixed In: (no value)



CC: schwern [...] pobox.com
Subject: foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Tue, 8 Jun 2004 16:38:34 -0400 (EDT)
To: perlbug [...] perl.org
From: schwern [...] windhund.schwern.org (Michael G Schwern)
Download (untitled) / with headers
text/plain 3.5k
This is a bug report for perl from schwern@pobox.com, generated with the help of perlbug 1.34 running under perl v5.8.1. ----------------------------------------------------------------- [Please enter your report here] A co-worker exhausted his machine's memory trying to compile this: for ( '', 'a'..'zzzzzzz' ) {} B::Deparse (using 'zzz') reveals that Perl is trying to build that entire list in memory. The range optimizer seems to only kick in when there's nothing but a range in there. Our work around is: for ( $i = ""; $i ne "aaa"; ($i = $i ? ++$i : "a") ) { print $i } Bug/optimizer-miss appears in 5.6.1, 5.8.1 and 5.8.3. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl v5.8.1: Configured by root at Fri Sep 12 19:46:46 PDT 2003. Summary of my perl5 (revision 5.0 version 8 subversion 1 RC3) configuration: Platform: osname=darwin, osvers=7.0, archname=darwin-thread-multi-2level uname='darwin hampsten 7.0 darwin kernel version 6.0: fri jul 25 16:58:41 pdt 2003; root:xnu-344.frankd.rootsxnu-344.frankd~objrelease_ppc power macintosh powerpc ' config_args='-ds -e -Dprefix=/usr -Dccflags=-g -pipe -Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-g -pipe -pipe -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include', optimize='-Os', cppflags='-no-cpp-precomp -g -pipe -pipe -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include' ccversion='', gccversion='3.3 20030304 (Apple Computer, Inc. build 1495)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='-L/usr/local/lib' libpth=/usr/local/lib /usr/lib libs=-ldbm -ldl -lm -lc perllibs=-ldl -lm -lc libc=/usr/lib/libc.dylib, so=dylib, useshrplib=true, libperl=libperl.dylib gnulibc_version='' Dynamic Linking: dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-bundle -undefined dynamic_lookup -L/usr/local/lib' Locally applied patches: RC3 --- @INC for perl v5.8.1: /sw/lib/perl5/5.8.1/darwin-thread-multi-2level /sw/lib/perl5/5.8.1 /sw/lib/perl5 /sw/lib/perl5/darwin /System/Library/Perl/5.8.1/darwin-thread-multi-2level /System/Library/Perl/5.8.1 /Library/Perl/5.8.1/darwin-thread-multi-2level /Library/Perl/5.8.1 /Library/Perl /Network/Library/Perl/5.8.1/darwin-thread-multi-2level /Network/Library/Perl/5.8.1 /Network/Library/Perl . --- Environment for perl v5.8.1: DYLD_LIBRARY_PATH (unset) HOME=/Users/schwern LANG (unset) LANGUAGE (unset) LC_ALL=C LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/Users/schwern/bin:/usr/local/bin:/Users/schwern/bin:/usr/local/bin:/sw/bin:/sw/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/schwern:/usr/X11R6/bin:.:. PERL5LIB=/sw/lib/perl5:/sw/lib/perl5/darwin PERL_BADLANG (unset) SHELL=/sw/bin/bash
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Thu, 9 Sep 2004 22:47:35 +0100
To: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 855b
On Tue, Jun 08, 2004 at 08:38:52PM -0000, Michael G Schwern wrote: Show quoted text
> A co-worker exhausted his machine's memory trying to compile this: > > for ( '', 'a'..'zzzzzzz' ) {} > > B::Deparse (using 'zzz') reveals that Perl is trying to build that > entire list in memory. The range optimizer seems to only kick in when > there's nothing but a range in there. > > Our work around is: > > for ( $i = ""; $i ne "aaa"; ($i = $i ? ++$i : "a") ) { print $i } > > Bug/optimizer-miss appears in 5.6.1, 5.8.1 and 5.8.3.
I'm not sure how this can be construed as a bug. In the general case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5 doesn't have P6's lazy lists. It just so happens that C<for (X..Y)> is special-cased, but I don't see why that should lead to a general expcatation of lazy-listness? Dave. -- Nothing ventured, nothing lost.
CC: perl5-porters [...] perl.org
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Fri, 10 Sep 2004 03:34:00 -0500
From: David Nicol <davidnicol [...] gmail.com>
Download (untitled) / with headers
text/plain 171b
Just curious -- does the range optimization in C<for> apply to slices? foreach(@Array[$left .. $right]){ ... -- David L Nicol IT Consulting since 1986
CC: perl5-porters [...] perl.org
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Fri, 10 Sep 2004 03:32:48 -0700
To: David Nicol <davidnicol [...] gmail.com>
From: Yitzchak Scott-Thoennes <sthoenna [...] efn.org>
Download (untitled) / with headers
text/plain 200b
On Fri, Sep 10, 2004 at 03:34:00AM -0500, David Nicol wrote: Show quoted text
> Just curious -- does the range optimization in C<for> apply to slices? > > foreach(@Array[$left .. $right]){ > ...
No.
CC: perl5-porters [...] perl.org
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Fri, 15 Oct 2004 03:02:22 -0400
To: Dave Mitchell <davem [...] iabyn.com>
From: Michael G Schwern <schwern [...] gmail.com>
Download (untitled) / with headers
text/plain 549b
On Thu, 9 Sep 2004 22:47:35 +0100, Dave Mitchell <davem@iabyn.com> wrote: Show quoted text
> I'm not sure how this can be construed as a bug. In the general > case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5 doesn't > have P6's lazy lists. It just so happens that C<for (X..Y)> is > special-cased, but I don't see why that should lead to a general > expcatation of lazy-listness?
I have high expectations. Call it a feature request if it makes you feel better. I reported it so its in the tracking system if someone wants to take a crack at it.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 692b
On Fri Oct 15 09:32:02 2004, schwern wrote: Show quoted text
> On Thu, 9 Sep 2004 22:47:35 +0100, Dave Mitchell <davem@iabyn.com> wrote:
> > I'm not sure how this can be construed as a bug. In the general > > case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
doesn't Show quoted text
> > have P6's lazy lists. It just so happens that C<for (X..Y)> is > > special-cased, but I don't see why that should lead to a general > > expcatation of lazy-listness?
> > I have high expectations. > > Call it a feature request if it makes you feel better. I reported it > so its in the tracking system if someone wants to take a crack at it. >
Schwern, Is this still on your wishlist? Thank you very much. Jim Keenan
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 785b
On Sat Sep 29 19:08:16 2012, jkeenan wrote: Show quoted text
> On Fri Oct 15 09:32:02 2004, schwern wrote:
> > On Thu, 9 Sep 2004 22:47:35 +0100, Dave Mitchell <davem@iabyn.com>
wrote: Show quoted text
> > > I'm not sure how this can be construed as a bug. In the general > > > case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
> doesn't
> > > have P6's lazy lists. It just so happens that C<for (X..Y)> is > > > special-cased, but I don't see why that should lead to a general > > > expcatation of lazy-listness?
> > > > I have high expectations. > > > > Call it a feature request if it makes you feel better. I reported it > > so its in the tracking system if someone wants to take a crack at it. > >
> > Schwern, > > Is this still on your wishlist?
It’s on *mine*. -- Father Chrysostomos
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Sat, 24 Nov 2012 10:02:52 -0500
To: perlbug-followup [...] perl.org
From: "Matthew Horsfall (alh)" <wolfsage [...] gmail.com>
Download (untitled) / with headers
text/plain 1.1k
On Sun, Sep 30, 2012 at 1:27 AM, Father Chrysostomos via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Sat Sep 29 19:08:16 2012, jkeenan wrote:
> On Fri Oct 15 09:32:02 2004, schwern wrote:
> > On Thu, 9 Sep 2004 22:47:35 +0100, Dave Mitchell <davem@iabyn.com>
wrote:
> > > I'm not sure how this can be construed as a bug. In the general
> > > case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
> doesn't
> > > have P6's lazy lists. It just so happens that C<for (X..Y)> is
> > > special-cased, but I don't see why that should lead to a general
> > > expcatation of lazy-listness?
> >
> > I have high expectations.
> >
> > Call it a feature request if it makes you feel better.  I reported it
> > so its in the tracking system if someone wants to take a crack at it.
> >
>
> Schwern,
>
> Is this still on your wishlist?

It’s on *mine*.

Is there anything blocking this from happening, or just the usual (time, effort, tuits)?

Is there any other case besides for/foreach that 'a..z' should iterate on the fly rather than immediately generate a list in memory?

-- Matthew Horsfall (alh)
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.3k
On Sat Nov 24 07:03:37 2012, alh wrote: Show quoted text
> On Sun, Sep 30, 2012 at 1:27 AM, Father Chrysostomos via RT < > perlbug-followup@perl.org> wrote: >
> > On Sat Sep 29 19:08:16 2012, jkeenan wrote:
> > > On Fri Oct 15 09:32:02 2004, schwern wrote:
> > > > On Thu, 9 Sep 2004 22:47:35 +0100, Dave Mitchell <davem@iabyn.com>
> > wrote:
> > > > > I'm not sure how this can be construed as a bug. In the general > > > > > case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
> > > doesn't
> > > > > have P6's lazy lists. It just so happens that C<for (X..Y)> is > > > > > special-cased, but I don't see why that should lead to a general > > > > > expcatation of lazy-listness?
> > > > > > > > I have high expectations. > > > > > > > > Call it a feature request if it makes you feel better. I
reported it Show quoted text
> > > > so its in the tracking system if someone wants to take a crack
at it. Show quoted text
> > > >
> > > > > > Schwern, > > > > > > Is this still on your wishlist?
> > > > It’s on *mine*. > >
> > Is there anything blocking this from happening, or just the usual (time, > effort, tuits)?
The usual, plus lack of inspiration. Show quoted text
> > Is there any other case besides for/foreach that 'a..z' should iterate on > the fly rather than immediately generate a list in memory?
I hadn’t thought of that, but if it could iterate on the fly in more places it would allow for more idiomatic code. -- Father Chrysostomos
CC: perl5-porters [...] perl.org
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Sat, 24 Nov 2012 17:09:27 +0100
To: perlbug-followup [...] perl.org
From: Leon Timmermans <fawaka [...] gmail.com>
Download (untitled) / with headers
text/plain 780b
On Sat, Nov 24, 2012 at 4:34 PM, Father Chrysostomos via RT <perlbug-followup@perl.org> wrote: Show quoted text
>> Is there anything blocking this from happening, or just the usual (time, >> effort, tuits)?
> > The usual, plus lack of inspiration. >
>> >> Is there any other case besides for/foreach that 'a..z' should iterate on >> the fly rather than immediately generate a list in memory?
> > I hadn’t thought of that, but if it could iterate on the fly in more > places it would allow for more idiomatic code.
It would be awesome if we had real lazy lists, but that would require coroutines first (which is known to be doable, but has many edge-cases, in particular dynamic scoping ones). Some kind of internal generator would be a good start though, but would still be non-trivial. Leon
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Sat, 24 Nov 2012 11:15:36 -0500
To: perlbug-followup [...] perl.org
From: "Matthew Horsfall (alh)" <wolfsage [...] gmail.com>
Download (untitled) / with headers
text/plain 764b
On Sat, Nov 24, 2012 at 10:34 AM, Father Chrysostomos via RT <perlbug-followup@perl.org> wrote:
Show quoted text
>
> Is there any other case besides for/foreach that 'a..z' should iterate on
> the fly rather than immediately generate a list in memory?

I hadn’t thought of that, but if it could iterate on the fly in more
places it would allow for more idiomatic code.


I see these as being sensible uses of iteration:

for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ }

my @found = grep { $_ =~ /./ } 1..10;

my %hash = map { $_ => 1 } 1..10;

But should this iterate at runtime or be pregenerated as it is now?

my @arr = (1..10);

If it's in a sub that's never hit, there's a memory saving there if it iterates when requested.

-- Matthew Horsfall (alh)
Subject: Fwd: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Sat, 24 Nov 2012 11:17:06 -0500
To: perlbug-followup [...] perl.org
From: "Matthew Horsfall (alh)" <wolfsage [...] gmail.com>
Download (untitled) / with headers
text/plain 1.3k
---------- Forwarded message ----------
From: Matthew Horsfall (alh) <wolfsage@gmail.com>
Date: Sat, Nov 24, 2012 at 11:16 AM
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
To: Leon Timmermans <fawaka@gmail.com>


On Sat, Nov 24, 2012 at 11:09 AM, Leon Timmermans <fawaka@gmail.com> wrote:
Show quoted text
On Sat, Nov 24, 2012 at 4:34 PM, Father Chrysostomos via RT
<perlbug-followup@perl.org> wrote:
>> Is there anything blocking this from happening, or just the usual (time,
>> effort, tuits)?
>
> The usual, plus lack of inspiration.
>
>>
>> Is there any other case besides for/foreach that 'a..z' should iterate on
>> the fly rather than immediately generate a list in memory?
>
> I hadn’t thought of that, but if it could iterate on the fly in more
> places it would allow for more idiomatic code.

It would be awesome if we had real lazy lists, but that would require
coroutines first (which is known to be doable, but has many
edge-cases, in particular dynamic scoping ones). Some kind of internal
generator would be a good start though, but would still be
non-trivial.

Leon

Yeah, that's what I was thinking it'd be nice to have. The logic is sort of there right now, but it's tied very closely to loops right now and ripping that out could be painful.

-- Matthew Horsfall (alh)

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 919b
On Sat Nov 24 08:16:13 2012, alh wrote: Show quoted text
> On Sat, Nov 24, 2012 at 10:34 AM, Father Chrysostomos via RT < > perlbug-followup@perl.org> wrote: >
> > > > > > Is there any other case besides for/foreach that 'a..z' should
> iterate on
> > > the fly rather than immediately generate a list in memory?
> > > > I hadn’t thought of that, but if it could iterate on the fly in more > > places it would allow for more idiomatic code. > > > >
> I see these as being sensible uses of iteration: > > for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ } > > my @found = grep { $_ =~ /./ } 1..10; > > my %hash = map { $_ => 1 } 1..10*;* > > But should this iterate at runtime or be pregenerated as it is now?
That’s a hard question. :-) Show quoted text
> > my @arr = (1..10); > > If it's in a sub that's never hit, there's a memory saving there if it > iterates when requested. > > -- Matthew Horsfall (alh)
-- Father Chrysostomos
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Sun, 25 Nov 2012 08:13:18 -0500
To: perlbug-followup [...] perl.org
From: "Matthew Horsfall (alh)" <wolfsage [...] gmail.com>
Download (untitled) / with headers
text/plain 563b
On Sat, Nov 24, 2012 at 3:57 PM, Father Chrysostomos via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Sat Nov 24 08:16:13 2012, alh wrote:
> I see these as being sensible uses of iteration:
>
> for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ }
>
> my @found = grep { $_ =~ /./ } 1..10;
>
> my %hash = map { $_ => 1 } 1..10*;*
>
> But should this iterate at runtime or be pregenerated as it is now?

That’s a hard question. :-)


I suppose we could slap together a C version of less.pm and then let users ;)

-- Matthew Horsfall (alh)
Subject: Re: [perl #30123] foreach ("", "a".."zzzzzz") confuses range optimizer
Date: Sun, 25 Nov 2012 17:20:29 +0100
To: perl5-porters [...] perl.org
From: "Dr.Ruud" <rvtol+usenet [...] isolution.nl>
Download (untitled) / with headers
text/plain 788b
On 2012-11-25 14:13, Matthew Horsfall (alh) wrote: Show quoted text
> On Sat, Nov 24, 2012 at 3:57 PM, Father Chrysostomos via RT > <perlbug-followup@perl.org <mailto:perlbug-followup@perl.org>> wrote:
Show quoted text
> On Sat Nov 24 08:16:13 2012, alh wrote:
> > I see these as being sensible uses of iteration: > > > > for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ } > > > > my @found = grep { $_ =~ /./ } 1..10; > > > > my %hash = map { $_ => 1 } 1..10*;* > > > > But should this iterate at runtime or be pregenerated as it is now?
> > That’s a hard question. :-) > > > I suppose we could slap together a C version of less.pm <http://less.pm> > and then let users ;)
Or add support for modifiers, like "a".."zzzzzzzzzzz":i ("i" for "iterator") -- Ruud


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org