Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

foreach ("", "a".."zzzzzz") confuses range optimizer #7352

Open
p5pRT opened this issue Jun 8, 2004 · 16 comments
Open

foreach ("", "a".."zzzzzz") confuses range optimizer #7352

p5pRT opened this issue Jun 8, 2004 · 16 comments

Comments

@p5pRT
Copy link

p5pRT commented Jun 8, 2004

Migrated from rt.perl.org#30123 (status was 'open')

Searchable as RT30123$

@p5pRT
Copy link
Author

p5pRT commented Jun 8, 2004

From @schwern

Created by @schwern

A co-worker exhausted his machine's memory trying to compile this​:

for ( '', 'a'..'zzzzzzz' ) {}

B​::Deparse (using 'zzz') reveals that Perl is trying to build that
entire list in memory. The range optimizer seems to only kick in when
there's nothing but a range in there.

Our work around is​:

for ( $i = ""; $i ne "aaa"; ($i = $i ? ++$i : "a") ) { print $i }

Bug/optimizer-miss appears in 5.6.1, 5.8.1 and 5.8.3.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl v5.8.1:

Configured by root at Fri Sep 12 19:46:46 PDT 2003.

Summary of my perl5 (revision 5.0 version 8 subversion 1 RC3) configuration:
  Platform:
    osname=darwin, osvers=7.0, archname=darwin-thread-multi-2level
    uname='darwin hampsten 7.0 darwin kernel version 6.0: fri jul 25 16:58:41 pdt 2003; root:xnu-344.frankd.rootsxnu-344.frankd~objrelease_ppc power macintosh powerpc '
    config_args='-ds -e -Dprefix=/usr -Dccflags=-g  -pipe  -Dldflags=-Dman3ext=3pm -Duseithreads -Duseshrplib'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-g -pipe -pipe -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include',
    optimize='-Os',
    cppflags='-no-cpp-precomp -g -pipe -pipe -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='3.3 20030304 (Apple Computer, Inc. build 1495)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='-L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib
    libs=-ldbm -ldl -lm -lc
    perllibs=-ldl -lm -lc
    libc=/usr/lib/libc.dylib, so=dylib, useshrplib=true, libperl=libperl.dylib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-bundle -undefined dynamic_lookup -L/usr/local/lib'

Locally applied patches:
    RC3


@INC for perl v5.8.1:
    /sw/lib/perl5/5.8.1/darwin-thread-multi-2level
    /sw/lib/perl5/5.8.1
    /sw/lib/perl5
    /sw/lib/perl5/darwin
    /System/Library/Perl/5.8.1/darwin-thread-multi-2level
    /System/Library/Perl/5.8.1
    /Library/Perl/5.8.1/darwin-thread-multi-2level
    /Library/Perl/5.8.1
    /Library/Perl
    /Network/Library/Perl/5.8.1/darwin-thread-multi-2level
    /Network/Library/Perl/5.8.1
    /Network/Library/Perl
    .


Environment for perl v5.8.1:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/schwern
    LANG (unset)
    LANGUAGE (unset)
    LC_ALL=C
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/schwern/bin:/usr/local/bin:/Users/schwern/bin:/usr/local/bin:/sw/bin:/sw/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/schwern:/usr/X11R6/bin:.:.
    PERL5LIB=/sw/lib/perl5:/sw/lib/perl5/darwin
    PERL_BADLANG (unset)
    SHELL=/sw/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Sep 9, 2004

From @iabyn

On Tue, Jun 08, 2004 at 08​:38​:52PM -0000, Michael G Schwern wrote​:

A co-worker exhausted his machine's memory trying to compile this​:

for ( '', 'a'..'zzzzzzz' ) {}

B​::Deparse (using 'zzz') reveals that Perl is trying to build that
entire list in memory. The range optimizer seems to only kick in when
there's nothing but a range in there.

Our work around is​:

for ( $i = ""; $i ne "aaa"; ($i = $i ? ++$i : "a") ) { print $i }

Bug/optimizer-miss appears in 5.6.1, 5.8.1 and 5.8.3.

I'm not sure how this can be construed as a bug. In the general
case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5 doesn't
have P6's lazy lists. It just so happens that C<for (X..Y)> is
special-cased, but I don't see why that should lead to a general
expcatation of lazy-listness?

Dave.

--
Nothing ventured, nothing lost.

@p5pRT
Copy link
Author

p5pRT commented Sep 9, 2004

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Sep 10, 2004

From @davidnicol

Just curious -- does the range optimization in C<for> apply to slices?

  foreach(@​Array[$left .. $right]){
  ...

--
David L Nicol
IT Consulting since 1986

@p5pRT
Copy link
Author

p5pRT commented Sep 10, 2004

From @ysth

On Fri, Sep 10, 2004 at 03​:34​:00AM -0500, David Nicol wrote​:

Just curious -- does the range optimization in C<for> apply to slices?

foreach\(@&#8203;Array\[$left \.\. $right\]\)\{
       \.\.\.

No.

@p5pRT
Copy link
Author

p5pRT commented Oct 15, 2004

From @schwern

On Thu, 9 Sep 2004 22​:47​:35 +0100, Dave Mitchell <davem@​iabyn.com> wrote​:

I'm not sure how this can be construed as a bug. In the general
case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5 doesn't
have P6's lazy lists. It just so happens that C<for (X..Y)> is
special-cased, but I don't see why that should lead to a general
expcatation of lazy-listness?

I have high expectations.

Call it a feature request if it makes you feel better. I reported it
so its in the tracking system if someone wants to take a crack at it.

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2012

From @jkeenan

On Fri Oct 15 09​:32​:02 2004, schwern wrote​:

On Thu, 9 Sep 2004 22​:47​:35 +0100, Dave Mitchell <davem@​iabyn.com> wrote​:

I'm not sure how this can be construed as a bug. In the general
case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
doesn't
have P6's lazy lists. It just so happens that C<for (X..Y)> is
special-cased, but I don't see why that should lead to a general
expcatation of lazy-listness?

I have high expectations.

Call it a feature request if it makes you feel better. I reported it
so its in the tracking system if someone wants to take a crack at it.

Schwern,

Is this still on your wishlist?

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2012

From @cpansprout

On Sat Sep 29 19​:08​:16 2012, jkeenan wrote​:

On Fri Oct 15 09​:32​:02 2004, schwern wrote​:

On Thu, 9 Sep 2004 22​:47​:35 +0100, Dave Mitchell <davem@​iabyn.com>
wrote​:

I'm not sure how this can be construed as a bug. In the general
case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
doesn't
have P6's lazy lists. It just so happens that C<for (X..Y)> is
special-cased, but I don't see why that should lead to a general
expcatation of lazy-listness?

I have high expectations.

Call it a feature request if it makes you feel better. I reported it
so its in the tracking system if someone wants to take a crack at it.

Schwern,

Is this still on your wishlist?

It’s on *mine*.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Nov 24, 2012

From @wolfsage

On Sun, Sep 30, 2012 at 1​:27 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Sat Sep 29 19​:08​:16 2012, jkeenan wrote​:

On Fri Oct 15 09​:32​:02 2004, schwern wrote​:

On Thu, 9 Sep 2004 22​:47​:35 +0100, Dave Mitchell <davem@​iabyn.com>
wrote​:

I'm not sure how this can be construed as a bug. In the general
case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
doesn't
have P6's lazy lists. It just so happens that C<for (X..Y)> is
special-cased, but I don't see why that should lead to a general
expcatation of lazy-listness?

I have high expectations.

Call it a feature request if it makes you feel better. I reported it
so its in the tracking system if someone wants to take a crack at it.

Schwern,

Is this still on your wishlist?

It’s on *mine*.

Is there anything blocking this from happening, or just the usual (time,
effort, tuits)?

Is there any other case besides for/foreach that 'a..z' should iterate on
the fly rather than immediately generate a list in memory?

-- Matthew Horsfall (alh)

@p5pRT
Copy link
Author

p5pRT commented Nov 24, 2012

From @cpansprout

On Sat Nov 24 07​:03​:37 2012, alh wrote​:

On Sun, Sep 30, 2012 at 1​:27 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Sat Sep 29 19​:08​:16 2012, jkeenan wrote​:

On Fri Oct 15 09​:32​:02 2004, schwern wrote​:

On Thu, 9 Sep 2004 22​:47​:35 +0100, Dave Mitchell <davem@​iabyn.com>
wrote​:

I'm not sure how this can be construed as a bug. In the general
case I'd expect 'a'..'zzzzzzz' to generate a large list, since P5
doesn't
have P6's lazy lists. It just so happens that C<for (X..Y)> is
special-cased, but I don't see why that should lead to a general
expcatation of lazy-listness?

I have high expectations.

Call it a feature request if it makes you feel better. I
reported it
so its in the tracking system if someone wants to take a crack
at it.

Schwern,

Is this still on your wishlist?

It’s on *mine*.

Is there anything blocking this from happening, or just the usual (time,
effort, tuits)?

The usual, plus lack of inspiration.

Is there any other case besides for/foreach that 'a..z' should iterate on
the fly rather than immediately generate a list in memory?

I hadn’t thought of that, but if it could iterate on the fly in more
places it would allow for more idiomatic code.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Nov 24, 2012

From @Leont

On Sat, Nov 24, 2012 at 4​:34 PM, Father Chrysostomos via RT
<perlbug-followup@​perl.org> wrote​:

Is there anything blocking this from happening, or just the usual (time,
effort, tuits)?

The usual, plus lack of inspiration.

Is there any other case besides for/foreach that 'a..z' should iterate on
the fly rather than immediately generate a list in memory?

I hadn’t thought of that, but if it could iterate on the fly in more
places it would allow for more idiomatic code.

It would be awesome if we had real lazy lists, but that would require
coroutines first (which is known to be doable, but has many
edge-cases, in particular dynamic scoping ones). Some kind of internal
generator would be a good start though, but would still be
non-trivial.

Leon

@p5pRT
Copy link
Author

p5pRT commented Nov 24, 2012

From @wolfsage

On Sat, Nov 24, 2012 at 10​:34 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

Is there any other case besides for/foreach that 'a..z' should iterate on
the fly rather than immediately generate a list in memory?

I hadn’t thought of that, but if it could iterate on the fly in more
places it would allow for more idiomatic code.

I see these as being sensible uses of iteration​:

for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ }

my @​found = grep { $_ =~ /./ } 1..10;

my %hash = map { $_ => 1 } 1..10*;*

But should this iterate at runtime or be pregenerated as it is now?

my @​arr = (1..10);

If it's in a sub that's never hit, there's a memory saving there if it
iterates when requested.

-- Matthew Horsfall (alh)

@p5pRT
Copy link
Author

p5pRT commented Nov 24, 2012

From @wolfsage

---------- Forwarded message ----------
From​: Matthew Horsfall (alh) <wolfsage@​gmail.com>
Date​: Sat, Nov 24, 2012 at 11​:16 AM
Subject​: Re​: [perl #30123] foreach ("", "a".."zzzzzz") confuses range
optimizer
To​: Leon Timmermans <fawaka@​gmail.com>

On Sat, Nov 24, 2012 at 11​:09 AM, Leon Timmermans <fawaka@​gmail.com> wrote​:

On Sat, Nov 24, 2012 at 4​:34 PM, Father Chrysostomos via RT
<perlbug-followup@​perl.org> wrote​:

Is there anything blocking this from happening, or just the usual (time,
effort, tuits)?

The usual, plus lack of inspiration.

Is there any other case besides for/foreach that 'a..z' should iterate
on
the fly rather than immediately generate a list in memory?

I hadn’t thought of that, but if it could iterate on the fly in more
places it would allow for more idiomatic code.

It would be awesome if we had real lazy lists, but that would require
coroutines first (which is known to be doable, but has many
edge-cases, in particular dynamic scoping ones). Some kind of internal
generator would be a good start though, but would still be
non-trivial.

Leon

Yeah, that's what I was thinking it'd be nice to have. The logic is sort of
there right now, but it's tied very closely to loops right now and ripping
that out could be painful.

-- Matthew Horsfall (alh)

@p5pRT
Copy link
Author

p5pRT commented Nov 24, 2012

From @cpansprout

On Sat Nov 24 08​:16​:13 2012, alh wrote​:

On Sat, Nov 24, 2012 at 10​:34 AM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

Is there any other case besides for/foreach that 'a..z' should
iterate on
the fly rather than immediately generate a list in memory?

I hadn’t thought of that, but if it could iterate on the fly in more
places it would allow for more idiomatic code.

I see these as being sensible uses of iteration​:

for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ }

my @​found = grep { $_ =~ /./ } 1..10;

my %hash = map { $_ => 1 } 1..10*;*

But should this iterate at runtime or be pregenerated as it is now?

That’s a hard question. :-)

my @​arr = (1..10);

If it's in a sub that's never hit, there's a memory saving there if it
iterates when requested.

-- Matthew Horsfall (alh)

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Nov 25, 2012

From @wolfsage

On Sat, Nov 24, 2012 at 3​:57 PM, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Sat Nov 24 08​:16​:13 2012, alh wrote​:

I see these as being sensible uses of iteration​:

for ('a'..'z', qw(cat dog mouse), 1..10) { print $_ }

my @​found = grep { $_ =~ /./ } 1..10;

my %hash = map { $_ => 1 } 1..10*;*

But should this iterate at runtime or be pregenerated as it is now?

That’s a hard question. :-)

I suppose we could slap together a C version of less.pm and then let users
;)

-- Matthew Horsfall (alh)

@p5pRT
Copy link
Author

p5pRT commented Nov 25, 2012

From @druud62

On 2012-11-25 14​:13, Matthew Horsfall (alh) wrote​:

On Sat, Nov 24, 2012 at 3​:57 PM, Father Chrysostomos via RT
<perlbug-followup@​perl.org <mailto​:perlbug-followup@​perl.org>> wrote​:

On Sat Nov 24 08&#8203;:16&#8203;:13 2012\, alh wrote&#8203;:
 > I see these as being sensible uses of iteration&#8203;:
 >
 > for \('a'\.\.'z'\, qw\(cat dog mouse\)\, 1\.\.10\) \{ print $\_ \}
 >
 > my @&#8203;found = grep \{ $\_ =~ /\./ \} 1\.\.10;
 >
 > my %hash = map \{ $\_ => 1 \} 1\.\.10\*;\*
 >
 > But should this iterate at runtime or be pregenerated as it is now?

That’s a hard question\. :\-\)

I suppose we could slap together a C version of less.pm <http​://less.pm>
and then let users ;)

Or add support for modifiers, like "a".."zzzzzzzzzzz"​:i
("i" for "iterator")

--
Ruud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants