Skip Menu |
Report information
Id: 132150
Status: pending release
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: mauke- <l.mai [at] web.de>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: core
Perl Version: 5.26.0
Fixed In: (no value)

Attachments
0001-parse-.-yadayada-as-a-term-not-an-operator-RT-132150.patch



To: perlbug [...] perl.org
From: l.mai [...] web.de
Subject: ... (yada-yada) parsing is inconsistent
Date: Sat, 23 Sep 2017 13:39:32 +0200
Download (untitled) / with headers
text/plain 4.4k
This is a bug report for perl from l.mai@web.de, generated with the help of perlbug 1.40 running under perl 5.26.0. ----------------------------------------------------------------- [Please describe your issue here] Perl parses '...' as 'die("Unimplemented")': $ perl -MO=Deparse -e '...' die 'Unimplemented'; -e syntax OK But it only does this at the beginning of a statement: $ perl -MO=Deparse -e '+ ...' syntax error at -e line 1, near "+ ..." -e had compilation errors. However, '...' is not by itself a statement: $ perl -MO=Deparse -e '... sub foo {}' syntax error at -e line 1, near "... sub foo " -e had compilation errors. In fact, it's an expression: $ perl -MO=Deparse -e '... . "hello"' die('Unimplemented') . 'hello'; -e syntax OK $ perl -MO=Deparse -e '... lt "hello"' die('Unimplemented') lt 'hello'; -e syntax OK $ perl -MO=Deparse -e '..., foo()' die('Unimplemented'), foo(); -e syntax OK However, it's a naughty expression that doesn't tell the parser that the next token should be an operator: $ perl -MO=Deparse -e '... + 0' syntax error at -e line 1, near "... +" -e had compilation errors. Here '+ 0' is parsed as a unary + applied to 0. The syntax error occurs because you can't have two terms in a row ('...', '+0') without an operator in between. This is especially noticeable with '/': $ perl -MO=Deparse -e '... / 1' Search pattern not terminated at -e line 1. $ perl -MO=Deparse -e '... / 1 /' syntax error at -e line 1, near "... / 1 /" -e had compilation errors. For consistency, either '...' should be made a statement of its own, or it should give operator context to the following token. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=low --- Site configuration information for perl 5.26.0: Configured by mauke at Fri Sep 22 13:28:36 CEST 2017. Summary of my perl5 (revision 5 version 26 subversion 0) configuration: Platform: osname=linux osvers=4.9.41-1-lts archname=i686-linux uname='linux simplicio 4.9.41-1-lts #1 smp mon aug 7 17:57:02 cest 2017 i686 gnulinux ' config_args='' hint=recommended useposix=true d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='cc' ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' optimize='-O2 -march=native' cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='' gccversion='7.2.0' gccosandvers='' intsize=4 longsize=4 ptrsize=4 doublesize=8 byteorder=1234 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=12 longdblkind=3 ivtype='long' ivsize=4 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=4 prototype=define Linker and Libraries: ld='cc' ldflags ='-fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/i686-pc-linux-gnu/7.2.0/include-fixed /usr/lib /lib libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.26.so so=so useshrplib=false libperl=libperl.a gnulibc_version='2.26' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E' cccdlflags='-fPIC' lddlflags='-shared -O2 -march=native -L/usr/local/lib -fstack-protector-strong' --- @INC for perl 5.26.0: /home/mauke/usr/lib/perl5/site_perl/5.26.0/i686-linux /home/mauke/usr/lib/perl5/site_perl/5.26.0 /home/mauke/usr/lib/perl5/5.26.0/i686-linux /home/mauke/usr/lib/perl5/5.26.0 --- Environment for perl 5.26.0: HOME=/home/mauke LANG=en_US.UTF-8 LANGUAGE=en_US LC_COLLATE=C LC_MONETARY=de_DE.UTF-8 LC_TIME=de_DE.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/mauke/perl5/perlbrew/bin:/home/mauke/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl PERLBREW_BASHRC_VERSION=0.73 PERLBREW_HOME=/home/mauke/.perlbrew PERLBREW_ROOT=/home/mauke/perl5/perlbrew PERL_BADLANG (unset) PERL_UNICODE=SAL SHELL=/bin/bash
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.6k
On Sat, 23 Sep 2017 04:39:49 -0700, mauke- wrote: Show quoted text
> > [Please describe your issue here] > > Perl parses '...' as 'die("Unimplemented")': > > $ perl -MO=Deparse -e '...' > die 'Unimplemented'; > -e syntax OK > > > But it only does this at the beginning of a statement: > > $ perl -MO=Deparse -e '+ ...' > syntax error at -e line 1, near "+ ..." > -e had compilation errors. > > > However, '...' is not by itself a statement: > > $ perl -MO=Deparse -e '... sub foo {}' > syntax error at -e line 1, near "... sub foo " > -e had compilation errors. > > > In fact, it's an expression: > > $ perl -MO=Deparse -e '... . "hello"' > die('Unimplemented') . 'hello'; > -e syntax OK > > $ perl -MO=Deparse -e '... lt "hello"' > die('Unimplemented') lt 'hello'; > -e syntax OK > > $ perl -MO=Deparse -e '..., foo()' > die('Unimplemented'), foo(); > -e syntax OK > > > However, it's a naughty expression that doesn't tell the parser that > the next > token should be an operator: > > $ perl -MO=Deparse -e '... + 0' > syntax error at -e line 1, near "... +" > -e had compilation errors. > > Here '+ 0' is parsed as a unary + applied to 0. The syntax error > occurs because > you can't have two terms in a row ('...', '+0') without an operator in > between. > > This is especially noticeable with '/': > > $ perl -MO=Deparse -e '... / 1' > Search pattern not terminated at -e line 1. > > $ perl -MO=Deparse -e '... / 1 /' > syntax error at -e line 1, near "... / 1 /" > -e had compilation errors. > > > For consistency, either '...' should be made a statement of its own, > or it > should give operator context to the following token.
Patch (to do the latter) attached. Thoughts?
Subject: 0001-parse-.-yadayada-as-a-term-not-an-operator-RT-132150.patch
From 02be2fc4ff7e4ede46381ee96b533ffe1aaa50ab Mon Sep 17 00:00:00 2001 From: Lukas Mai <l.mai@web.de> Date: Sat, 23 Sep 2017 14:25:32 +0200 Subject: [PATCH] parse ... (yadayada) as a term, not an operator (RT #132150) --- t/op/yadayada.t | 19 ++++++++++++++++++- toke.c | 2 +- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git t/op/yadayada.t t/op/yadayada.t index 861389f4c5..0013e37046 100644 --- t/op/yadayada.t +++ t/op/yadayada.t @@ -8,7 +8,7 @@ BEGIN { use strict; -plan 9; +plan 12; my $err; my $err1 = "Unimplemented at $0 line "; @@ -42,6 +42,23 @@ eval { @transformed = map {;... } @input; }; is $@, $err, "Disambiguation case 4"; $@ = ''; + +note("RT #132150: ... is a term, not an operator"); +$err = $err1 . ( __LINE__ + 1 ) . $err2; +eval { ... + 0 }; +is $@, $err, "... + 0 parses"; +$@ = ''; + +$err = $err1 . ( __LINE__ + 1 ) . $err2; +eval { ... % 1 }; +is $@, $err, "... % 1 parses"; +$@ = ''; + +$err = $err1 . ( __LINE__ + 1 ) . $err2; +eval { ... / 1 }; +is $@, $err, "... / 1 parses"; +$@ = ''; + # # Regression tests, making sure ... is still parsable as an operator. # diff --git toke.c toke.c index a91a4fcfbe..63f8d418ca 100644 --- toke.c +++ toke.c @@ -6856,7 +6856,7 @@ Perl_yylex(pTHX) } if (PL_expect == XSTATE && s[1] == '.' && s[2] == '.') { s += 3; - OPERATOR(YADAYADA); + TERM(YADAYADA); } if (PL_expect == XOPERATOR || !isDIGIT(s[1])) { char tmp = *s++; -- 2.14.1
Date: Sat, 23 Sep 2017 17:54:24 +0100
From: Zefram <zefram [...] fysh.org>
Subject: Re: [perl #132150] ... (yada-yada) parsing is inconsistent
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 493b
l.mai@web.de wrote: Show quoted text
>For consistency, either '...' should be made a statement of its own, or it >should give operator context to the following token.
The current behaviour is indeed inconsistent, but the statement-like option for making it consistent doesn't require that '...' be a complete statement. It would be perfectly consistent for '...;' to be the complete statement (with semicolon implicit before closing brace, as it already is) with '... sub foo {}' remaining illegal. -zefram
Subject: Re: [perl #132150] ... (yada-yada) parsing is inconsistent
From: Zefram <zefram [...] fysh.org>
Date: Thu, 2 Nov 2017 19:53:29 +0000
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 228b
l.mai@web.de via RT wrote: Show quoted text
>Patch (to do the latter) attached. Thoughts?
perly.y makes clear that yada-yada is intended to be an expression, not a statement. Patch applied as f5727a1c71878a34f6255eb1a506c0b21af7d36f. -zefram
Subject: Re: [perl #132150] ... (yada-yada) parsing is inconsistent
Date: Mon, 06 Nov 2017 16:51:05 +0000
From: Smylers <smylers [...] stripey.com>
To: <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 852b
Zefram writes: Show quoted text
> perly.y makes clear that yada-yada is intended to be an expression, > not a statement. Patch applied as > f5727a1c71878a34f6255eb1a506c0b21af7d36f.
perlsyn documents it as being the ellipsis statement, specifically saying: You can only use the elliptical statement to stand in for a complete statement. … The elliptical statement cannot stand in for an expression that is part of a larger statement, since the "..." is also the three-dot version of the flip-flop operator (see "Range Operators" in perlop). These examples of attempts to use an ellipsis are syntax errors: use v5.12; print ...; open(my $fh, ">", "/dev/passwd") or ...; if ($condition && ... ) { say "Howdy" }; Sounds like that is no longer true (and quite possibly that use of the word “since” there was never true). Smylers
From: Zefram <zefram [...] fysh.org>
To: perl5-porters [...] perl.org
Subject: Re: [perl #132150] ... (yada-yada) parsing is inconsistent
Date: Fri, 10 Nov 2017 01:06:29 +0000
Smylers wrote: Show quoted text
>perlsyn documents it as being the ellipsis statement,
Hmm, that's an embuggerance. The bit you quoted about ambiguity with the flip-flop operator doesn't actually make sense as a rationale, because we resolve those kinds of ambiguities all the time, for example between the "+" infix operator and the "+" prefix operator. But it turns out there's more inconsistency to the parsing code. Not only was PL_expect being set in a way hostile to treating "..." as a term, but the "..." token was not recognised in the contexts that would be required for it to be a term. So after Lukas's patch, although "... + 0" is now accepted, "0 + ..." is still not accepted. It would take more changes to make it consistently a term. Taking into account the documentation and these additional implementation issues, I reckon the old code (prior to Lukas's patch) was closer to having "...;" be a statement than "..." be an expression. It looks like the grammar in perly.y was actually the one thing inconsistent with that. I'll try changing it to a statement, and see how that goes. -zefram
From: Zefram <zefram [...] fysh.org>
To: perl5-porters [...] perl.org
Subject: Re: [perl #132150] ... (yada-yada) parsing is inconsistent
Date: Fri, 10 Nov 2017 02:27:07 +0000
Download (untitled) / with headers
text/plain 1.3k
I wrote: Show quoted text
>I'll try changing it to a statement, and see how that goes.
Looks like that works just fine. Only takes a small perly.y change (though I also reverted the toke.c change). The documentation in perlsyn was a bit ropey: in addition to the incorrect statement about ambiguity with flip-flop, it had some stuff about block/hash ambiguity in map, in a situation that since 5.22 has not parsed the way it claims. So I've removed the incorrect stuff and added a bit more clarification there. Also some test cases about when "..." should not parse as yada-yada. Applied to blead as commit 29d69c3c41c7e93f884256b1087face64d5fdd1e. I wasn't sure how to classify this for perldelta. On the one hand it's a bugfix; on the other it's making illegal some constructs that made at least partial sense and were previously accepted. I've conservatively classified it as an "incompatible change". However, if we think of it as incompatible, we need to decide whether a deprecation cycle is required. Implementing a deprecation would be a bit more involved, but not disproportionately so. The closest precedent I can think of for this sort of change is the removal of the implicit parens around qw, for which we did do a deprecation cycle, based on the knowledge that the thing being deprecated was moderately widespread. The bad uses of yada-yada are probably a lot less common. -zefram
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 727b
On Thu, 09 Nov 2017 17:06:43 -0800, zefram@fysh.org wrote: Show quoted text
> The bit you quoted about ambiguity with the > flip-flop operator doesn't actually make sense as a rationale, because > we resolve those kinds of ambiguities all the time, for example between > the "+" infix operator and the "+" prefix operator.
Introducing a new term or prefix operator that looks like an existing infix breaks existing syntax, of the form: substr $x, 1, . substr $y, 1; (I write code like that all the time.) If you put ... instead of . then you have an example of code that a ... term will break. It doesn’t happen the other way round. We can introduce a new infix that looks like an existing prefix or term. -- Father Chrysostomos
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 214b
On Thu, 09 Nov 2017 18:27:24 -0800, zefram@fysh.org wrote: Show quoted text
> Applied to blead as commit 29d69c3c41c7e93f884256b1087face64d5fdd1e.
I think that means this can be closed. I am doing that. -- Father Chrysostomos


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org