Skip Menu |
Report information
Id: 123737
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: brian.carpenter [at] gmail.com
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: unknown
Perl Version: (no value)
Fixed In: (no value)



Subject: S_no_op: Assertion `s >= oldbp' failed. (toke.c:536)
Download (untitled) / with headers
text/plain 3.9k
Built v5.21.9 (v5.21.8-200-ga57d3d4) using the following command line: ./Configure -des -Dusedevel -DDEBUGGING -Dcc=afl-gcc -Doptimize=-O2\ -g && AFL_HARDEN=1 make -j6 test-prep Bug found with AFL (http://lcamtuf.coredump.cx/afl) GDB output: [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Scalar found where operator expected at tokeabort line 1, near "0$" perl: toke.c:536: S_no_op: Assertion `s >= oldbp' failed. Program received signal SIGABRT, Aborted. [----------------------------------registers-----------------------------------] RAX: 0x0 RBX: 0x7fffffffe652 --> 0x6b6f74006c726570 ('perl') RCX: 0xffffffffffffffff RDX: 0x6 RSI: 0x2952 ('R)') RDI: 0x2952 ('R)') RBP: 0x7ffff6ea9a07 --> 0x257325732500203a (': ') RSP: 0x7fffffffd738 --> 0x7ffff6d923e0 (<*__GI_abort+384>: mov rdx,QWORD PTR fs:0x10) RIP: 0x7ffff6d8f165 (<*__GI_raise+53>: cmp rax,0xfffffffffffff000) R8 : 0x7ffff7fdd700 (0x00007ffff7fdd700) R9 : 0x6f5f6f6e5f53203a (': S_no_o') R10: 0x8 R11: 0x202 R12: 0xea9cc3 ("s >= oldbp") R13: 0xeb7225 --> 0x706f5f6f6e5f53 ('S_no_op') R14: 0x7ffff6ea9a07 --> 0x257325732500203a (': ') R15: 0x218 EFLAGS: 0x202 (carry parity adjust zero sign trap INTERRUPT direction overflow) [-------------------------------------code-------------------------------------] 0x7ffff6d8f15b <*__GI_raise+43>: movsxd rdi,eax 0x7ffff6d8f15e <*__GI_raise+46>: mov eax,0xea 0x7ffff6d8f163 <*__GI_raise+51>: syscall => 0x7ffff6d8f165 <*__GI_raise+53>: cmp rax,0xfffffffffffff000 0x7ffff6d8f16b <*__GI_raise+59>: ja 0x7ffff6d8f182 <*__GI_raise+82> 0x7ffff6d8f16d <*__GI_raise+61>: repz ret 0x7ffff6d8f16f <*__GI_raise+63>: nop 0x7ffff6d8f170 <*__GI_raise+64>: test eax,eax [------------------------------------stack-------------------------------------] 0000| 0x7fffffffd738 --> 0x7ffff6d923e0 (<*__GI_abort+384>: mov rdx,QWORD PTR fs:0x10) 0008| 0x7fffffffd740 --> 0xea9cc3 ("s >= oldbp") 0016| 0x7fffffffd748 --> 0x7ffff6eab9c1 --> 0x706c6568007325 ('%s') 0024| 0x7fffffffd750 --> 0x7fffffffd770 --> 0x3000000018 0032| 0x7fffffffd758 --> 0x218 0040| 0x7fffffffd760 --> 0x7fffffffd860 --> 0x7fffffffe652 --> 0x6b6f74006c726570 ('perl') 0048| 0x7fffffffd768 --> 0x7ffff6dc3fe6 (<__fxprintf+310>: lea rsp,[rbp-0x20]) 0056| 0x7fffffffd770 --> 0x3000000018 [------------------------------------------------------------------------------] Legend: code, data, rodata, value Stopped reason: SIGABRT 0x00007ffff6d8f165 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. gdb-peda$ bt #0 0x00007ffff6d8f165 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007ffff6d923e0 in *__GI_abort () at abort.c:92 #2 0x00007ffff6d88311 in *__GI___assert_fail (assertion=assertion@entry=0xea9cc3 "s >= oldbp", file=<optimized out>, file@entry=0xea9a0d "toke.c", line=line@entry=0x218, function=function@entry=0xeb7225 "S_no_op") at assert.c:81 #3 0x00000000005c2701 in S_no_op (what=what@entry=0xeaa39e "Scalar", s=s@entry=0x1182032 "{\n;") at toke.c:536 #4 0x0000000000651615 in Perl_yylex () at toke.c:5991 #5 0x000000000065c275 in Perl_yyparse (gramtype=<optimized out>) at perly.c:322 #6 0x000000000052d275 in S_parse_body (env=env@entry=0x0, xsinit=xsinit@entry=0x42d080 <xs_init>) at perl.c:2273 #7 0x000000000053324f in perl_parse (my_perl=<optimized out>, xsinit=0x42d080 <xs_init>, argc=<optimized out>, argv=<optimized out>, env=0x0) at perl.c:1607 #8 0x000000000042cc8c in main (argc=0x2, argv=0x7fffffffe3c8, env=0x7fffffffe3e0) at perlmain.c:114 #9 0x00007ffff6d7bead in __libc_start_main (main=<optimized out>, argc=<optimized out>, ubp_av=<optimized out>, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe3b8) at libc-start.c:244 #10 0x000000000042cfa5 in _start () Test case hexdump: 0000000 2430 0a7b 0000004
Subject: tokeabort.gz
Download tokeabort.gz
application/x-gzip 34b

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
Fixed in 488bc57. -- Father Chrysostomos
Subject: Re: [perl #123737] S_no_op: Assertion `s >= oldbp' failed. (toke.c:536)
To: perlbug-followup [...] perl.org
Date: Tue, 17 Mar 2015 03:51:22 -0500
From: Brian Carpenter <brian.carpenter [...] gmail.com>
Download (untitled) / with headers
text/plain 8.5k
Download (untitled) / with headers
text/html 14.9k

Message body is not shown because it is too large.

Download test43
application/octet-stream 6b

Message body not shown because it is not plain text.

Download test43-min
application/octet-stream 5b

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 465b
On Tue Mar 17 01:52:36 2015, brian.carpenter@gmail.com wrote: Show quoted text
> This bugger is back, albeit in a slight different part of toke.c.
I just noticed that the ticket was not reopened, I'll do that now. Brian's new test case is: % echo '0$#{' | ./miniperl -c Array length found where operator expected at - line 1, near "0$#" miniperl: toke.c:539: S_no_op: Assertion `s >= oldbp' failed. Aborted (core dumped) % Hopefully Father C will get time to take a look. Hugo
Date: Fri, 24 Apr 2015 13:06:41 -0400
From: "Matthew Horsfall (alh)" <wolfsage [...] gmail.com>
To: perlbug-followup [...] perl.org
CC: Perl5 Porters <perl5-porters [...] perl.org>
Subject: Re: [perl #123737] S_no_op: Assertion `s >= oldbp' failed. (toke.c:536)
Download (untitled) / with headers
text/plain 4.4k
On Sun, Mar 22, 2015 at 8:41 PM, Hugo van der Sanden via RT <perlbug-followup@perl.org> wrote:
Show quoted text
On Tue Mar 17 01:52:36 2015, brian.carpenter@gmail.com wrote:
> This bugger is back, albeit in a slight different part of toke.c.

I just noticed that the ticket was not reopened, I'll do that now.

Brian's new test case is:

% echo '0$#{' | ./miniperl -c
Array length found where operator expected at - line 1, near "0$#"
miniperl: toke.c:539: S_no_op: Assertion `s >= oldbp' failed.
Aborted (core dumped)
%


I'm not sure if the original fix was enough or the right place to fix things.

Back in 5.18.4, these both worked and reported the correct errors:

  mhorsfall@dory:~$ ~/dpppperls/debug/perl-5.18.4/bin/perl5.18.4 -ce '0$#{'
  Array length found where operator expected at -e line 1, near "0$#"
      (Missing operator before $#?)
  syntax error at -e line 1, near "0$#"
  Missing right curly or square bracket at -e line 1, at end of line
  -e had compilation errors.

  mhorsfall@dory:~$ ~/dpppperls/debug/perl-5.18.4/bin/perl5.18.4 -ce '0${'
  Scalar found where operator expected at -e line 1, near "0$"
      (Missing operator before $?)
  syntax error at -e line 1, near "0$"
  Missing right curly or square bracket at -e line 1, at end of line
  -e had compilation errors.

In 5.19.5 with the following commit, these started panicing:

   good - zero exit from ./perl -Ilib /home/mhorsfall/crash.pl
  a49b10d0a8dde2a4adb5a0a90e7b846b243e2514 is the first bad commit
  commit a49b10d0a8dde2a4adb5a0a90e7b846b243e2514
  Author: Brian Fraser <fraserbn@gmail.com>
  Date:   Sun Sep 1 20:41:26 2013 -0300
 
      toke.c, scan_ident(): use PEEKSPACE() to skip over whitespace.
     
      This fixes a number of bugs regarding whitespace and line numbers
      in scan_ident(), such as ${\nfoo\n} not increasing the line number,
      or ${\ntime\n[1]} not working.
     
      It goes through a number of hoops to get the correct line number for
      warnings emmitted from scan_ident, and reverts CopLINE to its
      original value if scan_ident() is giving up and returning from the
      point of the opening bracket, like in the case of ${\n\nfoo()}.
 
  :040000 040000 fea9796b35814ce4842f64bf81366bad5ee381ba 0bc66bc8eeb87e6160264cb0ae12e38e45803c1b M    t
  :100644 100644 53ad9f85ce0b819b1cf33fd53bf57c21b43b6c21 682fe67af183d23171c93ecc4499949d3fd2cfe2 M    toke.c
  bisect run success
  That took 742 seconds.

Later in 5.21.5, they started working again in non-debug builds, but begain reporting errors incorrectly:

("fixed" by:)

  commit 59685a4604d61709c23100c3754f52081ac007f1
  Author: Yves Orton <demerphq@gmail.com>
  Date:   Tue Sep 23 01:34:27 2014 +0200
 
      add an assert that the length arg for UTF8f is non-negative
     
      If we dont we will just hit a different more confusing assert
      later. In production builds we zero elen so the args is assumed
      empty.


  mhorsfall@dory:~/p5/perl$ runperls -dm 5.21.5 -e '0${'
   /home/mhorsfall/dpppperls/default/perl-5.21.5/bin/perl5.21.5 -e '0${' 2>&1
  Scalar found where operator expected at -e line 1, near "0$"
      (Missing operator before ?)
  syntax error at -e line 1, near "0$"
  Missing right curly or square bracket at -e line 1, at end of line
  Execution of -e aborted due to compilation errors.
  child exited with value 255

  mhorsfall@dory:~/p5/perl$ runperls -dm 5.21.5 -e '0$#{'
   /home/mhorsfall/dpppperls/default/perl-5.21.5/bin/perl5.21.5 -e '0$#{' 2>&1
  Array length found where operator expected at -e line 1, near "0$#"
      (Missing operator before ?)
  syntax error at -e line 1, near "0$#"
  Missing right curly or square bracket at -e line 1, at end of line
  Execution of -e aborted due to compilation errors.
  child exited with value 255
 
(Notice that "Missing operator before ?)" doesn't have the identifier anymore)

The fix for this ticket fixed the first case, but are these individual fixes needed for each case, or is there some more global fix that covers them all? (I'm really not sure)

Also, here's another broken one:

  mhorsfall@tworivers:~$ perl -e '0@'
  Array found where operator expected at -e line 1, at end of line
      (Missing operator before ?)
  syntax error at -e line 1, near "0@
  "
  Execution of -e aborted due to compilation errors.

Though that never appeared to report properly. Also the newline after the @ is strange...

In any case, I'm not sure if this needs to remain a blocker for 5.22 if we don't fix it in time since it's been broken since 5.20...

Cheers,

-- Matthew Horsfall (alh)


CC: perlbug-followup [...] perl.org, Perl5 Porters <perl5-porters [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #123737] S_no_op: Assertion `s >= oldbp' failed. (toke.c:536)
To: "Matthew Horsfall (alh)" <wolfsage [...] gmail.com>
Date: Sat, 25 Apr 2015 17:14:41 +0100
Download (untitled) / with headers
text/plain 413b
On Fri, Apr 24, 2015 at 01:06:41PM -0400, Matthew Horsfall (alh) wrote: Show quoted text
> In any case, I'm not sure if this needs to remain a blocker for 5.22 if we > don't fix it in time since it's been broken since 5.20...
I've just fixed the 0$#{ case with v5.21.11-17-g310a0d0, but I'll leave the ticket open in case someone wants to do a more general fix post 5.22. -- Never do today what you can put off till tomorrow.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.1k
On Fri Apr 24 10:07:10 2015, alh wrote: Show quoted text
> The fix for this ticket fixed the first case, but are these individual > fixes needed for each case, or is there some more global fix that covers > them all? (I'm really not sure)
I think they'll need individual fixes, since they depend on the handler for each token advancing the buffer pointer to provide enough context. Show quoted text
> Also, here's another broken one: > > mhorsfall@tworivers:~$ perl -e '0@' > Array found where operator expected at -e line 1, at end of line > (Missing operator before ?) > syntax error at -e line 1, near "0@ > " > Execution of -e aborted due to compilation errors. > > Though that never appeared to report properly. Also the newline after the @ > is strange...
The attached improves the Missing operator line, it doesn't try to handle '0@$foo' but helps for '0@foo'. The newline for the syntax error line isn't specific to @, it occurs for other similar synax errors too: $ ./perl -e '0$foo' Scalar found where operator expected at -e line 1, near "0$foo" (Missing operator before $foo?) syntax error at -e line 1, near "0$foo " Execution of -e aborted due to compilation errors. Tony
Subject: 0001-perl-123737-delay-reporting-a-missing-operator-for-a.patch
From 405867c6d927552e43332df4277784f77119b0e8 Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Wed, 25 Nov 2015 16:07:51 +1100 Subject: [perl #123737] delay reporting a missing operator for arrays Previously it was reported a the beginning of the '@' case, without even skipping the @ symbol. Make the code more similar to the scalar case and try to parse an identifier first. --- t/lib/croak/toke | 9 +++++++++ toke.c | 7 ++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/t/lib/croak/toke b/t/lib/croak/toke index 64012fb..50394da 100644 --- a/t/lib/croak/toke +++ b/t/lib/croak/toke @@ -37,6 +37,15 @@ syntax error at - line 1, near "0$#" Missing right curly or square bracket at - line 1, at end of line Execution of - aborted due to compilation errors. ######## +# NAME (Missing opertaor before @foo) [perl #123737] +0@foo +EXPECT +Array found where operator expected at - line 1, near "0@foo" + (Missing operator before @foo?) +syntax error at - line 1, near "0@foo +" +Execution of - aborted due to compilation errors. +######## # NAME Unterminated here-doc in string eval eval "<<foo"; die $@ EXPECT diff --git a/toke.c b/toke.c index 6d6975c..169c970 100644 --- a/toke.c +++ b/toke.c @@ -6353,11 +6353,12 @@ Perl_yylex(pTHX) TOKEN('$'); case '@': - if (PL_expect == XOPERATOR) - no_op("Array", s); - else if (PL_expect == XPOSTDEREF) POSTDEREF('@'); + if (PL_expect == XPOSTDEREF) + POSTDEREF('@'); PL_tokenbuf[0] = '@'; s = scan_ident(s, PL_tokenbuf + 1, sizeof PL_tokenbuf - 1, FALSE); + if (PL_expect == XOPERATOR) + no_op("Array", s); pl_yylval.ival = 0; if (!PL_tokenbuf[1]) { PREREF('@'); -- 2.1.4
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 640b
On Tue Nov 24 21:22:47 2015, tonyc wrote: Show quoted text
> > Also, here's another broken one: > > > > mhorsfall@tworivers:~$ perl -e '0@' > > Array found where operator expected at -e line 1, at end of line > > (Missing operator before ?) > > syntax error at -e line 1, near "0@ > > " > > Execution of -e aborted due to compilation errors. > > > > Though that never appeared to report properly. Also the newline after > > the @ > > is strange...
> > The attached improves the Missing operator line, it doesn't try to > handle '0@$foo' but helps for '0@foo'.
Applied as a7162bf74f38a7ede2efc930a958560392ce092f. Leaving open for further cases. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 416b
I see your fix in the git shortlog (http://perl5.git.perl.org/perl.git/shortlog), but Perl v5.23.7 (v5.23.6-104-g5dcc841) still SIGABRTs with perl -e '0@{': toke.c:539: S_no_op: Assertion `s >= oldbp' failed" Line 539 now as opposed to 536 in my original report. On Sun Jan 10 15:48:38 2016, tonyc wrote: Show quoted text
> > Applied as a7162bf74f38a7ede2efc930a958560392ce092f. > > Leaving open for further cases. > > Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 446b
On Mon Jan 11 16:55:50 2016, brian.carpenter@gmail.com wrote: Show quoted text
> I see your fix in the git shortlog > (http://perl5.git.perl.org/perl.git/shortlog), but Perl v5.23.7 > (v5.23.6-104-g5dcc841) still SIGABRTs with perl -e '0@{': > > toke.c:539: S_no_op: Assertion `s >= oldbp' failed" > > Line 539 now as opposed to 536 in my original report.
Oops, you're right, I got too into improving the message for 0@foo. The attached fixes it for me. Tony
Subject: 0001-perl-123737-handle-a-non-identifer-after-better-for-.patch
From 25dc4549efb21e888d4f0eaa858fa4fa2341562e Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Tue, 12 Jan 2016 15:39:00 +1100 Subject: [perl #123737] handle a non-identifer after @ better for a missing op Previously this would assert(). --- t/lib/croak/toke | 11 ++++++++++- toke.c | 10 ++++++++-- 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/t/lib/croak/toke b/t/lib/croak/toke index 50394da..18dfa24 100644 --- a/t/lib/croak/toke +++ b/t/lib/croak/toke @@ -37,7 +37,7 @@ syntax error at - line 1, near "0$#" Missing right curly or square bracket at - line 1, at end of line Execution of - aborted due to compilation errors. ######## -# NAME (Missing opertaor before @foo) [perl #123737] +# NAME (Missing operator before @foo) [perl #123737] 0@foo EXPECT Array found where operator expected at - line 1, near "0@foo" @@ -46,6 +46,15 @@ syntax error at - line 1, near "0@foo " Execution of - aborted due to compilation errors. ######## +# NAME (Missing operator before @{) [perl #123737] +0@{ +EXPECT +Array found where operator expected at - line 1, near "0@{" + (Missing operator before @{?) +syntax error at - line 1, near "0@" +Missing right curly or square bracket at - line 1, at end of line +Execution of - aborted due to compilation errors. +######## # NAME Unterminated here-doc in string eval eval "<<foo"; die $@ EXPECT diff --git a/toke.c b/toke.c index 95ce3fd..23c3521 100644 --- a/toke.c +++ b/toke.c @@ -6368,8 +6368,14 @@ Perl_yylex(pTHX) POSTDEREF('@'); PL_tokenbuf[0] = '@'; s = scan_ident(s, PL_tokenbuf + 1, sizeof PL_tokenbuf - 1, FALSE); - if (PL_expect == XOPERATOR) - no_op("Array", s); + if (PL_expect == XOPERATOR) { + d = s; + if (PL_bufptr > s) { + d = PL_bufptr-1; + PL_bufptr = PL_oldbufptr; + } + no_op("Array", d); + } pl_yylval.ival = 0; if (!PL_tokenbuf[1]) { PREREF('@'); -- 2.1.4
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 707b
On Mon Jan 11 20:39:59 2016, tonyc wrote: Show quoted text
> On Mon Jan 11 16:55:50 2016, brian.carpenter@gmail.com wrote:
> > I see your fix in the git shortlog > > (http://perl5.git.perl.org/perl.git/shortlog), but Perl v5.23.7 > > (v5.23.6-104-g5dcc841) still SIGABRTs with perl -e '0@{': > > > > toke.c:539: S_no_op: Assertion `s >= oldbp' failed" > > > > Line 539 now as opposed to 536 in my original report.
> > Oops, you're right, I got too into improving the message for 0@foo. > > The attached fixes it for me.
Applied as 61d30259f32e5bba52238d60608b6909208eb604. I also checked for similar problems in other calls to no_op() and didn't see any other cases I could make crash, so I'll close this ticket. Tony
Download (untitled) / with headers
text/plain 252b
Thank you for submitting this report. You have helped make Perl better. With the release of Perl 5.24.0 on May 9, 2016, this and 149 other issues have been resolved. Perl 5.24.0 may be downloaded via https://metacpan.org/release/RJBS/perl-5.24.0


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org