Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regular Expression matching in signal handler causes side-effects #12405

Closed
p5pRT opened this issue Sep 13, 2012 · 30 comments
Closed

Regular Expression matching in signal handler causes side-effects #12405

p5pRT opened this issue Sep 13, 2012 · 30 comments

Comments

@p5pRT
Copy link

p5pRT commented Sep 13, 2012

Migrated from rt.perl.org#114878 (status was 'resolved')

Searchable as RT114878$

@p5pRT
Copy link
Author

p5pRT commented Sep 13, 2012

From tim@electronghost.co.uk

Created by tim@electronghost.co.uk

Using Safe signals...
If a signal arrives while a regular expression is being processed, and the
signal handler contains an RE itself, then the original RE is silently not
executed, causing bugs which appear "impossible".

Here is a test case​:

====== CUT HERE ======
#!/usr/bin/env perl

sub sighup {
  my $bar="This-Has-Dashes-HUP";
  $bar=~s/.*-//;
  print "$bar\n";
 
}

$SIG{'HUP'}=\&sighup;

while (1) {
  my $foo="This​:Has​:Colons";
  $foo=~s/.*​://;
  if ($foo=~m/​:/) {
  print "BUG!!​: $foo\n";
  }
}
====== CUT HERE ======

Run this endless loop and arrange to send it a SIGHUP once per second.

On PERL 5.14 (and, presumably, later versions) you see the following output​:

bash$ perl ./t.pl
HUP
HUP
HUP
HUP
BUG!!​: This​:Has​:Colons
HUP
HUP
BUG!!​: This​:Has​:Colons
HUP
BUG!!​: This​:Has​:Colons
HUP
BUG!!​: This​:Has​:Colons
HUP
HUP
BUG!!​: This​:Has​:Colons
HUP
BUG!!​: This​:Has​:Colons
HUP
HUP
BUG!!​: This​:Has​:Colons
HUP
HUP
HUP
BUG!!​: This​:Has​:Colons

On PERL 5.12 and earlier, you see correct output​:

bash$ perl ./t.pl
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP
HUP

Silently failing to execute code seems like a critical failure, so marking
this critical :-)

Perl Info

Flags:
    category=core
    severity=critical

This perlbug was built using Perl 5.14.2 in the Fedora build system.
It is being executed now by Perl 5.14.2 - Wed May 30 14:55:10 UTC 2012.

Site configuration information for perl 5.14.2:

Configured by Red Hat, Inc. at Wed May 30 14:55:10 UTC 2012.

Summary of my perl5 (revision 5 version 14 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=2.6.32-220.4.1.el6.x86_64, archname=x86_64-linux-thread-multi
    uname='linux x86-14.phx2.fedoraproject.org 2.6.32-220.4.1.el6.x86_64 #1 smp thu jan 19 14:50:54 est 2012 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4  -m64 -mtune=generic -Dccdlflags=-Wl,--enable-new-dtags -Dlddlflags=-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4  -m64 -mtune=generic -Wl,-z,relro  -DDEBUGGING=-g -Dversion=5.14.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbin!
 perl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.7.0 20120507 (Red Hat 4.7.0-5)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -fstack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.15'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64/perl5/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wl,-z,relro '

Locally applied patches:
    


@INC for perl 5.14.2:
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5
    .


Environment for perl 5.14.2:
    HOME=/home/tim
    LANG=en_US.UTF-8
    LANGUAGE=
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/home/tim/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From mark.s.phillips@outlook.com

Also broken in 5.14.0 and 5.16.1.

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From [Unknown Contact. See original ticket]

Also broken in 5.14.0 and 5.16.1.

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From mark.s.phillips@outlook.com

To cut a long story short –
1) It works in the latest git bleeding edge 5.17.4+
2) It was fixed by changelist b93070e
3) I can confirm that backporting this changelist to 5.14.2 fixes the issue.

b93070e
“eliminate PL_reglast(close)?paren, PL_regoffs
 
  eliminate the three vars
  PL_reglastcloseparen
  PL_reglastparen
  PL_regoffs
  (which are actually aliases to PL_reg_state struct elements).
 
  These three vars always point to the corresponding fields within the
  currently executing regex; so just access those fields directly instead.
 
  This makes switching between regexes with (??{}) simpler​: just update
  rex, and everything automatically references the new fields.”

My guess is that what is happening is that somehow the cache variables
(which have been eliminated) are becoming corrupt when the signal happens.

Testing trail​:-

5.17.4 bleeding edge git - Works - version
b71c54b
7.17.3 git - Works - version 288edf5
5.17.2 git - Works - version 9895e5d
5.17.1 git - Works - version 955b6ac
2012/06/19

Works cec7265 2012/06/06

Works f6033a9 2012/05/25

Works f067efb 2012/05/21

Works b3fd53f 2012/05/15 eliminate
PL_reg_start_tmp, PL_reg_start_tmpl

Works b93070e 2012/05/15 eliminate
PL_reglast(close)?paren, PL_regoffs

Broken f8ff62a 2012/06/05 make Perl_...
and my_re_op_compile sigs match

Broken 5f616ea 2012/04/01

Broken 1147b55 2011/11/18

Broken c71dea7 2012/06/07

5.17.0 git - Broken - version 028eed1
2012/05/26

5.16.1 - Broken
5.14.2 - Broken
5.14.0 - Broken (still broken with -O0)
5.13.5 - Broken
5.12.4 - Works

@p5pRT
Copy link
Author

p5pRT commented Sep 14, 2012

From [Unknown Contact. See original ticket]

To cut a long story short –
1) It works in the latest git bleeding edge 5.17.4+
2) It was fixed by changelist b93070e
3) I can confirm that backporting this changelist to 5.14.2 fixes the issue.

b93070e
“eliminate PL_reglast(close)?paren, PL_regoffs
 
  eliminate the three vars
  PL_reglastcloseparen
  PL_reglastparen
  PL_regoffs
  (which are actually aliases to PL_reg_state struct elements).
 
  These three vars always point to the corresponding fields within the
  currently executing regex; so just access those fields directly instead.
 
  This makes switching between regexes with (??{}) simpler​: just update
  rex, and everything automatically references the new fields.”

My guess is that what is happening is that somehow the cache variables
(which have been eliminated) are becoming corrupt when the signal happens.

Testing trail​:-

5.17.4 bleeding edge git - Works - version
b71c54b
7.17.3 git - Works - version 288edf5
5.17.2 git - Works - version 9895e5d
5.17.1 git - Works - version 955b6ac
2012/06/19

Works cec7265 2012/06/06

Works f6033a9 2012/05/25

Works f067efb 2012/05/21

Works b3fd53f 2012/05/15 eliminate
PL_reg_start_tmp, PL_reg_start_tmpl

Works b93070e 2012/05/15 eliminate
PL_reglast(close)?paren, PL_regoffs

Broken f8ff62a 2012/06/05 make Perl_...
and my_re_op_compile sigs match

Broken 5f616ea 2012/04/01

Broken 1147b55 2011/11/18

Broken c71dea7 2012/06/07

5.17.0 git - Broken - version 028eed1
2012/05/26

5.16.1 - Broken
5.14.2 - Broken
5.14.0 - Broken (still broken with -O0)
5.13.5 - Broken
5.12.4 - Works

@p5pRT
Copy link
Author

p5pRT commented Sep 15, 2012

@jkeenan - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Sep 25, 2012

From mark.s.phillips@outlook.com

Unfortunately simply backporting the fix from 5.17 chagelist
b93070e breaks binary compatibility
with plugins.

In particular after re-compiling and installing the Fedora 17 perl RPM
(perl-5.14.2-212.fc17.src.rpm), the automake package started crashing.
The regression tests were all ok.

I have tweaked the back-port to maintain compatibility in three areas (I
am not sure which are important) and now it maintains compatibility and
works with other pre-compiled perl 5.14.2 packages.

The areas I tweaked were​:-
1) Modified regcppush/regcppop to push/pop the same values as before the
patch (in particular to still push/pop PL_regoffs, even though it is no
longer used).

2) Modified regcppush/regcppop to have the same prototype as before the
patch, except to make the argument to regcppop non-const. The pop code
writes to both rex and the (unused) PL_* variables.

3) Left the deprecrated PL variables in re_save_state so that the
interpretor object is unchanged - I suspect this is the critical change.

4) The PL_regoffs/PL_reglastparen/PL_reglastcloseparen variables are now
only used by regcppush. I have modified everywere regcppush is called to
set them correctly before regcppush...

I have attached the working patch which appears to pass regression
testing and allows a pre-compiled automake to work when using other
pre-compiled 5.14.2 libs.

@p5pRT
Copy link
Author

p5pRT commented Sep 25, 2012

From [Unknown Contact. See original ticket]

Unfortunately simply backporting the fix from 5.17 chagelist
b93070e breaks binary compatibility
with plugins.

In particular after re-compiling and installing the Fedora 17 perl RPM
(perl-5.14.2-212.fc17.src.rpm), the automake package started crashing.
The regression tests were all ok.

I have tweaked the back-port to maintain compatibility in three areas (I
am not sure which are important) and now it maintains compatibility and
works with other pre-compiled perl 5.14.2 packages.

The areas I tweaked were​:-
1) Modified regcppush/regcppop to push/pop the same values as before the
patch (in particular to still push/pop PL_regoffs, even though it is no
longer used).

2) Modified regcppush/regcppop to have the same prototype as before the
patch, except to make the argument to regcppop non-const. The pop code
writes to both rex and the (unused) PL_* variables.

3) Left the deprecrated PL variables in re_save_state so that the
interpretor object is unchanged - I suspect this is the critical change.

4) The PL_regoffs/PL_reglastparen/PL_reglastcloseparen variables are now
only used by regcppush. I have modified everywere regcppush is called to
set them correctly before regcppush...

I have attached the working patch which appears to pass regression
testing and allows a pre-compiled automake to work when using other
pre-compiled 5.14.2 libs.

@p5pRT
Copy link
Author

p5pRT commented Sep 25, 2012

From mark.s.phillips@outlook.com

Working patch attached

@p5pRT
Copy link
Author

p5pRT commented Sep 25, 2012

From mark.s.phillips@outlook.com

perl-5.14.2-kill.patch
diff -ur perl-5.14.2.orig/embed.fnc perl-5.14.2/embed.fnc
--- perl-5.14.2.orig/embed.fnc	2011-09-26 10:44:34.000000000 +0100
+++ perl-5.14.2/embed.fnc	2012-09-25 11:33:50.459083508 +0100
@@ -1862,7 +1862,7 @@
 ERs	|bool	|reginclass	|NULLOK const regexp * const prog|NN const regnode * const n|NN const U8 * const p|NULLOK STRLEN *lenp\
 				|bool const do_utf8sv_is_utf8
 Es	|CHECKPOINT|regcppush	|I32 parenfloor
-Es	|char*	|regcppop	|NN const regexp *rex
+Es	|char*	|regcppop	|NN regexp *rex
 ERsn	|U8*	|reghop3	|NN U8 *s|I32 off|NN const U8 *lim
 #ifdef XXX_dmq
 ERsn	|U8*	|reghop4	|NN U8 *s|I32 off|NN const U8 *llim \
diff -ur perl-5.14.2.orig/proto.h perl-5.14.2/proto.h
--- perl-5.14.2.orig/proto.h	2011-09-26 10:44:34.000000000 +0100
+++ perl-5.14.2/proto.h	2012-09-25 10:49:23.418716467 +0100
@@ -6245,7 +6245,7 @@
 #define PERL_ARGS_ASSERT_REG_CHECK_NAMED_BUFF_MATCHED	\
 	assert(rex); assert(scan)
 
-STATIC char*	S_regcppop(pTHX_ const regexp *rex)
+STATIC char*	S_regcppop(pTHX_ regexp *rex)
 			__attribute__nonnull__(pTHX_1);
 #define PERL_ARGS_ASSERT_REGCPPOP	\
 	assert(rex)
diff -ur perl-5.14.2.orig/regexec.c perl-5.14.2/regexec.c
--- perl-5.14.2.orig/regexec.c	2011-09-26 10:44:34.000000000 +0100
+++ perl-5.14.2/regexec.c	2012-09-25 11:13:18.672575209 +0100
@@ -389,7 +389,7 @@
     regcpblow(cp)
 
 STATIC char *
-S_regcppop(pTHX_ const regexp *rex)
+S_regcppop(pTHX_ regexp *rex)
 {
     dVAR;
     UV i;
@@ -403,8 +403,10 @@
     assert((i & SAVE_MASK) == SAVEt_REGCONTEXT); /* Check that the magic cookie is there. */
     i >>= SAVE_TIGHT_SHIFT; /* Parentheses elements to pop. */
     input = (char *) SSPOPPTR;
-    *PL_reglastcloseparen = SSPOPINT;
-    *PL_reglastparen = SSPOPINT;
+    rex->lastcloseparen = SSPOPINT;
+    PL_reglastcloseparen = &rex->lastcloseparen;
+    rex->lastparen = SSPOPINT;
+    PL_reglastparen = &rex->lastparen;
     PL_regsize = SSPOPINT;
     PL_regoffs=(regexp_paren_pair *) SSPOPPTR;
 
@@ -414,24 +416,24 @@
 	I32 tmps;
 	U32 paren = (U32)SSPOPINT;
 	PL_reg_start_tmp[paren] = (char *) SSPOPPTR;
-	PL_regoffs[paren].start = SSPOPINT;
+	rex->offs[paren].start = SSPOPINT;
 	tmps = SSPOPINT;
-	if (paren <= *PL_reglastparen)
-	    PL_regoffs[paren].end = tmps;
+	if (paren <= rex->lastparen)
+	    rex->offs[paren].end = tmps;
 	DEBUG_BUFFERS_r(
 	    PerlIO_printf(Perl_debug_log,
 			  "     restoring \\%"UVuf" to %"IVdf"(%"IVdf")..%"IVdf"%s\n",
-			  (UV)paren, (IV)PL_regoffs[paren].start,
+			  (UV)paren, (IV)rex->offs[paren].start,
 			  (IV)(PL_reg_start_tmp[paren] - PL_bostr),
-			  (IV)PL_regoffs[paren].end,
-			  (paren > *PL_reglastparen ? "(no)" : ""));
+			  (IV)rex->offs[paren].end,
+			  (paren > rex->lastparen ? "(no)" : ""));
 	);
     }
     DEBUG_BUFFERS_r(
-	if (*PL_reglastparen + 1 <= rex->nparens) {
+	if (rex->lastparen + 1 <= rex->nparens) {
 	    PerlIO_printf(Perl_debug_log,
 			  "     restoring \\%"IVdf"..\\%"IVdf" to undef\n",
-			  (IV)(*PL_reglastparen + 1), (IV)rex->nparens);
+			  (IV)(rex->lastparen + 1), (IV)rex->nparens);
 	}
     );
 #if 1
@@ -444,10 +446,10 @@
      * this code seems to be necessary or otherwise
      * this erroneously leaves $1 defined: "1" =~ /^(?:(\d)x)?\d$/
      * --jhi updated by dapm */
-    for (i = *PL_reglastparen + 1; i <= rex->nparens; i++) {
+    for (i = rex->lastparen + 1; i <= rex->nparens; i++) {
 	if (i > PL_regsize)
-	    PL_regoffs[i].start = -1;
-	PL_regoffs[i].end = -1;
+	    rex->offs[i].start = -1;
+	rex->offs[i].end = -1;
     }
 #endif
     return input;
@@ -2608,9 +2610,9 @@
      * --jhi updated by dapm */
 #if 1
     if (prog->nparens) {
-	regexp_paren_pair *pp = PL_regoffs;
+	regexp_paren_pair *pp = prog->offs;
 	register I32 i;
-	for (i = prog->nparens; i > (I32)*PL_reglastparen; i--) {
+	for (i = prog->nparens; i > (I32)prog->lastparen; i--) {
 	    ++pp;
 	    pp->start = -1;
 	    pp->end = -1;
@@ -2619,7 +2621,7 @@
 #endif
     REGCP_SET(lastcp);
     if (regmatch(reginfo, progi->program + 1)) {
-	PL_regoffs[0].end = PL_reginput - PL_bostr;
+	prog->offs[0].end = PL_reginput - PL_bostr;
 	return 1;
     }
     if (reginfo->cutpoint)
@@ -2955,8 +2957,8 @@
     PERL_ARGS_ASSERT_REG_CHECK_NAMED_BUFF_MATCHED;
 
     for ( n=0; n<SvIVX(sv_dat); n++ ) {
-        if ((I32)*PL_reglastparen >= nums[n] &&
-            PL_regoffs[nums[n]].end != -1)
+        if ((I32)rex->lastparen >= nums[n] &&
+            rex->offs[nums[n]].end != -1)
         {
             return nums[n];
         }
@@ -3132,14 +3134,14 @@
 
 	case KEEPS:
 	    /* update the startpoint */
-	    st->u.keeper.val = PL_regoffs[0].start;
+	    st->u.keeper.val = rex->offs[0].start;
 	    PL_reginput = locinput;
-	    PL_regoffs[0].start = locinput - PL_bostr;
+	    rex->offs[0].start = locinput - PL_bostr;
 	    PUSH_STATE_GOTO(KEEPS_next, next);
 	    /*NOT-REACHED*/
 	case KEEPS_next_fail:
 	    /* rollback the start point change */
-	    PL_regoffs[0].start = st->u.keeper.val;
+	    rex->offs[0].start = st->u.keeper.val;
 	    sayNO_SILENT;
 	    /*NOT-REACHED*/
 	case EOL:
@@ -3397,9 +3399,9 @@
 	case TRIE_next_fail: /* we failed - try next alternative */
             if ( ST.jump) {
                 REGCP_UNWIND(ST.cp);
-	        for (n = *PL_reglastparen; n > ST.lastparen; n--)
-		    PL_regoffs[n].end = -1;
-	        *PL_reglastparen = n;
+	        for (n = rex->lastparen; n > ST.lastparen; n--)
+		    rex->offs[n].end = -1;
+	        rex->lastparen = n;
 	    }
 	    if (!--ST.accepted) {
 	        DEBUG_EXECUTE_r({
@@ -3433,7 +3435,7 @@
             }
 
             if ( ST.jump) {
-                ST.lastparen = *PL_reglastparen;
+                ST.lastparen = rex->lastparen;
 	        REGCP_SET(ST.cp);
             }
 
@@ -4074,11 +4076,11 @@
 	    n = ARG(scan);  /* which paren pair */
 
 	  do_nref_ref_common:
-	    ln = PL_regoffs[n].start;
+	    ln = rex->offs[n].start;
 	    PL_reg_leftiter = PL_reg_maxiter;		/* Void cache */
-	    if (*PL_reglastparen < n || ln == -1)
+	    if (rex->lastparen < n || ln == -1)
 		sayNO;			/* Do not match unless seen CLOSEn. */
-	    if (ln == PL_regoffs[n].end)
+	    if (ln == rex->offs[n].end)
 		break;
 
 	    s = PL_bostr + ln;
@@ -4092,7 +4094,7 @@
 		    * not going off the end given by PL_regeol, and returns in
 		    * limit upon success, how much of the current input was
 		    * matched */
-		if (! foldEQ_utf8_flags(s, NULL, PL_regoffs[n].end - ln, utf8_target,
+		if (! foldEQ_utf8_flags(s, NULL, rex->offs[n].end - ln, utf8_target,
 				    locinput, &limit, 0, utf8_target, utf8_fold_flags))
 		{
 		    sayNO;
@@ -4107,7 +4109,7 @@
 		(type == REF ||
 		 UCHARAT(s) != fold_array[nextchr]))
 		sayNO;
-	    ln = PL_regoffs[n].end - ln;
+	    ln = rex->offs[n].end - ln;
 	    if (locinput + ln > PL_regeol)
 		sayNO;
 	    if (ln > 1 && (type == REF
@@ -4196,7 +4198,7 @@
 		DEBUG_STATE_r( PerlIO_printf(Perl_debug_log, 
 		    "  re_eval 0x%"UVxf"\n", PTR2UV(PL_op)) );
 		PAD_SAVE_LOCAL(old_comppad, (PAD*)rexi->data->data[n + 2]);
-		PL_regoffs[0].end = PL_reg_magic->mg_len = locinput - PL_bostr;
+		rex->offs[0].end = PL_reg_magic->mg_len = locinput - PL_bostr;
 
                 if (sv_yes_mark) {
                     SV *sv_mrk = get_sv("REGMARK", 1);
@@ -4318,14 +4320,13 @@
 
         eval_recurse_doit: /* Share code with GOSUB below this line */                		
 		/* run the pattern returned from (??{...}) */
+		
+		PL_reglastcloseparen = &rex->lastcloseparen;
+		PL_reglastparen = &rex->lastparen;
 		ST.cp = regcppush(0);	/* Save *all* the positions. */
 		REGCP_SET(ST.lastcp);
 		
-		PL_regoffs = re->offs; /* essentially NOOP on GOSUB */
-		
 		/* see regtry, specifically PL_reglast(?:close)?paren is a pointer! (i dont know why) :dmq */
-		PL_reglastparen = &re->lastparen;
-		PL_reglastcloseparen = &re->lastcloseparen;
 		re->lastparen = 0;
 		re->lastcloseparen = 0;
 
@@ -4372,12 +4373,6 @@
 	    cur_eval = ST.prev_eval;
 	    cur_curlyx = ST.prev_curlyx;
 
-	    /* rex was changed so update the pointer in PL_reglastparen and PL_reglastcloseparen */
-	    PL_reglastparen = &rex->lastparen;
-	    PL_reglastcloseparen = &rex->lastcloseparen;
-	    /* also update PL_regoffs */
-	    PL_regoffs = rex->offs;
-	    
 	    /* XXXX This is too dramatic a measure... */
 	    PL_reg_maxiter = 0;
             if ( nochange_depth )
@@ -4392,9 +4387,6 @@
 	    SETREX(rex_sv,ST.prev_rex);
 	    rex = (struct regexp *)SvANY(rex_sv);
 	    rexi = RXi_GET(rex); 
-	    /* rex was changed so update the pointer in PL_reglastparen and PL_reglastcloseparen */
-	    PL_reglastparen = &rex->lastparen;
-	    PL_reglastcloseparen = &rex->lastcloseparen;
 
 	    PL_reginput = locinput;
 	    REGCP_UNWIND(ST.lastcp);
@@ -4417,13 +4409,13 @@
 	    break;
 	case CLOSE:
 	    n = ARG(scan);  /* which paren pair */
-	    PL_regoffs[n].start = PL_reg_start_tmp[n] - PL_bostr;
-	    PL_regoffs[n].end = locinput - PL_bostr;
+	    rex->offs[n].start = PL_reg_start_tmp[n] - PL_bostr;
+	    rex->offs[n].end = locinput - PL_bostr;
 	    /*if (n > PL_regsize)
 		PL_regsize = n;*/
-	    if (n > *PL_reglastparen)
-		*PL_reglastparen = n;
-	    *PL_reglastcloseparen = n;
+	    if (n > rex->lastparen)
+		rex->lastparen = n;
+	    rex->lastcloseparen = n;
             if (cur_eval && cur_eval->u.eval.close_paren == n) {
 	        goto fake_end;
 	    }    
@@ -4438,14 +4430,14 @@
                     if ( OP(cursor)==CLOSE ){
                         n = ARG(cursor);
                         if ( n <= lastopen ) {
-                            PL_regoffs[n].start
+                            rex->offs[n].start
 				= PL_reg_start_tmp[n] - PL_bostr;
-                            PL_regoffs[n].end = locinput - PL_bostr;
+                            rex->offs[n].end = locinput - PL_bostr;
                             /*if (n > PL_regsize)
                             PL_regsize = n;*/
-                            if (n > *PL_reglastparen)
-                                *PL_reglastparen = n;
-                            *PL_reglastcloseparen = n;
+                            if (n > rex->lastparen)
+                                rex->lastparen = n;
+                            rex->lastcloseparen = n;
                             if ( n == ARG(scan) || (cur_eval &&
                                 cur_eval->u.eval.close_paren == n))
                                 break;
@@ -4457,7 +4449,7 @@
 	    /*NOTREACHED*/	    
 	case GROUPP:
 	    n = ARG(scan);  /* which paren pair */
-	    sw = cBOOL(*PL_reglastparen >= n && PL_regoffs[n].end != -1);
+	    sw = cBOOL(rex->lastparen >= n && rex->offs[n].end != -1);
 	    break;
 	case NGROUPP:
 	    /* reg_check_named_buff_matched returns 0 for no match */
@@ -4580,8 +4572,8 @@
 
 	    /* XXXX Probably it is better to teach regpush to support
 	       parenfloor > PL_regsize... */
-	    if (parenfloor > (I32)*PL_reglastparen)
-		parenfloor = *PL_reglastparen; /* Pessimization... */
+	    if (parenfloor > (I32)rex->lastparen)
+		parenfloor = rex->lastparen; /* Pessimization... */
 
 	    ST.prev_curlyx= cur_curlyx;
 	    cur_curlyx = st;
@@ -4641,6 +4633,8 @@
 	    /* First just match a string of min A's. */
 
 	    if (n < min) {
+		PL_reglastcloseparen = &rex->lastcloseparen;
+		PL_reglastparen = &rex->lastparen;
 		ST.cp = regcppush(cur_curlyx->u.curlyx.parenfloor);
 		cur_curlyx->u.curlyx.lastloc = locinput;
 		REGCP_SET(ST.lastcp);
@@ -4717,6 +4711,8 @@
 	    if (cur_curlyx->u.curlyx.minmod) {
 		ST.save_curlyx = cur_curlyx;
 		cur_curlyx = cur_curlyx->u.curlyx.prev_curlyx;
+		PL_reglastcloseparen = &rex->lastcloseparen;
+		PL_reglastparen = &rex->lastparen;
 		ST.cp = regcppush(ST.save_curlyx->u.curlyx.parenfloor);
 		REGCP_SET(ST.lastcp);
 		PUSH_YES_STATE_GOTO(WHILEM_B_min, ST.save_curlyx->u.curlyx.B);
@@ -4726,6 +4722,8 @@
 	    /* Prefer A over B for maximal matching. */
 
 	    if (n < max) { /* More greed allowed? */
+		PL_reglastcloseparen = &rex->lastcloseparen;
+		PL_reglastparen = &rex->lastparen;
 		ST.cp = regcppush(cur_curlyx->u.curlyx.parenfloor);
 		cur_curlyx->u.curlyx.lastloc = locinput;
 		REGCP_SET(ST.lastcp);
@@ -4811,6 +4809,8 @@
 	    /* Try grabbing another A and see if it helps. */
 	    PL_reginput = locinput;
 	    cur_curlyx->u.curlyx.lastloc = locinput;
+	    PL_reglastcloseparen = &rex->lastcloseparen;
+	    PL_reglastparen = &rex->lastparen;
 	    ST.cp = regcppush(cur_curlyx->u.curlyx.parenfloor);
 	    REGCP_SET(ST.lastcp);
 	    PUSH_STATE_GOTO(WHILEM_A_min,
@@ -4829,7 +4829,7 @@
 
 	case BRANCH:	    /*  /(...|A|...)/ */
 	    scan = NEXTOPER(scan); /* scan now points to inner node */
-	    ST.lastparen = *PL_reglastparen;
+	    ST.lastparen = rex->lastparen;
 	    ST.next_branch = next;
 	    REGCP_SET(ST.cp);
 	    PL_reginput = locinput;
@@ -4863,10 +4863,10 @@
 	        no_final = 0;
 	    }
 	    REGCP_UNWIND(ST.cp);
-	    for (n = *PL_reglastparen; n > ST.lastparen; n--)
-		PL_regoffs[n].end = -1;
-	    *PL_reglastparen = n;
-	    /*dmq: *PL_reglastcloseparen = n; */
+	    for (n = rex->lastparen; n > ST.lastparen; n--)
+		rex->offs[n].end = -1;
+	    rex->lastparen = n;
+	    /*dmq: rex->lastcloseparen = n; */
 	    scan = ST.next_branch;
 	    /* no more branches? */
 	    if (!scan || (OP(scan) != BRANCH && OP(scan) != BRANCHJ)) {
@@ -4905,8 +4905,8 @@
 		U32 paren = ST.me->flags;
 		if (paren > PL_regsize)
 		    PL_regsize = paren;
-		if (paren > *PL_reglastparen)
-		    *PL_reglastparen = paren;
+		if (paren > rex->lastparen)
+		    rex->lastparen = paren;
 		scan += NEXT_OFF(scan); /* Skip former OPEN. */
 	    }
 	    ST.A = scan;
@@ -5033,13 +5033,13 @@
 		/* mark current A as captured */
 		I32 paren = ST.me->flags;
 		if (ST.count) {
-		    PL_regoffs[paren].start
+		    rex->offs[paren].start
 			= HOPc(PL_reginput, -ST.alen) - PL_bostr;
-		    PL_regoffs[paren].end = PL_reginput - PL_bostr;
-		    /*dmq: *PL_reglastcloseparen = paren; */
+		    rex->offs[paren].end = PL_reginput - PL_bostr;
+		    /*dmq: rex->lastcloseparen = paren; */
 		}
 		else
-		    PL_regoffs[paren].end = -1;
+		    rex->offs[paren].end = -1;
 		if (cur_eval && cur_eval->u.eval.close_paren &&
 		    cur_eval->u.eval.close_paren == (U32)ST.me->flags) 
 		{
@@ -5074,12 +5074,12 @@
 #define CURLY_SETPAREN(paren, success) \
     if (paren) { \
 	if (success) { \
-	    PL_regoffs[paren].start = HOPc(locinput, -1) - PL_bostr; \
-	    PL_regoffs[paren].end = locinput - PL_bostr; \
-	    *PL_reglastcloseparen = paren; \
+	    rex->offs[paren].start = HOPc(locinput, -1) - PL_bostr; \
+	    rex->offs[paren].end = locinput - PL_bostr; \
+	    rex->lastcloseparen = paren; \
 	} \
 	else \
-	    PL_regoffs[paren].end = -1; \
+	    rex->offs[paren].end = -1; \
     }
 
 	case STAR:		/*  /A*B/ where A is width 1 */
@@ -5098,8 +5098,8 @@
 	    ST.paren = scan->flags;	/* Which paren to set */
 	    if (ST.paren > PL_regsize)
 		PL_regsize = ST.paren;
-	    if (ST.paren > *PL_reglastparen)
-		*PL_reglastparen = ST.paren;
+	    if (ST.paren > rex->lastparen)
+		rex->lastparen = ST.paren;
 	    ST.min = ARG1(scan);  /* min to match */
 	    ST.max = ARG2(scan);  /* max to match */
 	    if (cur_eval && cur_eval->u.eval.close_paren &&
@@ -5255,7 +5255,7 @@
 	case CURLY_B_min_known_fail:
 	    /* failed to find B in a non-greedy match where c1,c2 valid */
 	    if (ST.paren && ST.count)
-		PL_regoffs[ST.paren].end = -1;
+		rex->offs[ST.paren].end = -1;
 
 	    PL_reginput = locinput;	/* Could be reset... */
 	    REGCP_UNWIND(ST.cp);
@@ -5333,7 +5333,7 @@
 	case CURLY_B_min_fail:
 	    /* failed to find B in a non-greedy match where c1,c2 invalid */
 	    if (ST.paren && ST.count)
-		PL_regoffs[ST.paren].end = -1;
+		rex->offs[ST.paren].end = -1;
 
 	    REGCP_UNWIND(ST.cp);
 	    /* failed -- move forward one */
@@ -5380,7 +5380,7 @@
 	case CURLY_B_max_fail:
 	    /* failed to find B in a greedy match */
 	    if (ST.paren && ST.count)
-		PL_regoffs[ST.paren].end = -1;
+		rex->offs[ST.paren].end = -1;
 
 	    REGCP_UNWIND(ST.cp);
 	    /*  back up. */
@@ -5401,16 +5401,14 @@
 		PL_reg_flags ^= st->u.eval.toggle_reg_flags; 
 
 		st->u.eval.prev_rex = rex_sv;		/* inner */
+		PL_reglastcloseparen = &rex->lastcloseparen;
+		PL_reglastparen = &rex->lastparen;
+		st->u.eval.cp = regcppush(0); /* Save *all* the positions. */
 		SETREX(rex_sv,cur_eval->u.eval.prev_rex);
 		rex = (struct regexp *)SvANY(rex_sv);
 		rexi = RXi_GET(rex);
 		cur_curlyx = cur_eval->u.eval.prev_curlyx;
 		(void)ReREFCNT_inc(rex_sv);
-		st->u.eval.cp = regcppush(0);	/* Save *all* the positions. */
-
-		/* rex was changed so update the pointer in PL_reglastparen and PL_reglastcloseparen */
-		PL_reglastparen = &rex->lastparen;
-		PL_reglastcloseparen = &rex->lastcloseparen;
 
 		REGCP_SET(st->u.eval.lastcp);
 		PL_reginput = locinput;

@p5pRT
Copy link
Author

p5pRT commented Jan 15, 2013

From @jkeenan

On Tue Sep 25 05​:03​:58 2012, msp wrote​:

Working patch attached

P5P​: Can we get a ruling on whether we still accept patches for Perl 5.14?

And setting that aside, what do we think of the proposed patch?

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Jan 15, 2013

From @rjbs

* James E Keenan via RT <perlbug-followup@​perl.org> [2013-01-14T22​:02​:28]

On Tue Sep 25 05​:03​:58 2012, msp wrote​:

Working patch attached

P5P​: Can we get a ruling on whether we still accept patches for Perl 5.14?

Yes, we do. 5.14 is not general-fix-EOL until the release of 5.18. If there
are fixes for 5.14 waiting in maint-5.14 when 5.18.0 is released, 5.14 will
have probably a final release with them.

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented Jan 18, 2013

From @iabyn

On Mon, Jan 14, 2013 at 07​:02​:28PM -0800, James E Keenan via RT wrote​:

And setting that aside, what do we think of the proposed patch?

gets a +1 from me

--
A major Starfleet emergency breaks out near the Enterprise, but
fortunately some other ships in the area are able to deal with it to
everyone's satisfaction.
  -- Things That Never Happen in "Star Trek" #13

@p5pRT
Copy link
Author

p5pRT commented May 7, 2013

From mark.s.phillips@outlook.com

What is the status of this bug?
Any chance of getting it fixed because it is painful having to rebuild
perl every time redhat ships another broken version​:-)

@p5pRT
Copy link
Author

p5pRT commented May 7, 2013

From [Unknown Contact. See original ticket]

What is the status of this bug?
Any chance of getting it fixed because it is painful having to rebuild
perl every time redhat ships another broken version​:-)

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @khwilliamson

On 05/07/2013 07​:17 AM, Mark Phillips via RT wrote​:

What is the status of this bug?
Any chance of getting it fixed because it is painful having to rebuild
perl every time redhat ships another broken version​:-)

As another data point, I get this​:
§ myperl test.pl
HUP
HUP
panic​: sv_chop ptr=9486829, start=9494518, end=9494528 at test.pl line 13.

(In repeated runs)

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @khwilliamson

On 05/07/2013 06​:44 PM, Karl Williamson wrote​:

On 05/07/2013 07​:17 AM, Mark Phillips via RT wrote​:

What is the status of this bug?
Any chance of getting it fixed because it is painful having to rebuild
perl every time redhat ships another broken version​:-)

As another data point, I get this​:
§ myperl test.pl
HUP
HUP
panic​: sv_chop ptr=9486829, start=9494518, end=9494528 at test.pl line 13.

(In repeated runs)

I forgot to mention that myperl means blead.

And, in 5.16.0, there is no panic, but there is the bug stated in the
ticket.

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @rjbs

Nicholas has suggested that smoke-me/leave-scope-async-check could quite likely be relevant,
but that it also really needs another set of eyes.

Somebody?

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @iabyn

On Tue, May 07, 2013 at 06​:52​:45PM -0600, Karl Williamson wrote​:

On 05/07/2013 06​:44 PM, Karl Williamson wrote​:

On 05/07/2013 07​:17 AM, Mark Phillips via RT wrote​:

What is the status of this bug?
Any chance of getting it fixed because it is painful having to rebuild
perl every time redhat ships another broken version​:-)

As another data point, I get this​:
§ myperl test.pl
HUP
HUP
panic​: sv_chop ptr=9486829, start=9494518, end=9494528 at test.pl line 13.

(In repeated runs)

I forgot to mention that myperl means blead.

And, in 5.16.0, there is no panic, but there is the bug stated in
the ticket.

Just to confirm what you are saying​: that the OP's bug report code on
your *blead* causes the panic shown above?

If so, that's a new kettle of fish. The original assumption had been
that the bug was fixed in blead, and the discussion was merely about
backporting a modified version of the "fixing" commit to maint-5.14 and
maint-5.16.

FWIW, I can't reproduce any failures on blead.

What are the commit and the build options for your blead?

--
The Enterprise is captured by a vastly superior alien intelligence which
does not put them on trial.
  -- Things That Never Happen in "Star Trek" #10

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @iabyn

On Wed, May 08, 2013 at 04​:59​:40AM -0700, Ricardo SIGNES via RT wrote​:

Nicholas has suggested that smoke-me/leave-scope-async-check could quite likely be relevant,
but that it also really needs another set of eyes.

It looks okay, but I think the better approach is to finish removing regex
global state. I've been working on this piecemeal (struct re_save_state
had 23 fields in 5.10.0, has 17 now); I need to make a more concerted
effort once 5.18 is out.

--
The Enterprise successfully ferries an alien VIP from one place to another
without serious incident.
  -- Things That Never Happen in "Star Trek" #7

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @rjbs

* Dave Mitchell <davem@​iabyn.com> [2013-05-08T09​:21​:48]

Just to confirm what you are saying​: that the OP's bug report code on
your *blead* causes the panic shown above?

Yes.

Karl and I were playing with variants on this problem yesterday evening.

If I run the program included in the ticket and send it HUP a few times, I see​:

  HUP
  HUP
  panic​: sv_chop ptr=7fcb23c011d9, start=7fcb23c05b20, end=7fcb23c05b30 at test line 13.

This is avoided if I drop the s/// from the handler or the loop, and if I add a
sleep to the loop. Dropping the m// does not seem to matter. In other words,
it seems quite likely to me that "if we leave a s/// to enter a handler that
also does a s///, panic ensues."

  This is perl 5, version 18, subversion 0 (v5.18.0 (v5.17.11-110-g9d055c6)) built for darwin-2level

That's why I asked about Nick's fix​: to get this panic address.

Summary of my perl5 (revision 5 version 18 subversion 0) configuration​:
  Commit id​: 9d055c6596c43e1078ddc578f9f2db2e49e8b167
  Platform​:
  osname=darwin, osvers=12.3.0, archname=darwin-2level
  uname='darwin walrus.local 12.3.0 darwin kernel version 12.3.0​: sun jan 6 22​:37​:10 pst 2013; root​:xnu-2050.22.13~1release_x86_64 x86_64 '
  config_args='-de -Dprefix=/Users/rjbs/perl5/perlbrew/perls/18.0-RC0 -Dusedevel -Aeval​:scriptdir=/Users/rjbs/perl5/perlbrew/perls/18.0-RC0/bin'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include',
  optimize='-O3',
  cppflags='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include'
  ccversion='', gccversion='4.2.1 Compatible Apple Clang 4.0 ((tags/Apple/clang-421.0.57))', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector -L/usr/local/lib -L/opt/local/lib'
  libpth=/usr/local/lib /opt/local/lib /usr/lib
  libs=-lgdbm -ldbm -ldl -lm -lutil -lc
  perllibs=-ldl -lm -lutil -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
  PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PERL_SAWAMPERSAND
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO
  USE_PERL_ATOF
  Locally applied patches​:
  RC0
  Built under darwin
  Compiled at May 7 2013 10​:45​:52
  %ENV​:
  PERLBREW_BASHRC_VERSION="0.61"
  PERLBREW_HOME="/Users/rjbs/.perlbrew"
  PERLBREW_MANPATH="/Users/rjbs/perl5/perlbrew/perls/18.0-RC0/man"
  PERLBREW_PATH="/Users/rjbs/perl5/perlbrew/bin​:/Users/rjbs/perl5/perlbrew/perls/18.0-RC0/bin"
  PERLBREW_PERL="18.0-RC0"
  PERLBREW_ROOT="/Users/rjbs/perl5/perlbrew"
  PERLBREW_VERSION="0.61"
  PERLDOC="-n/opt/local/bin/groff"
  PERL_AUTOINSTALL="--skipdeps"
  PERL_MAILERS="sendmail​:/Users/rjbs/bin/sendmail"
  @​INC​:
  /Users/rjbs/perl5/perlbrew/perls/18.0-RC0/lib/site_perl/5.18.0/darwin-2level
  /Users/rjbs/perl5/perlbrew/perls/18.0-RC0/lib/site_perl/5.18.0
  /Users/rjbs/perl5/perlbrew/perls/18.0-RC0/lib/5.18.0/darwin-2level
  /Users/rjbs/perl5/perlbrew/perls/18.0-RC0/lib/5.18.0
  .

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @iabyn

On Wed, May 08, 2013 at 11​:01​:44AM -0400, Ricardo Signes wrote​:

If I run the program included in the ticket and send it HUP a few times, I see​:

HUP
HUP
panic​: sv_chop ptr=7fcb23c011d9, start=7fcb23c05b20, end=7fcb23c05b30 at test line 13.

I still can't reproduce :-(

If Nicholas's PERL_ASYNC_CHECK patch makes it go away, then perhaps there
is a case for applying it to blead now.

--
"I do not resent criticism, even when, for the sake of emphasis,
it parts for the time with reality".
  -- Winston Churchill, House of Commons, 22nd Jan 1941.

@p5pRT
Copy link
Author

p5pRT commented May 8, 2013

From @khwilliamson

On 05/08/2013 09​:50 AM, Dave Mitchell wrote​:

On Wed, May 08, 2013 at 11​:01​:44AM -0400, Ricardo Signes wrote​:

If I run the program included in the ticket and send it HUP a few times, I see​:

HUP
HUP
panic​: sv_chop ptr=7fcb23c011d9, start=7fcb23c05b20, end=7fcb23c05b30 at test line 13.

I still can't reproduce :-(

If Nicholas's PERL_ASYNC_CHECK patch makes it go away, then perhaps there
is a case for applying it to blead now.

I rebased and tried out this patch, and it appears to make the problem
go away. I'm smoking the rebased version now; it hadn't been rebased
since July

@p5pRT
Copy link
Author

p5pRT commented May 9, 2013

From @khwilliamson

On 05/08/2013 10​:58 AM, Karl Williamson wrote​:

On 05/08/2013 09​:50 AM, Dave Mitchell wrote​:

On Wed, May 08, 2013 at 11​:01​:44AM -0400, Ricardo Signes wrote​:

If I run the program included in the ticket and send it HUP a few
times, I see​:

HUP
HUP
panic​: sv_chop ptr=7fcb23c011d9, start=7fcb23c05b20,
end=7fcb23c05b30 at test line 13.

I still can't reproduce :-(

If Nicholas's PERL_ASYNC_CHECK patch makes it go away, then perhaps there
is a case for applying it to blead now.

I rebased and tried out this patch, and it appears to make the problem
go away. I'm smoking the rebased version now; it hadn't been rebased
since July

The smokes came back ok for this patch, so I've applied it to blead as
47c9d59. Looking at the code, it makes
sense that this patch would fix the problem.

I'm not confident enough in my knowledge of portability issues to write
a test case for this, so I'm not closing this bug report. Perhaps
someone else would contribute a test.

In the meantime, tim, perhaps you could compile blead and see if it
causes your problem to go away? Instructions are in perlrepository.

@p5pRT
Copy link
Author

p5pRT commented May 10, 2013

From tim@electronghost.co.uk

On Thursday 09 May 2013 09​:03​:40 Karl Williamson wrote​:

The smokes came back ok for this patch, so I've applied it to blead as
47c9d59. Looking at the code, it makes
sense that this patch would fix the problem.

I'm not confident enough in my knowledge of portability issues to write
a test case for this, so I'm not closing this bug report. Perhaps
someone else would contribute a test.

In the meantime, tim, perhaps you could compile blead and see if it
causes your problem to go away? Instructions are in perlrepository.

I will try to have a go at that next week.

--
Tim Smith <tim@​electronghost.co.uk>
SithLord Lingerie has won contract to supply Stormtrooper underwear in
seventeen sectors. (Some assembly required, Power Cells not included.)

@p5pRT
Copy link
Author

p5pRT commented May 14, 2013

From tim@electronghost.co.uk

On Fri, May 10, 2013 at 06​:47​:11PM +0100, Tim Smith wrote​:

On Thursday 09 May 2013 09​:03​:40 Karl Williamson wrote​:

The smokes came back ok for this patch, so I've applied it to blead as
47c9d59. Looking at the code, it makes
sense that this patch would fix the problem.

I'm not confident enough in my knowledge of portability issues to write
a test case for this, so I'm not closing this bug report. Perhaps
someone else would contribute a test.

In the meantime, tim, perhaps you could compile blead and see if it
causes your problem to go away? Instructions are in perlrepository.

I will try to have a go at that next week.

OK, I built bleed on a spare machine and ran my bug-provoker. No sign of the
problem after 2 hours. Normally I'd expect it to turn up within 10 seconds.

--
Tim Smith <tim@​electronghost.co.uk>
Sector Moff mutters insults behind Lord Vader's back; is demoted to corpse
almost instantly. 'That Vader, he kills us,' say Imperial personnel.

@p5pRT
Copy link
Author

p5pRT commented May 14, 2013

@iabyn - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed May 14, 2013
@p5pRT
Copy link
Author

p5pRT commented Jun 2, 2013

From @iabyn

I've now applied the following merge commit, which provides a more general
solution to the regex re-entrancy issue​:

commit 7d75537
Merge​: 3a74e0e 28d03b2
Author​: David Mitchell <davem@​iabyn.com>
AuthorDate​: Sun Jun 2 20​:59​:58 2013 +0100
Commit​: David Mitchell <davem@​iabyn.com>
CommitDate​: Sun Jun 2 22​:28​:56 2013 +0100

  [MERGE] get rid of (most) regex engine global state
 
  Historically, perl's regex engine was based on Henry Spencer's regex code,
  which was all contained within a single file and used a bunch of static
  variables to maintain the state of the current regex compile or execution.
 
  This was perfectly adequate when only a single thread could execute a
  regex, and where the regex engine couldn't be called re-entrantly.
 
  In 5.0, these vars were promoted to be full global vars as perl became
  embeddable; then in 5.5 they became part of the perl interpreter struct
  when MULTIPLICITY was introduced.
 
  In 5.6, the Perl_save_re_context() function was introduced that did a
  whole bunch of SAVEPPTR(PL_bostr) type stuff, and was called in various
  places where it was possible that the engine may be re-entered, to avoid
  overwriting the global state of the currently executing regex. This was
  particularly important now that Unicode had been introduced, and certain
  character classes could trigger a call to the perl-level SWASH code, which
  could itself execute a regex; and where /(?{ ... })/ code blocks could be
  called which could do likewise.
 
  In 5.10, The various PL_foo variables became fields within the new
  re_save_state struct, and a new interpreter var, PL_reg_state, was
  introduced which was of type struct re_save_state. Thus, all the
  individual vars were still global state, but it became easier to save them
  en-mass in Perl_save_re_context() by just copying the re_save_state stuct
  onto the save stack and marking it with the new SAVEt_RE_STATE type.
  Perl_save_re_context() was also expanded to be responsible for saving all
  the current $1 values.
 
  Up until now, that is roughly how things have remained, except for bug
  fixes such as discovering more places where Perl_save_re_context() needs
  to be called.
 
  Note that, philosophically speaking at least, this is broken in two ways.
  First, there's no good reason for the internal current state of the
  executing regex engine to be stored in a bunch of global vars; and
  secondly we're relying on potential callers of the regex engine (like the
  magic tie code for example), to be responsible for being aware that they
  *might* trigger re-entrancy in the regex engine, and to thus do
  Perl_save_re_context() as a precaution. This is error-prone and hard to
  prove correct. (As an example, Perl_save_re_context() is only called in
  the tie code if the tie code in question is doing a tied PRINT on STDERR;
  clearly an unusual use case that someone spotted was buggy at some point).
 
  The obvious fix, and the one performed by the series of commits in this
  merge, is to make all the global state local to the regex engine instead.
  Indeed, there is already a struct, regmatch_info, that is allocated as a
  local var in regexec(), then passed as an argument to the various
  lower-level functions called from regexec(). However, it only had limited
  use previously, so here we expand the number of functions where it is
  passed as an argument. In particular, it is now also created by
  re_intuit_start(), the other main run-time entry point to the regex
  engine.
 
  However, there is a problem with this, in that various regex vars need
  cleaning up on croak (e.g. they point to a malloced buffer). Since the
  regmatch_info struct is just a local var on the C stack, it will be lost
  by the longjmp done by a croak() before leave_scope() can clear up.
 
  To handle this, some fields that logically should go in regmatch_info,
  are instead added to two new structs​: regmatch_info_aux and
  regmatch_info_aux_eval; the first represents all the normal fields that
  need some sort of cleanup handling; the second represents extra fields
  (also possibly needing cleanup) that are only only needed if the pattern
  contains /(?{})/ code blocks. These two structs are allocated in the next
  two free PL_regmatch_state stack slots - since these slots are allocated
  in 4K slabs anyway, they are essentially free of charge. A single
  destructor function, S_cleanup_regmatch_info_aux() is then used with
  SAVEDESTRUCTOR_X() to perform all cleanup at the end of execution.
 
  In addition, all state and cleanup setup has been consolidated into a
  single point near the start of regexec(); previously it was spread across
  regexec(), regtry() and regmatch(). This used also to result in various
  inefficencies, such as PL_regmatch_state stack freeing all higher unused
  slabs at the end of each call to regmatch(), which might be called
  multiple times by regexec(). Now it just frees once.
 
  As part of this series of fixes it was necessary to change the API of
  Perl_re_intuit_start(). This is because the API was broken​: unlike
  Perl_regexec_flags(), it didn't have a strbeg arg, and would try to guess
  it from the SV (if any) passed to it. This could fail on overloaded SVs
  for example, or where its called without an SV (not done from core, but
  officially supported by the API). Note that this is likely to break
  Re​::Engine plugins, plus any code which directly calls intuit.
 
  Finally, note that although struct re_save_state and SAVEt_RE_STATE are
  gone, Perl_save_re_context() still does something useful​: the equivalent
  of local($1,$2...). Fixing that properly is a whole separate kettle of
  fish, not addressed here.
 
  As far as I'm aware, the only remaining global vars associated with the
  regex engine are
 
  PL_reg_curpm, PL_regmatch_state, PL_regmatch_slab, PL_colors, PL_colorse
 
  None of these are effected by re-entrancy. The state stack is, erm, a stack,
  so it can handle re-entrancy quite happily, and the others are safe too.

--
Overhead, without any fuss, the stars were going out.
  -- Arthur C Clarke

@p5pRT
Copy link
Author

p5pRT commented Jun 3, 2013

From david@kineticode.com

On Jun 2, 2013, at 2​:47 PM, Dave Mitchell <davem@​iabyn.com> wrote​:

I've now applied the following merge commit, which provides a more general
solution to the regex re-entrancy issue​:

This sounds like a great improvement, thanks Dave!

David

@p5pRT
Copy link
Author

p5pRT commented Jun 7, 2013

From @rjbs

* Dave Mitchell <davem@​iabyn.com> [2013-06-02T17​:47​:36]

I've now applied the following merge commit, which provides a more general
solution to the regex re-entrancy issue​:

commit 7d75537

Oho! Thanks, Dave. Exciting change!

--
rjbs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant