Regex qr// interpolation fails when chr(0) is used in a thread #7159

p5pRT · 2004-03-09T03:41:36Z

Migrated from rt.perl.org#27530 (status was 'resolved')

Searchable as RT27530$

p5pRT · 2004-03-09T03:41:37Z

From @jlokier

Created by @jlokier

Try this:

use threads;
$text = chr(0);
$regex = qr/[\Q$text\E]/;
async { /$regex /; } ->join;

The program should produce no output, but it produces this error message:

thread failed to start: Unmatched [ in regex; marked by <-- HERE in m/(?-xism:[
<-- HERE \/ at -e line 4.

This program is fine:

use threads;
$text = chr(0);
async { /[\Q$text\E]/; } ->join;

This too:

use threads;
$text = chr(0);
$regex = q/[\Q$text\E]/;
async { /$regex /; } ->join;

And this:

use threads;
$text = chr(0);
$regex = qr/[\Q$text\E]/;
async { /$regex/; } ->join;

Somehow, there is an interaction between threads and regex compilation
that is broken when a string containing a zero character is
interpolated from a qr// result into a new regex.

It's not as obscure as it sounds. The interpolation operator \Q...\E
is supposed to work wth _any_ characters - indeed, it's often used
with hostile external input or with binary data.

(In larger programs, from which this is a whittled test case, it's
been a real nuisance).

Thanks,
-- Jamie

Perl Info


Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.0:

Configured by bhcompile'
cf_email='bhcompile at Wed Aug 13 11:45:59 EDT 2003.

Summary of my rderl (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.21-1.1931.2.382.entsmp, archname=i386-linux-thread-multi
    uname='linux str'
    config_args='-des -Doptimize=-O2 -g -pipe -march=i386 -mcpu=i686 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5.8.0 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef'
 useithreads=define usemultiplicity=
    useperlio= d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=un uselongdouble=
    usemymalloc=, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.2.2 20030222 (Red Hat Linux 3.2.2-5)', gccosandvers=''
gccversion='3.2.2 200302'
    intsize=r, longsize=r, ptrsize=5, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long'
k', ivsize=4'
ivtype='l, nvtype='double'
o_nonbl', nvsize=, Off_t='', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc'
l', ldflags =' -L/u'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libper
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so', d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC'
ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5', lddlflags='s Unicode/Normalize XS/A'

Locally applied patches:
    MAINT18379


@INC for perl v5.8.0:
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    .


Environment for perl v5.8.0:
    HOME=/home/jamie
    LANG=en_GB.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/jamie/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash
    dlflags='-share (unset)

p5pRT · 2004-03-11T16:23:51Z

From @hvds

Jamie Lokier (via RT) <perlbug-followup@perl.org> wrote:
:Try this:
:
: use threads;
: $text = chr(0);
: $regex = qr/[\Q$text\E]/;
: async { /$regex /; } ->join;
:
:The program should produce no output, but it produces this error message:
:
:thread failed to start: Unmatched [ in regex; marked by <-- HERE in m/(?-xism:[
:<-- HERE \/ at -e line 4.

Thanks for the report. If I understand correctly, this occurs only in this
situation because the qr{} object gets copied from the source thread
memory space to that of the target thread, and it is during the copy that
the error occurs.

The patch below (to bleadperl, but also suitable for maint) should fix it.
I'm not sure where we put threads-specific tests for things like this,
but the example above is pretty close to an ideal test case.

Hugo

Inline Patch

--- sv.c.old	Wed Mar 10 20:59:52 2004
+++ sv.c	Thu Mar 11 16:16:48 2004
@@ -9715,7 +9715,7 @@
     New(0, ret->offsets, 2*len+1, U32);
     Copy(r->offsets, ret->offsets, 2*len+1, U32);
 
-    ret->precomp        = SAVEPV(r->precomp);
+    ret->precomp        = SAVEPVN(r->precomp, r->prelen);
     ret->refcnt         = r->refcnt;
     ret->minlen         = r->minlen;
     ret->prelen         = r->prelen;

p5pRT · 2004-03-11T16:23:54Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2004-03-11T23:28:45Z

From @rgs

hv@crypt.org wrote:

The patch below (to bleadperl, but also suitable for maint) should fix it.
I'm not sure where we put threads-specific tests for things like this,
but the example above is pretty close to an ideal test case.

Regression tests for cloning... perhaps in t/op/clone ?

--- sv.c.old Wed Mar 10 20:59:52 2004
+++ sv.c Thu Mar 11 16:16:48 2004
@@ -9715,7 +9715,7 @@
New(0, ret->offsets, 2*len+1, U32);
Copy(r->offsets, ret->offsets, 2*len+1, U32);

- ret->precomp = SAVEPV(r->precomp);
+ ret->precomp = SAVEPVN(r->precomp, r->prelen);

While we're at it, maybe this one is more complete :
(cargo-culting your fix)

==== //depot/perl/sv.c#726 - /opt/perl/p4/perl/sv.c ====

Inline Patch

--- /tmp/tmp.1393.0     Fri Mar 12 00:25:09 2004
+++ /opt/perl/p4/perl/sv.c      Fri Mar 12 00:25:02 2004
@@ -9715,7 +9715,7 @@
     New(0, ret->offsets, 2*len+1, U32);
     Copy(r->offsets, ret->offsets, 2*len+1, U32);
 
-    ret->precomp        = SAVEPV(r->precomp);
+    ret->precomp        = SAVEPVN(r->precomp, r->prelen);
     ret->refcnt         = r->refcnt;
     ret->minlen         = r->minlen;
     ret->prelen         = r->prelen;
@@ -9727,7 +9727,7 @@
     ret->sublen         = r->sublen;
 
     if (RX_MATCH_COPIED(ret))
-       ret->subbeg  = SAVEPV(r->subbeg);
+       ret->subbeg  = SAVEPVN(r->subbeg, r->sublen);
     else
        ret->subbeg = Nullch;
 #ifdef PERL_COPY_ON_WRITE

p5pRT · 2004-03-11T23:39:52Z

From @iabyn

On Fri, Mar 12, 2004 at 12:26:29AM +0100, Rafael Garcia-Suarez wrote:

hv@crypt.org wrote:

The patch below (to bleadperl, but also suitable for maint) should fix it.
I'm not sure where we put threads-specific tests for things like this,
but the example above is pretty close to an ideal test case.

Regression tests for cloning... perhaps in t/op/clone ?

well, there's always ext/threads/t/problems.t for random threads-related
problems.

--
Little fly, thy summer's play my thoughtless hand
has terminated with extreme prejudice.
(with apologies to William Blake)

p5pRT · 2004-03-12T03:42:19Z

From @hvds

Rafael Garcia-Suarez <rgarciasuarez@free.fr> wrote:
:While we're at it, maybe this one is more complete :
:(cargo-culting your fix)
[...]
:- ret->subbeg = SAVEPV(r->subbeg);
:+ ret->subbeg = SAVEPVN(r->subbeg, r->sublen);

Yes, that one is also needed.

Hugo

p5pRT · 2004-11-18T15:53:45Z

From @smpeters

[rafael - Thu Mar 11 15:28:45 2004]:

hv@crypt.org wrote:

The patch below (to bleadperl, but also suitable for maint) should
fix it.
I'm not sure where we put threads-specific tests for things like this,
but the example above is pretty close to an ideal test case.

Regression tests for cloning... perhaps in t/op/clone ?

--- sv.c.old Wed Mar 10 20:59:52 2004
+++ sv.c Thu Mar 11 16:16:48 2004
@@ -9715,7 +9715,7 @@
New(0, ret->offsets, 2*len+1, U32);
Copy(r->offsets, ret->offsets, 2*len+1, U32);

- ret->precomp = SAVEPV(r->precomp);
+ ret->precomp = SAVEPVN(r->precomp, r->prelen);

While we're at it, maybe this one is more complete :
(cargo-culting your fix)

==== //depot/perl/sv.c#726 - /opt/perl/p4/perl/sv.c ====
--- /tmp/tmp.1393.0 Fri Mar 12 00:25:09 2004
+++ /opt/perl/p4/perl/sv.c Fri Mar 12 00:25:02 2004
@@ -9715,7 +9715,7 @@
New(0, ret->offsets, 2*len+1, U32);
Copy(r->offsets, ret->offsets, 2*len+1, U32);

- ret->precomp = SAVEPV(r->precomp);
+ ret->precomp = SAVEPVN(r->precomp, r->prelen);
ret->refcnt = r->refcnt;
ret->minlen = r->minlen;
ret->prelen = r->prelen;
@@ -9727,7 +9727,7 @@
ret->sublen = r->sublen;
 if $RX\_MATCH\_COPIED\(ret$\)
- ret->subbeg = SAVEPV(r->subbeg);
+ ret->subbeg = SAVEPVN(r->subbeg, r->sublen);
else
ret->subbeg = Nullch;
#ifdef PERL_COPY_ON_WRITE

This fix was applied as change #22490.

p5pRT · 2004-11-18T15:53:47Z

@smpeters - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Nov 18, 2004

p5pRT added Severity High distro-Linux type-core labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regex qr// interpolation fails when chr(0) is used in a thread #7159

Regex qr// interpolation fails when chr(0) is used in a thread #7159

p5pRT commented Mar 9, 2004

p5pRT commented Mar 9, 2004

p5pRT commented Mar 11, 2004

p5pRT commented Mar 11, 2004

p5pRT commented Mar 11, 2004

p5pRT commented Mar 11, 2004

p5pRT commented Mar 12, 2004

p5pRT commented Nov 18, 2004

p5pRT commented Nov 18, 2004

Regex qr// interpolation fails when chr(0) is used in a thread #7159

Regex qr// interpolation fails when chr(0) is used in a thread #7159

Comments

p5pRT commented Mar 9, 2004

p5pRT commented Mar 9, 2004

From @jlokier

Created by @jlokier

p5pRT commented Mar 11, 2004

From @hvds

p5pRT commented Mar 11, 2004

p5pRT commented Mar 11, 2004

From @rgs

p5pRT commented Mar 11, 2004

From @iabyn

p5pRT commented Mar 12, 2004

From @hvds

p5pRT commented Nov 18, 2004

From @smpeters

p5pRT commented Nov 18, 2004