Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possibly memory leak when matching strings using pre-compiled regexes stored in hash key (perl >= v5.26) #16427

Closed
p5pRT opened this issue Feb 21, 2018 · 25 comments

Comments

@p5pRT
Copy link

p5pRT commented Feb 21, 2018

Migrated from rt.perl.org#132892 (status was 'open')

Searchable as RT132892$

@p5pRT
Copy link
Author

p5pRT commented Feb 21, 2018

From @rootkwok

Hello guys,

I have a long running perl script which allows the user configure the
program behavior with a hash that got pre-compiled regexps
as keys and the corresponding callback code as the value like below.
Recently I have upgraded to perl v5.26 and the memory
usage grows since then, so I have fall back to perl v5.24.3 as a temporary
solution. What I observed before fallback is that the
memory grows when I try to iterates the hash keys and matches them against
some text.

%action = (

  qr/^regex here/ => sub { print "match" },

);

I have written a small test for this problem, you can find it in the
attached perlbug.tar.xz. 20180221-mem-leak-poc.pl is the prof of
concept code. It will output the platform and perl version as the first
line, and make use of the ps command to retrieve the vsz and
rss of the perl interpreter before each iteration. The test result of
different platform are stored in the rslt/ sub-directory for your reference.
According to the results, with perl <v5.26, the size in memory remain
unchanged; While the memory usage keeps increasing when the
script is run with perl >=v5.26. I have limited the number of iteration,
the memory usage grows further if you increases the number of iteration.

|-- 20180221-mem-leak-poc.pl
`-- rslt
  |-- centos6-p5.10.1.log
  |-- centos6-p5.24.3.log
  |-- centos6-p5.26.1.log
  |-- centos6-p5.27.8.log
  |-- dfbsd-p5.24.3.log
  |-- dfbsd-p5.26.1.log
  |-- freebsd-p5.24.3.log
  `-- freebsd-p5.26.1.log

According to my empirical result, this problem occurs when using perl newer
than v5.26, the first line of perl -v of the affected perl
interpreters below.

This is perl 5, version 26, subversion 1 (v5.26.1) built for amd64-freebsd
This is perl 5, version 27, subversion 8 (v5.27.8) built for x86_64-linux
This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux

This is perl 5, version 26, subversion 1 (v5.26.1) built for

x86_64-dragonfly

I have traced the problem a little bit and I *guess* the problem is in the
function S_concat_pat of regcomp.c and the new SVs
copied out are immortal, so every time my program iterate through the keys
of hash to match against things, it creates more
copies of the regexes. I can try to dig deeper when I am free to build the
debug versions of perl, but that would be great if this
case can be taken care by the professionals here. Thanks a lot.
�

�
Best regards,
Baggio

@p5pRT
Copy link
Author

p5pRT commented Feb 21, 2018

From @rootkwok

code.png

@p5pRT
Copy link
Author

p5pRT commented Feb 21, 2018

From @rootkwok

perlbug.tar.xz

@p5pRT
Copy link
Author

p5pRT commented Feb 22, 2018

From @shlomif

On Wed, 21 Feb 2018 07​:49​:20 -0800
(via RT) <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by
# Please include the string​: [perl #132892]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132892 >

Hi all!

I reworked the example code to remove some potential problems, and verified
that htop sees its RAM consumption increasing​:

#!/usr/bin/perl

use strict;
use warnings;

# my $getmem = "ps -o vsz= -o rss= -p $$",

print "== Test result on $^O perl-$^V\n";

my %h = (
  qr{abcd} => [ 'ABCD', 2],
  qr{efgh} => [ 'EFGH', 1],
  qr{ijkl} => [ 'IJKL', 1],
  qr{mnop} => [ 'MNOP', 1],
  qr{qrst} => [ 'QRST', 1],
  qr{uvwx} => [ 'UVWX', 1],
);

my @​list = qw/ abcd abcd efgh ijkl mnop qrst uvwx /;

while (1) {
  # system($getmem);
  for my $r (keys %h) {
  my $c = 0;
  my ($name, $expected) = @​{$h{$r}};
  for my $e ( @​list ) {
  $c++ if $e =~ $r;
  }

  print ">>> Count mismatch for $name\n" if ( $c != $expected );
  }
}

Hello guys,

I have a long running perl script which allows the user configure the
program behavior with a hash that got pre-compiled regexps
as keys and the corresponding callback code as the value like below.
Recently I have upgraded to perl v5.26 and the memory
usage grows since then, so I have fall back to perl v5.24.3 as a temporary
solution. What I observed before fallback is that the
memory grows when I try to iterates the hash keys and matches them against
some text.

%action = (

qr/^regex here/ => sub { print "match" },

);

I have written a small test for this problem, you can find it in the
attached perlbug.tar.xz. 20180221-mem-leak-poc.pl is the prof of
concept code. It will output the platform and perl version as the first
line, and make use of the ps command to retrieve the vsz and
rss of the perl interpreter before each iteration. The test result of
different platform are stored in the rslt/ sub-directory for your reference.
According to the results, with perl <v5.26, the size in memory remain
unchanged; While the memory usage keeps increasing when the
script is run with perl >=v5.26. I have limited the number of iteration,
the memory usage grows further if you increases the number of iteration.

|-- 20180221-mem-leak-poc.pl
`-- rslt
|-- centos6-p5.10.1.log
|-- centos6-p5.24.3.log
|-- centos6-p5.26.1.log
|-- centos6-p5.27.8.log
|-- dfbsd-p5.24.3.log
|-- dfbsd-p5.26.1.log
|-- freebsd-p5.24.3.log
`-- freebsd-p5.26.1.log

According to my empirical result, this problem occurs when using perl newer
than v5.26, the first line of perl -v of the affected perl
interpreters below.

This is perl 5, version 26, subversion 1 (v5.26.1) built for amd64-freebsd
This is perl 5, version 27, subversion 8 (v5.27.8) built for x86_64-linux
This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux

This is perl 5, version 26, subversion 1 (v5.26.1) built for

x86_64-dragonfly

I have traced the problem a little bit and I *guess* the problem is in the
function S_concat_pat of regcomp.c and the new SVs
copied out are immortal, so every time my program iterate through the keys
of hash to match against things, it creates more
copies of the regexes. I can try to dig deeper when I am free to build the
debug versions of perl, but that would be great if this
case can be taken care by the professionals here. Thanks a lot.
�

�
Best regards,
Baggio

@p5pRT
Copy link
Author

p5pRT commented Feb 22, 2018

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Feb 22, 2018

From @dur-randir

Bisect points to commit b10cb25
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Sat Sep 17 20​:14​:53 2016 +0200

  regcomp.c​: S_concat_pat​: guard against missing trailing nulls

  The regex engine expects the pattern to have a null byte at
  SvEND(pat), but is not guaranteed to receive such a pattern
  when it is called, so S_concat_pat should guard against this
  case. It turns out this is only an issue when there is exactly
  one "argument" to the pattern. (Consider concatenation rules, etc).

@p5pRT
Copy link
Author

p5pRT commented Feb 22, 2018

From @demerphq

On 22 Feb 2018 12​:39, "via RT" <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by
# Please include the string​: [perl #132892]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132892 >

Hello guys,

I have a long running perl script which allows the user configure the
program behavior with a hash that got pre-compiled regexps
as keys and the corresponding callback code as the value like below.
Recently I have upgraded to perl v5.26 and the memory
usage grows since then, so I have fall back to perl v5.24.3 as a temporary
solution. What I observed before fallback is that the
memory grows when I try to iterates the hash keys and matches them against
some text.

%action = (

  qr/^regex here/ => sub { print "match" },

);

Regardless if the bug, perl hash keys are strings not objects. Perl will
compile these patterns, then stringify them to store them as keys, and then
when used later will recompile again. You need a better data structure.

Yves

I have written a small test for this problem, you can find it in the
attached perlbug.tar.xz. 20180221-mem-leak-poc.pl is the prof of
concept code. It will output the platform and perl version as the first
line, and make use of the ps command to retrieve the vsz and
rss of the perl interpreter before each iteration. The test result of
different platform are stored in the rslt/ sub-directory for your reference.
According to the results, with perl <v5.26, the size in memory remain
unchanged; While the memory usage keeps increasing when the
script is run with perl >=v5.26. I have limited the number of iteration,
the memory usage grows further if you increases the number of iteration.

|-- 20180221-mem-leak-poc.pl
`-- rslt
  |-- centos6-p5.10.1.log
  |-- centos6-p5.24.3.log
  |-- centos6-p5.26.1.log
  |-- centos6-p5.27.8.log
  |-- dfbsd-p5.24.3.log
  |-- dfbsd-p5.26.1.log
  |-- freebsd-p5.24.3.log
  `-- freebsd-p5.26.1.log

According to my empirical result, this problem occurs when using perl newer
than v5.26, the first line of perl -v of the affected perl
interpreters below.

This is perl 5, version 26, subversion 1 (v5.26.1) built for amd64-freebsd
This is perl 5, version 27, subversion 8 (v5.27.8) built for x86_64-linux
This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux

This is perl 5, version 26, subversion 1 (v5.26.1) built for

x86_64-dragonfly

I have traced the problem a little bit and I *guess* the problem is in the
function S_concat_pat of regcomp.c and the new SVs
copied out are immortal, so every time my program iterate through the keys
of hash to match against things, it creates more
copies of the regexes. I can try to dig deeper when I am free to build the
debug versions of perl, but that would be great if this
case can be taken care by the professionals here. Thanks a lot.
�

�
Best regards,
Baggio

@p5pRT
Copy link
Author

p5pRT commented Feb 23, 2018

From @demerphq

On 22 February 2018 at 12​:29, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

Bisect points to commit b10cb25
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Sat Sep 17 20​:14​:53 2016 +0200

regcomp\.c&#8203;: S\_concat\_pat&#8203;: guard against missing trailing nulls

The regex engine expects the pattern to have a null byte at
SvEND\(pat\)\, but is not guaranteed to receive such a pattern
when it is called\, so S\_concat\_pat should guard against this
case\. It turns out this is only an issue when there is exactly
one "argument" to the pattern\. \(Consider concatenation rules\, etc\)\.

Thanks, I forgot an sv_2mortal in that patch, I will push a fix soon.

Much obliged for the bisect!

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 23, 2018

From @demerphq

On 23 February 2018 at 04​:27, demerphq <demerphq@​gmail.com> wrote​:

On 22 February 2018 at 12​:29, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

Bisect points to commit b10cb25
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Sat Sep 17 20​:14​:53 2016 +0200

regcomp\.c&#8203;: S\_concat\_pat&#8203;: guard against missing trailing nulls

The regex engine expects the pattern to have a null byte at
SvEND\(pat\)\, but is not guaranteed to receive such a pattern
when it is called\, so S\_concat\_pat should guard against this
case\. It turns out this is only an issue when there is exactly
one "argument" to the pattern\. \(Consider concatenation rules\, etc\)\.

Thanks, I forgot an sv_2mortal in that patch, I will push a fix soon.

Much obliged for the bisect!

Fixed in 910a6a8

Unfortunately I don't know how to test for a memory leak, so no tests included.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 23, 2018

From @iabyn

On Fri, Feb 23, 2018 at 10​:34​:50AM +0100, demerphq wrote​:

Fixed in 910a6a8

Unfortunately I don't know how to test for a memory leak, so no tests included.

see t/op/svleak.t. It runs a bit of code several times in a loop and
tests whether PL_sv_count fails to remain constant.

--
Little fly, thy summer's play my thoughtless hand
has terminated with extreme prejudice.
  (with apologies to William Blake)

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 23 February 2018 at 14​:03, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, Feb 23, 2018 at 10​:34​:50AM +0100, demerphq wrote​:

Fixed in 910a6a8

Unfortunately I don't know how to test for a memory leak, so no tests included.

see t/op/svleak.t. It runs a bit of code several times in a loop and
tests whether PL_sv_count fails to remain constant.

Thanks, i just pushed 06f26dc which
does just that.

FWIW, there is still an open question why this happens at all. Keys
should be null terminated, and we should not fall into this code-path.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 25 February 2018 at 10​:50, demerphq <demerphq@​gmail.com> wrote​:

On 23 February 2018 at 14​:03, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, Feb 23, 2018 at 10​:34​:50AM +0100, demerphq wrote​:

Fixed in 910a6a8

Unfortunately I don't know how to test for a memory leak, so no tests included.

see t/op/svleak.t. It runs a bit of code several times in a loop and
tests whether PL_sv_count fails to remain constant.

Thanks, i just pushed 06f26dc which
does just that.

FWIW, there is still an open question why this happens at all. Keys
should be null terminated, and we should not fall into this code-path.

Apparently I am wrong​:

$ ./perl -Ilib -MDevel​::Peek -le'%h=("^foo"=>1); my ($re)=keys %h; Dump $re'
SV = PV(0x1009d08) at 0x103f0b8
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x1041478 "^foo"
  CUR = 4
  LEN = 0

This appears to be a side-effect of shared keys.

Another good reason not to put patterns in keys. :-(

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @dur-randir

On Fri, 23 Feb 2018 01​:35​:02 -0800, demerphq wrote​:

Fixed in 910a6a8

Since this is a regression in 5.26, should it also be added to voting list for the future 5.26.2? Or there're no plans for it?

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 25 February 2018 at 11​:00, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

On Fri, 23 Feb 2018 01​:35​:02 -0800, demerphq wrote​:

Fixed in 910a6a8

Since this is a regression in 5.26, should it also be added to voting list for the future 5.26.2? Or there're no plans for it?

Sure. I don't remember how to do that tho. Where does the voting list
live again?

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 25 February 2018 at 10​:57, demerphq <demerphq@​gmail.com> wrote​:

On 25 February 2018 at 10​:50, demerphq <demerphq@​gmail.com> wrote​:

On 23 February 2018 at 14​:03, Dave Mitchell <davem@​iabyn.com> wrote​:

On Fri, Feb 23, 2018 at 10​:34​:50AM +0100, demerphq wrote​:

Fixed in 910a6a8

Unfortunately I don't know how to test for a memory leak, so no tests included.

see t/op/svleak.t. It runs a bit of code several times in a loop and
tests whether PL_sv_count fails to remain constant.

Thanks, i just pushed 06f26dc which
does just that.

FWIW, there is still an open question why this happens at all. Keys
should be null terminated, and we should not fall into this code-path.

Apparently I am wrong​:

$ ./perl -Ilib -MDevel​::Peek -le'%h=("^foo"=>1); my ($re)=keys %h; Dump $re'
SV = PV(0x1009d08) at 0x103f0b8
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x1041478 "^foo"
CUR = 4
LEN = 0

This appears to be a side-effect of shared keys.

Another good reason not to put patterns in keys. :-(

Corroborating this I hacked a way to get an unshared hash, and then
checked what things look like, and the keys are not shared as
expected, and are null terminated.

$ ./perl -Ilib -MHash​::Util=new_unshared_hash -MDevel​::Peek -e'my
$hv=new_unshared_hash(); $hv->{foo}=1; Dump((keys %$hv)[0])'
SV = PV(0x2670c58) at 0x2699680
  REFCNT = 1
  FLAGS = (TEMP,POK,pPOK)
  PV = 0x269f9e8 "foo"\0
  CUR = 3
  LEN = 10

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @dur-randir

On Sun, 25 Feb 2018 01​:58​:15 -0800, demerphq wrote​:

$ ./perl -Ilib -MDevel​::Peek -le'%h=("^foo"=>1); my ($re)=keys %h;
Dump $re'
SV = PV(0x1009d08) at 0x103f0b8
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x1041478 "^foo"
CUR = 4
LEN = 0

This appears to be a side-effect of shared keys.

But the failed check in regcomp.c is not *(SvEND(msv)) == 0, but rather SvLEN(msv) > SvCUR(msv). But doesn't line 3134 in hv.c (HEK_KEY(hek)[len] = 0;) guarantee that it's implicitly null-terminated, just not shown by Devel​::Peek as such?

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @dur-randir

On Sun, 25 Feb 2018 02​:13​:12 -0800, demerphq wrote​:

Sure. I don't remember how to do that tho. Where does the voting list
live again?

Looks like votes-5.26.xml in a maint-votes branch.

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 25 February 2018 at 11​:19, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

On Sun, 25 Feb 2018 01​:58​:15 -0800, demerphq wrote​:

$ ./perl -Ilib -MDevel​::Peek -le'%h=("^foo"=>1); my ($re)=keys %h;
Dump $re'
SV = PV(0x1009d08) at 0x103f0b8
REFCNT = 1
FLAGS = (POK,IsCOW,pPOK)
PV = 0x1041478 "^foo"
CUR = 4
LEN = 0

This appears to be a side-effect of shared keys.

But the failed check in regcomp.c is not *(SvEND(msv)) == 0, but rather SvLEN(msv) > SvCUR(msv). But doesn't line 3134 in hv.c (HEK_KEY(hek)[len] = 0;) guarantee that it's implicitly null-terminated, just not shown by Devel​::Peek as such?

That sounds correct, but i dont think the regex engine has any way to
know that this LEN=0 case comes from a key. Since we don't know the
len we cant safely check past CUR.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @dur-randir

On Sun, 25 Feb 2018 02​:57​:09 -0800, demerphq wrote​:

That sounds correct, but i dont think the regex engine has any way to
know that this LEN=0 case comes from a key. Since we don't know the
len we cant safely check past CUR.

But you can check for SvIsCOW_shared_hash, which guarantees it to be from sharepvn.

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @xsawyerx

On 02/25/2018 12​:13 PM, demerphq wrote​:

On 25 February 2018 at 11​:00, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

On Fri, 23 Feb 2018 01​:35​:02 -0800, demerphq wrote​:

Fixed in 910a6a8
Since this is a regression in 5.26, should it also be added to voting list for the future 5.26.2? Or there're no plans for it?

It should be, yes!

Sure. I don't remember how to do that tho. Where does the voting list
live again?

The branch "maint-votes" in git. I've added them here​:

https://perl5.git.perl.org/perl.git/commitdiff/5f21d69b722f6a73f185e533ecf138e17adeabce

With a vote from both of us, Yves.

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 25 February 2018 at 16​:21, Sawyer X <xsawyerx@​gmail.com> wrote​:

On 02/25/2018 12​:13 PM, demerphq wrote​:

On 25 February 2018 at 11​:00, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

On Fri, 23 Feb 2018 01​:35​:02 -0800, demerphq wrote​:

Fixed in 910a6a8
Since this is a regression in 5.26, should it also be added to voting list for the future 5.26.2? Or there're no plans for it?

It should be, yes!

Sure. I don't remember how to do that tho. Where does the voting list
live again?

The branch "maint-votes" in git. I've added them here​:

https://perl5.git.perl.org/perl.git/commitdiff/5f21d69b722f6a73f185e533ecf138e17adeabce

With a vote from both of us, Yves.

Thanks, I was busy trying to test Sergey Aleynikov's suggestion of
using SvIsCOW_shared_hash when my perl build started throwing some
weird ass errors, so assuming that the reboot and clean build i just
did makes the weirdness go away, there might be one more commit
required. For something like this is it preferred to backport the full
patch sequence, or is it preferable to prepare a squashed version?

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @demerphq

On 25 February 2018 at 12​:03, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

On Sun, 25 Feb 2018 02​:57​:09 -0800, demerphq wrote​:

That sounds correct, but i dont think the regex engine has any way to
know that this LEN=0 case comes from a key. Since we don't know the
len we cant safely check past CUR.

But you can check for SvIsCOW_shared_hash, which guarantees it to be from sharepvn.

Thanks again for your help, i have pushed a patch to implement this suggestion​:

f1d945b

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Feb 25, 2018

From @xsawyerx

On 02/25/2018 05​:26 PM, demerphq wrote​:

On 25 February 2018 at 16​:21, Sawyer X <xsawyerx@​gmail.com> wrote​:

On 02/25/2018 12​:13 PM, demerphq wrote​:

On 25 February 2018 at 11​:00, Sergey Aleynikov via RT
<perlbug-followup@​perl.org> wrote​:

On Fri, 23 Feb 2018 01​:35​:02 -0800, demerphq wrote​:

Fixed in 910a6a8
Since this is a regression in 5.26, should it also be added to voting list for the future 5.26.2? Or there're no plans for it?
It should be, yes!

Sure. I don't remember how to do that tho. Where does the voting list
live again?
The branch "maint-votes" in git. I've added them here​:

https://perl5.git.perl.org/perl.git/commitdiff/5f21d69b722f6a73f185e533ecf138e17adeabce

With a vote from both of us, Yves.
Thanks, I was busy trying to test Sergey Aleynikov's suggestion of
using SvIsCOW_shared_hash when my perl build started throwing some
weird ass errors, so assuming that the reboot and clean build i just
did makes the weirdness go away, there might be one more commit
required. For something like this is it preferred to backport the full
patch sequence, or is it preferable to prepare a squashed version?

I think separate patches are better. It will align with blead.

@khwilliamson
Copy link
Contributor

What's the status of this bug?

@demerphq
Copy link
Collaborator

Looks to me like it can be closed. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants