Bleadperl v5.19.3-361-gde935cc breaks YAPPO/Data-Model-0.00008.tar.gz #13254

p5pRT · 2013-09-13T07:59:28Z

Migrated from rt.perl.org#119771 (status was 'rejected')

Searchable as RT119771$

p5pRT · 2013-09-13T07:59:28Z

From @andk

git bisect

commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700

Allow 64-bit array and stack offsets in entersub & goto

diagnostics

http://www.cpantesters.org/cpan/report/039a5aec-1c0b-11e3-b738-32d47e66b2dc

Failing reports come only from non-threaded perls. Threaded perls pass
all tests. The crucial test t/060_driver/memcached/serializer.t can be
skipped for various reasons, so make sure you have at least
Cache::Memcached::Fast and Data::MessagePack installed. Maybe there are
other optional dependencies, check for skip messages.

perl -V

Summary of my perl5 (revision 5 version 19 subversion 4) configuration:
Commit id: de935cc
Platform:
osname=linux, osvers=3.10-2-amd64, archname=x86_64-linux
uname='linux k83 3.10-2-amd64 #1 smp debian 3.10.7-1 (2013-08-17) x86_64 gnulinux '
config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.3-361-gde935cc/165a -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Uuseithreads -Uuselongdouble -DDEBUGGING=-g'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
ccversion='', gccversion='4.8.1', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
libc=, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.17'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO
USE_PERL_ATOF
Built under linux
Compiled at Sep 13 2013 08:39:52
@INC:
/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.3-361-gde935cc/165a/lib/site_perl/5.19.4/x86_64-linux
/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.3-361-gde935cc/165a/lib/site_perl/5.19.4
/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.3-361-gde935cc/165a/lib/5.19.4/x86_64-linux
/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.3-361-gde935cc/165a/lib/5.19.4
.

--
andreas

p5pRT · 2013-09-14T12:36:36Z

From @jkeenan

On Fri Sep 13 00:59:28 2013, andreas.koenig.7os6VVqR@franz.ak.mind.de wrote:

git bisect
----------
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700
Allow 64\-bit array and stack offsets in entersub & goto

The full commit message from the above was:

##########
$ git show de935cc
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700

Allow 64-bit array and stack offsets in entersub & goto

I don’t have enough memory to test this, but it needs to be done
eventually anyway.

#########

Shouldn't a plan have been developed to have *somebody* with enough
memory test this before it was committed to blead?

And where was its need "to be done eventually anyway" discussed?

Thank you very much.
Jim Keenan

diagnostics
-----------
http://www.cpantesters.org/cpan/report/039a5aec-1c0b-11e3-b738-
32d47e66b2dc

Failing reports come only from non-threaded perls. Threaded perls pass
all tests. The crucial test t/060_driver/memcached/serializer.t can be
skipped for various reasons, so make sure you have at least
Cache::Memcached::Fast and Data::MessagePack installed. Maybe there
are
other optional dependencies, check for skip messages.

p5pRT · 2013-09-14T12:36:37Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2013-09-14T14:12:57Z

From @cpansprout

On Sat Sep 14 05:36:36 2013, jkeenan wrote:

On Fri Sep 13 00:59:28 2013, andreas.koenig.7os6VVqR@franz.ak.mind.de
wrote:
git bisect
----------
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700
Allow 64\-bit array and stack offsets in entersub & goto
The full commit message from the above was:

##########
$ git show de935cc
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700
Allow 64\-bit array and stack offsets in entersub & goto

I don’t have enough memory to test this\, but it needs to be done
eventually anyway.

#########

Shouldn't a plan have been developed to have *somebody* with enough
memory test this before it was committed to blead?

This would be at least 30GB, possibly more, in this case. Most cases of
this class of bug would need hundreds of gigabytes.

And where was its need "to be done eventually anyway" discussed?

Ticket #72784.

If I’m working on an area of perl and I see that large integers are
going to wrap because the wrong internal types are going to be used,
then I’ll correct them. This is an incremental task. Every bit helps.

I haven’t looked into this failure yet, but I suspect this is a false
positive, or it’s uncovering some other bug.

--

Father Chrysostomos

p5pRT · 2013-09-14T21:34:44Z

From @tsee

On 09/14/2013 04:12 PM, Father Chrysostomos via RT wrote:

This would be at least 30GB, possibly more, in this case. Most cases of
this class of bug would need hundreds of gigabytes.

For the record, if anybody finds a case that really, really requires
testing on a machine with lots of RAM, then I can do that.

It doesn't come for free in terms of my time. I'd have to rig an
otherwise more bare-bones machine with all sorts of build tools, and
then obtain build the right perl. On top of that, there might be a delay
of days or even weeks. But I should be able to locate a machine with at
least 96GB, more likely 192GB if need be.

--Steffen

p5pRT · 2013-09-14T22:13:37Z

From victor@vsespb.ru

FYI there is Amazon EC2 hosting with 30Gb instances for ~ $1/hour (+ some
possible EBS storage + network traffic cost). (and, say 244Gb for $3.5/hour)

2013/9/15 Steffen Mueller <smueller@cpan.org>

For the record, if anybody finds a case that really, really requires
testing on a machine with lots of RAM, then I can do that.

It doesn't come for free in terms of my time. I'd have to rig an otherwise
more bare-bones machine with all sorts of build tools, and then obtain
build the right perl. On top of that, there might be a delay of days or
even weeks. But I should be able to locate a machine with at least 96GB,
more likely 192GB if need be.

--Steffen

p5pRT · 2013-09-15T11:24:42Z

From @tonycoz

On Sat, Sep 14, 2013 at 11:34:01PM +0200, Steffen Mueller wrote:

On 09/14/2013 04:12 PM, Father Chrysostomos via RT wrote:

This would be at least 30GB, possibly more, in this case. Most cases of
this class of bug would need hundreds of gigabytes.

For the record, if anybody finds a case that really, really requires
testing on a machine with lots of RAM, then I can do that.

It doesn't come for free in terms of my time. I'd have to rig an
otherwise more bare-bones machine with all sorts of build tools, and
then obtain build the right perl. On top of that, there might be a
delay of days or even weeks. But I should be able to locate a
machine with at least 96GB, more likely 192GB if need be.

The new dromedary has 96GB:

[tonyc@dromedary-001 ~]$ free
total used free shared buffers cached
Mem: 99050664 29417336 69633328 0 1336928 21840484
-/+ buffers/cache: 6239924 92810740
Swap: 2097144 19224 2077920

Tony

p5pRT · 2013-09-15T19:37:24Z

From @cpansprout

On Sun Sep 15 04:24:42 2013, tonyc wrote:

On Sat, Sep 14, 2013 at 11:34:01PM +0200, Steffen Mueller wrote:

On 09/14/2013 04:12 PM, Father Chrysostomos via RT wrote:

This would be at least 30GB, possibly more, in this case. Most
cases of
this class of bug would need hundreds of gigabytes.

For the record, if anybody finds a case that really, really requires
testing on a machine with lots of RAM, then I can do that.

It doesn't come for free in terms of my time. I'd have to rig an
otherwise more bare-bones machine with all sorts of build tools, and
then obtain build the right perl. On top of that, there might be a
delay of days or even weeks. But I should be able to locate a
machine with at least 96GB, more likely 192GB if need be.

The new dromedary has 96GB:

[tonyc@dromedary-001 ~]$ free
total used free shared buffers cached
Mem: 99050664 29417336 69633328 0 1336928 21840484
-/+ buffers/cache: 6239924 92810740
Swap: 2097144 19224 2077920

Tony

That’s useful to know. The last time I tried to use that much ram I
brought the whole system down. :-)

--

Father Chrysostomos

p5pRT · 2013-09-16T16:42:34Z

From @tsee

On 09/15/2013 01:24 PM, Tony Cook wrote:

On Sat, Sep 14, 2013 at 11:34:01PM +0200, Steffen Mueller wrote:

On 09/14/2013 04:12 PM, Father Chrysostomos via RT wrote:

This would be at least 30GB, possibly more, in this case. Most cases of
this class of bug would need hundreds of gigabytes.

For the record, if anybody finds a case that really, really requires
testing on a machine with lots of RAM, then I can do that.

It doesn't come for free in terms of my time. I'd have to rig an
otherwise more bare-bones machine with all sorts of build tools, and
then obtain build the right perl. On top of that, there might be a
delay of days or even weeks. But I should be able to locate a
machine with at least 96GB, more likely 192GB if need be.

The new dromedary has 96GB:

Ui, nice. Dennis++ for getting some of our production hardware for this. :)

--Steffen

p5pRT · 2013-09-17T06:28:19Z

From @cpansprout

On Fri Sep 13 00:59:28 2013, andreas.koenig.7os6VVqR@franz.ak.mind.de wrote:

git bisect
----------
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700
Allow 64\-bit array and stack offsets in entersub & goto
diagnostics
-----------
http://www.cpantesters.org/cpan/report/039a5aec-1c0b-11e3-b738-
32d47e66b2dc

Failing reports come only from non-threaded perls. Threaded perls pass
all tests. The crucial test t/060_driver/memcached/serializer.t can be
skipped for various reasons, so make sure you have at least
Cache::Memcached::Fast and Data::MessagePack installed. Maybe there
are
other optional dependencies, check for skip messages.

This appears to be a hash randomisation issue, though I have not yet
figured out why 5.18.1 is not triggering it. I am still looking into it.

This little script:

use Data::MessagePack;
$VAR1 = { "xxx" => undef, "zzz" => "hooo", "yyy" => "25c", "k_1" =>
undef, "k_2" => "b", "k_3" => "hooo" };
use Data::Dumper;
++$Data::Dumper::Useqq;
warn Dumper +Data::MessagePack->pack($VAR1)'

gives me different output each time (from 4 runs):

$VAR1 =
"\206\243yyy\24325c\243zzz\244hooo\243xxx\300\243k_1\300\243k_3\244hooo\243k_2\241b";
$VAR1 =
"\206\243xxx\300\243k_1\300\243k_2\241b\243zzz\244hooo\243yyy\24325c\243k_3\244hooo";
$VAR1 =
"\206\243xxx\300\243k_3\244hooo\243yyy\24325c\243zzz\244hooo\243k_1\300\243k_2\241b";
$VAR1 =
"\206\243k_2\241b\243xxx\300\243zzz\244hooo\243k_3\244hooo\243k_1\300\243yyy\24325c";

With PERL_HASH_SEED=0 the differences go away (for the one-liner, but
t/060_driver/memcached/serializer.t still fails).

With 5.18.1, the output is always the same. This is very strange....

--

Father Chrysostomos

p5pRT · 2013-09-19T13:29:41Z

From @cpansprout

On Mon Sep 16 23:28:19 2013, sprout wrote:

On Fri Sep 13 00:59:28 2013, andreas.koenig.7os6VVqR@franz.ak.mind.de
wrote:
git bisect
----------
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700
Allow 64\-bit array and stack offsets in entersub & goto
diagnostics
-----------
http://www.cpantesters.org/cpan/report/039a5aec-1c0b-11e3-b738-
32d47e66b2dc

Failing reports come only from non-threaded perls. Threaded perls
pass
all tests. The crucial test t/060_driver/memcached/serializer.t can
be
skipped for various reasons, so make sure you have at least
Cache::Memcached::Fast and Data::MessagePack installed. Maybe there
are
other optional dependencies, check for skip messages.
This appears to be a hash randomisation issue, though I have not yet
figured out why 5.18.1 is not triggering it. I am still looking into
it.

The randomisation appears to be the results of arguments (that
presumably tell it to sort the keys) getting dropped. I don’t know
exactly where that is happenning (i.e., which function is getting its
arguments dropped).

I just know that the attached patch fixes it, which makes absolutely no
sense to me.

I have to add the I32 cast in at least those two places for the bug to
go away. But how can if((I32)items) vs if(items) make any difference?

Hmm, could this have something to do with mg_size returning I32,
resulting in some optimisation with the if() that follows?

--

Father Chrysostomos

p5pRT · 2013-09-19T13:29:41Z

From @cpansprout

Inline Patch

diff --git a/pp_hot.c b/pp_hot.c
index 1155328..469a9ee 100644
--- a/pp_hot.c
+++ b/pp_hot.c
@@ -2717,12 +2717,12 @@ try_autoload:
 	    AV * const av = GvAV(PL_defgv);
 	    const SSize_t items = AvFILL(av) + 1;
 
-	    if (items) {
+	    if ((I32)items) {
 		SSize_t i = 0;
 		const bool m = cBOOL(SvRMAGICAL(av));
 		/* Mark is at the end of the stack. */
 		EXTEND(SP, items);
-		for (; i < items; ++i)
+		for (; i < (I32)items; ++i)
 		{
 		    SV *sv;
 		    if (m) {

p5pRT · 2013-09-19T20:16:59Z

From @cpansprout

On Thu Sep 19 06:29:41 2013, sprout wrote:

On Mon Sep 16 23:28:19 2013, sprout wrote:
On Fri Sep 13 00:59:28 2013, andreas.koenig.7os6VVqR@franz.ak.mind.de
wrote:
git bisect
----------
commit de935cc
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Sep 6 12:35:56 2013 -0700
Allow 64\-bit array and stack offsets in entersub & goto
diagnostics
-----------
http://www.cpantesters.org/cpan/report/039a5aec-1c0b-11e3-b738-
32d47e66b2dc

Failing reports come only from non-threaded perls. Threaded perls
pass
all tests. The crucial test t/060_driver/memcached/serializer.t can
be
skipped for various reasons, so make sure you have at least
Cache::Memcached::Fast and Data::MessagePack installed. Maybe there
are
other optional dependencies, check for skip messages.
This appears to be a hash randomisation issue, though I have not yet
figured out why 5.18.1 is not triggering it. I am still looking into
it.
The randomisation appears to be the results of arguments (that
presumably tell it to sort the keys) getting dropped. I don’t know
exactly where that is happenning (i.e., which function is getting its
arguments dropped).

That was just a guess based on the fact that the cast in if((I32)items)
fixed it. I suspected (without checking) that the if block was being
skipped.

I just know that the attached patch fixes it, which makes absolutely no
sense to me.

I have to add the I32 cast in at least those two places for the bug to
go away. But how can if((I32)items) vs if(items) make any difference?

Hmm, could this have something to do with mg_size returning I32,
resulting in some optimisation with the if() that follows?

I began to think after writing that that it smelt a bit like an
uninitialized value....

And now it appears to be exactly that: Data::MessagePack’s
xs-src/pack.c has this:

enc_t enc;
enc.sv = sv_2mortal(newSV(INIT_SIZE));
enc.cur = SvPVX(enc.sv);
enc.end = SvEND(enc.sv);
SvPOK_only(enc.sv);

// setup configuration
dMY_CXT;
enc.prefer_int = MY_CXT.prefer_int; // back compat
if(SvROK(self) && SvTYPE(SvRV(self)) == SVt_PVHV) {
HV* const hv = (HV*)SvRV(self);
SV** svp;

svp = hv_fetchs(hv, "prefer_integer", FALSE);
if(svp) {
enc.prefer_int = SvTRUE(*svp) ? true : false;
}

svp = hv_fetchs(hv, "canonical", FALSE);
if(svp) {
enc.canonical = SvTRUE(*svp) ? true : false;
}
}

There are no elses here to initialise enc.prefer_int and enc.canonical
to a default value.

So I have submitted a report to
<https://rt.cpan.org/Ticket/Display.html?id=88823>.

--

Father Chrysostomos

p5pRT · 2013-09-20T00:48:18Z

From @cpansprout

On Thu Sep 19 13:16:59 2013, sprout wrote:

So I have submitted a report [against Data::MessagePack] to
<https://rt.cpan.org/Ticket/Display.html?id=88823>.

Data::Model is also at fault, in expecting canonical order without
asking for it, so I have attached a patch to
<https://rt.cpan.org/Ticket/Display.html?id=76404.> and also filed
<https://github.com/yappo/p5-Data-Model/issues/1>.

This is not a new bug, as it has failed intermittently in the past. So
I am marking this as rejected, since nothing needs to be done in perl to
resolve it.

--

Father Chrysostomos

p5pRT · 2013-09-20T00:48:19Z

@cpansprout - Status changed from 'open' to 'rejected'

p5pRT · 2013-09-20T00:49:25Z

From @cpansprout

On Thu Sep 19 17:48:18 2013, sprout wrote:

On Thu Sep 19 13:16:59 2013, sprout wrote:

So I have submitted a report [against Data::MessagePack] to
<https://rt.cpan.org/Ticket/Display.html?id=88823>.

Data::Model is also at fault, in expecting canonical order without
asking for it, so I have attached a patch to
<https://rt.cpan.org/Ticket/Display.html?id=76404.>

I put the dot inside the angle brackets by mistake. Here is a clickable
version of the link (if your e-mail client makes it a link):

https://rt.cpan.org/Ticket/Display.html?id=76404

and also filed
<https://github.com/yappo/p5-Data-Model/issues/1>.

This is not a new bug, as it has failed intermittently in the past. So
I am marking this as rejected, since nothing needs to be done in perl to
resolve it.

--

Father Chrysostomos

p5pRT closed this as completed Sep 20, 2013

p5pRT added Severity Low distro-Linux i32-misuse labels Oct 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bleadperl v5.19.3-361-gde935cc breaks YAPPO/Data-Model-0.00008.tar.gz #13254

Bleadperl v5.19.3-361-gde935cc breaks YAPPO/Data-Model-0.00008.tar.gz #13254

p5pRT commented Sep 13, 2013

p5pRT commented Sep 13, 2013

p5pRT commented Sep 14, 2013

p5pRT commented Sep 14, 2013

p5pRT commented Sep 14, 2013

p5pRT commented Sep 14, 2013

p5pRT commented Sep 14, 2013

p5pRT commented Sep 15, 2013

p5pRT commented Sep 15, 2013

p5pRT commented Sep 16, 2013

p5pRT commented Sep 17, 2013

p5pRT commented Sep 19, 2013

p5pRT commented Sep 19, 2013

p5pRT commented Sep 19, 2013

p5pRT commented Sep 20, 2013

p5pRT commented Sep 20, 2013

p5pRT commented Sep 20, 2013

Bleadperl v5.19.3-361-gde935cc breaks YAPPO/Data-Model-0.00008.tar.gz #13254

Bleadperl v5.19.3-361-gde935cc breaks YAPPO/Data-Model-0.00008.tar.gz #13254

Comments

p5pRT commented Sep 13, 2013

p5pRT commented Sep 13, 2013

From @andk

p5pRT commented Sep 14, 2013

From @jkeenan

p5pRT commented Sep 14, 2013

p5pRT commented Sep 14, 2013

From @cpansprout

p5pRT commented Sep 14, 2013

From @tsee

p5pRT commented Sep 14, 2013

From victor@vsespb.ru

p5pRT commented Sep 15, 2013

From @tonycoz

p5pRT commented Sep 15, 2013

From @cpansprout

p5pRT commented Sep 16, 2013

From @tsee

p5pRT commented Sep 17, 2013

From @cpansprout

p5pRT commented Sep 19, 2013

From @cpansprout

p5pRT commented Sep 19, 2013

From @cpansprout

p5pRT commented Sep 19, 2013

From @cpansprout

p5pRT commented Sep 20, 2013

From @cpansprout

p5pRT commented Sep 20, 2013

p5pRT commented Sep 20, 2013

From @cpansprout