Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bleadperl v5.19.0-570-gb127e37 breaks PERLER/JavaScript-Dumper-0.011.tar.gz #13530

Closed
p5pRT opened this issue Jan 16, 2014 · 8 comments
Closed

Comments

@p5pRT
Copy link

p5pRT commented Jan 16, 2014

Migrated from rt.perl.org#121013 (status was 'resolved')

Searchable as RT121013$

@p5pRT
Copy link
Author

p5pRT commented Jan 16, 2014

From @andk

git


commit b127e37
Author​: Karl Williamson <public@​khwilliamson.com>
Date​: Mon May 13 07​:35​:35 2013 -0600

  PATCH​: [perl #108378] [perl #115800]

diagnostics


http​://www.cpantesters.org/cpan/report/d2dd3370-d826-11e2-bc6f-95dabcbf7481

Thanks to Slaven Rezić for finding this.

perl -V


  Summary of my perl5 (revision 5 version 19 subversion 1) configuration​:
  Commit id​: b127e37
  Platform​:
  osname=linux, osvers=3.10-3-amd64, archname=x86_64-linux-thread-multi-ld
  uname='linux k83 3.10-3-amd64 #1 smp debian 3.10.11-1 (2013-09-10) x86_64 gnulinux '
  config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Duseithreads -Duselongdouble -DDEBUGGING=-g'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.8.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.17'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
  PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_ITHREADS USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LONG_DOUBLE USE_PERLIO
  USE_PERL_ATOF USE_REENTRANT_API
  Built under linux
  Compiled at Jan 16 2014 06​:43​:25
  @​INC​:
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/site_perl/5.19.1/x86_64-linux-thread-multi-ld
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/site_perl/5.19.1
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/5.19.1/x86_64-linux-thread-multi-ld
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/5.19.1
  .

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Jan 28, 2014

From @khwilliamson

Here's the email we got for one ticket that would be nice to add to
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=116923
for 5.20. After 5.20 ships, we could manually add a ticket for 5.22 and
do something so that such emails point to that ticket.

It occurs to me that these emails may be machine generated, and if so,
we could request their originator to change the code to add them to the
current ticket for blockers, if that were feasible.

-------- Original Message --------
From​: - Thu Jan 16 00​:06​:33 2014
X-Account-Key​: account2
X-UIDL​: 0002896e4ad3b7bf
X-Mozilla-Status​: 0001
X-Mozilla-Status2​: 00000000
X-Mozilla-Keys​:
Received​: from x6.develooper.com (x6.develooper.com [207.171.7.86]) by
net.indra.com (8.14.7/8.14.7) with ESMTP id s0G6utEQ071871 for
<public@​khwilliamson.com>; Wed, 15 Jan 2014 23​:56​:57 -0700 (MST)
(envelope-from perl5-porters-return-211508-public=khwilliamson.com@​perl.org)
Received​: from lists-nntp.develooper.com (localhost.localdomain
[127.0.0.1]) by x6.develooper.com (Postfix) with SMTP id D80822A4E for
<public@​khwilliamson.com>; Wed, 15 Jan 2014 22​:56​:46 -0800 (PST)
Received​: (qmail 4640 invoked by uid 514); 16 Jan 2014 06​:56​:45 -0000
Mailing-List​: contact perl5-porters-help@​perl.org; run by ezmlm
Precedence​: bulk
list-help​: <mailto​:perl5-porters-help@​perl.org>
list-unsubscribe​: <mailto​:perl5-porters-unsubscribe@​perl.org>
list-post​: <mailto​:perl5-porters@​perl.org>
X-List-Archive​: <http​://nntp.perl.org/group/perl.perl5.porters/211508>
List-Id​: <perl5-porters.perl.org>
Delivered-To​: mailing list perl5-porters@​perl.org
Received​: (qmail 4628 invoked from network); 16 Jan 2014 06​:56​:45 -0000
Delivered-To​: perl5-porters@​perl.org
X-Spam-Status​: No, hits=-5.1 required=8.0
tests=BAYES_00,PERLBUG_CONF,SPF_NEUTRAL
X-Spam-Check-By​: la.mx.develooper.com
Delivered-To​: rt-perl5-testers@​x1.develooper.com
Mail-From​: perlbug-followup@​perl.org Wed Jan 15 22​:56​:40 2014
Delivered-To​: bugs-perl5-testers@​rt.perl.org
From​: (Andreas J. Koenig) (via RT) <perlbug-followup@​perl.org>
X-RT-NewTicket​: yes
To​: bugs-bitbucket@​rt.perl.org
Mail-Followup-To​: perl5-porters@​perl.org
Reply-To​: perl5-porters@​perl.org
X-RT-Will-Also-CC​: andreas.koenig.7os6VVqR@​franz.ak.mind.de,
Subject​: [perl #121013] Bleadperl v5.19.0-570-gb127e37 breaks
PERLER/JavaScript-Dumper-0.011.tar.gz
In-Reply-To​: <87ob3ctnxr.fsf@​k85.linux.bogus>
References​: <RT-Ticket-121013@​perl.org> <87ob3ctnxr.fsf@​k85.linux.bogus>
Message-ID​: <rt-4.0.18-29970-1389855395-1178.121013-75-0@​perl.org>
X-RT-Loop-Prevention​: perl
RT-Ticket​: perl #121013
Managed-BY​: RT 4.0.18 (http​://www.bestpractical.com/rt/)
RT-Originator​: andreas.koenig.7os6VVqR@​franz.ak.mind.de
MIME-Version​: 1.0
Content-Transfer-Encoding​: 8bit
Content-Type​: text/plain; charset="utf-8"
X-RT-Original-Encoding​: utf-8
Date​: Wed, 15 Jan 2014 22​:56​:35 -0800
Resent-To​: perl5-porters@​perl.org
X-Old-Spam-Check-By​: la.mx.develooper.com
X-Old-Spam-Status​: No, hits=-6.2 required=8.0
tests=BAYES_00,PERLBUG_CONF,RP_MATCHES_RCVD,SPF_PASS
Resent-Message-Id​: <20140116065646.D80822A4E@​x6.develooper.com>
Resent-Date​: Wed, 15 Jan 2014 22​:56​:46 -0800 (PST)
Resent-From​: perl5-porters-return-211508-public=khwilliamson.com@​perl.org
X-DCC-indra.com-Metrics​: net.indra.com; whitelist
X-Likely-Spam​: true
X-Likely-Spam​: true

# New Ticket Created by (Andreas J. Koenig)
# Please include the string​: [perl #121013]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121013 >

git


commit b127e37
Author​: Karl Williamson <public@​khwilliamson.com>
Date​: Mon May 13 07​:35​:35 2013 -0600

  PATCH​: [perl #108378] [perl #115800]

diagnostics


http​://www.cpantesters.org/cpan/report/d2dd3370-d826-11e2-bc6f-95dabcbf7481

Thanks to Slaven Rezić for finding this.

perl -V


  Summary of my perl5 (revision 5 version 19 subversion 1) configuration​:
  Commit id​: b127e37
  Platform​:
  osname=linux, osvers=3.10-3-amd64,
archname=x86_64-linux-thread-multi-ld
  uname='linux k83 3.10-3-amd64 #1 smp debian 3.10.11-1 (2013-09-10)
x86_64 gnulinux '

config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da
-Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des
-Ui_db -Duseithreads -Duselongdouble -DDEBUGGING=-g'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.8.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16,
Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib
/usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
-lgdbm_compat
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.17'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib
-fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
  PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_ITHREADS USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LONG_DOUBLE USE_PERLIO
  USE_PERL_ATOF USE_REENTRANT_API
  Built under linux
  Compiled at Jan 16 2014 06​:43​:25
  @​INC​:

/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/site_perl/5.19.1/x86_64-linux-thread-multi-ld

/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/site_perl/5.19.1

/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/5.19.1/x86_64-linux-thread-multi-ld

/home/sand/src/perl/repoperls/installed-perls/perl/v5.19.0-570-gb127e37/a2da/lib/5.19.1
  .

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Jan 28, 2014

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 5, 2014

From @khwilliamson

On Wed Jan 15 22​:56​:35 2014, andreas.koenig.7os6VVqR@​franz.ak.mind.de wrote​:

git
---
commit b127e37
Author​: Karl Williamson <public@​khwilliamson.com>
Date​: Mon May 13 07​:35​:35 2013 -0600

PATCH​: [perl #108378] [perl #115800]

diagnostics
-----------
http​://www.cpantesters.org/cpan/report/d2dd3370-d826-11e2-bc6f-
95dabcbf7481

Thanks to Slaven Rezić for finding this.

The test that fails in 5.19 is expecting a floating point number to be returned by JSON to be enclosed in double quotes, which it used to be, but after the blamed commit, no longer is. I have tracked this down to the following statement in JSON​::PP (JSON​:XS has a similar logic)​:

return $value # as is
  if $flags & ( B​::SVp_IOK | B​::SVp_NOK ) and !( $flags & B​::SVp_POK ); # SvTYPE is IV or NV?

The blamed commit changed things so that some scalars that are NVs no longer are also PVs. In the past this statement would not return and code further down would enclose it in double quotes, hence the old behavior, and the new.

But the logic here seems wrong to me. I don't understand the possible nuances and gotchas, but it seems to me that if something is an NV, it should be returned as an NV, without regard to if there is a stringified version available or not, which is what SVp_POK indicates.

My guess is that the tests for POK are to detect cases where something has a numeric value, but its string is different from that value and needs to be preserved, such as perhaps "00" (I don't know the internals here, so am just guessing.) But there is no restriction on somebody's deciding to make a numeric scalar POK. It can be done for good reasons, and it could be done for invalid ones. The modules shouldn't depend on this.

(In this case, prior to the blamed commit, a floating point number was made POK when its string form was needed, in order to avoid having to recalculate it each time. But this was found to create bugs where the decimal point varies in a program between, say, a comma and a dot because of locale differences. Making it POK cemented the decimal point to what it was at the time it was first stringified, not what it should be the next time a stringified version is needed. So it isn't made POK and is recalculated each time)

I think that both JSON​::PP and JSON​::XS should add logic to better detect whether something's string is different from its numeric value

--
Karl Williamson

@p5pRT
Copy link
Author

p5pRT commented Mar 6, 2014

From schmorp@schmorp.de

On Wed, Mar 05, 2014 at 11​:09​:47AM -0800, Karl Williamson via RT <perlbug-followup@​perl.org> wrote​:

The test that fails in 5.19 is expecting a floating point number to be returned by JSON to be enclosed in double quotes, which it used to be, but after the blamed commit, no longer is. I have tracked this down to the following statement in JSON​::PP (JSON​:XS has a similar logic)​:

Without seeing the test code nor the patch (both weren't easily
available), I can only make some general comments, however, there is a
good chance that the test itself is buggy.

Perl scalars are (for the most part) typeless - scalars are not NVs or PVs,
not even internally.

JSON​::XS (and JSON​::PP, which copied the logic) attempts the impossible
"and guesses" the type - the rules for that are described in the section
"MAPPING" in the JSON​::XS documentation.

The short version is that the last use/modification counts, and the string
form is preferred when in doubt, so if you printed your NV, or stringified
it, it will serialise as a string.

But the logic here seems wrong to me. I don't understand the possible nuances and gotchas, but it seems to me that if something is an NV, it should be returned as an NV, without regard to if there is a stringified version available or not, which is what SVp_POK indicates.

In your case, this is true, in other cases, this isn't. If you want to
make sure that your number is serialised as number, then you need to +=0
it for example.

The the treatment of "types" varies considerably within perl versions, and
the test as-is is the one that works reliably for perl versions since 5.8,
and errs on the safe side.

My guess is that the tests for POK are to detect cases where something
has a numeric value, but its string is different from that value and
needs to be preserved, such as perhaps "00" (I don't know the internals
here, so am just guessing.) But there is no restriction on somebody's
deciding to make a numeric scalar POK. It can be done for good reasons,
and it could be done for invalid ones. The modules shouldn't depend on
this.

The test si simply there because in JSON, the number 5 is different than
the string "5", while in Perl, there is no semantic difference between
them.

(In this case, prior to the blamed commit, a floating point number was
made POK when its string form was needed, in order to avoid having to
recalculate it each time. But this was found to create bugs where
the decimal point varies in a program between, say, a comma and a dot
because of locale differences.

The locale changes recently introduced in 5.19 cause a great deal of
breakage (probably again a case where the new p5ps don't care for
backwards compatibility). I hope these are sorted out before it gets
released.

I think that both JSON​::PP and JSON​::XS should add logic to better detect whether something's string is different from its numeric value

The problem is that it's not clear what to do in that case. JSON​::XS has to
decide on either string or number.

If the += 0 "trick" no longer works because perl stringifies the NV, then we
are indeed in trouble, as that makes it impossible to even semi-reliably
guess types.

However, such a change would break an enourmous number of programs (most
sql drivers use a similar form of guessing), and one would hope that p5p
would not introduce such a big change (although in the past, they didn't
seem to be reluctant to break things for no demonstrable gain).

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 10, 2014

From @khwilliamson

On 3/5/2014 10​:06 PM, Marc Lehmann wrote​:

The short version is that the last use/modification counts, and the string
form is preferred when in doubt, so if you printed your NV, or stringified
it, it will serialise as a string.

But the logic here seems wrong to me. I don't understand the possible nuances and gotchas, but it seems to me that if something is an NV, it should be returned as an NV, without regard to if there is a stringified version available or not, which is what SVp_POK indicates.
In your case, this is true, in other cases, this isn't. If you want to
make sure that your number is serialised as number, then you need to +=0
it for example.

The the treatment of "types" varies considerably within perl versions, and
the test as-is is the one that works reliably for perl versions since 5.8,
and errs on the safe side.

What if the code in JOSN​::PP instead did something like this​:

  if ($flags & ( B​::SVp_IOK | B​::SVp_NOK ) { # SvTYPE is IV or NV?
  return $value if ! $flags & B​::SVp_POK; # as is
  my $numeric = $value + 0;
  return $value if "$numeric" eq $value; # stringified is identical to
numeric
  }

Would that also work reliably? And insulate the code from us cretins on
p5p?

@p5pRT
Copy link
Author

p5pRT commented May 12, 2014

From @khwilliamson

I submitted a patch to the distribution
https://rt.cpan.org/Public/Bug/Display.html?id=95230

so am resolving this core ticket
--
Karl Williamson

@p5pRT p5pRT closed this as completed May 12, 2014
@p5pRT
Copy link
Author

p5pRT commented May 12, 2014

@khwilliamson - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant