Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Floating Point numbers do not roundtrip to string correctly. #16460

Open
p5pRT opened this issue Mar 9, 2018 · 6 comments
Open

Floating Point numbers do not roundtrip to string correctly. #16460

p5pRT opened this issue Mar 9, 2018 · 6 comments

Comments

@p5pRT
Copy link

p5pRT commented Mar 9, 2018

Migrated from rt.perl.org#132959 (status was 'open')

Searchable as RT132959$

@p5pRT
Copy link
Author

p5pRT commented Mar 9, 2018

From marco@nethype.de

Created by marco@nethype.de

On my debian 64 bit system with debians perl, perl fails to stringify floating point
numbers correctly. We found this because floating point did not roundtrip from and to a mysql
database correctly, but it can be reproduced on the commandline​:
 
  $ perl -e 'die 1520604616.472925'
  1520604616.47292 at -e line 1.

Note the missing "5" at the end. This is a bug because "1520604616.47292" and "1520604616.472925" get mapped
to different floating point numbers by perl​:

  $ perl -e 'die unpack "H*", pack "d", 1520604616.472925'
  67441e72a5a8d641 at -e line 1.
  $ perl -e 'die unpack "H*", pack "d", 1520604616.47292'
  52441e72a5a8d641 at -e line 1.

This is a not a problem with floating point itself, as there are distinct string and binary representations
for both numbers​:

  $ perl -e 'die unpack "H*", pack "d", 1520604616.472925*2'
  67441e72a5a8e641 at -e line 1.
  $ perl -e 'die unpack "H*", pack "d", 1520604616.47292*2'
  52441e72a5a8e641 at -e line 1.

Therefore I think this is a bug in perl - a floating point number should round trip to string and back
in cases where this is possible, and it is possible for the above number, but perl maps both numbers to the same string
on output (float->string), but not the same binary on input (string->float).

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.24.1:

Configured by Debian Project at Tue Sep 12 16:37:26 UTC 2017.

Summary of my perl5 (revision 5 version 24 subversion 1) configuration:
   
  Platform:
    osname=linux, osvers=3.16.0, archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-Gg9XxY/perl-5.24.1=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.24 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.24 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.24 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.24.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.24.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dusesitecustomize -Duse64bitint -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.24.1'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='x86_64-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='6.3.0 20170516', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='x86_64-linux-gnu-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.24.so, so=so, useshrplib=true, libperl=libperl.so.5.24
    gnulibc_version='2.24'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] https://bugs.debian.org/100195 Document the Net::SMTP 'Port' option
    DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the upstream Git repository
    DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.24.1-3+deb9u2 in patchlevel.h
    DEBPKG:debian/skip-kfreebsd-crash - https://bugs.debian.org/628493 [perl #96272] Skip a crashing test case in t/op/threads.t on GNU/kFreeBSD
    DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:debian/regen-skip - Skip a regeneration check in unrelated git repositories
    DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/devel-ppport-reproducibility - https://bugs.debian.org/801523 Sort the list of XS code files when generating RealPPPort.xs
    DEBPKG:debian/encode-unicode-bom-doc - https://bugs.debian.org/798727 Document Debian backport of Encode::Unicode fix
    DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub
    DEBPKG:fixes/crosscompile-no-targethost - [23695c0] [perl #127234] Fix the Configure escape with usecrosscompile but no targethost
    DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
    DEBPKG:fixes/ok-pod - Added encoding for pod.
    DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/nntp_docs - https://bugs.debian.org/51962 Net::NNTP: Correct innd/nnrpd confusion in relation to Reader option
    DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO
    DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:fixes/time_piece_doc - https://bugs.debian.org/817925 Time::Piece: Improve documentation for add_months and add_years
    DEBPKG:fixes/perlbug-refactor - https://bugs.debian.org/822463 [perl #128020] perlbug: Refactor duplicated file reading code
    DEBPKG:fixes/perlbug-linewrap - https://bugs.debian.org/822463 [perl #128020] perlbug: wrap overly long lines
    DEBPKG:fixes/hurd_sigaction - https://bugs.debian.org/825016 [d54f4ed] ext/POSIX/t/sigaction.t: Skip uid and pid tests on GNU/Hurd
    DEBPKG:fixes/hurd_hints - [4694301] https://bugs.debian.org/825020 [perl #128279] Modify hints for Hurd per Debian ticket 825020.
    DEBPKG:fixes/extutils-parsexs-reproducibility - [perl #128517] https://bugs.debian.org/829296 Make the output of ExtUtils::ParseXS reproducible
    DEBPKG:debian/CVE-2016-1238/sitecustomize-in-etc - Look for sitecustomize.pl in /etc/perl rather than sitelib on Debian systems
    DEBPKG:debian/CVE-2016-1238/test-suite-without-dot - [perl #127810] Patch unit tests to explicitly insert "." into @INC when needed.
    DEBPKG:debian/CVE-2016-1238/eumm-without-dot - [perl #127810] Add PERL_USE_UNSAFE_INC support to EU::MM for fortify_inc support.
    DEBPKG:debian/CVE-2016-1238/cpan-without-dot - [perl #127810] Set PERL_USE_UNSAFE_INC for cpan usage
    DEBPKG:debian/document_inc_removal - Document in perlvar that we remove '.' from @INC by default
    DEBPKG:fixes/extutils_makemaker_reproducible - https://bugs.debian.org/835815 https://bugs.debian.org/834190 Make perllocal.pod files reproducible
    DEBPKG:debian/CVE-2016-1238/remove-inc-test - Remove test for '.' in @INC as it might not be
    DEBPKG:fixes/file_path_hurd_errno - File-Path: Fix test failure in Hurd due to hard-coded ENOENT
    DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems
    DEBPKG:fixes/test-builder-warning - https://bugs.debian.org/840968 Silence a 'used only once' warning in Test::Builder
    DEBPKG:fixes/longdblinf-randomness - [dd68853] [perl #130133] https://bugs.debian.org/844752 Configure: fix garbage filtering with 80-bit long doubles
    DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
    DEBPKG:fixes/list_assign_leak - [1050723] [perl #130766] https://bugs.debian.org/855064 avoid a leak in list assign from/to magic values
    DEBPKG:fixes/perlfunc_inc_doc - [a03e9f8] https://bugs.debian.org/839536 [perl #130832] Documentation fixes for '.' possibly no longer being in @INC
    DEBPKG:fixes/file_path_chmod_race - https://bugs.debian.org/863870 [rt.cpan.org #121951] Prevent directory chmod race attack.
    DEBPKG:fixes/extutils_file_path_compat - Correct the order of tests of chmod(). (#294)
    DEBPKG:debian/customized - Update customized.dat for files patched in Debian
    DEBPKG:fixes/getopt-long-1 - https://bugs.debian.org/855532 [rt.cpan.org #114999] Fix bug RT#114999
    DEBPKG:fixes/getopt-long-2 - [rt.cpan.org #120300] Withdraw part of commit 5d9947fb445327c7299d8beb009d609bc70066c0, which tries to implement more GNU getopt_long campatibility. GNU
    DEBPKG:fixes/getopt-long-3 - provide a default value for optional arguments
    DEBPKG:fixes/getopt-long-4 - https://bugs.debian.org/864544 [rt.cpan.org #122068] Fix issue #122068.
    DEBPKG:fixes/fbm-instr-crash - [bb152a4] [perl #131575] https://bugs.debian.org/864782 don't call Perl_fbm_instr() with negative length
    DEBPKG:debian/CVE-2016-1238/base-pm-amends-pt2 - [1afa289] Limit dotless-INC effect on base.pm with guard:
    DEBPKG:fixes/CVE-2017-12837 - https://bugs.debian.org/875596 [perl #131582] [f7e5417] regcomp [perl #131582]
    DEBPKG:fixes/CVE-2017-12883 - https://bugs.debian.org/875597 [perl #131598] [40b3cda] PATCH: [perl #131598]


@INC for perl 5.24.1:
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.24.1
    /usr/local/share/perl/5.24.1
    /usr/lib/x86_64-linux-gnu/perl5/5.24
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.24
    /usr/share/perl/5.24
    /usr/local/lib/site_perl
    /usr/lib/x86_64-linux-gnu/perl-base


Environment for perl 5.24.1:
    HOME=/root
    LANG=en_DK.UTF-8
    LANGUAGE=en_US:en
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/opt/sbin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Mar 11, 2018

From @sisyphus

Le Fri, 09 Mar 2018 07​:21​:44 -0800, marco@​nethype.de a écrit :

On my debian 64 bit system with debians perl, perl fails to stringify
floating point
numbers correctly. We found this because floating point did not
roundtrip from and to a mysql
database correctly, but it can be reproduced on the commandline​:

$ perl -e 'die 1520604616.472925'
1520604616.47292 at -e line 1.

I think it should be fixed, though I'm not so sure that everyone agrees.
In any case, no-one has yet stepped up to do anything about it.

This is something that has been raised before - eg​:
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=127182

Assuming that the floating point type is 'double', the "round trip" will fail for many values if less than 17 decimal digits of precision are used.
Perl's print() only outputs a maximum of 15 digits, so avoid it if you need to do that "round trip".
Instead, use (s)printf() to deliver the requisite 17 digits.

So ... you go for the 17 digits​:

$ perl -le 'printf "%.16e\n", 1520604616.472925'
1.5206046164729249e+09

and you still end up with something that doesn't match.
Of course, it's just that 1520604616.472925 and 1.5206046164729249e+09 map to the very same double​:

$ perl -le 'printf "%a\n%a\n", 1520604616.472925, 1.5206046164729249e+09'
0x1.6a8a5721e4467p+30
0x1.6a8a5721e4467p+30

and it turns out that double is closer to 1.5206046164729249e+09 than it is to 1.520604616472925e+09.
The round trip succeeds - though the fact that 2 seemingly different values appeared can be a little surprising.)

Incidentally, in the bug report I linked to above, there's reference to an algorithm by Guy Steele that can be used to avoid such surprises.

Cheers,
Rob

@p5pRT
Copy link
Author

p5pRT commented Mar 11, 2018

The RT System itself - Status changed from 'new' to 'open'

@sisyphus
Copy link
Contributor

sisyphus commented Nov 25, 2023

There's no sign of this ever being fixed, and I don't think there are very many people (apart from me and the OP) who care about it.
I therefore suggest that this Issue can be closed.
(I don't have the required permissions to do that.)

As a brazen plug, I will mention a couple of extensions that do the roundtrip correctly && do it such that the string contains as few significant decimal digits as possible for the roundtrip to succeed.

One is Math::Ryu, which uses the Ryu algorithm (https://github.com/ulfjack/ryu). Math::Ryu's d2s() function works such that, for any double precision NV $d, the condition d2s($d) == $d will always be true (unless $d is a NaN), and the number of significant digits in the string returned by d2s($d) will be the minimum required for the condition to evaluate as true.

Examples:
d2s(0.1) returns the string 0.1
d2s(sqrt 2) returns the string 1.4142135623730951E0
d2s(1520604616.472925) returns the string 1.520604616472925E9
d2s(1.4 / 10) returns the string 1.3999999999999999E-1
(Perl knows damn well that 1.4/10 is not equivalent to the double 0.14, yet it happily interpolates the value of 1.4/10 to 0.14.)

Of course, having to do print d2s($d) is not as convenient as being able to do print $d, but at least the former always provides a value that will do the roundtrip.

The main problem with Math::Ryu is that it only builds on perls whose nvtype is "double". (If the nvtype is "double" then Math::Ryu should build and install straight out of the box.)

If perl's nvtype is not "double" then you'll need to use Math::MPFR's nvtoa() function - again which does the roundtrip using as few significant digits as possible. It uses a "dragon" implementation devised by Steele and White (https://lists.nongnu.org/archive/html/gcl-devel/2012-10/pdfkieTlklRzN.pdf).
Again, having to do print nvtoa($d) is not as convenient as being able to do print $d.
Also, Math::MPFR depends upon the gmp and mpfr C libraries.

@jkeenan
Copy link
Contributor

jkeenan commented Nov 25, 2023

There's no sign of this ever being fixed, and I don't think there are very many people (apart from me and the OP) who care about it. I therefore suggest that this Issue can be closed. (I don't have the required permissions to do that.)

Problem: If we were to simply close this ticket now, in a year or two someone else would stumble on this problem and open a new ticket, probably without searching old tickets. At the very least, we need to document this limitation and perhaps mention some extensions as you do.

Can you suggest some appropriate documentation?

Of course, if someone thinks this bug is fixable, then we should keep this ticket open.

@sisyphus
Copy link
Contributor

sisyphus commented Nov 26, 2023

I'm not sure that it's actually a "bug", though I may well have referred to it as such, from time to time.
It's really nothing more than an unfortunate consequence of deliberate design decisions that were made years ago.
Those decisions were that decimal stringification should round the value of the double precision NV to 15 significant decimal digits - also round to 18 significant decimal digits for the 80-bit extended precision long double NV, and round to 33 significant decimal digits for the __float 128 and IEEE 754 long double NVs.

I was mainly thinking that this issue could be closed because:

  1. there is just 1 post from the OP and just one reply;
  2. it is stalled and unlikely to be acted upon.

On further reflection, I'm thinking that the behaviour-changing aspect of any action would mean that we cannot do anything about this until perl 7.
If so, then that's another reason that this could be closed - especially if this particular forum is specifically for "perl 5" issues.

As regards documentation regarding this issue, I think it would be sufficient to insert into perlnumber.pod a paragraph that states something like:

In interpolating floating point (NV) values to decimal strings, the values are rounded as follows:
round to 15 significant digits if the NV is of type 'double';
round to 18 significant digits if the NV is of type 80-bit 'long double';
round to 33 significant digits if the NV is of type IEEE 754 128-bit 'long double' or '__float128'.
Consequently, the condition ("$nv" == $nv) is not always TRUE, even for non-NaN floating point values.

Maybe place that in the perlnumber documentation under the heading "Decimal Interpolation of Floating Point Values".

What do you think ?
Is there a more appropriate place for this ?
If we can sort out the wording here, then I'll do a PR.

To be clear - I don't mind at all if this issue remains open. It's just that I perceived there was an opportunity here to reduce the number of open issues by one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants