Strange and unwarranted underflow in string-to-number #13389

p5pRT · 2013-10-31T14:59:37Z

Migrated from rt.perl.org#120426 (status was 'open')

Searchable as RT120426$

p5pRT · 2013-10-31T14:59:37Z

From mwelinder@gmail.com

Created by mwelinder@gmail.com

Compare the following two commands. The first one thinks the
number is zero, the second one works fine. The whole difference
is an extra "0" in the mantissa.

# echo 0.1530e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
# echo 0.153e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=1.53e-306

The above is not supposed to underflow; there are still a couple of orders
of magnitude left:
# echo 0.1e-307 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=9.99999999999999e-309

Observed with 5.16.0
Not observed with 5.6.1
Not observed with 5.12.4

Observed with OpenSuSE build (5.16.0).

Perl Info


Flags:
    category=core
    severity=medium

This perlbug was built using Perl 5.16.0 - Mon Mar 11 10:57:50 UTC 2013
It is being executed now by  Perl 5.16.0 - Mon Mar 11 10:54:42 UTC 2013.

Site configuration information for perl 5.16.0:

Configured by abuild at Mon Mar 11 10:54:42 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 0) configuration:
   
  Platform:
    osname=linux, osvers=3.4.6-2.10-default, archname=x86_64-linux-thread-multi
    uname='linux build30 3.4.6-2.10-default #1 smp thu jul 26 09:36:26 utc 2012 (641c197) x86_64 x86_64 x86_64 gnulinux '
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
    ccversion='', gccversion='4.7.1 20120723 [gcc-4_7-branch revision 189773]', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
    libpth=/lib64 /usr/lib64 /usr/local/lib64
    libs=-lm -ldl -lcrypt -lpthread
    perllibs=-lm -ldl -lcrypt -lpthread
    libc=/lib64/libc-2.15.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.15'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.16.0/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches:
    


@INC for perl 5.16.0:
    /usr/lib/perl5/site_perl/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.0
    /usr/lib/perl5/vendor_perl/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.16.0
    /usr/lib/perl5/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/5.16.0
    /usr/lib/perl5/site_perl/5.16.0/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.0
    /usr/lib/perl5/site_perl
    .


Environment for perl 5.16.0:
    HOME=/home/welinder
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/usr/lib64/mpi/gcc/openmpi/lib64
    LOGDIR (unset)
    PATH=/home/welinder/bin/shell:/home/welinder/bin/shell:/usr/lib64/mpi/gcc/openmpi/bin:/home/welinder/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2013-12-02T15:21:51Z

From @iabyn

On Thu, Oct 31, 2013 at 07:59:37AM -0700, via RT wrote:

Compare the following two commands. The first one thinks the
number is zero, the second one works fine. The whole difference
is an extra "0" in the mantissa.

# echo 0.1530e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
# echo 0.153e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=1.53e-306

The above is not supposed to underflow; there are still a couple of orders
of magnitude left:
# echo 0.1e-307 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=9.99999999999999e-309

Observed with 5.16.0
Not observed with 5.6.1
Not observed with 5.12.4

Observed with OpenSuSE build (5.16.0).

I've (hopefully) fixed it with the following commit, which I've just
pushed for smoking:

commit 07079ca
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Mon Dec 2 15:04:49 2013 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Mon Dec 2 15:04:49 2013 +0000

[perl #120426] atof() errors with trailing 0's

For small numbers, trailing zeros in the fractional part of a
floating-point literal could cause rounding to zero.

From the bug report:

$ echo 0.1530e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.153e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=1.53e-306

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

--
print+qq&$}$"$/$s$,$a$d$g$s$@$.$q$,$:$.$q$^$,$@$a$~~$;$.$q$m&if+map{m,^\d{0\,},,${$::{$'}}=chr($"+=$&||1)}q&10m22,42}6:17a2~~2.3@3;^2dg3q/s"&=~m*\d\*.*g

p5pRT · 2013-12-02T15:21:52Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2013-12-02T15:43:50Z

From zefram@fysh.org

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero, such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0

-zefram

p5pRT · 2013-12-02T16:35:49Z

From @iabyn

On Mon, Dec 02, 2013 at 03:42:59PM +0000, Zefram wrote:

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero, such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0

Oh yeah :-(

I'll need to rethink that.

--
Red sky at night - gerroff my land!
Red sky at morning - gerroff my land!
-- old farmers' sayings #14

p5pRT · 2013-12-03T13:32:06Z

From @iabyn

On Mon, Dec 02, 2013 at 04:34:34PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 03:42:59PM +0000, Zefram wrote:

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero, such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0

Oh yeah :-(

I'll need to rethink that.

Second attempt now smoking as smoke-me/davem/float1:

commit 72cfc30
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Mon Dec 2 15:04:49 2013 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Tue Dec 3 12:30:01 2013 +0000

[perl #120426] atof() small value rounding errors

For something like 0.153e-305, which is small, but not quite the smallest
number (which is around 2.2e-308), adding extra digits to the fractional par
could cause unnecessary rounding to zero.

From the bug report:

$ echo 0.1530e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.153e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=1.53e-306

This was because 0.1234e-305 is calculated as

1234 / (10^309)

and 10^309 becomes infinity. In these edge cases, repeatedly decrement
the exponent and divide the mantissa by 10 until the exponent becomes in
range; in this case we instead calculate

123 / (10^308)

--
Technology is dominated by two types of people: those who understand what
they do not manage, and those who manage what they do not understand.

p5pRT · 2013-12-04T14:04:51Z

From mwelinder@gmail.com

If I understand you right, you are rolling your own atof. That is tricky
business. There are some tricky cases at

http://www.exploringbinary.com/incorrectly-rounded-conversions-in-gcc-and-glibc/

that glibc has only recently gotten right.

M.

On Tue, Dec 3, 2013 at 8:32 AM, Dave Mitchell via RT
<perlbug-followup@perl.org> wrote:

On Mon, Dec 02, 2013 at 04:34:34PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 03:42:59PM +0000, Zefram wrote:

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero, such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0

Oh yeah :-(

I'll need to rethink that.

Second attempt now smoking as smoke-me/davem/float1:

commit 72cfc30
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Mon Dec 2 15:04:49 2013 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Tue Dec 3 12:30:01 2013 +0000
\[perl \#120426\] atof small value rounding errors

For something like 0\.153e\-305\, which is small\, but not quite the smallest
number $which is around 2\.2e\-308$\, adding extra digits to the fractional par
could cause unnecessary rounding to zero\.

From the bug report&#8203;:

    $ echo 0\.1530e\-305 | perl \-e '$v = \<STDIN>; print "v="\, $v \+ 0\, "\\n";'
    v=0
    $ echo 0\.153e\-305  | perl \-e '$v = \<STDIN>; print "v="\, $v \+ 0\, "\\n";'
    v=1\.53e\-306

This was because 0\.1234e\-305 is calculated as

    1234 / $10^309$

and 10^309 becomes infinity\. In these edge cases\, repeatedly decrement
the exponent and divide the mantissa by 10 until the exponent becomes in
range; in this case we instead calculate

    123 / $10^308$
--
Technology is dominated by two types of people: those who understand what
they do not manage, and those who manage what they do not understand.

p5pRT · 2013-12-04T17:23:00Z

From @iabyn

On Tue, Dec 03, 2013 at 01:21:53PM -0500, Morten Welinder wrote:

If I understand you right, you are rolling your own atof.

Yep.

That is tricky business.

Yep.

There are some tricky cases at
http&#8203;://www\.exploringbinary\.com/incorrectly\-rounded\-conversions\-in\-gcc\-and\-glibc/
that glibc has only recently gotten right.

Thanks.

Does anyone know why we roll our own?

--
A major Starfleet emergency breaks out near the Enterprise, but
fortunately some other ships in the area are able to deal with it to
everyone's satisfaction.
-- Things That Never Happen in "Star Trek" #13

p5pRT · 2013-12-04T17:41:20Z

From @khwilliamson

On 12/04/2013 10:22 AM, Dave Mitchell wrote:

On Tue, Dec 03, 2013 at 01:21:53PM -0500, Morten Welinder wrote:

If I understand you right, you are rolling your own atof.

Yep.

That is tricky business.

Yep.
There are some tricky cases at
 http&#8203;://www\.exploringbinary\.com/incorrectly\-rounded\-conversions\-in\-gcc\-and\-glibc/
that glibc has only recently gotten right.
Thanks.

Does anyone know why we roll our own?

There is a comment in sv.c

/* This is evil, but floating point is even more evil */

It may be because we found plenty of bugs in vendor libc's and thought
we could do a better job. Also is the issue of locales. (That's why we
shouldn't be using atoi()).

p5pRT · 2013-12-04T18:39:59Z

From perl5-porters@perl.org

Dave Mitchell asked:

Does anyone know why we roll our own?

See the thread starting at <20010623225547.Z98663@plum.flirble.org>.

Unfortunately, the reimplementation is the cause of several RT tickets
that have been languishing open for years. (Whenever I care, I use
POSIX::strtod.)

p5pRT · 2013-12-05T04:21:20Z

From @sisyphus

-----Original Message-----
From: Father Chrysostomos

< (Whenever I care, I use POSIX::strtod.)

Similarly, could we also have a POSIX::strtold available to
"-Duselongdouble" builds of perl ?

Cheers,
Rob

p5pRT · 2013-12-05T17:57:28Z

From mwelinder@gmail.com

A simple solution to the overflow issues is to use the identity

10^n = 5^n * 2^n

5^n is much smaller and 2^n is harmless in the sense that multiplying
or dividing by it won't cause extra rounding errors (on a base 2 machine).

Morten

On Tue, Dec 3, 2013 at 8:32 AM, Dave Mitchell via RT
<perlbug-followup@perl.org> wrote:

On Mon, Dec 02, 2013 at 04:34:34PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 03:42:59PM +0000, Zefram wrote:

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero, such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0

Oh yeah :-(

I'll need to rethink that.

Second attempt now smoking as smoke-me/davem/float1:

commit 72cfc30
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Mon Dec 2 15:04:49 2013 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Tue Dec 3 12:30:01 2013 +0000
\[perl \#120426\] atof small value rounding errors

For something like 0\.153e\-305\, which is small\, but not quite the smallest
number $which is around 2\.2e\-308$\, adding extra digits to the fractional par
could cause unnecessary rounding to zero\.

From the bug report&#8203;:

    $ echo 0\.1530e\-305 | perl \-e '$v = \<STDIN>; print "v="\, $v \+ 0\, "\\n";'
    v=0
    $ echo 0\.153e\-305  | perl \-e '$v = \<STDIN>; print "v="\, $v \+ 0\, "\\n";'
    v=1\.53e\-306

This was because 0\.1234e\-305 is calculated as

    1234 / $10^309$

and 10^309 becomes infinity\. In these edge cases\, repeatedly decrement
the exponent and divide the mantissa by 10 until the exponent becomes in
range; in this case we instead calculate

    123 / $10^308$
--
Technology is dominated by two types of people: those who understand what
they do not manage, and those who manage what they do not understand.

p5pRT · 2013-12-25T21:51:27Z

From @iabyn

On Tue, Dec 03, 2013 at 01:31:27PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 04:34:34PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 03:42:59PM +0000, Zefram wrote:

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or fractional
component once its been detected that only zeroes are left (with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero, such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0, "\n";'
v=0

Oh yeah :-(

I'll need to rethink that.

Second attempt now smoking as smoke-me/davem/float1:

commit 72cfc30
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Mon Dec 2 15:04:49 2013 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Tue Dec 3 12:30:01 2013 +0000
\[perl \#120426\] atof small value rounding errors

For something like 0\.153e\-305\, which is small\, but not quite the smallest
number $which is around 2\.2e\-308$\, adding extra digits to the fractional par
could cause unnecessary rounding to zero\.

Now in blead as b27804d.

This doesn't address comments that appeared further in this thread.

--
Music lesson: a symbiotic relationship whereby a pupil's embellishments
concerning the amount of practice performed since the last lesson are
rewarded with embellishments from the teacher concerning the pupil's
progress over the corresponding period.

p5pRT · 2019-03-25T19:56:12Z

From @khwilliamson

On Wed, 25 Dec 2013 13:51:27 -0800, davem wrote:

On Tue, Dec 03, 2013 at 01:31:27PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 04:34:34PM +0000, Dave Mitchell wrote:

On Mon, Dec 02, 2013 at 03:42:59PM +0000, Zefram wrote:

Dave Mitchell wrote:

I fixed this by stopping the processing of the integer or
fractional
component once its been detected that only zeroes are left
(with a
suitable adjustment of the exponent).

Eww. That won't fix cases where the extra digits are non-zero,
such as

$ echo 0.15301e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0,
"\n";'
v=0
$ echo 0.15399e-305 | perl -e '$v = <STDIN>; print "v=", $v + 0,
"\n";'
v=0

Oh yeah :-(

I'll need to rethink that.

Second attempt now smoking as smoke-me/davem/float1:

commit 72cfc30
Author: David Mitchell <davem@iabyn.com>
AuthorDate: Mon Dec 2 15:04:49 2013 +0000
Commit: David Mitchell <davem@iabyn.com>
CommitDate: Tue Dec 3 12:30:01 2013 +0000

[perl #120426] atof() small value rounding errors

For something like 0.153e-305, which is small, but not quite the
smallest
number (which is around 2.2e-308), adding extra digits to the
fractional par
could cause unnecessary rounding to zero.

Now in blead as b27804d.

This doesn't address comments that appeared further in this thread.

I looked through this ticket, and don't see any comments that haven't been addressed. Please highlight those that haven't been addressed. If none, we can close this ticket

--
Karl Williamson

khwilliamson · 2019-11-26T19:00:30Z

No response after 6 years. Closing ticket

p5pRT added the khw label Oct 25, 2019

toddr removed the khw label Oct 25, 2019

khwilliamson closed this as completed Nov 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strange and unwarranted underflow in string-to-number #13389

Strange and unwarranted underflow in string-to-number #13389

p5pRT commented Oct 31, 2013

p5pRT commented Oct 31, 2013

p5pRT commented Dec 2, 2013

p5pRT commented Dec 2, 2013

p5pRT commented Dec 2, 2013

p5pRT commented Dec 2, 2013

p5pRT commented Dec 3, 2013

p5pRT commented Dec 4, 2013

p5pRT commented Dec 4, 2013

p5pRT commented Dec 4, 2013

p5pRT commented Dec 4, 2013

p5pRT commented Dec 5, 2013

p5pRT commented Dec 5, 2013

p5pRT commented Dec 25, 2013

p5pRT commented Mar 25, 2019

khwilliamson commented Nov 26, 2019

Strange and unwarranted underflow in string-to-number #13389

Strange and unwarranted underflow in string-to-number #13389

Comments

p5pRT commented Oct 31, 2013

p5pRT commented Oct 31, 2013

From mwelinder@gmail.com

Created by mwelinder@gmail.com

p5pRT commented Dec 2, 2013

From @iabyn

p5pRT commented Dec 2, 2013

p5pRT commented Dec 2, 2013

From zefram@fysh.org

p5pRT commented Dec 2, 2013

From @iabyn

p5pRT commented Dec 3, 2013

From @iabyn

p5pRT commented Dec 4, 2013

From mwelinder@gmail.com

p5pRT commented Dec 4, 2013

From @iabyn

p5pRT commented Dec 4, 2013

From @khwilliamson

p5pRT commented Dec 4, 2013

From perl5-porters@perl.org

p5pRT commented Dec 5, 2013

From @sisyphus

p5pRT commented Dec 5, 2013

From mwelinder@gmail.com

p5pRT commented Dec 25, 2013

From @iabyn

p5pRT commented Mar 25, 2019

From @khwilliamson

khwilliamson commented Nov 26, 2019