Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perl rounds inode in PP stat #15002

Closed
p5pRT opened this issue Oct 21, 2015 · 23 comments
Closed

perl rounds inode in PP stat #15002

p5pRT opened this issue Oct 21, 2015 · 23 comments

Comments

@p5pRT
Copy link

p5pRT commented Oct 21, 2015

Migrated from rt.perl.org#126414 (status was 'resolved')

Searchable as RT126414$

@p5pRT
Copy link
Author

p5pRT commented Oct 21, 2015

From @bulk88

Created by @bulk88

While I was thinking about implementing real inode info in Win32's PP stat()[1], I discovered that Perl rounds the inode integer.

http​://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file
-compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on 32b IV perl, it has the potential of being rounded if its > 2^53 and therefore is garbage/uninitialized. In a DB on FS scheme, bad things could happen if 2 files that aren't links in reality, "==" in perl as to being the same file.

An inode can be implemented in a couple ways by any FS.

-a 0 based offset into an array of something (offsets or structs) on the disk
-an absolute sector (units of 512 bytes) or byte position into the FS partition
-random looking hash number
-XOR against a secret of any of the 3 above

[1] https://rt.perl.org/Public/Bug/Display.html?id=65052
The last 2 implementations would give very frequent high bits that are > 2^53. Worst case scenario for 0 based inode FS is 2^53 bytes, which is 9007 TB of storage, which I dont think anyone would use 32 bit perl on a enterprise SAN/cluster, but that doesn't address the theoretical hash/xor/checksum based inode FSes.

Everyone agrees storing 64 bit C pointers in 64 bit doubles is forbidden, so why is perl storing inodes in NVs/doubles? Can something be done about this? Fatally error if its over 2^32 or over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and let the user in PP figure out what to do with it (Math​::Int64 it)?

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.23.4:

Configured by Owner at Fri Oct 16 16:33:21 2015.

Summary of my perl5 (revision 5 version 23 subversion 4) configuration:
  Derived from: c8b87727e0a3605fae65ba4f0811d649667abbf0
  Ancestor: 9b331ac6cfda2819646f52e55c4478f2f4123a3d
  Platform:
    osname=MSWin32, osvers=6.1, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32 -D_CONSOLE -DNO_STRICT -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS',
    optimize='-O1 -MD -Zi -DNDEBUG -GL',
    cppflags='-DWIN32'
    ccversion='18.00.31101', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234, doublekind=3
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p523\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'" -subsystem:console,"5.01"'
    libpth=\lib
    libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl523.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p523\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'" -subsystem:console,"5.01"'

Locally applied patches:
    uncommitted-changes
    c8b87727e0a3605fae65ba4f0811d649667abbf0


@INC for perl 5.23.4:
    C:/p523/site/lib
    C:/p523/lib
    .


Environment for perl 5.23.4:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\p523\site\bin;C:\p523\bin;C:\Program Files\ActiveState Komodo Edit 9\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\TortoiseGit\bin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\Program Files\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SDKs\TypeScript\1.0\;
    PERL_BADLANG (unset)
    SHELL (unset)

 		 	   		  

@p5pRT
Copy link
Author

p5pRT commented Oct 21, 2015

From @bulk88

On Tue Oct 20 18​:46​:47 2015, bulk88 wrote​:

This is a bug report for perl from bulk88@​hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.23.4.

-----------------------------------------------------------------
[Please describe your issue here]

While I was thinking about implementing real inode info in Win32's PP
stat()[1], I discovered that Perl rounds the inode integer.

http​://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file
-compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the
OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on
32b IV perl, it has the potential of being rounded if its > 2^53 and

Actually, MS bumped the inode size to 128 bits in Server 2012 OS for their nextgen enterpriseish FS. So for discussion, assume 128 bit inodes are real and have to be supported (personally Win32 Perl can stick with legacy 64 bit inodes for some years).

https://msdn.microsoft.com/en-us/library/windows/desktop/hh802691%28v=vs.85%29.aspx

See also https://msdn.microsoft.com/en-us/library/windows/desktop/aa363788%28v=vs.85%29.aspx

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2015

From @tonycoz

On Tue Oct 20 18​:46​:47 2015, bulk88 wrote​:

While I was thinking about implementing real inode info in Win32's PP
stat()[1], I discovered that Perl rounds the inode integer.

http​://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file
-compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the
OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on
32b IV perl, it has the potential of being rounded if its > 2^53 and
therefore is garbage/uninitialized. In a DB on FS scheme, bad things
could happen if 2 files that aren't links in reality, "==" in perl as
to being the same file.

An inode can be implemented in a couple ways by any FS.

-a 0 based offset into an array of something (offsets or structs) on
the disk
-an absolute sector (units of 512 bytes) or byte position into the FS
partition
-random looking hash number
-XOR against a secret of any of the 3 above

[1] https://rt.perl.org/Public/Bug/Display.html?id=65052
The last 2 implementations would give very frequent high bits that are

2^53. Worst case scenario for 0 based inode FS is 2^53 bytes, which
is 9007 TB of storage, which I dont think anyone would use 32 bit perl
on a enterprise SAN/cluster, but that doesn't address the theoretical
hash/xor/checksum based inode FSes.

Everyone agrees storing 64 bit C pointers in 64 bit doubles is
forbidden, so why is perl storing inodes in NVs/doubles? Can something
be done about this? Fatally error if its over 2^32 or over 2^53? Store
the 64 bit integer as a SVPV in printable ASCII and let the user in PP
figure out what to do with it (Math​::Int64 it)?

I can see us producing some sort of error if the inode number changes value when stored as a NV, perhaps with an option to disable that error, since stat() isn't only used to fetch a file's inode number.

As to how to behave when the inode number doesn't fit, we can look at existing implementations, from the Solaris stat() man page​:

  The stat(), fstat(), and lstat() functions may fail if​:

  EOVERFLOW One of the members is too large to store in the
  stat structure pointed to by buf.

Similarly on FreeBSD​:

  [EOVERFLOW] The file size in bytes cannot be represented correctly
  in the structure pointed to by sb.

If we follow the lead of Solaris/FreeBSD, perl's stat() would simply fail with EOVERFLOW (if available) if the inode number doesn't fit.

As to optionally disabling the error, we could use a variable like ${^WIN32_SLOPPY_STAT}, perhaps ${^SLOPPY_STAT_INO}, or something lexically scoped, to keep the sloppy behaviour restricted to a small amount of code.

Sort of related​: systems with large files can run into this issue for st_size​:

$ ./perl -le 'print +(stat "/home/tony/somefile.txt")[7]'
9.00819925474099e+16
$ ls -l ~/somefile.txt
-rw-r--r-- 1 tony tony 90081992547409912 Oct 27 03​:43 /home/tony/somefile.txt

(that's a sparse file on ZFS)

Tony

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2015

From @ap

* bulk88 <perlbug-followup@​perl.org> [2015-10-21 03​:50]​:

An inode is effectively an opaque pointer (integer) into a FS.

Like a zip code. That is also nominally a number but is meaningless
to do math on, and the only meaningful operation is testing equality.

Can something be done about this? Fatally error if its over 2^32 or
over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and
let the user in PP figure out what to do with it (Math​::Int64 it)?

If you mean converting the inode number 9876543210123456 to the string
'9876543210123456' (maybe only on overflow), then yes I would favour
that (along with updates to the docs telling people to compare inode
using eq and not ==). That way it would transparently Just Work, unlike
throwing an error. The only question I’m unclear on is what this would
do to XS code.

* Tony Cook via RT <perlbug-followup@​perl.org> [2015-10-27 04​:50]​:

I can see us producing some sort of error if the inode number changes
value when stored as a NV, perhaps with an option to disable that
error, since stat() isn't only used to fetch a file's inode number.

That would force all code not written for a controlled environment to
care about this every time it calls stat. Even more irritating, you have
to declare your *dis*interest in the inode number, which is by far the
more common case in my experience. (Most of the time I stat I want the
mtime, next most commonly probably the size. I’ve needed the inode too
but only very occasionally.)

If we’re going to go this route it would be nice to only force code that
actually looks at the inode to care, e.g. attaching magic to the scalar
that dies on access, so unaffected code doesn’t have to care.

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2015

From @bulk88

On Tue Oct 27 00​:54​:49 2015, aristotle wrote​:

* bulk88 <perlbug-followup@​perl.org> [2015-10-21 03​:50]​:

An inode is effectively an opaque pointer (integer) into a FS.

Like a zip code. That is also nominally a number but is meaningless
to do math on, and the only meaningful operation is testing equality.

Can something be done about this? Fatally error if its over 2^32 or
over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and
let the user in PP figure out what to do with it (Math​::Int64 it)?

If you mean converting the inode number 9876543210123456 to the string
'9876543210123456' (maybe only on overflow), then yes I would favour
that (along with updates to the docs telling people to compare inode
using eq and not ==). That way it would transparently Just Work, unlike
throwing an error. The only question I’m unclear on is what this would
do to XS code.

Although I am hesitant about returning PVs and "eq" instead of "==" for perf reasons, that might be the only sane way to support 128 bit inodes. If an inode number is really a string, maybe it should be raw 4/8/16 bytes of binary, not printable ascii. Integer math on it is illegal and makes no sense.

Lets take a 128bit number, 0x3EB8E79140631092131F93905E8CB80E, 32 bytes to print in HEX ASCII. 16 bytes in binary. 38 bytes in decimal ASCII.

Since inode comparison is rare, I was thinking either warn or die if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be returning >2^53 inodes all the time that makes PP stat unusable with continuous warnings and dies. Another choice is, if it overflows, the inode is simply 0, an invalid (is inode 0 legal per posix?) constant that any human should know something is wrong.

If we’re going to go this route it would be nice to only force code that
actually looks at the inode to care, e.g. attaching magic to the scalar
that dies on access, so unaffected code doesn’t have to care.

Attaching magic (upgrade to SVMG, allocate a MG body, fill it in, etc) is expensive unless there is an immortable global SV* that is "fetch forbidden" and set up once per perl process.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Oct 27, 2015

From @Abigail

On Tue, Oct 27, 2015 at 09​:51​:44AM -0700, bulk88 via RT wrote​:

Since inode comparison is rare, I was thinking either warn or die
if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs
are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be
returning >2^53 inodes all the time that makes PP stat unusable with
continuous warnings and dies. Another choice is, if it overflows, the
inode is simply 0, an invalid (is inode 0 legal per posix?) constant
that any human should know something is wrong.

From my hazy memory, 0 isn't illegal, but no file system [1] actually
uses 0 for a real inode. 0 is used to signal errors, or for special
cases. A directory entry for instance, may record inode 0 for a deleted
file, and a readdir() will skip over entries with inode 0.

Abigail
--
[1] As soon as you say 'no', someone will come up with a counter example.

@p5pRT
Copy link
Author

p5pRT commented Oct 28, 2015

From @ap

* bulk88 via RT <perlbug-followup@​perl.org> [2015-10-27 17​:55]​:

Although I am hesitant about returning PVs and "eq" instead of "=="
for perf reasons, that might be the only sane way to support 128 bit
inodes. If an inode number is really a string, maybe it should be raw
4/8/16 bytes of binary, not printable ascii. Integer math on it is
illegal and makes no sense.

Lets take a 128bit number, 0x3EB8E79140631092131F93905E8CB80E, 32
bytes to print in HEX ASCII. 16 bytes in binary. 38 bytes in decimal
ASCII.

Since inode comparison is rare,

Yes. It is rare. And since eq does its level best to short-circuit, the
difference will only matter when the inode numbers are equal length and
share a long prefix. How likely is that really? Also don’t forget that
“have I seen this before?” in practice means “let me look if this inode
number is a key in this hash over here”… which is string-based rather
than == anyway.

I proposed conversion to decimal for two reasons​:

1. You can leave non-overflowing inode numbers as IVs, so the cost is at
  least not paid at stat time (except on overflow). That way callers
  who don’t care about the inode can avoid paying for it. Inasmuch as
  there is noteworthy cost, surely this is the more important factor.

2. Usability basically. Say you want to print the inode number in some
  debug output. Or you want to log it. Having to fiddle a conversion
  into the code somewhere to get something more readable than raw bytes
  is a pain.

Converting to hex would satisfy the usability consideration but unless
you are willing to say “sometimes you get a number and sometimes a hex
string” then you must *always* do it, i.e. the usually-IV optimisation
becomes impossible and every stat call everywhere gets penalised.

Basically converting overflowing values to decimal strings seems to be
as close as we can probably get this interface to “user cannot use it
wrong”, and without being relevantly costly. All other options clearly
require users to be much smarter and/or more careful, and to write more
code, which makes this the obviously better option. Unless I’ve missed
some clunker of a problem with it; have I?

I was thinking either warn or die if it is > 2^53 (overflows NV). Even
if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs.
Someone with ZFS or whatever might be returning >2^53 inodes all the
time that makes PP stat unusable with continuous warnings and dies.

I’m not sure I follow here; first you seem to state a preference for
warning/dieing, then pointing out that on some systems that would suck;
but, I’m not sure what you are concluding…?

Another choice is, if it overflows, the inode is simply 0, an invalid
(is inode 0 legal per posix?) constant that any human should know
something is wrong.

I guess, but… that has the same problem as dying​: it makes stat useless
on affected systems. Why do that, when converting to a string will make
it easy to write code that works correctly now and forever?

--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

@p5pRT
Copy link
Author

p5pRT commented Feb 5, 2016

From @bulk88

On Tue Oct 20 18​:46​:47 2015, bulk88 wrote​:

This is a bug report for perl from bulk88@​hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.23.4.

-----------------------------------------------------------------
[Please describe your issue here]

While I was thinking about implementing real inode info in Win32's PP
stat()[1], I discovered that Perl rounds the inode integer.

Since the code freezes begun/coming, I wrote a patch with the most conservative solution (instead of "eq" instead of "==" solutions like PVs of hex strings or raw bytes or PVs of decimal integers) so this ticket can be closed. This patch that I attached is part of a near future win32_stat refactor patch that I split up.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Feb 5, 2016

From @bulk88

0001-perl-126414-add-warning-if-perl-rounds-inode-in-PP-s.patch
From e9dda1b0e3c7c0721d983604d685a6362a39f3f6 Mon Sep 17 00:00:00 2001
From: Daniel Dragan <bulk88@hotmail.com>
Date: Fri, 5 Feb 2016 02:43:36 -0500
Subject: [PATCH] [perl #126414] add warning if perl rounds inode in PP stat

Do the most conservative approach to fixing #126414, warn if rounding
occurred. Since on some CCs/platforms integer to FP casting is a
internally a function call, only do it once. See also #84590 and
commit 8d8cba8868 .
---
 pod/perldiag.pod |  7 +++++++
 pp_sys.c         | 13 ++++++++++++-
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/pod/perldiag.pod b/pod/perldiag.pod
index ae061f9..d22dc66 100644
--- a/pod/perldiag.pod
+++ b/pod/perldiag.pod
@@ -2733,6 +2733,13 @@ transparently promotes all numbers to a floating point representation
 internally--subject to loss of precision errors in subsequent
 operations.
 
+=item Integer overflow in stat inode
+
+(S overflow) Your system's inode type is larger than perl can represent
+on your system's archtecture, and this particular inode value returned by
+your system's stat was stored by perl as a floating pointer number with a loss
+of precision.
+
 =item Integer overflow in srand
 
 (S overflow) The number you have passed to srand is too big to fit
diff --git a/pp_sys.c b/pp_sys.c
index 15b4d8b..a7d99d1 100644
--- a/pp_sys.c
+++ b/pp_sys.c
@@ -2965,11 +2965,22 @@ PP(pp_stat)
 	RETURN;
     }
     if (max) {
+#if ST_INO_SIZE > IVSIZE
+	NV ino;
+#endif
 	EXTEND(SP, max);
 	EXTEND_MORTAL(max);
 	mPUSHi(PL_statcache.st_dev);
 #if ST_INO_SIZE > IVSIZE
-	mPUSHn(PL_statcache.st_ino);
+	ino = (NV)PL_statcache.st_ino;
+	if(
+#   if ST_INO_SIGN <= 0
+	  ino < -NV_OVERFLOWS_INTEGERS_AT ||
+#   endif
+	  ino > NV_OVERFLOWS_INTEGERS_AT)
+	    Perl_ck_warner_d(aTHX_ packWARN(WARN_OVERFLOW),
+			     "Integer overflow in stat inode");
+	mPUSHn(ino);
 #else
 #   if ST_INO_SIGN <= 0
 	mPUSHi(PL_statcache.st_ino);
-- 
1.9.5.msysgit.1

@p5pRT
Copy link
Author

p5pRT commented Feb 10, 2016

From @bulk88

On Thu Feb 04 23​:49​:50 2016, bulk88 wrote​:

Since the code freezes begun/coming, I wrote a patch with the most
conservative solution (instead of "eq" instead of "==" solutions like
PVs of hex strings or raw bytes or PVs of decimal integers) so this
ticket can be closed. This patch that I attached is part of a near
future win32_stat refactor patch that I split up.

Actually, I'm not sure if this patch should be applied before 5.24. In a separate patch, I added inode support to Win32 Perl, and it seems that NTFS inode numbers are a 64 bit bitfield/struct, not linear 64 bit counter, and therefore always > 2^53 and every PP stat() is generating an overflow warning on my 32 bit IV perl. Not sure how to proceed now.

Example output from a modified perl


Integer overflow in stat inode hi 00320000 lo 000451e6 fp 14073748835815910.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00390000 lo 000451d8 fp 16044073672790488.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 005b0000 lo 00044a00 fp 25614222880950784.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00440000 lo 00044bf3 fp 19140298416606196.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00240000 lo 00044a42 fp 10133099161864770.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00710000 lo 00044bde fp 31806672368585696.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00500000 lo 00044bf6 fp 22517998137134072.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00260000 lo 00045229 fp 10696049115288104.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002c0000 lo 000452c2 fp 12384898975552194.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00450000 lo 000452cc fp 19421773393318604.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00280000 lo 000452ce fp 11258999068709582.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00260000 lo 000452df fp 10696049115288288.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002c0000 lo 000452e1 fp 12384898975552224.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003f0000 lo 0004551f fp 17732923533055264.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 008a0000 lo 00055510 fp 38843546786419984.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00410000 lo 00048484 fp 18295873486488708.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002e0000 lo 00066d48 fp 12947848929111368.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00520000 lo 000488e0 fp 23080948090570976.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00350000 lo 00021c9c fp 14918173765803164.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00440000 lo 00045511 fp 19140298416608528.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003e0000 lo 0004550c fp 17451448556344588.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00460000 lo 00045509 fp 19703248370029832.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00330000 lo 00055a70 fp 14355223812594288.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00330000 lo 00021ca4 fp 14355223812381860.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003c0000 lo 0004522e fp 16888498602922542.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002f0000 lo 00045193 fp 13229323905683860.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002a0000 lo 000451be fp 11821949022130622.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00230000 lo 000451c3 fp 9851624185156036.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00250000 lo 000451c2 fp 10414574138577346.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00260000 lo 000451c8 fp 10696049115288008.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003b0000 lo 00066d1f fp 16607023626349856.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002b0000 lo 0000af09 fp 12103423998603016.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00220000 lo 00045213 fp 9570149208445460.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002b0000 lo 000452d3 fp 12103423998841556.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00270000 lo 0004535b fp 10977524091999068.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00200000 lo 0004549a fp 9007199255024794.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00570000 lo 000554f2 fp 24488322974176496.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00390000 lo 000554f5 fp 16044073672856820.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00420000 lo 000554f4 fp 18577348463252724.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00480000 lo 000554f7 fp 20266198323516664.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00380000 lo 0004579f fp 15762598696081312.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 007b0000 lo 0001e0f2 fp 34621422135533808.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002b0000 lo 00055530 fp 12103423998907696.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00250000 lo 00055533 fp 10414574138643764.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00270000 lo 00055532 fp 10977524092065074.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003f0000 lo 00047dbb fp 17732923533065660.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00230000 lo 00055556 fp 9851624185222486.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00200000 lo 00055558 fp 9007199255090520.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003c0000 lo 00048508 fp 16888498602935560.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00530000 lo 00048871 fp 23362423067281520.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00350000 lo 0004890f fp 14918173765962000.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00310000 lo 000489e0 fp 13792273859119584.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003e0000 lo 00048d3c fp 17451448556358972.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 003e0000 lo 00059702 fp 17451448556427010.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 009a0000 lo 00059846 fp 43347146413807688.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00340000 lo 00059738 fp 14636698789320504.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00b30000 lo 00059a5b fp 50384020831574616.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002f0000 lo 000546a3 fp 13229323905746596.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00ae0000 lo 0005a016 fp 48976645948022808.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00ab0000 lo 0005a105 fp 48132221017891080.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00280000 lo 00063723 fp 11258999068833572.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00210000 lo 00063735 fp 9288674231858996.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00270000 lo 0006372f fp 10977524092122928.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00200000 lo 0004508e fp 9007199255023758.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002c0000 lo 0004507c fp 12384898975551612.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00270000 lo 00066ceb fp 10977524092136684.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00210000 lo 00066cee fp 9288674231872750.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00220000 lo 00066ced fp 9570149208583404.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00200000 lo 00066cf2 fp 9007199255162098.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00310000 lo 000451ea fp 13792273859105258.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00250000 lo 00047ab4 fp 10414574138587828.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00360000 lo 000452b4 fp 15199648742658740.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00250000 lo 00044c8c fp 10414574138576012.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00230000 lo 00044cb1 fp 9851624185154736.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00270000 lo 00044cd2 fp 10977524091997394.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 004d0000 lo 00044bff fp 21673573207002112.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00360000 lo 00044cde fp 15199648742657246.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00270000 lo 00044ce4 fp 10977524091997412.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00370000 lo 00044c21 fp 15481123719367712.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 002e0000 lo 00044ce5 fp 12947848928972004.0000
00 at C​:/p523/src/lib/File/Find.pm line 435.
Integer overflow in stat inode hi 00230000 lo 00044cf2 fp 9851624185154802.00000
0 at C​:/p523/src/lib/File/Find.pm line 435.


--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Feb 11, 2016

From @ap

* bulk88 via RT <perlbug-followup@​perl.org> [2016-02-10 02​:30]​:

It seems that NTFS inode numbers are a 64 bit bitfield/struct, not
linear 64 bit counter, and therefore always > 2^53 and every PP stat()
is generating an overflow warning on my 32 bit IV perl.

Which implies that the inode field has always been broken. Not bad!

I must wonder how come nobody ran into this before, then. An impressive
feat of “works by accident” – but how?

Presumably by way of the rounding being deterministic. Possibly due to
the form of the bit struct aligning with the deterministic rounding in
such a way that it mitigates the problem in practical scenarios.

Not sure how to proceed now.

Test that hypothesis. Run a recursive walk over your FSs and record the
inodes and paths and check to see if you get false duplicates based on
comparing inode values. If not, then the urgency to solve this goes way
down and also the importance of solving it non-disruptively goes way up.
(Although it still ought to be solved, either way.)

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

@p5pRT
Copy link
Author

p5pRT commented Feb 12, 2016

From @bulk88

On Thu Feb 11 15​:55​:50 2016, aristotle wrote​:

* bulk88 via RT <perlbug-followup@​perl.org> [2016-02-10 02​:30]​:

It seems that NTFS inode numbers are a 64 bit bitfield/struct, not
linear 64 bit counter, and therefore always > 2^53 and every PP stat()
is generating an overflow warning on my 32 bit IV perl.

Which implies that the inode field has always been broken. Not bad!

I must wonder how come nobody ran into this before, then. An impressive
feat of “works by accident” – but how?

The windows libc ALWAYS sets st_ino to 0. Blead win32 perl's st_ino is always 0 ( there is an old feature request in https​://rt.perl.org/Ticket/Display.html?id=65052 asking to fix that I attempted to fix). I created an unpublished/yet to publish patch that gets the inode ("FileIndex" or "IndexNumber" https://msdn.microsoft.com/en-us/library/windows/hardware/ff540318%28v=vs.85%29.aspx ) from the MS Win32 API, since the MS libc API doesn't supply it and presents it on a Perl C level to upper Perl C layers. That is how I got inodes visible on a PP level in win32 perl and was able to test this overflow/rounding patch.

Presumably by way of the rounding being deterministic. Possibly due to
the form of the bit struct aligning with the deterministic rounding in
such a way that it mitigates the problem in practical scenarios.

The chances of a collision seem low, but not theoretically impossible (only comparing all 64 bits would be collision free). Since Im not a CS PHD, I wont try to estimate the chance. Any porters can analyze my raw data that i posted and guess the chance. The highest 16 bit short seems to be a bucket or checksum, while the low 32 bits seem to be an offset into an array. That raw data is probably from iterating over my perl git source code root dir and some of the offsets seem very close.

Not sure how to proceed now.

Test that hypothesis. Run a recursive walk over your FSs and record the
inodes and paths and check to see if you get false duplicates based on
comparing inode values. If not, then the urgency to solve this goes way
down and also the importance of solving it non-disruptively goes way up.
(Although it still ought to be solved, either way.)

Iterate over the entire FS, make a hash with the key being the FP and value 1 or ++. At the end of interating over the entire FS (or as much before 32 bit process mem limit hits), go over the hash, if a key has a value >1, either I have hard links on Win32 or rounding collision. Is this the correct algorithm?

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From zefram@fysh.org

Fixed in commit 2e8ea15. stat() now
returns a string of decimal digits if the inode number isn't representable
as a native integer.

-zefram

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From @shlomif

On Sat, 11 Nov 2017 07​:50​:52 +0000
Zefram <zefram@​fysh.org> wrote​:

Fixed in commit 2e8ea15. stat() now
returns a string of decimal digits if the inode number isn't representable
as a native integer.

-zefram

Thanks, Zefram!

--


Shlomi Fish http​://www.shlomifish.org/
UNIX Fortune Cookies - http​://www.shlomifish.org/humour/fortunes/

XSLT is the number one cause of programmers’ suicides since Visual Basic 1.0.
  — http​://www.shlomifish.org/humour/bits/facts/XSLT/

Please reply to list if it's a mailing list post - http​://shlom.in/reply .

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From @dur-randir

On Fri, 10 Nov 2017 23​:51​:30 -0800, zefram@​fysh.org wrote​:

Fixed in commit 2e8ea15. stat() now
returns a string of decimal digits if the inode number isn't representable
as a native integer.

This commit introduced a new build warning on my machine (64 bit Mac OS X)​:

pp_sys.c​:3014​:32​: warning​: comparison of unsigned expression < 0 is always false [-Wtautological-compare]
  neg = PL_statcache.st_ino < 0;
  ~~~~~~~~~~~~~~~~~~~ ^ ~

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From @jkeenan

On Sat, 11 Nov 2017 15​:27​:16 GMT, randir wrote​:

On Fri, 10 Nov 2017 23​:51​:30 -0800, zefram@​fysh.org wrote​:

Fixed in commit 2e8ea15. stat() now
returns a string of decimal digits if the inode number isn't
representable
as a native integer.

This commit introduced a new build warning on my machine (64 bit Mac
OS X)​:

pp_sys.c​:3014​:32​: warning​: comparison of unsigned expression < 0 is
always false [-Wtautological-compare]
neg = PL_statcache.st_ino < 0;
~~~~~~~~~~~~~~~~~~~ ^ ~

Can you attach the output of 'perl -V' from this build?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From @dur-randir

On Sat, 11 Nov 2017 08​:22​:22 -0800, jkeenan wrote​:

Can you attach the output of 'perl -V' from this build?

Summary of my perl5 (revision 5 version 27 subversion 6) configuration​:
  Commit id​: 7d65f65
  Platform​:
  osname=darwin
  osvers=13.4.0
  archname=darwin-thread-multi-2level
  uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0​: mon jan 11 18​:17​:34 pst 2016; root​:xnu-2422.115.15~1release_x86_64 x86_64 '
  config_args='-de -Dusedevel -DDEBUGGING -Duseithreads'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=define
  usemultiplicity=define
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='cc'
  ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -DPERL_USE_SAFE_PUTENV'
  optimize='-O3 -g'
  cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion=''
  gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='cc'
  ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'
  libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib /usr/local/lib /usr/lib
  libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
  perllibs=-lpthread -ldl -lm -lutil -lc
  libc=
  so=dylib
  useshrplib=false
  libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=bundle
  d_dlsymun=undef
  ccdlflags=' '
  cccdlflags=' '
  lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  DEBUGGING
  HAS_TIMES
  MULTIPLICITY
  PERLIO_LAYERS
  PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_IMPLICIT_CONTEXT
  PERL_MALLOC_WRAP
  PERL_OP_PARENT
  PERL_PRESERVE_IVUV
  PERL_TRACK_MEMPOOL
  PERL_USE_DEVEL
  PERL_USE_SAFE_PUTENV
  USE_64_BIT_ALL
  USE_64_BIT_INT
  USE_ITHREADS
  USE_LARGE_FILES
  USE_LOCALE
  USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC
  USE_LOCALE_TIME
  USE_PERLIO
  USE_PERL_ATOF
  USE_REENTRANT_API
  Built under darwin
  Compiled at Nov 11 2017 18​:30​:42
  %ENV​:
  PERLBREW_BASHRC_VERSION="0.78"
  PERLBREW_HOME="/Users/dur-randir/.perlbrew"
  PERLBREW_MANPATH="/Users/dur-randir/perlbrew/perls/perl-5.22.1/man"
  PERLBREW_PATH="/Users/dur-randir/perlbrew/bin​:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin"
  PERLBREW_PERL="perl-5.22.1"
  PERLBREW_ROOT="/Users/dur-randir/perlbrew"
  PERLBREW_VERSION="0.78"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.27.6/darwin-thread-multi-2level
  /usr/local/lib/perl5/site_perl/5.27.6
  /usr/local/lib/perl5/5.27.6/darwin-thread-multi-2level
  /usr/local/lib/perl5/5.27.6

@p5pRT
Copy link
Author

p5pRT commented Nov 11, 2017

From zefram@fysh.org

Sergey Aleynikov via RT wrote​:

This commit introduced a new build warning on my machine (64 bit Mac OS X)​:

Thanks. That should be fixed by commit
b889707.

-zefram

@p5pRT
Copy link
Author

p5pRT commented Nov 12, 2017

From @dur-randir

On Sat, 11 Nov 2017 15​:27​:55 -0800, zefram@​fysh.org wrote​:

Thanks. That should be fixed by commit
b889707.

Yes, this fixed it, thanks.

@p5pRT
Copy link
Author

p5pRT commented Nov 12, 2017

@cpansprout - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

p5pRT commented Jun 23, 2018

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0, this and 185 other issues have been
resolved.

Perl 5.28.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Author

p5pRT commented Jun 23, 2018

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant