Skip Menu |
Report information
Id: 126414
Status: pending release
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: bulk88 <bulk88 [at] hotmail.com>
Cc:
AdminCc:

Operating System: mswin32
PatchStatus: (no value)
Severity: low
Type: core
Perl Version: 5.23.4
Fixed In: (no value)

Attachments
0001-perl-126414-add-warning-if-perl-rounds-inode-in-PP-s.patch



From: bulk 88 <bulk88 [...] hotmail.com>
To: "perlbug [...] perl.org" <perlbug [...] perl.org>
Subject: perl rounds inode in PP stat
Date: Tue, 20 Oct 2015 21:46:29 -0400
Download (untitled) / with headers
text/plain 5.4k
This is a bug report for perl from bulk88@hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.23.4.


-----------------------------------------------------------------
[Please describe your issue here]

While I was thinking about implementing real inode info in Win32's PP stat()[1], I discovered that Perl rounds the inode integer.

http://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file
-compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on 32b IV perl, it has the potential of being rounded if its > 2^53 and therefore is garbage/uninitialized. In a DB on FS scheme, bad things could happen if 2 files that aren't links in reality, "==" in perl as to being the same file.

An inode can be implemented in a couple ways by any FS.

-a 0 based offset into an array of something (offsets or structs) on the disk
-an absolute sector (units of 512 bytes) or byte position into the FS partition
-random looking hash number
-XOR against a secret of any of the 3 above

[1] https://rt.perl.org/Public/Bug/Display.html?id=65052
The last 2 implementations would give very frequent high bits that are > 2^53. Worst case scenario for 0 based inode FS is 2^53 bytes, which is 9007 TB of storage, which I dont think anyone would use 32 bit perl on a enterprise SAN/cluster, but that doesn't address the theoretical hash/xor/checksum based inode FSes.

Everyone agrees storing 64 bit C pointers in 64 bit doubles is forbidden, so why is perl storing inodes in NVs/doubles? Can something be done about this? Fatally error if its over 2^32 or over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and let the user in PP figure out what to do with it (Math::Int64 it)?

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=low
---
Site configuration information for perl 5.23.4:

Configured by Owner at Fri Oct 16 16:33:21 2015.

Summary of my perl5 (revision 5 version 23 subversion 4) configuration:
  Derived from: c8b87727e0a3605fae65ba4f0811d649667abbf0
  Ancestor: 9b331ac6cfda2819646f52e55c4478f2f4123a3d
  Platform:
    osname=MSWin32, osvers=6.1, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32 -D_CONSOLE -DNO_STRICT -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS',
    optimize='-O1 -MD -Zi -DNDEBUG -GL',
    cppflags='-DWIN32'
    ccversion='18.00.31101', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234, doublekind=3
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p523\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'" -subsystem:console,"5.01"'
    libpth=\lib
    libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl523.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpath:"c:\p523\lib\CORE"  -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'" -subsystem:console,"5.01"'

Locally applied patches:
    uncommitted-changes
    c8b87727e0a3605fae65ba4f0811d649667abbf0

---
@INC for perl 5.23.4:
    C:/p523/site/lib
    C:/p523/lib
    .

---
Environment for perl 5.23.4:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\p523\site\bin;C:\p523\bin;C:\Program Files\ActiveState Komodo Edit 9\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\TortoiseGit\bin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\Program Files\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SDKs\TypeScript\1.0\;
    PERL_BADLANG (unset)
    SHELL (unset)

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.3k
On Tue Oct 20 18:46:47 2015, bulk88 wrote: Show quoted text
> This is a bug report for perl from bulk88@hotmail.com, > generated with the help of perlbug 1.40 running under perl 5.23.4. > > > ----------------------------------------------------------------- > [Please describe your issue here] > > While I was thinking about implementing real inode info in Win32's PP > stat()[1], I discovered that Perl rounds the inode integer. > > http://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e > > associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590 > > An inode has 2 uses AFAIK. > > -undelete a file > -compare hard if they are the same file > > An inode is effectively an opaque pointer (integer) into a FS. If the > OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on > 32b IV perl, it has the potential of being rounded if its > 2^53 and
Actually, MS bumped the inode size to 128 bits in Server 2012 OS for their nextgen enterpriseish FS. So for discussion, assume 128 bit inodes are real and have to be supported (personally Win32 Perl can stick with legacy 64 bit inodes for some years). https://msdn.microsoft.com/en-us/library/windows/desktop/hh802691%28v=vs.85%29.aspx See also https://msdn.microsoft.com/en-us/library/windows/desktop/aa363788%28v=vs.85%29.aspx -- bulk88 ~ bulk88 at hotmail.com
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 3.1k
On Tue Oct 20 18:46:47 2015, bulk88 wrote: Show quoted text
> While I was thinking about implementing real inode info in Win32's PP > stat()[1], I discovered that Perl rounds the inode integer. > > http://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e > > associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590 > > An inode has 2 uses AFAIK. > > -undelete a file > -compare hard if they are the same file > > An inode is effectively an opaque pointer (integer) into a FS. If the > OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on > 32b IV perl, it has the potential of being rounded if its > 2^53 and > therefore is garbage/uninitialized. In a DB on FS scheme, bad things > could happen if 2 files that aren't links in reality, "==" in perl as > to being the same file. > > An inode can be implemented in a couple ways by any FS. > > -a 0 based offset into an array of something (offsets or structs) on > the disk > -an absolute sector (units of 512 bytes) or byte position into the FS > partition > -random looking hash number > -XOR against a secret of any of the 3 above > > [1] https://rt.perl.org/Public/Bug/Display.html?id=65052 > The last 2 implementations would give very frequent high bits that are
> > 2^53. Worst case scenario for 0 based inode FS is 2^53 bytes, which
> is 9007 TB of storage, which I dont think anyone would use 32 bit perl > on a enterprise SAN/cluster, but that doesn't address the theoretical > hash/xor/checksum based inode FSes. > > Everyone agrees storing 64 bit C pointers in 64 bit doubles is > forbidden, so why is perl storing inodes in NVs/doubles? Can something > be done about this? Fatally error if its over 2^32 or over 2^53? Store > the 64 bit integer as a SVPV in printable ASCII and let the user in PP > figure out what to do with it (Math::Int64 it)?
I can see us producing some sort of error if the inode number changes value when stored as a NV, perhaps with an option to disable that error, since stat() isn't only used to fetch a file's inode number. As to how to behave when the inode number doesn't fit, we can look at existing implementations, from the Solaris stat() man page: The stat(), fstat(), and lstat() functions may fail if: EOVERFLOW One of the members is too large to store in the stat structure pointed to by buf. Similarly on FreeBSD: [EOVERFLOW] The file size in bytes cannot be represented correctly in the structure pointed to by sb. If we follow the lead of Solaris/FreeBSD, perl's stat() would simply fail with EOVERFLOW (if available) if the inode number doesn't fit. As to optionally disabling the error, we could use a variable like ${^WIN32_SLOPPY_STAT}, perhaps ${^SLOPPY_STAT_INO}, or something lexically scoped, to keep the sloppy behaviour restricted to a small amount of code. Sort of related: systems with large files can run into this issue for st_size: $ ./perl -le 'print +(stat "/home/tony/somefile.txt")[7]' 9.00819925474099e+16 $ ls -l ~/somefile.txt -rw-r--r-- 1 tony tony 90081992547409912 Oct 27 03:43 /home/tony/somefile.txt (that's a sparse file on ZFS) Tony
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
To: perl5-porters [...] perl.org
Subject: Re: [perl #126414] perl rounds inode in PP stat
Date: Tue, 27 Oct 2015 08:54:04 +0100
Download (untitled) / with headers
text/plain 1.7k
* bulk88 <perlbug-followup@perl.org> [2015-10-21 03:50]: Show quoted text
> An inode is effectively an opaque pointer (integer) into a FS.
Like a zip code. That is also nominally a number but is meaningless to do math on, and the only meaningful operation is testing equality. Show quoted text
> Can something be done about this? Fatally error if its over 2^32 or > over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and > let the user in PP figure out what to do with it (Math::Int64 it)?
If you mean converting the inode number 9876543210123456 to the string '9876543210123456' (maybe only on overflow), then yes I would favour that (along with updates to the docs telling people to compare inode using eq and not ==). That way it would transparently Just Work, unlike throwing an error. The only question I’m unclear on is what this would do to XS code. * Tony Cook via RT <perlbug-followup@perl.org> [2015-10-27 04:50]: Show quoted text
> I can see us producing some sort of error if the inode number changes > value when stored as a NV, perhaps with an option to disable that > error, since stat() isn't only used to fetch a file's inode number.
That would force all code not written for a controlled environment to care about this every time it calls stat. Even more irritating, you have to declare your *dis*interest in the inode number, which is by far the more common case in my experience. (Most of the time I stat I want the mtime, next most commonly probably the size. I’ve needed the inode too but only very occasionally.) If we’re going to go this route it would be nice to only force code that actually looks at the inode to care, e.g. attaching magic to the scalar that dies on access, so unaffected code doesn’t have to care. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.2k
On Tue Oct 27 00:54:49 2015, aristotle wrote: Show quoted text
> * bulk88 <perlbug-followup@perl.org> [2015-10-21 03:50]:
> > An inode is effectively an opaque pointer (integer) into a FS.
> > Like a zip code. That is also nominally a number but is meaningless > to do math on, and the only meaningful operation is testing equality. >
> > Can something be done about this? Fatally error if its over 2^32 or > > over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and > > let the user in PP figure out what to do with it (Math::Int64 it)?
> > If you mean converting the inode number 9876543210123456 to the string > '9876543210123456' (maybe only on overflow), then yes I would favour > that (along with updates to the docs telling people to compare inode > using eq and not ==). That way it would transparently Just Work, unlike > throwing an error. The only question I’m unclear on is what this would > do to XS code.
Although I am hesitant about returning PVs and "eq" instead of "==" for perf reasons, that might be the only sane way to support 128 bit inodes. If an inode number is really a string, maybe it should be raw 4/8/16 bytes of binary, not printable ascii. Integer math on it is illegal and makes no sense. Lets take a 128bit number, 0x3EB8E79140631092131F93905E8CB80E, 32 bytes to print in HEX ASCII. 16 bytes in binary. 38 bytes in decimal ASCII. Since inode comparison is rare, I was thinking either warn or die if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be returning >2^53 inodes all the time that makes PP stat unusable with continuous warnings and dies. Another choice is, if it overflows, the inode is simply 0, an invalid (is inode 0 legal per posix?) constant that any human should know something is wrong. Show quoted text
> If we’re going to go this route it would be nice to only force code that > actually looks at the inode to care, e.g. attaching magic to the scalar > that dies on access, so unaffected code doesn’t have to care.
Attaching magic (upgrade to SVMG, allocate a MG body, fill it in, etc) is expensive unless there is an immortable global SV* that is "fetch forbidden" and set up once per perl process. -- bulk88 ~ bulk88 at hotmail.com
Subject: Re: [perl #126414] perl rounds inode in PP stat
Date: Tue, 27 Oct 2015 18:11:50 +0100
CC: "OtherRecipients of perl Ticket #126414": ;, perl5-porters [...] perl.org
To: bulk88 via RT <perlbug-followup [...] perl.org>
From: Abigail <abigail [...] abigail.be>
Download (untitled) / with headers
text/plain 901b
On Tue, Oct 27, 2015 at 09:51:44AM -0700, bulk88 via RT wrote: Show quoted text
> > Since inode comparison is rare, I was thinking either warn or die > if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs > are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be > returning >2^53 inodes all the time that makes PP stat unusable with > continuous warnings and dies. Another choice is, if it overflows, the > inode is simply 0, an invalid (is inode 0 legal per posix?) constant > that any human should know something is wrong.
From my hazy memory, 0 isn't illegal, but no file system [1] actually uses 0 for a real inode. 0 is used to signal errors, or for special cases. A directory entry for instance, may record inode 0 for a deleted file, and a readdir() will skip over entries with inode 0. Abigail -- [1] As soon as you say 'no', someone will come up with a counter example.
Subject: Re: [perl #126414] perl rounds inode in PP stat
Date: Wed, 28 Oct 2015 01:48:56 +0100
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
To: perl5-porters [...] perl.org
* bulk88 via RT <perlbug-followup@perl.org> [2015-10-27 17:55]: Show quoted text
> Although I am hesitant about returning PVs and "eq" instead of "==" > for perf reasons, that might be the only sane way to support 128 bit > inodes. If an inode number is really a string, maybe it should be raw > 4/8/16 bytes of binary, not printable ascii. Integer math on it is > illegal and makes no sense. > > Lets take a 128bit number, 0x3EB8E79140631092131F93905E8CB80E, 32 > bytes to print in HEX ASCII. 16 bytes in binary. 38 bytes in decimal > ASCII. > > Since inode comparison is rare,
Yes. It is rare. And since eq does its level best to short-circuit, the difference will only matter when the inode numbers are equal length and share a long prefix. How likely is that really? Also don’t forget that “have I seen this before?” in practice means “let me look if this inode number is a key in this hash over here”… which is string-based rather than == anyway. I proposed conversion to decimal for two reasons: 1. You can leave non-overflowing inode numbers as IVs, so the cost is at least not paid at stat time (except on overflow). That way callers who don’t care about the inode can avoid paying for it. Inasmuch as there is noteworthy cost, surely this is the more important factor. 2. Usability basically. Say you want to print the inode number in some debug output. Or you want to log it. Having to fiddle a conversion into the code somewhere to get something more readable than raw bytes is a pain. Converting to hex would satisfy the usability consideration but unless you are willing to say “sometimes you get a number and sometimes a hex string” then you must *always* do it, i.e. the usually-IV optimisation becomes impossible and every stat call everywhere gets penalised. Basically converting overflowing values to decimal strings seems to be as close as we can probably get this interface to “user cannot use it wrong”, and without being relevantly costly. All other options clearly require users to be much smarter and/or more careful, and to write more code, which makes this the obviously better option. Unless I’ve missed some clunker of a problem with it; have I? Show quoted text
> I was thinking either warn or die if it is > 2^53 (overflows NV). Even > if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs. > Someone with ZFS or whatever might be returning >2^53 inodes all the > time that makes PP stat unusable with continuous warnings and dies.
I’m not sure I follow here; first you seem to state a preference for warning/dieing, then pointing out that on some systems that would suck; but, I’m not sure what you are concluding…? Show quoted text
> Another choice is, if it overflows, the inode is simply 0, an invalid > (is inode 0 legal per posix?) constant that any human should know > something is wrong.
I guess, but… that has the same problem as dying: it makes stat useless on affected systems. Why do that, when converting to a string will make it easy to write code that works correctly now and forever? -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: [PATCH] perl rounds inode in PP stat
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 774b
On Tue Oct 20 18:46:47 2015, bulk88 wrote: Show quoted text
> This is a bug report for perl from bulk88@hotmail.com, > generated with the help of perlbug 1.40 running under perl 5.23.4. > > > ----------------------------------------------------------------- > [Please describe your issue here] > > While I was thinking about implementing real inode info in Win32's PP > stat()[1], I discovered that Perl rounds the inode integer.
Since the code freezes begun/coming, I wrote a patch with the most conservative solution (instead of "eq" instead of "==" solutions like PVs of hex strings or raw bytes or PVs of decimal integers) so this ticket can be closed. This patch that I attached is part of a near future win32_stat refactor patch that I split up. -- bulk88 ~ bulk88 at hotmail.com
Subject: 0001-perl-126414-add-warning-if-perl-rounds-inode-in-PP-s.patch
From e9dda1b0e3c7c0721d983604d685a6362a39f3f6 Mon Sep 17 00:00:00 2001 From: Daniel Dragan <bulk88@hotmail.com> Date: Fri, 5 Feb 2016 02:43:36 -0500 Subject: [PATCH] [perl #126414] add warning if perl rounds inode in PP stat Do the most conservative approach to fixing #126414, warn if rounding occurred. Since on some CCs/platforms integer to FP casting is a internally a function call, only do it once. See also #84590 and commit 8d8cba8868 . --- pod/perldiag.pod | 7 +++++++ pp_sys.c | 13 ++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/pod/perldiag.pod b/pod/perldiag.pod index ae061f9..d22dc66 100644 --- a/pod/perldiag.pod +++ b/pod/perldiag.pod @@ -2733,6 +2733,13 @@ transparently promotes all numbers to a floating point representation internally--subject to loss of precision errors in subsequent operations. +=item Integer overflow in stat inode + +(S overflow) Your system's inode type is larger than perl can represent +on your system's archtecture, and this particular inode value returned by +your system's stat was stored by perl as a floating pointer number with a loss +of precision. + =item Integer overflow in srand (S overflow) The number you have passed to srand is too big to fit diff --git a/pp_sys.c b/pp_sys.c index 15b4d8b..a7d99d1 100644 --- a/pp_sys.c +++ b/pp_sys.c @@ -2965,11 +2965,22 @@ PP(pp_stat) RETURN; } if (max) { +#if ST_INO_SIZE > IVSIZE + NV ino; +#endif EXTEND(SP, max); EXTEND_MORTAL(max); mPUSHi(PL_statcache.st_dev); #if ST_INO_SIZE > IVSIZE - mPUSHn(PL_statcache.st_ino); + ino = (NV)PL_statcache.st_ino; + if( +# if ST_INO_SIGN <= 0 + ino < -NV_OVERFLOWS_INTEGERS_AT || +# endif + ino > NV_OVERFLOWS_INTEGERS_AT) + Perl_ck_warner_d(aTHX_ packWARN(WARN_OVERFLOW), + "Integer overflow in stat inode"); + mPUSHn(ino); #else # if ST_INO_SIGN <= 0 mPUSHi(PL_statcache.st_ino); -- 1.9.5.msysgit.1
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 10.8k
On Thu Feb 04 23:49:50 2016, bulk88 wrote: Show quoted text
> Since the code freezes begun/coming, I wrote a patch with the most > conservative solution (instead of "eq" instead of "==" solutions like > PVs of hex strings or raw bytes or PVs of decimal integers) so this > ticket can be closed. This patch that I attached is part of a near > future win32_stat refactor patch that I split up.
Actually, I'm not sure if this patch should be applied before 5.24. In a separate patch, I added inode support to Win32 Perl, and it seems that NTFS inode numbers are a 64 bit bitfield/struct, not linear 64 bit counter, and therefore always > 2^53 and every PP stat() is generating an overflow warning on my 32 bit IV perl. Not sure how to proceed now. Example output from a modified perl -------------------------------------------- Integer overflow in stat inode hi 00320000 lo 000451e6 fp 14073748835815910.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00390000 lo 000451d8 fp 16044073672790488.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 005b0000 lo 00044a00 fp 25614222880950784.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00440000 lo 00044bf3 fp 19140298416606196.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00240000 lo 00044a42 fp 10133099161864770.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00710000 lo 00044bde fp 31806672368585696.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00500000 lo 00044bf6 fp 22517998137134072.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00260000 lo 00045229 fp 10696049115288104.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002c0000 lo 000452c2 fp 12384898975552194.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00450000 lo 000452cc fp 19421773393318604.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00280000 lo 000452ce fp 11258999068709582.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00260000 lo 000452df fp 10696049115288288.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002c0000 lo 000452e1 fp 12384898975552224.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003f0000 lo 0004551f fp 17732923533055264.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 008a0000 lo 00055510 fp 38843546786419984.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00410000 lo 00048484 fp 18295873486488708.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002e0000 lo 00066d48 fp 12947848929111368.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00520000 lo 000488e0 fp 23080948090570976.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00350000 lo 00021c9c fp 14918173765803164.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00440000 lo 00045511 fp 19140298416608528.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003e0000 lo 0004550c fp 17451448556344588.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00460000 lo 00045509 fp 19703248370029832.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00330000 lo 00055a70 fp 14355223812594288.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00330000 lo 00021ca4 fp 14355223812381860.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003c0000 lo 0004522e fp 16888498602922542.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002f0000 lo 00045193 fp 13229323905683860.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002a0000 lo 000451be fp 11821949022130622.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 000451c3 fp 9851624185156036.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 000451c2 fp 10414574138577346.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00260000 lo 000451c8 fp 10696049115288008.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003b0000 lo 00066d1f fp 16607023626349856.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002b0000 lo 0000af09 fp 12103423998603016.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00220000 lo 00045213 fp 9570149208445460.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002b0000 lo 000452d3 fp 12103423998841556.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 0004535b fp 10977524091999068.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 0004549a fp 9007199255024794.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00570000 lo 000554f2 fp 24488322974176496.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00390000 lo 000554f5 fp 16044073672856820.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00420000 lo 000554f4 fp 18577348463252724.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00480000 lo 000554f7 fp 20266198323516664.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00380000 lo 0004579f fp 15762598696081312.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 007b0000 lo 0001e0f2 fp 34621422135533808.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002b0000 lo 00055530 fp 12103423998907696.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 00055533 fp 10414574138643764.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00055532 fp 10977524092065074.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003f0000 lo 00047dbb fp 17732923533065660.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 00055556 fp 9851624185222486.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 00055558 fp 9007199255090520.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003c0000 lo 00048508 fp 16888498602935560.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00530000 lo 00048871 fp 23362423067281520.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00350000 lo 0004890f fp 14918173765962000.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00310000 lo 000489e0 fp 13792273859119584.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003e0000 lo 00048d3c fp 17451448556358972.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003e0000 lo 00059702 fp 17451448556427010.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 009a0000 lo 00059846 fp 43347146413807688.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00340000 lo 00059738 fp 14636698789320504.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00b30000 lo 00059a5b fp 50384020831574616.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002f0000 lo 000546a3 fp 13229323905746596.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00ae0000 lo 0005a016 fp 48976645948022808.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00ab0000 lo 0005a105 fp 48132221017891080.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00280000 lo 00063723 fp 11258999068833572.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00210000 lo 00063735 fp 9288674231858996.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 0006372f fp 10977524092122928.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 0004508e fp 9007199255023758.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002c0000 lo 0004507c fp 12384898975551612.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00066ceb fp 10977524092136684.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00210000 lo 00066cee fp 9288674231872750.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00220000 lo 00066ced fp 9570149208583404.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 00066cf2 fp 9007199255162098.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00310000 lo 000451ea fp 13792273859105258.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 00047ab4 fp 10414574138587828.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00360000 lo 000452b4 fp 15199648742658740.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 00044c8c fp 10414574138576012.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 00044cb1 fp 9851624185154736.00000 0 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00044cd2 fp 10977524091997394.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 004d0000 lo 00044bff fp 21673573207002112.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00360000 lo 00044cde fp 15199648742657246.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00044ce4 fp 10977524091997412.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00370000 lo 00044c21 fp 15481123719367712.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002e0000 lo 00044ce5 fp 12947848928972004.0000 00 at C:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 00044cf2 fp 9851624185154802.00000 0 at C:/p523/src/lib/File/Find.pm line 435. -------------------------------------------- -- bulk88 ~ bulk88 at hotmail.com
Subject: Re: [perl #126414] perl rounds inode in PP stat
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
To: perl5-porters [...] perl.org
Date: Fri, 12 Feb 2016 00:55:28 +0100
* bulk88 via RT <perlbug-followup@perl.org> [2016-02-10 02:30]: Show quoted text
> It seems that NTFS inode numbers are a 64 bit bitfield/struct, not > linear 64 bit counter, and therefore always > 2^53 and every PP stat() > is generating an overflow warning on my 32 bit IV perl.
Which implies that the inode field has always been broken. Not bad! I must wonder how come nobody ran into this before, then. An impressive feat of “works by accident” – but how? Presumably by way of the rounding being deterministic. Possibly due to the form of the bit struct aligning with the deterministic rounding in such a way that it mitigates the problem in practical scenarios. Show quoted text
> Not sure how to proceed now.
Test that hypothesis. Run a recursive walk over your FSs and record the inodes and paths and check to see if you get false duplicates based on comparing inode values. If not, then the urgency to solve this goes way down and also the importance of solving it non-disruptively goes way up. (Although it still ought to be solved, either way.) Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.5k
On Thu Feb 11 15:55:50 2016, aristotle wrote: Show quoted text
> * bulk88 via RT <perlbug-followup@perl.org> [2016-02-10 02:30]:
> > It seems that NTFS inode numbers are a 64 bit bitfield/struct, not > > linear 64 bit counter, and therefore always > 2^53 and every PP stat() > > is generating an overflow warning on my 32 bit IV perl.
> > Which implies that the inode field has always been broken. Not bad! > > I must wonder how come nobody ran into this before, then. An impressive > feat of “works by accident” – but how?
The windows libc ALWAYS sets st_ino to 0. Blead win32 perl's st_ino is always 0 ( there is an old feature request in https://rt.perl.org/Ticket/Display.html?id=65052 asking to fix that I attempted to fix). I created an unpublished/yet to publish patch that gets the inode ("FileIndex" or "IndexNumber" https://msdn.microsoft.com/en-us/library/windows/hardware/ff540318%28v=vs.85%29.aspx ) from the MS Win32 API, since the MS libc API doesn't supply it and presents it on a Perl C level to upper Perl C layers. That is how I got inodes visible on a PP level in win32 perl and was able to test this overflow/rounding patch. Show quoted text
> Presumably by way of the rounding being deterministic. Possibly due to > the form of the bit struct aligning with the deterministic rounding in > such a way that it mitigates the problem in practical scenarios.
The chances of a collision seem low, but not theoretically impossible (only comparing all 64 bits would be collision free). Since Im not a CS PHD, I wont try to estimate the chance. Any porters can analyze my raw data that i posted and guess the chance. The highest 16 bit short seems to be a bucket or checksum, while the low 32 bits seem to be an offset into an array. That raw data is probably from iterating over my perl git source code root dir and some of the offsets seem very close. Show quoted text
> > Not sure how to proceed now.
> > Test that hypothesis. Run a recursive walk over your FSs and record the > inodes and paths and check to see if you get false duplicates based on > comparing inode values. If not, then the urgency to solve this goes way > down and also the importance of solving it non-disruptively goes way up. > (Although it still ought to be solved, either way.)
Iterate over the entire FS, make a hash with the key being the FP and value 1 or ++. At the end of interating over the entire FS (or as much before 32 bit process mem limit hits), go over the hash, if a key has a value >1, either I have hard links on Win32 or rounding collision. Is this the correct algorithm? -- bulk88 ~ bulk88 at hotmail.com
To: perl5-porters [...] perl.org
From: Zefram <zefram [...] fysh.org>
Date: Sat, 11 Nov 2017 07:50:52 +0000
Subject: Re: [perl #126414] perl rounds inode in PP stat
Download (untitled) / with headers
text/plain 175b
Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now returns a string of decimal digits if the inode number isn't representable as a native integer. -zefram
To: perl5-porters [...] perl.org
From: Shlomi Fish <shlomif [...] shlomifish.org>
Date: Sat, 11 Nov 2017 10:21:37 +0200
Subject: Re: [perl #126414] perl rounds inode in PP stat
Download (untitled) / with headers
text/plain 667b
On Sat, 11 Nov 2017 07:50:52 +0000 Zefram <zefram@fysh.org> wrote: Show quoted text
> Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now > returns a string of decimal digits if the inode number isn't representable > as a native integer. > > -zefram
Thanks, Zefram! -- ----------------------------------------------------------------- Shlomi Fish http://www.shlomifish.org/ UNIX Fortune Cookies - http://www.shlomifish.org/humour/fortunes/ XSLT is the number one cause of programmers’ suicides since Visual Basic 1.0. — http://www.shlomifish.org/humour/bits/facts/XSLT/ Please reply to list if it's a mailing list post - http://shlom.in/reply .
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 499b
On Fri, 10 Nov 2017 23:51:30 -0800, zefram@fysh.org wrote: Show quoted text
> Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now > returns a string of decimal digits if the inode number isn't representable > as a native integer.
This commit introduced a new build warning on my machine (64 bit Mac OS X): pp_sys.c:3014:32: warning: comparison of unsigned expression < 0 is always false [-Wtautological-compare] neg = PL_statcache.st_ino < 0; ~~~~~~~~~~~~~~~~~~~ ^ ~
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 693b
On Sat, 11 Nov 2017 15:27:16 GMT, randir wrote: Show quoted text
> On Fri, 10 Nov 2017 23:51:30 -0800, zefram@fysh.org wrote:
> > Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now > > returns a string of decimal digits if the inode number isn't > > representable > > as a native integer.
> > This commit introduced a new build warning on my machine (64 bit Mac > OS X): > > pp_sys.c:3014:32: warning: comparison of unsigned expression < 0 is > always false [-Wtautological-compare] > neg = PL_statcache.st_ino < 0; > ~~~~~~~~~~~~~~~~~~~ ^ ~
Can you attach the output of 'perl -V' from this build? Thank you very much. -- James E Keenan (jkeenan@cpan.org)
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 3.5k
On Sat, 11 Nov 2017 08:22:22 -0800, jkeenan wrote: Show quoted text
> Can you attach the output of 'perl -V' from this build?
Summary of my perl5 (revision 5 version 27 subversion 6) configuration: Commit id: 7d65f652cb34981f4cb53a56496f5712f740d496 Platform: osname=darwin osvers=13.4.0 archname=darwin-thread-multi-2level uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0: mon jan 11 18:17:34 pst 2016; root:xnu-2422.115.15~1release_x86_64 x86_64 ' config_args='-de -Dusedevel -DDEBUGGING -Duseithreads' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='cc' ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -DPERL_USE_SAFE_PUTENV' optimize='-O3 -g' cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='' gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='cc' ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib' libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib /usr/local/lib /usr/lib libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc perllibs=-lpthread -ldl -lm -lutil -lc libc= so=dylib useshrplib=false libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=bundle d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_TRACK_MEMPOOL PERL_USE_DEVEL PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Built under darwin Compiled at Nov 11 2017 18:30:42 %ENV: PERLBREW_BASHRC_VERSION="0.78" PERLBREW_HOME="/Users/dur-randir/.perlbrew" PERLBREW_MANPATH="/Users/dur-randir/perlbrew/perls/perl-5.22.1/man" PERLBREW_PATH="/Users/dur-randir/perlbrew/bin:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin" PERLBREW_PERL="perl-5.22.1" PERLBREW_ROOT="/Users/dur-randir/perlbrew" PERLBREW_VERSION="0.78" @INC: lib /usr/local/lib/perl5/site_perl/5.27.6/darwin-thread-multi-2level /usr/local/lib/perl5/site_perl/5.27.6 /usr/local/lib/perl5/5.27.6/darwin-thread-multi-2level /usr/local/lib/perl5/5.27.6
From: Zefram <zefram [...] fysh.org>
To: perl5-porters [...] perl.org
Subject: Re: [perl #126414] perl rounds inode in PP stat
Date: Sat, 11 Nov 2017 23:27:10 +0000
Download (untitled) / with headers
text/plain 200b
Sergey Aleynikov via RT wrote: Show quoted text
>This commit introduced a new build warning on my machine (64 bit Mac OS X):
Thanks. That should be fixed by commit b8897079199ab3165b21ca3aff78c7479842181a. -zefram
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 173b
On Sat, 11 Nov 2017 15:27:55 -0800, zefram@fysh.org wrote: Show quoted text
> Thanks. That should be fixed by commit > b8897079199ab3165b21ca3aff78c7479842181a.
Yes, this fixed it, thanks.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org