Skip Menu |
Report information
Id: 133673
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: jik [at] kamens.us
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: medium
Type: library
Perl Version: 5.26.2
Fixed In: (no value)

Attachments
0001-perl-133673-disable-the-File-Find-nlink-optimization.patch



To: perlbug [...] perl.org
From: jik [...] kamens.us
Subject: File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
Date: Tue, 20 Nov 2018 16:52:52 -0500 (EST)
This is a bug report for perl from jik@kamens.us, generated with the help of perlbug 1.40 running under perl 5.26.2. ----------------------------------------------------------------- I have a perl script which does `use File::Find;` and later calls `&find` on a directory that is mounted over CIFS from a Synology NAS. This script has worked fine in the past, but it is no longer iterating over subdirectories of the top-level directory that is passed into `&find`. I looked at the code in Find.pm and I see that it automatically avoids using the number of links to a directory when `stat` on the directory returns less than 2 for nlink. Unfortunately, it appears that directories on the NAS return 2 as the number of links for all directories. Observe (the directory "~/ogg" is on my NAS): $ cd ~/ogg $ mkdir foo $ mkdir foo/bar $ stat foo File: foo Size: 0 Blocks: 0 IO Block: 16384 directory Device: 42h/66d Inode: 8892 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 100/ users) Access: 2018-11-20 16:50:16.928421300 -0500 Modify: 2018-11-20 16:50:20.604421400 -0500 Change: 2018-11-20 16:50:20.604421400 -0500 Birth: - $ stat foo/bar File: foo/bar Size: 0 Blocks: 0 IO Block: 16384 directory Device: 42h/66d Inode: 8893 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 100/ users) Access: 2018-11-20 16:50:20.604421400 -0500 Modify: 2018-11-20 16:50:20.604421400 -0500 Change: 2018-11-20 16:50:20.604421400 -0500 Birth: - $ Compare with the local filesystem in /tmp: $ cd /tmp $ mkdir foo $ mkdir foo/bar $ stat foo File: foo Size: 4096 Blocks: 8 IO Block: 4096 directory Device: fd00h/64768d Inode: 39862292 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 1000/ jik) Access: 2018-11-20 16:50:44.132313739 -0500 Modify: 2018-11-20 16:50:46.620233036 -0500 Change: 2018-11-20 16:50:46.620233036 -0500 Birth: - $ stat foo/bar File: foo/bar Size: 4096 Blocks: 8 IO Block: 4096 directory Device: fd00h/64768d Inode: 39862306 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 1000/ jik) Access: 2018-11-20 16:50:46.620233036 -0500 Modify: 2018-11-20 16:50:46.620233036 -0500 Change: 2018-11-20 16:50:46.620233036 -0500 Birth: - $ I don't think this was caused by a change to the code in File::Find, because I went back several Perl releases going back years and tested all of them with the same results. I think for some reason the NAS is returning different a value for nlink than it used to, and this is breaking Perl's logic. So while the problem may not have started happening due to a change in Perl, the fact remains that the File::Find man page claims it's supposed to detect automatically when nlink can't be used, and it's not doing that successfully here. [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=library severity=medium module=File::Find --- Site configuration information for perl 5.26.2: Configured by Ubuntu at Sun Aug 12 08:26:41 UTC 2018. Summary of my perl5 (revision 5 version 26 subversion 2) configuration: Platform: osname=linux osvers=4.9.0 archname=x86_64-linux-gnu-thread-multi uname='linux localhost 4.9.0 #1 smp debian 4.9.0 x86_64 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-sbTH0k/perl-5.26.2=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-Bsymbolic-functions -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.26 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.26 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.26 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.26.2 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.26.2 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.26.2' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='x86_64-linux-gnu-gcc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' optimize='-O2 -g' cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='' gccversion='8.2.0' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='x86_64-linux-gnu-gcc' ldflags =' -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=libc-2.27.so so=so useshrplib=true libperl=libperl.so.5.26 gnulibc_version='2.27' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E' cccdlflags='-fPIC' lddlflags='-shared -L/usr/local/lib -fstack-protector-strong' Locally applied patches: DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN. DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check. DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information. DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories. DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes. DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking DEBPKG:fixes/respect_umask - Respect umask during installation DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets. DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor. DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy. DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable. DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.26.2-7 in patchlevel.h DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags} DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798 DEBPKG:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site DEBPKG:fixes/time_piece_doc - https://bugs.debian.org/817925 Time::Piece: Improve documentation for add_months and add_years DEBPKG:fixes/extutils_makemaker_reproducible - https://bugs.debian.org/835815 https://bugs.debian.org/834190 Make perllocal.pod files reproducible DEBPKG:fixes/file_path_hurd_errno - File-Path: Fix test failure in Hurd due to hard-coded ENOENT DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters DEBPKG:fixes/file_path_chmod_race - https://bugs.debian.org/863870 [rt.cpan.org #121951] Prevent directory chmod race attack. DEBPKG:fixes/extutils_file_path_compat - Correct the order of tests of chmod(). (#294) DEBPKG:fixes/getopt-long-2 - [rt.cpan.org #120300] Withdraw part of commit 5d9947fb445327c7299d8beb009d609bc70066c0, which tries to implement more GNU getopt_long campatibility. GNU DEBPKG:fixes/getopt-long-3 - provide a default value for optional arguments DEBPKG:fixes/getopt-long-4 - https://bugs.debian.org/864544 [rt.cpan.org #122068] Fix issue #122068. DEBPKG:fixes/test-builder-reset - https://bugs.debian.org/865894 Reset inside subtest maintains parent DEBPKG:debian/hppa_opmini_optimize_workaround - https://bugs.debian.org/869122 Lower the optimization level of opmini.c on hppa DEBPKG:debian/sh4_op_optimize_workaround - https://bugs.debian.org/869373 Also lower the optimization level of op.c and opmini.c on sh4 DEBPKG:fixes/json-pp-example - [rt.cpan.org #92793] https://bugs.debian.org/871837 fix RT-92793: bug in SYNOPSIS DEBPKG:debian/perldoc-pager - https://bugs.debian.org/870340 [rt.cpan.org #120229] Fix perldoc terminal escapes when sensible-pager is less DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need. DEBPKG:fixes/rename-filexp.U-phase1 - regen-configure: rename filexp.U to filexp_path.U, phase 1 DEBPKG:fixes/rename-filexp.U-phase2 - regen-configure: rename filexp.U to filexp_path.U, phase 2 DEBPKG:fixes/packaging_test_skips - Skip various tests if PERL_BUILD_PACKAGING is set DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian DEBPKG:fixes/encode-alias-regexp - https://bugs.debian.org/880085 fix https://github.com/dankogai/p5-encode/issues/127 DEBPKG:fixes/CVE-2018-6797-testcase - [perl #132227] (perl #132227) restart a node if we change to uni rules within the node and encounter a sharp S DEBPKG:fixes/test-printf-null - https://bugs.debian.org/893601 Fix pointer to integer cast in null_printf check DEBPKG:debian/configure-regen - https://bugs.debian.org/762638 Regenerate Configure et al. after probe unit changes DEBPKG:fixes/CVE-2018-12015-Archive-Tar-directory-traversal - https://bugs.debian.org/900834 [rt.cpan.org #125523] Remove existing files before overwriting them DEBPKG:debian/gdbm-fatal - [perl #133295] https://bugs.debian.org/904005 Temporarily skip GDBM_File fatal.t for gdbm >= 1.15 compatibility --- @INC for perl 5.26.2: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.2 /usr/local/share/perl/5.26.2 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/local/lib/x86_64-linux-gnu/perl/5.26.0 /usr/local/share/perl/5.26.0 /usr/lib/x86_64-linux-gnu/perl-base --- Environment for perl 5.26.2: HOME=/home/jik LANG=en_US.utf8 LANGUAGE=en_US:en_GB:en LC_ADDRESS=en_US.UTF-8 LC_COLLATE=C LC_IDENTIFICATION=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_NUMERIC=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_TELEPHONE=en_US.UTF-8 LC_TIME=en_US.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/jik/.cargo/bin:/home/jik/bin:/home/jik/bin:/home/jik/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jik/bin/.bash_aliases:/usr/lib/emacs/25.2/x86_64-linux-gnu PERL_BADLANG (unset) SHELL=/bin/bash
RT-Send-CC: perl5-porters [...] perl.org

Message body is not shown because it is too large.

CC: perlbug-followup [...] perl.org
Date: Wed, 21 Nov 2018 08:39:52 +0100
Subject: Re: [perl #133673] File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 7.1k
On Tue, 20 Nov 2018 18:32:32 -0800, "James E Keenan via RT" <perlbug-followup@perl.org> wrote: Show quoted text
> On Tue, 20 Nov 2018 22:51:54 GMT, jik@kamens.us wrote:
> > > > This is a bug report for perl from jik@kamens.us, > > generated with the help of perlbug 1.40 running under perl 5.26.2. > > ----------------------------------------------------------------- > > I have a perl script which does `use File::Find;` and later calls > > `&find` on a directory that is mounted over CIFS from a Synology NAS.
> > For those of us who, like myself, are unfamiliar with your > terminology ... are these the "CIFS" and "Synology NAS" you are > referring to? > > https://en.wikipedia.org/wiki/Server_Message_Block
CIFS ~= Samba : https://www.varonis.com/blog/cifs-vs-smb/ In short: CIFS is old, Samba is new HP-UX also uses CIFS to refer to cross-system mounts à la Windows, but it supports both Show quoted text
Here, it is relative important what DSM (the Synology NAS OS) version is running, as old(er) architectures and models do not "do" updates on the DSM anymore and Synology models support a wide range of CPU types See also README.synology in the perl root (which might need some updates) I know that the OP uses perl on a debian system to a SMB mount on the NAS, but here are my DSM perl version: On my DS218, I have 2 perl versions available: Linux 4.4.59+ [DSM 6.2-23824-1] DS218+ Celeron(R) CPU J3355 @ 2.00GHz/2001(2) x86_64 1828 Mb /opt/bin/perl This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux-glibc /usr/local/bin/perl This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-linux Show quoted text
> > This script has worked fine in the past, but it is no longer iterating > > over subdirectories of the top-level directory that is passed into > > `&find`. > > > > I looked at the code in Find.pm and I see that it automatically avoids > > using the number of links to a directory when `stat` on the directory > > returns less than 2 for nlink. > > > > Unfortunately, it appears that directories on the NAS return 2 as the > > number of links for all directories. Observe (the directory "~/ogg" is > > on my NAS):
I can confirm that, but have to add that this has nothing to do with SBM or CIFS, as the same thing shows on the NAS itself Show quoted text
> > $ cd ~/ogg > > $ mkdir foo > > $ mkdir foo/bar > > $ stat foo > > File: foo > > Size: 0 Blocks: 0 IO Block: 16384 directory > > Device: 42h/66d Inode: 8892 Links: 2 > > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 100/ > > users) > > Access: 2018-11-20 16:50:16.928421300 -0500 > > Modify: 2018-11-20 16:50:20.604421400 -0500 > > Change: 2018-11-20 16:50:20.604421400 -0500 > > Birth: -
/NAS/Tmp > stat foo File: foo Size: 0 Blocks: 0 IO Block: 16384 directory Device: 2eh/46d Inode: 11759612 Links: 1 Access: (0775/drwxrwxr-x) Uid: ( 9999/ merijn) Gid: ( 100/ users) Access: 2018-11-21 08:32:10.693179600 +0100 Modify: 2018-11-21 08:32:13.501215300 +0100 Change: 2018-11-21 08:32:13.501215300 +0100 Birth: - Show quoted text
> > $ stat foo/bar > > File: foo/bar > > Size: 0 Blocks: 0 IO Block: 16384 directory > > Device: 42h/66d Inode: 8893 Links: 2 > > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 100/ > > users) > > Access: 2018-11-20 16:50:20.604421400 -0500 > > Modify: 2018-11-20 16:50:20.604421400 -0500 > > Change: 2018-11-20 16:50:20.604421400 -0500 > > Birth: - > > $
/NAS/Tmp > stat foo/bar File: foo/bar Size: 0 Blocks: 0 IO Block: 16384 directory Device: 2eh/46d Inode: 11759613 Links: 1 Access: (0775/drwxrwxr-x) Uid: ( 9999/ merijn) Gid: ( 100/ users) Access: 2018-11-21 08:32:13.501215300 +0100 Modify: 2018-11-21 08:32:13.501215300 +0100 Change: 2018-11-21 08:32:13.501215300 +0100 Birth: - Show quoted text
> > Compare with the local filesystem in /tmp: > > > > $ cd /tmp > > $ mkdir foo > > $ mkdir foo/bar > > $ stat foo > > File: foo > > Size: 4096 Blocks: 8 IO Block: 4096 directory > > Device: fd00h/64768d Inode: 39862292 Links: 3 > > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 1000/ > > jik) > > Access: 2018-11-20 16:50:44.132313739 -0500 > > Modify: 2018-11-20 16:50:46.620233036 -0500 > > Change: 2018-11-20 16:50:46.620233036 -0500 > > Birth: -
/tmp 509 > stat foo File: foo Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 803h/2051d Inode: 48762113 Links: 3 Access: (0775/drwxrwxr-x) Uid: ( 9999/ merijn) Gid: ( 100/ users) Access: 2018-11-21 08:33:56.039678844 +0100 Modify: 2018-11-21 08:33:56.039678844 +0100 Change: 2018-11-21 08:33:56.039678844 +0100 Birth: - Show quoted text
> > $ stat foo/bar > > File: foo/bar > > Size: 4096 Blocks: 8 IO Block: 4096 directory > > Device: fd00h/64768d Inode: 39862306 Links: 2 > > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 1000/ > > jik) > > Access: 2018-11-20 16:50:46.620233036 -0500 > > Modify: 2018-11-20 16:50:46.620233036 -0500 > > Change: 2018-11-20 16:50:46.620233036 -0500 > > Birth: - > > $
/tmp 510 > stat foo/bar File: foo/bar Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 803h/2051d Inode: 48762200 Links: 2 Access: (0775/drwxrwxr-x) Uid: ( 9999/ merijn) Gid: ( 100/ users) Access: 2018-11-21 08:33:56.039678844 +0100 Modify: 2018-11-21 08:33:56.039678844 +0100 Change: 2018-11-21 08:33:56.039678844 +0100 Birth: - Show quoted text
> > I don't think this was caused by a change to the code in File::Find, > > because I went back several Perl releases going back years and tested > > all of them with the same results. I think for some reason the NAS is > > returning different a value for nlink than it used to, and this is > > breaking Perl's logic. > > > > So while the problem may not have started happening due to a change in > > Perl, the fact remains that the File::Find man page claims it's > > supposed to detect automatically when nlink can't be used, and it's > > not doing that successfully here.
Now on the NAS itself: /volume1/NAS/Tmp > stat foo File: ‘foo’ Size: 6 Blocks: 0 IO Block: 4096 directory Device: 22h/34d Inode: 11759612 Links: 1 Access: (0775/drwxrwxr-x) Uid: ( 9999/ merijn) Gid: ( 100/ users) Access: 2018-11-21 08:32:10.693179680 +0100 Modify: 2018-11-21 08:32:13.501215351 +0100 Change: 2018-11-21 08:32:13.501215351 +0100 Birth: - /volume1/NAS/Tmp > stat foo/bar File: ‘foo/bar’ Size: 0 Blocks: 0 IO Block: 4096 directory Device: 22h/34d Inode: 11759613 Links: 1 Access: (0775/drwxrwxr-x) Uid: ( 9999/ merijn) Gid: ( 100/ users) Access: 2018-11-21 08:32:13.501215351 +0100 Modify: 2018-11-21 08:32:13.501215351 +0100 Change: 2018-11-21 08:32:13.501215351 +0100 Birth: - This all has to do with the FS implementation of DSM and the access-control they use -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Download (untitled)
application/pgp-signature 473b

Message body not shown because it is not plain text.

From: Jonathan Kamens via perl5-porters <perl5-porters [...] perl.org>
Subject: Re: [perl #133673] File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
Date: Wed, 21 Nov 2018 11:50:50 -0500
To: perlbug-followup [...] perl.org

I have a DS415+ NAS running DSM 6.2.1-23824 Update 1, the current release.

  jik


RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 3.2k
On Tue, 20 Nov 2018 14:51:54 -0800, jik@kamens.us wrote: Show quoted text
> I have a perl script which does `use File::Find;` and later calls > `&find` on a directory that is mounted over CIFS from a Synology NAS. > > This script has worked fine in the past, but it is no longer iterating > over subdirectories of the top-level directory that is passed into > `&find`. > > I looked at the code in Find.pm and I see that it automatically avoids > using the number of links to a directory when `stat` on the directory > returns less than 2 for nlink. > > Unfortunately, it appears that directories on the NAS return 2 as the > number of links for all directories. Observe (the directory "~/ogg" is > on my NAS): > > $ cd ~/ogg > $ mkdir foo > $ mkdir foo/bar > $ stat foo > File: foo > Size: 0 Blocks: 0 IO Block: 16384 directory
I wonder if this directory size of zero can be used as a heuristic for distrusting st_nlink. Show quoted text
> Device: 42h/66d Inode: 8892 Links: 2 > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 100/ > users) > Access: 2018-11-20 16:50:16.928421300 -0500 > Modify: 2018-11-20 16:50:20.604421400 -0500 > Change: 2018-11-20 16:50:20.604421400 -0500 > Birth: - > $ stat foo/bar > File: foo/bar > Size: 0 Blocks: 0 IO Block: 16384 directory > Device: 42h/66d Inode: 8893 Links: 2 > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 100/ > users) > Access: 2018-11-20 16:50:20.604421400 -0500 > Modify: 2018-11-20 16:50:20.604421400 -0500 > Change: 2018-11-20 16:50:20.604421400 -0500 > Birth: - > $ > > Compare with the local filesystem in /tmp: > > $ cd /tmp > $ mkdir foo > $ mkdir foo/bar > $ stat foo > File: foo > Size: 4096 Blocks: 8 IO Block: 4096 directory > Device: fd00h/64768d Inode: 39862292 Links: 3 > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 1000/ > jik) > Access: 2018-11-20 16:50:44.132313739 -0500 > Modify: 2018-11-20 16:50:46.620233036 -0500 > Change: 2018-11-20 16:50:46.620233036 -0500 > Birth: - > $ stat foo/bar > File: foo/bar > Size: 4096 Blocks: 8 IO Block: 4096 directory > Device: fd00h/64768d Inode: 39862306 Links: 2 > Access: (0755/drwxr-xr-x) Uid: ( 1000/ jik) Gid: ( 1000/ > jik) > Access: 2018-11-20 16:50:46.620233036 -0500 > Modify: 2018-11-20 16:50:46.620233036 -0500 > Change: 2018-11-20 16:50:46.620233036 -0500 > Birth: - > $ > > I don't think this was caused by a change to the code in File::Find, > because I went back several Perl releases going back years and tested > all of them with the same results. I think for some reason the NAS is > returning different a value for nlink than it used to, and this is > breaking Perl's logic. > > So while the problem may not have started happening due to a change in > Perl, the fact remains that the File::Find man page claims it's > supposed to detect automatically when nlink can't be used, and it's > not doing that successfully here.
Unfortunately I don't see a simple fix for File::Find itself. As a workaround you could do: local $File::Find::dont_use_link = 1; before calling File::Find functions. In terms of fixing File::Find itself, all I can see is distrusting nlink on a given st_dev until we've found a directory with st_nlink > 2. Tony
To: perl5-porters [...] perl.org
From: Achim Gratz <Stromeko [...] nexgo.de>
Subject: Re: [perl #133673] File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
Date: Mon, 26 Nov 2018 19:34:15 +0100
Download (untitled) / with headers
text/plain 1.5k
"Tony Cook via RT" writes: Show quoted text
> I wonder if this directory size of zero can be used as a heuristic for > distrusting st_nlink.
I don't see the relation. Many non-UN*X file systems these days don't report a size for anything that's not a proper file. On Linux, several VFS do the same (/proc for instance) and some (e.g. /proc again) even do it for files. If directory links are not implemented, the reported number of links should be 1, not 2 however. Show quoted text
> Unfortunately I don't see a simple fix for File::Find itself. > > As a workaround you could do: > > local $File::Find::dont_use_link = 1; > > before calling File::Find functions. > > In terms of fixing File::Find itself, all I can see is distrusting > nlink on a given st_dev until we've found a directory with > st_nlink > 2.
Well, the first directory you find where nlink==2 and there is at least one other subdirectory tells you that you're looking at a buggy FS implementation. However, that determination can race with directory creation / deletion, so I think you need to err on the safe side and distrust nlink until you've fully resolved the first level _and_ it matches up with the directory count you've found. That potentially creates (much) more work if that first level is unusually large and has no directories in it, but I don't see what can be done about it if you want to work around buggy FS implementations. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
To: perlbug <perlbug-followup [...] perl.org>
Date: Mon, 26 Nov 2018 20:01:17 +0100
Subject: Re: [perl #133673] File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
From: Leon Timmermans <fawaka [...] gmail.com>
CC: Perl5 Porters <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 572b
On Mon, Nov 26, 2018 at 5:47 AM Tony Cook via RT <perlbug-followup@perl.org> wrote: Show quoted text
> Unfortunately I don't see a simple fix for File::Find itself. > > As a workaround you could do: > > local $File::Find::dont_use_link = 1; > > before calling File::Find functions. > > In terms of fixing File::Find itself, all I can see is distrusting nlink on a given st_dev until we've found a directory with st_nlink > 2.
Quite frankly, this optimization has given us sufficient headache over the years that I would argue for $File::Find::dont_use_link = 1 to be the default. Leon
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 676b
On Mon, 26 Nov 2018 11:01:52 -0800, LeonT wrote: Show quoted text
> On Mon, Nov 26, 2018 at 5:47 AM Tony Cook via RT > <perlbug-followup@perl.org> wrote:
> > Unfortunately I don't see a simple fix for File::Find itself. > > > > As a workaround you could do: > > > > local $File::Find::dont_use_link = 1; > > > > before calling File::Find functions. > > > > In terms of fixing File::Find itself, all I can see is distrusting > > nlink on a given st_dev until we've found a directory with st_nlink > > > 2.
> > Quite frankly, this optimization has given us sufficient headache over > the years that I would argue for $File::Find::dont_use_link = 1 to be > the default.
Like the attached. Tony
Subject: 0001-perl-133673-disable-the-File-Find-nlink-optimization.patch
From e3ad92c0a5ef0a707b5cb4e0d98fda313f3442c7 Mon Sep 17 00:00:00 2001 From: Tony Cook <tony@develop-help.com> Date: Mon, 10 Dec 2018 15:38:54 +1100 Subject: (perl #133673) disable the File::Find nlink optimization by default This will also fix 128894 and 126144. --- ext/File-Find/lib/File/Find.pm | 40 +++++++++------------------------------- 1 file changed, 9 insertions(+), 31 deletions(-) diff --git a/ext/File-Find/lib/File/Find.pm b/ext/File-Find/lib/File/Find.pm index 03dac9fbda..c052ceebb3 100644 --- a/ext/File-Find/lib/File/Find.pm +++ b/ext/File-Find/lib/File/Find.pm @@ -3,7 +3,7 @@ use 5.006; use strict; use warnings; use warnings::register; -our $VERSION = '1.34'; +our $VERSION = '1.35'; require Exporter; require Cwd; @@ -770,31 +770,11 @@ sub finddepth { $File::Find::skip_pattern = qr/^\.{1,2}\z/; $File::Find::untaint_pattern = qr|^([-+@\w./]+)$|; -# These are hard-coded for now, but may move to hint files. -if ($^O eq 'VMS') { - $Is_VMS = 1; - $File::Find::dont_use_nlink = 1; -} -elsif ($^O eq 'MSWin32') { - $Is_Win32 = 1; -} - # this _should_ work properly on all platforms # where File::Find can be expected to work $File::Find::current_dir = File::Spec->curdir || '.'; -$File::Find::dont_use_nlink = 1 - if $^O eq 'os2' || $^O eq 'dos' || $^O eq 'amigaos' || $Is_Win32 || - $^O eq 'interix' || $^O eq 'cygwin' || $^O eq 'qnx' || $^O eq 'nto'; - -# Set dont_use_nlink in your hint file if your system's stat doesn't -# report the number of links in a directory as an indication -# of the number of files. -# See e.g. hints/haiku.sh for Haiku. -unless ($File::Find::dont_use_nlink) { - require Config; - $File::Find::dont_use_nlink = 1 if ($Config::Config{'dont_use_nlink'}); -} +$File::Find::dont_use_nlink = 1; # We need a function that checks if a scalar is tainted. Either use the # Scalar::Util module's tainted() function or our (slower) pure Perl @@ -1106,17 +1086,15 @@ warnings. =item $dont_use_nlink -You can set the variable C<$File::Find::dont_use_nlink> to 1 if you want to -force File::Find to always stat directories. This was used for file systems -that do not have an C<nlink> count matching the number of sub-directories. -Examples are ISO-9660 (CD-ROM), AFS, HPFS (OS/2 file system), FAT (DOS file -system) and a couple of others. +You can set the variable C<$File::Find::dont_use_nlink> to 0 if you +are sure the filesystem you are scanning reflects the number of +subdirectories in the parent directory's C<nlink> count. -You shouldn't need to set this variable, since File::Find should now detect -such file systems on-the-fly and switch itself to using stat. This works even -for parts of your file system, like a mounted CD-ROM. +If you do set C<$File::Find::dont_use_nlink> to 0, you may notice an +improvement in speed at the risk of not recursing into subdirectories +if a filesystem doesn't populate C<nlink> as expected. -If you do set C<$File::Find::dont_use_nlink> to 1, you will notice slow-downs. +C<$File::Find::dont_use_nlink> now defaults to 1 on all platforms. =item symlinks -- 2.11.0
To: perlbug-followup [...] perl.org
Subject: Re: [perl #133673] File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
Date: Mon, 10 Dec 2018 05:04:44 +0000
From: Sevan Janiyan <venture37 [...] geeklan.co.uk>
Download (untitled) / with headers
text/plain 161b
On 10/12/2018 04:39, Tony Cook via RT wrote: Show quoted text
> Like the attached.
Does that not break functionality on current environments like QNX, Cygwin and Haiku? Sevan
To: Sevan Janiyan <venture37 [...] geeklan.co.uk>
Date: Mon, 10 Dec 2018 16:26:17 +1100
Subject: Re: [perl #133673] File::Find doesn't detect dont_use_nlink for CIFS-mounted Synology NAS filesystem
From: Tony Cook <tony [...] develop-help.com>
CC: perlbug-followup [...] perl.org
Download (untitled) / with headers
text/plain 532b
On Mon, Dec 10, 2018 at 05:04:44AM +0000, Sevan Janiyan wrote: Show quoted text
> On 10/12/2018 04:39, Tony Cook via RT wrote:
> > Like the attached.
> > Does that not break functionality on current environments like QNX, > Cygwin and Haiku?
-$File::Find::dont_use_nlink = 1 - if $^O eq 'os2' || $^O eq 'dos' || $^O eq 'amigaos' || $Is_Win32 || - $^O eq 'interix' || $^O eq 'cygwin' || $^O eq 'qnx' || $^O eq 'nto'; - is replaced: +$File::Find::dont_use_nlink = 1; further down. So there's no change on Cygwin, QNX, Haiku, etc. Tony
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 820b
On Sun, 09 Dec 2018 20:39:43 -0800, tonyc wrote: Show quoted text
> On Mon, 26 Nov 2018 11:01:52 -0800, LeonT wrote:
> > On Mon, Nov 26, 2018 at 5:47 AM Tony Cook via RT > > <perlbug-followup@perl.org> wrote:
> > > Unfortunately I don't see a simple fix for File::Find itself. > > > > > > As a workaround you could do: > > > > > > local $File::Find::dont_use_link = 1; > > > > > > before calling File::Find functions. > > > > > > In terms of fixing File::Find itself, all I can see is distrusting > > > nlink on a given st_dev until we've found a directory with st_nlink > > > > 2.
> > > > Quite frankly, this optimization has given us sufficient headache over > > the years that I would argue for $File::Find::dont_use_link = 1 to be > > the default.
> > Like the attached.
Applied as f75dfdbdb9d52b3024db9a0eb27405c2175af144. Tony
Download (untitled) / with headers
text/plain 313b
Thank you for filing this report. You have helped make Perl better. With the release today of Perl 5.30.0, this and 160 other issues have been resolved. Perl 5.30.0 may be downloaded via: https://metacpan.org/release/XSAWYERX/perl-5.30.0 If you find that the problem persists, feel free to reopen this ticket.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org