Skip Menu |
Report information
Id: 122820
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: jkeenan <jkeenan [at] cpan.org>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: unknown
Perl Version: (no value)
Fixed In: (no value)



Download (untitled) / with headers
text/plain 5.3k
With this ticket I bring into RT the discussion that began on list starting at: http://www.nntp.perl.org/group/perl.perl5.porters/2014/09/msg220237.html In this post I summarize that discussion, present current documentation relevant to the problem and describe one policy question which needs to be resolved before a solution can be implemented. I. Discussion of Problem Steve Hay initiated the discussion in the context of working on core on Win32: ##### "I just noticed that after doing a "nmake distclean" there are still many files left lying around that shouldn't be. Exactly which ones is shown by running "git clean -dfx" afterwards: That should output nothing, but instead cleaned up all the files below! "distclean certainly used to work at some time in the (recent?) past." ##### Subsequent posts confirmed that this problem is *not* limited to Win32 and has existed for some time. Some have suggested that 'make distclean' be made an alias for 'git clean -dfx', but others have pointed out that you cannot and should not assume that one is always working within a git checkout. 'make distclean' presumably DTRT for Perl 5 core developers before we switched to git as our revision control system. It should not have been allowed to fall into disrepair. [My editorial comments: There is no guarantee that git will be the last revision control system Perl will ever use, so we shouldn't tie ourselves too closely to it. TMTOWTDI encourages this and prudence recommends this. 'git clean -dfx' may DTRT for *many* Perl 5 core developers *much* of the time, the mere fact that we are having this discussion indicates that it does not DTRT for all core developers all of the time.] II. Documentation of Current Functionality A. 'make distclean' 'make distclean' is documented in INSTALL under the heading "Starting all over again": ##### If you wish to rebuild perl from the same build directory, you should clean it out with the command 'make distclean' or 'make realclean'[.] The only difference between the two is that 'make distclean' also removes your old config.sh and Policy.sh files. [Some paragraph reformatting -- jk.] ##### B. 'git clean -dfx' From 'man git-clean' for Git 1.9.l: ##### Cleans the working tree by recursively removing files that are not under version control, starting from the current directory. Normally, only files unknown to Git are removed, but if the -x option is specified, ignored files are also removed. This can, for example, be useful to remove all build products. ##### Here is what I believe to be the most relevant parts of the documentation of the three options: ##### -d: Remove untracked directories in addition to untracked files. -f: If the Git configuration variable clean.requireForce is not set to false, git clean will refuse to run unless given -f, -n or -i. -x: This allows removing all untracked files, including build products. This can be used (possibly in conjunction with git reset) to create a pristine working directory to test a clean build. ##### C. Attachments Attached are three files derived from a perl built on Linux x86_64 in blead. 'git clean -dfx' was run before a fetch from origin and rebase, then perl was configured only with '-des -Dusedevel', then perl was built and tested. * Output from 'git clean -dfx' after from-scratch build: git.clean.dfx.txt * Output from 'make distclean' after from-scratch build: make.distclean.txt * Output from 'git clean -dfx' after 'make distclean' has been run: activated.after.make.distclean.git.clean.dfx.txt Examination of the latter suggests that almost all of the files and directories which 'make distclean' is failing to remove and which we would presumably expect it to remove are found under 'lib/'. III. Policy Issues A. Scope of Defect Are we in agreement that, given an otherwise pristine working directory, 'make distclean' should remove the same directories and files as 'git clean -dfx'? In other words, after running 'make distclean' should a file like attachment activated.after.make.distclean.git.clean.dfx.txt be empty? B. Scope of 'make distclean' relative to 'git clean -dfx' 'make distclean' does not rely on the revision control system for the list of files to be removed. It relies on MANIFEST, Makefile and its predecessors, etc. As such, it is not documented to remove any files the user may have in the working directory which are not under revision control or are not build artifacts. Suppose I had in my working directory a little shell script to configure a DEBUGGING build. ##### $ cat mydebugconfigure.sh #/bin/sh ./Configure -des -Dusedevel -DDEBUGGING ##### 'make distclean' would presumably *not* remove that shell script as it is neither under revision control nor a build artifact. 'git clean', however, treats the working directory primarily as a git checkout and -- particularly with '-dfx' -- aims to get that checkout back to a pristine state from git's point of view. git's point of view is not necessarily perl's or the developer's point of view. 'git clean -dfx' would nuke that shell script wrapping around Configure. Once we have corrected the defect in III.A. above, are we happy with the fact that 'make distclean' and 'git clean -dfx' have somewhat different objectives and hence achieve different results? IV. Summary Once we've resolved III.A. and III.B., patches welcome! Thank you very much. Jim Keenan -- James E Keenan (jkeenan@cpan.org)
Subject: activated.after.make.distclean.git.clean.dfx.txt
Removing lib/Attribute/ Removing lib/AutoLoader.pm Removing lib/AutoSplit.pm Removing lib/B/Debug.pm Removing lib/CPAN/ Removing lib/Carp.pm Removing lib/Carp/ Removing lib/Config/Perl/ Removing lib/Devel/ Removing lib/Digest.pm Removing lib/Digest/ Removing lib/Dumpvalue.pm Removing lib/Env.pm Removing lib/Exporter.pm Removing lib/Exporter/ Removing lib/ExtUtils/CBuilder.pm Removing lib/ExtUtils/CBuilder/ Removing lib/ExtUtils/Command.pm Removing lib/ExtUtils/Constant.pm Removing lib/ExtUtils/Constant/ Removing lib/ExtUtils/Install.pm Removing lib/ExtUtils/Installed.pm Removing lib/ExtUtils/Miniperl.pm Removing lib/ExtUtils/Packlist.pm Removing lib/Fatal.pm Removing lib/File/Fetch.pm Removing lib/File/Find.pm Removing lib/File/Path.pm Removing lib/File/Temp.pm Removing lib/FileCache.pm Removing lib/Filter/ Removing lib/Getopt/Long.pm Removing lib/HTTP/ Removing lib/I18N/ Removing lib/IO/ Removing lib/IPC/ Removing lib/Locale/ Removing lib/Math/ Removing lib/Memoize.pm Removing lib/Memoize/ Removing lib/Module/ Removing lib/NEXT.pm Removing lib/Net/Ping.pm Removing lib/Params/ Removing lib/Parse/ Removing lib/Perl/ Removing lib/PerlIO/ Removing lib/Pod/Escapes.pm Removing lib/Pod/Simple.pm Removing lib/Pod/Simple.pod Removing lib/Pod/Simple/ Removing lib/Safe.pm Removing lib/Search/ Removing lib/SelfLoader.pm Removing lib/Term/ Removing lib/Test.pm Removing lib/Test/ Removing lib/Text/ Removing lib/Thread/ Removing lib/Tie/File.pm Removing lib/Tie/Memoize.pm Removing lib/Tie/RefHash.pm Removing lib/Time/Local.pm Removing lib/autodie.pm Removing lib/autodie/ Removing lib/autouse.pm Removing lib/base.pm Removing lib/bigint.pm Removing lib/bignum.pm Removing lib/bigrat.pm Removing lib/constant.pm Removing lib/encoding/ Removing lib/experimental.pm Removing lib/fields.pm Removing lib/if.pm Removing lib/parent.pm Removing lib/perlfaq.pm Removing lib/perlfaq.pod Removing lib/perlfaq1.pod Removing lib/perlfaq2.pod Removing lib/perlfaq3.pod Removing lib/perlfaq4.pod Removing lib/perlfaq5.pod Removing lib/perlfaq6.pod Removing lib/perlfaq7.pod Removing lib/perlfaq8.pod Removing lib/perlfaq9.pod Removing lib/perlglossary.pod Removing lib/version.pm Removing lib/version.pod Removing lib/version/ Removing tclean veryclean _verycleaner Removing tclean' or 'veryclean' targets are run.
Subject: git.clean.dfx.txt
Download git.clean.dfx.txt
text/plain 39.1k

Message body is not shown because it is too large.

Subject: make.distclean.txt
Download make.distclean.txt
text/plain 115.8k

Message body is not shown because it is too large.

RT-Send-CC: perl5-porters [...] perl.org
Subject: perl.V.txt
Download perl.V.txt
text/plain 3.2k
Summary of my perl5 (revision 5 version 21 subversion 5) configuration: Commit id: e96513a272994fa40337538729c370cb2b01f9fd Platform: osname=linux, osvers=3.13.0-35-generic, archname=x86_64-linux uname='linux zareason 3.13.0-35-generic #62-ubuntu smp fri aug 15 01:58:42 utc 2014 x86_64 x86_64 x86_64 gnulinux ' config_args='-des -Dusedevel -DDEBUGGING' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.8.2', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.19' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Built under linux Compiled at Sep 21 2014 09:47:45 %ENV: PERL5LIB="/home/jkeenan/perl5/lib/perl5" PERLBREW_BASHRC_VERSION="0.67" PERLBREW_HOME="/home/jkeenan/.perlbrew" PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.20.0/man" PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin:/home/jkeenan/perl5/perlbrew/perls/perl-5.20.0/bin" PERLBREW_PERL="perl-5.20.0" PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew" PERLBREW_VERSION="0.67" PERL_LOCAL_LIB_ROOT="/home/jkeenan/perl5" PERL_MB_OPT="--install_base "/home/jkeenan/perl5"" PERL_MM_OPT="INSTALL_BASE=/home/jkeenan/perl5" PERL_WORKDIR="gitwork/perl" @INC: lib /home/jkeenan/perl5/lib/perl5/x86_64-linux /home/jkeenan/perl5/lib/perl5 /usr/local/lib/perl5/site_perl/5.21.5/x86_64-linux /usr/local/lib/perl5/site_perl/5.21.5 /usr/local/lib/perl5/5.21.5/x86_64-linux /usr/local/lib/perl5/5.21.5 .
Subject: Re: [perl #122820]
CC: "Perl5 Porters (E-mail)" <perl5-porters [...] perl.org>
To: perlbug-comment [...] perl.org
Date: Sun, 21 Sep 2014 14:27:02 -0500
From: "Craig A. Berry" <craig.a.berry [...] gmail.com>
Download (untitled) / with headers
text/plain 529b
distclean also fails to clean up entirely with v5.20.1-3-g479259f and what's left looks pretty much the same as what's left in blead. v5.18.3-RC1-2-g77d4f3e distclean leaves some things behind but all of them are directories under lib/ (no .pm files) plus some DB_File test detritus. So I think both John and Steve are right. I think what John saw a couple of years ago was fixed by 727d4ce653261272b. I think what we're seeing now and what Steve reported appeared in the 5.19.x timeframe. Haven't narrowed it down further.
From: "Craig A. Berry" <craig.a.berry [...] gmail.com>
To: perlbug-comment [...] perl.org
Date: Sun, 21 Sep 2014 20:40:13 -0500
CC: "Perl5 Porters (E-mail)" <perl5-porters [...] perl.org>
Subject: Re: [perl #122820]
Download (untitled) / with headers
text/plain 606b
The problem could be readily reduced to: $ ./miniperl -Ilib make_ext.pl --target=all Exporter MAKE="make" $ ./miniperl -Ilib make_ext.pl --target=distclean Exporter MAKE="make" $ ls lib/Exporter.pm and observing that lib/Exporter.pm was still there after it was created by the first command and should have been (but wasn't) cleaned up by the second command. Fixed in 9fa5aa91434. Side note: "make clean" is even more seriously broken -- it removes "Config_git.pl" which it then depends on for nearly everything it does. But that's a different bug, and was the same before and after fixing this one.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.1k
On Sun Sep 21 18:40:52 2014, craig.a.berry@gmail.com wrote: Show quoted text
> The problem could be readily reduced to: > > $ ./miniperl -Ilib make_ext.pl --target=all Exporter MAKE="make" > $ ./miniperl -Ilib make_ext.pl --target=distclean Exporter MAKE="make" > $ ls lib/Exporter.pm > > and observing that lib/Exporter.pm was still there after it was > created by the first command and should have been (but wasn't) cleaned > up by the second command. > > Fixed in 9fa5aa91434. >
I'm amazed that Craig solved this so quickly. Craig's patch assumes -- correctly, IMO -- that 'make distclean' and 'git clean -dfx' have slightly different objectives. Going back to the example I provided in section III.B above, I: * Placed in my working directory a small shell script, mydebugconfigure.sh, wrapping around Configure. * Configured with this script, built and test perl at blead (9fa5aa9143). * Said: 'make distclean'. * Then said: 'git clean -dfxn' to perform a dry-run of what 'git clean -dfx' would still deem needing removal to keep git happy: ##### $ git clean -dfxn Would remove mydebugconfigure.sh ##### This DWIMs for me. Thank you very much. -- James E Keenan (jkeenan@cpan.org)
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 520b
On Sun Sep 21 18:40:52 2014, craig.a.berry@gmail.com wrote: Show quoted text
> The problem could be readily reduced to: > > $ ./miniperl -Ilib make_ext.pl --target=all Exporter MAKE="make" > $ ./miniperl -Ilib make_ext.pl --target=distclean Exporter MAKE="make" > $ ls lib/Exporter.pm > > and observing that lib/Exporter.pm was still there after it was > created by the first command and should have been (but wasn't) cleaned > up by the second command. > > Fixed in 9fa5aa91434. >
Excellent! Thanks for the fix. Closing ticket now.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org