Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contradictory advice and test on using Test::More #13231

Closed
p5pRT opened this issue Sep 5, 2013 · 10 comments
Closed

Contradictory advice and test on using Test::More #13231

p5pRT opened this issue Sep 5, 2013 · 10 comments

Comments

@p5pRT
Copy link

p5pRT commented Sep 5, 2013

Migrated from rt.perl.org#119619 (status was 'open')

Searchable as RT119619$

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From @Smylers

This is a bug report for perl from smylers@​stripey.com,
generated with the help of perlbug 1.39 running under perl 5.19.4.


perlhack, in its TESTING section, says that tests other than in specific
subdirectories should use Test​::More (rather than t/test.pl).

But t/porting/test_bootstrap.t specifically tests that no t/**.t files
use Test​::More — adding the line​:

  use Test​::More;

into a test in t/porting/ (not one of the directories perlhack mentions
as requiring ad-hoc testing or test.pl) makes test_bootstrap.t fail.

These obviously contradict each other, and I don't know what should be
used for a new test in t/porting/.

Either test_bootstrap.t needs restricting to the directories mentioned
in perlhack, or the advice in perlhack needs changing to say never to
use Test​::More.



Flags​:
  category=core
  severity=low


Site configuration information for perl 5.19.4​:

Configured by smylers at Tue Sep 3 10​:58​:31 BST 2013.

Summary of my perl5 (revision 5 version 19 subversion 4) configuration​:
  Commit id​: 5565c73
  Platform​:
  osname=linux, osvers=3.8.0-30-generic, archname=x86_64-linux
  uname='linux fozzie 3.8.0-30-generic #44-ubuntu smp thu aug 22 20​:52​:24 utc 2013 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2',
  cppflags='-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.7.3', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
  libs=-lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
  libc=, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.17'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'


@​INC for perl 5.19.4​:
  lib
  /home/smylers/lib/perl5/site_perl
  /home/smylers/lib/perl5
  /usr/local/lib/perl5/site_perl/5.19.4/x86_64-linux
  /usr/local/lib/perl5/site_perl/5.19.4
  /usr/local/lib/perl5/5.19.4/x86_64-linux
  /usr/local/lib/perl5/5.19.4
  .


Environment for perl 5.19.4​:
  HOME=/home/smylers
  LANG=en_GB.utf8
  LANGUAGE=en_GB​:en
  LC_COLLATE=C
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/home/smylers/bin​:/usr/local/sbin​:/usr/local/bin​:/sbin​:/bin​:/usr/sbin​:/usr/bin​:/usr/X11R6/bin​:/usr/games
  PERL5LIB=/home/smylers/lib/perl5/site_perl​:/home/smylers/lib/perl5
  PERL_BADLANG (unset)
  PERL_CPANM_OPT=--sudo --prompt
  SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From @jkeenan

On Thu Sep 05 04​:44​:55 2013, smylers@​stripey.com wrote​:

This is a bug report for perl from smylers@​stripey.com,
generated with the help of perlbug 1.39 running under perl 5.19.4.

-----------------------------------------------------------------

perlhack, in its TESTING section, says that tests other than in
specific
subdirectories should use Test​::More (rather than t/test.pl).

But t/porting/test_bootstrap.t specifically tests that no t/**.t files
use Test​::More — adding the line​:

use Test​::More;

into a test in t/porting/ (not one of the directories perlhack
mentions
as requiring ad-hoc testing or test.pl) makes test_bootstrap.t fail.

These obviously contradict each other, and I don't know what should be
used for a new test in t/porting/.

Either test_bootstrap.t needs restricting to the directories mentioned
in perlhack, or the advice in perlhack needs changing to say never to
use Test​::More.

I can confirm that adding or editing a file in t/porting which contains
'use Test​::More' will cause t/porting/test_bootstrap.t to fail.

Would the patch attached suffice?

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

From @jkeenan

0001-Add-t-porting-to-list-of-directories-which-use-t-tes.patch
From c0f09e3f907fe356fdf7d7ac36c9bea024894e0d Mon Sep 17 00:00:00 2001
From: James E Keenan <jkeenan@cpan.org>
Date: Fri, 6 Sep 2013 01:12:30 +0200
Subject: [PATCH] Add t/porting to list of directories which use t/test.pl.

For: RT #119619
---
 pod/perlhack.pod |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/pod/perlhack.pod b/pod/perlhack.pod
index ff685d2..028849c 100644
--- a/pod/perlhack.pod
+++ b/pod/perlhack.pod
@@ -754,7 +754,7 @@ tested.  Tests in F<t/opbasic>, for instance, have been placed there
 rather than in F<t/op> because they test functionality which
 F<t/test.pl> presumes has already been demonstrated to work.
 
-=item * F<t/cmd>, F<t/run>, F<t/io> and F<t/op>
+=item * F<t/cmd>, F<t/run>, F<t/io>, F<t/op> and F<t/porting>
 
 Now that basic require() and subroutines are tested, you can use the
 F<t/test.pl> library.
-- 
1.7.1

@p5pRT
Copy link
Author

p5pRT commented Sep 5, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Sep 6, 2013

From @Smylers

James E Keenan via RT writes​:

On Thu Sep 05 04​:44​:55 2013, smylers@​stripey.com wrote​:

perlhack, in its TESTING section, says that tests other than in
specific subdirectories should use Test​::More (rather than
t/test.pl).

But t/porting/test_bootstrap.t specifically tests that no t/**.t
files use Test​::More — adding the line​:

use Test​::More;

into a test in t/porting/ (not one of the directories perlhack
mentions as requiring ad-hoc testing or test.pl) makes
test_bootstrap.t fail.

Note that t/porting/ is simply an example which demonstrates the
contradiction; the problem isn't restricted to t/porting. My apologies
if that wasn't clear in my bug report.

Here's the relevant text from perlhack​:

  · t/base, t/comp and t/opbasic

  ... use ad hoc tests for these three ...
 
  · t/cmd, t/run, t/io and t/op

  Now that basic require() and subroutines are tested, you can use
  the t/test.pl library. ...

  · Everything else

  Now that the core of Perl is tested, Test​::More can and should be
  used. ...

And here are t/ subdirectories which include .t files but aren't
mentioned in either of the first two bullet points above​:

  lib x2p benchmark mro bigmem porting uni japh re test_pl win32

These obviously contradict each other, and I don't know what should
be used for a new test in t/porting/.

Either test_bootstrap.t needs restricting to the directories
mentioned in perlhack, or the advice in perlhack needs changing to
say never to use Test​::More.

I can confirm that adding or editing a file in t/porting which
contains 'use Test​::More' will cause t/porting/test_bootstrap.t to
fail.

Would the patch attached suffice?

That patch simply moves t/porting into the second bullet point's list,
which obviously doesn't resolve the contradiction in general.

As far as I can tell, test_bootstrap.t prevents any test in t/ from
using Test​::More, and as such the ‘everything else’ bullet point in
perlhack can't ever apply.

So making the documentation consistent with test_bootstrap.t would
require entirely removing that third bullet point which mentions
Test​::More, and renaming the second one to say ‘everything else’ (that
is, everything but the three files mentioned in the first bullet point).

But it isn't clear to me that that's the right thing to do. The author
of that advice intended that Test​::More could be used in some
circumstances, and the reasoning sound plausible to me. As such, it may
be that adjusting test_bootstrap.t to match the advice (only prohibiting
Test​::More selectively) would be better.

So before any solution can be effected, somebody needs to make a
judgement call about what's desired here.

Cheers

Smylers
--
Stop drug companies hiding negative research results.
Sign the AllTrials petition to get all clinical research results published.
Read more​: http​://www.alltrials.net/blog/the-alltrials-campaign/

@Smylers
Copy link
Contributor

Smylers commented Feb 3, 2020

This contradiction still exists in blead: perlhack says to use Test::More, but if you do, test_bootstrap.t will object that you have done so.

Given tests seem to be existing without using Test::More, shall I submit a doc patch reversing the advice and saying not to use it?

@jkeenan
Copy link
Contributor

jkeenan commented Feb 3, 2020

This contradiction still exists in blead: perlhack says to use Test::More, but if you do, test_bootstrap.t will object that you have done so.

Given tests seem to be existing without using Test::More, shall I submit a doc patch reversing the advice and saying not to use it?

Since the last time we discussed this issue was more than six years ago, I'll have to re-read the ticket to respond adequately.

Thank you very much.
Jim Keenan

jkeenan added a commit that referenced this issue Feb 3, 2020
As reported by Smylers in RT 119619 (now GH 13231), the guidance
provided in 'perlhack' for usage of Test::More is misleading.  In
practice, we do not use Test::More in test files in the
subdirectories underneath 't/'.  In these directories we test
Perl 5's "pre-modular" functionality -- functionality which does not
depend on our having demonstrated that we can load code via 'use
MODULE'.  Usage of 'use MODULE' in 't/*/*.t' files has been largely
prohibited by a unit test in t/porting/bootstrap.t.

This patch clarifies that guidance.

For:  #13231
jkeenan added a commit that referenced this issue Feb 3, 2020
One of the objectives of t/porting/bootstrap.t has been to prohibit use of
testing modules which need to be loaded via 'use MODULE' in test files found
in subdirectories under 't/'.  In those subdirectories, we do not presume that
module loading via 'use MODULE' has been demonstrated to work.  However, the
unit test in bootstrap.t was written solely to spot instances of 'use
Test::More'.  It did not consider the possibility that a test file under 't/'
might 'use Test::Simple'.

There was one such test file which use-d Test::Simple:
t/lib/overload_nomethod.t.  This patch rewrites that file to use 't/test.pl'
(as other t/lib/*.t files customarily do) and adapts t/porting/bootstrap.t to
rule out 'use Test::Simple' as well as 'use Test::More' from '*.t' files
underneath 't/'.

For:  #13231
@jkeenan
Copy link
Contributor

jkeenan commented Feb 3, 2020

This contradiction still exists in blead: perlhack says to use Test::More, but if you do, test_bootstrap.t will object that you have done so.
Given tests seem to be existing without using Test::More, shall I submit a doc patch reversing the advice and saying not to use it?

Since the last time we discussed this issue was more than six years ago, I'll have to re-read the ticket to respond adequately.

Thank you very much.
Jim Keenan

I've now re-read this ticket and believe I understand your points. In the first patch in the jkeenan/ghi-13231-test-more-advice branch and in a p.r. to follow, I have modified pod/perlhack.pod to reflect this.

In the course of demonstrating to myself the correctness of your argument, I found one gotcha: a test under t/lib which, though it did not use Test::More, did presume that Test:: modules could be loaded via use MODULE. The test file in question said use Test::Simple. So in the second patch in the branch I modify that file require t/test.pl and adapt t/porting/test_bootstrap.t accordingly.

Please review.

Thank you very much.
Jim Keenan

@jkeenan
Copy link
Contributor

jkeenan commented Feb 3, 2020

See: #17524

jkeenan added a commit that referenced this issue Feb 4, 2020
As reported by Smylers in RT 119619 (now GH 13231), the guidance
provided in 'perlhack' for usage of Test::More is misleading.  In
practice, we do not use Test::More in test files in the
subdirectories underneath 't/'.  In these directories we test
Perl 5's "pre-modular" functionality -- functionality which does not
depend on our having demonstrated that we can load code via 'use
MODULE'.  Usage of 'use MODULE' in 't/*/*.t' files has been largely
prohibited by a unit test in t/porting/bootstrap.t.

This patch clarifies that guidance.

POD correction per Tony Cook

For:  #13231
jkeenan added a commit that referenced this issue Feb 4, 2020
One of the objectives of t/porting/bootstrap.t has been to prohibit use of
testing modules which need to be loaded via 'use MODULE' in test files found
in subdirectories under 't/'.  In those subdirectories, we do not presume that
module loading via 'use MODULE' has been demonstrated to work.  However, the
unit test in bootstrap.t was written solely to spot instances of 'use
Test::More'.  It did not consider the possibility that a test file under 't/'
might 'use Test::Simple'.

There was one such test file which use-d Test::Simple:
t/lib/overload_nomethod.t.  This patch rewrites that file to use 't/test.pl'
(as other t/lib/*.t files customarily do) and adapts t/porting/bootstrap.t to
rule out 'use Test::Simple' as well as 'use Test::More' from '*.t' files
underneath 't/'.

For:  #13231
@jkeenan
Copy link
Contributor

jkeenan commented Feb 4, 2020

Closed vi commits 915b794 and
070720f

@jkeenan jkeenan closed this as completed Feb 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants