Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod::Html: t/htmldir3.t test produces spurious pass #12271

Closed
p5pRT opened this issue Jul 13, 2012 · 13 comments
Closed

Pod::Html: t/htmldir3.t test produces spurious pass #12271

p5pRT opened this issue Jul 13, 2012 · 13 comments
Assignees
Labels
ext/Pod-Html issues in the blead-upstream Pod-Html distribution

Comments

@p5pRT
Copy link

p5pRT commented Jul 13, 2012

Migrated from rt.perl.org#114126 (status was 'open')

Searchable as RT114126$

@p5pRT
Copy link
Author

p5pRT commented Jul 13, 2012

From @jkeenan

This is a bug report for perl from jkeenan@​cpan.org,
generated with the help of perlbug 1.39 running under perl 5.16.0.

In the course of working on my fork of Pod-Html, I have come to a point
where all the code has been refactored into an object-oriented interface
and where all the original tests (revised to have the OO interface) are
passing -- except one.

Further investigation suggests that the one failing test is *only
accidentally passing* in the test suite found in ext/Pod-Html/t/ in the
Perl 5 core distribution.

You can see this for yourself by cd-ing to ext/Pod-Html/ in blead.
Call​:

#####
prove -Ilib t/*.t
#####

All tests should pass.

Now, copy into ext/Pod-Html/t/ the two files attached to this RT​:
xhtmldir3.t and yhtmldir3.t. xhtmldir3.t simply removes the first of
two invocations of the testing function convert_n_test() from
xhtmldir3.t. yhtmldir3.t switches the order of those two invocations of
convert_n_test().

If all the tests and testing functions were well
formulated in the first place, you would expect all tests to pass. Now
call​:

#####
prove -Ilib t/*.t
#####

The sole test in xhtmldir3.t and the first test in yhtmldir3.t should
now fail, with output like this​:

#####
t/xhtmldir3.t .. 1/?
# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 78.
# --- htmldir3_expected.tmp 2012-07-12 13​:15​:14.000000000 -0400
# +++ /Users/jimk/gitwork/perl/ext/Pod-Html/t/htmldir3.html
2012-07-12 13​:15​:14.000000000 -0400
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a
href="Users/jimk/gitwork/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a
href="Users/jimk/gitwork/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 1 test of 1.
t/xhtmldir3.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
t/yhtmldir3.t .. 1/2
# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 78.
# --- htmldir3_expected.tmp 2012-07-12 13​:15​:14.000000000 -0400
# +++ /Users/jimk/gitwork/perl/ext/Pod-Html/t/htmldir3.html
2012-07-12 13​:15​:14.000000000 -0400
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a
href="Users/jimk/gitwork/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a
href="Users/jimk/gitwork/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
t/yhtmldir3.t .. 2/2 # Looks like you failed 1 test of 2.
t/yhtmldir3.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests

Test Summary Report


t/xhtmldir3.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
t/yhtmldir3.t (Wstat​: 256 Tests​: 2 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
Files=19, Tests=37, 15 wallclock secs ( 0.36 usr 0.21 sys + 8.94 cusr
  2.54 csys = 12.05 CPU)
Result​: FAIL
#####

The differences between the expected HTML and the generated HTML which
are being reported are exactly the same as in the one failing test in my
github branch.

#####
# -<p>a link to <a
href="Users/jimk/gitwork/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#####

This suggests to me that the problem is in the way t/htmldir3.t is
formulated. My hunch is that the first test in t/htmldir3.t changes the
environment in some way, not yet understood, that enables the second
test to pass. But when we make that test the first
one in a file, it fails.

Thank you very much.
Jim Keenan


Flags​:
  category=library
  severity=low
  module=Pod​::Html


Site configuration information for perl 5.16.0​:

Configured by jimk at Sun May 20 20​:01​:26 EDT 2012.

Summary of my perl5 (revision 5 version 16 subversion 0) configuration​:

  Platform​:
  osname=darwin, osvers=8.11.0, archname=darwin-2level
  uname='darwin macintosh-8.local 8.11.0 darwin kernel version
8.11.0​: wed oct 10 18​:26​:00 pdt 2007; root​:xnu-792.24.17~1release_ppc
power macintosh powerpc '
  config_args='-des'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -fno-strict-aliasing
-pipe -I/usr/local/include -I/opt/local/include',
  optimize='-O3',
  cppflags='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe
-I/usr/local/include -I/opt/local/include'
  ccversion='', gccversion='4.0.1 (Apple Computer, Inc. build 5250)',
gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='
-L/usr/local/lib -L/opt/local/lib'
  libpth=/usr/local/lib /opt/local/lib /usr/lib
  libs=-ldbm -ldl -lm -lc
  perllibs=-ldl -lm -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-L/usr/local/lib -L/opt/local/lib'

Locally applied patches​:


@​INC for perl 5.16.0​:
  /usr/local/lib/perl5/site_perl/5.16.0/darwin-2level
  /usr/local/lib/perl5/site_perl/5.16.0
  /usr/local/lib/perl5/5.16.0/darwin-2level
  /usr/local/lib/perl5/5.16.0
  /usr/local/lib/perl5/site_perl/5.14.2
  /usr/local/lib/perl5/site_perl/5.14.0
  /usr/local/lib/perl5/site_perl/5.12.0
  /usr/local/lib/perl5/site_perl/5.10.1
  /usr/local/lib/perl5/site_perl/5.10.0
  /usr/local/lib/perl5/site_perl
  .


Environment for perl 5.16.0​:

DYLD_LIBRARY_PATH=/Users/jimk/work/pseudoinstall/lib​:/Users/jimk/gitwork/parrot/blib/lib
  HOME=/Users/jimk
  LANG (unset)
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)

PATH=/usr/local/bin​:/opt/local/bin​:/opt/local/sbin​:/usr/local/bin​:/opt/local/bin​:/opt/local/sbin​:/bin​:/sbin​:/usr/bin​:/usr/sbin​:/Users/jimk/bin​:/Users/jimk/bin/perl​:/Users/jimk/bin/c​:/Users/jimk/bin/shell​:/sw/lib​:/sw/bin​:/Users/jimk/bin​:/Users/jimk/bin/perl​:/Users/jimk/bin/c​:/Users/jimk/bin/shell​:/sw/lib​:/sw/bin
  PERL_BADLANG (unset)
  SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Jul 13, 2012

From @jkeenan

#!/usr/bin/perl -w # -*- perl -*-

BEGIN {
  require "t/pod2html-lib.pl";
}

END {
  rem_test_dir();
}

use strict;
use Cwd;
use File​::Spec;
use File​::Spec​::Functions;
use Test​::More tests => 2;

SKIP​: {
  my $output = make_test_dir();
  skip "$output", 2 if $output;

  my $cwd = cwd();
  my ($v, $d) = splitpath($cwd, 1);
  my $relcwd = substr($d, length(File​::Spec->rootdir()));

  my $data_pos = tell DATA; # to read <DATA> twice

  convert_n_test("htmldir3", "test --htmldir and --htmlroot 3b",
  "--podpath=". catdir($relcwd, 't'),
  "--podroot=$v". File​::Spec->rootdir,
  "--htmldir=t",
  "--outfile=t/htmldir3.html",
  "--quiet",
  );

  seek DATA, $data_pos, 0; # to read <DATA> twice (expected output is the same)

  convert_n_test("htmldir3", "test --htmldir and --htmlroot 3a",
  "--podpath=$relcwd",
  "--podroot=$v". File​::Spec->rootdir,
  "--htmldir=". catdir($cwd, 't', ''), # test removal trailing slash,
  "--quiet",
  );
}

__DATA__
<?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http​://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http​://www.w3.org/1999/xhtml">
<head>
<title></title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<link rev="made" href="mailto​:[PERLADMIN]" />
</head>

<body style="background-color​: white">

<ul id="index">
  <li><a href="#NAME">NAME</a></li>
  <li><a href="#LINKS">LINKS</a></li>
</ul>

<h1 id="NAME">NAME</h1>

<p>htmldir - Test --htmldir feature</p>

<h1 id="LINKS">LINKS</h1>

<p>Normal text, a <a>link</a> to nowhere,</p>

<p>a link to <a href="[RELCURRENTWORKINGDIRECTORY]/testdir/test.lib/var-copy.html">var-copy</a>,</p>

<p><a href="[RELCURRENTWORKINGDIRECTORY]/t/htmlescp.html">htmlescp</a>,</p>

<p><a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>,</p>

<p>and another <a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>.</p>

</body>

</html>

@p5pRT
Copy link
Author

p5pRT commented Jul 13, 2012

From @jkeenan

#!/usr/bin/perl -w # -*- perl -*-

BEGIN {
  require "t/pod2html-lib.pl";
}

END {
  rem_test_dir();
}

use strict;
use Cwd;
use File​::Spec;
use File​::Spec​::Functions;
use Test​::More qw(no_plan); # tests => 2;

SKIP​: {
  my $output = make_test_dir();
  skip "$output", 1 if $output;

  my $cwd = cwd();
  my ($v, $d) = splitpath($cwd, 1);
  my $relcwd = substr($d, length(File​::Spec->rootdir()));

  convert_n_test("htmldir3", "test --htmldir and --htmlroot 3b",
  "--podpath=". catdir($relcwd, 't'),
  "--podroot=$v". File​::Spec->rootdir,
  "--htmldir=t",
  "--outfile=t/htmldir3.html",
  "--quiet",
  );
}

__DATA__
<?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http​://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http​://www.w3.org/1999/xhtml">
<head>
<title></title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<link rev="made" href="mailto​:[PERLADMIN]" />
</head>

<body style="background-color​: white">

<ul id="index">
  <li><a href="#NAME">NAME</a></li>
  <li><a href="#LINKS">LINKS</a></li>
</ul>

<h1 id="NAME">NAME</h1>

<p>htmldir - Test --htmldir feature</p>

<h1 id="LINKS">LINKS</h1>

<p>Normal text, a <a>link</a> to nowhere,</p>

<p>a link to <a href="[RELCURRENTWORKINGDIRECTORY]/testdir/test.lib/var-copy.html">var-copy</a>,</p>

<p><a href="[RELCURRENTWORKINGDIRECTORY]/t/htmlescp.html">htmlescp</a>,</p>

<p><a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>,</p>

<p>and another <a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>.</p>

</body>

</html>

@p5pRT
Copy link
Author

p5pRT commented Jul 23, 2012

From @nwc10

On Fri, Jul 13, 2012 at 05​:35​:19AM -0700, James E Keenan wrote​:

Further investigation suggests that the one failing test is *only
accidentally passing* in the test suite found in ext/Pod-Html/t/ in the
Perl 5 core distribution.

This suggests to me that the problem is in the way t/htmldir3.t is
formulated. My hunch is that the first test in t/htmldir3.t changes the
environment in some way, not yet understood, that enables the second
test to pass. But when we make that test the first
one in a file, it fails.

I'm not clear what you'd like to happen as a result of this ticket.

Is it a request for someone else to investigate to give a second opinion
on the cause, and to to propose a solution?

Or is this more a "note to self" to come back and fix it at some point?

Nicholas Clark

@p5pRT
Copy link
Author

p5pRT commented Jul 23, 2012

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jul 23, 2012

From @jkeenan

On Mon Jul 23 02​:18​:36 2012, nicholas wrote​:

On Fri, Jul 13, 2012 at 05​:35​:19AM -0700, James E Keenan wrote​:

Further investigation suggests that the one failing test is *only
accidentally passing* in the test suite found in ext/Pod-Html/t/ in the
Perl 5 core distribution.

This suggests to me that the problem is in the way t/htmldir3.t is
formulated. My hunch is that the first test in t/htmldir3.t changes
the
environment in some way, not yet understood, that enables the second
test to pass. But when we make that test the first
one in a file, it fails.

I'm not clear what you'd like to happen as a result of this ticket.

Is it a request for someone else to investigate to give a second opinion
on the cause, and to to propose a solution?

Or is this more a "note to self" to come back and fix it at some point?

Both.

At the hackathon held at DuckDuckGo in Pennsylvania last month, I had
occasion to discuss Pod​::Html with Ricardo Signes. He suggested that it
would worth considering "dual-lifing" this module and I set up a fork on
my github to explore what this would look like. In the course of that
exploration I've encountered several issues for which I've opened RTs.
While I've proposed patches for some of them, there's a lot about
Pod​::Html I don't yet understand. For those cases I'm flagging the
issues in the hope that someone else can diagnose the problem sooner
than I can.

In this particular case, it would be good to (a) get an explanation of
what the purpose of the misleading test is; (b) develop an understanding
of what is changing in the testing environment that produces the
spurious pass.

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Jul 24, 2012

From @rjbs

* James E Keenan via RT <perlbug-followup@​perl.org> [2012-07-23T08​:03​:47]

In this particular case, it would be good to (a) get an explanation of
what the purpose of the misleading test is; (b) develop an understanding
of what is changing in the testing environment that produces the
spurious pass.

It would be good to get the input of Marc Green, who has probably done the most
fiddling with Pod​::Html recently.

cc'd.

via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=114126

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented Jan 1, 2014

From @jkeenan

On Mon Jul 23 05​:03​:47 2012, jkeenan wrote​:

On Mon Jul 23 02​:18​:36 2012, nicholas wrote​:

On Fri, Jul 13, 2012 at 05​:35​:19AM -0700, James E Keenan wrote​:

Further investigation suggests that the one failing test is *only
accidentally passing* in the test suite found in ext/Pod-Html/t/ in the
Perl 5 core distribution.

This suggests to me that the problem is in the way t/htmldir3.t is
formulated. My hunch is that the first test in t/htmldir3.t changes
the
environment in some way, not yet understood, that enables the second
test to pass. But when we make that test the first
one in a file, it fails.

I'm not clear what you'd like to happen as a result of this ticket.

Is it a request for someone else to investigate to give a second opinion
on the cause, and to to propose a solution?

Or is this more a "note to self" to come back and fix it at some point?

Both.

At the hackathon held at DuckDuckGo in Pennsylvania last month, I had
occasion to discuss Pod​::Html with Ricardo Signes. He suggested that it
would worth considering "dual-lifing" this module and I set up a fork on
my github to explore what this would look like. In the course of that
exploration I've encountered several issues for which I've opened RTs.
While I've proposed patches for some of them, there's a lot about
Pod​::Html I don't yet understand. For those cases I'm flagging the
issues in the hope that someone else can diagnose the problem sooner
than I can.

In this particular case, it would be good to (a) get an explanation of
what the purpose of the misleading test is; (b) develop an understanding
of what is changing in the testing environment that produces the
spurious pass.

Thank you very much.
Jim Keenan

Two calendar years later ... and this just got weirder!

I created a branch on the dromedary test server today and added the two test files (with minor updatings) originally posted in this ticket.

I then configure, built and tested perl in that branch. As expected, the two additional files failed.

I then repeatedly ran this set of commands​:

#####
cd t;./perl harness -v ../ext/Pod-Html/t/*.t; cd-
#####

... and got different results each time!

t/xhtmldir3.t and t/yhtmldir3.t failed each time, as expected. But each repetition generated failures in 0 to 2 *other*, previously passing files! The output is in the attachment.

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Jan 1, 2014

From @jkeenan

On dromedary​: branch​: 114126-pod-html-accidental-pass
Forked from blead at 2d66f61

Test Summary Report


../ext/Pod-Html/t/xhtmldir3.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/yhtmldir3.t (Wstat​: 256 Tests​: 2 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
Files=2366, Tests=683642, 124 wallclock secs (71.71 usr 7.55 sys + 441.97 cusr 38.76 csys = 559.99 CPU)
Result​: FAIL
make​: *** [test_harness] Error 2

okay, that's per #114126.

But now let's run the harness over that directory​:

$ cd t;./perl harness -v ../ext/Pod-Html/t/*.t;cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 1;0 0/? 0/? 0/? 0/? 0/3 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 2;0 1/10 0/? 0/1 0/1 0/3 0/? 0/1 0/? )==============Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 4;0 2/10 0/1 1/1 0/1 0/3 0/? 0/1 0/? )==============
# Failed test 'Cache created'
# at t/cache.t line 33.
# got​: undef
# expected​: '1'
Cannot open cache file​: No such file or directory at t/cache.t line 34.
# Looks like you planned 10 tests but ran 3.
# Looks like you failed 1 test of 3 run.
# Looks like your test exited with 2 just after 3.
===( 6;0 3/10 0/1 1/1 1/1 0/3 0/? 0/1 0/? )==============ok
===( 6;0 3/10 0/1 1/1 0/3 0/? 0/1 0/? )===================Dubious, test returned 2 (wstat 512, 0x200)
Failed 8/10 subtests
===( 6;0 0/1 1/1 0/3 0/? 0/1 0/? 0/? )=====================ok
===( 8;0 0/1 1/3 0/2 1/1 0/? 0/? 0/? 0/? )================ok
===( 9;0 0/1 2/3 0/2 0/? 0/? 0/? 0/? )=====================ok
===( 10;0 0/1 0/2 0/? 0/? 0/? 0/? 0/? )=====================
# Failed test 'cross references'
# at t/pod2html-lib.pl line 82.
# --- crossref_expected.tmp 2014-01-01 21​:23​:59.768167006 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/crossref.html 2014-01-01 21​:23​:59.767167007 +0100
# @​@​ -35,15 +35,15 @​@​
#
# <p><a href="#non-existant-section">"non existant section"</a></p>
#
# -<p><a href="../testdir/test.lib/var-copy.html">var-copy</a></p>
# +<p><a>var-copy</a></p>
#
# -<p><a href="../testdir/test.lib/var-copy.html#pod">"$"" in var-copy</a></p>
# +<p><a>"$"" in var-copy</a></p>
#
# <p><code>var-copy</code></p>
#
# <p><code>var-copy/$"</code></p>
#
# -<p><a href="../testdir/test.lib/podspec-copy.html#First">"First​:" in podspec-copy</a></p>
# +<p><a>"First​:" in podspec-copy</a></p>
#
# <p><code>podspec-copy/First​:</code></p>
#
# Looks like you failed 1 test of 1.
===( 11;0 1/1 0/2 0/? 0/? 0/? 0/? 0/? 0/? )================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
===( 12;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 14;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 15;0 0/? 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 20;0 2/3 1/2 1/2 0/1 1/1 0/1 0/1 0/? )================ok
===( 22;0 3/3 1/2 2/2 0/1 0/1 0/1 0/? )=====================ok
===( 22;0 1/2 2/2 0/1 0/1 0/1 0/? 0/? )=====================ok
===( 24;0 2/2 1/1 0/1 0/1 0/? 0/? 0/? 0/? )================ok
===( 24;0 1/1 0/1 0/1 0/? 0/? 0/? 0/? )=====================ok
===( 26;0 1/1 1/1 0/? 0/? 0/? 0/? )==========================ok
===( 26;0 1/1 0/? 0/? 0/? 0/? )===============================ok
===( 27;0 1/1 0/? 0/? 0/? )====================================ok
===( 27;1 0/? 0/? 0/? )=========================================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 28;1 1/1 0/? 0/2 )=========================================ok
===( 28;1 0/? 0/2 )==============================================
# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 82.
===( 29;1 1/? 0/2 )==============================================# --- htmldir3_expected.tmp 2014-01-01 21​:24​:00.043166881 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:24​:00.043166880 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 1 test of 1.

# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 82.
===( 30;1 1/1 1/2 )==============================================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
../ext/Pod-Html/t/yhtmldir3.t .. # --- htmldir3_expected.tmp 2014-01-01 21​:24​:00.046166879 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:24​:00.046166879 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#

# Failed test 'test --htmldir and --htmlroot 3a'
# at t/pod2html-lib.pl line 82.

not ok 2 - test --htmldir and --htmlroot 3a
# --- htmldir3_expected.tmp 2014-01-01 21​:24​:00.064166871 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:24​:00.063166871 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 2 tests of 2.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests

Test Summary Report


../ext/Pod-Html/t/cache.t (Wstat​: 512 Tests​: 3 Failed​: 1)
  Failed test​: 3
  Non-zero exit status​: 2
  Parse errors​: Bad plan. You planned 10 tests but ran 3.
../ext/Pod-Html/t/crossref2.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/xhtmldir3.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/yhtmldir3.t (Wstat​: 512 Tests​: 2 Failed​: 2)
  Failed tests​: 1-2
  Non-zero exit status​: 2
Files=20, Tests=31, 1 wallclock secs ( 0.07 usr 0.02 sys + 2.38 cusr 0.30 csys = 2.77 CPU)
Result​: FAIL
/home/jkeenan/perl

2 more files are failing!

Repeat.

$ cd t;./perl harness -v ../ext/Pod-Html/t/*.t;cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 2;0 1/10 0/? 0/? 0/? 0/3 0/? 0/? 0/? )==============Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
t/feature2.t​: error opening ./pod2htmd.tmp for reading​: No such file or directory
# Looks like your test exited with 2 before it could output anything.
===( 10;0 8/10 0/1 0/1 0/1 1/3 0/2 0/1 0/? )==============Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests
===( 14;0 10/10 0/1 0/1 0/1 3/3 0/1 0/? )===================ok
===( 15;0 0/1 1/1 0/1 3/3 0/1 0/? 0/? 0/? )================ok
===( 15;0 0/1 1/1 0/1 0/1 0/? 0/? 0/? )=====================ok
===( 18;0 1/1 1/1 1/1 0/? 0/? 0/? 0/? 0/? )================ok
===( 18;0 1/1 1/1 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 18;0 1/1 0/? 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 19;1 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 20;1 0/3 0/2 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 26;1 2/3 1/2 1/2 1/1 1/1 0/1 0/1 0/? )================ok
===( 27;1 2/3 1/2 2/2 1/1 0/1 0/1 0/? )=====================ok
===( 27;1 2/3 1/2 2/2 0/1 0/1 0/? 0/? )=====================ok
===( 28;1 2/3 2/2 0/1 0/1 0/? 0/? 0/? )=====================ok
===( 28;1 2/3 0/1 0/1 0/? 0/? 0/? 0/? )=====================ok
===( 31;1 1/1 1/1 0/? 0/? 0/? 0/? )==========================ok
===( 31;1 1/1 0/? 0/? 0/? 0/? )===============================ok
===( 32;1 1/1 0/? 0/? 0/? )====================================ok
===( 32;1 0/? 0/? 0/? )=========================================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 33;1 1/1 0/? 0/2 )=========================================ok
===( 33;1 0/? 0/2 )==============================================
# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 82.
===( 34;1 1/? 0/2 )==============================================# --- htmldir3_expected.tmp 2014-01-01 21​:25​:01.240138807 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:25​:01.240138807 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 1 test of 1.

# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 82.
===( 35;1 1/1 1/2 )==============================================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
# --- htmldir3_expected.tmp 2014-01-01 21​:25​:01.245138803 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:25​:01.244138803 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
../ext/Pod-Html/t/yhtmldir3.t ..
# Failed test 'test --htmldir and --htmlroot 3a'
# at t/pod2html-lib.pl line 82.

not ok 2 - test --htmldir and --htmlroot 3a
# --- htmldir3_expected.tmp 2014-01-01 21​:25​:01.261138798 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:25​:01.260138798 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 2 tests of 2.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests

Test Summary Report


../ext/Pod-Html/t/feature2.t (Wstat​: 512 Tests​: 0 Failed​: 0)
  Non-zero exit status​: 2
  Parse errors​: Bad plan. You planned 2 tests but ran 0.
../ext/Pod-Html/t/xhtmldir3.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/yhtmldir3.t (Wstat​: 512 Tests​: 2 Failed​: 2)
  Failed tests​: 1-2
  Non-zero exit status​: 2
Files=20, Tests=36, 1 wallclock secs ( 0.06 usr 0.02 sys + 2.48 cusr 0.31 csys = 2.87 CPU)
Result​: FAIL
/home/jkeenan/perl

cache.t and crossref2.t are once again passing -- but now feature2.t is failing!

Repeat once more.

$ cd t;./perl harness -v ../ext/Pod-Html/t/*.t;cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 1;0 0/? 0/? 0/? 0/? 0/3 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 3;1 0/? 0/? 1/1 1/1 0/3 0/? 0/1 0/? )================ok
===( 3;1 0/? 0/? 1/1 0/3 0/? 0/1 0/? )=====================ok
===( 5;1 1/10 0/? 0/3 0/? 1/1 0/? 0/? 0/? )==============ok
===( 10;1 5/10 0/1 1/3 0/2 0/? 0/? 0/? 0/? )==============ok
===( 12;1 5/10 0/1 0/2 0/? 0/? 0/? 0/? )===================ok
===( 18;1 0/1 1/2 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 20;1 1/1 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 21;1 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 25;1 1/3 1/2 1/2 0/? 0/? 0/? 0/? 0/? )================ok
===( 28;1 3/3 1/2 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 28;1 1/2 0/? 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 30;1 0/1 1/1 0/? 0/1 0/1 0/? 0/? 0/? )================ok
===( 31;1 1/1 0/1 0/1 0/1 0/? 0/? 0/? )=====================ok
===( 32;1 0/1 0/1 1/1 0/? 0/? 0/? )==========================ok
===( 34;1 1/1 1/1 0/? 0/? 0/? )===============================ok
===( 34;1 1/1 0/? 0/? 0/? )====================================ok
===( 34;1 0/? 0/? 0/? )=========================================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.

# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 82.
===( 35;1 0/1 1/? 0/2 )=========================================# --- htmldir3_expected.tmp 2014-01-01 21​:26​:10.249107428 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:26​:10.248107429 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 1 test of 1.

# Failed test 'test --htmldir and --htmlroot 3b'
# at t/pod2html-lib.pl line 82.
===( 37;1 1/1 1/1 1/2 )=========================================# --- htmldir3_expected.tmp 2014-01-01 21​:26​:10.252107427 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:26​:10.252107426 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
===( 37;1 1/1 1/2 )==============================================ok
../ext/Pod-Html/t/yhtmldir3.t ..
# Failed test 'test --htmldir and --htmlroot 3a'
# at t/pod2html-lib.pl line 82.

not ok 2 - test --htmldir and --htmlroot 3a
# --- htmldir3_expected.tmp 2014-01-01 21​:26​:10.269107418 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir3.html 2014-01-01 21​:26​:10.268107419 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 2 tests of 2.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests

Test Summary Report


../ext/Pod-Html/t/xhtmldir3.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/yhtmldir3.t (Wstat​: 512 Tests​: 2 Failed​: 2)
  Failed tests​: 1-2
  Non-zero exit status​: 2
Files=20, Tests=38, 1 wallclock secs ( 0.07 usr 0.01 sys + 2.08 cusr 0.26 csys = 2.42 CPU)
Result​: FAIL
/home/jkeenan/perl

Now we're back to just x/y htmldir3.t failing!

Further repetitions vary. Often 3 tests fail​: x/y and one of cache, crossref2, feature2.

@p5pRT
Copy link
Author

p5pRT commented Jan 8, 2014

From @jkeenan

On Wed Jan 01 12​:35​:53 2014, jkeenan wrote​:

Two calendar years later ... and this just got weirder!

I created a branch on the dromedary test server today and added the
two test files (with minor updatings) originally posted in this
ticket.

I then configure, built and tested perl in that branch. As expected,
the two additional files failed.

I then repeatedly ran this set of commands​:

#####
cd t;./perl harness -v ../ext/Pod-Html/t/*.t; cd-
#####

... and got different results each time!

t/xhtmldir3.t and t/yhtmldir3.t failed each time, as expected. But
each repetition generated failures in 0 to 2 *other*, previously
passing files! The output is in the attachment.

More data. Tonight I built blead on the dromedary server. All tests, including those in ext/Pod-Html/t/, passed. (The command I run there is​:
TEST_JOBS=${TEST_JOBS} make -j${TEST_JOBS} test_harness
... with TEST_JOBS=8.)

I then ran those same tests with the harness 8 times -- and got 8 different results.

#####
cd t;./perl harness -v ../ext/Pod-Html/t/*.t; cd-
#####

Results attached.

It is interesting to note that many, but not all, of the test failures in the attachment are failures like these two​:

#####
# -<p><a href="../testdir/test.lib/var-copy.html">var-copy</a></p>
# +<p><a>var-copy</a></p>
#
# -<p><a href="../testdir/test.lib/var-copy.html#pod">"$"" in var-copy</a></p>
# +<p><a>"$"" in var-copy</a></p>
#####

That is, the output expected -- the '-' lines in the diff excerpt above -- is an <a> tag where something like the following has been interpolated in​:

#####
href="../testdir/test.lib/var-copy.html
#####

But what is actually generated is simply​: <a>

In my attempt to make Pod-Html CPANnable this type of test failure is one I'm getting in many files -- but at least I'm getting them consistently. And it's the only type of failure I'm still getting. So, while there may well be errors in my refactoring of the code, the fundamental problem may have been there all along but simply have been masked.

Thank you very much.
Jim Keenan

@p5pRT
Copy link
Author

p5pRT commented Jan 8, 2014

From @jkeenan

$ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 1;0 1/1 0/? 0/? 0/1 0/1 0/3 0/? 0/? )================ok
===( 4;0 2/10 0/? 1/1 0/1 0/3 0/? 0/1 0/? )==============ok
===( 4;0 2/10 0/? 0/1 0/3 0/? 0/1 0/? )===================
# Failed test 'podpath'
# at t/cache.t line 38.
# got​: 't'
# expected​: 'scooby​:shaggy​:fred​:velma​:daphne'
===( 7;0 3/10 0/1 1/1 0/3 0/? 1/1 0/? 0/? )==============ok
===( 7;0 3/10 0/1 0/3 0/? 1/1 0/? 0/? )===================ok
===( 7;0 3/10 0/1 0/3 0/2 0/? 0/? 0/? )===================# Looks like you failed 1 test of 10.
===( 9;0 4/10 0/1 1/3 0/2 0/? 0/? 0/? 0/? )==============ok
===( 17;0 10/10 0/1 0/2 0/? 0/? 0/? 0/? )===================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/10 subtests
===( 17;0 0/1 0/2 0/? 0/? 0/? 0/? 0/? )=====================
# Failed test 'cross references'
# at t/pod2html-lib.pl line 82.
===( 18;0 1/1 0/2 0/? 0/? 0/? 0/? 0/? 0/? )================# --- crossref_expected.tmp 2014-01-08 02​:58​:30.186060121 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/crossref.html 2014-01-08 02​:58​:30.185060124 +0100
# @​@​ -35,15 +35,15 @​@​
#
# <p><a href="#non-existant-section">"non existant section"</a></p>
#
# -<p><a href="../testdir/test.lib/var-copy.html">var-copy</a></p>
# +<p><a>var-copy</a></p>
#
# -<p><a href="../testdir/test.lib/var-copy.html#pod">"$"" in var-copy</a></p>
# +<p><a>"$"" in var-copy</a></p>
#
# <p><code>var-copy</code></p>
#
# <p><code>var-copy/$"</code></p>
#
# -<p><a href="../testdir/test.lib/podspec-copy.html#First">"First​:" in podspec-copy</a></p>
# +<p><a>"First​:" in podspec-copy</a></p>
#
# <p><code>podspec-copy/First​:</code></p>
#
# Looks like you failed 1 test of 1.
===( 19;0 1/1 1/2 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 20;0 1/1 0/? 0/? 0/? 0/? 0/? 0/? )=====================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
===( 23;0 2/2 1/3 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 23;0 1/3 0/? 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 25;0 0/? 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 26;0 0/2 0/2 0/1 1/1 0/? 0/1 0/? 0/? )================ok
===( 29;0 0/2 1/2 1/1 0/1 1/1 0/? 0/? )=====================ok
===( 30;0 1/2 1/2 1/1 0/1 0/? 0/? )==========================ok
===( 30;0 1/2 1/2 0/1 0/? 0/? )===============================ok
===( 31;0 1/2 0/1 0/? 0/? )====================================ok
===( 33;0 1/1 0/? 0/? )=========================================ok
===( 34;0 1/1 0/? )==============================================ok
../ext/Pod-Html/t/podnoerr.t ...
ok 1 - pod error section
ok

Test Summary Report


../ext/Pod-Html/t/cache.t (Wstat​: 256 Tests​: 10 Failed​: 1)
  Failed test​: 4
  Non-zero exit status​: 1
../ext/Pod-Html/t/crossref2.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
Files=18, Tests=35, 0 wallclock secs ( 0.06 usr 0.02 sys + 1.72 cusr 0.20 csys = 2.00 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 2;0 1/1 0/? 0/? 1/1 0/1 0/3 0/? 0/1 )================ok
===( 2;0 1/1 0/? 0/? 0/1 0/3 0/? 0/1 )=====================ok
===( 3;0 0/? 0/? 1/1 0/3 0/? 0/1 0/? 0/? )================ok
===( 4;0 0/? 0/? 0/3 0/? 1/1 0/? 0/? 0/? )================ok
===( 5;0 1/10 0/? 0/3 0/? 0/? 0/? 0/? 0/? )==============
# Failed test 'podpath'
# at t/cache.t line 38.
# got​: ''
# expected​: 'scooby​:shaggy​:fred​:velma​:daphne'

# Failed test 'podroot'
# at t/cache.t line 39.
# got​: '.'
# expected​: '/home/jkeenan/perl/ext/Pod-Html'
===( 10;0 5/10 0/1 1/3 0/2 0/? 0/? 0/? 0/? )==============ok
===( 13;0 5/10 1/1 0/2 0/? 0/? 0/? 0/? 0/? )==============# Looks like you failed 2 tests of 10.

# Failed test 'misc pod-html --verbose warnings'
# at t/feature2.t line 27.
# 'scanning for directory cache
# scanning for directory cache
# caching directories for later use
# Converting input file /home/jkeenan/perl/ext/Pod-Html/t/feature2.pod
# '
# doesn't match '(?^x​:
# \Acaching\ directories\ for\ later\ use\n
# Converting\ input\ file\ \S+[/\\\]]feature2\.pod\n\z
# )'
# Looks like you failed 1 test of 2.
ok
===( 18;0 10/10 0/2 0/? 0/? 0/? 0/? 0/? )===================Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/10 subtests
===( 19;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
===( 20;0 0/? 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.

# Failed test 'test --htmldir and --htmlroot 1a'
# at t/pod2html-lib.pl line 82.
===( 21;0 1/2 0/3 0/2 0/? 0/? 0/? 0/? 0/? )================# --- htmldir1_expected.tmp 2014-01-08 02​:58​:54.960990762 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir1.html 2014-01-08 02​:58​:54.960990762 +0100
# @​@​ -26,7 +26,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="/home/jkeenan/perl/ext/Pod-Html/t/htmlescp.html">htmlescp</a>,</p>
#
===( 24;0 1/2 2/3 1/2 0/? 0/? 0/? 0/? 0/? )================# Looks like you failed 1 test of 2.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
===( 25;0 2/3 1/2 0/2 0/? 0/? 0/? 0/? )=====================ok
===( 28;0 2/2 1/2 0/1 0/1 0/? 0/? 0/? 0/? )================ok
===( 28;0 1/2 0/1 0/1 0/? 0/? 0/? 0/? )=====================
# Failed test 'test --htmldir and --htmlroot 5'
# at t/pod2html-lib.pl line 82.
===( 29;0 1/2 1/1 0/1 0/1 0/1 0/? 0/? )=====================# --- htmldir5_expected.tmp 2014-01-08 02​:58​:54.991990675 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/htmldir5.html 2014-01-08 02​:58​:54.990990678 +0100
# @​@​ -24,7 +24,7 @​@​
#
# <p>Normal text, a <a>link</a> to nowhere,</p>
#
# -<p>a link to <a href="../testdir/test.lib/var-copy.html">var-copy</a>,</p>
# +<p>a link to <a>var-copy</a>,</p>
#
# <p><a href="./htmlescp.html">htmlescp</a>,</p>
#
# Looks like you failed 1 test of 1.
===( 30;0 1/2 1/1 1/1 0/1 0/1 0/? 0/? )=====================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
===( 30;0 1/2 1/1 0/1 0/1 0/? 0/? )==========================ok
===( 30;0 1/2 0/1 0/1 0/? 0/? )===============================ok
===( 33;1 1/1 1/1 0/? 0/? )====================================ok
===( 33;1 1/1 0/? 0/? )=========================================ok
===( 35;1 1/1 1/1 )==============================================ok
../ext/Pod-Html/t/podnoerr.t ... ok

Test Summary Report


../ext/Pod-Html/t/cache.t (Wstat​: 512 Tests​: 10 Failed​: 2)
  Failed tests​: 4-5
  Non-zero exit status​: 2
../ext/Pod-Html/t/feature2.t (Wstat​: 256 Tests​: 2 Failed​: 1)
  Failed test​: 2
  Non-zero exit status​: 1
../ext/Pod-Html/t/htmldir1.t (Wstat​: 256 Tests​: 2 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/htmldir5.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
Files=18, Tests=35, 1 wallclock secs ( 0.05 usr 0.01 sys + 1.68 cusr 0.21 csys = 1.95 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 2;0 1/10 0/? 0/? 0/? 0/3 0/? 0/? 0/? )==============Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
ok
===( 16;0 0/1 1/1 0/1 3/3 1/2 0/1 0/? 0/? )================ok
===( 18;0 1/1 1/1 0/1 3/3 0/1 0/? 0/? 0/? )================ok
===( 18;0 1/1 0/1 3/3 0/1 0/? 0/? 0/? )=====================ok
===( 19;0 1/1 3/3 0/1 0/? 0/? 0/? 0/? 0/? )================ok
===( 19;0 1/1 0/1 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 20;0 1/1 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 21;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 22;0 0/3 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 26;0 2/3 1/2 0/2 0/1 1/1 0/1 0/1 0/? )================ok
===( 28;0 2/3 1/2 1/2 1/1 0/1 0/1 0/? 0/? )================ok
===( 28;0 2/3 1/2 1/2 0/1 0/1 0/? 0/? )=====================ok
===( 31;0 3/3 1/2 1/1 0/1 0/? 0/? )==========================ok
===( 31;0 1/2 1/1 0/1 0/? 0/? )===============================ok
===( 32;0 1/2 1/1 0/? 0/? )====================================ok
===( 32;0 1/2 0/? 0/? )=========================================ok
===( 34;0 1/1 0/? )==============================================ok
../ext/Pod-Html/t/podnoerr.t ...
ok 1 - pod error section
ok
All tests successful.
Files=18, Tests=35, 0 wallclock secs ( 0.05 usr 0.01 sys + 1.48 cusr 0.18 csys = 1.72 CPU)
Result​: PASS
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 1;0 0/? 0/? 0/? 0/? 0/3 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 2;0 1/10 0/1 0/1 0/1 0/3 0/? 0/? 0/? )==============
# Failed test 'podpath'
# at t/cache.t line 38.
# got​: ''
# expected​: 'scooby​:shaggy​:fred​:velma​:daphne'

# Failed test 'podroot'
# at t/cache.t line 39.
# got​: '.'
# expected​: '/home/jkeenan/perl/ext/Pod-Html'

# Failed test 'cross references'
# at t/pod2html-lib.pl line 82.
===( 7;0 5/10 0/1 1/1 0/1 0/3 0/? 0/1 0/? )==============# --- crossref_expected.tmp 2014-01-08 02​:59​:14.307936532 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/crossref.html 2014-01-08 02​:59​:14.307936532 +0100
# @​@​ -35,15 +35,15 @​@​
#
# <p><a href="#non-existant-section">"non existant section"</a></p>
#
# -<p><a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a></p>
# +<p><a>var-copy</a></p>
#
# -<p><a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html#pod">"$"" in var-copy</a></p>
# +<p><a>"$"" in var-copy</a></p>
#
# <p><code>var-copy</code></p>
#
# <p><code>var-copy/$"</code></p>
#
# -<p><a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/podspec-copy.html#First">"First​:" in podspec-copy</a></p>
# +<p><a>"First​:" in podspec-copy</a></p>
#
# <p><code>podspec-copy/First​:</code></p>
#
# Looks like you failed 1 test of 1.
===( 8;0 5/10 0/1 1/1 0/1 1/3 0/2 0/1 0/? )==============ok
===( 10;0 5/10 0/1 1/1 0/1 0/2 0/1 0/? )===================# Looks like you failed 2 tests of 10.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
===( 15;0 7/10 1/1 1/1 0/2 1/1 0/? 0/? 0/? )==============ok
===( 16;0 8/10 1/1 0/2 1/1 0/? 0/? 0/? )===================ok
===( 17;0 9/10 0/2 1/1 0/? 0/? 0/? 0/? )===================ok
===( 18;0 10/10 0/2 0/? 0/? 0/? 0/? 0/? )===================Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/10 subtests
===( 19;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 21;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
Error removing temporary directory 'testdir/test.lib' at t/pod2html-lib.pl line 27.
===( 23;0 2/2 0/3 1/2 0/2 0/1 0/1 0/? 0/? )================ok
===( 27;0 2/3 1/2 1/2 0/1 1/1 0/1 0/? 0/? )================ok
===( 28;0 2/3 2/2 1/2 0/1 0/1 0/1 0/? )=====================ok
===( 29;0 2/3 1/2 1/1 0/1 0/1 0/? 0/? )=====================ok
===( 29;0 2/3 1/2 0/1 0/1 0/? 0/? )==========================ok
===( 31;0 2/2 0/1 0/1 0/? 0/? )===============================ok
===( 32;0 1/1 0/1 0/? 0/? )====================================ok
===( 33;0 1/1 0/? 0/? )=========================================ok
===( 35;0 1/1 1/1 )==============================================ok
../ext/Pod-Html/t/poderr.t ..... ok

Test Summary Report


../ext/Pod-Html/t/crossref3.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
../ext/Pod-Html/t/cache.t (Wstat​: 512 Tests​: 10 Failed​: 2)
  Failed tests​: 4-5
  Non-zero exit status​: 2
Files=18, Tests=35, 0 wallclock secs ( 0.06 usr 0.02 sys + 1.53 cusr 0.22 csys = 1.83 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
cannot remove directory for testdir/test.lib​: Directory not empty at t/pod2html-lib.pl line 27.
ok
===( 2;0 0/? 0/? 1/1 0/1 0/3 0/? 0/? 0/? )================ok
===( 3;0 1/10 0/? 0/1 0/3 0/? 0/1 0/? 0/? )==============
# Failed test 'cross references'
# at t/pod2html-lib.pl line 82.
===( 5;0 2/10 0/? 1/1 0/3 0/? 0/1 0/? 0/? )==============# --- crossref_expected.tmp 2014-01-08 02​:59​:24.314908458 +0100
# +++ /home/jkeenan/perl/ext/Pod-Html/t/crossref.html 2014-01-08 02​:59​:24.313908461 +0100
# @​@​ -35,15 +35,15 @​@​
#
# <p><a href="#non-existant-section">"non existant section"</a></p>
#
# -<p><a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html">var-copy</a></p>
# +<p><a>var-copy</a></p>
#
# -<p><a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/var-copy.html#pod">"$"" in var-copy</a></p>
# +<p><a>"$"" in var-copy</a></p>
#
# <p><code>var-copy</code></p>
#
# <p><code>var-copy/$"</code></p>
#
# -<p><a href="/home/jkeenan/perl/ext/Pod-Html/testdir/test.lib/podspec-copy.html#First">"First​:" in podspec-copy</a></p>
# +<p><a>"First​:" in podspec-copy</a></p>
#
# <p><code>podspec-copy/First​:</code></p>
#
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
===( 12;0 7/10 0/1 1/3 0/2 1/1 0/? 0/? 0/? )==============ok
===( 12;0 7/10 0/1 1/3 0/2 0/? 0/? 0/? )===================ok
===( 17;0 10/10 0/1 0/2 0/? 0/? 0/? 0/? )===================ok
===( 18;0 0/1 1/2 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 20;0 1/1 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 22;0 1/2 1/3 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 24;0 2/3 0/? 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 26;0 0/2 0/2 0/? 1/1 0/? 0/? 0/? 0/? )================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
ok
===( 29;0 1/2 1/2 1/1 0/1 0/1 0/? 0/? )=====================ok
===( 30;0 2/2 1/2 0/1 0/1 0/? 0/? )==========================ok
===( 31;0 2/2 0/1 0/1 0/? 0/? )===============================ok
===( 32;0 1/1 0/1 0/? 0/? )====================================ok
===( 33;0 1/1 0/? 0/? )=========================================ok
===( 35;0 1/1 1/1 )==============================================ok
../ext/Pod-Html/t/poderr.t ..... ok

Test Summary Report


../ext/Pod-Html/t/crossref.t (Wstat​: 256 Tests​: 1 Failed​: 1)
  Failed test​: 1
  Non-zero exit status​: 1
Files=18, Tests=35, 0 wallclock secs ( 0.05 usr 0.02 sys + 1.67 cusr 0.21 csys = 1.95 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 1;0 1/1 0/? 0/? 0/1 0/1 0/3 0/? 0/? )================ok
===( 4;0 2/10 0/? 1/1 0/1 0/3 0/? 0/1 0/? )==============ok
===( 4;0 2/10 0/1 0/1 0/3 0/? 0/1 0/? )===================
# Failed test 'podpath'
# at t/cache.t line 38.
# got​: 't'
# expected​: 'scooby​:shaggy​:fred​:velma​:daphne'
===( 6;0 3/10 0/1 1/1 0/3 0/? 0/1 0/? 0/? )==============ok
===( 7;0 3/10 0/1 0/3 0/? 1/1 0/? 0/? 0/? )==============ok
===( 8;0 3/10 0/1 1/3 0/2 0/? 0/? 0/? 0/? )==============ok
===( 10;0 3/10 0/1 0/2 0/? 0/? 0/? 0/? )===================# Looks like you failed 1 test of 10.
===( 14;0 6/10 1/1 0/2 0/? 0/? 0/? 0/? 0/? )==============Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/10 subtests
===( 18;0 1/1 0/2 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 19;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 20;0 0/? 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 21;0 1/2 0/3 0/2 0/2 0/? 0/? 0/? 0/? )================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 24;0 2/2 1/3 1/2 0/2 0/1 0/1 0/1 0/? )================ok
===( 28;0 2/3 1/2 1/2 1/1 1/1 0/1 0/? 0/? )================ok
===( 28;0 2/3 1/2 1/2 1/1 0/1 0/? 0/? )=====================ok
===( 31;0 3/3 2/2 1/2 1/1 0/1 0/? 0/? )=====================ok
===( 31;0 2/2 1/2 1/1 0/1 0/? 0/? )==========================ok
===( 31;0 1/2 1/1 0/1 0/? 0/? )===============================ok
===( 31;0 1/2 0/1 0/? 0/? )====================================ok
===( 33;0 1/1 0/? 0/? )=========================================ok
===( 35;0 1/1 1/1 )==============================================ok
../ext/Pod-Html/t/podnoerr.t ... ok

Test Summary Report


../ext/Pod-Html/t/cache.t (Wstat​: 256 Tests​: 10 Failed​: 1)
  Failed test​: 4
  Non-zero exit status​: 1
Files=18, Tests=35, 0 wallclock secs ( 0.05 usr 0.02 sys + 1.60 cusr 0.21 csys = 1.88 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 2;0 1/10 0/? 0/? 0/? 0/3 0/? 0/? 0/? )==============Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
ok
===( 13;0 1/1 0/1 0/1 1/3 0/2 0/? 0/? 0/? )================ok
===( 14;0 0/1 0/1 2/3 0/2 0/1 0/? 0/? )=====================
# Failed test 'misc pod-html --verbose warnings'
# at t/feature2.t line 27.
# 'scanning for directory cache
# scanning for directory cache
# caching directories for later use
# Converting input file /home/jkeenan/perl/ext/Pod-Html/t/feature2.pod
# '
# doesn't match '(?^x​:
# \Acaching\ directories\ for\ later\ use\n
# Converting\ input\ file\ \S+[/\\\]]feature2\.pod\n\z
# )'
# Looks like you failed 1 test of 2.
cannot open /home/jkeenan/perl/ext/Pod-Html/t/crossref.html​: No such file or directory at t/pod2html-lib.pl line 69, <DATA> chunk 1.
# Looks like your test exited with 2 before it could output anything.
===( 15;0 0/1 1/1 2/3 0/2 0/1 0/? 0/? 0/? )================ok
===( 16;0 0/1 1/1 0/2 0/1 0/? 0/? 0/? )=====================Dubious, test returned 2 (wstat 512, 0x200)
Failed 1/1 subtests
===( 16;0 1/1 0/2 0/1 0/? 0/? 0/? 0/? )=====================ok
===( 18;0 1/2 1/1 0/? 0/? 0/? 0/? 0/? 0/? )================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
===( 19;0 1/1 0/? 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 20;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 21;0 0/3 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 26;0 2/3 1/2 1/2 0/1 1/1 0/1 0/? 0/? )================ok
===( 27;0 2/3 1/2 1/2 1/1 0/1 0/? 0/? 0/? )================ok
===( 27;0 2/3 1/2 1/2 0/1 0/1 0/? 0/? )=====================ok
===( 30;0 3/3 2/2 0/1 0/1 0/? 0/? )==========================ok
===( 30;0 3/3 0/1 0/1 0/? 0/? )===============================ok
===( 31;0 1/1 0/1 0/? 0/? )====================================ok
===( 32;0 1/1 0/? 0/? )=========================================ok
===( 33;0 1/1 0/? )==============================================ok
../ext/Pod-Html/t/podnoerr.t ...
ok 1 - pod error section
ok

Test Summary Report


../ext/Pod-Html/t/crossref3.t (Wstat​: 512 Tests​: 0 Failed​: 0)
  Non-zero exit status​: 2
  Parse errors​: Bad plan. You planned 1 tests but ran 0.
../ext/Pod-Html/t/feature2.t (Wstat​: 256 Tests​: 2 Failed​: 1)
  Failed test​: 2
  Non-zero exit status​: 1
Files=18, Tests=34, 0 wallclock secs ( 0.05 usr 0.02 sys + 1.47 cusr 0.18 csys = 1.72 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 1;0 1/1 0/? 0/? 0/1 0/1 0/3 0/? 0/1 )================ok
===( 1;0 0/? 0/? 0/1 0/1 0/3 0/? 0/1 )=====================cannot remove directory for testdir/test.lib​: No such file or directory at t/pod2html-lib.pl line 27.
===( 3;0 0/? 0/? 1/1 1/1 0/3 0/? 0/1 0/? )================ok
===( 3;0 0/? 0/? 1/1 0/3 0/? 0/1 0/? )=====================ok
===( 4;0 0/10 0/? 0/3 0/? 1/1 0/? 0/? 0/? )==============ok
===( 10;0 5/10 0/1 1/3 0/2 0/? 0/? 0/? 0/? )==============ok
===( 13;0 6/10 0/1 0/2 0/? 0/? 0/? 0/? )===================ok
===( 18;0 1/1 0/2 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 19;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 20;0 0/? 0/? 0/? 0/? 0/? 0/? 0/? )=====================t/htmldir4.t​: error opening ./pod2htmd.tmp for reading​: No such file or directory
# Looks like your test exited with 2 before it could output anything.
===( 21;0 0/? 1/3 0/2 0/2 0/? 0/? 0/? 0/? )================Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests
===( 22;0 0/? 1/3 1/2 0/? 0/? 0/? 0/? 0/? )================ok
===( 24;0 0/? 2/3 0/? 0/? 0/? 0/? 0/? )=====================ok
===( 26;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 28;0 2/2 0/1 1/1 0/1 0/1 0/? 0/? )=====================ok
===( 28;0 0/1 1/1 0/1 0/1 0/? 0/? )==========================ok
===( 29;0 1/1 0/1 0/1 0/? 0/? )===============================ok
===( 30;0 0/1 1/1 0/? 0/? )====================================ok
===( 31;0 1/1 0/? 0/? )=========================================ok
===( 32;0 1/1 0/? )==============================================ok
../ext/Pod-Html/t/podnoerr.t ...
ok 1 - pod error section
ok

Test Summary Report


../ext/Pod-Html/t/htmldir4.t (Wstat​: 512 Tests​: 0 Failed​: 0)
  Non-zero exit status​: 2
  Parse errors​: Bad plan. You planned 2 tests but ran 0.
Files=18, Tests=33, 0 wallclock secs ( 0.04 usr 0.02 sys + 1.85 cusr 0.20 csys = 2.11 CPU)
Result​: FAIL
/home/jkeenan/perl
[dromedary-001​:perl] 1022 $ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd -
===( 1;0 1/1 0/? 0/? 0/? 0/? 0/3 0/? 0/? )================ok
===( 2;0 1/10 0/? 0/? 0/? 0/3 0/? 0/? 0/? )==============Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.

# Failed test 'podpath'
# at t/cache.t line 38.
# got​: ''
# expected​: 'scooby​:shaggy​:fred​:velma​:daphne'

# Failed test 'podroot'
# at t/cache.t line 39.
# got​: '.'
# expected​: '/home/jkeenan/perl/ext/Pod-Html'
Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 7;0 5/10 0/1 0/1 0/1 1/3 0/? 0/? 0/? )==============ok
===( 10;0 6/10 0/1 0/1 0/1 0/2 0/? 0/? )===================# Looks like you failed 2 tests of 10.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/10 subtests
===( 15;0 1/1 0/1 0/1 0/2 0/? 0/? 0/? 0/? )================ok
===( 15;0 0/1 0/1 0/2 0/1 0/? 0/? 0/? )=====================
# Failed test 'misc pod-html --verbose warnings'
# at t/feature2.t line 27.
# 'scanning for directory cache
# scanning for directory cache
# caching directories for later use
# Converting input file /home/jkeenan/perl/ext/Pod-Html/t/feature2.pod
# '
# doesn't match '(?^x​:
# \Acaching\ directories\ for\ later\ use\n
# Converting\ input\ file\ \S+[/\\\]]feature2\.pod\n\z
# )'
# Looks like you failed 1 test of 2.
===( 16;0 1/1 0/1 0/2 0/1 0/? 0/? 0/? 0/? )================ok
===( 17;0 1/1 0/2 0/1 0/? 0/? 0/? 0/? 0/? )================ok
===( 18;0 1/2 0/1 0/? 0/? 0/? 0/? 0/? 0/? )================Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
===( 20;0 1/1 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 21;0 1/2 0/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 22;0 0/3 0/2 0/? 0/? 0/? 0/? 0/? )=====================Directory 'test.lib' exists (it shouldn't yet) - removing it at t/pod2html-lib.pl line 14.
===( 28;0 2/3 1/2 1/2 1/1 1/1 0/1 0/1 0/? )================ok
===( 29;0 3/3 1/2 1/2 1/1 0/1 0/1 0/? )=====================ok
===( 30;0 3/3 2/2 1/2 0/1 0/1 0/? 0/? )=====================ok
===( 30;0 3/3 1/2 0/1 0/1 0/? 0/? )==========================ok
===( 30;0 1/2 0/1 0/1 0/? 0/? )===============================ok
===( 32;0 1/1 0/1 0/? 0/? )====================================ok
===( 33;0 1/1 0/? 0/? )=========================================ok
===( 34;0 1/1 0/? )==============================================ok
../ext/Pod-Html/t/podnoerr.t ...
ok 1 - pod error section
ok

Test Summary Report


../ext/Pod-Html/t/cache.t (Wstat​: 512 Tests​: 10 Failed​: 2)
  Failed tests​: 4-5
  Non-zero exit status​: 2
../ext/Pod-Html/t/feature2.t (Wstat​: 256 Tests​: 2 Failed​: 1)
  Failed test​: 2
  Non-zero exit status​: 1
Files=18, Tests=35, 1 wallclock secs ( 0.05 usr 0.01 sys + 1.45 cusr 0.18 csys = 1.69 CPU)
Result​: FAIL
/home/jkeenan/perl

@jkeenan jkeenan added the ext/Pod-Html issues in the blead-upstream Pod-Html distribution label Jan 31, 2020
@jkeenan
Copy link
Contributor

jkeenan commented Feb 24, 2021

At least some of the problems reported in this ticket, which I originally filed in 2012, are still present today. Example:

$ cd t; TEST_JOBS=4 ./perl harness ../ext/Pod-Html/t/*.t; cd -
../ext/Pod-Html/t/anchorify.t .. ok                                     
===(       1;0  0/?  0/?  0/? )=========================================Directory 'test.lib' exists (it shouldn't yet) - removing it at ./t/pod2html-lib.pl line 14.
../ext/Pod-Html/t/cache.t ...... ok                                     
../ext/Pod-Html/t/crossref3.t .. ok                                     
../ext/Pod-Html/t/crossref2.t .. ok                                     
../ext/Pod-Html/t/crossref.t ... ok                                     
===(      15;0  1/3  0/2  0/?  0/? )====================================
#   Failed test 'misc pod-html --verbose warnings'
#   at t/feature2.t line 26.
#                   'scanning for directory cache
# scanning for directory cache
# caching directories for later use
# Converting input file /home/jkeenan/gitwork/perl/ext/Pod-Html/t/feature2.pod
# Cannot find file "crossref.*" directly under podpath, cannot find suitable replacement: link remains unresolved.
# '
#     doesn't match '(?^x:
#     \Acaching\ directories\ for\ later\ use\n
#     Converting\ input\ file\ \S+[/\\\]]feature2\.pod\n
#     Cannot\ find\ file\ "crossref\.\*"\ directly\ under\ podpath,\ cannot\ find
#     \ suitable\ replacement:\ link\ remains\ unresolved\.\n\z
#     )'
# Looks like you failed 1 test of 2.                                    
../ext/Pod-Html/t/eol.t ........ ok
../ext/Pod-Html/t/feature2.t ... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests 
../ext/Pod-Html/t/feature.t .... ok                                     
../ext/Pod-Html/t/htmldir1.t ... ok                                     
../ext/Pod-Html/t/htmldir2.t ... ok                                     
../ext/Pod-Html/t/htmldir3.t ... ok                                     
../ext/Pod-Html/t/htmldir4.t ... ok                                     
../ext/Pod-Html/t/htmldir5.t ... ok                                     
../ext/Pod-Html/t/htmlescp.t ... ok                                     
../ext/Pod-Html/t/htmllink.t ... ok                                     
../ext/Pod-Html/t/htmlview.t ... ok                                     
../ext/Pod-Html/t/poderr.t ..... ok                                     
../ext/Pod-Html/t/podnoerr.t ... ok   

Test Summary Report
-------------------
../ext/Pod-Html/t/feature2.t (Wstat: 256 Tests: 2 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=18, Tests=35,  1 wallclock secs ( 0.04 usr  0.03 sys +  2.67 cusr  0.29 csys =  3.03 CPU)
Result: FAIL
/home/jkeenan/gitwork/perl

However, I no longer have the branch that I was working on when I filed the ticket. The laptop I was using has, for all practical purposes, been wiped. The "dromedary" development server I was using crashed and the code on it was unsalvageable.

I suspect the root problem here is that we're not using tempdirs and tempfiles to create the test .html files created during testing. Once some other Pod-Html tickets are resolved, I hope to explore that direction.

Thank you very much.
Jim Keenan

jkeenan added a commit to jkeenan/perl5 that referenced this issue Mar 7, 2021
The purpose of this branch is to explore the reasons behind the test
failures observed in Perl#12271.  In
this commit we simply add the two dummy test files which illustrate the
problem. xhthmldir3.t comments out the one good run; yhtmldir3.t
reverses the order of the two runs such that the bad one precedes the
good one.

We clearly know that one run is defective and will have to be deleted.
What we don't yet know is why it gives a spurious pass if it is preceded
by the good run.
jkeenan added a commit to jkeenan/perl5 that referenced this issue Mar 9, 2021
This function permits examination of the state of the Pod-Html cache
file between calls to xconvert().  This is being added to diagnose the
problems reported in Perl#12271.

2. Allow an 'expect_fail' option to be passed to xconvert().

This will be passed internally to _process_diff().  Within the latter,
if expect_fail is Perl-false, the unit test will FAIL and the diff
between the expectation and the result will be dumped as has always been
the case.  If, however, we expect the test to fail but want to indicate
a reason for that failure, we should (a) adjust the 'description' and
(b) set expect_fail to Perl-true.  When the calculated HTML fails to
match the expectation, the unit test will PASS but the description will
inform the user that we have an anomaly that has been diagnosed.

Adapt 3 t/*htmldir3.t test files to this interface modification.
jkeenan added a commit to jkeenan/perl5 that referenced this issue Mar 14, 2021
This function permits examination of the state of the Pod-Html cache
file between calls to xconvert().  This is being added to diagnose the
problems reported in Perl#12271.

2. Allow an 'expect_fail' option to be passed to xconvert().

This will be passed internally to _process_diff().  Within the latter,
if expect_fail is Perl-false, the unit test will FAIL and the diff
between the expectation and the result will be dumped as has always been
the case.  If, however, we expect the test to fail but want to indicate
a reason for that failure, we should (a) adjust the 'description' and
(b) set expect_fail to Perl-true.  When the calculated HTML fails to
match the expectation, the unit test will PASS but the description will
inform the user that we have an anomaly that has been diagnosed.

Adapt 3 t/*htmldir3.t test files to this interface modification.
jkeenan added a commit to jkeenan/perl5 that referenced this issue May 10, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue May 10, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue May 10, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue May 11, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue May 11, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
jkeenan added a commit to jkeenan/perl5 that referenced this issue May 30, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jun 15, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jun 15, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jun 15, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl#12271.
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jun 15, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
jkeenan added a commit that referenced this issue Jun 28, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
#12271.

Correct typo spotted by rjbs.
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jun 29, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jul 6, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
jkeenan added a commit to jkeenan/perl5 that referenced this issue Jul 6, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl#12271.
thibaultduponchelle pushed a commit to thibaultduponchelle/perl5 that referenced this issue Jul 9, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl#12271.

Correct typo spotted by rjbs.
thibaultduponchelle pushed a commit to thibaultduponchelle/perl5 that referenced this issue Jul 9, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
@jkeenan
Copy link
Contributor

jkeenan commented Aug 23, 2021

From @jkeenan

This is a bug report for perl from jkeenan@​cpan.org,
generated with the help of perlbug 1.39 running under perl 5.16.0.
[snip]

This suggests to me that the problem is in the way t/htmldir3.t is
formulated. My hunch is that the first test in t/htmldir3.t changes the
environment in some way, not yet understood, that enables the second
test to pass. But when we make that test the first
one in a file, it fails.

This problem should be resolved. In the recent series of pull requests refactoring ext/Pod-Html, I converted all the tests to create tempfiles for their execution. I also added an expect_fail KVP to the testing infrastructure, so that you can specify that you expect a given POD-to-HTML conversion not to "succeed" (that is, match the expected output exactly).

So ext/Pod-Html/t/htmldir3.t now does the right thing. I will close this ticket within 3 days unless there is an objection.

Thank you very much.
Jim Keenan

@jkeenan jkeenan self-assigned this Aug 23, 2021
@jkeenan jkeenan added the Closable? We might be able to close this ticket, but we need to check with the reporter label Aug 23, 2021
@jkeenan jkeenan closed this as completed Sep 1, 2021
@jkeenan jkeenan removed the Closable? We might be able to close this ticket, but we need to check with the reporter label Sep 1, 2021
nicomen pushed a commit to nicomen/pumpkin-perl that referenced this issue Oct 11, 2021
Until now, the tests for this library ran inside the ext/Pod-Html
directory in the core distribution.  Since the tests necessarily create
files while the harness runs, there was a possibility for race
conditions, which meant that we had to have a workaround in t/harness to
have the tests in this directory not run in parallel.

This commit restructures the testing process so that each test program
gets its own tempdir in which to create files and directories.

The test architecture is now found in t/lib/Testing.pm.
This module will export two functions on demand that are used in the
individual test files.  They guarantee that all files created when
running a test program will be created in a temporary directory rather
than underneath the core distribution.

t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be
removed.  The workaround has been removed from t/harness.

Change directory name from testdir/ to corpus/.  So as to more precisely
describe what the directory's purpose is.

Keep porting tests happy. Update MANIFEST and known_pod_issues database.

Acknowledge Craig Berry's recommendation in earlier branch

This should help in the resolution of
Perl/perl5#12271.

Correct typo spotted by rjbs.
nicomen pushed a commit to nicomen/pumpkin-perl that referenced this issue Oct 11, 2021
A method to assist in debugging cache problems.

Should assist in resolving Perl/perl5#12271.

Signed-off-by: James E Keenan <jkeenan@cpan.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ext/Pod-Html issues in the blead-upstream Pod-Html distribution
Projects
None yet
Development

No branches or pull requests

2 participants