New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pod::Html: t/htmldir3.t test produces spurious pass #12271
Comments
From @jkeenanThis is a bug report for perl from jkeenan@cpan.org, In the course of working on my fork of Pod-Html, I have come to a point Further investigation suggests that the one failing test is *only You can see this for yourself by cd-ing to ext/Pod-Html/ in blead. ##### All tests should pass. Now, copy into ext/Pod-Html/t/ the two files attached to this RT: If all the tests and testing functions were well ##### The sole test in xhtmldir3.t and the first test in yhtmldir3.t should ##### Test Summary Report t/xhtmldir3.t (Wstat: 256 Tests: 1 Failed: 1) The differences between the expected HTML and the generated HTML which ##### This suggests to me that the problem is in the way t/htmldir3.t is Thank you very much. Flags: Site configuration information for perl 5.16.0: Configured by jimk at Sun May 20 20:01:26 EDT 2012. Summary of my perl5 (revision 5 version 16 subversion 0) configuration: Platform: Locally applied patches: @INC for perl 5.16.0: Environment for perl 5.16.0: DYLD_LIBRARY_PATH=/Users/jimk/work/pseudoinstall/lib:/Users/jimk/gitwork/parrot/blib/lib PATH=/usr/local/bin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/Users/jimk/bin:/Users/jimk/bin/perl:/Users/jimk/bin/c:/Users/jimk/bin/shell:/sw/lib:/sw/bin:/Users/jimk/bin:/Users/jimk/bin/perl:/Users/jimk/bin/c:/Users/jimk/bin/shell:/sw/lib:/sw/bin |
From @jkeenan#!/usr/bin/perl -w # -*- perl -*- BEGIN { END { use strict; SKIP: { my $cwd = cwd(); my $data_pos = tell DATA; # to read <DATA> twice convert_n_test("htmldir3", "test --htmldir and --htmlroot 3b", seek DATA, $data_pos, 0; # to read <DATA> twice (expected output is the same) convert_n_test("htmldir3", "test --htmldir and --htmlroot 3a", __DATA__ <body style="background-color: white"> <ul id="index"> <h1 id="NAME">NAME</h1> <p>htmldir - Test --htmldir feature</p> <h1 id="LINKS">LINKS</h1> <p>Normal text, a <a>link</a> to nowhere,</p> <p>a link to <a href="[RELCURRENTWORKINGDIRECTORY]/testdir/test.lib/var-copy.html">var-copy</a>,</p> <p><a href="[RELCURRENTWORKINGDIRECTORY]/t/htmlescp.html">htmlescp</a>,</p> <p><a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>,</p> <p>and another <a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>.</p> </body> </html> |
From @jkeenan#!/usr/bin/perl -w # -*- perl -*- BEGIN { END { use strict; SKIP: { my $cwd = cwd(); convert_n_test("htmldir3", "test --htmldir and --htmlroot 3b", __DATA__ <body style="background-color: white"> <ul id="index"> <h1 id="NAME">NAME</h1> <p>htmldir - Test --htmldir feature</p> <h1 id="LINKS">LINKS</h1> <p>Normal text, a <a>link</a> to nowhere,</p> <p>a link to <a href="[RELCURRENTWORKINGDIRECTORY]/testdir/test.lib/var-copy.html">var-copy</a>,</p> <p><a href="[RELCURRENTWORKINGDIRECTORY]/t/htmlescp.html">htmlescp</a>,</p> <p><a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>,</p> <p>and another <a href="[RELCURRENTWORKINGDIRECTORY]/t/feature.html#Another-Head-1">"Another Head 1" in feature</a>.</p> </body> </html> |
From @nwc10On Fri, Jul 13, 2012 at 05:35:19AM -0700, James E Keenan wrote:
I'm not clear what you'd like to happen as a result of this ticket. Is it a request for someone else to investigate to give a second opinion Or is this more a "note to self" to come back and fix it at some point? Nicholas Clark |
The RT System itself - Status changed from 'new' to 'open' |
From @jkeenanOn Mon Jul 23 02:18:36 2012, nicholas wrote:
Both. At the hackathon held at DuckDuckGo in Pennsylvania last month, I had In this particular case, it would be good to (a) get an explanation of Thank you very much. |
From @rjbs* James E Keenan via RT <perlbug-followup@perl.org> [2012-07-23T08:03:47]
It would be good to get the input of Marc Green, who has probably done the most cc'd.
-- |
From @jkeenanOn Mon Jul 23 05:03:47 2012, jkeenan wrote:
Two calendar years later ... and this just got weirder! I created a branch on the dromedary test server today and added the two test files (with minor updatings) originally posted in this ticket. I then configure, built and tested perl in that branch. As expected, the two additional files failed. I then repeatedly ran this set of commands: ##### ... and got different results each time! t/xhtmldir3.t and t/yhtmldir3.t failed each time, as expected. But each repetition generated failures in 0 to 2 *other*, previously passing files! The output is in the attachment. Thank you very much. |
From @jkeenanOn dromedary: branch: 114126-pod-html-accidental-pass Test Summary Report ../ext/Pod-Html/t/xhtmldir3.t (Wstat: 256 Tests: 1 Failed: 1) okay, that's per #114126. But now let's run the harness over that directory: $ cd t;./perl harness -v ../ext/Pod-Html/t/*.t;cd - # Failed test 'test --htmldir and --htmlroot 3b' # Failed test 'test --htmldir and --htmlroot 3a' not ok 2 - test --htmldir and --htmlroot 3a Test Summary Report ../ext/Pod-Html/t/cache.t (Wstat: 512 Tests: 3 Failed: 1) 2 more files are failing! Repeat. $ cd t;./perl harness -v ../ext/Pod-Html/t/*.t;cd - # Failed test 'test --htmldir and --htmlroot 3b' not ok 2 - test --htmldir and --htmlroot 3a Test Summary Report ../ext/Pod-Html/t/feature2.t (Wstat: 512 Tests: 0 Failed: 0) cache.t and crossref2.t are once again passing -- but now feature2.t is failing! Repeat once more. $ cd t;./perl harness -v ../ext/Pod-Html/t/*.t;cd - # Failed test 'test --htmldir and --htmlroot 3b' # Failed test 'test --htmldir and --htmlroot 3b' not ok 2 - test --htmldir and --htmlroot 3a Test Summary Report ../ext/Pod-Html/t/xhtmldir3.t (Wstat: 256 Tests: 1 Failed: 1) Now we're back to just x/y htmldir3.t failing! Further repetitions vary. Often 3 tests fail: x/y and one of cache, crossref2, feature2. |
From @jkeenanOn Wed Jan 01 12:35:53 2014, jkeenan wrote:
More data. Tonight I built blead on the dromedary server. All tests, including those in ext/Pod-Html/t/, passed. (The command I run there is: I then ran those same tests with the harness 8 times -- and got 8 different results. ##### Results attached. It is interesting to note that many, but not all, of the test failures in the attachment are failures like these two: ##### That is, the output expected -- the '-' lines in the diff excerpt above -- is an <a> tag where something like the following has been interpolated in: ##### But what is actually generated is simply: <a> In my attempt to make Pod-Html CPANnable this type of test failure is one I'm getting in many files -- but at least I'm getting them consistently. And it's the only type of failure I'm still getting. So, while there may well be errors in my refactoring of the code, the fundamental problem may have been there all along but simply have been masked. Thank you very much. |
From @jkeenan$ cd t;./perl harness -v ../ext/Pod-Html/t/*t; cd - Test Summary Report ../ext/Pod-Html/t/cache.t (Wstat: 256 Tests: 10 Failed: 1) # Failed test 'podroot' # Failed test 'misc pod-html --verbose warnings' # Failed test 'test --htmldir and --htmlroot 1a' Test Summary Report ../ext/Pod-Html/t/cache.t (Wstat: 512 Tests: 10 Failed: 2) # Failed test 'podroot' # Failed test 'cross references' Test Summary Report ../ext/Pod-Html/t/crossref3.t (Wstat: 256 Tests: 1 Failed: 1) Test Summary Report ../ext/Pod-Html/t/crossref.t (Wstat: 256 Tests: 1 Failed: 1) Test Summary Report ../ext/Pod-Html/t/cache.t (Wstat: 256 Tests: 10 Failed: 1) Test Summary Report ../ext/Pod-Html/t/crossref3.t (Wstat: 512 Tests: 0 Failed: 0) Test Summary Report ../ext/Pod-Html/t/htmldir4.t (Wstat: 512 Tests: 0 Failed: 0) # Failed test 'podpath' # Failed test 'podroot' Test Summary Report ../ext/Pod-Html/t/cache.t (Wstat: 512 Tests: 10 Failed: 2) |
At least some of the problems reported in this ticket, which I originally filed in 2012, are still present today. Example:
However, I no longer have the branch that I was working on when I filed the ticket. The laptop I was using has, for all practical purposes, been wiped. The "dromedary" development server I was using crashed and the code on it was unsalvageable. I suspect the root problem here is that we're not using tempdirs and tempfiles to create the test Thank you very much. |
The purpose of this branch is to explore the reasons behind the test failures observed in Perl#12271. In this commit we simply add the two dummy test files which illustrate the problem. xhthmldir3.t comments out the one good run; yhtmldir3.t reverses the order of the two runs such that the bad one precedes the good one. We clearly know that one run is defective and will have to be deleted. What we don't yet know is why it gives a spurious pass if it is preceded by the good run.
This function permits examination of the state of the Pod-Html cache file between calls to xconvert(). This is being added to diagnose the problems reported in Perl#12271. 2. Allow an 'expect_fail' option to be passed to xconvert(). This will be passed internally to _process_diff(). Within the latter, if expect_fail is Perl-false, the unit test will FAIL and the diff between the expectation and the result will be dumped as has always been the case. If, however, we expect the test to fail but want to indicate a reason for that failure, we should (a) adjust the 'description' and (b) set expect_fail to Perl-true. When the calculated HTML fails to match the expectation, the unit test will PASS but the description will inform the user that we have an anomaly that has been diagnosed. Adapt 3 t/*htmldir3.t test files to this interface modification.
This function permits examination of the state of the Pod-Html cache file between calls to xconvert(). This is being added to diagnose the problems reported in Perl#12271. 2. Allow an 'expect_fail' option to be passed to xconvert(). This will be passed internally to _process_diff(). Within the latter, if expect_fail is Perl-false, the unit test will FAIL and the diff between the expectation and the result will be dumped as has always been the case. If, however, we expect the test to fail but want to indicate a reason for that failure, we should (a) adjust the 'description' and (b) set expect_fail to Perl-true. When the calculated HTML fails to match the expectation, the unit test will PASS but the description will inform the user that we have an anomaly that has been diagnosed. Adapt 3 t/*htmldir3.t test files to this interface modification.
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. This should help in the resolution of Perl#12271.
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. This should help in the resolution of Perl#12271.
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. This should help in the resolution of Perl#12271.
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl#12271.
A method to assist in debugging cache problems. Should assist in resolving Perl#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl#12271.
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl#12271.
A method to assist in debugging cache problems. Should assist in resolving Perl#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl#12271.
A method to assist in debugging cache problems. Should assist in resolving Perl#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of #12271. Correct typo spotted by rjbs.
A method to assist in debugging cache problems. Should assist in resolving Perl#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
A method to assist in debugging cache problems. Should assist in resolving Perl#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl#12271.
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl#12271. Correct typo spotted by rjbs.
A method to assist in debugging cache problems. Should assist in resolving Perl#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
This problem should be resolved. In the recent series of pull requests refactoring So Thank you very much. |
Until now, the tests for this library ran inside the ext/Pod-Html directory in the core distribution. Since the tests necessarily create files while the harness runs, there was a possibility for race conditions, which meant that we had to have a workaround in t/harness to have the tests in this directory not run in parallel. This commit restructures the testing process so that each test program gets its own tempdir in which to create files and directories. The test architecture is now found in t/lib/Testing.pm. This module will export two functions on demand that are used in the individual test files. They guarantee that all files created when running a test program will be created in a temporary directory rather than underneath the core distribution. t/pod2html-lib.pl is now superseded by t/lib/Testing.pm and can be removed. The workaround has been removed from t/harness. Change directory name from testdir/ to corpus/. So as to more precisely describe what the directory's purpose is. Keep porting tests happy. Update MANIFEST and known_pod_issues database. Acknowledge Craig Berry's recommendation in earlier branch This should help in the resolution of Perl/perl5#12271. Correct typo spotted by rjbs.
A method to assist in debugging cache problems. Should assist in resolving Perl/perl5#12271. Signed-off-by: James E Keenan <jkeenan@cpan.org>
Migrated from rt.perl.org#114126 (status was 'open')
Searchable as RT114126$
The text was updated successfully, but these errors were encountered: