[PATCH] parallel testing on Win32, now with working prototype, needs work #13367
Comments
From @bulk88Created by @bulk88Reposting Earlier post (not to P5P) I made about the idea that got no response Original P5P post below. I decided to get parallel testing working on Win32 Perl.It has never The commit disabling use of IO::Select in TAP::Parser::Multiplexer is NPFS I think is capable of supplying all information/APIs necessary to For my first try, I decided to just do the bare minimum for harness to I dont see myself implementing writability or exceptions parts of select So I am attaching 2 files. 1 is a patch to TAP::Parser::Multiplexer to The code *barely* works on Win32 with TEST_JOBS=8 and I get 8 advancing The code has a number of problems I dont know what to do about and I TAP::Parser::Multiplexer and TAP::Parser::Iterator::Process may have To get this into perl git, is another problem. It uses use Win32::API I've also found out, very very very often (if I turn on the reporting, Another problem is, TAP::Parser::Multiplexer is INCAPABLE of processing my $sel = $self->{select}; return sub { # Drain all the non-selectable parsers first unless (@ready) { my ( $h, $parser, $stash, @handles ) = @{ shift @ready unless ( defined $result ) { # Force another can_read - we may now have removed a handle return ( $parser, $stash, $result ); IDK what to do about this. Also TAP is constantly doing selects on closed file descriptors. _______________________________________________________________ ______________________________________________________________ I can't figure out what is causing all the "Use of uninitialized value _______________________________________________________________ my $bits = $vec->[VEC_BITS]; my $count = 0; I also attached a screenshot of console window during parallel testing. IO::Select::Win32Pipes needs a new name also. I dont have the tuits to Also the fork of IO::Select isn't the most efficient design. I kept Another problem is, Multiplexer/TAP has a problem servicing the child Need help. Lots of it. Perl Info |
From @bulk880001-TAP-Parser-Multiplexer-hacking.patchFrom 34be502a91fb50412fdfdc68f17fd5edb1311edf Mon Sep 17 00:00:00 2001
From: Daniel Dragan <bulk88@hotmail.com>
Date: Sat, 19 Oct 2013 13:58:52 -0400
Subject: [PATCH] TAP::Parser::Multiplexer hacking
---
cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm | 17 ++++++++++++++---
1 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm b/cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm
index 68457a8..4ae7a29 100644
--- a/cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm
+++ b/cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm
@@ -3,13 +3,24 @@ package TAP::Parser::Multiplexer;
use strict;
use warnings;
-use IO::Select;
+
use parent 'TAP::Object';
use constant IS_WIN32 => $^O =~ /^(MS)?Win32$/;
use constant IS_VMS => $^O eq 'VMS';
-use constant SELECT_OK => !( IS_VMS || IS_WIN32 );
+use constant SELECT_OK => !( IS_VMS);
+
+
+BEGIN {
+my @oldinc = @INC;
+local(@INC);
+@INC = @oldinc;
+unshift @INC, 'C:\perl519\site\lib';
+require IO::Select::Win32Pipes;
+#eval 1 ? "use IO::Select::Win32Pipes;"
+#: "use IO::Select;" ;
+}
=head1 NAME
@@ -58,7 +69,7 @@ Returns a new C<TAP::Parser::Multiplexer> object.
sub _initialize {
my $self = shift;
- $self->{select} = IO::Select->new;
+ $self->{select} = IS_WIN32 ? IO::Select::Win32Pipes->new : IO::Select->new;
$self->{avid} = []; # Parsers that can't select
$self->{count} = 0;
return $self;
--
1.7.9.msysgit.0
|
From @bulk88 |
From @bulk88On Wed Oct 23 17:35:23 2013, bulk88 wrote:
........................... Bump. -- |
From @LeontOn Thu, Oct 24, 2013 at 2:35 AM, bulk88 <perlbug-followup@perl.org> wrote:
The way it's intended to be done is by writing your own multiplexer class. In general, I'd recommend first prototyping this on CPAN, and when it works
I can imagine how that happens. The multiplexer needs some mechanism to Leon |
|
The RT System itself - Status changed from 'new' to 'open' |
From @tonycozOn Wed Oct 23 17:35:23 2013, bulk88 wrote:
Is there a .xs file missing from this? Tony |
From @bulk88On Sun Jun 15 17:23:01 2014, tonyc wrote:
I dont have the code locally anymore. The .xs file I think was empty and it isn't required, since the .pm only uses Win32::API and Win32API::File. The .xs file might have had native NT API testing, but an official MS API implementation would be best first, which is what the .pm has in it, then trying native API if there are problems using MS API. Should I retest this on 5.21 and submit an updated patch/code that is known to work on 5.21? It won't be prettier than what you see due to design problems/integration problems with TAP:: that nobody has had an answer for yet. It will require extra modules to be in the core, unless syscall() on Win32 becomes a Win32::API "light" clone that can call arbitrary c function pointers. Also can TAP depend on XS? Its architecture seems to be PP all the way through, except for IPC::Open3 I think. I would like to see parallel Win32 testing, but it can't move forward until someone other than me wants it in, and there is a path and plan for approval from both P5P and TAP's maintainers/owner. -- |
From @tonycozOn Sun, Jun 15, 2014 at 06:11:38PM -0700, bulk88 via RT wrote:
Thanks, I saw the load and wondered.
If we asked nicely, maybe the two functions you use could be added to
It currently depends on XS, TAP::Harness loads IO::Handle. minitest is currently broken on Win32 because of it: cd ..\t && ..\miniperl.exe -I..\lib harness base/*.t comp/*.t cmd/*.t i
I'd like to see parallel Win32 testing, but I'm not sure this is the Unfortunately I think a version which used Win32 pipes as they're Tony |
From @bulk88On Sun Jun 15 18:47:45 2014, tonyc wrote:
I didn't think of that. Should they be documented, minimally documented ("internal use only"), or undocumented for core use only?
I know the correct way to do it, its just that unix IO (and perl, with select()), are deeply based on non-blocking IO (getting notification of data before starting to read it, read() is as fast as the CPU will execute the asm code that makes up the read ()func), while Win32 IO is completely based, based on async IO (variable time to completion read()ing, read is slow (sync io), or read immediately returns, but the OS temporarily takes ownership of your pointer, and at some later point in time you find out your can have your pointer back from the OS). To fake the Unix model on Windows, requires doing a secret pre-read on the OS handle, and buffering in user space, to fake up a select() on pipes or files (AFAIK select is useless on files on posix, even if disk IO will block your thread for a few ms). This is really complicated. Native API allows you to register an event handle to trip when an event happens on the pipe (FSCTL_PIPE_ASSIGN_EVENT), then FSCTL_PIPE_QUERY_EVENT to find read/write/other status on the pipe that tripped the event handle. MS obviously put this in for their NT POSIX emulation in SFU/Interfix, so MS's Interix select() is POSIX compliant, unlike their Win32 select(). But, its Native API.... So Native API would allow your busy-loop-less sleep()-less design request. -- |
From @jkeenanOn Sun Jun 15 19:38:23 2014, bulk88 wrote:
I reviewed this older ticket this evening. There has been no correspondence in the ticket in more than 15 months. bulk88, if you are still interested in pursuing this, I'd like to ask you to take Leon T's earlier suggestion to work this out on CPAN first. I'm taking this ticket for the purpose of closing it in 7 days unless there is a compelling reason to keep it open. Thank you very much. -- |
From @jkeenanOn Mon Oct 05 19:15:48 2015, jkeenan wrote:
No one spoke up in favor of keeping this ticket open; closing. Thank you very much. -- |
|
@jkeenan - Status changed from 'open' to 'rejected' |
Migrated from rt.perl.org#120330 (status was 'rejected')
Searchable as RT120330$
The text was updated successfully, but these errors were encountered: