[patches] enhance bench.pl to test same perl under different options/args #15272
Comments
From @jimcCreated by @jimcPorting/bench.pl has 2 interrelated shortcomings: 1- it stores cachegrind results using {$testname}{$perlname}. 2- it cannot pass different options/args to each Perl-Under-Test. This patchset corrects those problems, allowing to test the same perl example 0: $] perl Porting/bench.pl --jobs=2 --verbose \ example 1: $] perl Porting/bench.pl --jobs=2 \ yields: perl5.18.2+strict perl5.18.0+dumper example 2: $] perl Porting/bench.pl --jobs=2 \ That wont actually work, cuz the -D<options> cause extra output which example 3: $] perl Porting/bench.pl --jobs=2 \ This uses the extended [=label=(ENV-name-value-pairs)] to add the patches: 0001-Porting-bench.pl-verbose-assist-on-tests-failure.patch they are also available at They are still WIP; they extend the use-cases as I intended, but I Perl Info |
From @jimc0001-Porting-bench.pl-verbose-assist-on-tests-failure.patchFrom 4b49008ac26be5da31dd54c4f3d546cbb5564f0e Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sat, 9 Apr 2016 22:33:10 -0600
Subject: [PATCH 01/10] Porting/bench.pl: --verbose assist on tests failure
with --verbose, --tests=foo will fail loudly, so user can pick a test.
---
Porting/bench.pl | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index f2fcf12..de8deac 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -417,7 +417,9 @@ sub filter_tests {
else {
my %t;
for (split /,/, $opt) {
- die "Error: no such test found: '$_'\n" unless exists $tests->{$_};
+ die "Error: no such test found: '$_'\n"
+ . ($OPTS{verbose} ? " have: @{[ sort keys %$tests ]}\n" : "")
+ unless exists $tests->{$_};
$t{$_} = 1;
}
for (keys %$tests) {
--
2.5.5
|
From @jimc0002-prep-grind-run-to-accept-args-from-PUTs.patchFrom 5d74351dc42bd2c5c388acc483264d8b8233624a Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sat, 9 Apr 2016 22:45:25 -0600
Subject: [PATCH 02/10] prep grind-run to accept args from PUTs
---
Porting/bench.pl | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index de8deac..639efe1 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -704,7 +704,7 @@ sub grind_run {
);
for my $p (@$perls) {
- my ($perl, $label) = @$p;
+ my ($perl, $label, @putargs) = @$p;
# Run both the empty loop and the active loop
# $counts->[0] and $counts->[1] times.
@@ -715,7 +715,7 @@ sub grind_run {
. "valgrind --tool=cachegrind --branch-sim=yes "
. "--cachegrind-out-file=/dev/null "
. "$OPTS{grindargs} "
- . "$perl $OPTS{perlargs} - $counts->[$j] 2>&1";
+ . "$perl $OPTS{perlargs} @putargs - $counts->[$j] 2>&1";
# for debugging and error messages
my $id = "$test/$perl "
. ($i ? "active" : "empty") . "/"
--
2.5.5
|
From @jimc0003-Porting-bench.pl-protect-against-data-loss.patchFrom 206d9fdfa34ba3ca5417032c9a919713649d0647 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sat, 9 Apr 2016 22:51:48 -0600
Subject: [PATCH 03/10] Porting/bench.pl: protect against data loss
Due to %data storage using $perl_name (rather than $label),
blead silently loses data when running like so:
perl Porting/bench.pl -- perl perl
patch complains about above, but allows following:
perl Porting/bench.pl -- perl=FOO perl=BAR
With this 1/2 fix, we can further probe the underlying key probs,
evident in the results from above
...
loop::for::lex_range4
empty for loop with lexical var and 4 integer range
BAR BAR
------ ------
Ir 100.00 100.00
Dr 100.00 100.00
---
Porting/bench.pl | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 639efe1..65ffddb 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -482,9 +482,12 @@ sub select_a_perl {
sub process_perls {
my @results;
+ my %seen;
for my $p (@_) {
my ($perl, $label) = split /=/, $p, 2;
$label //= $perl;
+ die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
+
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
die "Error: unable to execute '$perl': $r" if $r ne "ok\n";
push @results, [ $perl, $label ];
--
2.5.5
|
From @jimc0004-Porting-bench.pl-allow-per-PUT-perl-under-test-optio.patchFrom 1b2a12811f098bef6e0a608786f9512e5a86e51d Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sat, 9 Apr 2016 23:20:10 -0600
Subject: [PATCH 04/10] Porting/bench.pl: allow per-PUT (perl under test)
options and modules
Rework process_perls() to give a richer usage / API, allowing
additional command-line options, specific to each Perl-Under-Test.
For example:
bench.pl -- perl=plain perl=slower -Mstrict -DmpMA
The above runs the same perl-exe for 2 different tests (PUTS), but
adds expensive debugging options to only the 2nd PUT.
Do this by changing strategy; we scan the list backwards, and
test/treat each item as a perlexe (ie qx/$perlexe -e 'print "ok"/).
Instead of dieing on a not-perl, they're collected and submitted as a
PUT once a $perlexe is found.
Added 'require_order' to terminate arg processing when '--' is
encountered on cmdline; without it the PUT options are in-validated by
GetOptions.
---
Porting/bench.pl | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 65ffddb..9e6f6b9 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -209,7 +209,7 @@ Requires C<JSON::PP> to be available.
use 5.010000;
use warnings;
use strict;
-use Getopt::Long qw(:config no_auto_abbrev);
+use Getopt::Long qw(:config no_auto_abbrev require_order);
use IPC::Open2 ();
use IO::Select;
use IO::File;
@@ -480,19 +480,27 @@ sub select_a_perl {
# Validate the list of perl=label on the command line.
# Return a list of [ exe, label ] pairs.
-sub process_perls {
+sub process_puts {
my @results;
my %seen;
- for my $p (@_) {
+ my @putargs; # collect not-perls into args per PUT
+
+ for my $p (reverse @_) {
+ push @putargs, $p and next if $p =~ /^-/; # not-perl, dont send to qx//
+
my ($perl, $label) = split /=/, $p, 2;
$label //= $perl;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
- die "Error: unable to execute '$perl': $r" if $r ne "ok\n";
- push @results, [ $perl, $label ];
+ if ($r eq "ok\n") {
+ push @results, [ $perl, $label, reverse @putargs ];
+ @putargs = ();
+ } else {
+ push @putargs, $p; # not-perl
+ }
}
- return @results;
+ return reverse @results;
}
@@ -620,7 +628,7 @@ sub do_grind {
die "Error: only a single test may be specified with --bisect\n"
if defined $OPTS{bisect} and keys %$tests != 1;
- $perls = [ process_perls(@$perl_args) ];
+ $perls = [ process_puts(@$perl_args) ];
$results = grind_run($tests, $order, $perls, $loop_counts);
--
2.5.5
|
From @jimc0005-store-data-using-unique-label-not-perl-exe-which-may.patchFrom c8926bd35feda41219d501e4c92b69b3781794c1 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 00:06:50 -0600
Subject: [PATCH 05/10] store data using unique label, not perl-exe, which may
be used 2x
Since we want that a perl-name can be used in 2 separate PUTs, we
cannot use it alone to store PUT results. Instead, use the PUTs
labels, which are enforced unique.
And use "$test/$label" instead of "$test/$perl" in debug msg, since
thats how we store it now.
This change has downstream data-indexing ramifications, which are not
fully understood yet, and which are NOT handled here.
---
Porting/bench.pl | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 9e6f6b9..ab62069 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -728,7 +728,7 @@ sub grind_run {
. "$OPTS{grindargs} "
. "$perl $OPTS{perlargs} @putargs - $counts->[$j] 2>&1";
# for debugging and error messages
- my $id = "$test/$perl "
+ my $id = "$test/$label "
. ($i ? "active" : "empty") . "/"
. ($j ? "long" : "short") . " loop";
@@ -856,7 +856,7 @@ sub grind_run {
. "Output\n$o";
}
- $results{$j->{test}}{$j->{perl}}[$j->{active}][$j->{loopix}]
+ $results{$j->{test}}{$j->{plabel}}[$j->{active}][$j->{loopix}]
= parse_cachegrind($output, $j->{id}, $j->{perl});
}
--
2.5.5
|
From @jimc0006-grind_print-adjust-for-labels.patchFrom 7443e7176e23980882931a5dc72234bb2d55ee05 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 00:15:31 -0600
Subject: [PATCH 06/10] grind_print: adjust for labels
since HEAD~1, data is stored by the label, not the perlname.
simplify grind-print
---
Porting/bench.pl | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index ab62069..fdd4602 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -945,7 +945,7 @@ sub grind_process {
my %counts;
my %data;
- my $perl_norm = $perls->[$OPTS{norm}][0]; # the name of the reference perl
+ my $perl_norm = $perls->[$OPTS{norm}][1]; # the label of the reference perl
for my $test_name (keys %$res) {
my $res1 = $res->{$test_name};
@@ -1101,6 +1101,7 @@ sub grind_print {
my ($results, $averages, $perls, $tests, $order) = @_;
my @perl_names = map $_->[0], @$perls;
+ my @perl_labels = map $_->[1], @$perls;
my %perl_labels;
$perl_labels{$_->[0]} = $_->[1] for @$perls;
@@ -1108,7 +1109,8 @@ sub grind_print {
# Calculate the width to display for each column.
my $min_width = $OPTS{raw} ? 8 : 6;
my @widths = map { length($_) < $min_width ? $min_width : length($_) }
- @perl_labels{@perl_names};
+ #@perl_labels{@perl_names};
+ @perl_labels;
# Print standard header.
grind_blurb($perls);
@@ -1138,7 +1140,7 @@ sub grind_print {
print " " x $field_label_width;
for (0..$#widths) {
printf " %*s", $widths[$_],
- $i ? ('-' x$widths[$_]) : $perl_labels{$perl_names[$_]};
+ $i ? ('-' x$widths[$_]) : $perl_labels[$_];
}
print "\n";
}
@@ -1160,7 +1162,7 @@ sub grind_print {
print " " x $field_label_width;
for (0..$#widths) {
printf " %*s", $widths[$_],
- $i ? ('-' x$widths[$_]) : $perl_labels{$perl_names[$_]};
+ $i ? ('-' x$widths[$_]) : $perl_labels[$_];
}
print "\n";
}
@@ -1190,7 +1192,7 @@ sub grind_print {
}
for my $i (0..$#widths) {
- my $res2 = $res1->{$perl_names[$i]};
+ my $res2 = $res1->{$perl_labels[$i]};
my $p = $res2->{$field};
if (!defined $p) {
printf " %*s", $widths[$i], '-';
--
2.5.5
|
From @jimc0007-add-pod-for-PUT.patchFrom a1db85ddeef7b044d4e15e48bf6f8a263b937576 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 07:46:40 -0600
Subject: [PATCH 07/10] add pod for PUT
---
Porting/bench.pl | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index fdd4602..8da1de5 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -14,7 +14,11 @@ perls.
# Basic: run the tests in t/perf/benchmarks against two or
# more perls
- bench.pl [options] perl1[=label1] perl2[=label2] ...
+ bench.pl [options] perlA[=labelA] perlB[=labelB] ...
+
+ # Alt: run the tests against same perlA 2x, with and without extra options
+
+ bench.pl [options] -- perlA=fast PerlA=slow -Mstrict -Dpsltoc
# Run bench.pl's own built-in sanity tests
@@ -140,8 +144,8 @@ It defaults to the leftmost column.
--perlargs=I<foo>
-Optional command-line arguments to pass to each perl that is run as part of
-a cachegrind session. For example, C<--perlargs=-Ilib>.
+Optional command-line arguments to pass to each perl-under-test
+(perlA, perlB in synopsis) For example, C<--perlargs=-Ilib>.
=item *
--
2.5.5
|
From @jimc0008-s-results-res_puts-in-process_puts.patchFrom 619c4f38075bb987f690d721133de0e146199cc2 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 11:50:13 -0600
Subject: [PATCH 08/10] s/@results/@res_puts/ in process_puts
---
Porting/bench.pl | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 8da1de5..b474e58 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -481,11 +481,11 @@ sub select_a_perl {
}
-# Validate the list of perl=label on the command line.
-# Return a list of [ exe, label ] pairs.
+# Validate the list of perl=label (+ cmdline options) on the command line.
+# Return a list of [ exe, label, cmdline-options ] tuples, ie PUTs
sub process_puts {
- my @results;
+ my @res_puts; # returned, each item is [ perlexe, label, @putargs ]
my %seen;
my @putargs; # collect not-perls into args per PUT
@@ -498,13 +498,17 @@ sub process_puts {
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
if ($r eq "ok\n") {
- push @results, [ $perl, $label, reverse @putargs ];
+ push @res_puts, [ $perl, $label, reverse @putargs ];
@putargs = ();
+ warn "Added Perl-Under-Test: [ @{[@{$res_puts[-1]}]} ]\n"
+ if $OPTS{verbose};
} else {
+ warn "putargs: @putargs + $p, a not-perl: $r\n"
+ if $OPTS{verbose};
push @putargs, $p; # not-perl
}
}
- return reverse @results;
+ return reverse @res_puts;
}
--
2.5.5
|
From @jimc0009-add-label-feature-which-appends-the-label-to-perl-na.patchFrom 95d8480afb78adc18dbc74063c5c2926ec0bd546 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 11:58:11 -0600
Subject: [PATCH 09/10] add =+label feature, which appends the label to
perl-name
Using this reduces the burden of keeping each label unique,
simplifying the use of labels to name varying workloads created by
-Mfoo -M<something-heavy>
This enables an invocation like the following to run -DDEBUGGING
perls-A,B in bare mode, and heavily encumbered by -Dflag induced
overhead:
$] perl Porting/bench.pl --jobs=2 \
--tests=/loop::for::my/ \
-- \
perl-A=+bare perl-A=+workload -DpsltocPmfr \
perl-B=+bare perl-B=+workload -DpsltocPmfr
Unfortunately, that example falls over, because the -D<blah> output
upsets cachegrind_parser. That might be fixable by removing the
'2>&1' in the $cmd. Nonetheless, workloads are constructible.
more prosaically:
$] perl Porting/bench.pl --jobs=2 \
--tests=/loop::for::my/ \
-- perl5.18.2=+strict -Mstrict perl5.18.0=+dumper -MData::Dumper
yields:
Got eof for pid 11377 (loop::for::my_array4/perl5.18.0+dumper active/long loop)
...
loop::for::my_array4
empty for loop with my var and 4 integer array
perl5.18.2+strict perl5.18.0+dumper
----------------- -----------------
Ir 100.00 114.20
Dr 100.00 119.35
Dw 100.00 135.03
COND 100.00 99.71
IND 100.00 100.00
COND_m 100.00 119.64
IND_m 100.00 92.31
Ir_m1 100.00 100.00
Dr_m1 100.00 100.00
Dw_m1 100.00 100.00
Ir_mm 100.00 -
Dr_mm 100.00 -
Dw_mm 100.00 100.00
AVERAGE
perl5.18.2+strict perl5.18.0+dumper
----------------- -----------------
Ir 100.00 114.56
Dr 100.00 118.87
Dw 100.00 133.30
COND 100.00 100.41
IND 100.00 100.00
COND_m 100.00 69.14
IND_m 100.00 89.95
Ir_m1 100.00 100.00
Dr_m1 100.00 100.00
Dw_m1 100.00 100.00
Ir_mm 100.00 100.00
Dr_mm 100.00 100.00
Dw_mm 100.00 100.00
---
Porting/bench.pl | 1 +
1 file changed, 1 insertion(+)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index b474e58..0d98b3f 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -494,6 +494,7 @@ sub process_puts {
my ($perl, $label) = split /=/, $p, 2;
$label //= $perl;
+ $label = $perl.$label if $label =~ /^\+/;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
--
2.5.5
|
From @jimc0010-provide-per-PUT-environment.patchFrom 0b72fed626276f6f9f881ba983f445506f80436e Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 16:43:46 -0600
Subject: [PATCH 10/10] provide per-PUT environment
This patch lets user specify ENVAR=value pairs for individual PUTs,
thereby altering runtime behavior. For some combination of module and
ENVAR, this should result in measurable benchmark differences.
This example benchmarks a PERL_MEM_LOG configured perl, 1st PUT with
PERL_MEM_LOG disabled, but enabled in the 2nd:
$ perl Porting/bench.pl --jobs=2 \
--verbose --debug --tests=/loop::for::pkg_/ \
-- \
perl5.23.9=+bare \
perl5.23.9=+full=PERL_MEM_LOG=3mst -MData::Dumper
---
Porting/bench.pl | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 0d98b3f..0e2e5be 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -492,14 +492,18 @@ sub process_puts {
for my $p (reverse @_) {
push @putargs, $p and next if $p =~ /^-/; # not-perl, dont send to qx//
- my ($perl, $label) = split /=/, $p, 2;
+ my ($perl, $label, $env) = split /=/, $p, 3;
$label //= $perl;
$label = $perl.$label if $label =~ /^\+/;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
+ my %env;
+ if ($env) {
+ %env = split /[=:]/, $env;
+ }
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
if ($r eq "ok\n") {
- push @res_puts, [ $perl, $label, reverse @putargs ];
+ push @res_puts, [ $perl, $label, \%env, reverse @putargs ];
@putargs = ();
warn "Added Perl-Under-Test: [ @{[@{$res_puts[-1]}]} ]\n"
if $OPTS{verbose};
@@ -724,14 +728,18 @@ sub grind_run {
);
for my $p (@$perls) {
- my ($perl, $label, @putargs) = @$p;
+ my ($perl, $label, $env, @putargs) = @$p;
# Run both the empty loop and the active loop
# $counts->[0] and $counts->[1] times.
for my $i (0,1) {
for my $j (0,1) {
- my $cmd = "PERL_HASH_SEED=0 "
+ my $envstr = '';
+ if (ref $env) {
+ $envstr .= "$_=$env->{$_} " for sort keys %$env;
+ }
+ my $cmd = "PERL_HASH_SEED=0 $envstr"
. "valgrind --tool=cachegrind --branch-sim=yes "
. "--cachegrind-out-file=/dev/null "
. "$OPTS{grindargs} "
--
2.5.5
|
From @tonycozOn Tue Apr 12 09:35:30 2016, yoduh wrote:
0006-grind_print-adjust-for-labels.patch @@ -1108,7 +1109,8 @@ sub grind_print { # Print standard header. left over working edit I guess. 0010-provide-per-PUT-environment.patch + my %env; Pity I can't test PERLIO=:stdio vs PERLIO=:perlio, though I'm not sure what would be a better separator. Other than that it all looks good. Tony |
|
The RT System itself - Status changed from 'new' to 'open' |
From @jimc
Perhaps the right separator is a comma, So, I went with that, attaching v2 of 0006 - with new separators 0010-provide-per-PUT-environment.patch - as above
|
From @jimc0006-grind_print-adjust-for-labels.patchFrom c3cd0c0d83e57eac4ae1ecc433d43f4155d77125 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 00:15:31 -0600
Subject: [PATCH 06/11] grind_print: adjust for labels
since HEAD~1, data is stored by the label, not the perlname.
simplify grind-print based upon this.
---
Porting/bench.pl | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index ab62069..7cf0cd1 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -945,7 +945,7 @@ sub grind_process {
my %counts;
my %data;
- my $perl_norm = $perls->[$OPTS{norm}][0]; # the name of the reference perl
+ my $perl_norm = $perls->[$OPTS{norm}][1]; # the label of the reference perl
for my $test_name (keys %$res) {
my $res1 = $res->{$test_name};
@@ -1101,6 +1101,7 @@ sub grind_print {
my ($results, $averages, $perls, $tests, $order) = @_;
my @perl_names = map $_->[0], @$perls;
+ my @perl_labels = map $_->[1], @$perls;
my %perl_labels;
$perl_labels{$_->[0]} = $_->[1] for @$perls;
@@ -1108,7 +1109,7 @@ sub grind_print {
# Calculate the width to display for each column.
my $min_width = $OPTS{raw} ? 8 : 6;
my @widths = map { length($_) < $min_width ? $min_width : length($_) }
- @perl_labels{@perl_names};
+ @perl_labels;
# Print standard header.
grind_blurb($perls);
@@ -1138,7 +1139,7 @@ sub grind_print {
print " " x $field_label_width;
for (0..$#widths) {
printf " %*s", $widths[$_],
- $i ? ('-' x$widths[$_]) : $perl_labels{$perl_names[$_]};
+ $i ? ('-' x$widths[$_]) : $perl_labels[$_];
}
print "\n";
}
@@ -1160,7 +1161,7 @@ sub grind_print {
print " " x $field_label_width;
for (0..$#widths) {
printf " %*s", $widths[$_],
- $i ? ('-' x$widths[$_]) : $perl_labels{$perl_names[$_]};
+ $i ? ('-' x$widths[$_]) : $perl_labels[$_];
}
print "\n";
}
@@ -1190,7 +1191,7 @@ sub grind_print {
}
for my $i (0..$#widths) {
- my $res2 = $res1->{$perl_names[$i]};
+ my $res2 = $res1->{$perl_labels[$i]};
my $p = $res2->{$field};
if (!defined $p) {
printf " %*s", $widths[$i], '-';
--
2.5.5
|
From @jimc0010-provide-per-PUT-environment.patchFrom 10012becba8e65248234bc32846a1a49798979f7 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 16:43:46 -0600
Subject: [PATCH 10/11] provide per-PUT environment
This patch lets user specify ENVAR=value pairs for individual PUTs,
thereby altering runtime behavior. For some combination of module and
ENVAR, this should result in measurable benchmark differences.
This example benchmarks a PERL_MEM_LOG configured perl, 1st PUT with
PERL_MEM_LOG disabled, but enabled in the 2nd:
$ perl Porting/bench.pl --jobs=2 \
--verbose --debug --tests=/loop::for::pkg_/ \
-- \
perl5.23.9:+bare \
perl5.23.9=+full:PERL_MEM_LOG=3mst,PERLIO=:stdio -MData::Dumper
this will run:
Command: PERL_HASH_SEED=0 PERLIO=:stdio PERL_MEM_LOG=3mst valgrind --tool=cachegrind --branch-sim=yes --cachegrind-out-file=/dev/null perl5.23.9 -MData::Dumper - 20 2>&1
Note that ':' is now a valid separator between the perl & label, but
not between ENVAR=value pairs, where a ',' is required.
---
Porting/bench.pl | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 4fabb49..e48d866 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -492,19 +492,23 @@ sub process_puts {
for my $p (reverse @_) {
push @putargs, $p and next if $p =~ /^-/; # not-perl, dont send to qx//
- my ($perl, $label) = split /=/, $p, 2;
+ my ($perl, $label, $env) = split /[=:,]/, $p, 3;
$label //= $perl;
$label = $perl.$label if $label =~ /^\+/;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
+ my %env;
+ if ($env) {
+ %env = split /[=,]/, $env;
+ }
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
if ($r eq "ok\n") {
- push @res_puts, [ $perl, $label, reverse @putargs ];
+ push @res_puts, [ $perl, $label, \%env, reverse @putargs ];
@putargs = ();
warn "Added Perl-Under-Test: [ @{[@{$res_puts[-1]}]} ]\n"
if $OPTS{verbose};
} else {
- warn "putargs: @putargs + $p, a not-perl: $r\n"
+ warn "PUT-args: @putargs + a not-perl: $p $r\n"
if $OPTS{verbose};
push @putargs, $p; # not-perl
}
@@ -724,14 +728,18 @@ sub grind_run {
);
for my $p (@$perls) {
- my ($perl, $label, @putargs) = @$p;
+ my ($perl, $label, $env, @putargs) = @$p;
# Run both the empty loop and the active loop
# $counts->[0] and $counts->[1] times.
for my $i (0,1) {
for my $j (0,1) {
- my $cmd = "PERL_HASH_SEED=0 "
+ my $envstr = '';
+ if (ref $env) {
+ $envstr .= "$_=$env->{$_} " for sort keys %$env;
+ }
+ my $cmd = "PERL_HASH_SEED=0 $envstr"
. "valgrind --tool=cachegrind --branch-sim=yes "
. "--cachegrind-out-file=/dev/null "
. "$OPTS{grindargs} "
--
2.5.5
|
From @jimc0011-Porting-bench.pl-grindargs-can-be-abused-usefully.patchFrom 5b1b55b02075de6547acbf5a696c6b7e98cc9c90 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Tue, 12 Apr 2016 16:59:37 -0600
Subject: [PATCH 11/11] Porting/bench.pl --grindargs can be abused, usefully.
Add an example where --grindargs='3>foobar' opens an extra file
descriptor for writing, and PERL_MEM_LOG=3 causes perl to log to the
file instead of STDERR, so that the output doesnt cause
parse_cachegrind() to choke, which would abort the test.
The example also demonstrates overriding default grind options, ie
--cachegrind-out-file=junk.$$. This has no obvious utility at
present, and perhaps buries the lead.
---
Porting/bench.pl | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index e48d866..9c0f7b1 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -117,6 +117,37 @@ If only one field is selected, the output is in more compact form.
Optional command-line arguments to pass to cachegrind invocations.
+This option is appended to those which bench.pl uses for its own
+purposes; so it can be used to override them (see --debug output
+below), and can also be 'abused' to add redirects into the valgrind
+command invocation.
+
+For example, this writes PERL_MEM_LOG activity to foobar, because
+3>foobar redirects fd 3, then perl under PERL_MEM_LOG writes to fd 3.
+
+ $ perl Porting/bench.pl --jobs=2 --verbose --debug \
+ --tests=call::sub::amp_empty \
+ --perlargs='-MDevel::Peek' \
+ \
+ --grindargs='--cachegrind-out-file=junk.$$ 3>foobar' \
+ -- \
+ perl5.24.0 -e'"BEGIN{runops_debug(0)}"' \
+ perl5.24.0:+traceops:PERL_MEM_LOG=3mst -e'"BEGIN{runops_debug(1)}"'
+
+yields (with --debug, prettyfied)
+
+ Command: PERL_HASH_SEED=0
+ PERLIO=:stdio PERL_MEM_LOG=3mst PERL_TRACE_OPS=
+ valgrind --tool=cachegrind --branch-sim=yes
+ --cachegrind-out-file=/dev/null --cachegrind-out-file=junk.$$ 3>foobar
+ perl5.24.0 -MDevel::Peek -e"BEGIN{runops_debug(1)}" - 10 2>&1
+
+The result is that a set of junk.$$ files are written, foobar contains
+expected debug-output, and parse_cachegrind() never sees that output,
+so it doesnt reject it. This then allows the test to complete.
+Theres no obvious utility for those junk.$$ files, but you can have
+them anyway.
+
=item *
---help
--
2.5.5
|
From @jimcrevised patch 10/11 attached. now handles per-PUT envars in the order given, this lets them be used to insert redirections, $ perl Porting/bench.pl --jobs=2 \ |
From @jimc0010-provide-per-PUT-environment.patchFrom 19583debdbfe53b2c79d71afc8c27330e120b2e6 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 16:43:46 -0600
Subject: [PATCH 10/11] provide per-PUT environment
This patch lets user specify ENVAR=value pairs for individual PUTs,
thereby altering runtime behavior for it. For some combination of
module and ENVAR, this should result in measurable benchmark
differences.
This example benchmarks a PERL_MEM_LOG configured perl, 1st PUT with
PERL_MEM_LOG disabled, but enabled in the 2nd:
$ perl Porting/bench.pl --jobs=2 \
--verbose --debug --tests=/loop::for::pkg_/ \
-- \
perl5.23.9:+bare \
perl5.23.9=+full:PERL_MEM_LOG=3mst,PERLIO=:stdio,=3\>/dev/null -MData::Dumper
this will run:
Command: PERL_HASH_SEED=0 PERL_MEM_LOG=3mst PERLIO=:stdio 3>/dev/null valgrind -
-tool=cachegrind --branch-sim=yes --cachegrind-out-file=/dev/null perl5.24.0
-MData::Dumper - 10 2>&1
Notes:
':' is now a valid separator after the perl-name & label, but not
between ENVAR=value pairs, where a ',' is required. This allows pairs
like PERLIO=:stdio, but disallows values with commas.
Empty ENVARS are allowed, they are emitted as values only, not as
var=val pairs. This allows them to be used to insert redirections,
such as the '3>/dev/null' in the example above. Note the 3\> escape.
ENVARs are processed in the order given, allowing multiple redirects,
and since they can be order dependent.
---
Porting/bench.pl | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index e74fdeb..5f5c8c3 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -492,23 +492,29 @@ sub process_puts {
for my $p (reverse @_) {
push @putargs, $p and next if $p =~ /^-/; # not-perl, dont send to qx//
- my ($perl, $label) = split /=/, $p, 2;
+ my ($perl, $label, $env) = split /[=:,]/, $p, 3;
$label //= $perl;
$label = $perl.$label if $label =~ /^\+/;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
+ my @env;
+ if ($env) {
+ @env = split /[=,]/, $env;
+ }
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
if ($r eq "ok\n") {
- push @res_puts, [ $perl, $label, reverse @putargs ];
+ push @res_puts, [ $perl, $label, [@env], reverse @putargs ];
@putargs = ();
warn "Added Perl-Under-Test: [ @{[@{$res_puts[-1]}]} ]\n"
if $OPTS{verbose};
} else {
- warn "putargs: @putargs + $p, a not-perl: $r\n"
+ warn "PUT-args: @putargs + a not-perl: $p $r\n"
if $OPTS{verbose};
push @putargs, $p; # not-perl
}
}
+ warn "PUT-args: @putargs lack a PUT\n" if @putargs;
+
return reverse @res_puts;
}
@@ -724,14 +730,19 @@ sub grind_run {
);
for my $p (@$perls) {
- my ($perl, $label, @putargs) = @$p;
+ my ($perl, $label, $env, @putargs) = @$p;
# Run both the empty loop and the active loop
# $counts->[0] and $counts->[1] times.
for my $i (0,1) {
for my $j (0,1) {
- my $cmd = "PERL_HASH_SEED=0 "
+ my $envstr = '';
+ while (my ($var, $val) = splice(@$env, 0, 2)) {
+ $envstr .= ($var ne '')
+ ? "$var=$val " : "$val ";
+ }
+ my $cmd = "PERL_HASH_SEED=0 $envstr"
. "valgrind --tool=cachegrind --branch-sim=yes "
. "--cachegrind-out-file=/dev/null "
. "$OPTS{grindargs} "
--
2.5.5
|
From @tonycozOn Mon Apr 25 11:48:09 2016, yoduh wrote:
+ $ perl Porting/bench.pl --jobs=2 --verbose --debug \ where did the PERLIO=:stdio come from? Tony |
From @jimcoh dear, probly a cut-paste err I'll take a close look.. On Tue, May 10, 2016 at 6:01 PM, Tony Cook via RT
|
From @jimchi Tony, Ive simplified the --grindargs example, taking out not-relevant The prose has extra focus on bench.pl plumbing, extra 2 patches are tiny fixups, applicable to earlier patches. Whats missing ? I will be doing a benchrun of perl5.25.0 with PERL_TRACE_OPS on and off |
From @jimc0011-Porting-bench.pl-grindargs-can-be-abused-usefully.patchFrom ae3c703c08fe2481d4987f9dd7e48a0a17271a90 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Tue, 12 Apr 2016 16:59:37 -0600
Subject: [PATCH 11/13] Porting/bench.pl --grindargs can be abused, usefully.
Add an example where --grindargs='3>foobar' opens an extra file
descriptor for writing, and PERL_MEM_LOG=3 causes perl to log to the
file instead of STDERR, so that the output doesnt cause
parse_cachegrind() to choke, which would abort the test.
The example also demonstrates overriding default grind options, ie
--cachegrind-out-file=junk.$$. This has no obvious utility at
present, and perhaps buries the lead.
---
Porting/bench.pl | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 64 insertions(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 5f5c8c3..a635f95 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -115,7 +115,70 @@ If only one field is selected, the output is in more compact form.
--grindargs=I<foo>
-Optional command-line arguments to pass to cachegrind invocations.
+Optional command-line arguments to pass to all cachegrind invocations.
+
+This option is appended to those which bench.pl uses for its own
+purposes; so it can be used to override them (see --debug output
+below), and can also be 'abused' to add redirects into the valgrind
+command invocation.
+
+For example, this writes PERL_MEM_LOG activity to foobar.$$, because
+3>foobar.$$ redirects fd 3, then perl under PERL_MEM_LOG writes to fd 3.
+
+ $ perl Porting/bench.pl --jobs=2 --verbose --debug \
+ --tests=call::sub::amp_empty \
+ \
+ --grindargs='--cachegrind-out-file=junk.$$ 3>foobar.$$' \
+ -- \
+ perl5.24.0 perl5.24.0:+memlog:PERL_MEM_LOG=3mst
+
+for the +memlog tests, this executes as: (shown via --debug, then prettyfied)
+
+ Command: PERL_HASH_SEED=0 PERL_MEM_LOG=3mst
+ valgrind --tool=cachegrind --branch-sim=yes
+ --cachegrind-out-file=/dev/null --cachegrind-out-file=junk.$$ 3>foobar.$$
+ perl5.24.0 - 10 2>&1
+
+The result is that a set of junk.$$ files containing raw cachegrind
+output are written, and foobar.$$ contains the expected memlog output.
+
+Notes:
+
+Theres no obvious utility for those junk.$$ and foobar.$$ files, but
+you can have them anyway.
+
+The 3 in PERL_MEM_LOG=3mst is needed because the output would
+otherwize go to STDERR, and cause parse_cachegrind() to reject the
+test and die.
+
+The --grindargs redirect is needed to capture the memlog output;
+without it, the memlog output is written to fd3, around
+parse_cachegrind and effectively into /dev/null
+
+PERL_MEM_LOG is expensive when used.
+
+call::sub::amp_empty
+&foo function call with no args or body
+
+ perl5.24.0 perl5.24.0+memlog
+ ---------- -----------------
+ Ir 394.0 543477.5
+ Dr 161.0 146814.1
+ Dw 72.0 122304.6
+ COND 58.0 66796.4
+ IND 5.0 5537.7
+
+COND_m 0.0 6743.1
+ IND_m 5.0 1490.2
+
+ Ir_m1 0.0 683.7
+ Dr_m1 0.0 65.9
+ Dw_m1 0.0 8.5
+
+ Ir_mm 0.0 11.6
+ Dr_mm 0.0 10.6
+ Dw_mm 0.0 4.7
+
=item *
--
2.5.5
|
From @jimc0012-fixup-synopsis-from-add-pod-for-PUT.patchFrom 990c067d502cc4e52be5a6e7d811a3dc1abcd139 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Wed, 11 May 2016 09:44:08 -0600
Subject: [PATCH 12/13] fixup synopsis - from 'add pod for PUT'
---
Porting/bench.pl | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index a635f95..ee663ae 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -14,9 +14,9 @@ perls.
# Basic: run the tests in t/perf/benchmarks against two or
# more perls
- bench.pl [options] perlA[=labelA] perlB[=labelB] ...
+ bench.pl [options] -- perlA[=labelA] perlB[=labelB] ...
- # Alt: run the tests against same perlA 2x, with and without extra options
+ # run the tests against same perlA 2x, with and without extra options
bench.pl [options] -- perlA=fast PerlA=slow -Mstrict -Dpsltoc
--
2.5.5
|
From @jimc0013-fixup-usage-add-.-from-allow-per-PUT-perl-under-test.patchFrom 394293ae3a7e2b0361997e182af2f51f6a6460af Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Wed, 11 May 2016 09:45:02 -0600
Subject: [PATCH 13/13] fixup usage, add --. from allow per-PUT (perl under
test) options
---
Porting/bench.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index ee663ae..e27d055 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -294,7 +294,7 @@ my %VALID_FIELDS = map { $_ => 1 }
sub usage {
die <<EOF;
-usage: $0 [options] perl[=label] ...
+usage: $0 [options] -- perl[=label] ...
--action=foo What action to perform [default: grind].
--average Only display average, not individual test results.
--benchfile=foo File containing the benchmarks;
--
2.5.5
|
From @tonycozOn Wed May 11 11:22:16 2016, yoduh wrote:
This looks ok, a couple of podcheck errors occur, fixed by the attached. Tony |
From @tonycoz0001-fix-podcheck-errors-in-bench.pl.patchFrom d593b9b7b179c558641448f4d401ac2dce915e82 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Thu, 2 Jun 2016 12:05:27 +1000
Subject: [PATCH] fix podcheck errors in bench.pl
---
Porting/bench.pl | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 2363379..fb06040 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -16,7 +16,8 @@ perls.
bench.pl [options] -- perlA[=labelA] perlB[=labelB] ...
- # run the tests against same perlA 2x, with and without extra options
+ # run the tests against same perlA 2x, with and without extra
+ # options
bench.pl [options] -- perlA=fast PerlA=slow -Mstrict -Dpsltoc
@@ -136,8 +137,8 @@ for the +memlog tests, this executes as: (shown via --debug, then prettyfied)
Command: PERL_HASH_SEED=0 PERL_MEM_LOG=3mst
valgrind --tool=cachegrind --branch-sim=yes
- --cachegrind-out-file=/dev/null --cachegrind-out-file=junk.$$ 3>foobar.$$
- perl5.24.0 - 10 2>&1
+ --cachegrind-out-file=/dev/null --cachegrind-out-file=junk.$$
+ 3>foobar.$$ perl5.24.0 - 10 2>&1
The result is that a set of junk.$$ files containing raw cachegrind
output are written, and foobar.$$ contains the expected memlog output.
--
2.1.4
|
From @jimcOn Wed, Jun 1, 2016 at 10:06 PM, Tony Cook via RT <perlbug-followup@perl.org
I did run those tests, and was unable to see a difference between This left me uncertain as to the cause; Ive been away from home (and the patchset) for past 2 weeks,
I'll add this to the set. I did have some misgivings about parts of this example; plus the addition is somewhat counter to DaveM's brevity. |
From @tonycozOn Wed Jun 01 20:51:10 2016, yoduh wrote:
Thanks, I've squashed the fixup commits (include my podcheck fixes) into their base commits and applied these as commits 2429375 through 8a094fe. Tony |
|
@tonycoz - Status changed from 'open' to 'pending release' |
From @jimcOn Tue, Jun 14, 2016 at 8:01 PM, Tony Cook via RT <perlbug-followup@perl.org
oof. meanwhile I found a bug I was after, and made some further cleanups. Ive reworked them on top of blead now. 0001-fixup-remove-unused-vars.patch 2-3 are fixes for the disappearing-environment bug I was chasing |
From @jimc0001-fixup-remove-unused-vars.patchFrom 96fbb10f5915127f1b708e9df3ec99b03b475f67 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 16 Jun 2016 09:39:42 -0600
Subject: [PATCH 01/14] fixup: remove unused vars
this was noted and fixed previously, but lost in revise-resubmit
---
Porting/bench.pl | 2 --
1 file changed, 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index fb06040..75b919a 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -1183,8 +1183,6 @@ sub grind_print {
my @perl_names = map $_->[0], @$perls;
my @perl_labels = map $_->[1], @$perls;
- my %perl_labels;
- $perl_labels{$_->[0]} = $_->[1] for @$perls;
my $field_label_width = 6;
# Calculate the width to display for each column.
--
2.5.5
|
From @jimc0002-fixup-handle-env-correctly.patchFrom 6274c53cc19c7dba6427060a6ba35ef216172ad1 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 9 Jun 2016 16:13:08 -0600
Subject: [PATCH 02/14] fixup: handle env correctly
fix 2 flaws in ENV handling by 60858fe86c
- hoist $envstr population out of nested 0,1 0,1 loops
the $envstr pertains to all 4 empty/active short/long jobs.
- take my copy of @$env
splice @$env consumed the ENV parsed out of PUT spec,
so only 1st $job got proper environment set,
rest had empty @$env
Also, fix $envstr insertion with trailing space, and add it into $id
which is used in verbose/debug.
And in process_puts(), drop ',' as a split char. Its ok for correct
inputs, but will be misleading otherwise.
---
Porting/bench.pl | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 75b919a..3b7ba7e 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -556,7 +556,7 @@ sub process_puts {
for my $p (reverse @_) {
push @putargs, $p and next if $p =~ /^-/; # not-perl, dont send to qx//
- my ($perl, $label, $env) = split /[=:,]/, $p, 3;
+ my ($perl, $label, $env) = split /[=:]/, $p, 3;
$label //= $perl;
$label = $perl.$label if $label =~ /^\+/;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
@@ -794,22 +794,24 @@ sub grind_run {
for my $p (@$perls) {
my ($perl, $label, $env, @putargs) = @$p;
+ my $envstr = '';
+ my @env = @$env;
+ while (my ($var, $val) = splice(@env, 0, 2)) {
+ $envstr .= ($var ne '')
+ ? "$var=$val " : "$val ";
+ }
# Run both the empty loop and the active loop
# $counts->[0] and $counts->[1] times.
for my $i (0,1) {
for my $j (0,1) {
- my $envstr = '';
- if (ref $env) {
- $envstr .= "$_=$env->{$_} " for sort keys %$env;
- }
- my $cmd = "PERL_HASH_SEED=0 $envstr"
+ my $cmd = "PERL_HASH_SEED=0 $envstr "
. "valgrind --tool=cachegrind --branch-sim=yes "
. "--cachegrind-out-file=/dev/null "
- . "$OPTS{grindargs} "
+ . "$OPTS{grindargs} -- "
. "$perl $OPTS{perlargs} @putargs - $counts->[$j] 2>&1";
# for debugging and error messages
- my $id = "$test/$label "
+ my $id = "$test/$label $envstr "
. ($i ? "active" : "empty") . "/"
. ($j ? "long" : "short") . " loop";
--
2.5.5
|
From @jimc0003-handle-ENV-mods-in-order-w-array-better-diag-warn.patchFrom d7adac625d2e683757b913e8b1b71139d6a24b4f Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Wed, 15 Jun 2016 16:28:05 -0600
Subject: [PATCH 03/14] handle %ENV mods in order, w array, better diag/warn
In process_puts(), use @env not %env to process the $env string.
This preserves the order they were given by user.
Counter to normal expectations, order can matter here because
grind_run() has env handling which supports its use to specify
redirections (PERL_MEM_LOG=3mst,=3\>my-file), and redirections are
generally order dependent (even if not for this example).
---
Porting/bench.pl | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 3b7ba7e..94390ea 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -561,22 +561,27 @@ sub process_puts {
$label = $perl.$label if $label =~ /^\+/;
die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
- my %env;
+ my @env;
if ($env) {
- %env = split /[=,]/, $env;
+ @env = split /[=,]/, $env;
+ $env =~ s/,/ /g; # fixup for display
}
my $r = qx($perl -e 'print qq(ok\n)' 2>&1);
if ($r eq "ok\n") {
- push @res_puts, [ $perl, $label, \%env, reverse @putargs ];
- @putargs = ();
- warn "Added Perl-Under-Test: [ @{[@{$res_puts[-1]}]} ]\n"
+ @putargs = reverse @putargs;
+ push @res_puts, [ $perl, $label, [@env], @putargs ];
+ no warnings 'uninitialized';
+ warn "added Perl-Under-Test: [ $env $perl $label @putargs ]\n"
if $OPTS{verbose};
+ @putargs = ();
} else {
warn "PUT-args: @putargs + a not-perl: $p $r\n"
if $OPTS{verbose};
push @putargs, $p; # not-perl
}
}
+ warn "PUT-args: @putargs lack a PUT\n" if @putargs;
+
return reverse @res_puts;
}
--
2.5.5
|
From @jimc0004-parse_cachegrind-should-identify-failing-job-executi.patchFrom e0d6e138fd8a7f4c134cb09fbd34be14aae561f7 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Tue, 17 May 2016 14:01:03 -0600
Subject: [PATCH 04/14] parse_cachegrind should identify failing job executions
Pass command to parse_cachegrind so it can ID the failing command/job.
While parse_cachegrind has ID already (and could report with that),
command is better because it is what is actually running (and failing).
---
Porting/bench.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 94390ea..0d81a9a 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -609,7 +609,7 @@ EOF
# See do_selftest() for examples of the output format.
sub parse_cachegrind {
- my ($output, $id, $perl) = @_;
+ my ($output, $id, $perl, $cmd) = @_;
my %res;
@@ -617,7 +617,7 @@ sub parse_cachegrind {
for (@lines) {
unless (s/(==\d+==)|(--\d+--) //) {
die "Error: while executing $id:\n"
- . "unexpected code or cachegrind output:\n$_\n";
+ . "unexpected code or cachegrind output:\n$_\n while running: $cmd\n";
}
if (/I refs:\s+([\d,]+)/) {
$res{Ir} = $1;
@@ -945,7 +945,7 @@ sub grind_run {
}
$results{$j->{test}}{$j->{plabel}}[$j->{active}][$j->{loopix}]
- = parse_cachegrind($output, $j->{id}, $j->{perl});
+ = parse_cachegrind($output, $j->{id}, $j->{perl}, $j->{cmd});
}
# Reap finished jobs
--
2.5.5
|
From @jimc0005-if-write-existing-dir-given-write-a-new-data-file-th.patchFrom ab6a165d62d172645e3ce17b867e9a9787f825fd Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 9 Jun 2016 11:12:41 -0600
Subject: [PATCH 05/14] if --write <existing-dir> given, write a new data-file
there.
This option lets user "collect by default", automatically naming the
data-file as existing-dir/'seconds-since-epoch'-$$
---
Porting/bench.pl | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 0d81a9a..a381b5b 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -262,9 +262,11 @@ Display progress information.
-w I<file>
--write=I<file>
+--write=I<directory>
-Save the raw data to the specified file. It can be read back later with
-C<--read>.
+Save the raw data to the specified file or directory. The written file
+can be read back later with C<--read>. If a directory is given, a
+timestamp-pid file is written into the dir.
Requires C<JSON::PP> to be available.
@@ -730,6 +732,9 @@ sub do_grind {
select_a_perl($OPTS{'compact'}, $perls, "--compact");
}
if (defined $OPTS{write}) {
+ if (-d $OPTS{write}) {
+ $OPTS{write} = $OPTS{write} . '/' . time() .'-'.$$;
+ }
my $json = JSON::PP::encode_json({
version => $FORMAT_VERSION,
loop_counts => $loop_counts,
--
2.5.5
|
From @jimc0006-write-shouldnt-suppress-display-of-results.patchFrom c2050ac4639bb903c44d791fb305d03840adcbcd Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Wed, 8 Jun 2016 15:03:07 -0600
Subject: [PATCH 06/14] --write shouldnt suppress display of results
If user gives --write=<existing-dir> to collect data automatically,
she probably still wants to see the reports. Remove the else.
---
Porting/bench.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index a381b5b..24d0521 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -749,7 +749,7 @@ sub do_grind {
print $out $json or die "Error: writing to file '$OPTS{write}': $!\n";
close $out or die "Error: closing file '$OPTS{write}': $!\n";
}
- else {
+ {
my ($processed, $averages) =
grind_process($results, $perls, $loop_counts);
--
2.5.5
|
From @jimc0007-if-read-write-both-given-warn-dont-die.patchFrom 21b040e097cebd656b73c0e48ca2d20ebf050d99 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 9 Jun 2016 19:19:52 -0600
Subject: [PATCH 07/14] if --read --write both given, warn dont die.
Since we want --write=<existing-dir> to be a usable default, lets not
die if --read is also given. Instead just warn and unset --write
flag.
TBD: we could allow the write instead, esp if --write=<existing-dir>
is given, as this cant overwrite the --read file.
---
Porting/bench.pl | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 24d0521..631eb70 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -388,7 +388,8 @@ my %OPTS = (
if (defined $OPTS{read} and defined $OPTS{write}) {
- die "Error: can't specify both --read and --write options\n";
+ warn "Warning: can't specify --write with --read, suppressing it\n";
+ undef $OPTS{write};
}
if (defined $OPTS{read} or defined $OPTS{write}) {
--
2.5.5
|
From @jimc0008-add-quiet-option.patchFrom 76a0a4af343e7ff00f7f3c2f7203618970939249 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 9 Jun 2016 16:53:08 -0600
Subject: [PATCH 08/14] add --quiet option
Last patch made display and --write separate,
Now add --quiet so user can silence the display it optionally.
---
Porting/bench.pl | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 631eb70..635165b 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -213,6 +213,13 @@ Optional command-line arguments to pass to each perl-under-test
=item *
+-q
+--quiet
+
+Skip output printing.
+
+=item *
+
--raw
Display raw data counts rather than percentages in the outputs. This
@@ -325,6 +332,7 @@ usage: $0 [options] -- perl[=label] ...
[default: all tests].
--verbose Display progress information.
-w|--write=file Save the raw data to the specified file.
+ -q|--quiet Suppress printing of tabular result
--action is one of:
grind run the code under cachegrind
@@ -351,6 +359,7 @@ my %OPTS = (
jobs => 1,
norm => 0,
perlargs => '',
+ quiet => 0,
raw => 0,
read => undef,
sort => undef,
@@ -376,6 +385,7 @@ my %OPTS = (
'jobs|j=i' => \$OPTS{jobs},
'norm=s' => \$OPTS{norm},
'perlargs=s' => \$OPTS{perlargs},
+ 'quiet|q' => \$OPTS{quiet},
'raw' => \$OPTS{raw},
'read|r=s' => \$OPTS{read},
'sort=s' => \$OPTS{sort},
@@ -750,7 +760,7 @@ sub do_grind {
print $out $json or die "Error: writing to file '$OPTS{write}': $!\n";
close $out or die "Error: closing file '$OPTS{write}': $!\n";
}
- {
+ unless ($OPTS{quiet}) {
my ($processed, $averages) =
grind_process($results, $perls, $loop_counts);
--
2.5.5
|
From @jimc0009-add-v-to-verbose.patchFrom f8440b4dfc6224d661cf93c4e1b4539eb6a97c4e Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 9 Jun 2016 19:16:59 -0600
Subject: [PATCH 09/14] add -v to --verbose
---
Porting/bench.pl | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 635165b..6180ff0 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -261,6 +261,7 @@ expression. For example
=item *
+-v
--verbose
Display progress information.
@@ -330,7 +331,7 @@ usage: $0 [options] -- perl[=label] ...
--tests=FOO Select only the specified tests from the benchmarks file;
FOO may be either of the form 'foo,bar' or '/regex/';
[default: all tests].
- --verbose Display progress information.
+ -v|--verbose Display progress information.
-w|--write=file Save the raw data to the specified file.
-q|--quiet Suppress printing of tabular result
@@ -390,7 +391,7 @@ my %OPTS = (
'read|r=s' => \$OPTS{read},
'sort=s' => \$OPTS{sort},
'tests=s' => \$OPTS{tests},
- 'verbose' => \$OPTS{verbose},
+ 'verbose|v' => \$OPTS{verbose},
'write|w=s' => \$OPTS{write},
) or usage;
--
2.5.5
|
From @jimc0010-add-git-option-to-use-V-git_describe-as-label.patchFrom 48bf6d325d70018c7c6d6afd19a5832f392e5763 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Wed, 15 Jun 2016 16:25:23 -0600
Subject: [PATCH 10/14] add --git option, to use -V:git_describe as label
Add --git option, which changes the default label from exe-name to
`git describe`.
Since c385646f2, the label is used as a storage key in --write data
files. So use of --git also affects the key used to store the results
data.
---
Porting/bench.pl | 22 +++++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 6180ff0..451f736 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -114,6 +114,13 @@ If only one field is selected, the output is in more compact form.
=item *
+--git
+
+use -V:git_describe as a label in the test results. This also affects
+the label used to store the data written with --write
+
+=item *
+
--grindargs=I<foo>
Optional command-line arguments to pass to all cachegrind invocations.
@@ -355,6 +362,7 @@ my %OPTS = (
bisect => undef,
compact => undef,
debug => 0,
+ git => 0,
grindargs => '',
fields => undef,
jobs => 1,
@@ -380,6 +388,7 @@ my %OPTS = (
'bisect=s' => \$OPTS{bisect},
'compact=s' => \$OPTS{compact},
'debug' => \$OPTS{debug},
+ 'git|g' => \$OPTS{git},
'grindargs=s' => \$OPTS{grindargs},
'help' => \$OPTS{help},
'fields=s' => \$OPTS{fields},
@@ -571,9 +580,16 @@ sub process_puts {
push @putargs, $p and next if $p =~ /^-/; # not-perl, dont send to qx//
my ($perl, $label, $env) = split /[=:]/, $p, 3;
- $label //= $perl;
- $label = $perl.$label if $label =~ /^\+/;
- die "$label cannot be used on 2 different PUTs\n" if $seen{$label}++;
+ my $gdesc;
+ if ($OPTS{git}) {
+ $gdesc = qx($perl -Ilib -V::git_describe:);
+ $gdesc =~ s/' ?//g;
+ }
+ $gdesc //= $perl;
+ $label //= $gdesc;
+ $label = $gdesc.$label if $label =~ /^\+/;
+ die "label:$label cannot be used on 2 different PUTs\n"
+ if $seen{$label}++;
my @env;
if ($env) {
--
2.5.5
|
From @jimc0011-extend-label-feature-to-label.patchFrom 38830bf1dacc9666e80b4f2aeab8afbb5832bb0b Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 12 Jun 2016 06:53:33 -0600
Subject: [PATCH 11/14] extend +label feature to -label
Treat -label like +label, ie as an appending label.
.label form is avoided, despite its nmemonic appeal,
because then "./perl" looks like an appending label,
resulting in "./perl./perl" in the report.
---
Porting/bench.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index 451f736..f57718c 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -587,7 +587,7 @@ sub process_puts {
}
$gdesc //= $perl;
$label //= $gdesc;
- $label = $gdesc.$label if $label =~ /^\+/;
+ $label = $gdesc.$label if $label =~ /^[\+\-]/;
die "label:$label cannot be used on 2 different PUTs\n"
if $seen{$label}++;
--
2.5.5
|
From @jimc0012-if-verbose-given-print-workload-estimate.patchFrom 7700c9cd944e36c3e65e8b6e358ad937d71d506a Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sat, 11 Jun 2016 21:48:25 -0600
Subject: [PATCH 12/14] if --verbose given, print workload estimate
warn about #tests running on #PUTs -> #jobs queued.
Job count can be high, and runtimes long, let the user know.
---
Porting/bench.pl | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index f57718c..e1be7d1 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -742,6 +742,10 @@ sub do_grind {
$perls = [ process_puts(@$perl_args) ];
+ warn(scalar keys %$tests, " tests, on ", scalar @$perls,
+ " PUTs -> ", (scalar keys %$tests) * (scalar @$perls),
+ " jobs to be queued\n")
+ if $OPTS{verbose};
$results = grind_run($tests, $order, $perls, $loop_counts);
}
--
2.5.5
|
From @jimc0013-parse-PBENCH_PL-envar-for-bench.pl-default-options.patchFrom b38dc173bd0a0b758550f25ef993b11eba18d8d5 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Thu, 9 Jun 2016 12:52:44 -0600
Subject: [PATCH 13/14] parse PBENCH_PL envar for bench.pl default options
useful for --jobs=N etc, stuff you dont want to retype.
Ive been using:
--write=./bench-runs --tests=/call::sub/ --jobs=8 --verbose --git
---
Porting/bench.pl | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index e1be7d1..a03b5d5 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -381,6 +381,9 @@ my %OPTS = (
# process command-line args and call top-level action
{
+ if ($ENV{PBENCH_PL}) {
+ unshift @ARGV, split /\s+/, $ENV{PBENCH_PL};
+ }
GetOptions(
'action=s' => \$OPTS{action},
'average' => \$OPTS{average},
@@ -780,6 +783,7 @@ sub do_grind {
or die " Error: can't open $OPTS{write} for writing: $!\n";
print $out $json or die "Error: writing to file '$OPTS{write}': $!\n";
close $out or die "Error: closing file '$OPTS{write}': $!\n";
+ warn "wrote data to $OPTS{write}\n";
}
unless ($OPTS{quiet}) {
my ($processed, $averages) =
--
2.5.5
|
From @jimc0014-add-pod-for-PUT.patchFrom ad87327bb240bd41ab1199b1cce38cf79f369571 Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Sun, 10 Apr 2016 07:46:40 -0600
Subject: [PATCH 14/14] add pod for PUT
add =head2 PERLS UNDER TEST, right after DESCRIPTION
Its a bit light on =markup.
C<=label> didnt work (markup) for me.
Its also light on noting ':' vs '=' separator
---
Porting/bench.pl | 47 +++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 45 insertions(+), 2 deletions(-)
diff --git a/Porting/bench.pl b/Porting/bench.pl
index a03b5d5..1009bc0 100755
--- a/Porting/bench.pl
+++ b/Porting/bench.pl
@@ -21,6 +21,20 @@ perls.
bench.pl [options] -- perlA=fast PerlA=slow -Mstrict -Dpsltoc
+ # run tests on ./perl, in several ways
+ # use per PUT ENV and args to modify workload
+
+ export PBENCH_PL=' --verbose --write=./bench-runs/ --git '
+ bench.pl [opts] -- \
+ ./perl=+bare \
+ ./perl=+Dq -Dq \
+ ./perl:+traceops:PERL_TRACE_OPS=0 \
+ ./perl:+memlog:PERL_MEM_LOG=3mst \
+ ./perl:+trcDq:PERL_TRACE_OPS=0 -Dq \
+ ./perl:+memDq:PERL_MEM_LOG=3mst -Dq \
+ ./perl:+trcmem:PERL_TRACE_OPS=0,PERL_MEM_LOG=3mst \
+ ./perl:+trcmemDq:PERL_TRACE_OPS=0,PERL_MEM_LOG=3mst -Dq
+
# Run bench.pl's own built-in sanity tests
bench.pl --action=selftest
@@ -44,8 +58,37 @@ There are options to write the raw data to a file, and to read it back.
This means that you can view the same run data in different views with
different selection and sort options.
-The optional C<=label> after each perl executable is used in the display
-output.
+=head2 PERLS UNDER TEST
+
+One or more perls-under-test (PUTs) are expected on the command-line,
+after '--' terminates option processing.
+
+each PUT consists of:
+
+ a perl name or executable (path etc)
+ an optional =label (or :label :+label )
+ an optional ENV spec:
+ :PERL_MEM_LOG=3mst,PERL_TRACE_OPS=0
+
+ 0 or more additional arguments, options
+ for example:
+ -Dq -Ilib -MDevel::Peek -e'"BEGIN{runops_debug(1)}"'
+
+The label defaults to perl-name, and must be unique. It is used to
+store and display the results. :+label appends the label, rather than
+overriding the default.
+
+Arguments and Environment can be given to alter the workload imposed
+by the tests. See the 2nd SYNOPSIS example, and note the following:
+
+ each line after -- is a separate PUT
+ all PUTs are using ./perl, other perls on $PATH work too
+ 1st 2 PUTs use '=' to define labels
+ other PUTs use ':' alternative
+ all PUTs have distinct labels (required)
+ all PUTs use +label to append rather than override the default
+ 2nd PUT has -Dq option. this has surprising cost.
+ PERL_MEM_LOG=3mst is quite expensive (as one would expect)
=head1 OPTIONS
--
2.5.5
|
|
@tonycoz - Status changed from 'pending release' to 'open' |
From @iabynOn Tue, Apr 12, 2016 at 09:35:31AM -0700, Jim Cromie wrote:
I'm not entirely happy with this new syntax and implementation. Firstly, the documentation doesn't make it clear what are perls and perlA -perlA_arg1 -perlA_arg2 perlB -perlB_arg1 -perlB_arg2 Secondly it introduces a bug whereby a misspelled perl executable $ Porting/bench.pl nosuchperl perl5240 quite happily benchmarks just perl5240 with no warnings. Third, the determination of what is a perl executable and what is an arg Porting/bench.pl \ although I guess that could run afoul of executable pathnames with spaces -- |
From @ap* Dave Mitchell <davem@iabyn.com> [2016-09-27 17:12]:
Seems obvious enough to me… how about this? First, of all, benchmarking 3 perls, with no args possible, has Porting/bench.pl perlA perlB perlC Then, if you need args, add --cmd to say so, and separate the command Porting/bench.pl \ So the initial shorthand example can also be written longhand as Porting/bench.pl --cmd perlA :: perlB :: perlC Does that work for you? Regards, |
From @jimcOn Tue, Sep 27, 2016 at 9:02 AM, Dave Mitchell via RT <
yes, that is the intent. It may be that those examples are in the unapplied 2nd batch of patches. Secondly it introduces a bug whereby a misspelled perl executable
quite happily benchmarks just perl5240 with no warnings. OK. thats probably due to handling the perl-args in reverse order, Also, note that the bare '--' arg is needed to terminate the Getopt::Long
Ack - after reversing the post '--' arg-scan, I added the -dash check
Aristotle's suggestion to use prove syntax may provide a way forward
|
From @iabynOn Tue, Sep 27, 2016 at 04:01:34PM +0100, Dave Mitchell wrote:
I've just pushed a branch with about 24 commits that fixes up bench.pl a I decided to handle per-perl args and environment using --args and --env Porting/bench.pl \ Here's the merge commit message: commit 4c95ee9 [MERGE] test and fix up Porting/bench.pl -- |
|
@iabyn - Status changed from 'open' to 'resolved' |
Migrated from rt.perl.org#127885 (status was 'resolved')
Searchable as RT127885$
The text was updated successfully, but these errors were encountered: