From @khwilliamson

Standardize some mechanism so that tests that fail on smokers can be instead made TODO or skipped
For instance, we could tell Time​::Hi-Res that it was running on a loaded VM, so it wouldn't generate failures

--
Karl Williamson

From @jkeenan

On Thu, 07 Apr 2016 02​:37​:10 GMT, khw wrote​:

Standardize some mechanism so that tests that fail on smokers can be
instead made TODO or skipped
For instance, we could tell Time​::Hi-Res that it was running on a
loaded VM, so it wouldn't generate failures

In my FreeBSD-10.3 VM, the last test found in t/re/speed.t is the one that most frequently fails​:

#####
  {
  # [perl #127855] Slowdown in m//g on COW strings of certain lengths
  # this should take milliseconds, but took 10's of seconds.
  my $elapsed= -time;
  my $len= 4e6;
  my $zeros= 40000;
  my $str= ( "0" x $zeros ) . ( "1" x ( $len - $zeros ) );
  my $substr= substr( $str, 1 );
  1 while $substr=~m/0/g;
  $elapsed += time;
  ok( $elapsed <= 1, "should not COW on long string with substr and m//g");
  }
#####

See, e.g., http​://perl5.test-smoke.org/report/57470.

All smoke test runs on this platform are built with a threaded perl. The test seems to fail most frequently on DEBUGGING builds. Whenever I run the test file by itself, that is, outside the smoke-test run, it always PASSes.

How would you diagnose or address that?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

The RT System itself - Status changed from 'new' to 'open'

From @iabyn

On Sat, Sep 02, 2017 at 04​:56​:55PM -0700, James E Keenan via RT wrote​:

On Thu, 07 Apr 2016 02​:37​:10 GMT, khw wrote​:

Standardize some mechanism so that tests that fail on smokers can be
instead made TODO or skipped
For instance, we could tell Time​::Hi-Res that it was running on a
loaded VM, so it wouldn't generate failures

In my FreeBSD-10.3 VM, the last test found in t/re/speed.t is the one that most frequently fails​:

#####
{
# [perl #127855] Slowdown in m//g on COW strings of certain lengths
# this should take milliseconds, but took 10's of seconds.
my $elapsed= -time;
my $len= 4e6;
my $zeros= 40000;
my $str= ( "0" x $zeros ) . ( "1" x ( $len - $zeros ) );
my $substr= substr( $str, 1 );
1 while $substr=~m/0/g;
$elapsed += time;
ok( $elapsed <= 1, "should not COW on long string with substr and m//g");
}
#####

See, e.g., http​://perl5.test-smoke.org/report/57470.

All smoke test runs on this platform are built with a threaded perl. The test seems to fail most frequently on DEBUGGING builds. Whenever I run the test file by itself, that is, outside the smoke-test run, it always PASSes.

How would you diagnose or address that?

Is it actually failing test 58, or is it that the watchdog timeout is
triggering during test 58?

Try 'use Time​::HiRes qw(time);' and print the value of $elapsed just before
the ok test, e.g.

+++ b/t/re/speed.t
@​@​ -146,6 +146,7 @​@​ PROG
  {
  # [perl #127855] Slowdown in m//g on COW strings of certain lengths
  # this should take milliseconds, but took 10's of seconds.
+ use Time​::HiRes qw(time);
  my $elapsed= -time;
  my $len= 4e6;
  my $zeros= 40000;
@​@​ -153,6 +154,7 @​@​ PROG
  my $substr= substr( $str, 1 );
  1 while $substr=~m/0/g;
  $elapsed += time;
+ printf "e=%.6f\n", $elapsed;
  ok( $elapsed <= 1, "should not COW on long string with substr and m//g");
  }

Then run it several times and see what sort of values you get.
On my system is see values of around 0.02s; if your system is taking
nearer to 1s to run that test, then it could occasionally overlap two
1s clock ticks and fail the test. In that case it would be as simple as
changing the '$elapsed <= 1' condition to '$elapsed <= 2' or maybe 3.

--
In England there is a special word which means the last sunshine
of the summer. That word is "spring".

From @khwilliamson

On Mon, 04 Sep 2017 01​:29​:40 -0700, davem wrote​:

On Sat, Sep 02, 2017 at 04​:56​:55PM -0700, James E Keenan via RT wrote​:

On Thu, 07 Apr 2016 02​:37​:10 GMT, khw wrote​:

Standardize some mechanism so that tests that fail on smokers can
be
instead made TODO or skipped
For instance, we could tell Time​::Hi-Res that it was running on a
loaded VM, so it wouldn't generate failures

In my FreeBSD-10.3 VM, the last test found in t/re/speed.t is the one
that most frequently fails​:

#####
{
# [perl #127855] Slowdown in m//g on COW strings of certain
lengths
# this should take milliseconds, but took 10's of seconds.
my $elapsed= -time;
my $len= 4e6;
my $zeros= 40000;
my $str= ( "0" x $zeros ) . ( "1" x ( $len - $zeros ) );
my $substr= substr( $str, 1 );
1 while $substr=~m/0/g;
$elapsed += time;
ok( $elapsed <= 1, "should not COW on long string with substr and
m//g");
}
#####

See, e.g., http​://perl5.test-smoke.org/report/57470.

All smoke test runs on this platform are built with a threaded perl.
The test seems to fail most frequently on DEBUGGING builds. Whenever
I run the test file by itself, that is, outside the smoke-test run,
it always PASSes.

How would you diagnose or address that?

Is it actually failing test 58, or is it that the watchdog timeout is
triggering during test 58?

Try 'use Time​::HiRes qw(time);' and print the value of $elapsed just
before
the ok test, e.g.

+++ b/t/re/speed.t
@​@​ -146,6 +146,7 @​@​ PROG
{
# [perl #127855] Slowdown in m//g on COW strings of certain
lengths
# this should take milliseconds, but took 10's of seconds.
+ use Time​::HiRes qw(time);
my $elapsed= -time;
my $len= 4e6;
my $zeros= 40000;
@​@​ -153,6 +154,7 @​@​ PROG
my $substr= substr( $str, 1 );
1 while $substr=~m/0/g;
$elapsed += time;
+ printf "e=%.6f\n", $elapsed;
ok( $elapsed <= 1, "should not COW on long string with substr
and m//g");
}

Then run it several times and see what sort of values you get.
On my system is see values of around 0.02s; if your system is taking
nearer to 1s to run that test, then it could occasionally overlap two
1s clock ticks and fail the test. In that case it would be as simple
as
changing the '$elapsed <= 1' condition to '$elapsed <= 2' or maybe 3.

Was there any follow up to this?
--
Karl Williamson

From @khwilliamson

On Wed, 06 Apr 2016 19​:37​:10 -0700, khw wrote​:

Standardize some mechanism so that tests that fail on smokers can be
instead made TODO or skipped
For instance, we could tell Time​::Hi-Res that it was running on a
loaded VM, so it wouldn't generate failures

What if we create an environment variable something like PERL_IS_VM or PERL_RUNNING_IN_VM
and then create subs like skip_if_vm() todo_if_vm()
--
Karl Williamson