From @zoffixznet

Recently, we have had about ~2.7% overall performance loss, when running stresstest/spectest that a couple of devs noticed.
Tux's bench also showed a slowdown.

Based on stresstest timings I post ( https://irclog.perlgeek.de/perl6-dev/search/?nick=&q=ZOFVM ), the slowdown happened in
some commits made between 2017-05-31 and 2017-06-01. I tried narrowing down what brought the slowdown, but couldn't quite
pinpoint it, because there's some randomness in timings of some tests. Here's my results of test times​:

SHA Stresstest (s) Spectest (s)
aebd0fa 123
01d948d 122
2500e50 121 / 122 86
79b8ab9 119 / 118 83
b879060 117 / 119 82 / 82
3f5cc5a 117
3488914 118
fb7dd8a 119

As you can see, the middle commits is where the slowage appears to occur, with 2500e50 being the most likely culprit,
even though, the code itself isn't something that would be even remotely hot.

I went and rewrote it in NQP (attached) to see if it drops the numbers at all, and... it didn't really.
I got a 120s stresstest and a 85s spectest.

I gave up with this.

I think we need some extensive bench without any random sleeps or delays in it and comprehensive enough to test more than
just a few text methods. Perhaps, something that can be cooked up with https://map.perl6.party/

And with such a bench in hand, it might become easier to hunt commits that slow stuff down considerably.

From @zoffixznet

exception.diff

From @lizmat

On 5 Jun 2017, at 19​:30, Zoffix Znet (via RT) <perl6-bugs-followup@​perl.org> wrote​:

# New Ticket Created by Zoffix Znet
# Please include the string​: [perl #​131515]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl6/Ticket/Display.html?id=131515 >

Recently, we have had about ~2.7% overall performance loss, when running stresstest/spectest that a couple of devs noticed.
Tux's bench also showed a slowdown.

Based on stresstest timings I post ( https://irclog.perlgeek.de/perl6-dev/search/?nick=&q=ZOFVM ), the slowdown happened in
some commits made between 2017-05-31 and 2017-06-01. I tried narrowing down what brought the slowdown, but couldn't quite
pinpoint it, because there's some randomness in timings of some tests. Here's my results of test times​:

SHA Stresstest (s) Spectest (s)
aebd0fa 123
01d948d 122
2500e50 121 / 122 86
79b8ab9 119 / 118 83
b879060 117 / 119 82 / 82
3f5cc5a 117
3488914 118
fb7dd8a 119

As you can see, the middle commits is where the slowage appears to occur, with 2500e50 being the most likely culprit,
even though, the code itself isn't something that would be even remotely hot.

I went and rewrote it in NQP (attached) to see if it drops the numbers at all, and... it didn't really.
I got a 120s stresstest and a 85s spectest.

I gave up with this.

I think we need some extensive bench without any random sleeps or delays in it and comprehensive enough to test more than
just a few text methods. Perhaps, something that can be cooked up with https://map.perl6.party/

And with such a bench in hand, it might become easier to hunt commits that slow stuff down considerably.<exception.diff>

Generally, we don’t care about performance once we have an unrecoverable exception that needs to be reported. If the stresstest regression is caused by trying to do Levenstein on method names (which can be a lot on some objects), then maybe the performance of spectest/stresstest can be accounted for by *just* the tests that throw Method Not Found errors?

The RT System itself - Status changed from 'new' to 'open'

From @zoffixznet

On Mon, 05 Jun 2017 13​:28​:00 -0700, elizabeth wrote​:

Generally, we don’t care about performance once we have an
unrecoverable exception that needs to be reported. If the stresstest
regression is caused by trying to do Levenstein on method names (which
can be a lot on some objects), then maybe the performance of
spectest/stresstest can be accounted for by *just* the tests that
throw Method Not Found errors?

Yeah, I think the nqp version of the exception is proof that's the wrong commit, otherwise
it'd've got faster than even pre-commit.

Something else was the cause, but I ran out of steam hunting it.

From @AlexDaniel

I'll take a look a bit later. It feels like the commit you mentioned is the right one, which makes me think that we can reject this ticket, but at that pace the spectest will get to turtle speeds even though all our changes will be justified.

Anyway, I'll bisect it first and then we'll have a look at what can be done to Make the Spectest Fast Again.

On 2017-06-05 14​:27​:28, cpan@​zoffix.com wrote​:

On Mon, 05 Jun 2017 13​:28​:00 -0700, elizabeth wrote​:

Generally, we don’t care about performance once we have an
unrecoverable exception that needs to be reported. If the stresstest
regression is caused by trying to do Levenstein on method names
(which
can be a lot on some objects), then maybe the performance of
spectest/stresstest can be accounted for by *just* the tests that
throw Method Not Found errors?

Yeah, I think the nqp version of the exception is proof that's the
wrong commit, otherwise
it'd've got faster than even pre-commit.

Something else was the cause, but I ran out of steam hunting it.

From @zoffixznet

So it *is* the commit originally found, but the Exception.message call was later optimized away to being called only when needed, and not when the exception is thrown, so this don't matter anymore.

Commit timing results​: https://irclog.perlgeek.de/perl6-dev/2017-10-14#i_15303228

@zoffixznet - Status changed from 'open' to 'resolved'