Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursively .emit-ing from the tap of the same supply bails out #6610

Open
p6rt opened this issue Oct 14, 2017 · 5 comments
Open

Recursively .emit-ing from the tap of the same supply bails out #6610

p6rt opened this issue Oct 14, 2017 · 5 comments

Comments

@p6rt
Copy link

p6rt commented Oct 14, 2017

Migrated from rt.perl.org#132292 (status was 'open')

Searchable as RT132292$

@p6rt
Copy link
Author

p6rt commented Oct 14, 2017

From @AlexDaniel

Code​:
my $s1 = Supplier.new; $s1.Supply.tap​: { say $_; $s1.emit(2) if $++ < 5; say "here" }; $s1.emit(1)

¦2017.06​:
1
2
2
2
2
2
here
here
here
here
here
here

¦HEAD(012c80f)​:
1
here
2

Possible IRC discussion​: https://irclog.perlgeek.de/perl6-dev/2017-09-21#i_15197905

The behavior change two times, first it started hanging after
(2017-09-18) rakudo/rakudo@4a8038c
And then the hang was resolved (incorrectly?) in
(2017-09-22) rakudo/rakudo@5478392

I think the output on 2017.06 makes more sense.

@p6rt
Copy link
Author

p6rt commented Oct 16, 2017

From @jnthn

On Fri, 13 Oct 2017 20​:43​:42 -0700, alex.jakimenko@​gmail.com wrote​:

Code​:
my $s1 = Supplier.new; $s1.Supply.tap​: { say $_; $s1.emit(2) if $++ <
5; say "here" }; $s1.emit(1)

¦2017.06​:
1
2
2
2
2
2
here
here
here
here
here
here

¦HEAD(012c80f)​:
1
here
2

Possible IRC discussion​: https://irclog.perlgeek.de/perl6-dev/2017-09-
21#i_15197905

The behavior change two times, first it started hanging after
(2017-09-18)
rakudo/rakudo@4a8038c
And then the hang was resolved (incorrectly?) in
(2017-09-22)
rakudo/rakudo@5478392

I think the output on 2017.06 makes more sense.

Actually the 2017.06 behavior was clearly wrong, because it violates the principle that a Supply chain will process a message at a time. That every "here" comes out at the end illustrates that the tap block was reentered. That was not an intended behavior, but rather an accident resulting through use of a reentrant mutex for some (not all) Supply concurrency management.

The commits in question and those around them introduced a unified concurrency model for all Supply operations, including `supply` blocks, based around Lock​::Async. The changes fixed many other issues, but also forced a revisit of the question of recursion - effectively, a supply sending a message to itself. This is a tricky problem, because there's some competing design goals around supplies​:

1. Serial message processing (as mentioned above)
2. Back-pressure​: those who emit into a Supply chain pay the cost of the message processing.
3. Fairness​: Messages are processed in the order they arrive.
4. No concurrency unless requested

I think 1 is pretty non-negotiable, because it's hard to write reliable concurrent code if you don't know what your "transaction scope" is. I really should have noticed the issue with reentrant mutexes sooner, though I guess that's my own usage biases to blame​: I very rarely use `.tap` and instead use supply/react/whenever where this issue could never happen (but - big issue - the old supply block mechanism violated goal 2 and arguably 3).

So the interesting question is how many we can have out of 2, 3, and 4. The current solution, on recursion, is to schedule the recursive message handling using the current $*SCHEDULER. This makes sure that we get 1 and 3. Unfortunately, it violates 2 (if we expect it to be transitively applied) and 4, and it's 4 that results in the effects reported here.

Note that if it's rewritten as​:

my $s1 = Supplier.new; react { whenever $s1 { say $_; $++ < 5 ?? $s1.emit(2) !! $s1.done; say "here" }; $s1.emit(1) }

Then it works fine​:

1
here
2
here
2
here
2
here
2
here
2
here

Also if you sleep a bit after the original​:

my $s1 = Supplier.new; $s1.Supply.tap​: { say $_; $s1.emit(2) if $++ < 5; say "here" }; $s1.emit(1); sleep 1

Then the output is the same as the above also (of course, the react block way is the correct one, not sleeping!)

So the question is if we can find a way to have 2 and 4, while retaining 1 and 3, and at what cost.

@p6rt
Copy link
Author

p6rt commented Oct 16, 2017

The RT System itself - Status changed from 'new' to 'open'

@p6rt
Copy link
Author

p6rt commented Oct 16, 2017

From @jnthn

On Mon, 16 Oct 2017 07​:42​:06 -0700, jnthn@​jnthn.net wrote​:

So the question is if we can find a way to have 2 and 4, while
retaining 1 and 3, and at what cost.

Also noting that in order to preserve 3, then in a situation like​:

* "Thread" 1 sends message A
* Handler for message A starts running
* "Thread" 2 sends message B
* Handler for message A emits a recursive message C
* Handler for message A completes

Then at this point, the next thing that needs to happen is for message B to be processed, for fairness. This means that "Thread" 1 needs to (non-blockingly, if in the pool) wait until message B has been processed, before it can do message C. If we can arrange for that to happen then I guess things would work out OK enough​: the sender of A will have to wait a while, but the recursive message send of C was "its fault" for sending A.

@p6rt
Copy link
Author

p6rt commented Oct 16, 2017

From @AlexDaniel

Right. Then I guess it's not a regression. Tag removed.

On 2017-10-16 07​:42​:06, jnthn@​jnthn.net wrote​:

On Fri, 13 Oct 2017 20​:43​:42 -0700, alex.jakimenko@​gmail.com wrote​:

Code​:
my $s1 = Supplier.new; $s1.Supply.tap​: { say $_; $s1.emit(2) if $++ <
5; say "here" }; $s1.emit(1)

¦2017.06​:
1
2
2
2
2
2
here
here
here
here
here
here

¦HEAD(012c80f)​:
1
here
2

Possible IRC discussion​: https://irclog.perlgeek.de/perl6-dev/2017-
09-
21#i_15197905

The behavior change two times, first it started hanging after
(2017-09-18)
rakudo/rakudo@4a8038c
And then the hang was resolved (incorrectly?) in
(2017-09-22)
rakudo/rakudo@5478392

I think the output on 2017.06 makes more sense.

Actually the 2017.06 behavior was clearly wrong, because it violates
the principle that a Supply chain will process a message at a time.
That every "here" comes out at the end illustrates that the tap block
was reentered. That was not an intended behavior, but rather an
accident resulting through use of a reentrant mutex for some (not all)
Supply concurrency management.

The commits in question and those around them introduced a unified
concurrency model for all Supply operations, including `supply`
blocks, based around Lock​::Async. The changes fixed many other issues,
but also forced a revisit of the question of recursion - effectively,
a supply sending a message to itself. This is a tricky problem,
because there's some competing design goals around supplies​:

1. Serial message processing (as mentioned above)
2. Back-pressure​: those who emit into a Supply chain pay the cost of
the message processing.
3. Fairness​: Messages are processed in the order they arrive.
4. No concurrency unless requested

I think 1 is pretty non-negotiable, because it's hard to write
reliable concurrent code if you don't know what your "transaction
scope" is. I really should have noticed the issue with reentrant
mutexes sooner, though I guess that's my own usage biases to blame​: I
very rarely use `.tap` and instead use supply/react/whenever where
this issue could never happen (but - big issue - the old supply block
mechanism violated goal 2 and arguably 3).

So the interesting question is how many we can have out of 2, 3, and
4. The current solution, on recursion, is to schedule the recursive
message handling using the current $*SCHEDULER. This makes sure that
we get 1 and 3. Unfortunately, it violates 2 (if we expect it to be
transitively applied) and 4, and it's 4 that results in the effects
reported here.

Note that if it's rewritten as​:

my $s1 = Supplier.new; react { whenever $s1 { say $_; $++ < 5 ??
$s1.emit(2) !! $s1.done; say "here" }; $s1.emit(1) }

Then it works fine​:

1
here
2
here
2
here
2
here
2
here
2
here

Also if you sleep a bit after the original​:

my $s1 = Supplier.new; $s1.Supply.tap​: { say $_; $s1.emit(2) if $++ <
5; say "here" }; $s1.emit(1); sleep 1

Then the output is the same as the above also (of course, the react
block way is the correct one, not sleeping!)

So the question is if we can find a way to have 2 and 4, while
retaining 1 and 3, and at what cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant