Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO.slurp throws exception when run in threads #6638

Open
p6rt opened this issue Nov 14, 2017 · 9 comments
Open

IO.slurp throws exception when run in threads #6638

p6rt opened this issue Nov 14, 2017 · 9 comments

Comments

@p6rt
Copy link

p6rt commented Nov 14, 2017

Migrated from rt.perl.org#132447 (status was 'open')

Searchable as RT132447$

@p6rt
Copy link
Author

p6rt commented Nov 14, 2017

From @lefth

The exception is​:
Cannot assign to an immutable value in method slurp at
SETTING​::src/core/IO/Handle.pm line 698
  in method slurp at SETTING​::src/core/IO/Path.pm line 603
  in block at ./concurrency-test.p6 line 40
  in block at SETTING​::src/core/Promise.pm line 217
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 284
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 173
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 166
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 163

I've observed this error on both Linux and Windows 10, on the September and
October Rakudo releases. It's intermittent, so I use a shell loop to run my
test case repeatedly until the problem occurs​:

while perl6 ./concurrency-test.p6 $DIR; do :; done
(where $DIR is any directory that contains several thousand files.)

This problem only occurs with slurp. When I instead open a file handle,
read its full contents with `.lines.cache`, then close the handle, there is
no exception.

The test case I used to show this issue is uploaded here​:
https://gist.github.com/lefth/6d71ca714ca2dc184220a91ceb41334d

@p6rt
Copy link
Author

p6rt commented Nov 15, 2017

From @zoffixznet

Hi,

Having trouble reproducing this on Linux on 2017.10-177-g77048b6 in a dir with 33k files that have one line of text each.

- Can you inlcude the exact version (perl6 -v) to go with the --ll-exception stack trace so we can examine the referenced core code? I tried following the line numbers in the trace you gave with 2017.09 and 2017.10 releases, but they don't seem to quite match up
- Does your directory also have other directories? Does the crash happen with just files in the directory?
- What's the typical size of the files in your directory?

This problem only occurs with slurp. When I instead open a file handle,
read its full contents with `.lines.cache`, then close the handle, there is
no exception.

Does it happen if you use the handle's .slurp method? i.e.

  open($file-path).slurp​: :close;

Let us know.

Thanks.

@p6rt
Copy link
Author

p6rt commented Nov 15, 2017

The RT System itself - Status changed from 'new' to 'open'

@p6rt
Copy link
Author

p6rt commented Nov 15, 2017

From @lefth

It seems this has already been resolved on the master branch. After making
a new build, I cannot reproduce the problem. Can you close the bug report?
Thanks!

On Wed, Nov 15, 2017 at 2​:59 AM perl6 via RT <perl6-bugs-followup@​perl.org>
wrote​:

Greetings,

This message has been automatically generated in response to the
creation of a trouble ticket regarding​:
"IO.slurp throws exception when run in threads",
a summary of which appears below.

There is no need to reply to this message right now. Your ticket has been
assigned an ID of [perl #​132447].

Please include the string​:

     \[perl #&#8203;132447\]

in the subject line of all future correspondence about this issue. To do
so,
you may reply to this message.

                    Thank you,
                    perl6\-bugs\-followup@&#8203;perl\.org

-------------------------------------------------------------------------
The exception is​:
Cannot assign to an immutable value in method slurp at
SETTING​::src/core/IO/Handle.pm line 698
in method slurp at SETTING​::src/core/IO/Path.pm line 603
in block at ./concurrency-test.p6 line 40
in block at SETTING​::src/core/Promise.pm line 217
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 284
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 173
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 166
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 163

I've observed this error on both Linux and Windows 10, on the September and
October Rakudo releases. It's intermittent, so I use a shell loop to run my
test case repeatedly until the problem occurs​:

while perl6 ./concurrency-test.p6 $DIR; do :; done
(where $DIR is any directory that contains several thousand files.)

This problem only occurs with slurp. When I instead open a file handle,
read its full contents with `.lines.cache`, then close the handle, there is
no exception.

The test case I used to show this issue is uploaded here​:
https://gist.github.com/lefth/6d71ca714ca2dc184220a91ceb41334d

@p6rt
Copy link
Author

p6rt commented Nov 15, 2017

From @AlexDaniel

Marked as 「 testneeded」 to make sure we have tests for it.

If there are tests already, then it should be linked and the ticket can be closed.

On 2017-11-15 14​:43​:51, dan@​zwell.net wrote​:

It seems this has already been resolved on the master branch. After
making
a new build, I cannot reproduce the problem. Can you close the bug
report?
Thanks!

On Wed, Nov 15, 2017 at 2​:59 AM perl6 via RT <perl6-bugs-
followup@​perl.org>
wrote​:

Greetings,

This message has been automatically generated in response to the
creation of a trouble ticket regarding​:
"IO.slurp throws exception when run in threads",
a summary of which appears below.

There is no need to reply to this message right now. Your ticket has
been
assigned an ID of [perl #​132447].

Please include the string​:

[perl #​132447]

in the subject line of all future correspondence about this issue. To
do
so,
you may reply to this message.

Thank you,
perl6-bugs-followup@​perl.org

-------------------------------------------------------------------------
The exception is​:
Cannot assign to an immutable value in method slurp at
SETTING​::src/core/IO/Handle.pm line 698
in method slurp at SETTING​::src/core/IO/Path.pm line 603
in block at ./concurrency-test.p6 line 40
in block at SETTING​::src/core/Promise.pm line 217
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 284
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 173
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 166
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 163

I've observed this error on both Linux and Windows 10, on the
September and
October Rakudo releases. It's intermittent, so I use a shell loop to
run my
test case repeatedly until the problem occurs​:

while perl6 ./concurrency-test.p6 $DIR; do :; done
(where $DIR is any directory that contains several thousand files.)

This problem only occurs with slurp. When I instead open a file
handle,
read its full contents with `.lines.cache`, then close the handle,
there is
no exception.

The test case I used to show this issue is uploaded here​:
https://gist.github.com/lefth/6d71ca714ca2dc184220a91ceb41334d

@p6rt
Copy link
Author

p6rt commented Nov 17, 2017

From @lefth

After more careful checking, I found the bug fix did make it into the
October release. A bisect showed it was fixed it in
commit 6af44f8d38a02bbd0d68cfd014165d6e33e4d89a. So, in the prior commit,
the --version 2017.09-490-g3f595acfb built on MoarVM version 2017.10, and
the exception (with --ll-exception) is​:

Got exception​: Cannot assign to an immutable value in method slurp at
SETTING​::src/core/IO/Handle.pm line 735
  in method slurp at SETTING​::src/core/IO/Path.pm line 609
  in block at ./concurrency-test.p6 line 40
  in block at SETTING​::src/core/Promise.pm line 241
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 659
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 224
  in method run-one at SETTING​::src/core/ThreadPoolScheduler.pm line 224
  in method dispatch​:<!> at SETTING​::src/core/Mu.pm line 806
  in block at SETTING​::src/core/ThreadPoolScheduler.pm line 258

Died
  at SETTING​::src/core/Exception.pm​:57
(D​:\cygwin64\home\p6\rakudo\install\share/perl6/runtime/CORE.setting.moarvm​:throw)
from SETTING​::src/core/control.pm​:170
(D​:\cygwin64\home\p6\rakudo\install\share/perl6/runtime/CORE.setting.moarvm​:die)
from SETTING​::src/core/control.pm​:166
(D​:\cygwin64\home\p6\rakudo\install\share/perl6/runtime/CORE.setting.moarvm​:die)
from ./concurrency-test.p6​:54 (<ephemeral file>​:MAIN)
from SETTING​::src/core/Main.pm​:198
(D​:\cygwin64\home\p6\rakudo\install\share/perl6/runtime/CORE.setting.moarvm​:MAIN_HELPER)
from ./concurrency-test.p6​:33 (<ephemeral file>​:<unit>)
from ./concurrency-test.p6​:1 (<ephemeral file>​:<unit-outer>)
from gen\moar\stage2\NQPHLL.nqp​:1542
(D​:\cygwin64\home\p6\rakudo\install\share\nqp\lib/NQPHLL.moarvm​:eval)
from gen\moar\stage2\NQPHLL.nqp​:1779
(D​:\cygwin64\home\p6\rakudo\install\share\nqp\lib/NQPHLL.moarvm​:evalfiles)
from gen\moar\stage2\NQPHLL.nqp​:1704
(D​:\cygwin64\home\p6\rakudo\install\share\nqp\lib/NQPHLL.moarvm​:command_eval)
from src/Perl6/Compiler.nqp​:42
(D​:\cygwin64\home\p6\rakudo\install\share\nqp\lib/Perl6/Compiler.moarvm​:command_eval)
from gen\moar\stage2\NQPHLL.nqp​:1630
(D​:\cygwin64\home\p6\rakudo\install\share\nqp\lib/NQPHLL.moarvm​:command_line)
from gen/moar/main.nqp​:47
(D​:\cygwin64\home\p6\rakudo\install\share\perl6\runtime\perl6.moarvm​:MAIN)
from gen/moar/main.nqp​:38
(D​:\cygwin64\home\p6\rakudo\install\share\perl6\runtime\perl6.moarvm​:<mainline>)
from <unknown>​:1
(D​:\cygwin64\home\p6\rakudo\install\share\perl6\runtime\perl6.moarvm​:<main>)
from <unknown>​:1
(D​:\cygwin64\home\p6\rakudo\install\share\perl6\runtime\perl6.moarvm​:<entry>)

I found I can reproduce the error within 1-4 iterations when running the
script on the Rakudo tree (built with --gen-moar and --gen-nqp and
installed without a prefix to ./install/).

Slurping on the file handle you described still produces the
exception​: open($file-path).slurp​: :close;
but slurping from a filehandle without :close, followed by $fh.close, is
fine.

On Thu, Nov 16, 2017 at 7​:58 AM Aleks-Daniel Jakimenko-Aleksejev via RT <
perl6-bugs-followup@​perl.org> wrote​:

Marked as 「testneeded」 to make sure we have tests for it.

If there are tests already, then it should be linked and the ticket can be
closed.

On 2017-11-15 14​:43​:51, dan@​zwell.net wrote​:

It seems this has already been resolved on the master branch. After
making
a new build, I cannot reproduce the problem. Can you close the bug
report?
Thanks!

On Wed, Nov 15, 2017 at 2​:59 AM perl6 via RT <perl6-bugs-
followup@​perl.org>
wrote​:

Greetings,

This message has been automatically generated in response to the
creation of a trouble ticket regarding​:
"IO.slurp throws exception when run in threads",
a summary of which appears below.

There is no need to reply to this message right now. Your ticket has
been
assigned an ID of [perl #​132447].

Please include the string​:

[perl #​132447]

in the subject line of all future correspondence about this issue. To
do
so,
you may reply to this message.

Thank you,
perl6-bugs-followup@​perl.org

-------------------------------------------------------------------------

The exception is​:
Cannot assign to an immutable value in method slurp at
SETTING​::src/core/IO/Handle.pm line 698
in method slurp at SETTING​::src/core/IO/Path.pm line 603
in block at ./concurrency-test.p6 line 40
in block at SETTING​::src/core/Promise.pm line 217
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 284
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 173
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 166
in block at SETTING​::src/core/ThreadPoolScheduler.pm line 163

I've observed this error on both Linux and Windows 10, on the
September and
October Rakudo releases. It's intermittent, so I use a shell loop to
run my
test case repeatedly until the problem occurs​:

while perl6 ./concurrency-test.p6 $DIR; do :; done
(where $DIR is any directory that contains several thousand files.)

This problem only occurs with slurp. When I instead open a file
handle,
read its full contents with `.lines.cache`, then close the handle,
there is
no exception.

The test case I used to show this issue is uploaded here​:
https://gist.github.com/lefth/6d71ca714ca2dc184220a91ceb41334d

@p6rt
Copy link
Author

p6rt commented Nov 17, 2017

From @zoffixznet

On Fri, 17 Nov 2017 09​:26​:10 -0800, dan@​zwell.net wrote​:

After more careful checking, I found the bug fix did make it into the
October release. A bisect showed it was fixed it in
commit 6af44f8d38a02bbd0d68cfd014165d6e33e4d89a.
[...]
Slurping on the file handle you described still produces the
exception​: open($file-path).slurp​: :close;
but slurping from a filehandle without :close, followed by $fh.close, is fine.

Great. The behaviour you describe matches the commit; without :close the buggy path is not taken.

I found I can reproduce the error within 1-4 iterations when running
the
script on the Rakudo tree (built with --gen-moar and --gen-nqp and
installed without a prefix to ./install/).

I tried writing the test to cover this bug, but out of dozens of things I tried (including trying your original script on rakudo's dir), I just can't reproduce the failure.

Does this crash on your computer when using commits before the fix, by any chance?

  given $*TMPDIR.add​: 'RT132447.test' { $^p.spurt​: ''; await (start $p.slurp) xx 40_000 };

@p6rt
Copy link
Author

p6rt commented Nov 18, 2017

From @lefth

Oh, and when the list of filenames is the simpler `"$*TMPDIR/RT132447.test"
xx 100`, the problem also appears, but it seemed to take many more
iterations to crash. That could be just chance, though.

On Sat, Nov 18, 2017 at 4​:01 PM Dan Zwell <dan@​zwell.net> wrote​:

Thanks for taking the time to look into this! I can't reproduce it with
that snippet, even if I make the file nonempty. But I can reproduce it with
the following two snippets. (I could not reproduce when I populated the
input file in the same script that does the await loop.)

perl6 -e '"$*TMPDIR/RT132447.test".IO.spurt​: "a" x 400_000;'
while perl6 --ll-exception -e 'my @​files = gather { for ^100 { take
"$*TMPDIR/RT132447.test" } }; await do for @​files { start .IO.slurp }'; do
echo iteration done; done

On Sat, Nov 18, 2017 at 3​:28 AM Zoffix Znet via RT <
perl6-bugs-followup@​perl.org> wrote​:

On Fri, 17 Nov 2017 09​:26​:10 -0800, dan@​zwell.net wrote​:

After more careful checking, I found the bug fix did make it into the
October release. A bisect showed it was fixed it in
commit 6af44f8d38a02bbd0d68cfd014165d6e33e4d89a.
[...]
Slurping on the file handle you described still produces the
exception​: open($file-path).slurp​: :close;
but slurping from a filehandle without :close, followed by $fh.close,
is fine.

Great. The behaviour you describe matches the commit; without :close the
buggy path is not taken.

I found I can reproduce the error within 1-4 iterations when running
the
script on the Rakudo tree (built with --gen-moar and --gen-nqp and
installed without a prefix to ./install/).

I tried writing the test to cover this bug, but out of dozens of things I
tried (including trying your original script on rakudo's dir), I just can't
reproduce the failure.

Does this crash on your computer when using commits before the fix, by
any chance?

given $\*TMPDIR\.add&#8203;: 'RT132447\.test' \{ $^p\.spurt&#8203;: ''; await \(start

$p.slurp) xx 40_000 };

@p6rt
Copy link
Author

p6rt commented Nov 18, 2017

From @lefth

Thanks for taking the time to look into this! I can't reproduce it with
that snippet, even if I make the file nonempty. But I can reproduce it with
the following two snippets. (I could not reproduce when I populated the
input file in the same script that does the await loop.)

perl6 -e '"$*TMPDIR/RT132447.test".IO.spurt​: "a" x 400_000;'
while perl6 --ll-exception -e 'my @​files = gather { for ^100 { take
"$*TMPDIR/RT132447.test" } }; await do for @​files { start .IO.slurp }'; do
echo iteration done; done

On Sat, Nov 18, 2017 at 3​:28 AM Zoffix Znet via RT <
perl6-bugs-followup@​perl.org> wrote​:

On Fri, 17 Nov 2017 09​:26​:10 -0800, dan@​zwell.net wrote​:

After more careful checking, I found the bug fix did make it into the
October release. A bisect showed it was fixed it in
commit 6af44f8d38a02bbd0d68cfd014165d6e33e4d89a.
[...]
Slurping on the file handle you described still produces the
exception​: open($file-path).slurp​: :close;
but slurping from a filehandle without :close, followed by $fh.close, is
fine.

Great. The behaviour you describe matches the commit; without :close the
buggy path is not taken.

I found I can reproduce the error within 1-4 iterations when running
the
script on the Rakudo tree (built with --gen-moar and --gen-nqp and
installed without a prefix to ./install/).

I tried writing the test to cover this bug, but out of dozens of things I
tried (including trying your original script on rakudo's dir), I just can't
reproduce the failure.

Does this crash on your computer when using commits before the fix, by any
chance?

given $\*TMPDIR\.add&#8203;: 'RT132447\.test' \{ $^p\.spurt&#8203;: ''; await \(start

$p.slurp) xx 40_000 };

@p6rt p6rt added the testneeded label Jan 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant