Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why are threads discouraged? #14691

Open
p5pRT opened this issue May 5, 2015 · 69 comments
Open

Why are threads discouraged? #14691

p5pRT opened this issue May 5, 2015 · 69 comments

Comments

@p5pRT
Copy link

p5pRT commented May 5, 2015

Migrated from rt.perl.org#125106 (status was 'open')

Searchable as RT125106$

@p5pRT
Copy link
Author

p5pRT commented May 5, 2015

From @ikegami

threads.pm says​:

The use of interpreter-based threads in perl is officially discouraged.

Why is that? Is it just because they're rather heavy? If so, why not just
say so and let the user decide whether he wants to use them or not? After
all, heavy threads aren't a problem if you reuse them (as in a worker
model).

@p5pRT
Copy link
Author

p5pRT commented May 5, 2015

From @ribasushi

On 05/05/2015 05​:23 PM, Eric Brine (via RT) wrote​:

# New Ticket Created by "Eric Brine"
# Please include the string​: [perl #125106]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106 >

threads.pm says​:

The use of interpreter-based threads in perl is officially discouraged.

Why is that? Is it just because they're rather heavy? If so, why not just
say so and let the user decide whether he wants to use them or not?

+1

@p5pRT
Copy link
Author

p5pRT commented May 5, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented May 5, 2015

From @leonerd

On Tue, 5 May 2015 08​:23​:15 -0700
"Eric Brine" (via RT) <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by "Eric Brine"
# Please include the string​: [perl #125106]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106 >

threads.pm says​:

The use of interpreter-based threads in perl is officially
discouraged.

Why is that? Is it just because they're rather heavy? If so, why not
just say so and let the user decide whether he wants to use them or
not? After all, heavy threads aren't a problem if you reuse them (as
in a worker model).

Or people could help me finish writing/editing the perlconcurrency doc,
so one day it will explain in more detail​:

  http​://www.leonerd.org.uk/code/perlconcurrency.pod

--
Paul "LeoNerd" Evans

leonerd@​leonerd.org.uk
http​://www.leonerd.org.uk/ | https://metacpan.org/author/PEVANS

@p5pRT
Copy link
Author

p5pRT commented May 5, 2015

From @rcaputo

On May 5, 2015, at 11​:23, Eric Brine <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by "Eric Brine"
# Please include the string​: [perl #125106]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106 >

threads.pm says​:

The use of interpreter-based threads in perl is officially discouraged.

Why is that? Is it just because they're rather heavy? If so, why not just
say so and let the user decide whether he wants to use them or not? After
all, heavy threads aren't a problem if you reuse them (as in a worker
model).

If your documentation is anything like mine, the paragraph before that statement tries to explain why. Perhaps the discouragement should begin with "Because of this, the use of...." to hint that the reader may have skipped something important and to discourage people from taking a provocative statement out of context.

The discouragement was added on Mar 2, 2014 after an intense deliberation on perl5-porters​:

"RFC​: add discouragement warning to perl threads documentation"
http​://www.gossamer-threads.com/lists/perl/porters/305035

The RFC was a product of an observation in irc.perl.org #p5p that most people in the Perl community, when asked for help with threads in Perl, will harshly denounce their viability and recommend that the problematic program be gutted and rewritten with almost anything but.

The #p5p IRC discussion began on Feb 19, 2014, with "Perl's threads are socially weird. They exist and are supported, but they (and their users) are heavily deprecated in the community."

Or it began with an observation about 90 minutes earlier in a #p5p discussion with/about a Perl novice​:

right, because most channels don't answers threads questions
because most channels are full of people who don't use threads.pm
because 'use threads;' almost always indicates brain damage

--
Rocco Caputo <rcaputo@​pobox.com>

@p5pRT
Copy link
Author

p5pRT commented May 10, 2016

From @ikegami

On Tue, May 5, 2015 at 1​:49 PM, Rocco Caputo <rcaputo@​pobox.com> wrote​:

On May 5, 2015, at 11​:23, Eric Brine <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by "Eric Brine"
# Please include the string​: [perl #125106]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106 >

threads.pm says​:

The use of interpreter-based threads in perl is officially discouraged.

Why is that? Is it just because they're rather heavy? If so, why not just
say so and let the user decide whether he wants to use them or not? After
all, heavy threads aren't a problem if you reuse them (as in a worker
model).

If your documentation is anything like mine, the paragraph before that
statement tries to explain why. Perhaps the discouragement should begin
with "Because of this, the use of...." to hint that the reader may have
skipped something important and to discourage people from taking a
provocative statement out of context.

One might think that, except that the word "discouraged" is linked to a
definition that means something quite different than "not to be used
without understanding what it means that Perl threads are heavy". It
actually defines "discouraged" as "we'd like to get rid of them, but we're
not doing it now"

  - *discouraged*

  From time to time, we may mark language constructs and features which we
  consider to have been mistakes as *discouraged*. Discouraged features
  aren't currently candidates for removal, but we may later deprecate them if
  they're found to stand in the way of a significant improvement to the Perl
  core.

@p5pRT
Copy link
Author

p5pRT commented May 11, 2016

From @xsawyerx

On 05/10/2016 10​:22 PM, Eric Brine wrote​:

On Tue, May 5, 2015 at 1​:49 PM, Rocco Caputo <rcaputo@​pobox.com
<mailto​:rcaputo@​pobox.com>>wrote​:

> On May 5\, 2015\, at 11&#8203;:23\, Eric Brine \<perlbug\-followup@&#8203;perl\.org
\<mailto&#8203;:perlbug\-followup@&#8203;perl\.org>> wrote&#8203;:
>
> \# New Ticket Created by  "Eric Brine"
> \# Please include the string&#8203;:  \[perl \#125106\]
> \# in the subject line of all future correspondence about this issue\.
> \# \<URL&#8203;: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106
\<https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106>>
>
>
> threads\.pm \<http&#8203;://threads\.pm>says&#8203;:
>
> The use of interpreter\-based threads in perl is officially
discouraged\.
>
>
> Why is that? Is it just because they're rather heavy? If so\, why
not just
> say so and let the user decide whether he wants to use them or
not? After
> all\, heavy threads aren't a problem if you reuse them \(as in a
worker
> model\)\.

If your documentation is anything like mine\, the paragraph before
that statement tries to explain why\.  Perhaps the discouragement
should begin with "Because of this\, the use of\.\.\.\." to hint that
the reader may have skipped something important and to discourage
people from taking a provocative statement out of context\.

One might think that, except that the word "discouraged" is linked to
a definition that means something quite different than "not to be used
without understanding what it means that Perl threads are heavy". It
actually defines "discouraged" as "we'd like to get rid of them, but
we're not doing it now"

* *discouraged*

From time to time\, we may mark language constructs and features
which we consider to have been mistakes as \*discouraged\*\.
Discouraged features aren't currently candidates for removal\, but
we may later deprecate them if they're found to stand in the way
of a significant improvement to the Perl core\.

I agree this is a confusing description.

I also like Rocco's comments on describing *why* something is the way it
is, even if briefly. I found an example in the thread Rocco linked[1],
by Aristotle, on how to explain things briefly and clearly​:

"complex data structures are hard to share properly"

The quote is out of context and does not constitute as the patch I would
envision, but it's an example of how to describe something complicated
succinctly. Perhaps we can form a patch describing what we mean by
"discouraged". Having context might excuse us from trying to find a
better term for this than "discouraged".

[1] Thanks, Rocco!

@p5pRT
Copy link
Author

p5pRT commented May 14, 2016

From @leonerd

On Wed, 11 May 2016 19​:54​:37 +0200
Sawyer X <xsawyerx@​gmail.com> wrote​:

I agree this is a confusing description.

I also like Rocco's comments on describing *why* something is the way
it is, even if briefly. I found an example in the thread Rocco
linked[1], by Aristotle, on how to explain things briefly and clearly​:

"complex data structures are hard to share properly"

The quote is out of context and does not constitute as the patch I
would envision, but it's an example of how to describe something
complicated succinctly. Perhaps we can form a patch describing what
we mean by "discouraged". Having context might excuse us from trying
to find a better term for this than "discouraged".

[1] Thanks, Rocco!

If we have some better wording I could put it into perlconcurrency.pod
as well...

--
Paul "LeoNerd" Evans

leonerd@​leonerd.org.uk
http​://www.leonerd.org.uk/ | https://metacpan.org/author/PEVANS

@p5pRT
Copy link
Author

p5pRT commented Mar 24, 2018

From choroba@matfyz.cz

The documentation of threads.pm says that perl threads are
"discouraged". This causes lots of confusion, as the term is defined
in perlpolicy in the following way​:

  discouraged
  From time to time, we may mark language constructs and
  features which we consider to have been mistakes as
  discouraged. Discouraged features aren't currently candidates
  for removal, but we may later deprecate them if they're found
  to stand in the way of a significant improvement to the Perl
  core.

See also #125106, which seems to lead nowhere, maybe because it was
too ambitious. This patch provides just the minimal change.

See also http​://www.perlmonks.org/?node_id=1107534 .

Perl Info

Flags:
     category=library
     severity=medium
     module=threads

Site configuration information for perl 5.27.11:

Configured by choroba at Sat Mar 24 15:15:34 CET 2018.

Summary of my perl5 (revision 5 version 27 subversion 11) configuration:
   Commit id: e77195360c1a2b081a806df017630b3165e2aeac
   Platform:
     osname=linux
     osvers=4.4.120-45-default
     archname=x86_64-linux-thread-multi
     uname='linux still 4.4.120-45-default #1 smp wed mar 14 20:51:49 utc 2018 (623211f) x86_64 x86_64 x86_64 gnulinux '
     config_args='-Dusedevel -de -Dusethreads -Dprefix=/home/choroba/localperl'
     hint=recommended
     useposix=true
     d_sigaction=define
     useithreads=define
     usemultiplicity=define
     use64bitint=define
     use64bitall=define
     uselongdouble=undef
     usemymalloc=n
     default_inc_excludes_dot=define
     bincompat5005=undef
   Compiler:
     cc='cc'
     ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
     optimize='-O2'
     cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
     ccversion=''
     gccversion='4.8.5'
     gccosandvers=''
     intsize=4
     longsize=8
     ptrsize=8
     doublesize=8
     byteorder=12345678
     doublekind=3
     d_longlong=define
     longlongsize=8
     d_longdbl=define
     longdblsize=16
     longdblkind=3
     ivtype='long'
     ivsize=8
     nvtype='double'
     nvsize=8
     Off_t='off_t'
     lseeksize=8
     alignbytes=8
     prototype=define
   Linker and Libraries:
     ld='cc'
     ldflags =' -fstack-protector -L/usr/local/lib'
     libpth=/usr/local/lib /usr/lib64/gcc/x86_64-suse-linux/4.8/include-fixed /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
     libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
     perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
     libc=libc-2.22.so
     so=so
     useshrplib=false
     libperl=libperl.a
     gnulibc_version='2.22'
   Dynamic Linking:
     dlsrc=dl_dlopen.xs
     dlext=so
     d_dlsymun=undef
     ccdlflags='-Wl,-E'
     cccdlflags='-fPIC'
     lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'



@INC for perl 5.27.11:
     /home/choroba/localperl/lib/site_perl/5.27.11/x86_64-linux-thread-multi
     /home/choroba/localperl/lib/site_perl/5.27.11
     /home/choroba/localperl/lib/5.27.11/x86_64-linux-thread-multi
     /home/choroba/localperl/lib/5.27.11


Environment for perl 5.27.11:
     HOME=/home/choroba
     LANG=en_US.UTF-8
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=/home/choroba/perl5/bin:/home/choroba/bin:/bin:/sbin:/usr/bin:/usr/local/bin:/usr/games:/usr/X11R6/bin:/opt/gnome/bin:.
     PERL_BADLANG (unset)
     SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Mar 24, 2018

From choroba@matfyz.cz

0001-Remove-the-word-discouraged-from-threads.patch
From deedc55618b1f84d03e39a88fc89cf4dfb01744d Mon Sep 17 00:00:00 2001
From: "E. Choroba" <choroba@still.ms.mff.cuni.cz>
Date: Sat, 24 Mar 2018 16:46:50 +0100
Subject: [PATCH] Remove the word "discouraged" from threads

Its meaning is defined in perlpolicy and means something else. See
also #125106.
---
 dist/threads/lib/threads.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/dist/threads/lib/threads.pm b/dist/threads/lib/threads.pm
index 1b99567ef2..5e5016e3d7 100644
--- a/dist/threads/lib/threads.pm
+++ b/dist/threads/lib/threads.pm
@@ -143,8 +143,8 @@ system for multitasking that one might expect or hope for.  Threads are
 implemented in a way that make them easy to misuse.  Few people know how to
 use them correctly or will be able to provide help.
 
-The use of interpreter-based threads in perl is officially
-L<discouraged|perlpolicy/discouraged>.
+Therefore, the use of interpreter-based threads in perl is officially
+not recommended.
 
 =head1 SYNOPSIS
 
-- 
2.13.6

@p5pRT
Copy link
Author

p5pRT commented Mar 25, 2018

From @Grinnz

On Sat, Mar 24, 2018 at 12​:31 PM, E. Choroba <perlbug-followup@​perl.org>
wrote​:

# New Ticket Created by "E. Choroba"
# Please include the string​: [perl #133021]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133021 >

-----------------------------------------------------------------
[Please describe your issue here]

The documentation of threads.pm says that perl threads are
"discouraged". This causes lots of confusion, as the term is defined
in perlpolicy in the following way​:

discouraged
From time to time, we may mark language constructs and
features which we consider to have been mistakes as
discouraged. Discouraged features aren't currently candidates
for removal, but we may later deprecate them if they're found
to stand in the way of a significant improvement to the Perl
core.

See also #125106, which seems to lead nowhere, maybe because it was
too ambitious. This patch provides just the minimal change.

See also http​://www.perlmonks.org/?node_id=1107534 .

'not recommended' seems too lenient. 'strongly recommended against'?

-Dan

@p5pRT
Copy link
Author

p5pRT commented Mar 25, 2018

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 25, 2018

From @dur-randir

On Sat, 24 Mar 2018 09​:31​:06 -0700, choroba@​matfyz.cz wrote​:

 From time to time\, we may mark language constructs and
 features which we consider to have been mistakes as
 discouraged\. 

Threads in their current implementation are arguable a mistake. You loose ~30% of performance and who-measured-how-much memory by just using perl build with threads. And most of XS CPAN is up to some degree threads-unsafe (pure-perl CPAN is safe, so the goals for threads is reached, but still).

Discouraged features aren't currently candidates
 for removal\, but we may later deprecate them if they're found
 to stand in the way of a significant improvement to the Perl
 core\.

Threads are not going anywhere by this definition. Just discouraged.

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2018

From @iabyn

On Sun, Mar 25, 2018 at 04​:20​:13PM -0700, Sergey Aleynikov via RT wrote​:

Threads in their current implementation are arguable a mistake. You
loose ~30% of performance and who-measured-how-much memory by just using
perl build with threads.

The test suite on blead on my system runs 17% slower under ithreads, but
that includes running a bunch of ithreads-only tests too. So I think 30%
is overstating it.

I'm not aware that a normal single-threaded process uses significantly
more memory under ithreads.

--
Hofstadter's Law​: It always takes longer than you expect, even when you
take into account Hofstadter's Law.

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2018

From @wchristian

Since this change doesn't have a ticket attached to it, i figured it would be useful to provide the context of what led to its creation.

This discouragement was introduced by rjbs in commit 10a4597. I've extracted some of the chat log that led to it.

https://gist.github.com/wchristian/97525932a89d0785e07371c60d84209c

And here's the p5p mailing list thread that followed afterwards​:

https://www.nntp.perl.org/group/perl.perl5.porters/2014/03/msg213382.html

It's a lot of material, but i hope it'll help understand the context here.

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2018

From @dur-randir

On Mon, 26 Mar 2018 01​:36​:54 -0700, davem wrote​:

The test suite on blead on my system runs 17% slower under ithreads, but
that includes running a bunch of ithreads-only tests too. So I think 30%
is overstating it.

It depends on the operation type. I've measured hash accesses and got ~10% loss on threaded build for 5.26.1.

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2018

From @iabyn

On Mon, Mar 26, 2018 at 04​:52​:54AM -0700, Sergey Aleynikov via RT wrote​:

On Mon, 26 Mar 2018 01​:36​:54 -0700, davem wrote​:

The test suite on blead on my system runs 17% slower under ithreads, but
that includes running a bunch of ithreads-only tests too. So I think 30%
is overstating it.

It depends on the operation type. I've measured hash accesses and got
~10% loss on threaded build for 5.26.1.

10% isn't 30%.

--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From @jdhedden

As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.

I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature.

Just my two cents.

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From [Unknown Contact. See original ticket]

As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.

I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature.

Just my two cents.

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From @khwilliamson

On 03/26/2018 06​:54 PM, Jerry D. Hedden via RT wrote​:

As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.

I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature.

Just my two cents.

I agree with the above. And I really don't like 'not recommended' as I
think it is too strong. Maybe just list the problems

Even a 30% slowdown will be fully acceptable if you can divide the work
up into 8 or 16 parallel pieces. The gain far outweighs the cost.

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From @Grinnz

On Mon, Mar 26, 2018 at 8​:54 PM, Jerry D. Hedden via RT <
perlbug-comment@​perl.org> wrote​:

As one of several people who have maintained the various threads modules
over the years, I regret that I missed being part of the original
discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do
not have any bugs preventing their use. I acknowledge that not all Perl
modules are thread-safe, but there is sufficient documentation to that
affect in the POD.

I also acknowledge that the threads implementation is not ideal nor
optimal. Nonetheless, threads are useful, and are being used in the wild.
(I, for one, have even used them to good effect in production code.
<Gasp!>) Yes, if you don't know what you're doing, threads can be
problematic. ("There be dragons...," and all that.) However, the same can
be argued to greater or lesser degrees of any programming language feature
in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the
WARNING heading is alarmist. I feel strongly that the heading should be
changed to NOTICE, and the word 'discouraged' be changed to 'not
recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several
emails from professional Perl developers from around the world expressing
concern about it. I expressed to them the same opinions I have given
above, namely that threads work (but you have to know what you're doing),
and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based
threads". Don't use them as you so choose. After all, Perl has always
been about options. However, there ARE Perl developers that do feel they
are a VERY useful feature.

Just my two cents.

While all of this is true, the other audience for this documentation
section is the new Perl user who thinks threads sound like a nice
lightweight way to spread work around without breaking all of their code
(because most of the rest of Perl works this way, you can just throw things
together and they all work). These users do not read far enough to realize
that threads are not the optimal solution to their requirements without a
strong warning. In my opinion, severe discouragement (if not the perlpolicy
defined term) is warranted; if someone can achieve this while still making
it clear that threads are not going to be deprecated for valid and optimal
use cases for them, this would be ideal.

-Dan

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From @iabyn

On Mon, Mar 26, 2018 at 07​:20​:29PM -0600, Karl Williamson wrote​:

On 03/26/2018 06​:54 PM, Jerry D. Hedden via RT wrote​:

As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.

I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature.

Just my two cents.

+1

I agree with the above. And I really don't like 'not recommended' as I
think it is too strong. Maybe just list the problems

+1

--
The Enterprise's efficient long-range scanners detect a temporal vortex
distortion in good time, allowing it to be safely avoided via a minor
course correction.
  -- Things That Never Happen in "Star Trek" #21

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From @rcaputo

On Mar 27, 2018, at 07​:15, Dave Mitchell <davem@​iabyn.com> wrote​:

On Mon, Mar 26, 2018 at 07​:20​:29PM -0600, Karl Williamson wrote​:

On 03/26/2018 06​:54 PM, Jerry D. Hedden via RT wrote​:

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

I started the discussion that led to the addition of the discouragement warning. Its original purpose was to manage users' expectations of the kind of help they'd get from the larger Perl community for trying to use this feature.

Toning down the notice doesn't break its original purpose.

I agree with the above. And I really don't like 'not recommended' as I
think it is too strong. Maybe just list the problems

Getting rid of the notice entirely would resume failing to manage users' expectations.

In case someone on-list isn't aware of the advice given off-list, I'm closing with a small list of quotes from freenode #perl. These are pieces of real advice given to Perl users looking for help using threads. Some of you may recognize things you've said.

Some of you may be tempted to say that IRC isn't representative of a larger perl community, but many of the vocal opponents on IRC also participate in other areas. They use and advocate Perl professionally. They attend or speak at Perl conferences. They are perl5 porters.

This list includes both old and recent statements​:

don't use threads

When someone says "I need to use threads" in perl, they (99.9% of the time) mean "I don't know what I'm talking about, please (re-)educate me"

But at least you didn't start out trying to use threads.

Yes.. Don't use threads. At all.

The simple answer is "don't use threads." Just fork if you must.

no, because threads are usually a stupid answer.

do not use threads.

'use threads' is "please let me use the windows fork hack code"

also, don't use threads in Perl; if your problem requires threads, don't use Perl

you seem to have missed the point earlier that was made about "dont use threads"

the thread lecture is don't use threads in Perl, or don't use Perl

before anything else, notice that people here will start telling you about how you probably shouldn't use threads in Perl.

Hi. Please don't use threads.

I wonder how many times we've told qubit not to use threads and how mayn times they've ignored us.

Because threads are not what you want. Threads in Perl, doubly so

(Almost) Everyone who wants to use threads in perl understands them incorrectly.

very few professional perl devs will use threads for real code

because threads are a stupid way to do concurrency in perl.

I really wish there was this little genie like the little green dude in The Flintstones that would pop up and educate people when they think they want to use threads in perl.

Don't use threads, would be our suggestion

If you want to wrap your head in crazy concurrency bugs and random failures, then use threads.

Suggestion​: don't use threads

don't use threads in Perl

perlbot​: threads is Friends don't let friends use threads ; Elizabeth on Threads - http​://www.perlmonks.org/index.pl?node_id=288022

basically​: stop trying to use threads.

and really you don't want to use threads at all

or you can use threads, and we will ignore you when it all breaks.

step 1​: don't use threads

The response is don't use threads

never use threads in perl

So then they're always asking "how can I do threads" and we say "don't use threads" and they complain and say "waaaaa I need them for doing lots of things at once" and then we say "no you dno't, you can do other things".. and they give a look of complete utter confusion.

yes but in your case it is because threads are known to be broken and so will cause you problems

First convince your department not to use threads.

I don't imagine you'll get much help.. Our advice is "don't use threads"... hardly anyone here does much if anything with threads.. So likely nobody here really knows

This may be the definitive idea of why allowing people to use threads is a bad idea.

you should not use threads.

don't use threads. problem solved.

don't use threads is the answer

You don't want to use threads.

ah, the truth "I use threads because I'm too dumb to understand select"

But if you're wanting to use threads, odds are you are doing something wrong :)

the answer is invariably "don't use threads" :)

ah, well, your first mistake was to actually seriously use threads in Perl

look, we can't actually advise you on a concurrency solution unless you tell us what the program's going to do - although the odds are pretty good that we'll say "don't use threads" since that's almost always the worst choice

code containing 'use Switch' or 'use threads' can be assumed to be broken

the best advice I can give is to not use threads in Perl

some idiot did 'use threads;' and now you have a broken program that needs replacing

DON'T USE THREADS, DON'T TELL OTHERS TO USE THREADS.

also, #perl will almost always refuse to help with threads. because threads in perl are shit.

--
Rocco Caputo <rcaputo@​pobox.com>

@p5pRT
Copy link
Author

p5pRT commented Mar 27, 2018

From @lizmat

Perhaps a reference to the “forks” module on CPAN could be added?

  https://metacpan.org/pod/forks

Liz

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2018

From @Leont

On Tue, Mar 27, 2018 at 3​:20 AM, Karl Williamson
<public@​khwilliamson.com> wrote​:

I agree with the above. And I really don't like 'not recommended' as I
think it is too strong. Maybe just list the problems

I think the main problem with the current wording is that it
discourages without explaining why. In particular it doesn't explain
that threads.pm does something different than what many people expect
it to do.

People expect it to be good for "share all the data" scenarios
(because pretty much anything else called threads is), even though it
is terrible at that. This confusion isn't helped by threads
implementations in commonly used implementations of languages
occupying the same niches as we do (e.g. cpython, Ruby MRI) sucking in
exactly the opposite way (they're good at sharing but useless at
actually being parallel).

It's not so much threads.pm that needs a big fat warnings, it's
threads​::shared that does.

Leon

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2018

From @dur-randir

On Mon, 26 Mar 2018 06​:06​:47 -0700, davem wrote​:

10% isn't 30%.

My idea was to show that you can't attribute all the test run slowdown to additional threading tests. Probably, I should've made this clearer in that message. As for the total slowdown figures, it depends on the application, but it's still "free" - you get it even if you don't start any threads.

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2018

From @dur-randir

On Mon, 26 Mar 2018 18​:21​:02 -0700, public@​khwilliamson.com wrote​:

I agree with the above. And I really don't like 'not recommended' as
I
think it is too strong. Maybe just list the problems

Even a 30% slowdown will be fully acceptable if you can divide the
work
up into 8 or 16 parallel pieces. The gain far outweighs the cost.

But if you use processes for parallelism and not threads, you don't get this slowdown at all and get the same multicore boost. And you don't have to guess which XS module will segfault next. The only case when I find using threads reasonable is writing GUI, where you _have to_ be in the same address space.

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2018

From @ppisar

On 2018-03-27, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

Perhaps a reference to the “forks” module on CPAN could be added?

Which does not work either
<https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of
thread-safety fixes in File​::Glob).

-- Petr

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2018

From @lizmat

On 28 Mar 2018, at 12​:36, Petr Pisar <ppisar@​redhat.com> wrote​:
On 2018-03-27, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

Perhaps a reference to the “forks” module on CPAN could be added?
Which does not work either
<https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of
thread-safety fixes in File​::Glob).

Way back when in 2002 / 2003, I had some free time on my hands and decided to see how I could use that to the advancement of Perl. The 5.8.0 release was still in the works, and it had some fine new features. One of them being ithreads.

Having always been very interested in executing stuff asynchronously / in parallel, I decided to cut my teeth into it, coming from having been exposed to pthreads in C.

This resulted in quite a number of Thread​:: modules on CPAN [1]. When I finally found out *how* Perl ithreads were implemented, I was *very* disappointed.

Having become considered a bit of a “ithreads” wizard on PerlMonks at the time, I basically explained the situation *almost 15 years ago* [2]

Apart from bug fixes, I don’t think the situation since then has changed significantly.

So, back to the documentation issue. Perhaps the “Performance considerations” paragraph from that post could serve as inspiration.

Liz

[1] https://metacpan.org/author/ELIZABETH
[2] http​://www.perlmonks.org/?node_id=288022

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2018

From choroba@matfyz.cz

Talking about PerlMonks, try reading some of the posts from BrowserUk, the current threads "wizard" there.

http​://www.perlmonks.org/?node_id=3989;BIT=threads;a=BrowserUk

On Wed, 28 Mar 2018 04​:04​:44 -0700, elizabeth wrote​:

On 28 Mar 2018, at 12​:36, Petr Pisar <ppisar@​redhat.com> wrote​:
On 2018-03-27, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

Perhaps a reference to the “forks” module on CPAN could be added?
Which does not work either
<https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of
thread-safety fixes in File​::Glob).

Way back when in 2002 / 2003, I had some free time on my hands and
decided to see how I could use that to the advancement of Perl. The
5.8.0 release was still in the works, and it had some fine new
features. One of them being ithreads.

Having always been very interested in executing stuff asynchronously /
in parallel, I decided to cut my teeth into it, coming from having
been exposed to pthreads in C.

This resulted in quite a number of Thread​:: modules on CPAN [1]. When
I finally found out *how* Perl ithreads were implemented, I was *very*
disappointed.

Having become considered a bit of a “ithreads” wizard on PerlMonks at
the time, I basically explained the situation *almost 15 years ago*
[2]

Apart from bug fixes, I don’t think the situation since then has
changed significantly.

So, back to the documentation issue. Perhaps the “Performance
considerations” paragraph from that post could serve as inspiration.

Liz

[1] https://metacpan.org/author/ELIZABETH
[2] http​://www.perlmonks.org/?node_id=288022

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2018

From @lizmat

On 9 Apr 2018, at 19​:00, Dan Book <grinnz@​gmail.com> wrote​:
On Mon, Apr 9, 2018 at 12​:05 PM, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

My point was about Perl code running in separate OS-threads, what I understand (perhaps
wrongly) with things like Future. So inside of a single interpreter.
Future has nothing to do with threading of any sort (though it could be leveraged in combination, such as what IO​::Async​::Function does on Windows). It's a mechanism for cooperative multitasking. I'm not sure what OS threads you are talking about that would run "inside of a single interpreter." pthreads run outside the interpreter.

Ok, so if I understand you correctly​: if you run some pthreads outside of a Perl interpreter thread, you will never be able to run any Perl code in that pthreads thread that could potentially access variables from a Perl interpreter running in another thread?

Liz

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2018

From @cpansprout

On Mon, 09 Apr 2018 08​:02​:10 -0700, davem wrote​:

(I don't know what OOC stands for).

Probably ‘out of curiosity’. Or maybe it’s a Dutch word.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Apr 9, 2018

From @wchristian

On Mon, 09 Apr 2018 12​:37​:54 +0200, Dave Mitchell <davem@​iabyn.com> wrote​:

Is this based on the idea that the ithreads implementation itself is
inherently buggy, or that some modules may not be thread-safe.

My current assumption is the latter. However i cannot speculate at all as to whether it might be some of column A or not.

2) If one's threaded code randomly craps out due to a 3rd party module
being non-thread-safe, or supposedly thread-safe but buggy, then that's an
issue with the third-party module. In principle any programming language
will struggle with threads if used with a 3rd-party library that isn't
thread-safe.

This has dangerous assumptions in it​:

- Thread safety is not a binary on/off switch. Modules can appear to be thread-safe under some conditions and not be so under others. That's why i called it a heisenbug in my previous email.

- Putting the blame on 3rd party modules is fine for most languages where it's easier to switch to other implementations or roll your own. However this is Perl. The language lives on CPAN, not in the interpreter. Asking someone who just spent hours implementing a parallel web downloader to gradually replace any of the bits they used will not be taken well.

- Very few cpan maintainers know how to do thread-safety. Heck, most don't even practice string/binary safety for filehandles. Most of them are Perl developers first and you're asking them to move at least 2 degrees of separation outside of their zone of comfort. For a lot of modules there's simply a good chance this is not going to happen.

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Mon, Apr 09, 2018 at 07​:26​:30PM +0200, Elizabeth Mattijsen wrote​:

Ok, so if I understand you correctly​: if you run some pthreads outside
of a Perl interpreter thread, you will never be able to run any Perl
code in that pthreads thread that could potentially access variables
from a Perl interpreter running in another thread?

I'm not sure I understand that sentence, but the basic rule of doing any
sort of threading in perl is that two OS-level threads must not
simultaneously access/modify the same perl interpreter struct (and all the
data hanging off it, e.g. SVs etc).

This was why the 5005THREADS model was abandoned. Nobody could think of a
way of avoiding all the side-effects that any trivial usage of an SV might
give rise to (e.g. changing the ref count, converting from an IV to PVIV),
without having a lock in about every second line of code in the perl
core.

I vaguely understand that the approach in Python is to have one big
central lock so only one thread can execute at any one time.

--
"I used to be with it, but then they changed what ‘it’ was, and now what
I’m with isn’t it. And what’s ‘it’ seems weird and scary to me."
  -- Grandpa Simpson
(It will happen to you too.)

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Mon, Apr 09, 2018 at 11​:01​:22PM +0200, Christian Walde wrote​:

2) If one's threaded code randomly craps out due to a 3rd party module
being non-thread-safe, or supposedly thread-safe but buggy, then that's an
issue with the third-party module. In principle any programming language
will struggle with threads if used with a 3rd-party library that isn't
thread-safe.

This has dangerous assumptions in it​:

- Thread safety is not a binary on/off switch. Modules can appear to be
thread-safe under some conditions and not be so under others. That's why
i called it a heisenbug in my previous email.

- Putting the blame on 3rd party modules is fine for most languages
where it's easier to switch to other implementations or roll your own.
However this is Perl. The language lives on CPAN, not in the
interpreter. Asking someone who just spent hours implementing a parallel
web downloader to gradually replace any of the bits they used will not
be taken well.

- Very few cpan maintainers know how to do thread-safety. Heck, most
don't even practice string/binary safety for filehandles. Most of them
are Perl developers first and you're asking them to move at least 2
degrees of separation outside of their zone of comfort. For a lot of
modules there's simply a good chance this is not going to happen.

But everything you've said above applies just as much to any 3rd-party
library/module in any programming language which supports concurrency.
I really fail to see how CPAN is different in this regard.

Most CPAN authors don't need to worry about thread-safety, as long as it's
a perl-level module. Perl is thread-safe at the language level, and you
don't need to do anything special to make Perl code thread-safe. Or at
least, nothing more than you would have to do to make your module safe
across forks (which is what ithreads is emulating after all). For example,
if a thread is created, the module's variables will be cloned, and a
destructor might be called for a variable in each of the threads. And of
course if the module is updating something external like a file, then it
may need to exclusively lock that file.

XS code is of course a lot harder, because now you're doing concurrency
programming in C. But even here, its easier than general C concurrency,
because of perl's ITHREADS "not shared by default" model. You mainly have
to avoid global/static mutable data, and be prepared for any of your
data stored in SVs (e.g. DB handles) being cloned when a new thread is
created.

--
All wight. I will give you one more chance. This time, I want to hear
no Wubens. No Weginalds. No Wudolf the wed-nosed weindeers.
  -- Life of Brian

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From tom@binary.com

On 10 April 2018 at 05​:01, Christian Walde <walde.christian@​gmail.com>
wrote​:

On Mon, 09 Apr 2018 12​:37​:54 +0200, Dave Mitchell <davem@​iabyn.com> wrote​:

Is this based on the idea that the ithreads implementation itself is

inherently buggy, or that some modules may not be thread-safe.

My current assumption is the latter. However i cannot speculate at all as
to whether it might be some of column A or not.

The official threads.pm documentation claims that the current code has what
I'd class as "bugs"​:

  Even with the latest version of Perl, it is known that certain
  constructs with threads may result in warning messages concerning
  leaked scalars or unreferenced scalars. However, such warnings are
  harmless, and may safely be ignored.

If they are leaking, this has bad implications for long-running code.
If they're not leaking, why the warning?
If this information is out of date, then the bugs are in the documentation!

2) If one's threaded code randomly craps out due to a 3rd party module

being non-thread-safe, or supposedly thread-safe but buggy, then that's an
issue with the third-party module. In principle any programming language
will struggle with threads if used with a 3rd-party library that isn't
thread-safe.

This has dangerous assumptions in it​:

- Thread safety is not a binary on/off switch. Modules can appear to be
thread-safe under some conditions and not be so under others. That's why i
called it a heisenbug in my previous email.

Indeed, this is arguably worse than clear "the code crashed so it doesn't
work" cases.

We also don't have good documentation on how to make modules thread-safe​:
I've encountered quite a few Perl developers who are confident that thread
safety is only an issue with XS.

As an example​: far as I recall, the threads.pm documention and perlmod page
to which they link never explicitly state that ref addresses will be
different in each thread​:

  perl -e'use threads; use Scalar​::Util qw(refaddr); my %x; $x{refaddr
$_} = $_ for [qw(x)], [qw(y)]; threads->create(sub { warn threads->tid . "
=> " . join ", ", map { "$_ (" . refaddr($x{$_}) . ")" } sort keys %x
})->join for 1..2'
  1 => 30822192 (32355200), 30948440 (32355272) at -e line 1.
  2 => 30822192 (32355456), 30948440 (32355528) at -e line 1.

I know much of my CPAN code will be susceptible to this type of issue. Any
time someone asks about thread safety, I point to the "discouraged" line in
the official documentation and explain that I don't expect my code to work
when multiple threads are active - "maybe try again if we have a new
threads implementation in the future".

Downgrading that warning to something less clear could have unfortunate
consequences for people learning (or supporting those who learn) Perl. I'm
also not entirely convinced that "discouraged" is the wrong word - taking
the phrases from the official policy​:

From time to time, we may mark language constructs and features which we

consider to have been mistakes as discouraged.

- Windows support​: not a mistake. Exposing the API to end users without
warnings and clear information about how threads differs from other
languages, on the other hand - maybe not ideal.

Discouraged features aren't currently candidates for removal, but we may

later deprecate them if they're found to stand in the way of a significant
improvement to the Perl core.

- surely this is the case? If we come up with a better way to implement
threads while retaining the ability for Windows to support some form of
process emulation and related features, wouldn't this be something that's
welcomed? I don't get the impression that there's widespread satisfaction
with the current state of affairs on either threads.pm or threads​::shared.

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @wchristian

On Tue, 10 Apr 2018 13​:03​:45 +0200, Dave Mitchell <davem@​iabyn.com> wrote​:

I really fail to see how CPAN is different in this regard.

Because this is Perl where not using CPAN is not an option and CPAN library consumers rely on CPAN libraries being almost entirely rock-solid, or at least the author having easy and quick ways to fix bugs.

If you really look at all 3 points i made, in aggregate, and don't see how this is a problem and a danger, then i don't think i can come up with other word combinations to make you see it.

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @dur-randir

On Tue, 10 Apr 2018 04​:04​:06 -0700, davem wrote​:

The problem is in people's expectations. When most part of something is thread-safe, they'll just assume "everything is thread-safe" and then'll be hit very hard. One of the examples I like is Mouse, which was thread-unsafe until the year 2015. While it's not as popular as Moose, it's still a module with large a large user base. And your chances to encounter bugs in not so popular XS modules are much, much higher. What's even worse, you're likely to hit them only under some unlucky circumstances (see example below).

As for the core, I agree that it's now much more thread-stable than before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no threads" with some nice crash examples in it - they crash no more. But still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just because of no one has discovered that. And the following still fails loudly (though not dumps a core);

while (1) {
  push @​foo, threads->create(sub {
  require IO​::Handle;
  });
  $_->detach for(splice @​foo);
}

So while yes, perl is much more thread-safe inherently as it used to be, I won't recommend using threads in it to anyone.

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Tue, Apr 10, 2018 at 02​:59​:50PM +0200, Christian Walde wrote​:

On Tue, 10 Apr 2018 13​:03​:45 +0200, Dave Mitchell <davem@​iabyn.com> wrote​:

I really fail to see how CPAN is different in this regard.

Because this is Perl where not using CPAN is not an option and CPAN
library consumers rely on CPAN libraries being almost entirely
rock-solid, or at least the author having easy and quick ways to fix
bugs.

You still haven't differentiated perl+CPAN from some_other_language +
3rd_party_libraries_needed_to_get_the_job_done.

If using a programming language other than perl, then it is likely that​:

1) not using 3rd-party libraries is not an option;
2) consumers of 3rd-party libraries rely on those libraries being almost
  entirely rock-solid, or at least the author having easy and quick ways
  to fix bugs.

If you really look at all 3 points i made, in aggregate, and don't see
how this is a problem and a danger, then i don't think i can come up
with other word combinations to make you see it.

Can you come up with a hypothetical scenario, e.g. a multi-threaded
program that uses libraries to connect to a database and retrieve and
parse some XML data. Then got through it step by step so that I can see
why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
XML library is safe? What is the crucial difference between the two that
flips it from being safe to unsafe?

--
If life gives you lemons, you'll probably develop a citric acid allergy.

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From tom@binary.com

On 10 April 2018 at 21​:59, Dave Mitchell <davem@​iabyn.com> wrote​:

On Tue, Apr 10, 2018 at 02​:59​:50PM +0200, Christian Walde wrote​:

If you really look at all 3 points i made, in aggregate, and don't see
how this is a problem and a danger, then i don't think i can come up
with other word combinations to make you see it.

Can you come up with a hypothetical scenario, e.g. a multi-threaded
program that uses libraries to connect to a database and retrieve and
parse some XML data. Then got through it step by step so that I can see
why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
XML library is safe? What is the crucial difference between the two that
flips it from being safe to unsafe?

Here's one trivial hacked-together example of code which I would argue
leads to "unexpected" results, at least from the perspective of a C or Java
programmer experienced in the ways of threads​:

#!/usr/bin/perl
use strict;
use warnings;

use threads;
use threads​::shared;

my %items : shared;
my $lockvar : shared;

my $t = threads->create(sub {
  lock $lockvar;
  cond_wait($lockvar);
  my @​data : shared = qw(example content here);
  $items{0 + \@​data} = \@​data;
});
my $seen_item;
while(1) {
  do {
  lock $lockvar;
  cond_broadcast($lockvar);
  } until keys %items;
  print "Item with address " . $_ . " actually refers to " . (0 +
$items{0 + $_}) . "\n" for keys %items;
  last if defined $seen_item and $seen_item ne (values %items)[0];
  ($seen_item) = values %items;
}
$t->join;

=pod

Sample output (5.26.1)​:

  Item with address 31700632 actually refers to 31360376
  Item with address 31700632 actually refers to 30235168

=cut

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Tue, Apr 10, 2018 at 08​:40​:45PM +0800, Tom Molesworth via perl5-porters wrote​:

The official threads.pm documentation claims that the current code has what
I'd class as "bugs"​:

    Even with the latest version of Perl\, it is known that certain
    constructs with threads may result in warning messages concerning
    leaked scalars or unreferenced scalars\. However\, such warnings are
    harmless\, and may safely be ignored\.

If they are leaking, this has bad implications for long-running code.
If they're not leaking, why the warning?
If this information is out of date, then the bugs are in the documentation!

That text is 11 years old and should be removed.

This has dangerous assumptions in it​:

- Thread safety is not a binary on/off switch. Modules can appear to be
thread-safe under some conditions and not be so under others. That's why i
called it a heisenbug in my previous email.

Indeed, this is arguably worse than clear "the code crashed so it doesn't
work" cases.

Any threaded program in any programming language is susceptible to subtle
heisenbugs. I'd argue that perl is less susceptible than many languages
due to its 'not shared be default' nature.

We also don't have good documentation on how to make modules thread-safe​:
I've encountered quite a few Perl developers who are confident that thread
safety is only an issue with XS.

I'd be one of those, in the sense that the perl language itself is
thread-safe, and doesn't normally need locks or special handling; but in
any environment where there are multiple concurrently running threads,
there will be some extra considerations required.

As an example​: far as I recall, the threads.pm documention and perlmod page
to which they link never explicitly state that ref addresses will be
different in each thread​:

Since the whole basic principle of ithreads is that data isn't shared by
default, it would be astonishing if two (non-shared) refs in two different
threads had the same address.

I know much of my CPAN code will be susceptible to this type of issue.

Do you think there are many such issues, or is ref addresses the main one?

Any
time someone asks about thread safety, I point to the "discouraged" line in
the official documentation and explain that I don't expect my code to work
when multiple threads are active - "maybe try again if we have a new
threads implementation in the future".

Fine - add to the pod in your modules that they're not thread-safe.

- Windows support​: not a mistake.

Yet I suspect many complaints about bad threading behaviour in perl (such
as ref addresses changing in a sub-thread) applies just as much to the
Windows fork emulation - e.g. do a pseudo-fork on Windows and all the
ref addresses change. I don't see you we can praise one and condemn the
other.

Exposing the API to end users without
warnings and clear information about how threads differs from other
languages, on the other hand - maybe not ideal.

I am all in favour of having, at the same location as the current
'discouraged' text, but instead of it, a big flashing neon sign saying
that perl threads are a bit different from what you might expect and only
use them if you understand that (e.g. each thread is a non-shared clone
of the parent, with memory and start-up-cost implications).

Discouraged features aren't currently candidates for removal, but we may

later deprecate them if they're found to stand in the way of a significant
improvement to the Perl core.

- surely this is the case? If we come up with a better way to implement
threads while retaining the ability for Windows to support some form of
process emulation and related features, wouldn't this be something that's
welcomed? I don't get the impression that there's widespread satisfaction
with the current state of affairs on either threads.pm or threads​::shared.

Even if we ended up doing Windows fork() differently, I wouldn't (at the
moment) advocate deprecating or removing threads.pm threads​::shared, or
the underlying core implementation.

--
"I do not resent criticism, even when, for the sake of emphasis,
it parts for the time with reality".
  -- Winston Churchill, House of Commons, 22nd Jan 1941.

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From Eirik-Berg.Hanssen@allverden.no

On Tue, Apr 10, 2018 at 4​:27 PM, Dave Mitchell <davem@​iabyn.com> wrote​:

On Tue, Apr 10, 2018 at 08​:40​:45PM +0800, Tom Molesworth via perl5-porters
wrote​:

We also don't have good documentation on how to make modules thread-safe​:
I've encountered quite a few Perl developers who are confident that
thread
safety is only an issue with XS.

I'd be one of those, in the sense that the perl language itself is
thread-safe, and doesn't normally need locks or special handling; but in
any environment where there are multiple concurrently running threads,
there will be some extra considerations required.

  And yet, documented behaviour of pure-Perl modules breaks under threads.

  Because of this​:

As an example​: far as I recall, the threads.pm documention and perlmod
page
to which they link never explicitly state that ref addresses will be
different in each thread​:

Since the whole basic principle of ithreads is that data isn't shared by
default, it would be astonishing if two (non-shared) refs in two different
threads had the same address.

  (And quite a few pure-Perl modules do not expect this. I don't have a
count of how common it is on CPAN though.)

Any
time someone asks about thread safety, I point to the "discouraged" line
in
the official documentation and explain that I don't expect my code to
work
when multiple threads are active - "maybe try again if we have a new
threads implementation in the future".

Fine - add to the pod in your modules that they're not thread-safe.

  How about we make thread support opt-in instead of opt-out?

Exposing the API to end users without
warnings and clear information about how threads differs from other
languages, on the other hand - maybe not ideal.

I am all in favour of having, at the same location as the current
'discouraged' text, but instead of it, a big flashing neon sign saying
that perl threads are a bit different from what you might expect and only
use them if you understand that (e.g. each thread is a non-shared clone
of the parent, with memory and start-up-cost implications).

  That is, add a warning that threads are unsupported by CPAN modules
unless such support is advertised?

Eirik

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Tue, Apr 10, 2018 at 06​:42​:44AM -0700, Sergey Aleynikov via RT wrote​:

As for the core, I agree that it's now much more thread-stable than
before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no
threads" with some nice crash examples in it - they crash no more. But
still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just
because of no one has discovered that. And the following still fails
loudly (though not dumps a core);

while (1) {
push @​foo, threads->create(sub {
require IO​::Handle;
});
$_->detach for(splice @​foo);
}

Oh, that's fun, Looks like _create_getline_subs in IO.xs is directly
modifying the global PL_check[] rather than via the official API which
does the necessary locking.

So while yes, perl is much more thread-safe inherently as it used to be,
I won't recommend using threads in it to anyone.

Perl has bugs. Perl's threading has bugs. I haven't seen any particular
evidence yet that the number of threaded bugs is disproportionately large
on recent perls.

--
The Enterprise successfully ferries an alien VIP from one place to another
without serious incident.
  -- Things That Never Happen in "Star Trek" #7

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @rcaputo

On Apr 10, 2018, at 10​:27, Dave Mitchell <davem@​iabyn.com> wrote​:

I am all in favour of having, at the same location as the current
'discouraged' text, but instead of it, a big flashing neon sign saying
that perl threads are a bit different from what you might expect and only
use them if you understand that (e.g. each thread is a non-shared clone
of the parent, with memory and start-up-cost implications).

Also that you will mostly be on your own navigating the differences,
as Perl's users on average are reluctant at best to help you with them
and will often "actively encourage" you to seek other implementations.

--
Rocco Caputo <rcaputo@​pobox.com>

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @khwilliamson

On 04/10/2018 07​:42 AM, Sergey Aleynikov via RT wrote​:

On Tue, 10 Apr 2018 04​:04​:06 -0700, davem wrote​:

The problem is in people's expectations. When most part of something is thread-safe, they'll just assume "everything is thread-safe" and then'll be hit very hard. One of the examples I like is Mouse, which was thread-unsafe until the year 2015. While it's not as popular as Moose, it's still a module with large a large user base. And your chances to encounter bugs in not so popular XS modules are much, much higher. What's even worse, you're likely to hit them only under some unlucky circumstances (see example below).

As for the core, I agree that it's now much more thread-stable than before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no threads" with some nice crash examples in it - they crash no more. But still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just because of no one has discovered that.

Actually threaded pure perl programs are still unsafe in 5.26. perl
switches the locale behind your back, even if you follow our admonitions
to not explicitly use locales. I'm to blame for some of these, in my
earlier naivete, and some have been there for a long time. Sergey found
one case that I added in 5.24 I believe, and contributed a test file to
verify it's still fixed

5.28 uses thread safe locales if available on the system. On other
systems, I avoid switching locales, and added a mutex for those cases
where switching is still done.

When perl starts up, it reads the environment to see what each locale
category should be set to. If not all categories are set to the same
thing, this would cause perl to potentially switch locales to gather
information about the outliers. This is potentially problematic on
unsafe threaded builds. I solved this by gathering the information at
start up, and caching it. This is one of the ways 5.28 avoids switching
locales.

In researching this, I looked in the POSIX standard for functions it
allows to be non-thread safe. I have a WIP to add cautions about these
to XS writers. I also noticed that the Linux man pages indicated that
they have failed to implement correctly some that are supposed to be
thread-safe. Other systems may implement these safely, but not others.

I searched the perl source code for instances of these calls. And then
manually started to examine them to see if there was a problem. I have
not finished (and of course I may make mistakes in my analysis). The
glaring case where there is a problem is in accessing the environment
(getenv() et.al.) These need to be protected by a mutex, but it's only
a problem if another thread is changing the environment at the same
time, a much less common occurrence. My guess is that these aren't
crashing things because most environment changes would tend to be done
at start-up, even before thread creation. Things you might not expect
to, without thinking about it, like tzset(), do access the environment,
and there is a potential race if the environment is changed by another
thread during tzset's execution. And tzset is called from places that
at first glance you wouldn't expect.

  And the following still fails loudly (though nott dumps a core);

while (1) {
push @​foo, threads->create(sub {
require IO​::Handle;
});
$_->detach for(splice @​foo);
}

So while yes, perl is much more thread-safe inherently as it used to be, I won't recommend using threads in it to anyone.

---
via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=125106

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Tue, Apr 10, 2018 at 10​:26​:21PM +0800, Tom Molesworth via perl5-porters wrote​:

On 10 April 2018 at 21​:59, Dave Mitchell <davem@​iabyn.com> wrote​:

Can you come up with a hypothetical scenario, e.g. a multi-threaded
program that uses libraries to connect to a database and retrieve and
parse some XML data. Then got through it step by step so that I can see
why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
XML library is safe? What is the crucial difference between the two that
flips it from being safe to unsafe?

Here's one trivial hacked-together example of code which I would argue
leads to "unexpected" results, at least from the perspective of a C or Java
programmer experienced in the ways of threads​:

That isn't a reply to the asked question, which was trying to eke out an
understanding of why Christian believes the CPAN ecosystem in some way
makes threading more hazardous on perl than in other languages (which it
may, but I can't grasp the point he's trying to make).

The example code you've given seems to be (at a quick glance), just the
fact (again) that refs in different threads have different addresses in
perl.

Yes of course this will be confusing to anyone who expects the perl
threading model to be the same as C or Java's. And yes I agree that the
docs should explain the model early on to manage expectations.

--
This is a great day for France!
  -- Nixon at Charles De Gaulle's funeral

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @iabyn

On Tue, Apr 10, 2018 at 04​:46​:13PM +0200, Eirik Berg Hanssen wrote​:

That is, add a warning that threads are unsupported by CPAN modules
unless such support is advertised?

I'd be happy with that.

--
Lear​: Dost thou call me fool, boy?
Fool​: All thy other titles thou hast given away; that thou wast born with.

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @wchristian

On Tue, 10 Apr 2018 15​:59​:53 +0200, Dave Mitchell <davem@​iabyn.com> wrote​:

why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
XML library is safe? What is the crucial difference between the two that
flips it from being safe to unsafe?

Very short version​: Java has a much more bigger userbase, money invested, and things like a database module or an xml parser under java will be written under the assumption of being required to work with threads, as Java uses threads like skittles.

Also in Java both of these are in the standard library, not 3rd party.

Additionally, to give you a more concrete example of a simple Perl threads fuck-up​:

In 2011 i attempted to write a simple parallel http downloader, using LWP and some thread queue manager module. It worked 100% fine under windows, no matter what i did. On Fedora it seemed to work fine, but crashed randomly when attempting to handle too many downloads.

This would be completely inconceivable in Java.

--
With regards,
Christian Walde

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From tom@binary.com

On 10 April 2018 at 23​:11, Dave Mitchell <davem@​iabyn.com> wrote​:

On Tue, Apr 10, 2018 at 10​:26​:21PM +0800, Tom Molesworth via perl5-porters
wrote​:

On 10 April 2018 at 21​:59, Dave Mitchell <davem@​iabyn.com> wrote​:

Can you come up with a hypothetical scenario, e.g. a multi-threaded
program that uses libraries to connect to a database and retrieve and
parse some XML data. Then got through it step by step so that I can see
why using perl and CPAN is dangerous, but using (e.g.) java and a DB
and
XML library is safe? What is the crucial difference between the two
that
flips it from being safe to unsafe?

Here's one trivial hacked-together example of code which I would argue
leads to "unexpected" results, at least from the perspective of a C or
Java
programmer experienced in the ways of threads​:

That isn't a reply to the asked question, which was trying to eke out an
understanding of why Christian believes the CPAN ecosystem in some way
makes threading more hazardous on perl than in other languages (which it
may, but I can't grasp the point he's trying to make).

The example code you've given seems to be (at a quick glance), just the
fact (again) that refs in different threads have different addresses in
perl.

Related but slightly different​: this time it's 2 different addresses
*within the same thread*, none of which match the refaddr in the other
thread. For something that's supposedly "shared", I'd call having 3
addresses surprising at best!

What I'm trying to demonstrate is it's easy to hit implementation details
in Perl, ones that go against the common understanding of "threads". I
don't think people should have to read Shared.xs before they can implement
a multithreaded XML+DB application. Documentation might help, but there's
an uphill struggle at the first word​: what we call "threads", other
languages might call a nearly-unrecognisable special case!

(Eirik and Christian have already addressed the other comments I was going
to make, so I'll stop here - sorry for derailing the thread)

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @Leont

On Tue, Apr 10, 2018 at 2​:40 PM, Tom Molesworth via perl5-porters
<perl5-porters@​perl.org> wrote​:

Discouraged features aren't currently candidates for removal, but we may
later deprecate them if they're found to stand in the way of a significant
improvement to the Perl core.

- surely this is the case? If we come up with a better way to implement
threads while retaining the ability for Windows to support some form of
process emulation and related features, wouldn't this be something that's
welcomed? I don't get the impression that there's widespread satisfaction
with the current state of affairs on either threads.pm or threads​::shared.

A crucial thing to understand in this idea is that ithreads and
threads.pm are not the same thing. ithreads is a C-level feature in
the implementation, threads.pm and pseudoforks are end-user level
features that are built upon ithreads.

One can build other abstractions on top of them. ithreads is not a
good abstraction for anything resembling shared-memory architectures,
but does fit a number of other concurrency models. threads​::lite did
proof that concept I hope.

If that's not what one wants, it's probably possible to implement GIL
threading instead (though I'm not sure if we'd really want that
either). I think that's unexplored territory though.

Leon

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @xenu

On Tue, 10 Apr 2018, at 16​:46, Eirik Berg Hanssen wrote​:

How about we make thread support opt-in instead of opt-out?

It's already the case. By default perl builds without ithreads.

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From Eirik-Berg.Hanssen@allverden.no

On Wed, Apr 11, 2018 at 12​:33 AM, Tomasz Konojacki <me@​xenu.pl> wrote​:

On Tue, 10 Apr 2018, at 16​:46, Eirik Berg Hanssen wrote​:

How about we make thread support opt-in instead of opt-out?

It's already the case. By default perl builds without ithreads.

  Context​:

Fine - add to the pod in your modules that they're not thread-safe.

How about we make thread support opt-in instead of opt-out?

  That is, I was referring to thread support in CPAN modules. I'm
suggesting we document, in big fat (blinking marquee etc ...)​:

  Unless a module explicitly states it supports threads, and you use it
with threads, you're on your own and get to keep both pieces when it breaks.

Eirik

@p5pRT
Copy link
Author

p5pRT commented Apr 10, 2018

From @xenu

On Wed, 11 Apr 2018, at 00​:44, Eirik Berg Hanssen wrote​:

  Context​: 

Fine - add to the pod in your modules that they're not thread-safe.

  How about we make thread support opt-in instead of opt-out?

  That is, I was referring to thread support in CPAN modules.  I'm suggesting we document, in big fat (blinking marquee etc ...)​:

  Unless a module explicitly states it supports threads, and you use it with threads, you're on your own and get to keep both pieces when it breaks.

Eirik

Oh, I see, I missed the last sentence of your message. Sorry for the noise.

@p5pRT
Copy link
Author

p5pRT commented Apr 11, 2018

From @bulk88

On Tue, 10 Apr 2018 15​:25​:00 -0700, LeonT wrote​:

If that's not what one wants, it's probably possible to implement GIL
threading instead (though I'm not sure if we'd really want that
either). I think that's unexplored territory though.

Wouldnt one of the existing fibers/future/promises/Coro/async-but-not-async-with-explicit-yields modules that swap Perl stacks inside the same interp/same perl thread be perl's already-implemented GIL concept?

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Apr 11, 2018

From @Leont

On Wed, Apr 11, 2018 at 8​:45 AM, bulk88 via RT
<perlbug-followup@​perl.org> wrote​:

On Tue, 10 Apr 2018 15​:25​:00 -0700, LeonT wrote​:

If that's not what one wants, it's probably possible to implement GIL
threading instead (though I'm not sure if we'd really want that
either). I think that's unexplored territory though.

Wouldnt one of the existing fibers/future/promises/Coro/async-but-not-async-with-explicit-yields modules that swap Perl stacks inside the same interp/same perl thread be perl's already-implemented GIL concept?

AFAIK those are all "one interpreter one os-thread", wheras GIL
threading is "one interpreter many os-threads"

Leon

@MaxPerl
Copy link

MaxPerl commented Nov 6, 2022

Please remove the discourage warning. The threads implementation has limitations, of course. But they are explained into the deep, so that everybody can decide whether or not to use threads. At the moment the warning frightens developers to use and try threads (I was frightened and begin only after a long time to study threads). But the threads implementation only can better in Perl if it is used and bugs can be found.

A problem indeed is that many modules doesn't support threads. The perlmod docs explain a simple mechanism for making a module threadsafe by adding CLONE_SKIP to the module. I found it helpful to make modules not used in the worker threads threadsafe at runtime with following BEGIN block:

BEGIN {
	*PackageNameNotUsedInThreads::CLONE_SKIP = sub {print "no cloning \n"; 1;}
}

Perhaps this is a useful trick for others and could be documented, too?

Thanks in advance and best wishes,
Max

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants