Skip Menu |
Report information
Id: 125106
Status: open
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: choroba [at] matfyz.cz
ikegami [at] adaelis.com
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: unknown
Perl Version: (no value)
Fixed In: (no value)

Attachments
0001-Remove-the-word-discouraged-from-threads.patch



Date: Tue, 5 May 2015 11:23:03 -0400
From: Eric Brine <ikegami [...] adaelis.com>
To: perlbug [...] perl.org
Subject: Why are threads discouraged?
Download (untitled) / with headers
text/plain 318b
Show quoted text
The use of interpreter-based threads in perl is officially discouraged.

Why is that? Is it just because they're rather heavy? If so, why not just say so and let the user decide whether he wants to use them or not? After all, heavy threads aren't a problem if you reuse them (as in a worker model).

To: perl5-porters [...] perl.org
From: Peter Rabbitson <rabbit-p5p [...] rabbit.us>
Date: Tue, 05 May 2015 17:37:29 +0200
Subject: Re: [perl #125106] Why are threads discouraged?
Download (untitled) / with headers
text/plain 525b
On 05/05/2015 05:23 PM, Eric Brine (via RT) wrote: Show quoted text
> # New Ticket Created by "Eric Brine" > # Please include the string: [perl #125106] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=125106 > > > > threads.pm says: > > The use of interpreter-based threads in perl is officially discouraged. > > > Why is that? Is it just because they're rather heavy? If so, why not just > say so and let the user decide whether he wants to use them or not?
+1
CC: bugs-bitbucket [...] rt.perl.org
To: "Eric Brine\" (via RT) <perlbug-followup [...] perl.org>" [...] x6.develooper.com
Date: Tue, 5 May 2015 17:20:44 +0100
From: "Paul \"LeoNerd\" Evans" <leonerd [...] leonerd.org.uk>
Subject: Re: [perl #125106] Why are threads discouraged?
Download (untitled) / with headers
text/plain 944b
On Tue, 5 May 2015 08:23:15 -0700 "Eric Brine" (via RT) <perlbug-followup@perl.org> wrote: Show quoted text
> # New Ticket Created by "Eric Brine" > # Please include the string: [perl #125106] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=125106 > > > > threads.pm says: > > The use of interpreter-based threads in perl is officially > discouraged. > > > Why is that? Is it just because they're rather heavy? If so, why not > just say so and let the user decide whether he wants to use them or > not? After all, heavy threads aren't a problem if you reuse them (as > in a worker model).
Or people could help me finish writing/editing the perlconcurrency doc, so one day it will explain in more detail: http://www.leonerd.org.uk/code/perlconcurrency.pod -- Paul "LeoNerd" Evans leonerd@leonerd.org.uk http://www.leonerd.org.uk/ | https://metacpan.org/author/PEVANS
Download (untitled)
application/pgp-signature 213b

Message body not shown because it is not plain text.

From: Rocco Caputo <rcaputo [...] pobox.com>
CC: bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #125106] Why are threads discouraged?
Date: Tue, 5 May 2015 13:49:52 -0400
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.9k
Show quoted text
> On May 5, 2015, at 11:23, Eric Brine <perlbug-followup@perl.org> wrote: > > # New Ticket Created by "Eric Brine" > # Please include the string: [perl #125106] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/Ticket/Display.html?id=125106 > > > > threads.pm says: > > The use of interpreter-based threads in perl is officially discouraged. > > > Why is that? Is it just because they're rather heavy? If so, why not just > say so and let the user decide whether he wants to use them or not? After > all, heavy threads aren't a problem if you reuse them (as in a worker > model).
If your documentation is anything like mine, the paragraph before that statement tries to explain why. Perhaps the discouragement should begin with "Because of this, the use of...." to hint that the reader may have skipped something important and to discourage people from taking a provocative statement out of context. The discouragement was added on Mar 2, 2014 after an intense deliberation on perl5-porters: "RFC: add discouragement warning to perl threads documentation" http://www.gossamer-threads.com/lists/perl/porters/305035 The RFC was a product of an observation in irc.perl.org #p5p that most people in the Perl community, when asked for help with threads in Perl, will harshly denounce their viability and recommend that the problematic program be gutted and rewritten with almost anything but. The #p5p IRC discussion began on Feb 19, 2014, with "Perl's threads are socially weird. They exist and are supported, but they (and their users) are heavily deprecated in the community." Or it began with an observation about 90 minutes earlier in a #p5p discussion with/about a Perl novice: right, because most channels don't answers threads questions because most channels are full of people who don't use threads.pm because 'use threads;' almost always indicates brain damage -- Rocco Caputo <rcaputo@pobox.com>
To: Rocco Caputo <rcaputo [...] pobox.com>
CC: perl5 porters <perl5-porters [...] perl.org>, "bugs-bitbucket [...] rt.perl.org" <bugs-bitbucket [...] rt.perl.org>
From: Eric Brine <ikegami [...] adaelis.com>
Subject: Re: [perl #125106] Why are threads discouraged?
Date: Tue, 10 May 2016 16:22:00 -0400
Download (untitled) / with headers
text/plain 1.6k
On Tue, May 5, 2015 at 1:49 PM, Rocco Caputo <rcaputo@pobox.com> wrote:
Show quoted text
> On May 5, 2015, at 11:23, Eric Brine <perlbug-followup@perl.org> wrote:
>
> # New Ticket Created by  "Eric Brine"
> # Please include the string:  [perl #125106]
> # in the subject line of all future correspondence about this issue.
> # <URL: https://rt.perl.org/Ticket/Display.html?id=125106 >
>
>
> threads.pm says:
>
> The use of interpreter-based threads in perl is officially discouraged.
>
>
> Why is that? Is it just because they're rather heavy? If so, why not just
> say so and let the user decide whether he wants to use them or not? After
> all, heavy threads aren't a problem if you reuse them (as in a worker
> model).

If your documentation is anything like mine, the paragraph before that statement tries to explain why.  Perhaps the discouragement should begin with "Because of this, the use of...." to hint that the reader may have skipped something important and to discourage people from taking a provocative statement out of context.

One might think that, except that the word "discouraged" is linked to a definition that means something quite different than "not to be used without understanding what it means that Perl threads are heavy". It actually defines "discouraged" as "we'd like to get rid of them, but we're not doing it now"
  • discouraged

    From time to time, we may mark language constructs and features which we consider to have been mistakes as discouraged. Discouraged features aren't currently candidates for removal, but we may later deprecate them if they're found to stand in the way of a significant improvement to the Perl core.

Date: Wed, 11 May 2016 19:54:37 +0200
Subject: Re: [perl #125106] Why are threads discouraged?
From: Sawyer X <xsawyerx [...] gmail.com>
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.6k
On 05/10/2016 10:22 PM, Eric Brine wrote: Show quoted text
> On Tue, May 5, 2015 at 1:49 PM, Rocco Caputo <rcaputo@pobox.com > <mailto:rcaputo@pobox.com>>wrote: >
> > On May 5, 2015, at 11:23, Eric Brine <perlbug-followup@perl.org
> <mailto:perlbug-followup@perl.org>> wrote:
> > > > # New Ticket Created by "Eric Brine" > > # Please include the string: [perl #125106] > > # in the subject line of all future correspondence about this issue. > > # <URL: https://rt.perl.org/Ticket/Display.html?id=125106
> <https://rt.perl.org/Ticket/Display.html?id=125106>>
> > > > > > threads.pm <http://threads.pm>says: > > > > The use of interpreter-based threads in perl is officially
> discouraged.
> > > > > > Why is that? Is it just because they're rather heavy? If so, why
> not just
> > say so and let the user decide whether he wants to use them or
> not? After
> > all, heavy threads aren't a problem if you reuse them (as in a
> worker
> > model).
> > If your documentation is anything like mine, the paragraph before > that statement tries to explain why. Perhaps the discouragement > should begin with "Because of this, the use of...." to hint that > the reader may have skipped something important and to discourage > people from taking a provocative statement out of context. > > > One might think that, except that the word "discouraged" is linked to > a definition that means something quite different than "not to be used > without understanding what it means that Perl threads are heavy". It > actually defines "discouraged" as "we'd like to get rid of them, but > we're not doing it now" > > * *discouraged* > > From time to time, we may mark language constructs and features > which we consider to have been mistakes as *discouraged*. > Discouraged features aren't currently candidates for removal, but > we may later deprecate them if they're found to stand in the way > of a significant improvement to the Perl core. >
I agree this is a confusing description. I also like Rocco's comments on describing *why* something is the way it is, even if briefly. I found an example in the thread Rocco linked[1], by Aristotle, on how to explain things briefly and clearly: "complex data structures are hard to share properly" The quote is out of context and does not constitute as the patch I would envision, but it's an example of how to describe something complicated succinctly. Perhaps we can form a patch describing what we mean by "discouraged". Having context might excuse us from trying to find a better term for this than "discouraged". [1] Thanks, Rocco!
To: perl5-porters [...] perl.org
Subject: Re: [perl #125106] Why are threads discouraged?
From: "Paul \"LeoNerd\" Evans" <leonerd [...] leonerd.org.uk>
Date: Sat, 14 May 2016 12:31:26 +0100
Download (untitled) / with headers
text/plain 943b
On Wed, 11 May 2016 19:54:37 +0200 Sawyer X <xsawyerx@gmail.com> wrote: Show quoted text
> I agree this is a confusing description. > > I also like Rocco's comments on describing *why* something is the way > it is, even if briefly. I found an example in the thread Rocco > linked[1], by Aristotle, on how to explain things briefly and clearly: > > "complex data structures are hard to share properly" > > The quote is out of context and does not constitute as the patch I > would envision, but it's an example of how to describe something > complicated succinctly. Perhaps we can form a patch describing what > we mean by "discouraged". Having context might excuse us from trying > to find a better term for this than "discouraged". > > [1] Thanks, Rocco!
If we have some better wording I could put it into perlconcurrency.pod as well... -- Paul "LeoNerd" Evans leonerd@leonerd.org.uk http://www.leonerd.org.uk/ | https://metacpan.org/author/PEVANS
Download (untitled)
application/pgp-signature 181b

Message body not shown because it is not plain text.

To: perlbug [...] perl.org
Date: Sat, 24 Mar 2018 16:57:35 +0100 (CET)
From: "E. Choroba" <choroba [...] matfyz.cz>
Subject: Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 3.7k
----------------------------------------------------------------- [Please describe your issue here] The documentation of threads.pm says that perl threads are "discouraged". This causes lots of confusion, as the term is defined in perlpolicy in the following way: discouraged From time to time, we may mark language constructs and features which we consider to have been mistakes as discouraged. Discouraged features aren't currently candidates for removal, but we may later deprecate them if they're found to stand in the way of a significant improvement to the Perl core. See also #125106, which seems to lead nowhere, maybe because it was too ambitious. This patch provides just the minimal change. See also http://www.perlmonks.org/?node_id=1107534 . [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=library severity=medium module=threads --- Site configuration information for perl 5.27.11: Configured by choroba at Sat Mar 24 15:15:34 CET 2018. Summary of my perl5 (revision 5 version 27 subversion 11) configuration: Commit id: e77195360c1a2b081a806df017630b3165e2aeac Platform: osname=linux osvers=4.4.120-45-default archname=x86_64-linux-thread-multi uname='linux still 4.4.120-45-default #1 smp wed mar 14 20:51:49 utc 2018 (623211f) x86_64 x86_64 x86_64 gnulinux ' config_args='-Dusedevel -de -Dusethreads -Dprefix=/home/choroba/localperl' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='cc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2' optimize='-O2' cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='' gccversion='4.8.5' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='cc' ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /usr/lib64/gcc/x86_64-suse-linux/4.8/include-fixed /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64 libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.22.so so=so useshrplib=false libperl=libperl.a gnulibc_version='2.22' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,-E' cccdlflags='-fPIC' lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' --- @INC for perl 5.27.11: /home/choroba/localperl/lib/site_perl/5.27.11/x86_64-linux-thread-multi /home/choroba/localperl/lib/site_perl/5.27.11 /home/choroba/localperl/lib/5.27.11/x86_64-linux-thread-multi /home/choroba/localperl/lib/5.27.11 --- Environment for perl 5.27.11: HOME=/home/choroba LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/choroba/perl5/bin:/home/choroba/bin:/bin:/sbin:/usr/bin:/usr/local/bin:/usr/games:/usr/X11R6/bin:/opt/gnome/bin:. PERL_BADLANG (unset) SHELL=/bin/bash

Message body is not shown because sender requested not to inline it.

CC: bugs-bitbucket [...] rt.perl.org
Date: Sun, 25 Mar 2018 16:27:42 -0400
From: Dan Book <grinnz [...] gmail.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: Perl5 Porters <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 1.1k
On Sat, Mar 24, 2018 at 12:31 PM, E. Choroba <perlbug-followup@perl.org> wrote:
Show quoted text
# New Ticket Created by  "E. Choroba"
# Please include the string:  [perl #133021]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=133021 >


-----------------------------------------------------------------
[Please describe your issue here]

The documentation of threads.pm says that perl threads are
"discouraged". This causes lots of confusion, as the term is defined
in perlpolicy in the following way:

   discouraged
        From time to time, we may mark language constructs and
        features which we consider to have been mistakes as
        discouraged. Discouraged features aren't currently candidates
        for removal, but we may later deprecate them if they're found
        to stand in the way of a significant improvement to the Perl
        core.

See also #125106, which seems to lead nowhere, maybe because it was
too ambitious. This patch provides just the minimal change.

See also http://www.perlmonks.org/?node_id=1107534 .

'not recommended' seems too lenient. 'strongly recommended against'?

-Dan 

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 775b
On Sat, 24 Mar 2018 09:31:06 -0700, choroba@matfyz.cz wrote: Show quoted text
> From time to time, we may mark language constructs and > features which we consider to have been mistakes as > discouraged.
Threads in their current implementation are arguable a mistake. You loose ~30% of performance and who-measured-how-much memory by just using perl build with threads. And most of XS CPAN is up to some degree threads-unsafe (pure-perl CPAN is safe, so the goals for threads is reached, but still). Show quoted text
> Discouraged features aren't currently candidates > for removal, but we may later deprecate them if they're found > to stand in the way of a significant improvement to the Perl > core.
Threads are not going anywhere by this definition. Just discouraged.
Date: Mon, 26 Mar 2018 09:36:20 +0100
From: Dave Mitchell <davem [...] iabyn.com>
To: Sergey Aleynikov via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 625b
On Sun, Mar 25, 2018 at 04:20:13PM -0700, Sergey Aleynikov via RT wrote: Show quoted text
> Threads in their current implementation are arguable a mistake. You > loose ~30% of performance and who-measured-how-much memory by just using > perl build with threads.
The test suite on blead on my system runs 17% slower under ithreads, but that includes running a bunch of ithreads-only tests too. So I think 30% is overstating it. I'm not aware that a normal single-threaded process uses significantly more memory under ithreads. -- Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: bugs-bitbucket [...] rt.perl.org, "E. Choroba" <perlbug-followup [...] perl.org> (via RT), perl5-porters [...] perl.org
From: "Christian Walde" <walde.christian [...] gmail.com>
Date: Mon, 26 Mar 2018 12:43:05 +0200
Download (untitled) / with headers
text/plain 571b
Since this change doesn't have a ticket attached to it, i figured it would be useful to provide the context of what led to its creation. This discouragement was introduced by rjbs in commit 10a4597703f4b044d3f968bf3923644ea2387f5c. I've extracted some of the chat log that led to it. https://gist.github.com/wchristian/97525932a89d0785e07371c60d84209c And here's the p5p mailing list thread that followed afterwards: https://www.nntp.perl.org/group/perl.perl5.porters/2014/03/msg213382.html It's a lot of material, but i hope it'll help understand the context here.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 330b
On Mon, 26 Mar 2018 01:36:54 -0700, davem wrote: Show quoted text
> The test suite on blead on my system runs 17% slower under ithreads, but > that includes running a bunch of ithreads-only tests too. So I think 30% > is overstating it.
It depends on the operation type. I've measured hash accesses and got ~10% loss on threaded build for 5.26.1.
CC: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Date: Mon, 26 Mar 2018 14:06:18 +0100
To: Sergey Aleynikov via RT <perlbug-followup [...] perl.org>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 540b
On Mon, Mar 26, 2018 at 04:52:54AM -0700, Sergey Aleynikov via RT wrote: Show quoted text
> On Mon, 26 Mar 2018 01:36:54 -0700, davem wrote:
> > The test suite on blead on my system runs 17% slower under ithreads, but > > that includes running a bunch of ithreads-only tests too. So I think 30% > > is overstating it.
> > It depends on the operation type. I've measured hash accesses and got > ~10% loss on threaded build for 5.26.1.
10% isn't 30%. -- A walk of a thousand miles begins with a single step... then continues for another 1,999,999 or so.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.7k
As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message. The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD. I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users. I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch). Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated. I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature. Just my two cents.
From: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perlbug-comment [...] perl.org
CC: perl5-porters [...] perl.org
Date: Mon, 26 Mar 2018 19:20:29 -0600
On 03/26/2018 06:54 PM, Jerry D. Hedden via RT wrote: Show quoted text
> As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message. > > The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD. > > I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users. > > I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch). > > Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated. > > I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature. > > Just my two cents. >
I agree with the above. And I really don't like 'not recommended' as I think it is too strong. Maybe just list the problems Even a 30% slowdown will be fully acceptable if you can divide the work up into 8 or 16 parallel pieces. The gain far outweighs the cost.
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perlbug-comment [...] perl.org
From: Dan Book <grinnz [...] gmail.com>
Date: Mon, 26 Mar 2018 21:27:10 -0400
Download (untitled) / with headers
text/plain 2.5k
On Mon, Mar 26, 2018 at 8:54 PM, Jerry D. Hedden via RT <perlbug-comment@perl.org> wrote:
Show quoted text
As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message.

The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use.  I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD.

I also acknowledge that the threads implementation is not ideal nor optimal.  Nonetheless, threads are useful, and are being used in the wild.  (I, for one, have even used them to good effect in production code.  <Gasp!>)  Yes, if you don't know what you're doing, threads can be problematic.  ("There be dragons...," and all that.)  However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users.

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist.  I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it.  I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated.

I'm not trying to convert anyone who doesn't like "interpreter-based threads".  Don't use them as you so choose.  After all, Perl has always been about options.  However, there ARE Perl developers that do feel they are a VERY useful feature.

Just my two cents.

While all of this is true, the other audience for this documentation section is the new Perl user who thinks threads sound like a nice lightweight way to spread work around without breaking all of their code (because most of the rest of Perl works this way, you can just throw things together and they all work). These users do not read far enough to realize that threads are not the optimal solution to their requirements without a strong warning. In my opinion, severe discouragement (if not the perlpolicy defined term) is warranted; if someone can achieve this while still making it clear that threads are not going to be deprecated for valid and optimal use cases for them, this would be ideal.

-Dan
Date: Tue, 27 Mar 2018 12:15:18 +0100
CC: perlbug-comment [...] perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: Karl Williamson <public [...] khwilliamson.com>
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 2.2k
On Mon, Mar 26, 2018 at 07:20:29PM -0600, Karl Williamson wrote: Show quoted text
> On 03/26/2018 06:54 PM, Jerry D. Hedden via RT wrote:
> > As one of several people who have maintained the various threads modules over the years, I regret that I missed being part of the original discussion that lead to the inclusion of the 'discouraged' message. > > > > The fact is that threads work, they are maintained, and they currently do not have any bugs preventing their use. I acknowledge that not all Perl modules are thread-safe, but there is sufficient documentation to that affect in the POD. > > > > I also acknowledge that the threads implementation is not ideal nor optimal. Nonetheless, threads are useful, and are being used in the wild. (I, for one, have even used them to good effect in production code. <Gasp!>) Yes, if you don't know what you're doing, threads can be problematic. ("There be dragons...," and all that.) However, the same can be argued to greater or lesser degrees of any programming language feature in the hands of unsophisticated users. > > > > I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist. I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch). > > > > Since the addition of the 'discouraged' message, I have received several emails from professional Perl developers from around the world expressing concern about it. I expressed to them the same opinions I have given above, namely that threads work (but you have to know what you're doing), and that threads have not been deprecated. > > > > I'm not trying to convert anyone who doesn't like "interpreter-based threads". Don't use them as you so choose. After all, Perl has always been about options. However, there ARE Perl developers that do feel they are a VERY useful feature. > > > > Just my two cents.
+1 Show quoted text
> I agree with the above. And I really don't like 'not recommended' as I > think it is too strong. Maybe just list the problems
+1 -- The Enterprise's efficient long-range scanners detect a temporal vortex distortion in good time, allowing it to be safely avoided via a minor course correction. -- Things That Never Happen in "Star Trek" #21
Date: Tue, 27 Mar 2018 10:32:54 -0400
CC: Karl Williamson <public [...] khwilliamson.com>, perlbug-comment [...] perl.org, perl5-porters [...] perl.org
To: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Rocco Caputo <rcaputo [...] pobox.com>
Download (untitled) / with headers
text/plain 5.1k
Download (untitled) / with headers
text/html 14.1k
Show quoted text
On Mar 27, 2018, at 07:15, Dave Mitchell <davem@iabyn.com> wrote:

On Mon, Mar 26, 2018 at 07:20:29PM -0600, Karl Williamson wrote:
On 03/26/2018 06:54 PM, Jerry D. Hedden via RT wrote:

I feel that, while the wording of the POD notice is reasonable, the WARNING heading is alarmist.  I feel strongly that the heading should be changed to NOTICE, and the word 'discouraged' be changed to 'not recommended' (as per the original poster's patch).

I started the discussion that led to the addition of the discouragement warning.  Its original purpose was to manage users' expectations of the kind of help they'd get from the larger Perl community for trying to use this feature.

Toning down the notice doesn't break its original purpose.

Show quoted text
I agree with the above.  And I really don't like 'not recommended' as I
think it is too strong.  Maybe just list the problems

Getting rid of the notice entirely would resume failing to manage users' expectations.

In case someone on-list isn't aware of the advice given off-list, I'm closing with a small list of quotes from freenode #perl.  These are pieces of real advice given to Perl users looking for help using threads.  Some of you may recognize things you've said.

Some of you may be tempted to say that IRC isn't representative of a larger perl community, but many of the vocal opponents on IRC also participate in other areas.  They use and advocate Perl professionally.  They attend or speak at Perl conferences.  They are perl5 porters.

This list includes both old and recent statements:

Show quoted text
don't use threads

Show quoted text
When someone says "I need to use threads" in perl, they (99.9% of the time) mean "I don't know what I'm talking about, please (re-)educate me"

Show quoted text
But at least you didn't start out trying to use threads.

Show quoted text
Yes.. Don't use threads. At all.

Show quoted text
The simple answer is "don't use threads." Just fork if you must.

Show quoted text
no, because threads are usually a stupid answer.

Show quoted text
do not use threads.

Show quoted text
'use threads' is "please let me use the windows fork hack code"

Show quoted text
also, don't use threads in Perl; if your problem requires threads, don't use Perl

Show quoted text
you seem to have missed the point earlier that was made about "dont use threads"

Show quoted text
the thread lecture is don't use threads in Perl, or don't use Perl

Show quoted text
before anything else, notice that people here will start telling you about how you probably shouldn't use threads in Perl.

Show quoted text
Hi. Please don't use threads.

Show quoted text
I wonder how many times we've told qubit not to use threads and how mayn times they've ignored us.

Show quoted text
Because threads are not what you want. Threads in Perl, doubly so

Show quoted text
(Almost) Everyone who wants to use threads in perl understands them incorrectly.

Show quoted text
very few professional perl devs will use threads for real code

Show quoted text
because threads are a stupid way to do concurrency in perl.

Show quoted text
I really wish there was this little genie like the little green dude in The Flintstones that would pop up and educate people when they think they want to use threads in perl.

Show quoted text
Don't use threads, would be our suggestion
Show quoted text

Show quoted text
If you want to wrap your head in crazy concurrency bugs and random failures, then use threads.

Show quoted text
Suggestion: don't use threads

Show quoted text
don't use threads in Perl

Show quoted text
perlbot: threads is Friends don't let friends use threads ; Elizabeth on Threads - http://www.perlmonks.org/index.pl?node_id=288022

Show quoted text
basically: stop trying to use threads.

Show quoted text
and really you don't want to use threads at all

Show quoted text
or you can use threads, and we will ignore you when it all breaks.

Show quoted text
step 1: don't use threads

Show quoted text
The response is don't use threads

Show quoted text
never use threads in perl

Show quoted text
So then they're always asking "how can I do threads" and we say "don't use threads" and they complain and say "waaaaa I need them for doing lots of things at once" and then we say "no you dno't, you can do other things".. and they give a look of complete utter confusion.

Show quoted text
yes but in your case it is because threads are known to be broken and so will cause you problems

Show quoted text
First convince your department not to use threads.

Show quoted text
I don't imagine you'll get much help.. Our advice is "don't use threads"... hardly anyone here does much if anything with threads.. So likely nobody here really knows

Show quoted text
This may be the definitive idea of why allowing people to use threads is a bad idea.

Show quoted text
you should not use threads.

Show quoted text
don't use threads. problem solved.

Show quoted text
don't use threads is the answer

Show quoted text
You don't want to use threads.

Show quoted text
ah, the truth "I use threads because I'm too dumb to understand select"

Show quoted text
But if you're wanting to use threads, odds are you are doing something wrong :)

Show quoted text
the answer is invariably "don't use threads"    :)

Show quoted text
ah, well, your first mistake was to actually seriously use threads in Perl

Show quoted text
look, we can't actually advise you on a concurrency solution unless you tell us what the program's going to do - although the odds are pretty good that we'll say "don't use threads" since that's almost always the worst choice

Show quoted text
code containing 'use Switch' or 'use threads' can be assumed to be broken

Show quoted text
the best advice I can give is to not use threads in Perl

Show quoted text
some idiot did 'use threads;' and now you have a broken program that needs replacing

Show quoted text
DON'T USE THREADS, DON'T TELL OTHERS TO USE THREADS.

Show quoted text
also, #perl will almost always refuse to help with threads. because threads in perl are shit.

-- 
Rocco Caputo <rcaputo@pobox.com>
Show quoted text
From: Elizabeth Mattijsen <liz [...] dijkmat.nl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perl5-porters [...] perl.org
Date: Tue, 27 Mar 2018 18:31:14 +0200
Download (untitled) / with headers
text/plain 110b
Perhaps a reference to the “forks” module on CPAN could be added? https://metacpan.org/pod/forks Liz
To: Karl Williamson <public [...] khwilliamson.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Leon Timmermans <fawaka [...] gmail.com>
Date: Wed, 28 Mar 2018 02:23:59 +0200
CC: Randy Stauner via RT <perlbug-comment [...] perl.org>, Perl5 Porters <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 939b
On Tue, Mar 27, 2018 at 3:20 AM, Karl Williamson <public@khwilliamson.com> wrote: Show quoted text
> I agree with the above. And I really don't like 'not recommended' as I > think it is too strong. Maybe just list the problems
I think the main problem with the current wording is that it discourages without explaining why. In particular it doesn't explain that threads.pm does something different than what many people expect it to do. People expect it to be good for "share all the data" scenarios (because pretty much anything else called threads is), even though it is terrible at that. This confusion isn't helped by threads implementations in commonly used implementations of languages occupying the same niches as we do (e.g. cpython, Ruby MRI) sucking in exactly the opposite way (they're good at sharing but useless at actually being parallel). It's not so much threads.pm that needs a big fat warnings, it's threads::shared that does. Leon
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 363b
On Mon, 26 Mar 2018 06:06:47 -0700, davem wrote: Show quoted text
> 10% isn't 30%.
My idea was to show that you can't attribute all the test run slowdown to additional threading tests. Probably, I should've made this clearer in that message. As for the total slowdown figures, it depends on the application, but it's still "free" - you get it even if you don't start any threads.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 656b
On Mon, 26 Mar 2018 18:21:02 -0700, public@khwilliamson.com wrote: Show quoted text
> I agree with the above. And I really don't like 'not recommended' as > I > think it is too strong. Maybe just list the problems > > Even a 30% slowdown will be fully acceptable if you can divide the > work > up into 8 or 16 parallel pieces. The gain far outweighs the cost.
But if you use processes for parallelism and not threads, you don't get this slowdown at all and get the same multicore boost. And you don't have to guess which XS module will segfault next. The only case when I find using threads reasonable is writing GUI, where you _have to_ be in the same address space.
From: Petr Pisar <ppisar [...] redhat.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perl5-porters [...] perl.org
Date: 28 Mar 2018 10:36:35 -0000
Download (untitled) / with headers
text/plain 273b
On 2018-03-27, Elizabeth Mattijsen <liz@dijkmat.nl> wrote: Show quoted text
> Perhaps a reference to the “forks” module on CPAN could be added? >
Which does not work either <https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of thread-safety fixes in File::Glob). -- Petr
From: Elizabeth Mattijsen <liz [...] dijkmat.nl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perl5-porters [...] perl.org
Date: Wed, 28 Mar 2018 13:02:44 +0200
Download (untitled) / with headers
text/plain 1.3k
Show quoted text
> On 28 Mar 2018, at 12:36, Petr Pisar <ppisar@redhat.com> wrote: > On 2018-03-27, Elizabeth Mattijsen <liz@dijkmat.nl> wrote:
>> Perhaps a reference to the “forks” module on CPAN could be added?
> Which does not work either > <https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of > thread-safety fixes in File::Glob).
Way back when in 2002 / 2003, I had some free time on my hands and decided to see how I could use that to the advancement of Perl. The 5.8.0 release was still in the works, and it had some fine new features. One of them being ithreads. Having always been very interested in executing stuff asynchronously / in parallel, I decided to cut my teeth into it, coming from having been exposed to pthreads in C. This resulted in quite a number of Thread:: modules on CPAN [1]. When I finally found out *how* Perl ithreads were implemented, I was *very* disappointed. Having become considered a bit of a “ithreads” wizard on PerlMonks at the time, I basically explained the situation *almost 15 years ago* [2] Apart from bug fixes, I don’t think the situation since then has changed significantly. So, back to the documentation issue. Perhaps the “Performance considerations” paragraph from that post could serve as inspiration. Liz [1] https://metacpan.org/author/ELIZABETH [2] http://www.perlmonks.org/?node_id=288022
Download (untitled) / with headers
text/plain 1.6k
Talking about PerlMonks, try reading some of the posts from BrowserUk, the current threads "wizard" there. http://www.perlmonks.org/?node_id=3989;BIT=threads;a=BrowserUk On Wed, 28 Mar 2018 04:04:44 -0700, elizabeth wrote: Show quoted text
> > On 28 Mar 2018, at 12:36, Petr Pisar <ppisar@redhat.com> wrote: > > On 2018-03-27, Elizabeth Mattijsen <liz@dijkmat.nl> wrote:
> >> Perhaps a reference to the “forks” module on CPAN could be added?
> > Which does not work either > > <https://rt.cpan.org/Public/Bug/Display.html?id=123248> (because of > > thread-safety fixes in File::Glob).
> > Way back when in 2002 / 2003, I had some free time on my hands and > decided to see how I could use that to the advancement of Perl. The > 5.8.0 release was still in the works, and it had some fine new > features. One of them being ithreads. > > Having always been very interested in executing stuff asynchronously / > in parallel, I decided to cut my teeth into it, coming from having > been exposed to pthreads in C. > > This resulted in quite a number of Thread:: modules on CPAN [1]. When > I finally found out *how* Perl ithreads were implemented, I was *very* > disappointed. > > Having become considered a bit of a “ithreads” wizard on PerlMonks at > the time, I basically explained the situation *almost 15 years ago* > [2] > > Apart from bug fixes, I don’t think the situation since then has > changed significantly. > > So, back to the documentation issue. Perhaps the “Performance > considerations” paragraph from that post could serve as inspiration. > > > Liz > > > [1] https://metacpan.org/author/ELIZABETH > [2] http://www.perlmonks.org/?node_id=288022
Date: Thu, 29 Mar 2018 22:46:40 +0200
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
From: Aristotle Pagaltzis <pagaltzis [...] gmx.de>
* E. Choroba <perlbug-followup@perl.org> [2018-03-25 21:31]: Show quoted text
> See also #125106, which seems to lead nowhere, maybe because it was > too ambitious. This patch provides just the minimal change.
The minimal change would be to simply remove the sentence, and that is what I would favour to begin with. * Christian Walde <walde.christian@gmail.com> [2018-03-26 12:49]: Show quoted text
> Since this change doesn't have a ticket attached to it, i figured it > would be useful to provide the context of what led to its creation.
Thank you. I vaguely remembered having been a part of that discussion and am glad to have the context back; I am also relieved to see that my current position is close to my previous stance. Show quoted text
> This discouragement was introduced by rjbs in commit > 10a4597703f4b044d3f968bf3923644ea2387f5c. I've extracted some of the > chat log that led to it.
I regret not catching this at the time, but the reason given for calling the feature discouraged does not meet the technical definition of the term in perlpolicy. So the statement in the threads.pm POD is simply not true. I must admit my own anti-ithreads bias here, and that this is probably what led me to skip over that issue. I am not unhappy to see threads discouraged-in-the-colloquial-sense and have always avoided them myself (with success). However, for some people they are the only option. And they are not ever being removed from Perl, even as a far-future possibility. They are thus not discouraged by the definition of that term in perlpolicy. However contentious everything else may be, that much is unambiguously true. * Jerry D. Hedden via RT <perlbug-comment@perl.org> [2018-03-27 02:59]: Show quoted text
> As one of several people who have maintained the various threads > modules over the years, I regret that I missed being part of the > original discussion that lead to the inclusion of the 'discouraged' > message.
Ditto. Show quoted text
> The fact is that threads work, they are maintained, and they currently > do not have any bugs preventing their use.
Ditto. Show quoted text
> I acknowledge that not all Perl modules are thread-safe, but there is > sufficient documentation to that affect in the POD.
Ditto. Show quoted text
> I also acknowledge that the threads implementation is not ideal nor > optimal. Nonetheless, threads are useful, and are being used in the > wild. (I, for one, have even used them to good effect in production > code. <Gasp!>) Yes, if you don't know what you're doing, threads can > be problematic. ("There be dragons...," and all that.) However, the > same can be argued to greater or lesser degrees of any programming > language feature in the hands of unsophisticated users.
Here we somewhat depart. To me the main point is that some people some of the time have no alternative to ithreads. And so if you need to use them, well then you need to use them. Show quoted text
> I feel that, while the wording of the POD notice is reasonable, the > WARNING heading is alarmist. I feel strongly that the heading should > be changed to NOTICE, and the word 'discouraged' be changed to 'not > recommended' (as per the original poster's patch).
The discouragement notice just belongs deleted. I’m not sure that the heading would be too strong without it so I’m -0 on changing that. Show quoted text
> Since the addition of the 'discouraged' message, I have received > several emails from professional Perl developers from around the world > expressing concern about it. I expressed to them the same opinions > I have given above, namely that threads work (but you have to know > what you're doing), and that threads have not been deprecated.
+1 Show quoted text
> I'm not trying to convert anyone who doesn't like "interpreter-based > threads". Don't use them as you so choose. After all, Perl has always > been about options. However, there ARE Perl developers that do feel > they are a VERY useful feature.
This may depend on perspective. If I were in a situation without the choice of something else, I could imagine finding them VERY useful. :-) * Leon Timmermans <fawaka@gmail.com> [2018-03-28 02:30]: Show quoted text
> I think the main problem with the current wording is that it > discourages without explaining why.
There is a long list of appropriately neutral explanation of footguns, but only miles down the page under BUGS AND LIMITATIONS. There has to be a reference to that. Show quoted text
> In particular it doesn't explain that threads.pm does something > different than what many people expect it to do. > > People expect it to be good for "share all the data" scenarios > (because pretty much anything else called threads is), even though it > is terrible at that. This confusion isn't helped by threads > implementations in commonly used implementations of languages > occupying the same niches as we do (e.g. cpython, Ruby MRI) sucking in > exactly the opposite way (they're good at sharing but useless at > actually being parallel). > > It's not so much threads.pm that needs a big fat warnings, it's > threads::shared that does.
+1 I would be in favour of a second patch which adds this up top, which can be bikeshedded if people feel strongly enough about it. I am willing to propose new language for that. First up I’d want to remove the untrue statement, however. * Rocco Caputo <rcaputo@pobox.com> [2018-03-27 16:35]: Show quoted text
> I started the discussion that led to the addition of the > discouragement warning. Its original purpose was to manage users' > expectations of the kind of help they'd get from the larger Perl > community for trying to use this feature. […] Getting rid of the > notice entirely would resume failing to manage users' expectations.
The notice was simply incorrect and therefore could not possibly manage users’ expectations correctly. If the fact that you’re talking about is that using threads will get you yelled at by some, then how is any user expectation managed correctly by *not* telling the user that and instead just telling them to not use threads? Show quoted text
> In case someone on-list isn't aware of the advice given off-list, I'm > closing with a small list of quotes from freenode #perl. These are > pieces of real advice given to Perl users looking for help using > threads. Some of you may recognize things you've said.
How heavy is the tail on the long tail graph of who these quotes came from? How many counterexamples of users getting real help with threads are there? Show quoted text
> Some of you may be tempted to say that IRC isn't representative of > a larger perl community, but many of the vocal opponents on IRC also > participate in other areas. They use and advocate Perl professionally. > They attend or speak at Perl conferences. They are perl5 porters.
Are you aware that what comes after your “but” doesn’t rebut what comes before it? :-) * Sergey Aleynikov via RT <perlbug-followup@perl.org> [2018-03-26 01:29]: Show quoted text
> On Sat, 24 Mar 2018 09:31:06 -0700, choroba@matfyz.cz wrote:
> > From time to time, we may mark language constructs and > > features which we consider to have been mistakes as > > discouraged.
> > Threads in their current implementation are arguable a mistake. You > loose ~30% of performance and who-measured-how-much memory by just > using perl build with threads. And most of XS CPAN is up to some > degree threads-unsafe (pure-perl CPAN is safe, so the goals for > threads is reached, but still).
Any feasible implementation would be a mistake of a different kind. So faced with the problem of needing some kind of threads but having only bad choices for their implementation, Perl made a different bad choice than other similar languages. (Cf. Leon’s reply.) And no random scary numbers pulled out of a hat please. I don’t mind mentioning that thread-supporting perls are slower at an above-noise level, but if you’re going to quote specific numbers, they must come with actual benchmarks to contextualise them. Just “30% slower!!1!” is FUD. (I myself compile my perls without threads, because if I never ever use threads then why pay for them? That is an argument that holds even in absence of any exact figures.) Show quoted text
> > Discouraged features aren't currently candidates > > for removal, but we may later deprecate them if they're found > > to stand in the way of a significant improvement to the Perl > > core.
> > Threads are not going anywhere by this definition.
Indeed. Show quoted text
> Just discouraged.
In the colloquial sense maybe, but as you found, not in the perlpolicy sense. * Sergey Aleynikov via RT <perlbug-followup@perl.org> [2018-03-28 03:04]: Show quoted text
> On Mon, 26 Mar 2018 18:21:02 -0700, public@khwilliamson.com wrote:
> > I agree with the above. And I really don't like 'not recommended' as > > I think it is too strong. Maybe just list the problems > > > > Even a 30% slowdown will be fully acceptable if you can divide the > > work up into 8 or 16 parallel pieces. The gain far outweighs the > > cost.
> > But if you use processes for parallelism and not threads, you don't > get this slowdown at all and get the same multicore boost. And you > don't have to guess which XS module will segfault next. The only case > when I find using threads reasonable is writing GUI, where you _have > to_ be in the same address space.
Except some people do not have the option. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: Aristotle Pagaltzis <pagaltzis [...] gmx.de>, perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
From: Karl Williamson <public [...] khwilliamson.com>
Date: Sun, 1 Apr 2018 09:33:42 -0600
Download (untitled) / with headers
text/plain 9.4k
On 03/29/2018 02:46 PM, Aristotle Pagaltzis wrote: Show quoted text
> * E. Choroba <perlbug-followup@perl.org> [2018-03-25 21:31]:
>> See also #125106, which seems to lead nowhere, maybe because it was >> too ambitious. This patch provides just the minimal change.
> > The minimal change would be to simply remove the sentence, and that is > what I would favour to begin with. > > > * Christian Walde <walde.christian@gmail.com> [2018-03-26 12:49]:
>> Since this change doesn't have a ticket attached to it, i figured it >> would be useful to provide the context of what led to its creation.
> > Thank you. I vaguely remembered having been a part of that discussion > and am glad to have the context back; I am also relieved to see that my > current position is close to my previous stance. >
>> This discouragement was introduced by rjbs in commit >> 10a4597703f4b044d3f968bf3923644ea2387f5c. I've extracted some of the >> chat log that led to it.
> > I regret not catching this at the time, but the reason given for calling > the feature discouraged does not meet the technical definition of the > term in perlpolicy. So the statement in the threads.pm POD is simply not > true. > > I must admit my own anti-ithreads bias here, and that this is probably > what led me to skip over that issue. I am not unhappy to see threads > discouraged-in-the-colloquial-sense and have always avoided them myself > (with success). > > However, for some people they are the only option. > > And they are not ever being removed from Perl, even as a far-future > possibility. They are thus not discouraged by the definition of that > term in perlpolicy. However contentious everything else may be, that > much is unambiguously true. > > > * Jerry D. Hedden via RT <perlbug-comment@perl.org> [2018-03-27 02:59]:
>> As one of several people who have maintained the various threads >> modules over the years, I regret that I missed being part of the >> original discussion that lead to the inclusion of the 'discouraged' >> message.
> > Ditto. >
>> The fact is that threads work, they are maintained, and they currently >> do not have any bugs preventing their use.
> > Ditto. >
>> I acknowledge that not all Perl modules are thread-safe, but there is >> sufficient documentation to that affect in the POD.
> > Ditto. >
>> I also acknowledge that the threads implementation is not ideal nor >> optimal. Nonetheless, threads are useful, and are being used in the >> wild. (I, for one, have even used them to good effect in production >> code. <Gasp!>) Yes, if you don't know what you're doing, threads can >> be problematic. ("There be dragons...," and all that.) However, the >> same can be argued to greater or lesser degrees of any programming >> language feature in the hands of unsophisticated users.
> > Here we somewhat depart. To me the main point is that some people some > of the time have no alternative to ithreads. And so if you need to use > them, well then you need to use them. >
>> I feel that, while the wording of the POD notice is reasonable, the >> WARNING heading is alarmist. I feel strongly that the heading should >> be changed to NOTICE, and the word 'discouraged' be changed to 'not >> recommended' (as per the original poster's patch).
> > The discouragement notice just belongs deleted. I’m not sure that the > heading would be too strong without it so I’m -0 on changing that. >
>> Since the addition of the 'discouraged' message, I have received >> several emails from professional Perl developers from around the world >> expressing concern about it. I expressed to them the same opinions >> I have given above, namely that threads work (but you have to know >> what you're doing), and that threads have not been deprecated.
> > +1 >
>> I'm not trying to convert anyone who doesn't like "interpreter-based >> threads". Don't use them as you so choose. After all, Perl has always >> been about options. However, there ARE Perl developers that do feel >> they are a VERY useful feature.
> > This may depend on perspective. If I were in a situation without the > choice of something else, I could imagine finding them VERY useful. :-) > > > * Leon Timmermans <fawaka@gmail.com> [2018-03-28 02:30]:
>> I think the main problem with the current wording is that it >> discourages without explaining why.
> > There is a long list of appropriately neutral explanation of footguns, > but only miles down the page under BUGS AND LIMITATIONS. There has to > be a reference to that. >
>> In particular it doesn't explain that threads.pm does something >> different than what many people expect it to do. >> >> People expect it to be good for "share all the data" scenarios >> (because pretty much anything else called threads is), even though it >> is terrible at that. This confusion isn't helped by threads >> implementations in commonly used implementations of languages >> occupying the same niches as we do (e.g. cpython, Ruby MRI) sucking in >> exactly the opposite way (they're good at sharing but useless at >> actually being parallel). >> >> It's not so much threads.pm that needs a big fat warnings, it's >> threads::shared that does.
> > +1 > > I would be in favour of a second patch which adds this up top, which can > be bikeshedded if people feel strongly enough about it. I am willing to > propose new language for that. > > First up I’d want to remove the untrue statement, however. > > > * Rocco Caputo <rcaputo@pobox.com> [2018-03-27 16:35]:
>> I started the discussion that led to the addition of the >> discouragement warning. Its original purpose was to manage users' >> expectations of the kind of help they'd get from the larger Perl >> community for trying to use this feature. […] Getting rid of the >> notice entirely would resume failing to manage users' expectations.
> > The notice was simply incorrect and therefore could not possibly manage > users’ expectations correctly. If the fact that you’re talking about is > that using threads will get you yelled at by some, then how is any user > expectation managed correctly by *not* telling the user that and instead > just telling them to not use threads? >
>> In case someone on-list isn't aware of the advice given off-list, I'm >> closing with a small list of quotes from freenode #perl. These are >> pieces of real advice given to Perl users looking for help using >> threads. Some of you may recognize things you've said.
> > How heavy is the tail on the long tail graph of who these quotes came > from? How many counterexamples of users getting real help with threads > are there? >
>> Some of you may be tempted to say that IRC isn't representative of >> a larger perl community, but many of the vocal opponents on IRC also >> participate in other areas. They use and advocate Perl professionally. >> They attend or speak at Perl conferences. They are perl5 porters.
> > Are you aware that what comes after your “but” doesn’t rebut what comes > before it? :-) > > > * Sergey Aleynikov via RT <perlbug-followup@perl.org> [2018-03-26 01:29]:
>> On Sat, 24 Mar 2018 09:31:06 -0700, choroba@matfyz.cz wrote:
>>> From time to time, we may mark language constructs and >>> features which we consider to have been mistakes as >>> discouraged.
>> >> Threads in their current implementation are arguable a mistake. You >> loose ~30% of performance and who-measured-how-much memory by just >> using perl build with threads. And most of XS CPAN is up to some >> degree threads-unsafe (pure-perl CPAN is safe, so the goals for >> threads is reached, but still).
> > Any feasible implementation would be a mistake of a different kind. So > faced with the problem of needing some kind of threads but having only > bad choices for their implementation, Perl made a different bad choice > than other similar languages. (Cf. Leon’s reply.) > > And no random scary numbers pulled out of a hat please. I don’t mind > mentioning that thread-supporting perls are slower at an above-noise > level, but if you’re going to quote specific numbers, they must come > with actual benchmarks to contextualise them. Just “30% slower!!1!” is > FUD. (I myself compile my perls without threads, because if I never ever > use threads then why pay for them? That is an argument that holds even > in absence of any exact figures.) >
>>> Discouraged features aren't currently candidates >>> for removal, but we may later deprecate them if they're found >>> to stand in the way of a significant improvement to the Perl >>> core.
>> >> Threads are not going anywhere by this definition.
> > Indeed. >
>> Just discouraged.
> > In the colloquial sense maybe, but as you found, not in the perlpolicy > sense. > > > * Sergey Aleynikov via RT <perlbug-followup@perl.org> [2018-03-28 03:04]:
>> On Mon, 26 Mar 2018 18:21:02 -0700, public@khwilliamson.com wrote:
>>> I agree with the above. And I really don't like 'not recommended' as >>> I think it is too strong. Maybe just list the problems >>> >>> Even a 30% slowdown will be fully acceptable if you can divide the >>> work up into 8 or 16 parallel pieces. The gain far outweighs the >>> cost.
>> >> But if you use processes for parallelism and not threads, you don't >> get this slowdown at all and get the same multicore boost. And you >> don't have to guess which XS module will segfault next. The only case >> when I find using threads reasonable is writing GUI, where you _have >> to_ be in the same address space.
> > Except some people do not have the option. > > > Regards, >
It turns out that #125106 also is about this topic, and I have merged the two tickets. You might want to look at the discussion there.
From: Sawyer X <xsawyerx [...] gmail.com>
Date: Thu, 5 Apr 2018 11:39:44 +0300
To: Aristotle Pagaltzis <pagaltzis [...] gmx.de>, perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
[Top-posted] I agree with pretty much every point Aristotle made here. I want to highlight some issues to move forward on: * As Aristotle explains, this definition of "discouraged" does not match what it means in perlpolicy. It might meet what we refer to in day-to-day communication, but this creates a gap between that and what our policy states it is. This is important because our technical decisions must conform to the policy. We must then remove the "discouraged" label. * There is still room on how to define this recommendation to avoid. I'm personally fond of the short-but-honest get-to-the-core descriptions, such as "This might not be what you want" or "This will likely not work the way you expect it to" or "This isn't what you would this it is" or "Unless you know how this works, you will want to stay clear of it." There are far better English speakers who could provide a nuanced phrasing and far better explanation of why this isn't what you wish, technically. * I think we definitely need to add more explanations (both on threads and threads::shared) for the difference in behavior vs. expected behavior, allowing users to jump from short-and-sweet to the details, helping them make a decision of whether they want to use threads.pm or not - and if so, how. * As Aristotle requested: We should not put numbers without benchmarks. We can say "noticeably slower" or "noticeably slower for <usages>." This is honest and far more useful, IMHO. * It would also be good to do the counter of suggesting not to use threads (or pointing at the grain of salt you have to take when wanting to use it), which is to note when it *is* useful to use threads, providing the narrow cases in which it shines or works the way you expect it to (or the way it's designed to). Did I miss anything? On 03/29/2018 11:46 PM, Aristotle Pagaltzis wrote: Show quoted text
> * E. Choroba <perlbug-followup@perl.org> [2018-03-25 21:31]:
>> See also #125106, which seems to lead nowhere, maybe because it was >> too ambitious. This patch provides just the minimal change.
> The minimal change would be to simply remove the sentence, and that is > what I would favour to begin with. > > > * Christian Walde <walde.christian@gmail.com> [2018-03-26 12:49]:
>> Since this change doesn't have a ticket attached to it, i figured it >> would be useful to provide the context of what led to its creation.
> Thank you. I vaguely remembered having been a part of that discussion > and am glad to have the context back; I am also relieved to see that my > current position is close to my previous stance. >
>> This discouragement was introduced by rjbs in commit >> 10a4597703f4b044d3f968bf3923644ea2387f5c. I've extracted some of the >> chat log that led to it.
> I regret not catching this at the time, but the reason given for calling > the feature discouraged does not meet the technical definition of the > term in perlpolicy. So the statement in the threads.pm POD is simply not > true. > > I must admit my own anti-ithreads bias here, and that this is probably > what led me to skip over that issue. I am not unhappy to see threads > discouraged-in-the-colloquial-sense and have always avoided them myself > (with success). > > However, for some people they are the only option. > > And they are not ever being removed from Perl, even as a far-future > possibility. They are thus not discouraged by the definition of that > term in perlpolicy. However contentious everything else may be, that > much is unambiguously true. > > > * Jerry D. Hedden via RT <perlbug-comment@perl.org> [2018-03-27 02:59]:
>> As one of several people who have maintained the various threads >> modules over the years, I regret that I missed being part of the >> original discussion that lead to the inclusion of the 'discouraged' >> message.
> Ditto. >
>> The fact is that threads work, they are maintained, and they currently >> do not have any bugs preventing their use.
> Ditto. >
>> I acknowledge that not all Perl modules are thread-safe, but there is >> sufficient documentation to that affect in the POD.
> Ditto. >
>> I also acknowledge that the threads implementation is not ideal nor >> optimal. Nonetheless, threads are useful, and are being used in the >> wild. (I, for one, have even used them to good effect in production >> code. <Gasp!>) Yes, if you don't know what you're doing, threads can >> be problematic. ("There be dragons...," and all that.) However, the >> same can be argued to greater or lesser degrees of any programming >> language feature in the hands of unsophisticated users.
> Here we somewhat depart. To me the main point is that some people some > of the time have no alternative to ithreads. And so if you need to use > them, well then you need to use them. >
>> I feel that, while the wording of the POD notice is reasonable, the >> WARNING heading is alarmist. I feel strongly that the heading should >> be changed to NOTICE, and the word 'discouraged' be changed to 'not >> recommended' (as per the original poster's patch).
> The discouragement notice just belongs deleted. I’m not sure that the > heading would be too strong without it so I’m -0 on changing that. >
>> Since the addition of the 'discouraged' message, I have received >> several emails from professional Perl developers from around the world >> expressing concern about it. I expressed to them the same opinions >> I have given above, namely that threads work (but you have to know >> what you're doing), and that threads have not been deprecated.
> +1 >
>> I'm not trying to convert anyone who doesn't like "interpreter-based >> threads". Don't use them as you so choose. After all, Perl has always >> been about options. However, there ARE Perl developers that do feel >> they are a VERY useful feature.
> This may depend on perspective. If I were in a situation without the > choice of something else, I could imagine finding them VERY useful. :-) > > > * Leon Timmermans <fawaka@gmail.com> [2018-03-28 02:30]:
>> I think the main problem with the current wording is that it >> discourages without explaining why.
> There is a long list of appropriately neutral explanation of footguns, > but only miles down the page under BUGS AND LIMITATIONS. There has to > be a reference to that. >
>> In particular it doesn't explain that threads.pm does something >> different than what many people expect it to do. >> >> People expect it to be good for "share all the data" scenarios >> (because pretty much anything else called threads is), even though it >> is terrible at that. This confusion isn't helped by threads >> implementations in commonly used implementations of languages >> occupying the same niches as we do (e.g. cpython, Ruby MRI) sucking in >> exactly the opposite way (they're good at sharing but useless at >> actually being parallel). >> >> It's not so much threads.pm that needs a big fat warnings, it's >> threads::shared that does.
> +1 > > I would be in favour of a second patch which adds this up top, which can > be bikeshedded if people feel strongly enough about it. I am willing to > propose new language for that. > > First up I’d want to remove the untrue statement, however. > > > * Rocco Caputo <rcaputo@pobox.com> [2018-03-27 16:35]:
>> I started the discussion that led to the addition of the >> discouragement warning. Its original purpose was to manage users' >> expectations of the kind of help they'd get from the larger Perl >> community for trying to use this feature. […] Getting rid of the >> notice entirely would resume failing to manage users' expectations.
> The notice was simply incorrect and therefore could not possibly manage > users’ expectations correctly. If the fact that you’re talking about is > that using threads will get you yelled at by some, then how is any user > expectation managed correctly by *not* telling the user that and instead > just telling them to not use threads? >
>> In case someone on-list isn't aware of the advice given off-list, I'm >> closing with a small list of quotes from freenode #perl. These are >> pieces of real advice given to Perl users looking for help using >> threads. Some of you may recognize things you've said.
> How heavy is the tail on the long tail graph of who these quotes came > from? How many counterexamples of users getting real help with threads > are there? >
>> Some of you may be tempted to say that IRC isn't representative of >> a larger perl community, but many of the vocal opponents on IRC also >> participate in other areas. They use and advocate Perl professionally. >> They attend or speak at Perl conferences. They are perl5 porters.
> Are you aware that what comes after your “but” doesn’t rebut what comes > before it? :-) > > > * Sergey Aleynikov via RT <perlbug-followup@perl.org> [2018-03-26 01:29]:
>> On Sat, 24 Mar 2018 09:31:06 -0700, choroba@matfyz.cz wrote:
>>> From time to time, we may mark language constructs and >>> features which we consider to have been mistakes as >>> discouraged.
>> Threads in their current implementation are arguable a mistake. You >> loose ~30% of performance and who-measured-how-much memory by just >> using perl build with threads. And most of XS CPAN is up to some >> degree threads-unsafe (pure-perl CPAN is safe, so the goals for >> threads is reached, but still).
> Any feasible implementation would be a mistake of a different kind. So > faced with the problem of needing some kind of threads but having only > bad choices for their implementation, Perl made a different bad choice > than other similar languages. (Cf. Leon’s reply.) > > And no random scary numbers pulled out of a hat please. I don’t mind > mentioning that thread-supporting perls are slower at an above-noise > level, but if you’re going to quote specific numbers, they must come > with actual benchmarks to contextualise them. Just “30% slower!!1!” is > FUD. (I myself compile my perls without threads, because if I never ever > use threads then why pay for them? That is an argument that holds even > in absence of any exact figures.) >
>>> Discouraged features aren't currently candidates >>> for removal, but we may later deprecate them if they're found >>> to stand in the way of a significant improvement to the Perl >>> core.
>> Threads are not going anywhere by this definition.
> Indeed. >
>> Just discouraged.
> In the colloquial sense maybe, but as you found, not in the perlpolicy > sense. > > > * Sergey Aleynikov via RT <perlbug-followup@perl.org> [2018-03-28 03:04]:
>> On Mon, 26 Mar 2018 18:21:02 -0700, public@khwilliamson.com wrote:
>>> I agree with the above. And I really don't like 'not recommended' as >>> I think it is too strong. Maybe just list the problems >>> >>> Even a 30% slowdown will be fully acceptable if you can divide the >>> work up into 8 or 16 parallel pieces. The gain far outweighs the >>> cost.
>> But if you use processes for parallelism and not threads, you don't >> get this slowdown at all and get the same multicore boost. And you >> don't have to guess which XS module will segfault next. The only case >> when I find using threads reasonable is writing GUI, where you _have >> to_ be in the same address space.
> Except some people do not have the option. > > > Regards,
CC: Aristotle Pagaltzis <pagaltzis [...] gmx.de>, perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
Date: Thu, 5 Apr 2018 09:33:08 -0400
From: Rocco Caputo <rcaputo [...] pobox.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: Sawyer X <xsawyerx [...] gmail.com>
Download (untitled) / with headers
text/plain 622b
On Apr 5, 2018, at 04:39, Sawyer X <xsawyerx@gmail.com> wrote:
Show quoted text

Did I miss anything?

I think so.  The only part of the existing paragraph that tries to warn the reader about the difference between "supported" and "you will receive support":

"Few people ... will be able to provide help."

Even that is an understatement.  Maybe it should say "willing or able".

In the context of the discussions leading up to the change, telling people that they won't get much help is the most important part of that paragraph.  The rest of it merely attempts to justify why it has to be said.

-- 
Rocco Caputo <rcaputo@pobox.com>
From: Sawyer X <xsawyerx [...] gmail.com>
Date: Fri, 6 Apr 2018 22:41:20 +0300
CC: Aristotle Pagaltzis <pagaltzis [...] gmx.de>, perl5-porters [...] perl.org, bugs-bitbucket [...] rt.perl.org
To: Rocco Caputo <rcaputo [...] pobox.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 1.2k
On 04/05/2018 04:33 PM, Rocco Caputo wrote: Show quoted text
> On Apr 5, 2018, at 04:39, Sawyer X <xsawyerx@gmail.com > <mailto:xsawyerx@gmail.com>> wrote:
>> >> Did I miss anything?
> > I think so.  The only part of the existing paragraph that tries to > warn the reader about the difference between "supported" and "you will > receive support": > > "Few people ... will be able to provide help." > > Even that is an understatement.  Maybe it should say "willing or able". > > In the context of the discussions leading up to the change, telling > people that they won't get much help is the most important part of > that paragraph.  The rest of it merely attempts to justify why it has > to be said.
You're right. That is a good point. I was thinking about this "discouraged" thing more and thought maybe a phrasing of, "Unless you're [intimately] familiar with how this module achieves threaded behavior, it is ill-advised to use it." or something that suggests "You should probably understand this if you want to use it." We could add that without knowledge, it can be unwieldy or something. Jerry, if you could also help us pin down a description that is not completely alarmist on the other, but is honest about the inherent risks this adds to anyone who does not understand how it works - that would be great.
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: Sawyer X <xsawyerx [...] gmail.com>
CC: Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
From: Christian Walde <walde.christian [...] gmail.com>
Date: Sun, 08 Apr 2018 22:23:20 +0000
Download (untitled) / with headers
text/plain 952b
There's one thing you're missing and why i think it's bad to call it alarmist. It is a simple matter if fact that even if theoretically the threads implementation is bug free and the foot guns documented, actually using threads often surfaces foot guns as heisenbugs:

- perl threads can work fine under a certain load, but crash randomly once the load becomes too big
- perl threads can work fine on one computer, but crash randomly on another due to slight module differences

This leads to a situation where the most accurate way to describe threads is:

Threads are fine to use in production only once the people working with them know and understand exactly and perfectly how they are implemented, what the drawbacks are, and how to handle problems arising from cpan modules that were not written by such people or have mistakes regarding threads in them.

On Thu, Apr 5, 2018, 10:40 Sawyer X <xsawyerx@gmail.com> wrote:
Show quoted text
Did I miss anything?
Date: Mon, 9 Apr 2018 08:51:15 +0100
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Sawyer X <xsawyerx [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
To: Christian Walde <walde.christian [...] gmail.com>
Download (untitled) / with headers
text/plain 642b
On Sun, Apr 08, 2018 at 10:23:20PM +0000, Christian Walde wrote: Show quoted text
> - perl threads can work fine under a certain load, but crash randomly once > the load becomes too big > - perl threads can work fine on one computer, but crash randomly on another > due to slight module differences
Is this based on the idea that the ithreads implementation itself is inherently buggy, or that some modules may not be thread-safe. I'd be surprised if the former, and if the latter, surely that applies to any "real" threading implementation in any programming language? -- Modern art: "That's easy, I could have done that!" "Ah, but you didn't!"
To: Dave Mitchell <davem [...] iabyn.com>, Christian Walde <walde.christian [...] gmail.com>
Date: Mon, 9 Apr 2018 12:18:10 +0300
CC: Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Sawyer X <xsawyerx [...] gmail.com>
Download (untitled) / with headers
text/plain 656b
On 04/09/2018 10:51 AM, Dave Mitchell wrote: Show quoted text
> On Sun, Apr 08, 2018 at 10:23:20PM +0000, Christian Walde wrote:
>> - perl threads can work fine under a certain load, but crash randomly once >> the load becomes too big >> - perl threads can work fine on one computer, but crash randomly on another >> due to slight module differences
> Is this based on the idea that the ithreads implementation itself is > inherently buggy, or that some modules may not be thread-safe. > > I'd be surprised if the former, and if the latter, surely that applies > to any "real" threading implementation in any programming language?
Could you please expand on this, Dave?
To: Sawyer X <xsawyerx [...] gmail.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Christian Walde <walde.christian [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Date: Mon, 9 Apr 2018 11:37:54 +0100
Download (untitled) / with headers
text/plain 1.8k
On Mon, Apr 09, 2018 at 12:18:10PM +0300, Sawyer X wrote: Show quoted text
> On 04/09/2018 10:51 AM, Dave Mitchell wrote:
> > On Sun, Apr 08, 2018 at 10:23:20PM +0000, Christian Walde wrote:
> >> - perl threads can work fine under a certain load, but crash randomly once > >> the load becomes too big > >> - perl threads can work fine on one computer, but crash randomly on another > >> due to slight module differences
> > Is this based on the idea that the ithreads implementation itself is > > inherently buggy, or that some modules may not be thread-safe. > > > > I'd be surprised if the former, and if the latter, surely that applies > > to any "real" threading implementation in any programming language?
> > Could you please expand on this, Dave?
1) I think the perl ithreads implementation is fairly bug-free and stable these days. 2) If one's threaded code randomly craps out due to a 3rd party module being non-thread-safe, or supposedly thread-safe but buggy, then that's an issue with the third-party module. In principle any programming language will struggle with threads if used with a 3rd-party library that isn't thread-safe. Perl's threads are supposedly safe at the perl language level. For example, if two threads simultaneously try to push a value to the same array, then a) the internal representation of the array (AV etc) won't be corrupted; b) both elements will get pushed; c) perl won't guarantee which order they're pushed in. This is a lot better guarantee than threads in C for example. The main danger area is with XS modules; pure perl modules should at least not corrupt the interpreter state; whether they work correctly (e.g. two threads both trying to update the same data file) is down to whether the module's author does the correct locking etc. -- No matter how many dust sheets you use, you will get paint on the carpet.
From: Elizabeth Mattijsen <liz [...] dijkmat.nl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Date: Mon, 9 Apr 2018 14:34:21 +0200
To: perl5-porters [...] perl.org
Show quoted text
> On 9 Apr 2018, at 12:37, Dave Mitchell <davem@iabyn.com> wrote: > On Mon, Apr 09, 2018 at 12:18:10PM +0300, Sawyer X wrote:
>> Could you please expand on this, Dave?
> Perl's threads are supposedly safe at the perl language level. For > example, if two threads simultaneously try to push a value to the same > array, then > a) the internal representation of the array (AV etc) won't be > corrupted; > b) both elements will get pushed;
Is that something that was fixed in the past 10 years or so? I distinctly remember this only being true for *shared* arrays, which involve a lot of overhead. OOC, is this also true for hashes to which keys are being added by several threads? Also OOC, if multiple threads increase / decrease the reference count of something, is that also threadsafe by default? AKA, will you never get double frees (increment missed) or DESTROY never getting called (decrement missed)? FWIW, I was discussing these things the other day at the German Perl Workshop, so this caught my eye having thought about these things for the first time in a long while. Liz
Date: Mon, 9 Apr 2018 16:00:12 +0100
Subject: Fwd: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Elizabeth Mattijsen <liz [...] dijkmat.nl>
From: Dave Mitchell <davem [...] iabyn.com>
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.4k
(missed out Ccing p5p) ----- Forwarded message from Dave Mitchell <davem@iabyn.com> ----- Date: Mon, 9 Apr 2018 15:59:14 +0100 From: Dave Mitchell <davem@iabyn.com> To: Elizabeth Mattijsen <liz@dijkmat.nl> Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation Message-ID: <20180409145914.GN14242@iabyn.com> On Mon, Apr 09, 2018 at 02:34:21PM +0200, Elizabeth Mattijsen wrote: Show quoted text
> Is that something that was fixed in the past 10 years or so? I > distinctly remember this only being true for *shared* arrays, which > involve a lot of overhead.
Well, non-shared arrays can only be accessed by a single thread, so they are not an issue. Running the following code, which has 100 threads pushing and popping things off a shared array shows that it ends with the same number of elements as when it started. use threads; use threads::shared; my @a : shared = (1..100); sub inc { for (1..10_000) { push @a, $_; pop @a; } } my @t; push @t, threads->new(\&inc) for 1..100; $_->join for @t; printf "size=%d\n", scalar(@a); I've run this code on 5.10.1 and 5.27.10 and in both cases it finishes with 100 elements. Show quoted text
> OOC, is this also true for hashes to which keys are being added by > several threads?
(I don't know what OOC stands for). If the different threads are using disjoint sets of keys when accessing the same shared hash, they won't interfere with each other. If multiple threads are adding, modifying and deleting the same key in a shared hash, then of course the final result for that key in the hash will depend on the timing; but it won't be corrupt. Show quoted text
> Also OOC, if multiple threads increase / decrease the reference count of > something, is that also threadsafe by default? AKA, will you never get > double frees (increment missed) or DESTROY never getting called > (decrement missed)?
The whole point of ithreads is that each thread gets its own interpreter, with its own copy of every SV. So it's not possible for multiple threads to access, let alone modify, the reference count of an SV. threads::shared uses locking behind the scenes to mediate access to shared variables, so again, ref-counting isn't an issue. -- "You're so sadly neglected, and often ignored. A poor second to Belgium, When going abroad." -- Monty Python, "Finland" Show quoted text
----- End forwarded message ----- -- Please note that ash-trays are provided for the use of smokers, whereas the floor is provided for the use of all patrons. -- Bill Royston
Date: Mon, 9 Apr 2018 18:05:06 +0200
From: Elizabeth Mattijsen <liz [...] dijkmat.nl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 758b
Show quoted text
> On 9 Apr 2018, at 17:00, Dave Mitchell <davem@iabyn.com> wrote: > On Mon, Apr 09, 2018 at 02:34:21PM +0200, Elizabeth Mattijsen wrote:
>> Is that something that was fixed in the past 10 years or so? I >> distinctly remember this only being true for *shared* arrays, which >> involve a lot of overhead.
> Well, non-shared arrays can only be accessed by a single thread, so they > are not an issue.
Ah, ok, so you’re not talking OS-threads, but perl ithreads. My point was about Perl code running in separate OS-threads, what I understand (perhaps wrongly) with things like Future. So inside of a single interpreter. I’m quite aware how Perl ithreads work, for the past 15 years or so at least: http://www.perlmonks.org/?node_id=288022 Liz
Date: Mon, 9 Apr 2018 13:00:30 -0400
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Perl5 Porters <perl5-porters [...] perl.org>
From: Dan Book <grinnz [...] gmail.com>
To: Elizabeth Mattijsen <liz [...] dijkmat.nl>
Download (untitled) / with headers
text/plain 1.1k
On Mon, Apr 9, 2018 at 12:05 PM, Elizabeth Mattijsen <liz@dijkmat.nl> wrote:
Show quoted text

> On 9 Apr 2018, at 17:00, Dave Mitchell <davem@iabyn.com> wrote:
> On Mon, Apr 09, 2018 at 02:34:21PM +0200, Elizabeth Mattijsen wrote:
>> Is that something that was fixed in the past 10 years or so?  I
>> distinctly remember this only being true for *shared* arrays, which
>> involve a lot of overhead.
> Well, non-shared arrays can only be accessed by a single thread, so they
> are not an issue.

Ah, ok, so you’re not talking OS-threads, but perl ithreads.

My point was about Perl code running in separate OS-threads, what I understand (perhaps wrongly) with things like Future.  So inside of a single interpreter.


I’m quite aware how Perl ithreads work, for the past 15 years or so at least:

    http://www.perlmonks.org/?node_id=288022


Liz

Future has nothing to do with threading of any sort (though it could be leveraged in combination, such as what IO::Async::Function does on Windows). It's a mechanism for cooperative multitasking. I'm not sure what OS threads you are talking about that would run "inside of a single interpreter." pthreads run outside the interpreter.

-Dan
From: Elizabeth Mattijsen <liz [...] dijkmat.nl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Date: Mon, 9 Apr 2018 19:26:30 +0200
To: Perl5 Porters <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 911b
Show quoted text
> On 9 Apr 2018, at 19:00, Dan Book <grinnz@gmail.com> wrote: > On Mon, Apr 9, 2018 at 12:05 PM, Elizabeth Mattijsen <liz@dijkmat.nl> wrote:
> > My point was about Perl code running in separate OS-threads, what I understand (perhaps > > wrongly) with things like Future. So inside of a single interpreter.
> Future has nothing to do with threading of any sort (though it could be leveraged in combination, such as what IO::Async::Function does on Windows). It's a mechanism for cooperative multitasking. I'm not sure what OS threads you are talking about that would run "inside of a single interpreter." pthreads run outside the interpreter.
Ok, so if I understand you correctly: if you run some pthreads outside of a Perl interpreter thread, you will never be able to run any Perl code in that pthreads thread that could potentially access variables from a Perl interpreter running in another thread? Liz
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 178b
On Mon, 09 Apr 2018 08:02:10 -0700, davem wrote: Show quoted text
> (I don't know what OOC stands for).
Probably ‘out of curiosity’. Or maybe it’s a Dutch word. -- Father Chrysostomos
From: "Christian Walde" <walde.christian [...] gmail.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: "Aristotle Pagaltzis" <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
Date: Mon, 09 Apr 2018 23:01:22 +0200
To: "Sawyer X" <xsawyerx [...] gmail.com>, "Dave Mitchell" <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 1.5k
On Mon, 09 Apr 2018 12:37:54 +0200, Dave Mitchell <davem@iabyn.com> wrote: Show quoted text
> Is this based on the idea that the ithreads implementation itself is > inherently buggy, or that some modules may not be thread-safe.
My current assumption is the latter. However i cannot speculate at all as to whether it might be some of column A or not. Show quoted text
> 2) If one's threaded code randomly craps out due to a 3rd party module > being non-thread-safe, or supposedly thread-safe but buggy, then that's an > issue with the third-party module. In principle any programming language > will struggle with threads if used with a 3rd-party library that isn't > thread-safe.
This has dangerous assumptions in it: - Thread safety is not a binary on/off switch. Modules can appear to be thread-safe under some conditions and not be so under others. That's why i called it a heisenbug in my previous email. - Putting the blame on 3rd party modules is fine for most languages where it's easier to switch to other implementations or roll your own. However this is Perl. The language lives on CPAN, not in the interpreter. Asking someone who just spent hours implementing a parallel web downloader to gradually replace any of the bits they used will not be taken well. - Very few cpan maintainers know how to do thread-safety. Heck, most don't even practice string/binary safety for filehandles. Most of them are Perl developers first and you're asking them to move at least 2 degrees of separation outside of their zone of comfort. For a lot of modules there's simply a good chance this is not going to happen. -- With regards, Christian Walde
Date: Tue, 10 Apr 2018 11:50:46 +0100
To: Elizabeth Mattijsen <liz [...] dijkmat.nl>
From: Dave Mitchell <davem [...] iabyn.com>
CC: Perl5 Porters <perl5-porters [...] perl.org>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 1.1k
On Mon, Apr 09, 2018 at 07:26:30PM +0200, Elizabeth Mattijsen wrote: Show quoted text
> Ok, so if I understand you correctly: if you run some pthreads outside > of a Perl interpreter thread, you will never be able to run any Perl > code in that pthreads thread that could potentially access variables > from a Perl interpreter running in another thread?
I'm not sure I understand that sentence, but the basic rule of doing any sort of threading in perl is that two OS-level threads must not simultaneously access/modify the same perl interpreter struct (and all the data hanging off it, e.g. SVs etc). This was why the 5005THREADS model was abandoned. Nobody could think of a way of avoiding all the side-effects that any trivial usage of an SV might give rise to (e.g. changing the ref count, converting from an IV to PVIV), without having a lock in about every second line of code in the perl core. I vaguely understand that the approach in Python is to have one big central lock so only one thread can execute at any one time. -- "I used to be with it, but then they changed what ‘it’ was, and now what I’m with isn’t it. And what’s ‘it’ seems weird and scary to me." -- Grandpa Simpson (It will happen to you too.)
Date: Tue, 10 Apr 2018 12:03:45 +0100
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Sawyer X <xsawyerx [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
To: Christian Walde <walde.christian [...] gmail.com>
Download (untitled) / with headers
text/plain 2.6k
On Mon, Apr 09, 2018 at 11:01:22PM +0200, Christian Walde wrote: Show quoted text
> > 2) If one's threaded code randomly craps out due to a 3rd party module > > being non-thread-safe, or supposedly thread-safe but buggy, then that's an > > issue with the third-party module. In principle any programming language > > will struggle with threads if used with a 3rd-party library that isn't > > thread-safe.
> > This has dangerous assumptions in it: > > - Thread safety is not a binary on/off switch. Modules can appear to be > thread-safe under some conditions and not be so under others. That's why > i called it a heisenbug in my previous email. > > - Putting the blame on 3rd party modules is fine for most languages > where it's easier to switch to other implementations or roll your own. > However this is Perl. The language lives on CPAN, not in the > interpreter. Asking someone who just spent hours implementing a parallel > web downloader to gradually replace any of the bits they used will not > be taken well. > > - Very few cpan maintainers know how to do thread-safety. Heck, most > don't even practice string/binary safety for filehandles. Most of them > are Perl developers first and you're asking them to move at least 2 > degrees of separation outside of their zone of comfort. For a lot of > modules there's simply a good chance this is not going to happen.
But everything you've said above applies just as much to any 3rd-party library/module in any programming language which supports concurrency. I really fail to see how CPAN is different in this regard. Most CPAN authors don't need to worry about thread-safety, as long as it's a perl-level module. Perl is thread-safe at the language level, and you don't need to do anything special to make Perl code thread-safe. Or at least, nothing more than you would have to do to make your module safe across forks (which is what ithreads is emulating after all). For example, if a thread is created, the module's variables will be cloned, and a destructor might be called for a variable in each of the threads. And of course if the module is updating something external like a file, then it may need to exclusively lock that file. XS code is of course a lot harder, because now you're doing concurrency programming in C. But even here, its easier than general C concurrency, because of perl's ITHREADS "not shared by default" model. You mainly have to avoid global/static mutable data, and be prepared for any of your data stored in SVs (e.g. DB handles) being cloned when a new thread is created. -- All wight. I will give you one more chance. This time, I want to hear no Wubens. No Weginalds. No Wudolf the wed-nosed weindeers. -- Life of Brian
CC: bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Tom Molesworth <tom [...] binary.com>
Date: Tue, 10 Apr 2018 20:40:45 +0800
To: Christian Walde <walde.christian [...] gmail.com>, Perl5 Porters <perl5-porters [...] perl.org>
Download (untitled) / with headers
text/plain 3.7k
On 10 April 2018 at 05:01, Christian Walde <walde.christian@gmail.com> wrote:
Show quoted text
On Mon, 09 Apr 2018 12:37:54 +0200, Dave Mitchell <davem@iabyn.com> wrote:

Is this based on the idea that the ithreads implementation itself is
inherently buggy, or that some modules may not be thread-safe.

My current assumption is the latter. However i cannot speculate at all as to whether it might be some of column A or not.

The official threads.pm documentation claims that the current code has what I'd class as "bugs":

        Even with the latest version of Perl, it is known that certain
        constructs with threads may result in warning messages concerning
        leaked scalars or unreferenced scalars. However, such warnings are
        harmless, and may safely be ignored.

If they are leaking, this has bad implications for long-running code.
If they're not leaking, why the warning?
If this information is out of date, then the bugs are in the documentation!
 
Show quoted text

2) If one's threaded code randomly craps out due to a 3rd party module
being non-thread-safe, or supposedly thread-safe but buggy, then that's an
issue with the third-party module. In principle any programming language
will struggle with threads if used with a 3rd-party library that isn't
thread-safe.

This has dangerous assumptions in it:

- Thread safety is not a binary on/off switch. Modules can appear to be thread-safe under some conditions and not be so under others. That's why i called it a heisenbug in my previous email.

Indeed, this is arguably worse than clear "the code crashed so it doesn't work" cases.

We also don't have good documentation on how to make modules thread-safe: I've encountered quite a few Perl developers who are confident that thread safety is only an issue with XS.

As an example: far as I recall, the threads.pm documention and perlmod page to which they link never explicitly state that ref addresses will be different in each thread:

    perl -e'use threads; use Scalar::Util qw(refaddr); my %x; $x{refaddr $_} = $_ for [qw(x)], [qw(y)]; threads->create(sub { warn threads->tid . " => " . join ", ", map { "$_ (" . refaddr($x{$_}) . ")" } sort keys %x })->join for 1..2'
    1 => 30822192 (32355200), 30948440 (32355272) at -e line 1.
    2 => 30822192 (32355456), 30948440 (32355528) at -e line 1.

I know much of my CPAN code will be susceptible to this type of issue. Any time someone asks about thread safety, I point to the "discouraged" line in the official documentation and explain that I don't expect my code to work when multiple threads are active - "maybe try again if we have a new threads implementation in the future".

Downgrading that warning to something less clear could have unfortunate consequences for people learning (or supporting those who learn) Perl. I'm also not entirely convinced that "discouraged" is the wrong word - taking the phrases from the official policy:

Show quoted text
From time to time, we may mark language constructs and features which we consider to have been mistakes as discouraged.

- Windows support: not a mistake. Exposing the API to end users without warnings and clear information about how threads differs from other languages, on the other hand - maybe not ideal.

Show quoted text
Discouraged features aren't currently candidates for removal, but we may later deprecate them if they're found to stand in the way of a significant improvement to the Perl core.

- surely this is the case? If we come up with a better way to implement threads while retaining the ability for Windows to support some form of process emulation and related features, wouldn't this be something that's welcomed? I don't get the impression that there's widespread satisfaction with the current state of affairs on either threads.pm or threads::shared.


To: "Dave Mitchell" <davem [...] iabyn.com>
Date: Tue, 10 Apr 2018 14:59:50 +0200
From: "Christian Walde" <walde.christian [...] gmail.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: "Sawyer X" <xsawyerx [...] gmail.com>, "Aristotle Pagaltzis" <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 569b
On Tue, 10 Apr 2018 13:03:45 +0200, Dave Mitchell <davem@iabyn.com> wrote: Show quoted text
> I really fail to see how CPAN is different in this regard.
Because this is Perl where not using CPAN is not an option and CPAN library consumers rely on CPAN libraries being almost entirely rock-solid, or at least the author having easy and quick ways to fix bugs. If you really look at all 3 points i made, in aggregate, and don't see how this is a problem and a danger, then i don't think i can come up with other word combinations to make you see it. -- With regards, Christian Walde
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.1k
On Tue, 10 Apr 2018 04:04:06 -0700, davem wrote: The problem is in people's expectations. When most part of something is thread-safe, they'll just assume "everything is thread-safe" and then'll be hit very hard. One of the examples I like is Mouse, which was thread-unsafe until the year 2015. While it's not as popular as Moose, it's still a module with large a large user base. And your chances to encounter bugs in not so popular XS modules are much, much higher. What's even worse, you're likely to hit them only under some unlucky circumstances (see example below). As for the core, I agree that it's now much more thread-stable than before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no threads" with some nice crash examples in it - they crash no more. But still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just because of no one has discovered that. And the following still fails loudly (though not dumps a core); while (1) { push @foo, threads->create(sub { require IO::Handle; }); $_->detach for(splice @foo); } So while yes, perl is much more thread-safe inherently as it used to be, I won't recommend using threads in it to anyone.
CC: Sawyer X <xsawyerx [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Dave Mitchell <davem [...] iabyn.com>
To: Christian Walde <walde.christian [...] gmail.com>
Date: Tue, 10 Apr 2018 14:59:53 +0100
Download (untitled) / with headers
text/plain 1.4k
On Tue, Apr 10, 2018 at 02:59:50PM +0200, Christian Walde wrote: Show quoted text
> On Tue, 10 Apr 2018 13:03:45 +0200, Dave Mitchell <davem@iabyn.com> wrote: >
> > I really fail to see how CPAN is different in this regard.
> > Because this is Perl where not using CPAN is not an option and CPAN > library consumers rely on CPAN libraries being almost entirely > rock-solid, or at least the author having easy and quick ways to fix > bugs.
You still haven't differentiated perl+CPAN from some_other_language + 3rd_party_libraries_needed_to_get_the_job_done. If using a programming language other than perl, then it is likely that: 1) not using 3rd-party libraries is not an option; 2) consumers of 3rd-party libraries rely on those libraries being almost entirely rock-solid, or at least the author having easy and quick ways to fix bugs. Show quoted text
> If you really look at all 3 points i made, in aggregate, and don't see > how this is a problem and a danger, then i don't think i can come up > with other word combinations to make you see it.
Can you come up with a hypothetical scenario, e.g. a multi-threaded program that uses libraries to connect to a database and retrieve and parse some XML data. Then got through it step by step so that I can see why using perl and CPAN is dangerous, but using (e.g.) java and a DB and XML library is safe? What is the crucial difference between the two that flips it from being safe to unsafe? -- If life gives you lemons, you'll probably develop a citric acid allergy.
To: Dave Mitchell <davem [...] iabyn.com>
CC: Christian Walde <walde.christian [...] gmail.com>, Sawyer X <xsawyerx [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, Perl5 Porters <perl5-porters [...] perl.org>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Tom Molesworth <tom [...] binary.com>
Date: Tue, 10 Apr 2018 22:26:21 +0800
Download (untitled) / with headers
text/plain 1.6k
On 10 April 2018 at 21:59, Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text
On Tue, Apr 10, 2018 at 02:59:50PM +0200, Christian Walde wrote:
> If you really look at all 3 points i made, in aggregate, and don't see
> how this is a problem and a danger, then i don't think i can come up
> with other word combinations to make you see it.

Can you come up with a hypothetical scenario, e.g.  a multi-threaded
program that uses libraries to connect to a database and retrieve and
parse some XML data. Then got through it step by step so that I can see
why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
XML library is safe? What is the crucial difference between the two that
flips it from being safe to unsafe?

Here's one trivial hacked-together example of code which I would argue leads to "unexpected" results, at least from the perspective of a C or Java programmer experienced in the ways of threads:

#!/usr/bin/perl 
use strict;
use warnings;

use threads;
use threads::shared;

my %items : shared;
my $lockvar : shared;

my $t = threads->create(sub {
    lock $lockvar;
    cond_wait($lockvar);
    my @data : shared = qw(example content here);
    $items{0 + \@data} = \@data;
});
my $seen_item;
while(1) {
    do {
        lock $lockvar;
        cond_broadcast($lockvar);
    } until keys %items;
    print "Item with address " . $_ . " actually refers to " . (0 + $items{0 + $_}) . "\n" for keys %items;
    last if defined $seen_item and $seen_item ne (values %items)[0];
    ($seen_item) = values %items;
}
$t->join;

=pod

Sample output (5.26.1):

    Item with address 31700632 actually refers to 31360376
    Item with address 31700632 actually refers to 30235168

=cut

Date: Tue, 10 Apr 2018 15:27:45 +0100
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Christian Walde <walde.christian [...] gmail.com>, Perl5 Porters <perl5-porters [...] perl.org>, bugs-bitbucket [...] rt.perl.org
To: Tom Molesworth <tom [...] binary.com>
Download (untitled) / with headers
text/plain 4.2k
On Tue, Apr 10, 2018 at 08:40:45PM +0800, Tom Molesworth via perl5-porters wrote: Show quoted text
> The official threads.pm documentation claims that the current code has what > I'd class as "bugs": > > Even with the latest version of Perl, it is known that certain > constructs with threads may result in warning messages concerning > leaked scalars or unreferenced scalars. However, such warnings are > harmless, and may safely be ignored. > > If they are leaking, this has bad implications for long-running code. > If they're not leaking, why the warning? > If this information is out of date, then the bugs are in the documentation!
That text is 11 years old and should be removed. Show quoted text
> > This has dangerous assumptions in it: > > > > - Thread safety is not a binary on/off switch. Modules can appear to be > > thread-safe under some conditions and not be so under others. That's why i > > called it a heisenbug in my previous email. > >
> > Indeed, this is arguably worse than clear "the code crashed so it doesn't > work" cases.
Any threaded program in any programming language is susceptible to subtle heisenbugs. I'd argue that perl is less susceptible than many languages due to its 'not shared be default' nature. Show quoted text
> We also don't have good documentation on how to make modules thread-safe: > I've encountered quite a few Perl developers who are confident that thread > safety is only an issue with XS.
I'd be one of those, in the sense that the perl language itself is thread-safe, and doesn't normally need locks or special handling; but in any environment where there are multiple concurrently running threads, there will be some extra considerations required. Show quoted text
> As an example: far as I recall, the threads.pm documention and perlmod page > to which they link never explicitly state that ref addresses will be > different in each thread:
Since the whole basic principle of ithreads is that data isn't shared by default, it would be astonishing if two (non-shared) refs in two different threads had the same address. Show quoted text
> I know much of my CPAN code will be susceptible to this type of issue.
Do you think there are many such issues, or is ref addresses the main one? Show quoted text
> Any > time someone asks about thread safety, I point to the "discouraged" line in > the official documentation and explain that I don't expect my code to work > when multiple threads are active - "maybe try again if we have a new > threads implementation in the future".
Fine - add to the pod in your modules that they're not thread-safe. Show quoted text
> - Windows support: not a mistake.
Yet I suspect many complaints about bad threading behaviour in perl (such as ref addresses changing in a sub-thread) applies just as much to the Windows fork emulation - e.g. do a pseudo-fork on Windows and all the ref addresses change. I don't see you we can praise one and condemn the other. Show quoted text
> Exposing the API to end users without > warnings and clear information about how threads differs from other > languages, on the other hand - maybe not ideal.
I am all in favour of having, at the same location as the current 'discouraged' text, but instead of it, a big flashing neon sign saying that perl threads are a bit different from what you might expect and only use them if you understand that (e.g. each thread is a non-shared clone of the parent, with memory and start-up-cost implications). Show quoted text
> Discouraged features aren't currently candidates for removal, but we may
> > later deprecate them if they're found to stand in the way of a significant > > improvement to the Perl core.
> > > - surely this is the case? If we come up with a better way to implement > threads while retaining the ability for Windows to support some form of > process emulation and related features, wouldn't this be something that's > welcomed? I don't get the impression that there's widespread satisfaction > with the current state of affairs on either threads.pm or threads::shared.
Even if we ended up doing Windows fork() differently, I wouldn't (at the moment) advocate deprecating or removing threads.pm threads::shared, or the underlying core implementation. -- "I do not resent criticism, even when, for the sake of emphasis, it parts for the time with reality". -- Winston Churchill, House of Commons, 22nd Jan 1941.
Date: Tue, 10 Apr 2018 16:46:13 +0200
CC: Tom Molesworth <tom [...] binary.com>, Christian Walde <walde.christian [...] gmail.com>, Perl5 Porters <perl5-porters [...] perl.org>, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Eirik Berg Hanssen <Eirik-Berg.Hanssen [...] allverden.no>
To: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 2.2k
On Tue, Apr 10, 2018 at 4:27 PM, Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text
On Tue, Apr 10, 2018 at 08:40:45PM +0800, Tom Molesworth via perl5-porters wrote:
> We also don't have good documentation on how to make modules thread-safe:
> I've encountered quite a few Perl developers who are confident that thread
> safety is only an issue with XS.

I'd be one of those, in the sense that the perl language itself is
thread-safe, and doesn't normally need locks or special handling; but in
any environment where there are multiple concurrently running threads,
there will be some extra considerations required.

  And yet, documented behaviour of pure-Perl modules breaks under threads.

  Because of this:
 
Show quoted text
> As an example: far as I recall, the threads.pm documention and perlmod page
> to which they link never explicitly state that ref addresses will be
> different in each thread:

Since the whole basic principle of ithreads is that data isn't shared by
default, it would be astonishing if two (non-shared) refs in two different
threads had the same address.

  (And quite a few pure-Perl modules do not expect this.  I don't have a count of how common it is on CPAN though.)
 
Show quoted text
> Any
> time someone asks about thread safety, I point to the "discouraged" line in
> the official documentation and explain that I don't expect my code to work
> when multiple threads are active - "maybe try again if we have a new
> threads implementation in the future".

Fine - add to the pod in your modules that they're not thread-safe.

  How about we make thread support opt-in instead of opt-out?
 
Show quoted text
> Exposing the API to end users without
> warnings and clear information about how threads differs from other
> languages, on the other hand - maybe not ideal.

I am all in favour of having, at the same location as the current
'discouraged' text, but instead of it, a big flashing neon sign saying
that perl threads are a bit different from what you might expect and only
use them if you understand that (e.g. each thread is a non-shared clone
of the parent, with memory and start-up-cost implications).

  That is, add a warning that threads are unsupported by CPAN modules unless such support is advertised?


Eirik
Date: Tue, 10 Apr 2018 15:59:27 +0100
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #125106] Why are threads discouraged?
CC: perl5-porters [...] perl.org
To: Sergey Aleynikov via RT <perlbug-followup [...] perl.org>
Download (untitled) / with headers
text/plain 1.1k
On Tue, Apr 10, 2018 at 06:42:44AM -0700, Sergey Aleynikov via RT wrote: Show quoted text
> As for the core, I agree that it's now much more thread-stable than > before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no > threads" with some nice crash examples in it - they crash no more. But > still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just > because of no one has discovered that. And the following still fails > loudly (though not dumps a core); > > while (1) { > push @foo, threads->create(sub { > require IO::Handle; > }); > $_->detach for(splice @foo); > }
Oh, that's fun, Looks like _create_getline_subs in IO.xs is directly modifying the global PL_check[] rather than via the official API which does the necessary locking. Show quoted text
> So while yes, perl is much more thread-safe inherently as it used to be, > I won't recommend using threads in it to anyone.
Perl has bugs. Perl's threading has bugs. I haven't seen any particular evidence yet that the number of threaded bugs is disproportionately large on recent perls. -- The Enterprise successfully ferries an alien VIP from one place to another without serious incident. -- Things That Never Happen in "Star Trek" #7
To: Dave Mitchell <davem [...] iabyn.com>
Date: Tue, 10 Apr 2018 11:07:17 -0400
From: Rocco Caputo <rcaputo [...] pobox.com>
CC: Tom Molesworth <tom [...] binary.com>, Christian Walde <walde.christian [...] gmail.com>, Perl5 Porters <perl5-porters [...] perl.org>, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 673b
On Apr 10, 2018, at 10:27, Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text

I am all in favour of having, at the same location as the current
'discouraged' text, but instead of it, a big flashing neon sign saying
that perl threads are a bit different from what you might expect and only
use them if you understand that (e.g. each thread is a non-shared clone
of the parent, with memory and start-up-cost implications).

Also that you will mostly be on your own navigating the differences,
as Perl's users on average are reluctant at best to help you with them
and will often "actively encourage" you to seek other implementations.

-- 
Rocco Caputo <rcaputo@pobox.com>
Show quoted text
Date: Tue, 10 Apr 2018 09:08:56 -0600
From: Karl Williamson <public [...] khwilliamson.com>
CC: perl5-porters [...] perl.org
Subject: Re: [perl #125106] Why are threads discouraged?
To: perlbug-followup [...] perl.org
Download (untitled) / with headers
text/plain 3.5k
On 04/10/2018 07:42 AM, Sergey Aleynikov via RT wrote: Show quoted text
> On Tue, 10 Apr 2018 04:04:06 -0700, davem wrote: > > The problem is in people's expectations. When most part of something is thread-safe, they'll just assume "everything is thread-safe" and then'll be hit very hard. One of the examples I like is Mouse, which was thread-unsafe until the year 2015. While it's not as popular as Moose, it's still a module with large a large user base. And your chances to encounter bugs in not so popular XS modules are much, much higher. What's even worse, you're likely to hit them only under some unlucky circumstances (see example below). > > As for the core, I agree that it's now much more thread-stable than before. Somewhere around 5.12 or 5.14 I've prepared a talk named "no threads" with some nice crash examples in it - they crash no more. But still, locale handling was thread-unsafe until 5.24 (or 5.26?) - just because of no one has discovered that.
Actually threaded pure perl programs are still unsafe in 5.26. perl switches the locale behind your back, even if you follow our admonitions to not explicitly use locales. I'm to blame for some of these, in my earlier naivete, and some have been there for a long time. Sergey found one case that I added in 5.24 I believe, and contributed a test file to verify it's still fixed 5.28 uses thread safe locales if available on the system. On other systems, I avoid switching locales, and added a mutex for those cases where switching is still done. When perl starts up, it reads the environment to see what each locale category should be set to. If not all categories are set to the same thing, this would cause perl to potentially switch locales to gather information about the outliers. This is potentially problematic on unsafe threaded builds. I solved this by gathering the information at start up, and caching it. This is one of the ways 5.28 avoids switching locales. In researching this, I looked in the POSIX standard for functions it allows to be non-thread safe. I have a WIP to add cautions about these to XS writers. I also noticed that the Linux man pages indicated that they have failed to implement correctly some that are supposed to be thread-safe. Other systems may implement these safely, but not others. I searched the perl source code for instances of these calls. And then manually started to examine them to see if there was a problem. I have not finished (and of course I may make mistakes in my analysis). The glaring case where there is a problem is in accessing the environment (getenv() et.al.) These need to be protected by a mutex, but it's only a problem if another thread is changing the environment at the same time, a much less common occurrence. My guess is that these aren't crashing things because most environment changes would tend to be done at start-up, even before thread creation. Things you might not expect to, without thinking about it, like tzset(), do access the environment, and there is a potential race if the environment is changed by another thread during tzset's execution. And tzset is called from places that at first glance you wouldn't expect. And the following still fails loudly (though nott dumps a core); Show quoted text
> > while (1) { > push @foo, threads->create(sub { > require IO::Handle; > }); > $_->detach for(splice @foo); > } > > So while yes, perl is much more thread-safe inherently as it used to be, I won't recommend using threads in it to anyone. > > --- > via perlbug: queue: perl5 status: open > https://rt.perl.org/Ticket/Display.html?id=125106 >
To: Tom Molesworth <tom [...] binary.com>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
CC: Christian Walde <walde.christian [...] gmail.com>, Sawyer X <xsawyerx [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, Perl5 Porters <perl5-porters [...] perl.org>
From: Dave Mitchell <davem [...] iabyn.com>
Date: Tue, 10 Apr 2018 16:11:34 +0100
Download (untitled) / with headers
text/plain 1.4k
On Tue, Apr 10, 2018 at 10:26:21PM +0800, Tom Molesworth via perl5-porters wrote: Show quoted text
> On 10 April 2018 at 21:59, Dave Mitchell <davem@iabyn.com> wrote:
> > Can you come up with a hypothetical scenario, e.g. a multi-threaded > > program that uses libraries to connect to a database and retrieve and > > parse some XML data. Then got through it step by step so that I can see > > why using perl and CPAN is dangerous, but using (e.g.) java and a DB and > > XML library is safe? What is the crucial difference between the two that > > flips it from being safe to unsafe? > >
> > Here's one trivial hacked-together example of code which I would argue > leads to "unexpected" results, at least from the perspective of a C or Java > programmer experienced in the ways of threads:
That isn't a reply to the asked question, which was trying to eke out an understanding of why Christian believes the CPAN ecosystem in some way makes threading more hazardous on perl than in other languages (which it may, but I can't grasp the point he's trying to make). The example code you've given seems to be (at a quick glance), just the fact (again) that refs in different threads have different addresses in perl. Yes of course this will be confusing to anyone who expects the perl threading model to be the same as C or Java's. And yes I agree that the docs should explain the model early on to manage expectations. -- This is a great day for France! -- Nixon at Charles De Gaulle's funeral
From: Dave Mitchell <davem [...] iabyn.com>
CC: Tom Molesworth <tom [...] binary.com>, Christian Walde <walde.christian [...] gmail.com>, Perl5 Porters <perl5-porters [...] perl.org>, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Date: Tue, 10 Apr 2018 16:12:42 +0100
To: Eirik Berg Hanssen <Eirik-Berg.Hanssen [...] allverden.no>
Download (untitled) / with headers
text/plain 317b
On Tue, Apr 10, 2018 at 04:46:13PM +0200, Eirik Berg Hanssen wrote: Show quoted text
> That is, add a warning that threads are unsupported by CPAN modules > unless such support is advertised?
I'd be happy with that. -- Lear: Dost thou call me fool, boy? Fool: All thy other titles thou hast given away; that thou wast born with.
To: "Dave Mitchell" <davem [...] iabyn.com>
CC: "Sawyer X" <xsawyerx [...] gmail.com>, "Aristotle Pagaltzis" <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, perl5-porters [...] perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: "Christian Walde" <walde.christian [...] gmail.com>
Date: Tue, 10 Apr 2018 17:39:27 +0200
On Tue, 10 Apr 2018 15:59:53 +0200, Dave Mitchell <davem@iabyn.com> wrote: Show quoted text
> why using perl and CPAN is dangerous, but using (e.g.) java and a DB and > XML library is safe? What is the crucial difference between the two that > flips it from being safe to unsafe?
Very short version: Java has a much more bigger userbase, money invested, and things like a database module or an xml parser under java will be written under the assumption of being required to work with threads, as Java uses threads like skittles. Also in Java both of these are in the standard library, not 3rd party. Additionally, to give you a more concrete example of a simple Perl threads fuck-up: In 2011 i attempted to write a simple parallel http downloader, using LWP and some thread queue manager module. It worked 100% fine under windows, no matter what i did. On Fedora it seemed to work fine, but crashed randomly when attempting to handle too many downloads. This would be completely inconceivable in Java. -- With regards, Christian Walde
Date: Tue, 10 Apr 2018 23:58:15 +0800
From: Tom Molesworth <tom [...] binary.com>
CC: Christian Walde <walde.christian [...] gmail.com>, Sawyer X <xsawyerx [...] gmail.com>, Aristotle Pagaltzis <pagaltzis [...] gmx.de>, bugs-bitbucket [...] rt.perl.org, Perl5 Porters <perl5-porters [...] perl.org>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
To: Dave Mitchell <davem [...] iabyn.com>

On 10 April 2018 at 23:11, Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text
On Tue, Apr 10, 2018 at 10:26:21PM +0800, Tom Molesworth via perl5-porters wrote:
> On 10 April 2018 at 21:59, Dave Mitchell <davem@iabyn.com> wrote:
> > Can you come up with a hypothetical scenario, e.g.  a multi-threaded
> > program that uses libraries to connect to a database and retrieve and
> > parse some XML data. Then got through it step by step so that I can see
> > why using perl and CPAN is dangerous, but using (e.g.) java and a DB and
> > XML library is safe? What is the crucial difference between the two that
> > flips it from being safe to unsafe?
> >
>
> Here's one trivial hacked-together example of code which I would argue
> leads to "unexpected" results, at least from the perspective of a C or Java
> programmer experienced in the ways of threads:

That isn't a reply to the asked question, which was trying to eke out an
understanding of why Christian believes the CPAN ecosystem in some way
makes threading more hazardous on perl than in other languages (which it
may, but I can't grasp the point he's trying to make).

The example code you've given seems to be (at a quick glance), just the
fact (again) that refs in different threads have different addresses in
perl.

Related but slightly different: this time it's 2 different addresses *within the same thread*, none of which match the refaddr in the other thread. For something that's supposedly "shared", I'd call having 3 addresses surprising at best!

What I'm trying to demonstrate is it's easy to hit implementation details in Perl, ones that go against the common understanding of "threads". I don't think people should have to read Shared.xs before they can implement a multithreaded XML+DB application. Documentation might help, but there's an uphill struggle at the first word: what we call "threads", other languages might call a nearly-unrecognisable special case!

(Eirik and Christian have already addressed the other comments I was going to make, so I'll stop here - sorry for derailing the thread) 
To: Tom Molesworth <tom [...] binary.com>
CC: Christian Walde <walde.christian [...] gmail.com>, Perl5 Porters <perl5-porters [...] perl.org>, bugs-bitbucket [...] rt.perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Leon Timmermans <fawaka [...] gmail.com>
Date: Wed, 11 Apr 2018 00:24:35 +0200
Download (untitled) / with headers
text/plain 1.2k
On Tue, Apr 10, 2018 at 2:40 PM, Tom Molesworth via perl5-porters <perl5-porters@perl.org> wrote: Show quoted text
>> Discouraged features aren't currently candidates for removal, but we may >> later deprecate them if they're found to stand in the way of a significant >> improvement to the Perl core.
> > - surely this is the case? If we come up with a better way to implement > threads while retaining the ability for Windows to support some form of > process emulation and related features, wouldn't this be something that's > welcomed? I don't get the impression that there's widespread satisfaction > with the current state of affairs on either threads.pm or threads::shared.
A crucial thing to understand in this idea is that ithreads and threads.pm are not the same thing. ithreads is a C-level feature in the implementation, threads.pm and pseudoforks are end-user level features that are built upon ithreads. One can build other abstractions on top of them. ithreads is not a good abstraction for anything resembling shared-memory architectures, but does fit a number of other concurrency models. threads::lite did proof that concept I hope. If that's not what one wants, it's probably possible to implement GIL threading instead (though I'm not sure if we'd really want that either). I think that's unexplored territory though. Leon
To: perl5-porters [...] perl.org
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Tomasz Konojacki <me [...] xenu.pl>
Date: Wed, 11 Apr 2018 00:33:40 +0200
Download (untitled) / with headers
text/plain 187b
On Tue, 10 Apr 2018, at 16:46, Eirik Berg Hanssen wrote:
Show quoted text
  How about we make thread support opt-in instead of opt-out?

It's already the case. By default perl builds without ithreads.
CC: Perl5 Porters <perl5-porters [...] perl.org>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
From: Eirik Berg Hanssen <Eirik-Berg.Hanssen [...] allverden.no>
To: Tomasz Konojacki <me [...] xenu.pl>
Date: Wed, 11 Apr 2018 00:44:09 +0200
Download (untitled) / with headers
text/plain 710b
On Wed, Apr 11, 2018 at 12:33 AM, Tomasz Konojacki <me@xenu.pl> wrote:
Show quoted text
On Tue, 10 Apr 2018, at 16:46, Eirik Berg Hanssen wrote:
  How about we make thread support opt-in instead of opt-out?

It's already the case. By default perl builds without ithreads.

  Context: 

Show quoted text
Fine - add to the pod in your modules that they're not thread-safe.

  How about we make thread support opt-in instead of opt-out?

  That is, I was referring to thread support in CPAN modules.  I'm suggesting we document, in big fat (blinking marquee etc ...):

  Unless a module explicitly states it supports threads, and you use it with threads, you're on your own and get to keep both pieces when it breaks.


Eirik
Date: Wed, 11 Apr 2018 01:13:39 +0200
To: perl5-porters [...] perl.org
From: Tomasz Konojacki <me [...] xenu.pl>
Subject: Re: [perl #133021] Removed the word "discouraged" from threads' documentation
Download (untitled) / with headers
text/plain 607b
On Wed, 11 Apr 2018, at 00:44, Eirik Berg Hanssen wrote: Show quoted text
>   Context:  >
>>
>>> Fine - add to the pod in your modules that they're not thread-safe.
>> >> >>   How about we make thread support opt-in instead of opt-out?
> >   That is, I was referring to thread support in CPAN modules.  I'm suggesting we document, in big fat (blinking marquee etc ...): > >   Unless a module explicitly states it supports threads, and you use it with threads, you're on your own and get to keep both pieces when it breaks. > > Eirik
Oh, I see, I missed the last sentence of your message. Sorry for the noise.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 488b
On Tue, 10 Apr 2018 15:25:00 -0700, LeonT wrote: Show quoted text
> If that's not what one wants, it's probably possible to implement GIL > threading instead (though I'm not sure if we'd really want that > either). I think that's unexplored territory though.
Wouldnt one of the existing fibers/future/promises/Coro/async-but-not-async-with-explicit-yields modules that swap Perl stacks inside the same interp/same perl thread be perl's already-implemented GIL concept? -- bulk88 ~ bulk88 at hotmail.com
Date: Wed, 11 Apr 2018 10:55:12 +0200
To: perlbug <perlbug-followup [...] perl.org>
From: Leon Timmermans <fawaka [...] gmail.com>
CC: Perl5 Porters <perl5-porters [...] perl.org>
Subject: Re: [perl #125106] Why are threads discouraged?
Download (untitled) / with headers
text/plain 661b
On Wed, Apr 11, 2018 at 8:45 AM, bulk88 via RT <perlbug-followup@perl.org> wrote: Show quoted text
> On Tue, 10 Apr 2018 15:25:00 -0700, LeonT wrote:
>> If that's not what one wants, it's probably possible to implement GIL >> threading instead (though I'm not sure if we'd really want that >> either). I think that's unexplored territory though.
> > Wouldnt one of the existing fibers/future/promises/Coro/async-but-not-async-with-explicit-yields modules that swap Perl stacks inside the same interp/same perl thread be perl's already-implemented GIL concept?
AFAIK those are all "one interpreter one os-thread", wheras GIL threading is "one interpreter many os-threads" Leon


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org