Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re: Bleadperl v5.17.9-200-g0e0ab62 breaks MLEHMANN/JSON-XS-2.33.tar.gz #12864

Closed
p5pRT opened this issue Mar 21, 2013 · 77 comments
Closed

Re: Bleadperl v5.17.9-200-g0e0ab62 breaks MLEHMANN/JSON-XS-2.33.tar.gz #12864

p5pRT opened this issue Mar 21, 2013 · 77 comments

Comments

@p5pRT
Copy link

p5pRT commented Mar 21, 2013

Migrated from rt.perl.org#117239 (status was 'resolved')

Searchable as RT117239$

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @andk

git bisect


commit 0e0ab62
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Sun Mar 17 20​:19​:09 2013 +0100

  Harden hashes against hash seed discovery by randomizing hash iteration

sample fail report


http​://www.cpantesters.org/cpan/report/48924bc0-9027-11e2-869e-dc2c3b384401

The test failing​:

t/19_incr.t (Wstat​: 12288 Tests​: 697 Failed​: 48)
  Failed tests​: 8, 11, 14, 17, 20, 29, 35, 38, 41, 44, 53
  56, 68, 71, 89, 95, 101, 104, 113, 122
  137, 140, 146, 152, 158, 164, 167, 170
  176, 179, 182, 185, 188, 194, 197, 200
  203, 206, 212, 233, 260, 263, 266, 269
  275, 284, 287, 290
  Non-zero exit status​: 48

The line it reports in the diagnostics​:

https://metacpan.org/source/MLEHMANN/JSON-XS-2.33/t/19_incr.t#L22

When the test is running twice, then the reported fails are usually
different ones.

perl -V


Summary of my perl5 (revision 5 version 17 subversion 10) configuration​:
  Commit id​: a7b39f8
  Platform​:
  osname=linux, osvers=3.2.0-4-amd64, archname=x86_64-linux-thread-multi-ld
  uname='linux k83 3.2.0-4-amd64 #1 smp debian 3.2.39-2 x86_64 gnulinux '
  config_args='-Dprefix=/home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Duseithreads -Duselongdouble -DDEBUGGING=-g'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.7.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.13'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT
  PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PERL_SAWAMPERSAND
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_ITHREADS USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LONG_DOUBLE USE_PERLIO
  USE_PERL_ATOF USE_REENTRANT_API
  Built under linux
  Compiled at Mar 19 2013 00​:37​:12
  %ENV​:
  PERL5LIB=""
  PERL5OPT=""
  PERL5_CPANPLUS_IS_RUNNING="3177"
  PERL5_CPAN_IS_RUNNING="3177"
  @​INC​:
  /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/site_perl/5.17.10/x86_64-linux-thread-multi-ld
  /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/site_perl/5.17.10
  /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/5.17.10/x86_64-linux-thread-multi-ld
  /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/5.17.10
  .

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @andk

(Andreas J. Koenig) (via RT) <perlbug-followup@​perl.org> writes​:

# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >

Harden hashes against hash seed discovery by randomizing hash iteration

Other candidates that start failing around the same time and should get
a closer examination​:

JEEN/Acme-CPANAuthors-Korean-0.09.tar.gz
SHARYANTO/Data-Schema-0.135.tar.gz
SMIRNIOS/DBD-SQLAnywhere-2.08.tar.gz
TIMB/DBI-1.623.tar.gz
ANDK/Devel-Symdump-2.08.tar.gz
OPI/IO-Handle-Record-0.14.tar.gz
MAKAMAKA/JSON-PPdev-2.27100.tar.gz
JEEN/Lingua-KO-TypoCorrector-0.03.tar.gz
JROBINSON/Locale-Object-0.79.tar.gz
PSCUST/Parse-FSM-1.06.tar.gz
MAGGIEXYZ/PDL-Stats-0.6.2.tar.gz
ADAMK/Perl-Squish-1.06.tar.gz
SATOH/Plack-Middleware-StaticShared-0.05.tar.gz
VOJ/RDF-NS-20130208.tar.gz
MWS/ResourcePool-1.0106.tar.gz
JSIRACUSA/Rose-HTML-Objects-0.617.tar.gz
MKUTTER/SOAP-Lite-0.714.tar.gz
MKUTTER/SOAP-Transport-TCP-0.715.tar.gz
TADAM/Test-Mock-LWP-Dispatch-0.03.tar.gz
MSCHWERN/Time-y2038-20100403.tar.gz
JEEN/WebService-Aladdin-0.0706.tar.gz

I have not had time to verify that they all start failing on the same
commit. At least for DBI I have evidence that it starts failing on
v5.17.9-201-g3078e10. But since randomness is involved this finding may
be biased.

I have not verified others due to lack of time.

--
andreas

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 06​:41​:28AM +0100, Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

Harden hashes against hash seed discovery by randomizing hash iteration

Thanks andreas for pointing this out, hi Yves.

I am talking a bit out of my ass here (call it a rant if you wish), and
can only guess whats going on. But in short​: this change is wrong.

- Unless I am mistaken, this is to avoid a DOS attack. DOS attacks are easy
  to protect against, and this won't protect me against basically any form
  of attack, so it's useless. Far better methods exist that prevent every
  form of memory resource and/or cpu time starvation.
- I'd say this simply breaks perl, as perl documented the order is fixed,
  at least between runs (I would argue even the earlier fix to randomise
  between runs is a bug). Sure, the wording is ambiguous, but that doesn't
  make it any better.
- Perl hash already are very slow (just write yourself some XS functions to
  interface to gcc's unordered_hash and watch the speed - despite the
  overhead of XS).
- Taking out the inability to write a proper hash on all the perl developers
  out there is just wrong - Perl is the runtime, and should get it right,
  if possible.

So please reconsider your choice of making perl worse again.

On a tangent, the current regime of perl development - making perl worse,
breaking CPAN more and more, adding lots of incompatible changes while at
the same time requiring use feature is in the wrong for quite some time
now, and, frankly, the only reason I don't fork perl is simply that I
am not prepared to maintain yet another software package. Breaking CPAN
packages every 6 months just doesn't cut it for me, and creates way too
much needless work on my side. Not even mentioning the train wrecks of
design hoisted on us in recent releases (smart matches, given/when being
lexical when everything else is dynamical in perl, mandatory use strict
backfitted, yet more version string madness etc. etc. - I know designing
these things is not easy, but if it's hard, don't make hacks and release
them every few months. OR, hey, even the regex changes which look nice,
but I have yet to see something that is either faster or more readable
when using them to implement something in the real world).

I'd much prefer if the current perl5porters would go back to maintaining
perl _5_ properly and taking out their dreams on breaking perl and making
it "better" on Perl 6.

But hey, for what I know I am probably in the minority and other people
embrace this, but I am certainly not alone in these sentiments.

So again, please reconsider, and think about either writing a proper
hash that doesn't suffer from simple attacks (a bit more speed would be
appreciated as well), or forget about making ineffective partial fixes for
problems that make writing perl harder, and fix nothing.

Yves, I know you are not to blame, at least not alone, but it's really
getting too much for me as a user of perl. I honestly wish back the lazy
days of perl 5.8, which had many bugs, was broken a lot, but at least
respected Perl as a language and CPAN for its code, instead of treating
both as slightly broken anyways.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @demerphq

On 21 March 2013 09​:28, Marc Lehmann <schmorp@​schmorp.de> wrote​:

On Thu, Mar 21, 2013 at 06​:41​:28AM +0100, Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

Harden hashes against hash seed discovery by randomizing hash iteration

Thanks andreas for pointing this out, hi Yves.

I am talking a bit out of my ass here (call it a rant if you wish), and
can only guess whats going on. But in short​: this change is wrong.

Your opinion is noted. However given you appear to be uninformed as to
the details it is not worth much.

- Unless I am mistaken, this is to avoid a DOS attack. DOS attacks are easy
to protect against, and this won't protect me against basically any form
of attack, so it's useless. Far better methods exist that prevent every
form of memory resource and/or cpu time starvation.

You are mistaken.

- I'd say this simply breaks perl, as perl documented the order is fixed,
at least between runs (I would argue even the earlier fix to randomise
between runs is a bug). Sure, the wording is ambiguous, but that doesn't
make it any better.

I am unaware of any such documentation. Please state your sources.

- Perl hash already are very slow (just write yourself some XS functions to
interface to gcc's unordered_hash and watch the speed - despite the
overhead of XS).

Yes, and this is a very tiny drop in a large bucket. If you think that
the changes I made are going to affect hash performance in any kind of
significant way then please provide evidence.

- Taking out the inability to write a proper hash on all the perl developers
out there is just wrong - Perl is the runtime, and should get it right,
if possible.

This doesn't make grammatical sense, so I don't know what you are trying to say.

So please reconsider your choice of making perl worse again.

I don't think I have made Perl worse, so currently I have nothing to reconsider.

On a tangent, the current regime of perl development - making perl worse,
breaking CPAN more and more, adding lots of incompatible changes while at
the same time requiring use feature is in the wrong for quite some time
now, and, frankly, the only reason I don't fork perl is simply that I
am not prepared to maintain yet another software package. Breaking CPAN
packages every 6 months just doesn't cut it for me, and creates way too
much needless work on my side. Not even mentioning the train wrecks of
design hoisted on us in recent releases (smart matches, given/when being
lexical when everything else is dynamical in perl, mandatory use strict
backfitted, yet more version string madness etc. etc. - I know designing
these things is not easy, but if it's hard, don't make hacks and release
them every few months. OR, hey, even the regex changes which look nice,
but I have yet to see something that is either faster or more readable
when using them to implement something in the real world).

I'd much prefer if the current perl5porters would go back to maintaining
perl _5_ properly and taking out their dreams on breaking perl and making
it "better" on Perl 6.

But hey, for what I know I am probably in the minority and other people
embrace this, but I am certainly not alone in these sentiments.

So again, please reconsider, and think about either writing a proper
hash that doesn't suffer from simple attacks (a bit more speed would be
appreciated as well), or forget about making ineffective partial fixes for
problems that make writing perl harder, and fix nothing.

Yves, I know you are not to blame, at least not alone, but it's really
getting too much for me as a user of perl. I honestly wish back the lazy
days of perl 5.8, which had many bugs, was broken a lot, but at least
respected Perl as a language and CPAN for its code, instead of treating
both as slightly broken anyways.

Thanks for your feedback.

I am sure the perl5porters who are regular contributors will give your
opinions on how they use their time the full attention and
consideration that they deserve.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @lizmat

On Mar 21, 2013, at 8​:33 AM, Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

(Andreas J. Koenig) (via RT) <perlbug-followup@​perl.org> writes​:

# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >

Harden hashes against hash seed discovery by randomizing hash iteration

Other candidates that start failing around the same time and should get
a closer examination​:

JEEN/Acme-CPANAuthors-Korean-0.09.tar.gz
SHARYANTO/Data-Schema-0.135.tar.gz
SMIRNIOS/DBD-SQLAnywhere-2.08.tar.gz
TIMB/DBI-1.623.tar.gz
ANDK/Devel-Symdump-2.08.tar.gz
OPI/IO-Handle-Record-0.14.tar.gz
MAKAMAKA/JSON-PPdev-2.27100.tar.gz
JEEN/Lingua-KO-TypoCorrector-0.03.tar.gz
JROBINSON/Locale-Object-0.79.tar.gz
PSCUST/Parse-FSM-1.06.tar.gz
MAGGIEXYZ/PDL-Stats-0.6.2.tar.gz
ADAMK/Perl-Squish-1.06.tar.gz
SATOH/Plack-Middleware-StaticShared-0.05.tar.gz
VOJ/RDF-NS-20130208.tar.gz
MWS/ResourcePool-1.0106.tar.gz
JSIRACUSA/Rose-HTML-Objects-0.617.tar.gz
MKUTTER/SOAP-Lite-0.714.tar.gz
MKUTTER/SOAP-Transport-TCP-0.715.tar.gz
TADAM/Test-Mock-LWP-Dispatch-0.03.tar.gz
MSCHWERN/Time-y2038-20100403.tar.gz
JEEN/WebService-Aladdin-0.0706.tar.gz

I have not had time to verify that they all start failing on the same
commit. At least for DBI I have evidence that it starts failing on
v5.17.9-201-g3078e10. But since randomness is involved this finding may
be biased.

I have not verified others due to lack of time.

So, to get this straight​:

my @​key= keys %hash;
my @​value= values %hash;

With this change, there is no guarantee anymore that $key[N] => $value[N] (in the same process) ?

I'm not 100% sure, but FWIW I think I have used this property of keys() and values() in production code in the past.

Liz

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @demerphq

On 21 March 2013 10​:15, Elizabeth Mattijsen <liz@​dijkmat.nl> wrote​:

On Mar 21, 2013, at 8​:33 AM, Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

(Andreas J. Koenig) (via RT) <perlbug-followup@​perl.org> writes​:

# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >

Harden hashes against hash seed discovery by randomizing hash iteration

Other candidates that start failing around the same time and should get
a closer examination​:

JEEN/Acme-CPANAuthors-Korean-0.09.tar.gz
SHARYANTO/Data-Schema-0.135.tar.gz
SMIRNIOS/DBD-SQLAnywhere-2.08.tar.gz
TIMB/DBI-1.623.tar.gz
ANDK/Devel-Symdump-2.08.tar.gz
OPI/IO-Handle-Record-0.14.tar.gz
MAKAMAKA/JSON-PPdev-2.27100.tar.gz
JEEN/Lingua-KO-TypoCorrector-0.03.tar.gz
JROBINSON/Locale-Object-0.79.tar.gz
PSCUST/Parse-FSM-1.06.tar.gz
MAGGIEXYZ/PDL-Stats-0.6.2.tar.gz
ADAMK/Perl-Squish-1.06.tar.gz
SATOH/Plack-Middleware-StaticShared-0.05.tar.gz
VOJ/RDF-NS-20130208.tar.gz
MWS/ResourcePool-1.0106.tar.gz
JSIRACUSA/Rose-HTML-Objects-0.617.tar.gz
MKUTTER/SOAP-Lite-0.714.tar.gz
MKUTTER/SOAP-Transport-TCP-0.715.tar.gz
TADAM/Test-Mock-LWP-Dispatch-0.03.tar.gz
MSCHWERN/Time-y2038-20100403.tar.gz
JEEN/WebService-Aladdin-0.0706.tar.gz

I have not had time to verify that they all start failing on the same
commit. At least for DBI I have evidence that it starts failing on
v5.17.9-201-g3078e10. But since randomness is involved this finding may
be biased.

I have not verified others due to lack of time.

So, to get this straight​:

my @​key= keys %hash;
my @​value= values %hash;

With this change, there is no guarantee anymore that $key[N] => $value[N] (in the same process) ?

Hi Liz! This hasn't changed.

I'm not 100% sure, but FWIW I think I have used this property of keys() and values() in production code in the past.

Yep, you sure have, we all have I should think, at some point or another. :-)

And the behavior has not changed. Any code that is currently correct
will continue to work as before.

What has changed is that in earlier perls you could do something like this​:

my @​keys= keys %hash;
my %copy= %hash;
my @​values= %copy;

and most of the time (but not always) get away with assuming that
$keys[$n] corresponds to $values[$n].

In 5.18 you are virtually guaranteed that $keys[$n] will NOT
correspond to $values[$n].

So all that has happened is that buggy code is now going to
demonstrate the bugs pretty much always instead of very very rarely.

cheers
Yves
ps​: Hope you and Wendy are well!

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @demerphq

On 21 March 2013 10​:25, demerphq <demerphq@​gmail.com> wrote​:

my @​keys= keys %hash;
my %copy= %hash;
my @​values= %copy;

That should have been

my @​values= values %copy;

Sorry.
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @lizmat

On Mar 21, 2013, at 10​:26 AM, demerphq <demerphq@​gmail.com> wrote​:

On 21 March 2013 10​:25, demerphq <demerphq@​gmail.com> wrote​:

my @​keys= keys %hash;
my %copy= %hash;
my @​values= %copy;

That should have been

my @​values= values %copy;

Sorry.

No problem, I realised that already.

And *phew*. :-) Thanks for the explanation.

Liz

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 10​:04​:24AM +0100, demerphq <demerphq@​gmail.com> wrote​:

Your opinion is noted. However given you appear to be uninformed as to
the details it is not worth much.

Then please state where I was wrong, I am always open to learn the facts.

- Unless I am mistaken, this is to avoid a DOS attack. DOS attacks are easy
to protect against, and this won't protect me against basically any form
of attack, so it's useless. Far better methods exist that prevent every
form of memory resource and/or cpu time starvation.

You are mistaken.

You are really claiming that this is one of the best methods to prevent
memory resource and/or cpu time starvation? Because that's what the satteemnt
you quoted says.

Or maybe you meant I am mistaken in something else? Without telling me
what it is you are refering to it is hard to tell.

- I'd say this simply breaks perl, as perl documented the order is fixed,
at least between runs (I would argue even the earlier fix to randomise
between runs is a bug). Sure, the wording is ambiguous, but that doesn't
make it any better.

I am unaware of any such documentation. Please state your sources.

The perlfunc 5.16 documentation on "keys". Older versions are even more
explicit.

all them are ambiguous, but I already mentioned that.

- Perl hash already are very slow (just write yourself some XS functions to
interface to gcc's unordered_hash and watch the speed - despite the
overhead of XS).

Yes, and this is a very tiny drop in a large bucket. If you think that
the changes I made are going to affect hash performance in any kind of
significant way then please provide evidence.

I don't think that, and haven't made that claim. I think you can make
faster hashes with better properties without the drawbacks.

- Taking out the inability to write a proper hash on all the perl developers
out there is just wrong - Perl is the runtime, and should get it right,
if possible.

This doesn't make grammatical sense,

I think it's a perfectly fine english sentence. Maybe a bit complicated for
you to understand though.

so I don't know what you are trying to say.

I am trying to say that the solution for bad hash tables is better hash
tables, or a sense of proportion, not breaking perfectly fine existing
code.

So please reconsider your choice of making perl worse again.

I don't think I have made Perl worse, so currently I have nothing to reconsider.

To me, breaking existing code for questionable or no reason is making it
worse. It is obvious that you disagree and think you didn't make it worse,
but at best, you have to disagree with my perfectly valid opinion on this
topic-

I am sure the perl5porters who are regular contributors will give your
opinions on how they use their time the full attention and consideration
that they deserve.

I am fully aware of them being volunteers, but so am I (and I am a regular
contributor as well), please keep that in mind.

The perl5porters are not "better" than me as a major contributor to perl
in any way, and have no higher moral ground. Acting uppity because you are
part of a mailinglist and contribute more often than me is proving nothing
really.

I can't "the people who maintainer perl5" (as opposed to perl5porters) to
do what I want, and neither do I try, but sarcaasm and ignorance on your
part is wrong too. My concerns are valid, and while you have all the right
in the world to ignore them, or even ridicule them, that doesn't make them
invalid.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @demerphq

On 21 March 2013 11​:02, Marc Lehmann <schmorp@​schmorp.de> wrote​:

On Thu, Mar 21, 2013 at 10​:04​:24AM +0100, demerphq <demerphq@​gmail.com> wrote​:

Your opinion is noted. However given you appear to be uninformed as to
the details it is not worth much.

Then please state where I was wrong, I am always open to learn the facts.

- Unless I am mistaken, this is to avoid a DOS attack. DOS attacks are easy
to protect against, and this won't protect me against basically any form
of attack, so it's useless. Far better methods exist that prevent every
form of memory resource and/or cpu time starvation.

You are mistaken.

You are really claiming that this is one of the best methods to prevent
memory resource and/or cpu time starvation? Because that's what the satteemnt
you quoted says.

I am sorry, I seem to be missing some context here. What statement did
I quote? I just checked this thread I did not quote anything.

The purpose of hash iterator randomization is make it harder, and
hopefully outright impossible to discover the hash seed a given
process is using.

Once an attacker can determine the hash seed they can launch
algorithmic complexity attacks on our hash implementation. However the
iterator randomization does not in any way prevent a DOS, it prevents
obtaining the information that makes a DOS possible.

Or maybe you meant I am mistaken in something else? Without telling me
what it is you are refering to it is hard to tell.

- I'd say this simply breaks perl, as perl documented the order is fixed,
at least between runs (I would argue even the earlier fix to randomise
between runs is a bug). Sure, the wording is ambiguous, but that doesn't
make it any better.

I am unaware of any such documentation. Please state your sources.

The perlfunc 5.16 documentation on "keys". Older versions are even more
explicit.

all them are ambiguous, but I already mentioned that.

Well, I beg to differ. I think you are conflating "does not say
anything on the subject" and "says something ambiguous".

The documentation for keys(), and as far as I know all other
documentation relating to hashes says nothing about consistency,
either cross process or in between hashes. It says that the keys are
returned in an apparently random order, that the random order is
subject to change in future versions of perl, and the only guarantee
of any form provided is that the keys() function will return items in
the same order that each() and values would return them assuming the
hash has not been modified (a secondary clause in the documentation
for "each" specifies it is safe to delete the key that was just
returned by each()).

Furthermore I cannot find any evidence that the wording for this has
changed in terms of specificity in the lifetime of perlfunc.pod​:

In bleadperl it says​:

aeedbbe (Nicholas Clark 2007-12-21 17​:58​:03
+0000 3109) The keys of a hash are returned in an apparently random
order. The actual
3b10bc6 (brian d foy 2010-01-13 16​:19​:33
+0100 3110) random order is subject to change in future versions of
Perl, but it
504f80c (Jarkko Hietaniemi 2003-06-26 05​:32​:02
+0000 3111) is guaranteed to be the same order as either the C<values>
or C<each>
4546b9e (Jarkko Hietaniemi 2003-07-27 20​:21​:40
+0000 3112) function produces (given that the hash has not been
modified). Since
c5f61d2 (Peter J. Holzer 2010-12-04 11​:40​:23
-0800 3113) Perl 5.8.1 the ordering can be different even between
different runs of
4546b9e (Jarkko Hietaniemi 2003-07-27 20​:21​:40
+0000 3114) Perl for security reasons (see L<perlsec/"Algorithmic
Complexity
d6df370 (Ronald J. Kimball 2003-07-02 07​:43​:05
-0400 3115) Attacks">).

In Perl 5.14.2 it says this​:

  The keys of a hash are returned in an apparently random
order. The actual random order is subject to change in future
versions of Perl, but it is guaranteed to be the same order as either
  the "values" or "each" function produces (given that
the hash has not been modified). Since Perl 5.8.1 the ordering is
different even between different runs of Perl for security reasons
  (see "Algorithmic Complexity Attacks" in perlsec).

In perl 5.8.5 it says this​:

  The keys are returned in an apparently random order.
The actual random order is subject to change in future versions of
perl, but it is guaranteed to be the same order as either the "val-
  ues" or "each" function produces (given that the hash
has not been modified). Since Perl 5.8.1 the ordering is different
even between different runs of Perl for security reasons (see "Algo-
  rithmic Complexity Attacks" in perlsec).

Here is what it said in 1998/99​:

19799a2 (Gurusamy Sarathy 1999-05-24 07​:24​:11
+0000 2314) Returns a list consisting of all the keys of the named
hash. (In
1d2dff6 (Gurusamy Sarathy 1998-05-29 02​:31​:44
+0000 2315) scalar context, returns the number of keys.) The keys are
returned in
ab19240 (Gurusamy Sarathy 1998-11-28 09​:36​:40
+0000 2316) an apparently random order. The actual random order is
subject to
ab19240 (Gurusamy Sarathy 1998-11-28 09​:36​:40
+0000 2317) change in future versions of perl, but it is guaranteed to
be the same
19799a2 (Gurusamy Sarathy 1999-05-24 07​:24​:11
+0000 2318) order as either the C<values> or C<each> function produces
(given
ab19240 (Gurusamy Sarathy 1998-11-28 09​:36​:40
+0000 2319) that the hash has not been modified). As a side effect,
it resets
ab19240 (Gurusamy Sarathy 1998-11-28 09​:36​:40
+0000 2320) HASH's iterator.

And in 97​:

aa68939 (Perl 5 Porters 1997-02-22 04​:41​:00
+1200 1710) Returns a normal array consisting of all the keys of the
named hash. (In
aa68939 (Perl 5 Porters 1997-02-22 04​:41​:00
+1200 1711) a scalar context, returns the number of keys.) The keys
are returned in
aa68939 (Perl 5 Porters 1997-02-22 04​:41​:00
+1200 1712) an apparently random order, but it is the same order as
either the
aa68939 (Perl 5 Porters 1997-02-22 04​:41​:00
+1200 1713) values() or each() function produces (given that the hash
has not been
aa68939 (Perl 5 Porters 1997-02-22 04​:41​:00
+1200 1714) modified). As a side effect, it resets HASH's iterator.

And in 94 (5.000)

a0d0e21 (Larry Wall 1994-10-17 23​:00​:00 +0000 1578) Returns a
normal array consisting of all the keys of the named
a0d0e21 (Larry Wall 1994-10-17 23​:00​:00 +0000 1579)
associative array. (In a scalar context, returns the number of keys.)
a0d0e21 (Larry Wall 1994-10-17 23​:00​:00 +0000 1580) The keys
are returned in an apparently random order, but it is the same
a0d0e21 (Larry Wall 1994-10-17 23​:00​:00 +0000 1581) order as
either the values() or each() function produces (given that
a0d0e21 (Larry Wall 1994-10-17 23​:00​:00 +0000 1582) the
associative array has not been modified). Here is yet another way
a0d0e21 (Larry Wall 1994-10-17 23​:00​:00 +0000 1583) to print
your environment​:

perlfunc.pod didnt exist prior to that commit.

I see nothing ambiguous in the documentation, nor do I see it becoming
more or less specific over time.

- Perl hash already are very slow (just write yourself some XS functions to
interface to gcc's unordered_hash and watch the speed - despite the
overhead of XS).

Yes, and this is a very tiny drop in a large bucket. If you think that
the changes I made are going to affect hash performance in any kind of
significant way then please provide evidence.

I don't think that, and haven't made that claim. I think you can make
faster hashes with better properties without the drawbacks.

So then why did you bring it up in this thread? If you have issues
with hash performance and think you can do better then please start a
new thread, preferably by posting patches.

- Taking out the inability to write a proper hash on all the perl developers
out there is just wrong - Perl is the runtime, and should get it right,
if possible.

This doesn't make grammatical sense,

I think it's a perfectly fine english sentence. Maybe a bit complicated for
you to understand though.

I guess you meant "your" when you said "the".

If so then I will just say that I find this comment rude and
uniformed. I am not responsible for our current hash implementation,
nor do I think you could do any better. I am pretty sure that the only
way one can improve perls hash performance is to make it do less, and
that by doing so you would upset far more people than you would make
happy by making it a nanosecond faster.

But please, go ahead and prove me wrong. Post some patches.

so I don't know what you are trying to say.

I am trying to say that the solution for bad hash tables is better hash
tables, or a sense of proportion, not breaking perfectly fine existing
code.

I find it really hard to take you seriously given how often you make
silly statements like "not breaking perfectly fine existing code". If
your code broke because of this change it is because you were making
unwarranted assumptions about how perl's hash function operated, and
as such your code wasn't perfectly fine. In fact if your code is
breaking due to this change then I could trivially contrive code in an
older version of perl that would also break your assumptions.

So please reconsider your choice of making perl worse again.

I don't think I have made Perl worse, so currently I have nothing to reconsider.

To me, breaking existing code for questionable or no reason is making it
worse. It is obvious that you disagree and think you didn't make it worse,
but at best, you have to disagree with my perfectly valid opinion on this
topic-

I have to disagree? What? Did you mean "agree"?

Anyway, it doesnt matter. I don't break existing code for no reason.
A) I have reasons, B) any code broken was already broken but the
author did not know it C) there is a reason the docs say "The actual
random order is subject to change in future versions of perl" and make
no other guarantees that the limited ones I enumerated above. The
reason of course is to allow the hash function to be changed or
improved over time.

I am sure the perl5porters who are regular contributors will give your
opinions on how they use their time the full attention and consideration
that they deserve.

I am fully aware of them being volunteers, but so am I (and I am a regular
contributor as well), please keep that in mind.

Perhaps you are, I never advanced an opinion on the subject.

The perl5porters are not "better" than me as a major contributor to perl
in any way, and have no higher moral ground. Acting uppity because you are
part of a mailinglist and contribute more often than me is proving nothing
really.

I never made any comments about anyones individuals merits. Nor do I
know what I did that was "uppity".

I can't "the people who maintainer perl5" (as opposed to perl5porters) to
do what I want, and neither do I try, but sarcaasm and ignorance on your
part is wrong too. My concerns are valid, and while you have all the right
in the world to ignore them, or even ridicule them, that doesn't make them
invalid.

Well I have responded to the only cogent concern that I have seen you
express, which is that you think that these changes contradict the
documentation. The documentation does not support you in this regard.

Is there some other matter I should respond to?

Oh, I suppose I forgot. Afaik, you don't even have to fix this, just
take the patches I did to the JSON​::PP tests and apply them.

See​: https://rt.cpan.org/Public/Bug/Display.html?id=83421

Thanks for your feedback.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 12​:28​:54PM +0100, demerphq <demerphq@​gmail.com> wrote​:
+-> > >> > - Unless I am mistaken, this is to avoid a DOS attack. DOS attacks are easy
+-> > >> > to protect against, and this won't protect me against basically any form
+-> > >> > of attack, so it's useless. Far better methods exist that prevent every
+-> > >> > form of memory resource and/or cpu time starvation.
| > >>
| > >> You are mistaken.
| > >
| > > You are really claiming that this is one of the best methods to prevent
| > > memory resource and/or cpu time starvation? Because that's what the satteemnt
| > > you quoted says.
| >
| > I am sorry, I seem to be missing some context here. What statement did
| > I quote? I just checked this thread I did not quote anything.
|
+-- These ones (hope you can read my ascii art).

I interpreted it to mean to refer to the last one you quoted.

Now, if you didn't quote anything, what would your statement refer to?

The purpose of hash iterator randomization is make it harder, and
hopefully outright impossible to discover the hash seed a given
process is using.

That in itself is obviously a no-goal. No, I don't believe that is the real
purpose. It must be something else, like​: "to make it harder to use DOS
attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal, I can discover these hash seeds
with a debugger, or an XS module, and I doubt your change will stop me
from doing it (to the extent this still exists).

Once an attacker can determine the hash seed they can launch
algorithmic complexity attacks on our hash implementation. However the
iterator randomization does not in any way prevent a DOS, it prevents
obtaining the information that makes a DOS possible.

I think you are splitting hairs, but you get it wrong.

The purpose of your change is to make it harder to apply such DOS attacks.
Saying that's not the purpose, but the purpose is just to hide this
information is dishonest, because hiding this information isn't a useful
purpose in itself.

So it seems I was completely right after all.

Thank you for spelling it out for me to verify.

The perlfunc 5.16 documentation on "keys". Older versions are even more
explicit.

all them are ambiguous, but I already mentioned that.

Well, I beg to differ. I think you are conflating "does not say anything
on the subject" and "says something ambiguous".

I think it strongly implies it. Beat me if you wish, but your
interpretation is as good as mine, regardless of the intent behind it.

The documentation for keys(), and as far as I know all other
documentation relating to hashes says nothing about consistency,
either cross process or in between hashes.

Sorry, but "guaranteed to be the same as either the keys ..." clearly
implies some consistency, at least between those. Also saying the order
is the same for the "same hash" is ambiguous, as "same" has tow meanings,
namely, "identical" (of same identity) and "similar in some kind" (e.g.
same contents).

It says that the keys are returned in an apparently random order, that
the random order is subject to change in future versions of perl,

No problem with that.

the only guarantee of any form provided is that the keys() function
will return items in the same order that each() and values would return
them assuming the hash has not been modified (a secondary clause in the
documentation for "each" specifies it is safe to delete the key that was
just returned by each()).

It also says the ordering is the same for the same hash.

I know how it's meant, but I also know what is said, and assuming that
"the same hash will result in the same ordering" is not unreasonable.

Note that the earlier randomisation patch already broke the semantics, as
the ordering changed even within the same version (but different program
runs).

Furthermore I cannot find any evidence that the wording for this has
changed in terms of specificity in the lifetime of perlfunc.pod​:

Saying it changes in future versions but is the same for the same hash
STRONGLY implies that the ordering is the same for the same version of
perl, which is already not true.

I give you, as I said, that the wording is ambiguous, and that it is
actually an older change that broke it, but this interprettaion is
entirely valid, and I am not the only one who used it.

perlfunc.pod didnt exist prior to that commit.

And it clearly changed, if you took notice.

I see nothing ambiguous in the documentation, nor do I see it becoming
more or less specific over time.

Saying you don'T se something has no value whatsoever. People tend to chose
one interpretation of another.

You'd have to make the stronger claim that no other interprettaion is
reasonably possible, and I don't think you can, simply because "same", in
english, is ambiguous as to whether mean identity or equality.

I don't think that, and haven't made that claim. I think you can make
faster hashes with better properties without the drawbacks.

So then why did you bring it up in this thread? If you have issues
with hash performance and think you can do better then please start a
new thread, preferably by posting patches.

I don't think that, and haven't made that claim. I think you can make
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I said it before, and Is aid it again​: I didn't bring up what you put into my
mouth, and I explain it again what I meant, and this time, please either
believe what I write, or argue against it, but please don't attack strawmen​:

I think one can make faster and better hashes which fix this problem without
breaking code.

That means the breakage is not necessary.

- Taking out the inability to write a proper hash on all the perl developers
out there is just wrong - Perl is the runtime, and should get it right,
if possible.

This doesn't make grammatical sense,

I think it's a perfectly fine english sentence. Maybe a bit complicated for
you to understand though.

I guess you meant "your" when you said "the".

No, I specifically meant "the", because I didn't want to say "your".

If so then I will just say that I find this comment rude and
uniformed.

Since your assumption is wrong, your conclusion is irrelevant fprtunately.

I am not responsible for our current hash implementation,

And I didn't claim so. On purpose, in fact.

nor do I think you could do any better. I am pretty sure that the only
way one can improve perls hash performance is to make it do less, and
that by doing so you would upset far more people than you would make
happy by making it a nanosecond faster.

But please, go ahead and prove me wrong. Post some patches.

I already did some benchmarks with unordered hashes. I wonder what they
do less than perl hashes, specially now that almost all conssitency in
ordering is gone, which is usually the only problem.

I am trying to say that the solution for bad hash tables is better hash
tables, or a sense of proportion, not breaking perfectly fine existing
code.

I find it really hard to take you seriously given how often you make
silly statements like "not breaking perfectly fine existing code".

If you want me to respect your opinions on what is fine code, please
respect mine.

your code broke because of this change it is because you were making
unwarranted assumptions about how perl's hash function operated,

I don't know if they are unwarranted, but at least they were documented,
even if that turns out to be ambiguous wording.

as such your code wasn't perfectly fine. In fact if your code is
breaking due to this change then I could trivially contrive code in an
older version of perl that would also break your assumptions.

Well, the code in question (which defines my assumptions) works with older
versions of perl, so I call your bluff.

To me, breaking existing code for questionable or no reason is making it
worse. It is obvious that you disagree and think you didn't make it worse,
but at best, you have to disagree with my perfectly valid opinion on this
topic-

I have to disagree? What? Did you mean "agree"?

No, I mean "disagree".

You seem to be very combative, and I guess this is why you make so many
simple mistakes​: I usually really mean what I write (and even then, I do make
mistakes).

I _really_ mean that you disagree, and I really mean that you have to
disagree with me if you have another opinion.

Anyway, it doesnt matter. I don't break existing code for no reason.

Sure. I am saying they are bad reasons, and badly executed, not that it is
without reason.

A) I have reasons,

Never said anything to the contrary.

B) any code broken was already broken but the author did not know it

Empty claim.

C) there is a reason the docs say "The actual random order is subject to
change in future versions of perl"

The actual random ordering changes with each make test run, too, but the
testsuite still passes. That is cleatrly not the bug here.

and make no other guarantees that the limited ones I enumerated
above. The reason of course is to allow the hash function to be changed
or improved over time.

It says the ordering is the same for the same hash.

Changing the hash function doesn't break the code. Changing the hash
function within the same version of perl, and within the same process, is
what changes it.

At this point I must actually admit that I didn't even check that this is
true, I only assume it.

So if the ordering is really the same for the same hash, within the same
process (using the same perl), then I am indeed totally wrong.

Is that the case?

I am fully aware of them being volunteers, but so am I (and I am a regular
contributor as well), please keep that in mind.

Perhaps you are, I never advanced an opinion on the subject.

Well, I am. I tell you so. What are your reasons do doubt me so much?

The perl5porters are not "better" than me as a major contributor to perl
in any way, and have no higher moral ground. Acting uppity because you are
part of a mailinglist and contribute more often than me is proving nothing
really.

I never made any comments about anyones individuals merits. Nor do I
know what I did that was "uppity".

I interpreted your comment as irony, as am pretty confident it was meant
like it. If you really honetsly say you meant it as-is, I will, however,
believe you and admit my reaction to your last comment was wrong.

Well I have responded to the only cogent concern that I have seen you
express, which is that you think that these changes contradict the
documentation. The documentation does not support you in this regard.

Well, it clearly does. It's your interpretation that doesn't support mine,
and now the code.

Is there some other matter I should respond to?

You don't have to respond at all, but if you do so, I personally expect
you to use more than empty claims, and actually use evidence (which you
have done, for the most part).

Oh, I suppose I forgot. Afaik, you don't even have to fix this, just
take the patches I did to the JSON​::PP tests and apply them.

See​: https://rt.cpan.org/Public/Bug/Display.html?id=83421

But the bug is in the perl core. Nothing I apply can fix that, becasue I
can't apply fixes to the perl core.

Besides, it makes no sense to me to say "you don't have to fix that, just
apply the fix". What you mean is "you don't have to make the work to invent a
fix, it has already been done".

The problem is that I think the code is fine, and even if not (by sheer power
of force by you applying a buggy patch and claiming it's a good and
reasonable and corretc patch and my code is broken), I think it *should* be
correct, because taking away a useful feature that doesn't (in itself) gain
you anything

All I urge you, again, is to consider doing the "hardening" either better,
in a compatible way (which I think is possible), or consider the trade-off
between a rather ineffective patch (that doesn't shield against similar
DOS attacks) and breaking code that ought to work.

I don't have any strong hope for that, and think you will opt for the
ignore-and-break-it route, which is a valid way to proceed from your side
of things. I did feel the moral obligation to tell you why it is wrong,
and if you still do it, you at least do it in an informed way.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @tsee

On 03/21/2013 02​:00 PM, schmorp@​schmorp.de (via RT) wrote​:

I think one can make faster and better hashes which fix this problem without
breaking code.

That means the breakage is not necessary.

Alternative solutions, with implementation to prove their validity,
would be most welcome. Honestly.

You seem to be very combative, and I guess this is why you make so many
simple mistakes​: I usually really mean what I write (and even then, I do make
mistakes).

I _really_ mean that you disagree, and I really mean that you have to
disagree with me if you have another opinion.

Umm, who is splitting hair now?

You don't have to respond at all, but if you do so, I personally expect
you to use more than empty claims, and actually use evidence (which you
have done, for the most part).

So why imply otherwise in the first part of your sentence? Shove your
passive aggressiveness, Marc.

But the bug is in the perl core. Nothing I apply can fix that, becasue I
can't apply fixes to the perl core.

So let's rephrase​: If you care for JSON​::XS to continue working, then
you can work around what *you* consider a bug in perl. Many others sure
don't.

Besides, it makes no sense to me to say "you don't have to fix that, just
apply the fix". What you mean is "you don't have to make the work to invent a
fix, it has already been done".

Hair! In tiny fragments!

--Steffen

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @ribasushi

On Thu, Mar 21, 2013 at 06​:00​:56AM -0700, schmorp@​schmorp.de wrote​:

That in itself is obviously a no-goal. No, I don't believe that is the real
purpose. It must be something else, like​: "to make it harder to use DOS
attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal, I can discover these hash seeds
with a debugger, or an XS module, and I doubt your change will stop me
from doing it (to the extent this still exists).

Marc, you actually are misunderstanding here. The purpose of the changes
is to make it very hard to determine the hash seed by *remotely*
observing how the hash key order behaves. Think long running process
which exposes the hash key order of parameters submitted via a web
request.

There is no access to the running process itself, no XS, no debugging
tools. Just "black box" observation. The point of the changes is to make
such type of observations futile, nothing more.

Chers

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @Leont

On Thu, Mar 21, 2013 at 2​:00 PM, schmorp@​schmorp.de
<perlbug-followup@​perl.org> wrote​:

The purpose of hash iterator randomization is make it harder, and
hopefully outright impossible to discover the hash seed a given
process is using.

That in itself is obviously a no-goal. No, I don't believe that is the real
purpose. It must be something else, like​: "to make it harder to use DOS
attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal, I can discover these hash seeds
with a debugger, or an XS module, and I doubt your change will stop me
from doing it (to the extent this still exists).

Well, if you can insert an XS module or a debugger you've already got
control over the process, no need for a DOS there.

Sorry, but "guaranteed to be the same as either the keys ..." clearly
implies some consistency, at least between those. Also saying the order
is the same for the "same hash" is ambiguous, as "same" has tow meanings,
namely, "identical" (of same identity) and "similar in some kind" (e.g.
same contents).

IMO same hash clearly means same identity, not similarly valued (if
only because the latter is rather poorly defined).

Saying it changes in future versions but is the same for the same hash
STRONGLY implies that the ordering is the same for the same version of
perl, which is already not true.

It suggests this changed wasn't planned years ahead of time, nothing
more nothing less.

Saying you don'T se something has no value whatsoever. People tend to chose
one interpretation of another.

You'd have to make the stronger claim that no other interprettaion is
reasonably possible, and I don't think you can, simply because "same", in
english, is ambiguous as to whether mean identity or equality.

Alternative interpretations are unfortunate and sometimes something to
deal with, but they are not authoritative.

If you want me to respect your opinions on what is fine code, please
respect mine.

perl doesn't agree it's fine.

your code broke because of this change it is because you were making
unwarranted assumptions about how perl's hash function operated,

I don't know if they are unwarranted, but at least they were documented,
even if that turns out to be ambiguous wording.

as such your code wasn't perfectly fine. In fact if your code is
breaking due to this change then I could trivially contrive code in an
older version of perl that would also break your assumptions.

Well, the code in question (which defines my assumptions) works with older
versions of perl, so I call your bluff.

You're assuming that similar hashes always return their content in the
same order. This is already false in older perls in at least two
circumstances​:

1) The values don't have the same bucket-size
2) The rehash mechanism has kicked in
3) The hash is tied

Neither case is very likely in most code, but they can both
realistically happen.

But the bug is in the perl core. Nothing I apply can fix that, becasue I
can't apply fixes to the perl core.

Besides, it makes no sense to me to say "you don't have to fix that, just
apply the fix". What you mean is "you don't have to make the work to invent a
fix, it has already been done".

The problem is that I think the code is fine, and even if not (by sheer power
of force by you applying a buggy patch and claiming it's a good and
reasonable and corretc patch and my code is broken), I think it *should* be
correct, because taking away a useful feature that doesn't (in itself) gain
you anything

All I urge you, again, is to consider doing the "hardening" either better,
in a compatible way (which I think is possible), or consider the trade-off
between a rather ineffective patch (that doesn't shield against similar
DOS attacks) and breaking code that ought to work.

I don't have any strong hope for that, and think you will opt for the
ignore-and-break-it route, which is a valid way to proceed from your side
of things. I did feel the moral obligation to tell you why it is wrong,
and if you still do it, you at least do it in an informed way.

Well, one of the two has to change, and it is unlikely to be perl.

Leon

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 06​:45​:11AM -0700, Steffen Mueller via RT <perlbug-followup@​perl.org> wrote​:

That means the breakage is not necessary.

Alternative solutions, with implementation to prove their validity,
would be most welcome. Honestly.

Honestly, I do so much for perl, and after my previous experiences I don't
have the time to argue for ages again to get such a patch through. I am
not maintaining perl5, and I think when one can't do a job properly, it's
likely not worth doing it at all.

You seem to be very combative, and I guess this is why you make so many
simple mistakes​: I usually really mean what I write (and even then, I do make
mistakes).

I _really_ mean that you disagree, and I really mean that you have to
disagree with me if you have another opinion.

Umm, who is splitting hair now?

Obviously not me - would you do me a favour and just read the whole mail,
without picking things out of context and accusing me of splitting hairs?

He made a wrong assumption/accusation, and all I did was point out that
it's not true.

You don't have to respond at all, but if you do so, I personally expect
you to use more than empty claims, and actually use evidence (which you
have done, for the most part).

So why imply otherwise in the first part of your sentence?

I am not aware of implying anything otherwise. When it comes to it, it
likely only exists in your mind.

Shove your passive aggressiveness, Marc.

I honestly don't think I am passive aggressive, Steffen.

But the fact that you attack me and ignore the actual issue completely tells
me you are just trolling here.

I'd take somebody like me, who actually argues, any time over somebody
like you, who only insults.

Can you do better? Sure you can.

But the bug is in the perl core. Nothing I apply can fix that, becasue I
can't apply fixes to the perl core.

So let's rephrase​: If you care for JSON​::XS to continue working, then
you can work around what *you* consider a bug in perl. Many others sure
don't.

And many others sure do. Apart from trolling and personal attacks, do
you actually have anything to say on the actual issue? Why are you even
responding if you are incapable of contributing anything of value?

Besides, it makes no sense to me to say "you don't have to fix that, just
apply the fix". What you mean is "you don't have to make the work to invent a
fix, it has already been done".

Hair! In tiny fragments!

That's exactly what I was pointing out. Count yourself a genius for
realising it.

In any case, I won't reply to obvious troll attempts by you anymore.

I have no problem with a bit of drama, but please, make it worthwuoe by
adding *some* substance.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 06​:54​:52AM -0700, Peter Rabbitson via RT <perlbug-followup@​perl.org> wrote​:

Marc, you actually are misunderstanding here.

No, I don't.

The purpose of the changes
is to make it very hard to determine the hash seed by *remotely*
observing how the hash key order behaves.

First, that's not what yves said (but sure, he is wrong).

Secondly, if your ead my mail, it is painfully obviously thta I know that - I
pointed it out myself.

It was yves who disagrees and said it's purpose to to avoid discovering
the value, not me.

Think long running process which exposes the hash key order of
parameters submitted via a web request.

I am perfectly aware of that, thank you very much. Again, pleae do
ourselves a favour and actually read my mail fully, not in tiny and
misleading excerpts.

I even brought up an example of how to exploiut this using a DOS attack
originally.

There is no access to the running process itself, no XS, no debugging
tools. Just "black box" observation. The point of the changes is to make
such type of observations futile, nothing more.

I know. Please read the whole thread before making comments, thank you.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @demerphq

On 21 March 2013 15​:00, Leon Timmermans <fawaka@​gmail.com> wrote​:

You're assuming that similar hashes always return their content in the
same order. This is already false in older perls in at least two
circumstances​:

Thats 3. :-)

1) The values don't have the same bucket-size
2) The rehash mechanism has kicked in
3) The hash is tied

Neither case is very likely in most code, but they can both
realistically happen.

Ah but those are the rare examples. The common example is this​:

%copy= %hash;

Any items that collide will change order in %copy as compared to hash
in every version of perl prior to
3078e10 (ie a few days ago). With
that patch applied there is a very small chance that %copy would have
the same key order as %hash. Amazingly tiny small.

This demonstration should work on any perl prior to the new hash randomization​:

$ perl -le'my %hash=(13=>1,19=>1); print join " ", keys %hash; my
%copy= %hash; print join " ",keys %copy'
19 13
13 19

So in the old days it was not even unlikely that the two "similar"
hashes will have different orders, on the contrary, simply copying a
hash would almost guarantee a different key order.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @ribasushi

On Thu, Mar 21, 2013 at 03​:15​:27PM +0100, Marc Lehmann wrote​:

There is no access to the running process itself, no XS, no debugging
tools. Just "black box" observation. The point of the changes is to make
such type of observations futile, nothing more.

I know. Please read the whole thread before making comments, thank you.

I am confused... If you know there is no access to the running
interpreter, why did you bring up XS/debugging in the first place...?

Not trolling - honest question.

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @Tux

On Thu, 21 Mar 2013 15​:12​:33 +0100, Marc Lehmann <schmorp@​schmorp.de>
wrote​:

On Thu, Mar 21, 2013 at 06​:45​:11AM -0700, Steffen Mueller via RT <perlbug-followup@​perl.org> wrote​:

That means the breakage is not necessary.

Alternative solutions, with implementation to prove their validity,
would be most welcome. Honestly.

Honestly, I do so much for perl,

Name anything that I cannot live without?
Anything I should know about?

$ cat $HOME/.cpan/prefs/MLEHMAN.yml


comment​: "use common sense"
match​:
  distribution​: "^MLEHMAN"
disabled​: 1
$

Sorry, you never convinced me your contributions should change that

and after my previous experiences I don't have the time to argue for
ages again to get such a patch through.

And what makes you think we *do* have the time to argue?

I am not maintaining perl5, and I think when one can't do a job
properly, it's likely not worth doing it at all.

It is currently done properly and done well. By (very) competent people.
p5p currently manages monthly development releases and on-time
production releases that are secure and safe.

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.17 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 07​:02​:01AM -0700, Leon Timmermans via RT <perlbug-followup@​perl.org> wrote​:

In case you wonder why it is a no-goal, I can discover these hash seeds
with a debugger, or an XS module, and I doubt your change will stop me
from doing it (to the extent this still exists).

Well, if you can insert an XS module or a debugger you've already got
control over the process, no need for a DOS there.

Again, it was yves who brought this up, not me. I was merely arguing that
it's not the purpose of the change, as do you and others.

IMO same hash clearly means same identity, not similarly valued (if
only because the latter is rather poorly defined).

Your opinion is irrelevant. I am not claiming it's the only valid
interpretation, I am claiming it is a valid interpretation.

Same hahs clearly means "same hash". Ask some people by showing them​:

  my %a = (a => 1, b => 2);
  my %b = (a => 1, b => 2);

Are you sure that "clearly" %a and %b are not the same? Do you really think
that this wording is totally unambiguous.

I don't think so, but you can surprise me.

Note that you are conveneitnly ignoring the other arguments I made.

Documentation is mostly irrelevant. Even if documentation clearly said it
one way or another, there might be good and valid reasons to break it.

The documentation just explains why relying in this behaviour is not
unreasonable. It does not matter what the intended menaing is, or that there
are other interpretations.

I was pointing out that code that relies on this is not broken, or buggy,
or silly, or anything else the code was called to be, but reasonable.

Furthermore, I claim this is a useful feature, and breaking it shouldn't be
done lightly.

And rantily, I pointed out that the low regard for bakcwards compatibility
and continued breakage is a big drain, and, I think, not necessary.

Saying it changes in future versions but is the same for the same hash
STRONGLY implies that the ordering is the same for the same version of
perl, which is already not true.

It suggests this changed wasn't planned years ahead of time, nothing
more nothing less.

The documentation cleaims it's the same for the same hash, but the ordering
might change in future versions.

I think you are talking bullshit semantics here. A normal reader will
clearly imply that the algorithm (and ordering) is the same within the same
version of perl.

Which it was.

You'd have to make the stronger claim that no other interprettaion is
reasonably possible, and I don't think you can, simply because "same", in
english, is ambiguous as to whether mean identity or equality.

Alternative interpretations are unfortunate and sometimes something to
deal with, but they are not authoritative.

I never claimed they are authoritative. You (as a group) however claimed that
it's silly, wrong, unreasonable, invalid whatever.

And that isn't true. It's a reasonable interpretation, and a useful
feature that was implemented for a very long time.

It shouldn't be broken lightly.

If you want me to respect your opinions on what is fine code, please
respect mine.

perl doesn't agree it's fine.

development versions don't, all released versions do, by documentation and
execution both.

What you mean is that some developer(s) disagree. That is indeed fine, but
definitely not the same thing.

Well, the code in question (which defines my assumptions) works with older
versions of perl, so I call your bluff.

You're assuming that similar hashes always return their content in the
same order.

But I don't, and haven't said so. I claim the code in question is fine and
it is a reasonable tand useful to have it work.

Neither case is very likely in most code, but they can both
realistically happen.

All of these are not relevant to this discussion though.

I don't have any strong hope for that, and think you will opt for the
ignore-and-break-it route, which is a valid way to proceed from your side
of things. I did feel the moral obligation to tell you why it is wrong,
and if you still do it, you at least do it in an informed way.

Well, one of the two has to change,

And why would that be? You hold a gun to my face? You take away
maintainership?

and it is unlikely to be perl.

I think so, too.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 07​:18​:13AM -0700, yves orton via RT <perlbug-followup@​perl.org> wrote​:

This demonstration should work on any perl prior to the new hash randomization​:

It is also unrelated to the code in question.

So in the old days it was not even unlikely that the two "similar"
hashes will have different orders, on the contrary, simply copying a
hash would almost guarantee a different key order.

A pity, but not the code (and topic) in question here.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Fri, Mar 22, 2013 at 01​:20​:40AM +1100, Peter Rabbitson <rabbit-p5p@​rabbit.us> wrote​:

There is no access to the running process itself, no XS, no debugging
tools. Just "black box" observation. The point of the changes is to make
such type of observations futile, nothing more.

I know. Please read the whole thread before making comments, thank you.

I am confused... If you know there is no access to the running
interpreter, why did you bring up XS/debugging in the first place...?

Because I originally said it's about avoiding DOS attacks (and later
refining that to mean cpu/memory starvation), and yves said, no, it's
purpose is to make it hard to discover the seed, after which I pointed out
that this is unlikely the real purpose, because you can already discover
this seed in other ways (for example, by setting an env variable before
starting the program).

Now, the DOS attack is only an example of an algorithmic complexity attack
that this causes, but my guess is that coming up with something that isn't
a DOS attack, using this technique, that can't be prohibited by resource
limits and monitoring is hard to impossible.

So, if the difference between the DOS attack I was talking about, and the
algorithmic complexity attack is indeed more than just irrelevant (and I
have not seen any indication here), then I wonder if the same protections
wouldn't help here either.

There are all sorts of attacks one can launch remotely. It is good that perl
makes a lot of these hard or impossible, but perl cannot make all of these
hard and impossible.

It is always a trade-off between usefulness of the language and resistance
against such an attack.

If such a remote attack would be a significant danger and wouldn't be caught
by prudent uses of other mechanisms (resource control...), then I might agree
that, regardless of documentation or code, it is likely good to pay the price
of braking code.

But the evidence is lacking, and this looks more like blind action. I also
wonder why the hash itself couldn't be made hardneed - surely there are
hashes with (relatively) stable iterating behaviour and collision resistance.

The problem here is likely the over-reliance on power-of-two hashing
(which requires good hash algorithms, which historically has been a
problem) on the lack of scalable collision handling.

I hope I answered your question in detail.

Not trolling - honest question.

And surely I had no indications otherwise. I can differentiate between
people who contribute nothing except personal attacks and others.

It is unfortunate that Steffen feels he has to poison the atmosphere that
way, but this is clearly Steffen, and nobody else.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @ribasushi

On Thu, Mar 21, 2013 at 04​:03​:54PM +0100, Marc Lehmann wrote​:

On Fri, Mar 22, 2013 at 01​:20​:40AM +1100, Peter Rabbitson <rabbit-p5p@​rabbit.us> wrote​:

There is no access to the running process itself, no XS, no debugging
tools. Just "black box" observation. The point of the changes is to make
such type of observations futile, nothing more.

I know. Please read the whole thread before making comments, thank you.

I am confused... If you know there is no access to the running
interpreter, why did you bring up XS/debugging in the first place...?

Because I originally said it's about avoiding DOS attacks (and later
refining that to mean cpu/memory starvation), and yves said, no, it's
purpose is to make it hard to discover the seed, after which I pointed out
that this is unlikely the real purpose, because you can already discover
this seed in other ways (for example, by setting an env variable before
starting the program).

Now, the DOS attack is only an example of an algorithmic complexity attack
that this causes, but my guess is that coming up with something that isn't
a DOS attack, using this technique, that can't be prohibited by resource
limits and monitoring is hard to impossible.

So, if the difference between the DOS attack I was talking about, and the
algorithmic complexity attack is indeed more than just irrelevant (and I
have not seen any indication here), then I wonder if the same protections
wouldn't help here either.

There are all sorts of attacks one can launch remotely. It is good that perl
makes a lot of these hard or impossible, but perl cannot make all of these
hard and impossible.

It is always a trade-off between usefulness of the language and resistance
against such an attack.

If such a remote attack would be a significant danger and wouldn't be caught
by prudent uses of other mechanisms (resource control...), then I might agree
that, regardless of documentation or code, it is likely good to pay the price
of braking code.

But the evidence is lacking, and this looks more like blind action. I also
wonder why the hash itself couldn't be made hardneed - surely there are
hashes with (relatively) stable iterating behaviour and collision resistance.

The problem here is likely the over-reliance on power-of-two hashing
(which requires good hash algorithms, which historically has been a
problem) on the lack of scalable collision handling.

I hope I answered your question in detail.

You did, thank you. However I fail to understand how do you find
*apparent* hash key order stability for semantically identical [1]
hashes a good thing.

That is if a hash order is stable 99.9% of the time, and then in the
last 0.01% all hell breaks loose (because of the many assumptions that
the 99.9 == 100) - how is an average programmer supposed to debug that?

Could you elaborate?

[1] Just to make sure I understand - from what I gather this​:

  `perl -MData​::Dumper -e '$Data​::Dumper​::Indent = 0; print Dumper {(1..20)}; print "\n"'`

producing identical output on all perls between 5.8.2 and 5.16.3 is
something you consider a feature. You do not consider it a bug. Am I
correct?

Cheers

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 07​:24​:19AM -0700, "H. Merijn Brand via RT" <perlbug-followup@​perl.org> wrote​:

On Thu, Mar 21, 2013 at 06​:45​:11AM -0700, Steffen Mueller via RT <perlbug-followup@​perl.org> wrote​:

That means the breakage is not necessary.

Alternative solutions, with implementation to prove their validity,
would be most welcome. Honestly.

Honestly, I do so much for perl,

Name anything that I cannot live without?

And what is the relevance for this topic and why would I care?

Anything I should know about?

No, I honestly don't care if you use my packages.

The reason you stopped using my packages, as you yourself told me, is that
you wanted to force me to port JSON​::XS to some very old vendor compiler
that has been superseded, on a platform that was well-supported by gcc
(was it some old HP-UX maybe? I really liked that platform and worked many
years on it...).

I said no, and said it's not unreasonable to ask you to upgrade, because a
working compiler is available for your system.

I have no problem that you don't want to use my module(s) because of
that. I have few problems with you threatening me to "warn people" about
my software, and few problems about you doing posts like this one.

Again, everybody is *fine* to not use my software. This is something
that is apparently hard to udnerstand. I don't publish software for
self-gratification, I publish software because I hope it is useful for
others, and it clearly is for a LOT of people - there is no shortage of
feedback on that.

But I do consider your behaviour wholly unreasonable, and if the
requirement for you to use my software is that I port it to your pet
compiler because you threaten me, then you just have to live without it.

And the arrogance of trying to threaten me into doing your work for free
is breathtaking.

So, no, my modules are not for you, and you clearly don't deserve it -
after all, you are the person suffering most for that decision...

(But I of course have no problems with you using it, according to their
license).

In any case, folks, ranting is all fine and good, but why don't you at
last attempt to stay a bit on-topic?

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Fri, Mar 22, 2013 at 02​:15​:53AM +1100, Peter Rabbitson <rabbit-p5p@​rabbit.us> wrote​:

I hope I answered your question in detail.

You did, thank you. However I fail to understand how do you find
*apparent* hash key order stability for semantically identical [1]
hashes a good thing.

That is if a hash order is stable 99.9% of the time, and then in the
last 0.01% all hell breaks loose (because of the many assumptions that
the 99.9 == 100) - how is an average programmer supposed to debug that?

I don't find that usefil, I find the actual behaviour useful. Old perls
arrived at the same order when I built them in the same way each time.

New perls do it only when setting an env variable, or within one process -
fair enough, not grat, but managable.

[1] Just to make sure I understand - from what I gather this​:

`perl -MData​::Dumper -e '$Data​::Dumper​::Indent = 0; print Dumper {(1..20)}; print "\n"'`

producing identical output on all perls between 5.8.2 and 5.16.3 is

No, and I made it explicit that I am fine with that (and it's clearly
documented, at leats for me, since 5.8.1 or so).

I have a problem (or rather, my module has one) with this code not
producing identical output​:

  use feature 'say';

  my %a = (a => 1, b => 2);
  my %b = (a => 1, b => 2);

  say keys %a;
  say keys %b;

Besides, I didn't know these perls would produce identical output - but
they seem to, wow, I didn't expect that, but cool - though I wouldn't rely
on that.

You do not consider it a bug.

Are you sure anybody would consider it a bug to get the same output for the
same perl snippet on multiple versions of perl?

I would say that's the default expectation, don't you agree? At least not
without good reasons for them to differ.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @Tux

On Thu, 21 Mar 2013 16​:17​:20 +0100, Marc Lehmann <schmorp@​schmorp.de>
wrote​:

On Thu, Mar 21, 2013 at 07​:24​:19AM -0700, "H. Merijn Brand via RT" <perlbug-followup@​perl.org> wrote​:

On Thu, Mar 21, 2013 at 06​:45​:11AM -0700, Steffen Mueller via RT <perlbug-followup@​perl.org> wrote​:

That means the breakage is not necessary.

Alternative solutions, with implementation to prove their validity,
would be most welcome. Honestly.

Honestly, I do so much for perl,

Name anything that I cannot live without?

And what is the relevance for this topic and why would I care?

Anything I should know about?

No, I honestly don't care if you use my packages.

The reason you stopped using my packages, as you yourself told me, is that
you wanted to force me to port JSON​::XS to some very old vendor compiler

No, I wanted you to conform to C89, which is what the rest of perl
requires. I never expected you to take the time machine to work around
failing compilers of the stone-age (xlc on AIX-4 is a good example of a
compiler that *clain=ms* to be ANSI compliant but is not)

that has been superseded, on a platform that was well-supported by gcc
(was it some old HP-UX maybe? I really liked that platform and worked many
years on it...).

I used HP C-ANSI-C as an *example* to why your code should use ANSI
comments instead of c++ style comments. IIRC there were only two or
three minor issues that could be fixed easily without breaking
anything, but you refused in blaming me not to know the specs.

I said no, and said it's not unreasonable to ask you to upgrade, because a
working compiler is available for your system.

I think it is not reasonable to ask to upgrade a compiler that conforms
to the minimal requirements for building perl (and still does so up
till today​: I am about to release perl5.16.3 on that box soon).
Especially when there are no updates available. Of course one could
install another compiler, like GNU gcc, but remember that on that
architecture, it comes with quite a high performance penalty compared
to building with the native compiler. I don't think an overall
performance hit *ever* warrants the use of // comments in C/XS.

I have no problem that you don't want to use my module(s) because of
that. I have few problems with you threatening me to "warn people" about
my software, and few problems about you doing posts like this one.

Again, everybody is *fine* to not use my software. This is something
that is apparently hard to udnerstand. I don't publish software for
self-gratification, I publish software because I hope it is useful for
others, and it clearly is for a LOT of people - there is no shortage of
feedback on that.

Then how hard would it be to conform to C89 and let the rest of the
people that enjoy perl in?

But I do consider your behaviour wholly unreasonable, and if the
requirement for you to use my software is that I port it to your pet
compiler because you threaten me, then you just have to live without it.

When did I ever threaten you?

And the arrogance of trying to threaten me into doing your work for free
is breathtaking.

Hrmph. IIRC my quest came with a (simple) patch. there was NO work for
you involved AT ALL.

So, no, my modules are not for you, and you clearly don't deserve it -
after all, you are the person suffering most for that decision...

I'm not suffering at all. Moreover, I think I am quite amused by the
whole situation.

(But I of course have no problems with you using it, according to their
license).

In any case, folks, ranting is all fine and good, but why don't you at
last attempt to stay a bit on-topic?

I think I am, but you seem to misunderstand a lot of other people too.

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.17 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @demerphq

On 21 March 2013 14​:00, Marc Lehmann <schmorp@​schmorp.de> wrote​:

On Thu, Mar 21, 2013 at 12​:28​:54PM +0100, demerphq <demerphq@​gmail.com> wrote​:
+-> > >> > - Unless I am mistaken, this is to avoid a DOS attack. DOS attacks are easy
+-> > >> > to protect against, and this won't protect me against basically any form
+-> > >> > of attack, so it's useless. Far better methods exist that prevent every
+-> > >> > form of memory resource and/or cpu time starvation.
| > >>
| > >> You are mistaken.
| > >
| > > You are really claiming that this is one of the best methods to prevent
| > > memory resource and/or cpu time starvation? Because that's what the satteemnt
| > > you quoted says.
| >
| > I am sorry, I seem to be missing some context here. What statement did
| > I quote? I just checked this thread I did not quote anything.
|
+-- These ones (hope you can read my ascii art).

I interpreted it to mean to refer to the last one you quoted.

Now, if you didn't quote anything, what would your statement refer to?

Ah, right, I didn't consider that "quoting" even tho of course it is.

The purpose of hash iterator randomization is make it harder, and
hopefully outright impossible to discover the hash seed a given
process is using.

That in itself is obviously a no-goal. No, I don't believe that is the real
purpose. It must be something else, like​: "to make it harder to use DOS
attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal, I can discover these hash seeds
with a debugger, or an XS module, and I doubt your change will stop me
from doing it (to the extent this still exists).

Peter Rabbitson already replied to this. Here is the dialog that ensued​:

On 21 March 2013 16​:03, Marc Lehmann <schmorp@​schmorp.de> wrote​:

On Fri, Mar 22, 2013 at 01​:20​:40AM +1100, Peter Rabbitson <rabbit-p5p@​rabbit.us> wrote​:

There is no access to the running process itself, no XS, no debugging
tools. Just "black box" observation. The point of the changes is to make
such type of observations futile, nothing more.

I know. Please read the whole thread before making comments, thank you.

I am confused... If you know there is no access to the running
interpreter, why did you bring up XS/debugging in the first place...?

Because I originally said it's about avoiding DOS attacks (and later
refining that to mean cpu/memory starvation), and yves said, no, it's
purpose is to make it hard to discover the seed, after which I pointed out
that this is unlikely the real purpose, because you can already discover
this seed in other ways (for example, by setting an env variable before
starting the program).

You know that this is not about XS nor having access to a debugger,
nor the ability to set the environment var, yet bring it up anyway.

So you are trolling.

Once an attacker can determine the hash seed they can launch
algorithmic complexity attacks on our hash implementation. However the
iterator randomization does not in any way prevent a DOS, it prevents
obtaining the information that makes a DOS possible.

I think you are splitting hairs, but you get it wrong.

Your attitude makes me want to split hairs with you. You employ a
highly disagreeable style of communicating with people and should not
be surprised when they do not want to be helpful towards you.

The patch to make hash iterator traversal random is specifically to
make it harder for an attacker to determine the hash seed.

The purpose of your change is to make it harder to apply such DOS attacks.
Saying that's not the purpose, but the purpose is just to hide this
information is dishonest, because hiding this information isn't a useful
purpose in itself.

I haven't been in any way dishonest. There is nothing false about my
statement at all.

The perlfunc 5.16 documentation on "keys". Older versions are even more
explicit.

all them are ambiguous, but I already mentioned that.

Well, I beg to differ. I think you are conflating "does not say anything
on the subject" and "says something ambiguous".

I think it strongly implies it. Beat me if you wish, but your
interpretation is as good as mine, regardless of the intent behind it.

No, my interpretation is correct and yours is a figment of your
imagination. There is no equivalence there.

The documentation for keys(), and as far as I know all other
documentation relating to hashes says nothing about consistency,
either cross process or in between hashes.

Sorry, but "guaranteed to be the same as either the keys ..." clearly
implies some consistency, at least between those. Also saying the order
is the same for the "same hash" is ambiguous, as "same" has tow meanings,
namely, "identical" (of same identity) and "similar in some kind" (e.g.
same contents).

No, no, no. There is no consistency, there never has been, there never
will be, and arguing about it is unproductive.

It says that the keys are returned in an apparently random order, that
the random order is subject to change in future versions of perl,

No problem with that.

the only guarantee of any form provided is that the keys() function
will return items in the same order that each() and values would return
them assuming the hash has not been modified (a secondary clause in the
documentation for "each" specifies it is safe to delete the key that was
just returned by each()).

It also says the ordering is the same for the same hash.

Yes, as in identity. If you find that

keys(%hash)

returns items in a different order than

values(%hash)

does then please file a bug.

I know how it's meant, but I also know what is said, and assuming that
"the same hash will result in the same ordering" is not unreasonable.

The docs say "the same hash unmodified", so that implies very strongly
that it means identity.

Note that the earlier randomisation patch already broke the semantics, as
the ordering changed even within the same version (but different program
runs).

No, it didnt break any semantics. Stop making stuff up.

Furthermore I cannot find any evidence that the wording for this has
changed in terms of specificity in the lifetime of perlfunc.pod​:

Saying it changes in future versions but is the same for the same hash
STRONGLY implies that the ordering is the same for the same version of
perl, which is already not true.

No it doesn't.

I give you, as I said, that the wording is ambiguous, and that it is
actually an older change that broke it, but this interprettaion is
entirely valid, and I am not the only one who used it.

perlfunc.pod didnt exist prior to that commit.

And it clearly changed, if you took notice.

Not in any way relevant to this discussion, if you took notice.

I see nothing ambiguous in the documentation, nor do I see it becoming
more or less specific over time.

Saying you don'T se something has no value whatsoever. People tend to chose
one interpretation of another.

You'd have to make the stronger claim that no other interprettaion is
reasonably possible, and I don't think you can, simply because "same", in
english, is ambiguous as to whether mean identity or equality.

There is nothing here to argue about, you are wrong, and there your
interpretation is wrong, and that is all there is to it.

I don't think that, and haven't made that claim. I think you can make
faster hashes with better properties without the drawbacks.

So then why did you bring it up in this thread? If you have issues
with hash performance and think you can do better then please start a
new thread, preferably by posting patches.

I don't think that, and haven't made that claim. I think you can make
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Are you really so stupid as to claim you didnt say something only a
few lines after you say it?

Did you not say​:

"I think you can make faster hashes with better properties without the
drawbacks."

And was I not replying to you saying it?

I said it before, and Is aid it again​: I didn't bring up what you put into my
mouth, and I explain it again what I meant, and this time, please either
believe what I write, or argue against it, but please don't attack strawmen​:

I think one can make faster and better hashes which fix this problem without
breaking code.

That means the breakage is not necessary.

So prove it. Patches welcome.

- Taking out the inability to write a proper hash on all the perl developers
out there is just wrong - Perl is the runtime, and should get it right,
if possible.

This doesn't make grammatical sense,

I think it's a perfectly fine english sentence. Maybe a bit complicated for
you to understand though.

I guess you meant "your" when you said "the".

No, I specifically meant "the", because I didn't want to say "your".

If so then I will just say that I find this comment rude and
uniformed.

Since your assumption is wrong, your conclusion is irrelevant fprtunately.

I am not responsible for our current hash implementation,

And I didn't claim so. On purpose, in fact.

nor do I think you could do any better. I am pretty sure that the only
way one can improve perls hash performance is to make it do less, and
that by doing so you would upset far more people than you would make
happy by making it a nanosecond faster.

But please, go ahead and prove me wrong. Post some patches.

I already did some benchmarks with unordered hashes. I wonder what they
do less than perl hashes, specially now that almost all conssitency in
ordering is gone, which is usually the only problem.

Which just shows you really have no idea at all how Perls hashes work.

Which pretty much ends this discussion.

I am trying to say that the solution for bad hash tables is better hash
tables, or a sense of proportion, not breaking perfectly fine existing
code.

I find it really hard to take you seriously given how often you make
silly statements like "not breaking perfectly fine existing code".

If you want me to respect your opinions on what is fine code, please
respect mine.

I don't really give a shit about your opinion actually. Especially as
you frame it rudely, make personal aspersions, and act like a jerk
when you express it.

your code broke because of this change it is because you were making
unwarranted assumptions about how perl's hash function operated,

I don't know if they are unwarranted, but at least they were documented,
even if that turns out to be ambiguous wording.

As I said before, there is nothing ambiguous about saying nothing
about something. Ambiguity comes when there are more than one option,
here there are zero options. There is no reasonable interpretation of
the documentation that supports your position.

as such your code wasn't perfectly fine. In fact if your code is
breaking due to this change then I could trivially contrive code in an
older version of perl that would also break your assumptions.

Well, the code in question (which defines my assumptions) works with older
versions of perl, so I call your bluff.

So sure, I can make it break with a one line change.

To me, breaking existing code for questionable or no reason is making it
worse. It is obvious that you disagree and think you didn't make it worse,
but at best, you have to disagree with my perfectly valid opinion on this
topic-

I have to disagree? What? Did you mean "agree"?

No, I mean "disagree".

You seem to be very combative, and I guess this is why you make so many
simple mistakes​: I usually really mean what I write (and even then, I do make
mistakes).

You are rude and insulting in your manner, you should not be surprised
when people you communicate with are combative.

I _really_ mean that you disagree, and I really mean that you have to
disagree with me if you have another opinion.

Anyway, it doesnt matter. I don't break existing code for no reason.

Sure. I am saying they are bad reasons, and badly executed, not that it is
without reason.

A) I have reasons,

Never said anything to the contrary.

You certainly implied it with "breaking existing code for questionable
or no reason is making it worse".

B) any code broken was already broken but the author did not know it

Empty claim.

Ive already demonstrated how your assumptions are broken by a simple hash copy.

C) there is a reason the docs say "The actual random order is subject to
change in future versions of perl"

The actual random ordering changes with each make test run, too, but the
testsuite still passes. That is cleatrly not the bug here.

and make no other guarantees that the limited ones I enumerated
above. The reason of course is to allow the hash function to be changed
or improved over time.

It says the ordering is the same for the same hash.

And it is. (So long as you dont modify it by inserting new items.)

Changing the hash function doesn't break the code. Changing the hash
function within the same version of perl, and within the same process, is
what changes it.

We don't change the hash function during the process.

At this point I must actually admit that I didn't even check that this is
true, I only assume it.

Well you are wrong.

So if the ordering is really the same for the same hash, within the same
process (using the same perl), then I am indeed totally wrong.

Is that the case?

@​keys1= keys %hash;
@​keys2= keys %hash;

will always return identical @​keys1 and @​keys2 assuming %hash is a
"normal" perl hash.

I am fully aware of them being volunteers, but so am I (and I am a regular
contributor as well), please keep that in mind.

Perhaps you are, I never advanced an opinion on the subject.

Well, I am. I tell you so. What are your reasons do doubt me so much?

Perhaps you are, I never advanced an opinion on the subject.

But since you bring it up again I will note I checked the log and you
are mentioned a whole 6 times.

$ git log --grep "Marc Lehmann" --pretty=oneline
a5b310e Update Digest-SHA to CPAN version 5.83
d9a4b45 Restore building Encode's
subextensions for a static build.
50c1ac0 Pod nit in Encode.pm, found
by Marc Lehmann in RT #36949.
1a08a6a [perl #22141] patch for
Time​::HiRes to get rid of -lrt on linux From​: Marc Lehmann (via RT)
<perlbug-followup@​perl.org> Message-Id​:
<rt-22141-56710.3.69543054121962@​bugs6
5911668 In the UTF-8 branch of
crypt() the extra \0 byte is required, found by Marc Lehmann.
bf8afc6 Fix for a segfault, from Marc Lehmann.

The perl5porters are not "better" than me as a major contributor to perl
in any way, and have no higher moral ground. Acting uppity because you are
part of a mailinglist and contribute more often than me is proving nothing
really.

I never made any comments about anyones individuals merits. Nor do I
know what I did that was "uppity".

I interpreted your comment as irony, as am pretty confident it was meant
like it. If you really honetsly say you meant it as-is, I will, however,
believe you and admit my reaction to your last comment was wrong.

I meant every word in that sentence literally.

Well I have responded to the only cogent concern that I have seen you
express, which is that you think that these changes contradict the
documentation. The documentation does not support you in this regard.

Well, it clearly does. It's your interpretation that doesn't support mine,
and now the code.

There is no "clearly does" involved here. The documentation simply
does not say what you seem to think it says. If it did it would say it
explicitly, but given how perls hash works (which I do know, and which
you clearly do not), I am very very certain that no person that did
understand how perls hashes work would ever say what you think the
documentation says.

Is there some other matter I should respond to?

You don't have to respond at all, but if you do so, I personally expect
you to use more than empty claims, and actually use evidence (which you
have done, for the most part).

Oh, I suppose I forgot. Afaik, you don't even have to fix this, just
take the patches I did to the JSON​::PP tests and apply them.

See​: https://rt.cpan.org/Public/Bug/Display.html?id=83421

But the bug is in the perl core. Nothing I apply can fix that, becasue I
can't apply fixes to the perl core.

Excuse me? No the bug is in your code, expecting to snapshot a
serialized hash and have that be the same as some other serialized
hash with the same keys. There has never been any reasonable basis to
expect this to always happen.

Besides, it makes no sense to me to say "you don't have to fix that, just
apply the fix". What you mean is "you don't have to make the work to invent a
fix, it has already been done".

If you prefer that wording then so be it.

The problem is that I think the code is fine, and even if not (by sheer power
of force by you applying a buggy patch and claiming it's a good and
reasonable and corretc patch and my code is broken), I think it *should* be
correct, because taking away a useful feature that doesn't (in itself) gain
you anything

All I urge you, again, is to consider doing the "hardening" either better,
in a compatible way (which I think is possible), or consider the trade-off
between a rather ineffective patch (that doesn't shield against similar
DOS attacks) and breaking code that ought to work.

I don't have any strong hope for that, and think you will opt for the
ignore-and-break-it route, which is a valid way to proceed from your side
of things. I did feel the moral obligation to tell you why it is wrong,
and if you still do it, you at least do it in an informed way.

Post some patches or shut up. I dont care what your opinion is on this
subject anymore.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 08​:36​:34AM -0700, "H. Merijn Brand via RT" <perlbug-followup@​perl.org> wrote​:

IIRC there were only two or three minor issues that could be fixed
easily without breaking anything, but you refused in blaming me not to
know the specs.

I don't remember it that way, sorry. It wasn't just minor, and as people here
love to say "where's your patch"? Not that I would likely apply it, for
reasons stated.

I think it is not reasonable to ask to upgrade a compiler that conforms
to the minimal requirements for building perl (and still does so up

Right, asking people to write their C++ or Fortran code in C is not
reasonable, because you say so. Your statement is wrong in general, and wrong
in particular.

But well, you are entitled to your opinion and not using my module, but
what you do is way out of proportions. But again, your choice.

Then how hard would it be to conform to C89 and let the rest of the
people that enjoy perl in?

The rest of the people happily enjoy it, last I heard from them.

But I do consider your behaviour wholly unreasonable, and if the
requirement for you to use my software is that I port it to your pet
compiler because you threaten me, then you just have to live without it.

When did I ever threaten you?

In your mail merijin. And now you execute your threat (probably not the
first time, but I am just speculating). In any case, feel free.

And the arrogance of trying to threaten me into doing your work for free
is breathtaking.

Hrmph. IIRC

You don't.

my quest came with a (simple) patch. there was NO work for

It did not.

And making up bullshit like "it was NO work AT ALL" doesn't make it
better. It would have been no work for you, but it would be continued work
for me, because I take maintaining my package seriously.

If it was no work at all, why didn't you do it? It's free software after
all.

The lack of respect you show for the work others have done for you to use,
freely is staggering.

In reality, you are just whining because I don't support your pet
compiler. But it really is my decision on where I set the cut-off point
of how antique language standards I support, and you are entitled to do it
better, or not use my stuff. Complaining like you do is just whining.

What's next, I need to support perl 5.001?

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From schmorp@schmorp.de

On Thu, Mar 21, 2013 at 12​:01​:13PM -0700, "Deven T. Corzine via RT" <perlbug-followup@​perl.org> wrote​:

Just because a word has multiple meanings does not make them synonymous.

Exactly, the meanings are not synonymous.

However, I agree with Andy that this thread is not productive anymore.

I gave my arguments, and was mostly ignored without hearing others.

If you want a reply from me, you have to write to me privately, so I can
see your e-mail address, and I will consider responding tot he best of my
abilities.

This is true for anybody else - keep in mind, I didn't post to the list
and don't receive your original mail, so can't reply to you privately.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 21, 2013

From @rurban

On Thu, Mar 21, 2013 at 8​:44 AM, Steffen Mueller <smueller@​cpan.org> wrote​:

On 03/21/2013 02​:00 PM, schmorp@​schmorp.de (via RT) wrote​:

I think one can make faster and better hashes which fix this problem
without
breaking code.

That means the breakage is not necessary.

Alternative solutions, with implementation to prove their validity, would be
most welcome. Honestly.

For the public facing cPanel binaries I solved the problem by using
the 5.8.1 hash approach
to randomize the seed always at init-time, not just when extending the hash.
This is much simpler, and does break less code than the 5.8.2 (and the
now fixed 5.18)
approach to randomize on hsplit.

Of course a collision-free hash without the fallback to a linked list
would be better,
but I'm not a hash expert.
And since p5p did not appreciate my ideas of const hashes and making
them perfect (faster access) I go with always randomized, but being
consistent at run-time.

The typed/const branch is at my github, the 5.8.1 patch for 5.6 is on
the p5p ML.
--
Reini Urban
http​://cpanel.net/ http​://www.perl-compiler.org/

@p5pRT
Copy link
Author

p5pRT commented Mar 22, 2013

From @bulk88

On Thu Mar 21 07​:02​:00 2013, LeonT wrote​:

On Thu, Mar 21, 2013 at 2​:00 PM, schmorp@​schmorp.de
<perlbug-followup@​perl.org> wrote​:

The purpose of hash iterator randomization is make it harder, and
hopefully outright impossible to discover the hash seed a given
process is using.

That in itself is obviously a no-goal. No, I don't believe that is
the real
purpose. It must be something else, like​: "to make it harder to use
DOS
attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal, I can discover these hash
seeds
with a debugger, or an XS module, and I doubt your change will stop
me
from doing it (to the extent this still exists).

Well, if you can insert an XS module or a debugger you've already got
control over the process, no need for a DOS there.

No need for XS or a debugger, "unpack('P'" and "(\%hash)+0" and you read
all process memory. With the my_perl being in memory right before every
PV buffer in the process, because of struct perl_memory_debug_header,
you can find out anything you want from Pure Perl. Getting such code to
be cross-perl version and cross platform safe is to much work to
seriously use the idea.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Mar 22, 2013

From @iabyn

On Thu, Mar 21, 2013 at 04​:52​:49PM +0100, Marc Lehmann wrote​:

The lack of respect you show for the work others have done for you to use,
freely is staggering.

Marc, I think you have beautifully summed up why so many of the people who
actively contribute to the perl 5 interpreter dislike your posts so
much.

--
You're only as old as you look.

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From @ikegami

On Thu, Mar 21, 2013 at 9​:00 AM, schmorp@​schmorp.de <
perlbug-followup@​perl.org> wrote​:

Also saying the order
is the same for the "same hash" is ambiguous, as "same" has tow meanings,
namely, "identical" (of same identity) and "similar in some kind" (e.g.
same contents).

"Same hash" is not used in the documentation. The documentation says "a
hash". If you have *two* hashes with similar values, you clearly don't have
*a* hash. Your ambiguity is manufactured.

Note that the earlier randomisation patch already broke the semantics, as

the ordering changed even within the same version (but different program
runs).

No semantics were changed. It's always been the case that C<keys> could
return two different list of keys for two hashes with identical keys. Even
before 5.8.1.

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From @ikegami

On Sat, Mar 23, 2013 at 5​:00 AM, Eric Brine <ikegami@​adaelis.com> wrote​:

It's always been the case that C<keys> could return two different list of
keys for two hashes with identical keys. Even before 5.8.1.

For example,

my %a = map { $_ => 1 } 'a'..'g';
my %b = map { $_ => 1 } 'a'..'g';
$b{h} = 1;
delete $b{h};

print keys(%a), "\n"; # ecagbdf
print keys(%b), "\n"; # eadcbgf

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From schmorp@schmorp.de

On Fri, Mar 22, 2013 at 11​:34​:15AM -0700, Dave Mitchell via RT <perlbug-followup@​perl.org> wrote​:

On Thu, Mar 21, 2013 at 04​:52​:49PM +0100, Marc Lehmann wrote​:

The lack of respect you show for the work others have done for you to use,
freely is staggering.

Marc, I think you have beautifully summed up why so many of the people who
actively contribute to the perl 5 interpreter dislike your posts so
much.

Besides, even if it were true - look at whom you cc'ed. Yep, the person who,
long ago, showed definite lack of respect for my work and tried to threaten
me to do his work for him, for free, while treading me, well, "unfriendly" to
use an euphemism.

So, while I respect the work of others, I wouldn't be surprised if that
respect would be slowly eroded by the behaviour I get in return. Why
measure me with higher standards than others?

If at all, I should be praised for not going down to that level, but I
make no illusions of that (and I don't like, nor care, for praise).

If you have any grain of integrity in you, then think about that. Maybe
even research history a bit. Then you will see I am right.

Sure that sounds arrogant, but after people telling me my code is shoddy
because investing more than half my waking life into it is not enough
(while they invest a lot less, and with far worse results) and that I
have to support every single compiler out there, no matter how old and
superceded it is, there is little left to me.

I will continue to provide quality software that is useful for an
enourmous number of perl users despite what some freaks think of it. I'm
doing it for the users, not for them.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From schmorp@schmorp.de

On Fri, Mar 22, 2013 at 11​:34​:15AM -0700, Dave Mitchell via RT <perlbug-followup@​perl.org> wrote​:

I am sorry that my last reply has gone to perlbug-followup@​perl.org -
I take the suggestion that this thread is dead seriously, but others
obviously don't, and I hit reply without checking the address, so don't
expect a public reaction (unless in error...).

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From @iabyn

On Sat, Mar 23, 2013 at 12​:07​:51PM +0100, Marc Lehmann wrote​:

On Fri, Mar 22, 2013 at 11​:34​:15AM -0700, Dave Mitchell via RT <perlbug-followup@​perl.org> wrote​:

On Thu, Mar 21, 2013 at 04​:52​:49PM +0100, Marc Lehmann wrote​:

The lack of respect you show for the work others have done for you to use,
freely is staggering.

Marc, I think you have beautifully summed up why so many of the people who
actively contribute to the perl 5 interpreter dislike your posts so
much.

Besides, even if it were true

You seem uncertain. I can assure you it is true.

- look at whom you cc'ed. Yep, the person who,
long ago, showed definite lack of respect for my work and tried to threaten
me to do his work for him, for free, while treading me, well, "unfriendly" to
use an euphemism.

Lets look at

  https://rt.cpan.org/Public/Bug/Display.html?id=42462

He made a bug report about your code not supporting older compilers
(admittedly not phrased well), and provided a small patch. Let's
look at your very first reply​:

  what a bunch of bullshit claims, the code is of course fully ANSI-C
  compliant, get a good book about C, or read the standards document.
  and tone down your tone before you make such idiotic claims, you'll
  only make yourself look like an idiot.

Straight off the bat, rude and attacking the *person*.

But I am more concerned with your initial response to this hash thread.
You got a message showing that a recent change to blead had caused some of
the tests in one of your modules to fail. Your response was a complete
rant. You made no attempt to enquire about the background to the changes,
you just charged in and accused everyone of being completely wrong, then
ranted about how we spend all our time breaking everything in perl. So, I
would repeat, "The lack of respect you show for the work others have done
for you to use, freely is staggering."

If at all, I should be praised for not going down to that level, but I
make no illusions of that (and I don't like, nor care, for praise).

But you immediately rush to the lowest possible, most combative level, at
every available opportunity. As I have shown above.

If you have any grain of integrity in you, then think about that. Maybe
even research history a bit. Then you will see I am right.

Ah, after one post, already questioning my integrity.
I am familiar with your postings to p5p over the years. I think you are
wrong.

I will continue to provide quality software that is useful for an
enourmous number of perl users despite what some freaks think of it. I'm
doing it for the users, not for them.

I will continue providing a quality perl interpreter for an enormous
number of perl users despite what what you think of it.

--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From schmorp@schmorp.de

On Sat, Mar 23, 2013 at 01​:02​:03PM +0000, Dave Mitchell <davem@​iabyn.com> wrote​:

Dave, you took my *private* reply and again posted it to the list. Unlike
my mistake, this was hardly a simple mistake by you, because all I did is
reply to your original mail, while you *manually* added the list address
back. (If anybody is confused​: I sent two mails, and for the first, I
manually edited out the list, and that is the one he reposted publicly,
while for the second, I forgot about the list and accidentally left it
in).

Since you are dragging this back to the list *again*, I need to answer it
here.

No thanks for making this list an even worse place, but I am sure you feel
like some angel of justice or so, even though socially your behaviour is
that of a troll.

Lets look at

https://rt.cpan.org/Public/Bug/Display.html?id=42462

He made a bug report about your code not supporting older compilers
(admittedly not phrased well), and provided a small patch.

There was no patch as has been pointed out before. Making up lies like
these just makes clear to me that you are not honest.

Straight off the bat, rude and attacking the *person*.

That's not "straight off the bat", that's a reply to a rude and
unnecessary mail by merjin, and not the first one either.

You might claim I have little tolerance to bullshit, but looking at only
part of the mail exchange and then making conclusions is unwarranted.

It's merely an attempt at character assassination.

rt.cpan.org *is* a common place to go to when the maintainer refuses some
change, in the hope of reaching more publicity, or other people. It is,
however, creating unnecesary extra work, and the only thing you can blame
me for is not to be dishonest because when I grow impatient I don't give
people a facade of politically correct bullshit but clearly tell them it's
wrong what they are doing.

The problems with, in general, rt.cpan.org are a whole different chapter
that has been topic here already.

You got a message showing that a recent change to blead had caused some of
the tests in one of your modules to fail. Your response was a complete
rant.

It definitely was a rant (as I wrote myself), but while I have no clue
what you mean with "complete rant", you are probably wrong about that.

The continued breakage in recent years, paired with bad decisions,
honestly made me annoyed with how perl is maintained these days - there
is little catering to stability, while at the same time, introducing
incompatible and/or badly designed extensions in a rush.

You made no attempt to enquire about the background to the changes,

I was well informed about the problem as-is (the problem itself is old),
and as it turned out, I was right.

What exactly should I have attempted, and what exactly would that have
changed? Was I factually wrong about anything related to that change?

No, I wasn't, so you implying that I would have had to enquire more first
is just dishonest.

you just charged in and accused everyone of being completely wrong,

No, you confuse me with yves. I charged in and made arguments on why this
is wrong.

Most everybody else (but not vereybody) charged in and accused me of being
completely wrong, without ever giving even a shred of evidence of it. Some
people then broguht up unrelated stuff ("yeha, I don't like this module
either because of") as if it had anything to do with the issue.

At least the current perl maintainer agreed with me on my points (that the
wording is ambiguous and that a better patch without breakage would be
preferred). And quite a lot of module maintainers seem to have fallen into
the same trap.

ranted about how we spend all our time breaking everything in perl.

Making this up doesn't make it true. Nowhere did I say or imply that, as
you you are well aware. You are a liar, which to me is the worst thing
really.

So, I would repeat, "The lack of respect you show for the work others
have done for you to use, freely is staggering."

And I would answer that you lie in public, not even stopping from
reposting private replies, just to get more attention for your lies.

If at all, I should be praised for not going down to that level, but I
make no illusions of that (and I don't like, nor care, for praise).

But you immediately rush to the lowest possible, most combative level,

That's not true. I might be dumb enough to only leave traces of that
publicly, but that doesn't make it so.

at every available opportunity. As I have shown above.

You haven't shown even a single case (because your conclusion is based on
lies you amde up and wrong information).

But hey, even two cases makes "every available opportunity".

The staggering amount of bullshit that you pull out of your arse is
staggering.

If you have any grain of integrity in you, then think about that. Maybe
even research history a bit. Then you will see I am right.

Ah, after one post, already questioning my integrity.

Another lie, I didn't question your integrity, I told you that if you have
some, then you will do some better research.

Didn't happen, so now I do question your integrity.

I will continue to provide quality software that is useful for an
enourmous number of perl users despite what some freaks think of it. I'm
doing it for the users, not for them.

I will continue providing a quality perl interpreter for an enormous
number of perl users despite what what you think of it.

Reality check​: "providing a perl interpreter". Well, I provide multiple
perl interpreters, while I doubt you currently provide any perl
interpreter to anybody.

There is no doubt that you worked a lot more on perl than me and most
people on this plane, but accusing me of disrespect and then disregarding
the masses of people who worked on perl looks hipocritical to me.

I think you have a hyperinflated ego.

But that's fine. Spreading lies like a machine is not. If don't think I
cna respect you anymore.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From @Tux

On Sat, 23 Mar 2013 15​:32​:07 +0100, Marc Lehmann <schmorp@​schmorp.de>
wrote​:

That's not "straight off the bat", that's a reply to a rude and
unnecessary mail by merjin, and not the first one either.

Somehow you cannot bring up the decency and respect to write my name as
it is supposed to be written.

Furthermore, I have never ever had the intention to be rude. The
rudeness you read is purely in your perception. If any of my mails have
ever come as a threat or rude to you, I can do nothing but apologize,
and I will blame my improper use of English language, which is not my
native tongue.

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.17 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From schmorp@schmorp.de

On Sat, Mar 23, 2013 at 07​:41​:48AM -0700, "H. Merijn Brand via RT" <perlbug-followup@​perl.org> wrote​:

That's not "straight off the bat", that's a reply to a rude and
unnecessary mail by merjin, and not the first one either.

Somehow you cannot bring up the decency and respect to write my name as
it is supposed to be written.

I honestly apologize for that - it definitely wasn't on purpose, and had
nothing to do with lack of respect for your name. I of course don't have
any big respect for you, but that wouldn't cause me to mistype your name
on purpose.

You could accuse me of not being diligent enough, and that is true,
obviously I fucked it up, more than once, after almost vowing to do
better, and I really didn't plan this, and apologise, whether you believe
me or not.

I consider mistyping a name on purpose as childish, and would not do so,
and have never done so in my life.

Merijn, Merijn, Merijn. Indeed, hard to type for me, and I guess I'll have
to copy&paste as to not fuck up again.

Furthermore, I have never ever had the intention to be rude.

Neither me then.

rudeness you read is purely in your perception.

Same here then. I always try to reply in form.

If any of my mails have ever come as a threat or rude to you

Wholeheartedly accepted. You might be pleased to know that I did invest
an effort to remove most C99 features from JSON​::XS in the last year, and
that it compiles even with MSVC6 now. It still won't compile with your
compiler(s), but at least it's much nearer now. Ok, since nothing changed
for you, you might not be pleased. Whatever.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From @iabyn

On Sat, Mar 23, 2013 at 03​:32​:07PM +0100, Marc Lehmann wrote​:

Dave, you took my *private* reply and again posted it to the list. Unlike
my mistake, this was hardly a simple mistake by you, because all I did is
reply to your original mail, while you *manually* added the list address
back.

Completely untrue. I took your public email​:

  Message-ID​: <20130323110751.GB2845@​schmorp.de>
  To​: Dave Mitchell via RT <perlbug-followup@​perl.org>

and made a public reply to it, to <perlbug-followup@​perl.org>.
I subsequently saw your follow-up email where you said you had made
that email public by mistake; if I had seen that in time, I would have
manually altered my reply to be private.

You sent a second email to me privately, which I replied to privately​:

  Message-ID​: <20130323131406.GE2409@​iabyn.com>
  To​: Marc Lehmann <schmorp@​schmorp.de>
  In-Reply-To​: <20130323110243.GA2845@​schmorp.de>

At no point did I change any of the To/CC fields manually.

Since you are dragging this back to the list *again*, I need to answer it
here.

You imply that I have made private threads public at least twice. I have
n fact done so zero times.

No thanks for making this list an even worse place, but I am sure you feel
like some angel of justice or so, even though socially your behaviour is
that of a troll.

No, I try very hard to be polite and accommodating on public lists,
although like all humans, I have bad days. You have a consistent pattern of
being inflammatory and showing no consideration for others.

I have never *once* made a deliberately trolling post in my 30 years online.

Lets look at

https://rt.cpan.org/Public/Bug/Display.html?id=42462

He made a bug report about your code not supporting older compilers
(admittedly not phrased well), and provided a small patch.

There was no patch as has been pointed out before.

It was clearly and unambiguously a patch. It wasn't in diff format, but it
was clearly indicating that to make your code more portable, you could
change the line​:

  HE *hes [count];
to
  #if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
  HE **hes = Perl_malloc (count * sizeof (HE));
  #else
  HE *hes [count];
  #endif

I have no opinion as to whether it was a good or complete patch, but it
was a patch.

Making up lies like
these just makes clear to me that you are not honest.

A classic Marc-ism. Take a slight argument over semantics and immediately
accuse your opponent of lying.

Straight off the bat, rude and attacking the *person*.

That's not "straight off the bat", that's a reply to a rude and
unnecessary mail by merjin, and not the first one either.

You might claim I have little tolerance to bullshit, but looking at only
part of the mail exchange and then making conclusions is unwarranted.

I am just going on the evidence that was presented to me​: a bug report
that was replied to in an astonishingly rude manner.

It's merely an attempt at character assassination.

No, its just an accurate comment​: it was your initial reply to the ticket,
it was rude, and it attacked the person.

You got a message showing that a recent change to blead had caused some of
the tests in one of your modules to fail. Your response was a complete
rant.

It definitely was a rant (as I wrote myself), but while I have no clue
what you mean with "complete rant", you are probably wrong about that.

By that, I meant nearly everything in your initial email contained broad,
sweeping criticisms.

You made no attempt to enquire about the background to the changes,

I was well informed about the problem as-is (the problem itself is old),
and as it turned out, I was right.

What exactly should I have attempted, and what exactly would that have
changed? Was I factually wrong about anything related to that change?

The hash issue has seen extensive discussions recently, both in the
general community, in the public perl community, and in private
discussions on the perl security list. Set against that you glibly state
that there are far better ways to aviod DoS, and that yves work is
"useless". But you don't don't go on to explain what these better methods
are, or how they they will always be better than hardening the hash code.

So you weren't entering into a constructive dialogue, you were just bashing.

No, I wasn't, so you implying that I would have had to enquire more first
is just dishonest.

You didn't appear to be aware the technical details of the motivation
behind the insertion order perturbation (avoiding leaking details of the
hash seed).

And there you immediately leap in with another ad Homenem. A difference of
opinion as to whether you were well-enough informed is immediately
"dishonest".

At least the current perl maintainer agreed with me on my points (that the
wording is ambiguous and that a better patch without breakage would be
preferred).

He also stated that the change "has been in progress for quite some time,
and has been thoroughly discussed, and it is going to stay in.

ranted about how we spend all our time breaking everything in perl.

Making this up doesn't make it true. Nowhere did I say or imply that, as
you you are well aware. You are a liar, which to me is the worst thing
really.

To quote from your original email​:

"the current regime of perl development - making perl worse,
breaking CPAN more and more, adding lots of incompatible changes while at
the same time requiring use feature is in the wrong for quite some time
now, and, frankly, the only reason I don't fork perl is simply that I
am not prepared to maintain yet another software package. Breaking CPAN
packages every 6 months just doesn't cut it for me"

My one-line summary may have been a touch hyperbolic, but I think it
fairly accurately implies that you made a large, sweeping claim about
perl's current development; one which, by the way, I completely
disagree with.

Once again you accuse me of being a liar. Doing a personal attack appears
to always your first choice. I think it perfectly clear that I have not
lied a single time in my emails in this thread.

So, I would repeat, "The lack of respect you show for the work others
have done for you to use, freely is staggering."

And I would answer that you lie in public, not even stopping from
reposting private replies, just to get more attention for your lies.

And as I already pointed out, I did not reply in public to a private
email. But unlike you, I am prepared to accept that you may have been
mistaken, and will not call you a liar.

Oh, and please stop calling me a liar. It is unjustified and deeply
offensive

If at all, I should be praised for not going down to that level, but I
make no illusions of that (and I don't like, nor care, for praise).

But you immediately rush to the lowest possible, most combative level,

That's not true. I might be dumb enough to only leave traces of that
publicly, but that doesn't make it so.

Your lack of self-awareness is astonishing. At the first sign of
disagreement, you immediately attack your opponent​: "liar", "troll",
"idiot", etc.

You haven't shown even a single case (because your conclusion is based on
lies you amde up and wrong information).

There you go again.

The staggering amount of bullshit that you pull out of your arse is
staggering.

Oh look, there you go again.

If you have any grain of integrity in you, then think about that. Maybe
even research history a bit. Then you will see I am right.

Ah, after one post, already questioning my integrity.

Another lie, I didn't question your integrity, I told you that if you have
some, then you will do some better research.

There you go again.

Didn't happen, so now I do question your integrity.

There you go again

I will continue to provide quality software that is useful for an
enourmous number of perl users despite what some freaks think of it. I'm
doing it for the users, not for them.

I will continue providing a quality perl interpreter for an enormous
number of perl users despite what what you think of it.

Reality check​: "providing a perl interpreter". Well, I provide multiple
perl interpreters, while I doubt you currently provide any perl
interpreter to anybody.

So the 5.14.3 tarball just spontaneously self-assembled?

There is no doubt that you worked a lot more on perl than me and most
people on this plane, but accusing me of disrespect and then disregarding
the masses of people who worked on perl looks hipocritical to me.

You are an intelligent person. I don't for one *second* believe you you
read my sentence as claiming I am the only person working on the perl
interpreter, or that only my contributions matter.

I think you have a hyperinflated ego.

Oh look, there you go again.

Just for the record, since attacking your opponent seem De Rigour in
Marc's world, I would just like to say that I think you are a deeply
unpleasant person.

But that's fine. Spreading lies like a machine is not. If don't think I
cna respect you anymore.

Once again I will point out that I have not lied a single time.

This is my final communication with you.

--
Dave's first rule of Opera​:
If something needs saying, say it​: don't warble it.

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From vadim.konovalov@alcatel-lucent.com

It was clearly and unambiguously a patch. It wasn't in diff
format, but it
was clearly indicating that to make your code more portable, you could
change the line​:

HE \*hes \[count\];

to
#if defined(__BORLANDC__) || defined(_MSC_VER) ||
defined(__STDC__)
HE **hes = Perl_malloc (count * sizeof (HE));
#else
HE *hes [count];
#endif

1)
an overlook - it should be
  HE **hes = Perl_malloc (count * sizeof (HE*));

it appears that HE is larger than HE* and allocating more memory, hence
misbehaviour is not noticed, but anyway, this isn't correct

2) when that chunk of memory freed? TIA :)

@p5pRT
Copy link
Author

p5pRT commented Mar 23, 2013

From @demerphq

On 21 March 2013 06​:42, Andreas J. Koenig via RT
<perlbug-followup@​perl.org> wrote​:

# New Ticket Created by (Andreas J. Koenig)
# Please include the string​: [perl #117239]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >

git bisect
----------
commit 0e0ab62
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Sun Mar 17 20​:19​:09 2013 +0100

Harden hashes against hash seed discovery by randomizing hash iteration

sample fail report
------------------
http​://www.cpantesters.org/cpan/report/48924bc0-9027-11e2-869e-dc2c3b384401

The test failing​:

t/19_incr.t (Wstat​: 12288 Tests​: 697 Failed​: 48)
Failed tests​: 8, 11, 14, 17, 20, 29, 35, 38, 41, 44, 53
56, 68, 71, 89, 95, 101, 104, 113, 122
137, 140, 146, 152, 158, 164, 167, 170
176, 179, 182, 185, 188, 194, 197, 200
203, 206, 212, 233, 260, 263, 266, 269
275, 284, 287, 290
Non-zero exit status​: 48

The line it reports in the diagnostics​:

https://metacpan.org/source/MLEHMANN/JSON-XS-2.33/t/19_incr.t#L22

When the test is running twice, then the reported fails are usually
different ones.

A patch to fix this can be found in​:

https://rt.cpan.org/Public/Bug/Display.html?id=84151

Which is the equivalent of​:

Inline Patch
diff --git a/t/19_incr.t b/t/19_incr.t
index 119a80a..2ff7ce0 100644
--- a/t/19_incr.t
+++ b/t/19_incr.t
@@ -24,8 +24,8 @@ sub splitter {
    }
 }

-splitter +JSON::XS->new              , '
["x\\"","\\u1000\\\\n\\nx",1,{"\\\\" :5 , "": "x"}]'; \-splitter \+JSON​::XS\->new \, '\[ "x\\\\""\,"\\\\u1000\\\\\\\\n\\\\nx" \, 1\,\{"\\\\\\\\ " :5 \, ""​: " x"\} \] '; \+splitter \+JSON​::XS\->new\->canonical\(1\)\, ' \["x\\\\""\,"\\\\u1000\\\\\\\\n\\\\nx"\,1\,\{"\\\\\\\\" :5 \, ""​: "x"\}\]'; \+splitter \+JSON​::XS\->new\->canonical\(1\)\, '\[ "x\\\\""\,"\\\\u1000\\\\\\\\n\\\\nx" \, 1\,\{"\\\\\\\\ " :5 \, ""​: " x"\} \] '; splitter \+JSON​::XS\->new\->allow\_nonref\, '"test"'; splitter \+JSON​::XS\->new\->allow\_nonref\, ' "5" ';

@p5pRT
Copy link
Author

p5pRT commented Mar 24, 2013

From schmorp@schmorp.de

On Sat, Mar 23, 2013 at 09​:34​:17AM -0700, Dave Mitchell via RT <perlbug-followup@​perl.org> wrote​:

Dave, you took my *private* reply and again posted it to the list. Unlike
my mistake, this was hardly a simple mistake by you, because all I did is
reply to your original mail, while you *manually* added the list address
back.

Completely untrue. I took your public email​:

I stand corrected, and apologise for that.

He made a bug report about your code not supporting older compilers
(admittedly not phrased well), and provided a small patch.

There was no patch as has been pointed out before.

It was clearly and unambiguously a patch. It wasn't in diff format, but it
was clearly indicating that to make your code more portable,

He quoted a compiler error message, gave some lines of C code, and some
cryptic comment with no indication on what to change to what, or what he
changed into what.

you could change the line

HE \*hes \[count\];

to
[...]

He wrote no such thing, neither clearly not implied nor even hinting at
that.

You just made this up by actually editing what he wrote and putting it
into a context you made up, so it suits your story.

How low can you go, Dave?

While in hindsight he might have intended his mail to be a patch, he
clearly failed to indicate it.

I have no opinion as to whether it was a good or complete patch, but it
was a patch.

A patch is, at the very least, is a description of changes to apply.

Quoting a compiler error, writing some random lines of C and then claiming
"Already seems to be much more portable for that part." is not, by any
stretch, such a description of changes.

Making up lies like
these just makes clear to me that you are not honest.

A classic Marc-ism. Take a slight argument over semantics and immediately
accuse your opponent of lying.

If you don't want to be accused of lieing you just have to stop doing it.

The above example of falsifying the e-mail you talk about is a good
example.

It's merely an attempt at character assassination.

No, its just an accurate comment​: it was your initial reply to the ticket,
it was rude, and it attacked the person.

Yeah, because you didn't even try to look at the whole story. You see part of
the evidence, and then decide that making untrue accusations is ok.

You might be forgiven to come to the wrong conclusion, but the conclusion
is still wrong.

It definitely was a rant (as I wrote myself), but while I have no clue
what you mean with "complete rant", you are probably wrong about that.

By that, I meant nearly everything in your initial email contained broad,
sweeping criticisms.

I disagree, but the way you misrepresent mails to suit your needs doesn't
surprise me.

What exactly should I have attempted, and what exactly would that have
changed? Was I factually wrong about anything related to that change?

Since you skirted answwering the questions, I assume I wasn't wrong after
all, so your implication that I should have enquired more was wrong.

discussions on the perl security list. Set against that you glibly state
that there are far better ways to aviod DoS, and that yves work is
"useless". But you don't don't go on to explain what these better methods
are, or how they they will always be better than hardening the hash code.

Given that these methods are well known, trivial, and explained by me later
on, I didn't see the need to explain them.

However, I gave far more explanations than Yves in his reply, or any later
reply, ever did.

All that the responsible person was capable of saying was "you are wrong",
while refusing to explain why.

My initial mail was, _in comparison_, an encyclopedia of mitigation
methods.

So you weren't entering into a constructive dialogue, you were just bashing.

As my interaction with reasonable members of the list has shown, I did enter
into a constructive dialog.

It's hard to enter in a constructive dialog if the majority of p5p, and
the person responsoble for the patch, makes wrong claims and can't be
bothered to show any evidence whatsoever.

No, I wasn't, so you implying that I would have had to enquire more first
is just dishonest.

You didn't appear to be aware the technical details of the motivation
behind the insertion order perturbation (avoiding leaking details of the
hash seed).

I don't think I gave you any reason to think that, and the person who
apparently did the patch stated a completely different reason than you
did just now (avoid an algorithmic complexity attack).

Note also that leaking the details of the hash seed is not the primary
motivation behind the patch at all. The primary motivation is to not allow
an algorithmic complexity attack, and not leaking the seed was seen as the
best way to mitigate this.

And there you immediately leap in with another ad Homenem. A difference
of opinion as to whether you were well-enough informed is immediately
"dishonest".

Except this is not about a difference in opinion, but about you making an
unfounded (and wrong) statement about me.

Misrepresenting what you actually said as an opinion, of course, fits into
the pattern of misrepresentig facts so they fit your narrative.

Making this up doesn't make it true. Nowhere did I say or imply that, as

To quote from your original email​:

[snip]

My one-line summary may have been a touch hyperbolic,

Also simply wrong, so it seems you admit that you made it up.

fairly accurately implies that you made a large, sweeping claim about
perl's current development; one which, by the way, I completely
disagree with.

Maybe, but that's not what you said. What you said was just made up, which
is the point.

Once again you accuse me of being a liar.

It seems to be factual, since you are well aware that the many things you
claim are not actually true.

Doing a personal attack appears to always your first choice.

It's not a personal attack if it's true, and it was clearly not my first
choice.

I think it perfectly clear that I have not lied a single time in my
emails in this thread.

Sure, you were just bending the truth unti it broke or so. We might disagree
about some subtle points about what "lie" means, but making false statements
while you are aware that you don't know their truth, and misquoting mails,
certainly counts as a lie, to me.

Oh, and please stop calling me a liar. It is unjustified and deeply
offensive

If you find it offensive then stop doing it, Dave. If you would refrain
from massaging quotes, grossly misrepresenting what was said and stop
making unfounded claims as if you knew they were true you might have a
point.

That's not true. I might be dumb enough to only leave traces of that
publicly, but that doesn't make it so.

Your lack of self-awareness is astonishing. At the first sign of
disagreement, you immediately attack your opponent​: "liar", "troll",
"idiot", etc.

Another lie.

The staggering amount of bullshit that you pull out of your arse is
staggering.

Oh look, there you go again.

Yeah, it gets old quickly. As yves put it... "you made me say that".

If you wouldn't make up all this bullshit, I wouldn't have to call it
bullshit.

Seriously, you don't pull anything out of your arse, but you still make it
up nevertheless.

Ah, after one post, already questioning my integrity.

Another lie, I didn't question your integrity, I told you that if you have
some, then you will do some better research.

There you go again.

I assume your refusal to back up your claim means that you admit you made
it up.

Didn't happen, so now I do question your integrity.

There you go again

Great argument to back up your claims.

Reality check​: "providing a perl interpreter". Well, I provide multiple
perl interpreters, while I doubt you currently provide any perl
interpreter to anybody.

So the 5.14.3 tarball just spontaneously self-assembled?

I don't think so.

There is no doubt that you worked a lot more on perl than me and most
people on this plane, but accusing me of disrespect and then disregarding
the masses of people who worked on perl looks hipocritical to me.

You are an intelligent person. I don't for one *second* believe you you
read my sentence as claiming I am the only person working on the perl
interpreter, or that only my contributions matter.

And I didn't say so, Dave, so take your strawmen ad burn it somewhere else.

What I do believe, and what should have been clear from what I stated so
clearly, is that you have an inflated ego. Your statement clearly sounds as
if you are doing basically all the work.

Once again I will point out that I have not lied a single time.

Let me point out then that I never in my life wrote a single ad hominem,
ever.

This is my final communication with you.

If your assortment of deliberate misinformation could even be called
"communication". But I guess, literally, it is a form of communication. Your
form of communication.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 24, 2013

From @Tux

On Sun, 24 Mar 2013 04​:05​:20 +0100, Marc Lehmann <schmorp@​schmorp.de>
wrote​:

snipping all but what I want to comment upon …

On Sat, Mar 23, 2013 at 09​:34​:17AM -0700, Dave Mitchell via RT <perlbug-followup@​perl.org> wrote​:

He made a bug report about your code not supporting older compilers
(admittedly not phrased well), and provided a small patch.

There was no patch as has been pointed out before.

It was clearly and unambiguously a patch. It wasn't in diff format, but it
was clearly indicating that to make your code more portable,

He quoted a compiler error message, gave some lines of C code, and some
cryptic comment with no indication on what to change to what, or what he
changed into what.

The first post in that thread was what I - as an XS module author -
expect in a ticket. The second post shows that I did some research,
hoping in that could be an entry to a discussion on how to make that
into a patch. More on "patch" below. I have failed in that second
post to express that intention, but the reply took away all my wish
in continuation of that intention as you hopefully understand.

you could change the line

HE \*hes \[count\];

to
[...]

He wrote no such thing, neither clearly not implied nor even hinting at
that.

Come on Marc. You care for your software, so you are able to combine an
error message and two lines of code to see the intention.

While in hindsight he might have intended his mail to be a patch, he
clearly failed to indicate it.

guilty as charged

I have no opinion as to whether it was a good or complete patch, but it
was a patch.

A patch is, at the very least, is a description of changes to apply.

And a test case, which in view of what Vadim noted would have shown
that if it were indeed a patch, it would have been a wrong and
incomplete patch​: HE instead of HE* and no free. An ideal patch would
have included a real-life test case showing that the changed code has
or has no influence on the performance. That first post was not a
patch. Rereading it, I don't think I claimed it to be a patch​:

Already seems to be much more portable for that part.
  ^^^^^^^^^^^^^

Where do I claim that to be a patch?

Quoting a compiler error, writing some random lines of C and then claiming
"Already seems to be much more portable for that part." is not, by any
stretch, such a description of changes.

Fully agree

Making up lies like these just makes clear to me that you are not honest.

In how (other) people interpret written text, and reflecting their
feelings on those interpretation doesn't make them liars. Dave and me
both try to read past that first impression and are maybe more
optimistic about intentions then you are. Perception is subjective, not
objective. Dave is no liar.

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.17 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Mar 24, 2013

From schmorp@schmorp.de

On Sun, Mar 24, 2013 at 03​:19​:42AM -0700, "H. Merijn Brand via RT" <perlbug-followup@​perl.org> wrote​:

you could change the line

HE \*hes \[count\];

to
[...]

He wrote no such thing, neither clearly not implied nor even hinting at
that.

Come on Marc. You care for your software, so you are able to combine an
error message and two lines of code to see the intention.

The simple fact is, I was not able to, for whatever reason. In isolation
and in hindsight, things often look "obvious" that aren't.

And I had to wade through a lot of bullshit, such as things being
"unacceptable", code not being "ANSI-C" etc. etc. and the first time it
occured to me that you might have meant this as some kind of patch occured
to me in this very thread.

So no, your intentions weren't at all clear to me. It still sounds as if
you just wanted to troll me a bit with your comments (comments that, I am
confident, you know were wrong and/or misleading).

Maybe you wanted to rant, because you were obviously pissed off by somebody
having the audacity to use "unacceptable" versions of the C language.

I don't mind when somebody acts like that (that would be rather
hypocritical of me...), but then he or she better is prepared to get some
clear response.

A patch is, at the very least, is a description of changes to apply.

And a test case, which in view of what Vadim noted would have shown

I have no such requireemnts for patches. Neither do patches have to have
any specific format nor is a test case required. In fact, I very much
prefer an explanation of a prose description over a patch, as patches
are almost never directly applicable. A clear explanation of whats wrong
is way more useful to me than a patch, because the quality requirements
_I_ put on myself require me to understand what I am doing, and blindly
applying patches does not work for me.

Making up lies like these just makes clear to me that you are not honest.

In how (other) people interpret written text, and reflecting their
feelings on those interpretation doesn't make them liars.

I fully agree, but things are a bit different here.

People can be confused about meanings, and can misinterpret things as much as
they want. That does not make them liars.

But when you tell them it isn't true, and give reasons, and they ignore
that and just repeat them, then they actively lie.

Just look at this sad thread. I don't know how many times it was pointed
out that changing the C99-style comments to C89 comments does not make the
module it compile, or more portable.

Yet I still receive private mails where people tell me that all I need to
do is change the comments, and the problem is solved.

Fair enough you might say, and to some extent, that's true. But when they
keep making this false claim after being explicitly and personally told it
isn't true, and why (because the module used a lot more extensions that
are found in C99), then they start becoming liars, it's that simple.

And there also has to be a limit. When somebody replies to this thread,
I can reasonably expect him or her not to repeat something that has been
shown to be false many times over an extended period of time.

Also, make no mistake. This list has been used before to spread lies about
me and/or my modules, and is quite welcoming to this kind of behaviour. I
am the only one to defend when it happens (after all, I am old enough to
speak for myself, and don't expect the correctness police to come into
play and fix things for me).

Dave and me both try to read past that first impression and are maybe
more optimistic about intentions then you are. Perception is subjective,
not objective. Dave is no liar.

Let's not get hung up over an apparently politically-incorrect word.

Dave made untrue statements while well knowing that they are likely
untrue, without an effort to correct them. To me, that's a lie, but to
him, that might be business as usual.

The words aren't important, it's whats being done that is.

Or rather, that's wrong. Seems for some people the words are more
important than whats done (or is fact), but I am definitely not one of
them.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 24, 2013

From @bulk88

Konovalov, Vadim (Vadim)** CTR ** wrote​:

It was clearly and unambiguously a patch. It wasn't in diff
format, but it
was clearly indicating that to make your code more portable, you could
change the line​:

HE \*hes \[count\];

to
#if defined(__BORLANDC__) || defined(_MSC_VER) ||
defined(__STDC__)
HE **hes = Perl_malloc (count * sizeof (HE));
#else
HE *hes [count];
#endif

1)
an overlook - it should be
HE **hes = Perl_malloc (count * sizeof (HE*));

it appears that HE is larger than HE* and allocating more memory, hence
misbehaviour is not noticed, but anyway, this isn't correct

2) when that chunk of memory freed? TIA :)

alloca, it can never leak. A non-const length auto C array calls alloca
anyway on C99 under the hood.

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2013

From @ikegami

On Sun, Mar 24, 2013 at 4​:00 PM, bulk88 <bulk88@​hotmail.com> wrote​:

Konovalov, Vadim (Vadim)** CTR ** wrote​:

It was clearly and unambiguously a patch. It wasn't in diff format, but it

was clearly indicating that to make your code more portable, you could
change the line​:

HE \*hes \[count\];

to
#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
HE **hes = Perl_malloc (count * sizeof (HE));
#else
HE *hes [count];
#endif

1)
an overlook - it should be HE **hes = Perl_malloc (count * sizeof
(HE*));

it appears that HE is larger than HE* and allocating more memory, hence
misbehaviour is not noticed, but anyway, this isn't correct

2) when that chunk of memory freed? TIA :)

alloca, it can never leak. A non-const length auto C array calls alloca
anyway on C99 under the hood.

I think he's asking about the newly introduced Perl_malloc

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2013

From @Tux

On Mon, 25 Mar 2013 20​:47​:39 -0400, Eric Brine <ikegami@​adaelis.com>
wrote​:

On Sun, Mar 24, 2013 at 4​:00 PM, bulk88 <bulk88@​hotmail.com> wrote​:

Konovalov, Vadim (Vadim)** CTR ** wrote​:

It was clearly and unambiguously a patch. It wasn't in diff format, but it

was clearly indicating that to make your code more portable, you could
change the line​:

HE \*hes \[count\];

to
#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
HE **hes = Perl_malloc (count * sizeof (HE));
#else
HE *hes [count];
#endif

1)
an overlook - it should be HE **hes = Perl_malloc (count * sizeof
(HE*));

it appears that HE is larger than HE* and allocating more memory, hence
misbehaviour is not noticed, but anyway, this isn't correct

2) when that chunk of memory freed? TIA :)

alloca, it can never leak. A non-const length auto C array calls alloca
anyway on C99 under the hood.

I think he's asking about the newly introduced Perl_malloc

I tested this yesterday with this code​:
--8<---
my $n = 100;
foreach my $f (qw( test1.json test2.json )) {
  say "Testing $f";
  my $jsn = do { open my $fh, "<", $f; local $/; <$fh> };
  printf "Length​: %9d\n", length $jsn;
  my $x = decode_json ($jsn);
  printf "Size D​: %9d\n", total_size ($x);
  printf "Size E​: %9d\n", length encode_json ([$x]);

  my $td = 0;
  for (1..$n) {
  my $t0 = [ gettimeofday ];
  $x = decode_json ($jsn);
  $td += tv_interval ($t0);
  }
  printf "%s​: decode %10.6f s/run\n", $f, $td / $n;

  my $te = 0;
  for (1..$n) {
  my $t0 = [ gettimeofday ];
  my $j = encode_json ([$x]);
  $te += tv_interval ($t0);
  }

  printf "%s​: encode %10.6f s/run\n", $f, $te / $n;
  }
-->8---

on two relative big json files test1.json is unformatted (no newlines
and whitespace) test2.json is formatted (prettied)

-rw-rw-rw- 1 merijn users 716642 Mar 25 18​:40 test1.json
-rw-rw-rw- 1 merijn users 14776790 Mar 25 18​:40 test2.json

Testing test1.json
Length​: 716642
Size D​: 2240594
Size E​: 716589
test1.json​: decode 0.010089 s/run
test1.json​: encode 0.003995 s/run
Testing test2.json
Length​: 14776790
Size D​: 28672598
Size E​: 8198965
test2.json​: decode 0.157494 s/run
test2.json​: encode 0.042437 s/run

Then changed the code to (note the (HE *) and the dropped _ for
Linux/gcc. Used alloca, not Perl_malloc. I know I should look at
*en*coding timings only.

#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
  HE **hes = alloca (count * sizeof (HE *));
#else
  HE *hes [count]; // if your compiler dies here, you need to enable C99 mode
#endif

and got

Testing test1.json
Length​: 716642
Size D​: 2240594
Size E​: 716589
test1.json​: decode 0.010471 s/run
test1.json​: encode 0.003938 s/run
Testing test2.json
Length​: 14776790
Size D​: 28672598
Size E​: 8198965
test2.json​: decode 0.156661 s/run
test2.json​: encode 0.041112 s/run

Then with Perl_malloc

#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
  HE **hes = Perl_malloc (count * sizeof (HE *));
#else
  HE *hes [count]; // if your compiler dies here, you need to enable C99 mode
#endif

which resulted in

Testing test1.json
Length​: 716642
Size D​: 2240594
Size E​: 716589
test1.json​: decode 0.009939 s/run
test1.json​: encode 0.003932 s/run
Testing test2.json
Length​: 14776790
Size D​: 28672598
Size E​: 8198965
test2.json​: decode 0.157021 s/run
test2.json​: encode 0.041158 s/run

which is still faster than the original code.

gcc (SUSE Linux) 4.7.2 20130108 [gcc-4_7-branch revision 195012]
This is perl 5, version 16, subversion 3 (v5.16.3) built for i686-linux-64int
Linux 3.7.10-1.1-desktop [openSUSE 12.3 (Dartmouth)] i386 Core(TM) i7-2620M CPU @​ 2.70GHz/800(4) i686 8042 Mb

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using perl5.00307 .. 5.17 porting perl5 on HP-UX, AIX, and openSUSE
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2013

From @demerphq

On 26 March 2013 08​:32, H.Merijn Brand <h.m.brand@​xs4all.nl> wrote​:

On Mon, 25 Mar 2013 20​:47​:39 -0400, Eric Brine <ikegami@​adaelis.com>
wrote​:

On Sun, Mar 24, 2013 at 4​:00 PM, bulk88 <bulk88@​hotmail.com> wrote​:

Konovalov, Vadim (Vadim)** CTR ** wrote​:

It was clearly and unambiguously a patch. It wasn't in diff format, but it

was clearly indicating that to make your code more portable, you could
change the line​:

HE \*hes \[count\];

to
#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
HE **hes = Perl_malloc (count * sizeof (HE));
#else
HE *hes [count];
#endif

1)
an overlook - it should be HE **hes = Perl_malloc (count * sizeof
(HE*));

it appears that HE is larger than HE* and allocating more memory, hence
misbehaviour is not noticed, but anyway, this isn't correct

2) when that chunk of memory freed? TIA :)

alloca, it can never leak. A non-const length auto C array calls alloca
anyway on C99 under the hood.

I think he's asking about the newly introduced Perl_malloc

I tested this yesterday with this code​:
--8<---
my $n = 100;
foreach my $f (qw( test1.json test2.json )) {
say "Testing $f";
my $jsn = do { open my $fh, "<", $f; local $/; <$fh> };
printf "Length​: %9d\n", length $jsn;
my $x = decode_json ($jsn);
printf "Size D​: %9d\n", total_size ($x);
printf "Size E​: %9d\n", length encode_json ([$x]);

my $td = 0;
for \(1\.\.$n\) \{
    my $t0 = \[ gettimeofday \];
    $x = decode\_json \($jsn\);
    $td \+= tv\_interval \($t0\);
    \}
printf "%s&#8203;: decode %10\.6f s/run\\n"\, $f\, $td / $n;

my $te = 0;
for \(1\.\.$n\) \{
    my $t0 = \[ gettimeofday \];
    my $j = encode\_json \(\[$x\]\);
    $te \+= tv\_interval \($t0\);
    \}

printf "%s&#8203;: encode %10\.6f s/run\\n"\, $f\, $te / $n;
\}

-->8---

on two relative big json files test1.json is unformatted (no newlines
and whitespace) test2.json is formatted (prettied)

-rw-rw-rw- 1 merijn users 716642 Mar 25 18​:40 test1.json
-rw-rw-rw- 1 merijn users 14776790 Mar 25 18​:40 test2.json

Testing test1.json
Length​: 716642
Size D​: 2240594
Size E​: 716589
test1.json​: decode 0.010089 s/run
test1.json​: encode 0.003995 s/run
Testing test2.json
Length​: 14776790
Size D​: 28672598
Size E​: 8198965
test2.json​: decode 0.157494 s/run
test2.json​: encode 0.042437 s/run

Then changed the code to (note the (HE *) and the dropped _ for
Linux/gcc. Used alloca, not Perl_malloc. I know I should look at
*en*coding timings only.

#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
HE **hes = alloca (count * sizeof (HE *));
#else
HE *hes [count]; // if your compiler dies here, you need to enable C99 mode
#endif

and got

Testing test1.json
Length​: 716642
Size D​: 2240594
Size E​: 716589
test1.json​: decode 0.010471 s/run
test1.json​: encode 0.003938 s/run
Testing test2.json
Length​: 14776790
Size D​: 28672598
Size E​: 8198965
test2.json​: decode 0.156661 s/run
test2.json​: encode 0.041112 s/run

Then with Perl_malloc

#if defined(__BORLANDC__) || defined(_MSC_VER) || defined(__STDC__)
HE **hes = Perl_malloc (count * sizeof (HE *));
#else
HE *hes [count]; // if your compiler dies here, you need to enable C99 mode
#endif

which resulted in

Testing test1.json
Length​: 716642
Size D​: 2240594
Size E​: 716589
test1.json​: decode 0.009939 s/run
test1.json​: encode 0.003932 s/run
Testing test2.json
Length​: 14776790
Size D​: 28672598
Size E​: 8198965
test2.json​: decode 0.157021 s/run
test2.json​: encode 0.041158 s/run

which is still faster than the original code.

gcc (SUSE Linux) 4.7.2 20130108 [gcc-4_7-branch revision 195012]
This is perl 5, version 16, subversion 3 (v5.16.3) built for i686-linux-64int
Linux 3.7.10-1.1-desktop [openSUSE 12.3 (Dartmouth)] i386 Core(TM) i7-2620M CPU @​ 2.70GHz/800(4) i686 8042 Mb

IMO the differences there are in the noise.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2013

From schmorp@schmorp.de

On Tue, Mar 26, 2013 at 02​:21​:20AM -0700, "H. Merijn Brand via RT" <perlbug-followup@​perl.org> wrote​:

I tested this yesterday with this code​:

Your test wouldn't even execute the code, so any difference you measure
would be noise or worse.

which is still faster than the original code.

Since the code you benchmark should be effectively identical this just
means you don't know how to interpret your own results, or you are running
into a compiler quirk (where change at one placed strongly affects code in
another, unrelated place).

Just for fun, and to waste more time that I could have used to actually do
productive and useful stuff for Perl, I made a real benchmark, albeit with
the current version of JSON​::XS.

http​://ue.tst.eu/c511fb5d9e4ebc2cd389d7716c706242.txt

I measured the runtime of the full program, not just the encodes.

With STACK_HES defined to 64 (the default), I get​:

real 0m0.808s

With STACK_HES defined to 0 (basically disabling using the stack and always
using malloc)​:

real 0m0.938s

That's a 16% increase in runtime.

That probably doesn't mean much to p5pers who don't mind slowing down perl
by a factor of up to two because its fun to run your own mmu emulation in
the process emulation code, but it means a lot to JSON​::XS.

Things get worse when you don't use method calls to call encode, and then
worse when you actually measure the calls only and then worse again when
you have many hashes and not just one, as all of these further reduce the
overhead. Things might get slightly better if alloca were used, depending
on the architecture and the stack layout used, but it would likely result
in even bigger differences, as addressing can potentially be done with one
indirection less.

All in all, my mini benchmark strongly favours malloc.

I never benchmarked this before, of course, but assuming that the
equivalent of alloca can't beat malloc by a large margin is naive at best.

The very fact that your test is apparently slightly faster with a much
slower function call should have been an enourmous wake-up call to you -
at the very least, you could have added an abort() to atcually check if
its actually being executed. But hey, it fooled Yves too.

I just hope when designing patches for perl and talking about differences
being in the noise, a bit more taste and common sense is applied. Or maybe
a tad more experience in programming.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2013

From @demerphq

On 26 March 2013 13​:14, Marc Lehmann <schmorp@​schmorp.de> wrote​:

The very fact that your test is apparently slightly faster with a much
slower function call should have been an enourmous wake-up call to you -
at the very least, you could have added an abort() to atcually check if
its actually being executed. But hey, it fooled Yves too.

If you are referring to the timing results that Merijn posted then I
dont know what you mean by "fooled" me. I looked at the numbers and
concluded that the variance was in the noise. Such a description is
exactly what you expect if the code is the same for both. I did not
closely look at Merijn's code, and made and make no comment on whether
it is a valid benchmark.

As for /your/ "benchmark", well, it doesn't seem to be very useful
either. "real 0m0.808s" is the output of the bash time function,
and does not constitute a benchmark in any way. I suggest you read

http​://blogs.perl.org/users/steffen_mueller/2010/09/your-benchmarks-suck.html

which could almost be tailor written for this exact case.

Try running under dumbbench for a while and see what it says. THAT
would be a real benchmark. What you posted is a discrete timing, and
on its own tells us almost nothing.

Lastly I would just like to say that comments like this​:

"That probably doesn't mean much to p5pers who don't mind slowing down perl
by a factor of up to two because its fun to run your own mmu emulation in
the process emulation code, but it means a lot to JSON​::XS."

are unhelpful and just plain rude. We care a lot about slowing down
perl, and if we do slow down perl we do only because we have good
reasons, like correctness or security.

You are clearly an intelligent person. Anyone reading your code knows
you are both an experienced C programmer and an intelligent guy.
However when you say snotty things like me being fooled, or that we
don't care about performance, you come across as an extremely
unpleasant person, which in turn makes people like me quite unwilling
to help you. The fact is I could mention a few things that would grind
an extra few percent out of JSON​::XS, which you seem to care about,
but I probably won't because I am unwilling to help someone who is
repeatedly rude and insulting to me and my colleagues. You need to
think about that for a while. You are missing out on the support of
other clever people only because you cant keep a civil tongue in your
head. That is sad.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2013

From schmorp@schmorp.de

On Tue, Mar 26, 2013 at 01​:35​:52PM +0100, demerphq <demerphq@​gmail.com> wrote​:

If you are referring to the timing results that Merijn posted then I
dont know what you mean by "fooled" me. I looked at the numbers and
concluded that the variance was in the noise.

The obvious conclusion is that result is suspicious and needs better
verification, not that the variance is in the noise.

As for /your/ "benchmark", well, it doesn't seem to be very useful
either. "real 0m0.808s" is the output of the bash time function,
and does not constitute a benchmark in any way.

Of course it does, don't get silly.

It might not be a very exact benchmark, but unless you can point out an
error that cancels out 16% despite being a stable value, then there is
nothing wrong with that.

http​://blogs.perl.org/users/steffen_mueller/2010/09/your-benchmarks-suck.html

which could almost be tailor written for this exact case.

None of that applies to this case.

Try running under dumbbench for a while and see what it says. THAT
would be a real benchmark.

Do it yourself, prove that my benchmark is wrong, and you have a
point. Until then, you are just spreading FUD to make my results look
somehow unreliable.

What you posted is a discrete timing, and on its own tells us almost
nothing.

It tells you that at least under some conditions, the code is 16% slower,
with the effect being smeared out over two million runs. Any variances in the
iteration are as likely part of a valid real-world timing as they are true
outliers.

Cache colouring can have an effect this large, but the naive blog posting
you mentioned doesn't seem to be aware of that - the proposed robust
estimation doesn't take this into account.

Massaging your data set to throw out values you don't like without
evidence for them being actually false is unsound.

Recompiling and repeating the test, as I did, does take all this into
account.

"That probably doesn't mean much to p5pers who don't mind slowing down perl
by a factor of up to two because its fun to run your own mmu emulation in
the process emulation code, but it means a lot to JSON​::XS."

are unhelpful and just plain rude. We care a lot about slowing down
perl, and if we do slow down perl we do only because we have good
reasons, like correctness or security.

The example in my "rude" statement is neither about correctness or about
security, so your statement is triivally shown to be wrong. You are just
trying to be difficult...

However when you say snotty things like me being fooled, or that we
don't care about performance, you come across as an extremely
unpleasant person,

Well, the evidence supports both.

If you find reality unpleasant, then I am the wrong person to complain to.

to help you. The fact is I could mention a few things that would grind
an extra few percent out of JSON​::XS, which you seem to care about,

I am quite sure there are bug gains to be gotten. I have a few ideas
myself, and have consistently found significant gains in the past.

But frankly, either show your goods or shut up, because from what I see
of your experience here, it is as likely that you are mistaken as you are
right (did you benchmark your "few things", or do did you just make this
up?)-.

but I probably won't because I am unwilling to help someone who is
repeatedly rude and insulting to me and my colleagues. You need to

I think I am just pointing out the obvious.

think about that for a while. You are missing out on the support of
other clever people only because you cant keep a civil tongue in your
head. That is sad.

I never had reason to believe that I miss out on support of other clever
people. Maybe I did, but that seems to be an empty threat.

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

@p5pRT
Copy link
Author

p5pRT commented Aug 3, 2013

From @cpansprout

JSON​::XS 2.34 works with 5.18.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Aug 3, 2013

@cpansprout - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant