From @andk

-------------------- Start of partially forwarded message --------------------
list-help​: <mailto​:perl5-porters-help@​perl.org>
list-unsubscribe​: <mailto​:perl5-porters-unsubscribe@​perl.org>
list-post​: <mailto​:perl5-porters@​perl.org>
X-List-Archive​: <http​://nntp.perl.org/group/perl.perl5.porters/220138>
List-Id​: <perl5-porters.perl.org>
Delivered-To​: mailing list perl5-porters@​perl.org
Date​: Thu, 18 Sep 2014 23​:02​:28 +0200
Message-ID​: <CANgJU+VQeZosf_6sUh0_-f-d4Om81+4Qv15DGt2y8c-vZ1TG7Q@​mail.gmail.com>
Subject​: Re​: fix for CVE-2014-4330 present in blead
From​: demerphq <demerphq@​gmail.com>
To​: Perl5 Porteros <perl5-porters@​perl.org>

[...]
but DD is primarily intended as a debugging tool, I doubt it will
matter. (And if people are using it a serialization tool then they need
their head examined
[...]

cheers,
Yves
-------------------- End of forwarded message --------------------

How come this is not documented? It should be (if true).

--
andreas

From @demerphq

On 19 September 2014 05​:25, Andreas J. Koenig via RT <
perlbug-followup@​perl.org> wrote​:

# New Ticket Created by (Andreas J. Koenig)
# Please include the string​: [perl #122807]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122807 >

-------------------- Start of partially forwarded message
--------------------
list-help​: <mailto​:perl5-porters-help@​perl.org>
list-unsubscribe​: <mailto​:perl5-porters-unsubscribe@​perl.org>
list-post​: <mailto​:perl5-porters@​perl.org>
X-List-Archive​: <http​://nntp.perl.org/group/perl.perl5.porters/220138>
List-Id​: <perl5-porters.perl.org>
Delivered-To​: mailing list perl5-porters@​perl.org
Date​: Thu, 18 Sep 2014 23​:02​:28 +0200
Message-ID​: <
CANgJU+VQeZosf_6sUh0_-f-d4Om81+4Qv15DGt2y8c-vZ1TG7Q@​mail.gmail.com>
Subject​: Re​: fix for CVE-2014-4330 present in blead
From​: demerphq <demerphq@​gmail.com>
To​: Perl5 Porteros <perl5-porters@​perl.org>

[...]
but DD is primarily intended as a debugging tool, I doubt it will
matter. (And if people are using it a serialization tool then they need
their head examined
[...]

cheers,
Yves
-------------------- End of forwarded message --------------------

How come this is not documented? It should be (if true).

I'm not sure exactly what you mean by "if true".

I expressed an opinion that DD is generally unsuitable as a serialization
tool, perhaps a touch inappropriately worded mind you.

The reason are as follows​:

DD is incapable of serializing correctly many of the data structures Perl
can produce.
DD is incapable of round tripping many data structures, even ones as simple
as strings properly.
DD requires eval to deserialize, making its use as a serialization format
insecure. Data​::Undump somewhat addresses this, but only to a certain
extent.
DD is generally inefficient, both in implementation and in terms of output
size, and eval is slow as well.

As a debugging/diagnostics tool DD is pretty good, but even for that
purpose there are better tools like Data​::Dump​::Streamer, assuming
performance is not an issue (even for debugging performance can be an
issue.)

If you want to serialize Perl data then IMO you have a number of other
choices with different trade offs, all of which are pretty much superior to
DD​:

1. Storable
2. JSON
3. YAML
4. Sereal

Storable can represent *any* Perl data structure possible and is fast. It
suffers from the disadvantage that its serialization and deserialization
code is in one module, meaning one cannot upgrade the decoder independent
from the encoder. Compounding that problem is the fact it is distributed
with the Perl core, leading to the "storable upgrade trap", where if you
use it for persistent storage the only way you can upgrade the module or
Perl is to upgrade *every* perl install that uses that persistent store *at
once*. If you don't then you run the risk that the Storable bundled with a
newer Perl produces documents that cannot be read by the Storable bundled
with the older Perls. On top of that the Storable protocol and
implementation has various security issues, which some might consider to be
fairly serious. Storable does not have a well defined format.

JSON is a relatively uncomplicated encoding format incapable of
representing many Perl data structures. Nevertheless its simplicity, wide
portability, and efficiency (in particular via Marc Lehmann's excellent
JSON​::XS) makes it a good choice for many applications. JSON is
standardized and is supported by many languages.

YAML appears to be able to represent most of not all (of the sane) Perl
data structures. It is portable to other languages, includes a more or less
secure deserialization infrastructure. YAML is standardized and is
supported by many languages.

Sereal is designed as a replacement for Storable. It is more efficient both
in terms of speed and size, includes built in compression, allows for the
decoder and encoder to be upgraded independently and thus does not suffer
from the upgrade trap and is capable of representing nearly all "sane" Perl
data structures -- it does not support dualvars nor globs, but does support
pretty much everything else (aliases, regexps, etc). It includes features
to close all of the Storable security holes known to its authors at the
time it was written. For instance it is robust against "DESTROY" based
attacks, forbids duplicate keys in serialized hashes, delays blessing items
until it knows it has deserialized the entire data structure and will never
auto-require a module. Sereal includes built in compression (fast Snappy
and slow Gzip), and a wide variety of other features not worth mentioning
here. (In the interest of full disclosure I should note I am one of authors
of Sereal).

So from my point of view for pure serialization purposes Sereal is the best
general choice by a long margin.

If you also want human readable output then you should pick YAML or JSON.

If you just want to debug stuff then DD is plenty fine, although DDS is
probably better, albeit much slower.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

The RT System itself - Status changed from 'new' to 'open'

From @andk

On Fri, 19 Sep 2014 06​:22​:04 +0200, demerphq <demerphq@​gmail.com> said​:

  > I expressed an opinion that DD is generally unsuitable as a serialization tool,
  > perhaps a touch inappropriately worded mind you.

Thank you.

  > The reason are as follows​:

  > DD is incapable of serializing correctly many of the data structures Perl can
  > produce.

This is documented.

  > DD is incapable of round tripping many data structures,

This is documented.

  > even ones as simple as strings properly.

I could not find this documented, can you please elaborate?

  > DD requires eval to deserialize, making its use as a serialization format
  > insecure.

This is documented.

  > Data​::Undump somewhat addresses this, but only to a certain extent.
  > DD is generally inefficient, both in implementation and in terms of output
  > size, and eval is slow as well.

This may be relevant but does not necessarily need to be documented.

  > As a debugging/diagnostics tool DD is pretty good, but even for that purpose
  > there are better tools like Data​::Dump​::Streamer, assuming performance is not
  > an issue (even for debugging performance can be an issue.)

Ditto.

  > If you want to serialize Perl data then IMO you have a number of other choices
  > with different trade offs, all of which are pretty much superior to
  > DD​:

  > [...]

I don't think that the manpage for DD needs to mention all alternatives
there are on CPAN as long as DD itself is basically doing its job.

--
andreas

From @demerphq

On 20 September 2014 05​:16, Andreas Koenig <
andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

On Fri, 19 Sep 2014 06​:22​:04 +0200, demerphq <demerphq@​gmail.com>
said​:

I expressed an opinion that DD is generally unsuitable as a
serialization tool,
perhaps a touch inappropriately worded mind you.

Thank you.

You are welcome. But I have to say I am bit confused as to the purpose of
this ticket. Was it to find out if there was undocumented reasons?

The reason are as follows​:

DD is incapable of serializing correctly many of the data structures
Perl can
produce.

This is documented.

Is it? I am unaware of them. An example of something DD doesnt dump right​:

perl -MData​::Dumper -le'my @​array; $array[0]=\$array[1];
$array[1]=\$array[0]; print
Data​::Dumper->new([\@​array],[qw(*array)])->Purity(1)->Dump'
@​array = (
  \\do{my $o},
  do{my $o}
  );
${${$array[0]}} = $array[0];
$array[1] = ${$array[0]};

DD is incapable of round tripping many data structures,

This is documented.

Again, I am not familiar with where.

even ones as simple as strings properly.

I could not find this documented, can you please elaborate?

$ perl -MData​::Dumper -le'my $str="hello\tthere\n"; utf8​::upgrade($str);
print Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'
$str = "hello\tthere\n";

That $str is not going to eval back with the utf8 flag set.

Another example​:

$ perl -MData​::Dumper -le'my $str="\0\1\2"; utf8​::upgrade($str); print
Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'
$str = "\0\1\2";

That one actually I think is a regression, IIRC we used to output this as​:

$str = "\x{0}\x{1}\x{2}";

Which a module like Data​::Undump uses to know that the string should end up
utf8 on. Anyway, the point still holds, the utf8ness of the string is not
going to be preserved.

Another example​:

perl -MData​::Dumper -le'my $str="\0\1\2"; Internals​::SvREADONLY($str,1);
Dump($str); print
Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'

The readonly flag will not round trip.

DD requires eval to deserialize, making its use as a serialization
format
insecure.

This is documented.

Yes, I rather suspect it is.

Data​::Undump somewhat addresses this, but only to a certain extent.
DD is generally inefficient, both in implementation and in terms of
output
size, and eval is slow as well.

This may be relevant but does not necessarily need to be documented.

Ok.

As a debugging/diagnostics tool DD is pretty good, but even for that
purpose
there are better tools like Data​::Dump​::Streamer, assuming performance
is not
an issue (even for debugging performance can be an issue.)

Ditto.

I would say that DD should probably reference DDS as an alternative.

If you want to serialize Perl data then IMO you have a number of other
choices
with different trade offs, all of which are pretty much superior to
DD​:

[...]

I don't think that the manpage for DD needs to mention all alternatives
there are on CPAN as long as DD itself is basically doing its job.

Well, I thought the point of this thread was why people shouldnt think of
DD as a good tool for serializing data. It really really isnt.

And if it isnt it should probably reference modules that are.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

From @andk

On Sat, 20 Sep 2014 05​:47​:14 +0200, demerphq <demerphq@​gmail.com> said​:

  > On 20 September 2014 05​:16, Andreas Koenig <
  > andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

On Fri, 19 Sep 2014 06​:22​:04 +0200, demerphq <demerphq@​gmail.com>
  > said​:

  >>> I expressed an opinion that DD is generally unsuitable as a
  >>> serialization tool, perhaps a touch inappropriately worded mind
  >>> you.

  >> Thank you.

  > You are welcome. But I have to say I am bit confused as to the purpose of this
  > ticket. Was it to find out if there was undocumented reasons?

The purpose of this ticket is trivial​: you say DD is broken, I find this
is undocumented, so it must have consequences.

That said, I'm not sure where this ticket will lead us. If Data​::Dumper
is seriously broken we need to (1) either remove it from core or (2) fix
it or (3) document where it fails or (4) find some other settlement that
may be worked out in this ticket. I was not the one who brought the
topic up, I opened the ticket because is is an unacceptable situation.

  >>> The reason are as follows​:

  >>> DD is incapable of serializing correctly many of the data
  >>> structures Perl can produce.

  >> This is documented.
 
  > Is it? I am unaware of them. An example of something DD doesnt dump right​:

  > perl -MData​::Dumper -le'my @​array; $array[0]=\$array[1]; $array[1]=\$array[0];
  > print Data​::Dumper->new([\@​array],[qw(*array)])->Purity(1)->Dump'
  > @​array = (
  >            \\do{my $o},
  >            do{my $o}
  >          );
  > ${${$array[0]}} = $array[0];
  > $array[1] = ${$array[0]};

I stand corrected, this seems undocumented.

  >>> DD is incapable of round tripping many data structures,

  >> This is documented.
 
  > Again, I am not familiar with where.

I suppose this extra paragraph is a subcase of the above example? I
don't know.

  >>> even ones as simple as strings properly.

  >> I could not find this documented, can you please elaborate?
 
  > $ perl -MData​::Dumper -le'my $str="hello\tthere\n"; utf8​::upgrade($str); print
  > Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'
  > $str = "hello\tthere\n";

  > That $str is not going to eval back with the utf8 flag set.

  > Another example​:

  > $ perl -MData​::Dumper -le'my $str="\0\1\2"; utf8​::upgrade($str); print
  > Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'
  > $str = "\0\1\2";

  > That one actually I think is a regression, IIRC we used to output this as​:

  > $str = "\x{0}\x{1}\x{2}";

  > Which a module like Data​::Undump uses to know that the string should end up
  > utf8 on. Anyway, the point still holds, the utf8ness of the string is not going
  > to be preserved.

  > Another example​:

  > perl -MData​::Dumper -le'my $str="\0\1\2"; Internals​::SvREADONLY($str,1); Dump
  > ($str); print Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'

  > The readonly flag will not round trip.
  >

Aha, these exampes look like you were talking about internal flags.
Maybe that would not be too hard to document that internal flags like
those are not preserved. What's your opinion?

  >>> DD requires eval to deserialize, making its use as a serialization
  >>> format insecure.

  >> This is documented.

  > Yes, I rather suspect it is.

Good.
 
  >>> Data​::Undump somewhat addresses this, but only to a certain
  >>> extent. DD is generally inefficient, both in implementation and in
  >>> terms of output size, and eval is slow as well.

  >> This may be relevant but does not necessarily need to be documented.
 
  > Ok.

Great.

  >>> As a debugging/diagnostics tool DD is pretty good, but even for
  >>> that purpose there are better tools like Data​::Dump​::Streamer,
  >>> assuming performance is not an issue (even for debugging
  >>> performance can be an issue.)

  >> Ditto.
 
  > I would say that DD should probably reference DDS as an alternative.

Fine by me.

  >>> If you want to serialize Perl data then IMO you have a number of
  >>> other choices with different trade offs, all of which are pretty
  >>> much superior to DD​:

  >>> [...]

  >> I don't think that the manpage for DD needs to mention all alternatives
  >> there are on CPAN as long as DD itself is basically doing its job.

  > Well, I thought the point of this thread was why people shouldnt think of DD as
  > a good tool for serializing data. It really really isnt.

The point of this ticket is to fix bugs, be it documentations bugs,
packaging bugs, code bugs or bugs that fall in any other category.

  > And if it isnt it should probably reference modules that are.

Fine by me.

Would you mind providing a patch?

--
andreas

From @khwilliamson

On Sat Sep 20 05​:07​:10 2014, andreas.koenig.7os6VVqR@​franz.ak.mind.de wrote​:

On Sat, 20 Sep 2014 05​:47​:14 +0200, demerphq <demerphq@​gmail.com>
said​:

On 20 September 2014 05​:16, Andreas Koenig <
andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

On Fri, 19 Sep 2014 06​:22​:04 +0200, demerphq
<demerphq@​gmail.com>
said​:

I expressed an opinion that DD is generally unsuitable as a
serialization tool, perhaps a touch inappropriately worded mind
you.

Thank you.

You are welcome. But I have to say I am bit confused as to the
purpose of this
ticket. Was it to find out if there was undocumented reasons?

The purpose of this ticket is trivial​: you say DD is broken, I find
this
is undocumented, so it must have consequences.

That said, I'm not sure where this ticket will lead us. If
Data​::Dumper
is seriously broken we need to (1) either remove it from core or (2)
fix
it or (3) document where it fails or (4) find some other settlement
that
may be worked out in this ticket. I was not the one who brought the
topic up, I opened the ticket because is is an unacceptable situation.

The reason are as follows​:

DD is incapable of serializing correctly many of the data
structures Perl can produce.

This is documented.

Is it? I am unaware of them. An example of something DD doesnt dump
right​:

perl -MData​::Dumper -le'my @​array; $array[0]=\$array[1];
$array[1]=\$array[0];
print Data​::Dumper->new([\@​array],[qw(*array)])->Purity(1)->Dump'
@​array = (
           \\do{my $o},
           do{my $o}
         );
${${$array[0]}} = $array[0];
$array[1] = ${$array[0]};

I stand corrected, this seems undocumented.

DD is incapable of round tripping many data structures,

This is documented.

Again, I am not familiar with where.

I suppose this extra paragraph is a subcase of the above example? I
don't know.

even ones as simple as strings properly.

I could not find this documented, can you please elaborate?

$ perl -MData​::Dumper -le'my $str="hello\tthere\n";
utf8​::upgrade($str); print
Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'
$str = "hello\tthere\n";

That $str is not going to eval back with the utf8 flag set.

Another example​:

$ perl -MData​::Dumper -le'my $str="\0\1\2"; utf8​::upgrade($str);
print
Data​::Dumper->new([$str],["*str"])->Purity(1)->Useqq(1)->Dump()'
$str = "\0\1\2";

That one actually I think is a regression, IIRC we used to output
this as​:

$str = "\x{0}\x{1}\x{2}";

Which a module like Data​::Undump uses to know that the string should
end up
utf8 on. Anyway, the point still holds, the utf8ness of the string is
not going
to be preserved.

Another example​:

perl -MData​::Dumper -le'my $str="\0\1\2";
Internals​::SvREADONLY($str,1); Dump
($str); print Data​::Dumper->new([$str],["*str"])->Purity(1)-

Useqq(1)->Dump()'

The readonly flag will not round trip.

Aha, these exampes look like you were talking about internal flags.
Maybe that would not be too hard to document that internal flags like
those are not preserved. What's your opinion?

DD requires eval to deserialize, making its use as a serialization
format insecure.

This is documented.

Yes, I rather suspect it is.

Good.

Data​::Undump somewhat addresses this, but only to a certain
extent. DD is generally inefficient, both in implementation and in
terms of output size, and eval is slow as well.

This may be relevant but does not necessarily need to be documented.

Ok.

Great.

As a debugging/diagnostics tool DD is pretty good, but even for
that purpose there are better tools like Data​::Dump​::Streamer,
assuming performance is not an issue (even for debugging
performance can be an issue.)

Ditto.

I would say that DD should probably reference DDS as an alternative.

Fine by me.

If you want to serialize Perl data then IMO you have a number of
other choices with different trade offs, all of which are pretty
much superior to DD​:

[...]

I don't think that the manpage for DD needs to mention all
alternatives
there are on CPAN as long as DD itself is basically doing its job.

Well, I thought the point of this thread was why people shouldnt
think of DD as
a good tool for serializing data. It really really isnt.

The point of this ticket is to fix bugs, be it documentations bugs,
packaging bugs, code bugs or bugs that fall in any other category.

And if it isnt it should probably reference modules that are.

Fine by me.

Would you mind providing a patch?

A CVE related bug seems like we should be addressing it
--
Karl Williamson