From schmorp@schmorp.de

Created by schmorp@schmorp.de

perldoc -f pack tells us​:

  w A BER compressed integer (not an ASN.1 BER, see perlpacktut for details).

Ok, it's not an ASN.1 BER integer, still, it's an ASN.1 BER subidentifier
(and is happily used to decode these in e.g. Net​::SNMP).

Anyways, heading to perlpacktut to see why there is strong negation of
it's ASN.1 BER qualities, I find... nothing.

The only thing I can find is "Another Portable Binary Encoding", which tells us​:

  (Details can be found at <http​://Casbah.org/>, the Scarab project.)

Apparently, that domain fell off the internet in 2008. Looking at it
in the wayback machine still shows that there are no details at that
URL. At all. If there ever was, it must have been buried deep inside that
domain at some other URL. It looks more like some shameless plug for some
project, and in any case, perl now refers to some kind of recycling shop
or so.

Furthermore​:

  There is no size limit to BER encoding, but Perl won't go to extremes.

Well, last I looked, perl went very well to extremes when decoding these
gems, as it implements it's own multi-precision multiplication function
so it can decode BER integers of arbitrary lentgh into decimal string
form. Of course, whether that's going to extremes is another question, but
to me, that's quite surprising, and, if officially supported, quite a nice
touch.

Here are my suggestions to improve this situation​:

1. drop the "not an ASN.1 BER" comment, or at least qualify it, like
  "like an ASN.1 BER subidentifier, not a BER integer".

  At the very least the description of BER subidentifier encoding in
  X.209​:198811 6.22 is much more concise than any I found in the perl
  documentation, so saying it isn't an ASN.1 BER is more confusing than
  helpful, because by all means it is.

2. remove the url in perlpacktut.

3. document the fact that perl will go to multiprecision arithmetic when
  decoding these values (but not when encoding them), and that treating
  these decoded values as numbers in perl might reduce their precision.

Greetings,

Flags:
    category=core
    severity=none

Site configuration information for perl 5.16.3:

Configured by Marc Lehmann at Wed Mar 13 11:30:24 CET 2013.

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:
   
  Platform:
    osname=linux, osvers=3.2.0-4-amd64, archname=x86_64-linux
    uname='linux cerebro 3.2.0-4-amd64 #1 smp debian 3.2.35-2 x86_64 gnulinux '
    config_args='-Duselargefiles -Duse64bitint -Dusemymalloc=n -Dstatic_ext=Fcntl -Dcc=gcc -Dccflags=-DPERL_DISABLE_PMC -DPERL_ARENA_SIZE=16376 -USITEARCH_EXP -USITELIB_EXP -UARCHLIB_EXP -D_GNU_SOURCE  -I/opt/include -ggdb -gdwarf-2 -g3 -Doptimize=-O6 -fno-asynchronous-unwind-tables -fno-strict-aliasing -Dcccdlflags=-fPIC -Dldflags=-L/opt/perl/lib -L/opt/lib -Dlibs=-ldl -lm -lcrypt -Dprefix=/opt/perl -Dprivlib=/opt/perl/lib/perl5 -Darchlib=/opt/perl/lib/perl5 -Uusevendorprefix -Dsiteprefix=/opt/perl -Dsitelib=/opt/perl/lib/perl5 -Dsitearch=/opt/perl/lib/perl5 -Dsitebin=/opt/perl/bin -Dman1dir=/opt/perl/man/man1 -Dman3dir=/opt/perl/man/man3 -Dsiteman1dir=/opt/perl/man/man1 -Dsiteman3dir=/opt/perl/man/man3 -Dman1ext=1 -Dman3ext=3 -Dpager=/usr/bin/less -Uafs -Uusesfio -Uusenm -Uuseshrplib -Ud_dosuid -Dusethreads=undef -Duse5005threads=undef -Duseithreads=undef -Dusemultiplicity=undef -Demail=perl-binary@plan9.de -Dcf_email=perl-binary@plan9.de -Dcf_by=Marc Lehmann -Dlocincpth=/opt/perl/include /opt/include -Dmyhostname=localhost -Dmultiarch=undef -Dbin=/opt/perl/bin -Dxxxusedevel -DxxxDEBUGGING -Dxxxuse_debugging_perl -Dxxxuse_debugmalloc -dEs'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-DPERL_DISABLE_PMC -DPERL_ARENA_SIZE=16376 -USITEARCH_EXP -USITELIB_EXP -UARCHLIB_EXP -D_GNU_SOURCE -I/opt/include -ggdb -gdwarf-2 -g3 -fno-strict-aliasing -pipe -I/opt/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O6 -fno-asynchronous-unwind-tables -fno-strict-aliasing',
    cppflags='-DPERL_DISABLE_PMC -DPERL_ARENA_SIZE=16376 -USITEARCH_EXP -USITELIB_EXP -UARCHLIB_EXP -D_GNU_SOURCE -I/opt/include -ggdb -gdwarf-2 -g3 -fno-strict-aliasing -pipe -I/opt/include'
    ccversion='', gccversion='4.7.2', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-L/opt/perl/lib -L/opt/lib -L/usr/local/lib'
    libpth=/usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib
    libs=-ldl -lm -lcrypt
    perllibs=-ldl -lm -lcrypt
    libc=, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.13'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O6 -fno-asynchronous-unwind-tables -fno-strict-aliasing -L/opt/perl/lib -L/opt/lib -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.16.3:
    /root/src/sex
    /root/pserv/lib/perl5
    /opt/perl/lib/perl5
    /opt/perl/lib/perl5
    .


Environment for perl 5.16.3:
    HOME=/root
    LANG (unset)
    LANGUAGE (unset)
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/root/s2:/root/s:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11/bin:/usr/games:/usr/local/bin:/usr/local/sbin:/root/pserv:.
    PERL5LIB=/root/src/sex:/root/pserv/lib/perl5
    PERL5_CPANPLUS_CONFIG=/root/.cpanplus/config
    PERLDB_OPTS=ornaments=0
    PERL_ANYEVENT_DBI_TESTS=1
    PERL_ANYEVENT_LOOP_TESTS=1
    PERL_ANYEVENT_NET_TESTS=1
    PERL_BADLANG (unset)
    PERL_UNICODE=E
    SHELL=/bin/bash

From @tonycoz

On Thu, Apr 18, 2013 at 08​:04​:39AM -0700, schmorp@​schmorp.de (via RT) wrote​:

[Please describe your issue here]

perldoc -f pack tells us​:

w A BER compressed integer (not an ASN.1 BER, see perlpacktut for details).

Ok, it's not an ASN.1 BER integer, still, it's an ASN.1 BER subidentifier
(and is happily used to decode these in e.g. Net​::SNMP).

Anyways, heading to perlpacktut to see why there is strong negation of
it's ASN.1 BER qualities, I find... nothing.

The only thing I can find is "Another Portable Binary Encoding", which tells us​:

(Details can be found at <http​://Casbah.org/>, the Scarab project.)

Apparently, that domain fell off the internet in 2008. Looking at it
in the wayback machine still shows that there are no details at that
URL. At all. If there ever was, it must have been buried deep inside that
domain at some other URL. It looks more like some shameless plug for some
project, and in any case, perl now refers to some kind of recycling shop
or so.

I suspect it refers to this, or a similar definition​:

https://github.com/mworks-project/mw_scarab/blob/master/Scarab-0.1.00d19/doc/binary-serialization.txt#L201

There's another note in the same file complaining about the usage​:

https://github.com/mworks-project/mw_scarab/blob/master/Scarab-0.1.00d19/doc/binary-serialization.txt#L151

Here are my suggestions to improve this situation​:

1. drop the "not an ASN.1 BER" comment, or at least qualify it, like
"like an ASN.1 BER subidentifier, not a BER integer".

At the very least the description of BER subidentifier encoding in
X.209​:198811 6.22 is much more concise than any I found in the perl
documentation, so saying it isn't an ASN.1 BER is more confusing than
helpful, because by all means it is.

"the integer encoding used for ASN.1 subidentifiers"

2. remove the url in perlpacktut.

I wasn't able to find a good linkable reference either.

http​://luca.ntop.org/Teaching/Appunti/asn1.html

provides a good description, but I didn't see any fragment ids.

Tony

The RT System itself - Status changed from 'new' to 'open'

From schmorp@schmorp.de

On Thu, Apr 18, 2013 at 04​:00​:00PM -0700, Tony Cook via RT <perlbug-followup@​perl.org> wrote​:

1. drop the "not an ASN.1 BER" comment, or at least qualify it, like
"like an ASN.1 BER subidentifier, not a BER integer".

At the very least the description of BER subidentifier encoding in
X.209​:198811 6.22 is much more concise than any I found in the perl
documentation, so saying it isn't an ASN.1 BER is more confusing than
helpful, because by all means it is.

"the integer encoding used for ASN.1 subidentifiers"

The BER encoding, not the integer encoding. I don't think there strictly
is something like an integer encoding for ASN.1 subidentifiers -
subidentifiers *are* integers, and BER is the encoding used by perl (there
are many encodings for ASN.1) to encode such subidentifiers (or any other
non-negative integer within range).

There also is a BER integer encoding, but it is indeed different (and can
represent negative integers as well).

It's also quite clear by the references you dug out that these integers
are in fact supposed to be BER encoded, and the "BER" term used by scarab
is the asn.1 ber, just that the author of the document was confused.

So, saying it is the ASN.1 BER subidentifier encoding might be very helpful for
some people, and saying it isn't the BER intger encoding

2. remove the url in perlpacktut.

I wasn't able to find a good linkable reference either.

I don't think there needs to be one, or even should one. I think it would be
very bad for perl to refer to external documents to describe it's
functionality. If it's built in, perl should document it.

And I think it is documented reasonably well. What's not documented is why
it isn't ASN.1 BER (when it is), and the fix for that would likely be to
just drop the errornous statement.

http​://luca.ntop.org/Teaching/Appunti/asn1.html

provides a good description, but I didn't see any fragment ids.

Well, X.209 is freely downloadable from
http​://www.itu.int/rec/T-REC-X.209-198811-W/en

However, BER subidentifier coding is so trivial that having any external
reference would be like hitting the perl user into the face with "yeah,
we could have described it in two sentences, but it's so much more fun to
force you to go online and wade through big documents".

Really, the integers is just grouped into 7-bit groups from most
significant to least signficant and all but the last have the 8th bit set.

It's so trivial it's not even neccessary to explain that this is asn.1 ber
subidentifier coding, it would just be a nice touch for those people who
need to decode them, but frankly, that's probably just me and two other
people or so, and even I use "w" a lot more for other uses (such as length
prefixes or a compact integer encoding).

But saying it isn't asn.1 ber encoding *is* confusing, because I kept
wondering about it, because I couldn't see why not.

:)

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

From @rurban

Marc,

Can you please prepare a patch with the proposed changes.
Thanks

From schmorp@schmorp.de

On Fri, Apr 19, 2013 at 02​:21​:40PM -0700, Reini Urban via RT <perlbug-followup@​perl.org> wrote​:

Marc,

Can you please prepare a patch with the proposed changes.

I was under the impression I already gave one in my initial bug report(?)

--
  The choice of a Deliantra, the free code+content MORPG
  -----==- _GNU_ http​://www.deliantra.net
  ----==-- _ generation
  ---==---(_)__ __ ____ __ Marc Lehmann
  --==---/ / _ \/ // /\ \/ / schmorp@​schmorp.de
  -=====/_/_//_/\_,_/ /_/\_\

From @rurban

On Sat Apr 20 01​:43​:31 2013, schmorp@​schmorp.de wrote​:

On Fri, Apr 19, 2013 at 02​:21​:40PM -0700, Reini Urban via RT <perlbug-
followup@​perl.org> wrote​:

Marc,

Can you please prepare a patch with the proposed changes.

I was under the impression I already gave one in my initial bug
report(?)

No sorry.
No attachments appeared at https://rt-archive.perl.org/perl5/Ticket/Display.html?
id=117661
--
Reini Urban

From @rjbs

I have applied your suggestions in a branch here​:
http​://perl5.git.perl.org/perl.git/commitdiff/e2b02bb

If they are acceptable to all, I will apply them to blead soon.

--
rjbs