Skip Menu |
Report information
Id: 51936
Status: resolved
Priority: 0/
Queue: perl5

Owner: khw <khw [at] cpan.org>
Requestors: chris_hall <chris.hall [at] highwayman.com>
Cc: jgmyers <jgmyers [at] proofpoint.com>
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: Unicode
Perl Version: (no value)
Fixed In: (no value)



Subject: Inconsistent handling of characters with value > 0x7FFF_FFFF and other issues
Date: Thu, 20 Mar 2008 14:10:33 +0000
To: perlbug [...] perl.org
From: Chris Hall <chris.hall [...] highwayman.com>
Download (untitled) / with headers
text/plain 13.8k
This is a bug report for perl from chris.hall@highwayman.com, generated with the help of perlbug 1.36 running under perl 5.10.0. ----------------------------------------------------------------- [Please enter your report here] Amongst the issues: * Character values > 0x7FFF_FFFF are not consistently handled. IMO: the handling is so broken that it would be much better to draw the line at 0x7FFF_FFFF. * chr and pack respond differently to large and out of range values. * pack can generate strings that unpack will not process. * warnings about 'illegal' non-characters are arguably spurious. Certainly there are many cases which are more illegal where no warnings are issued. Treating 0xFFFF_FFFF as a non-character is interesting. * IMO: chr(-1) complete nonsense == undef, not "a character I cannot handle" == U+FFFD. Perl strings containing characters >0x7FFF_FFFF use a non-standard extension to UTF-8. Strictly speaking, UTF-8 stops at U+10FFFF. However, sequences up to 0x7FFF_FFFF are well defined. Bits of Perl are happier with these non-standard sequences than others. Consider: 1: use strict ; 2: use warnings ; 3: 4: warn "__Runtime__" ; 5: 6: my $q = chr(0x7FFF_FFFF).chr(0xE0).chr(0x8000_0000).chr(0xFFFF_FFFD) ; 7: my $v = utf8::valid($q) ? 'Valid' : 'Invalid' ; 8: my $l = length($q) ; 9: my $r = $q.$q ; 10: $q =~ s/\x{E0}/ / ; 11: $q =~ s/\x{7FFF_FFFF}/Hello/ ; 12: $q =~ s/\x{8000_0000}/World/ ; 13: $q =~ s/\x{FFFF_FFFD}/ !/ ; 14: print "$v($l): '$q'\n" ; 15: 16: $r = substr($r, 3, 4) ; 17: print "\$r=", hx(sc($r)), "\n" ; 18: my @w = unpack('U*', $r) ; 19: print "\@w=", hx(@w), "\n" ; 20: 21: $r = pack('U*', sc($r), 0x1_1234_5678) ; 22: print "\$r=", hx(sc($r)), "\n" ; 23: @w = unpack('U*', $r) ; 24: print "\@w=", hx(@w), "\n" ; 25: 26: sub sc { map ord, split(//, $_[0]) ; } ; 27: sub hx { map sprintf('\\x{%X}', $_), @_ ; } ; which generates: A: Unicode character 0x7fffffff is illegal at tb.pl line 11. B: Malformed UTF-8 character (byte 0xfe) at tb.pl line 12. C: Malformed UTF-8 character (byte 0xfe) at tb.pl line 13. D: Integer overflow in hexadecimal number at tb.pl line 21. E: Hexadecimal number > 0xffffffff non-portable at tb.pl line 21. --: __Runtime__ at tb.pl line 4. a: Unicode character 0x7fffffff is illegal at tb.pl line 6. b: Invalid(4): 'Hello World !' c: $r=\x{FFFFFFFD}\x{7FFFFFFF}\x{E0}\x{80000000} d: Malformed UTF-8 character (byte 0xfe) in unpack at tb.pl line 18. e: Malformed UTF-8 character (unexpected continuation byte 0x83, with no : preceding start byte) in unpack at tb.pl line 18. ... repeated for 0xbf, 0xbf, 0xbf, 0xbf, 0xbd f: Malformed UTF-8 character (byte 0xfe) in unpack at tb.pl line 18. g: Malformed UTF-8 character (unexpected continuation byte 0x82, with no : preceding start byte) in unpack at tb.pl line 18. ... repeated for 0x80, 0x80, 0x80, 0x80, 0x80 h: @w=\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{7FFFFFFF}\x{E0} : \x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0} i: Unicode character 0x7fffffff is illegal at tb.pl line 21. j: Unicode character 0xffffffff is illegal at tb.pl line 21. k: $r=\x{FFFFFFFD}\x{7FFFFFFF}\x{E0}\x{80000000}\x{FFFFFFFF} l: Malformed UTF-8 character (byte 0xfe) in unpack at tb.pl line 23. m: Malformed UTF-8 character (unexpected continuation byte 0x83, with no : preceding start byte) in unpack at tb.pl line 23. ... repeated for 0xbf, 0xbf, 0xbf, 0xbf, 0xbd n: Malformed UTF-8 character (byte 0xfe) in unpack at tb.pl line 23. o: Malformed UTF-8 character (unexpected continuation byte 0x82, with no : preceding start byte) in unpack at tb.pl line 23. ... repeated for 0x80, 0x80, 0x80, 0x80, 0x80 p: Malformed UTF-8 character (byte 0xfe) in unpack at tb.pl line 23. q: Malformed UTF-8 character (unexpected continuation byte 0x83, with no : preceding start byte) in unpack at tb.pl line 23. ... repeated for 0xbf, 0xbf, 0xbf, 0xbf, 0xbf r: @w=\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{7FFFFFFF}\x{E0} : \x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0}\x{0} NOTES: 1. chr(n) is happy with characters > 0x7FFF_FFFF BUT: note the runtime warning about 0x7FFF_FFFF itself -- output line a. Unicode defines characters U+xxFFFF as non-characters, for all xx from 0x00 to 0x10 -- the (current) Unicode range. These characters are NOT illegal. Unicode states: "Noncharacter code points are reserved for internal use, such as for sentinel values. They should never be interchanged. They do, however, have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange." Characters > 0x10_FFFF are not known to Unicode. IMO, chr(n) should not be issuing warnings about non-characters at all. IMO, to project non-characters beyond the Unicode range is doubly perverse. FURTHER: although characters > 0x10_FFFF are beyond Unicode, and characters > 0x7FFF_FFFF are beyond UTF-8, chr(n) is only warning about actual and invented non-characters (and surrogates). 2. Similarly "\x{8000_0000} and "\x{7FFF_FFFF}" -- output line A. 3. HOWEVER: utf8::valid() considers a string containing characters which are > 0x7FFF_FFFF to be *invalid* -- see code lines 7 & 14 and output line b. IMO allowing for characters 0x7FFF_FFFF in the first place is a mistake. But having allowed them, why flag the string as invalid ? 4. However: length() is happy, and issues no warning. Either length() is accepting the non-standard encoding, or some other mechanism means that it's not scanning the string. 5. Lines 12 & 13 generate warnings about malformed UTF-8, at compile time. However, the run-time copes with these super-large characters. 6. substr is happy with the super-large characters -- line 16. 7. split is happy with the super-large characters -- line 26. 8. ord is happy with the super-large characters -- line 26. 9. unpack 'U' throws up all over super-large characters ! See lines 18 & 23, and output d-h and l-r. unpack has no idea about the non-standard encoding of characters greater than 0x7FFF_FFFF, and unpacks each 'invalid' byte as 0x00. 10. pack 'U' complains about character values in much the same way as chr does -- output i & j. However, pack and chr are by no means consistent with each other, see below. 11. pack 'U' is generating stuff that unpack 'U' cannot cope with ! See lines 21-24 and output k-r Show quoted text
___________________________________________________________ Looking further at chr and pack: 1: use strict ; 2: use warnings ; 3: 4: warn "__Runtime__" ; 5: 6: my $q = chr(0xD800).chr(0xFFFF).chr(0x7FFF_FFFF) ; 7: my $v = utf8::valid($q) ? 'Valid' : 'Invalid' ; 8: print "\$q = ", hx(sc($q)), " -- $v\n" ; 9: 10: my @t = (0x1_2345_6789, -1, -10, 0xD800) ; 11: my $r = join '', map(chr, @t) ; 12: print "\$r=", hx(sc($r)), "\n" ; 13: 14: my $s = pack('U*', @t) ; 15: print "\$s=", hx(sc($s)), "\n" ; 16: 17: sub sc { map ord, split(//, $_[0]) ; } ; 18: sub hx { map sprintf('\\x{%X}', $_), @_ ; } ; On a 64-bit v5.8.8: A: UTF-16 surrogate 0xd800 at tb2.pl line 6. B: Unicode character 0xffff is illegal at tb2.pl line 6. C: Unicode character 0x7fffffff is illegal at tb2.pl line 6. D: Hexadecimal number > 0xffffffff non-portable at tb2.pl line 10. -- __Runtime__ at tb2.pl line 4. a: $q = \x{D800}\x{FFFF}\x{7FFFFFFF} -- Valid b: Unicode character 0xffffffffffffffff is illegal at tb2.pl line 11. c: UTF-16 surrogate 0xd800 at tb2.pl line 11. d: $r=\x{123456789}\x{FFFFFFFFFFFFFFFF}\x{FFFFFFFFFFFFFFF6}\x{D800} e: Unicode character 0xffffffff is illegal at tb2.pl line 14. f: UTF-16 surrogate 0xd800 at tb2.pl line 14. g: $s=\x{23456789}\x{FFFFFFFF}\x{FFFFFFF6}\x{D800} * chr(-1) generates a warning, not because it's complete rubbish, but because 0xffffffffffffffff is a non-character !!! chr(-3) doesn't merit a warning. * note that surrogates and non-characters are OK as far as utf8::valid is concerned -- no warnings, even. * pack is masking stuff to 32 bit unsigned !! * both chr and pack are throwing warnings about surrogates On a 32-bit v5.10.0: A: Integer overflow in hexadecimal number at tb2.pl line 10. B: Hexadecimal number > 0xffffffff non-portable at tb2.pl line 10. -- __Runtime__ at tb2.pl line 4. a: UTF-16 surrogate 0xd800 at tb2.pl line 6. b: Unicode character 0xffff is illegal at tb2.pl line 6. c: Unicode character 0x7fffffff is illegal at tb2.pl line 6. d: $q = \x{D800}\x{FFFF}\x{7FFFFFFF} -- Valid e: Unicode character 0xffffffff is illegal at tb2.pl line 11. f: UTF-16 surrogate 0xd800 at tb2.pl line 11. g: $r=\x{FFFFFFFF}\x{FFFD}\x{FFFD}\x{D800} h: Unicode character 0xffffffff is illegal at tb2.pl line 14. i: Unicode character 0xffffffff is illegal at tb2.pl line 14. j: UTF-16 surrogate 0xd800 at tb2.pl line 14. k: $s=\x{FFFFFFFF}\x{FFFFFFFF}\x{FFFFFFF6}\x{D800} * chr is mapping -ve values to U+FFFD -- without warning. This is as per documentation. However, character 0xFFFF_FFFF, merits a warning, but does NOT get translated to U+FFFD !! IMO: this is a dog's dinner. I think: - non-characters and surrogates should not trouble chr (any more than they trouble utf8::valid) - values that are invalid should generate undef, not U+FFFD replacement characters: a) cannot distinguish chr(0xFFFD) and chr(-10) b) U+FFFD is a replacement for a character that we don't know -- it's not a replacement for something that just isn't a character in the first place ! [-1 is a banana. U+FFFD is an orange, which we may substitute for another form of orange.] - limiting characters to 0x7FFF_FFFF is no great loss, and avoids a ton of portability and non-standard-ness issues. * pack 'U' is NOT mapping -ve values to U+FFFD !! [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=medium --- Site configuration information for perl 5.10.0: Configured by SYSTEM at Thu Jan 10 11:00:30 2008. Summary of my perl5 (revision 5 version 10 subversion 0) configuration: Platform: osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE -DNO_STRICT -DHAVE_DES_FCRYPT -DUSE_SITECUSTOMIZE - DPRIVLIB_LAST_IN_INC -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX', optimize='-MD -Zi -DNDEBUG -O1', cppflags='-DWIN32' ccversion='12.00.8804', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -libpath:"C:\Program Files\Perl\lib\CORE" -machine:x86' libpth=\lib libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msvcrt.lib perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl510.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -libpath:"C:\Program Files\Perl\lib\CORE" -machine:x86' Locally applied patches: ACTIVEPERL_LOCAL_PATCHES_ENTRY 32809 Load 'loadable object' with non-default file extension 32728 64-bit fix for Time::Local --- @INC for perl 5.10.0: d:\gmch_root\gmch perl lib d:\gmch_root\gmch perl lib\windows C:/Program Files/Perl/site/lib C:/Program Files/Perl/lib . --- Environment for perl 5.10.0: HOME (unset) LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=C:\Program Files\Perl\site\bin;C:\Program Files\Perl\bin;C:\PROGRAM FILES\_BATCH;C:\PROGRAM FILES\_BIN;C:\PROGRAM FILES\ARM\BIN\WIN_32- PENTIUM;C:\PROGRAM FILES\PERL\BIN\;C:\WINDOWS\SYSTEM32;C:\WINDOWS;C:\WINDOWS\SYSTEM32\WBEM;C:\PROGRAM FILES\ATI TECHNOLOGIES\ATI CONTROL PANEL;C:\PROGRAM FILES\MICROSOFT SQL SERVER\80\TOOLS\BINN\;C:\PROGRAM FILES\ARM\UTILITIES\FLEXLM\10.8.0\12\WIN_32-PENTIUM;C:\PROGRAM FILES\ARM\R VCT\PROGRAMS\3.0\441\EVAL2-SC\WIN_32-PENTIUM;C:\PROGRAM FILES\ARM\RVD\CORE\3.0\675\EVAL2-SC\WIN_32-PENTIUM\BIN;C:\PROGRAM FILES\SUPPORT TOOLS\;C:\Program Files\QuickTime\QTSystem\ PERLLIB=d:\gmch_root\gmch perl lib;d:\gmch_root\gmch perl lib\windows PERL_BADLANG (unset) SHELL (unset) -- Chris Hall highwayman.com
Download signature.asc
application/pgp-signature 470b

Message body not shown because it is not plain text.

Download (untitled) / with headers
text/plain 828b
This is similar to bug #43294. I agree that allowing characters above the Unicode maximum of U+10FFFF is a mistake. It serves no useful purpose and just causes trouble for those of us who are trying to process externally-provided UTF-8 data. To safely process untrusted UTF-8 data, we poor implementors need to learn all of the dark corners of Perl's nonstandard UTF-8 processing and somehow deal with the fact that Perl doesn't even agree with itself as to what is valid UTF-8. (see also bug 38722). Allowing surrogates and the non-character U+FFFE in UTF-8 is a security problem in much the same way that allowing non-shortest form sequences (such as C0 80) is. For that reason, chr() should not be permitted to create a surrogate or noncharacter--such a character cannot be represented in a well-formed UTF-8 sequence.
Download (untitled) / with headers
text/plain 2.9k
On Thu Mar 20 15:25:57 2008, jgmyers@proofpoint.com wrote: Show quoted text
> This is similar to bug #43294. > > I agree that allowing characters above the Unicode maximum of U+10FFFF > is a mistake. It serves no useful purpose and just causes trouble for > those of us who are trying to process externally-provided UTF-8 data. > To safely process untrusted UTF-8 data, we poor implementors need to > learn all of the dark corners of Perl's nonstandard UTF-8 processing
and Show quoted text
> somehow deal with the fact that Perl doesn't even agree with itself as > to what is valid UTF-8. (see also bug 38722).
Oh dear. I was actually trying to argue for decoupling general characters from Unicode and strict UTF-8. I think Perl's character general character handling has been mixed up with the handling of Unicode exchange formats. Partly because the internal form is utf8-like and stuff is called utf8 ! IMO Perl should, internally, handle characters with values 0x0..0x7FFF_FFFF -- interpretting the subset which is Unicode as Unicode when required, and only when required. I would dispense with all the broken and incomplete handling of "illegal" Unicode values, and the OTT values > 0x7FFF_FFFF, which I imagine would simplify things. Separately there is clearly the need to filter strict UTF-8 for exchange. Encode's strict-UTF handling isn't complete, but I don't think the requirements are either simple or consistent across applications. Seems to me that the current code falls between two stools and is not fully satisfying either the needs of general character string handling or the needs of strict interchange. Show quoted text
> Allowing surrogates and the non-character U+FFFE in UTF-8 is a
security Show quoted text
> problem in much the same way that allowing non-shortest form sequences > (such as C0 80) is. For that reason, chr() should not be permitted to > create a surrogate or noncharacter--such a character cannot be > represented in a well-formed UTF-8 sequence.
I don't think it's chr's job to police anything. The current inconsistencies etc IMO indicate that once you start trying to police these things you hit conflicting requirements, e.g.: * non-characters are OK for internal use, but not for external interchange. * use of strings for things other than Unicode [I note that printf '%vX' is suggested for IPv6. This implies holding IPv6 addresses in 8 characters, each 0x0..0xFFFF. Which would be impossible if strings refused to allow values that aren't kosher UTF-8 !] * processing UTF-16 as strings of characters 0..0xFFFF. and trying to do two things at once -- e.g. allowing chr() to generate surrogates but throw warnings about them I doubt satisfies anyone ! Applications that need strict UTF-8 (and possibly subsets thereof) need a layer of support for filtering and dealing with stuff that is application-invalid. But I don't think the needs of strict UTF-8 should get in the way of simple, general string handling. -- Chris Hall
Download (untitled) / with headers
text/plain 3.7k
On Thu Mar 20 17:56:04 2008, chris_hall wrote: Show quoted text
> I think Perl's character general character handling has been mixed up > with the handling of Unicode exchange formats. Partly because the > internal form is utf8-like and stuff is called utf8 ! > > IMO Perl should, internally, handle characters with values > 0x0..0x7FFF_FFFF -- interpretting the subset which is Unicode as > Unicode when required, and only when required.
I disagree--Perl should adopt and conform to the Unicode standard. Implementing something which is similar but nonconforming is just laying traps for unwary developers. Particularly heinous is the concept of calling something "utf8" that violates the conformance requirements that Unicode places on UTF-8. Some of the conformance requirements that Perl violates, those against decoding surrogates or U+FFFE, are necessary for security. (Actually, Unicode does not define the UTF-8 byte sequence for U+FFFE as being ill-formed, even though doing so is necessary for security. If you have a security syntax check followed by encoding and decoding in UTF-16, then an attacker could use U+FFFE to trick the UTF-16 decoder into byteswapping the data and having it interpreted differently than what was checked. I have not been able to get the Unicode Consortium to acknowledge this error.) Show quoted text
> I would dispense with all the broken and incomplete handling > of "illegal" Unicode values, and the OTT values > 0x7FFF_FFFF, which I > imagine would simplify things.
By allowing values that are not permitted by Unicode, you are laying a trap for developers not wary of getting get such illegal input. Show quoted text
> Separately there is clearly the need to filter strict UTF-8 for > exchange. Encode's strict-UTF handling isn't complete, but I don't > think the requirements are either simple or consistent across > applications.
The requirements with respect to ill-formed sequences, including surrogates and values above 10FFFF, are clearly specified in the Unicode standard. The requirements with respect to noncharacters are admittedly complex and obscure. Because of the U+FFFE issue, my experience has been that it is best to simply disallow them all. Show quoted text
> Seems to me that the current code falls between two stools and is not > fully satisfying either the needs of general character string handling > or the needs of strict interchange.
I would agree with this, though I disagree about the need for non-interchange strings. How are developers to keep straight which interfaces are to allow or disallow non-interchange strings? Show quoted text
> I don't think it's chr's job to police anything.
I disagree. It is chr's job to police chr('orange'). Similarly, it should police chr(0x7FFF_FFFF). Show quoted text
> The current > inconsistencies etc IMO indicate that once you start trying to police > these things you hit conflicting requirements, e.g.: > > * non-characters are OK for internal use, but not for external > interchange.
Which then begs the question of what is "internal use" versus "external interchange" and what a module must do when given "internal" data and must "interchange" it. Show quoted text
> * use of strings for things other than Unicode > > [I note that printf '%vX' is suggested for IPv6. This implies > holding IPv6 addresses in 8 characters, each 0x0..0xFFFF. > Which would be impossible if strings refused to allow > values that aren't kosher UTF-8 !]
To use strings for things other than Unicode, one should use byte sequences. To use Unicode characters for things that are not Unicode characters is a mistake. Show quoted text
> * processing UTF-16 as strings of characters 0..0xFFFF.
UTF-16 is a character encoding scheme and should be processed as a byte sequence just like every other character encoding scheme is. Surrogates are not characters as defined in chapter 2 of Unicode 5.0.
Subject: Re: [perl #51936] Inconsistent handling of characters with value > 0x7FFF_FFFF and other issues
Date: Sun, 23 Mar 2008 17:39:01 +0000
To: perlbug-followup [...] perl.org
From: Chris Hall <chris.hall [...] highwayman.com>
Download (untitled) / with headers
text/plain 6.1k
On Fri, 21 Mar 2008 you wrote Show quoted text
>On Thu Mar 20 17:56:04 2008, chris_hall wrote:
... Show quoted text
>(Actually, Unicode does not define the UTF-8 byte sequence for U+FFFE as >being ill-formed, even though doing so is necessary for security. If >you have a security syntax check followed by encoding and decoding in >UTF-16, then an attacker could use U+FFFE to trick the UTF-16 decoder >into byteswapping the data and having it interpreted differently than >what was checked. I have not been able to get the Unicode Consortium to >acknowledge this error.)
The standard already says that non-characters should not be exchanged externally -- so a careful UTF-8 decoder would intercept U+FFFE (what it would do with it might vary from application to application). I don't quite understand why you'd want to apply a UTF-16 decoder after a UTF-8 one. Or why a UTF-16 decoder would worry about byte-swapping after the BOM. What am I missing, here ? Show quoted text
>> I would dispense with all the broken and incomplete handling >> of "illegal" Unicode values, and the OTT values > 0x7FFF_FFFF, which I >> imagine would simplify things.
Show quoted text
>By allowing values that are not permitted by Unicode, you are laying a >trap for developers not wary of getting get such illegal input.
No, I'm suggesting removing all the clutter from simple character handling, which gets in the way of some applications. Applications that don't trust their input have their own issues, which I think need to be treated separately, with facilities for different applications to specify (a) what is invalid for them and (b) how to deal with invalid input. Show quoted text
>> Separately there is clearly the need to filter strict UTF-8 for >> exchange. Encode's strict-UTF handling isn't complete, but I don't >> think the requirements are either simple or consistent across >> applications.
Show quoted text
>The requirements with respect to ill-formed sequences, including >surrogates and values above 10FFFF, are clearly specified in the Unicode >standard.
It says they are ill-formed. It doesn't mandate what your application might do with them when they appear. A quick and dirty application might just throw rubbish away, and might get away with it. Another application might convert rubbish to U+FFFD and later whinge about unspecified errors in the input. Yet another application might which to give more specific diagnostics, either at the time the data is first received, or at some later time. Sequences between 0x10_FFFF and 0x7FFF_FFFF are well understood, though UTF-8 declares them ill-formed (while noting that ISO-10646 accepts them). Should an entire 4, 5 or 6 byte sequence beyond 0x10_FFFF be treated as ill-formed, or each individual byte as being part of an ill-formed sequence ? Similarly, the redundant longer forms, which UTF-8 says are ill-formed, different applications may wish to handle differently. Show quoted text
>The requirements with respect to noncharacters are admittedly complex >and obscure. Because of the U+FFFE issue, my experience has been that >it is best to simply disallow them all.
Except that non-characters are entirely legal, and may be essential to some applications. Then there's what to do with (a) unassigned characters, (b) private use characters when exchanging data between unconnected parties, (c) characters not known to the recipient, (d) control characters, etc. etc. ... Show quoted text
>> The current >> inconsistencies etc IMO indicate that once you start trying to police >> these things you hit conflicting requirements, e.g.: >> >> * non-characters are OK for internal use, but not for external >> interchange.
Show quoted text
>Which then begs the question of what is "internal use" versus "external >interchange" and what a module must do when given "internal" data and >must "interchange" it.
Indeed, and that may vary from application to application. So, not only is it (a) more general and (b) conceptually simpler to treat strings as sequences of abstract entities, but we can see that as soon as we try to do more, we run into (i) definition issues and (ii) application-dependent issues. Show quoted text
>> * use of strings for things other than Unicode >> >> [I note that printf '%vX' is suggested for IPv6. This implies >> holding IPv6 addresses in 8 characters, each 0x0..0xFFFF. >> Which would be impossible if strings refused to allow >> values that aren't kosher UTF-8 !]
Show quoted text
>To use strings for things other than Unicode, one should use byte >sequences. To use Unicode characters for things that are not Unicode >characters is a mistake.
I don't see why handling an IPv6 address as a short sequence of 16 bit "characters" (that is, things that go in strings) is any less reasonable than handling IPv4 addresses as short sequences of 8 bit "characters". In the old 8-bit character world what one did with characters was not limited by any given character set interpretation. The new world of 31 (or more) bit characters should not be limited either. Show quoted text
>> * processing UTF-16 as strings of characters 0..0xFFFF.
Show quoted text
>UTF-16 is a character encoding scheme and should be processed as a byte >sequence just like every other character encoding scheme is.
Not really. UTF-16 is defined in terms of 16 bit values. I can use strings for byte sequences. Why not word (16 bit) sequences ? Show quoted text
> Surrogates >are not characters as defined in chapter 2 of Unicode 5.0.
Well, this is it in a nut-shell. I don't think that Perl characters (that is, things that are components of Perl strings) are, or should be, defined to be Unicode characters. I think they should be abstract values -- with a 1-to-1 mapping to/from 31- or 32- bit unsigned integers. [OK, the size here is arbitrary... 31-bits fits with the well understood encoding, 32-bits would be trivially portable, 64-bits seems OTT, but you could argue for that too.] [Even if Perl characters were exactly Unicode characters, there would still be the application specific issues of what to do with non-characters, private use characters, undefined characters, etc. etc.] On top of a generic string data structure there should clearly be extensive support for Unicode. On top of that comes the need for controlled exchange of the various Unicode encoding formats -- for which different applications may have different requirements. Chris -- Chris Hall highwayman.com
Download signature.asc
application/pgp-signature 470b

Message body not shown because it is not plain text.

Subject: Re: [perl #51936] Inconsistent handling of characters with value > 0x7FFF_FFFF and other issues
Date: Tue, 25 Mar 2008 15:27:59 -0700
To: perlbug-followup [...] perl.org
From: John Gardiner Myers <jgmyers [...] proofpoint.com>
Download (untitled) / with headers
text/plain 7.4k
Chris Hall via RT wrote: Show quoted text
> It says they are ill-formed. It doesn't mandate what your application > might do with them when they appear. >
Unicode 5.0 conformance requirement C10 does mandate a restriction on what an application might do with ill-formed sequences. It states "When a process interprets a code unit sequence which purports to be in a Unicode character encoding form, it shall treat ill-formed code unit sequences as an error condition and shall not interpret such sequences as characters." Show quoted text
> A quick and dirty application might just throw rubbish away, and might > get away with it. > > Another application might convert rubbish to U+FFFD and later whinge > about unspecified errors in the input. >
Unicode permits such behavior. Show quoted text
> Sequences between 0x10_FFFF and 0x7FFF_FFFF are well understood, though > UTF-8 declares them ill-formed (while noting that ISO-10646 accepts > them).
ISO-10646 Amendment 2 no longer accepts characters above U+10FFFF. Show quoted text
> Should an entire 4, 5 or 6 byte sequence beyond 0x10_FFFF be > treated as ill-formed, or each individual byte as being part of an > ill-formed sequence ? >
Unicode permits either behavior. Show quoted text
> Similarly, the redundant longer forms, which UTF-8 says are ill-formed, > different applications may wish to handle differently. >
Again, Unicode conformance requirement C10 prohibits applications from interpreting such sequence of characters. To interpret such sequences as characters leaves applications vulnerable to serious security holes. See Unicode 5.0 section 5.19, Unicode Security, which addresses this very issue. Show quoted text
>
>> (Actually, Unicode does not define the UTF-8 byte sequence for U+FFFE as >> being ill-formed, even though doing so is necessary for security. If >> you have a security syntax check followed by encoding and decoding in >> UTF-16, then an attacker could use U+FFFE to trick the UTF-16 decoder >> into byteswapping the data and having it interpreted differently than >> what was checked. I have not been able to get the Unicode Consortium to >> acknowledge this error.) >>
> > I don't quite understand why you'd want to apply a UTF-16 decoder after > a UTF-8 one. Or why a UTF-16 decoder would worry about byte-swapping > after the BOM. What am I missing, here ? >
Software is sometimes constructed by connecting together modules that were previously made elsewhere. So an application might, after passing data through the security syntax check (or "security watchdog module" in Unicode section 5.19), process it through a separate module that writes the data out in UTF-16BE. That UTF-16BE data might in turn be processed by a third module that interprets its input as UTF-16. The potential attack does require a UTF-16 decoder that is sloppier than the one in Encode--either willing to switch endianness mid-stream or willing to treat an initial BOM as optional. Show quoted text
>
>> By allowing values that are not permitted by Unicode, you are laying a >> trap for developers not wary of getting get such illegal input. >>
> > No, I'm suggesting removing all the clutter from simple character > handling, which gets in the way of some applications. > > Applications that don't trust their input have their own issues, which I > think need to be treated separately, with facilities for different > applications to specify (a) what is invalid for them and (b) how to deal > with invalid input. >
Ill-formed sequences are invalid for everybody. By pushing the responsibility for handling such non-obvious character handling issues from the Perl core to individual applications, you would be significantly increasing the number of applications that fail to handle such issues as needed. This is laying traps. Even you seem to have been unaware of the seriously adverse security impact of handling the redundant longer forms as characters. How do you expect a run-of-the-mill Perl script writer to even know that they might have to run extra Unicode-specific validity checks? The current distinction that Perl makes between "utf8" and "utf-8" is quite obscure. Show quoted text
>> The requirements with respect to noncharacters are admittedly complex >> and obscure. Because of the U+FFFE issue, my experience has been that >> it is best to simply disallow them all. >>
> > Except that non-characters are entirely legal, and may be essential to > some applications. >
Please provide an example of a reasonable application to which a non-character is essential. There is no shortage of private use characters--I find it hard to believe that the loss of 66 potential characters is quite so catastrophic. Show quoted text
> Then there's what to do with (a) unassigned characters, (b) private use > characters when exchanging data between unconnected parties, (c) > characters not known to the recipient, (d) control characters, etc. etc. >
There are no such problems with any of these categories. Show quoted text
> > So, not only is it (a) more general and (b) conceptually simpler to > treat strings as sequences of abstract entities, but we can see that as > soon as we try to do more, we run into (i) definition issues and (ii) > application-dependent issues. >
No, it's the converse. When you fail to provide a consistent definition across the language, you run into issues with mismatched and inconsistent definitions within and across applications. Show quoted text
> I don't see why handling an IPv6 address as a short sequence of 16 bit > "characters" (that is, things that go in strings) is any less reasonable > than handling IPv4 addresses as short sequences of 8 bit "characters". >
In neither case are they characters. Show quoted text
> In the old 8-bit character world what one did with characters was not > limited by any given character set interpretation. The new world of 31 > (or more) bit characters should not be limited either. >
The old 8-bit character world is hardly a model of reasonableness. One didn't necessarily know what the character encoding scheme was, so one was quite likely to give the data the wrong interpretation. Some schemes, such as ISO 2022, were an absolute nightmare. The ability to store arbitrary 16 bit quantities in UTF-8 strings is hardly an overriding concern. The overriding concern is to handle text correctly. Show quoted text
>
>> UTF-16 is a character encoding scheme and should be processed as a byte >> sequence just like every other character encoding scheme is. >>
> > Not really. UTF-16 is defined in terms of 16 bit values. I can use > strings for byte sequences. Why not word (16 bit) sequences ? >
Perl strings are not word (16 bit) sequences. UTF-16 was a bad idea to begin with. Let it die a natural death, just like UTF-7. Show quoted text
>> Surrogates are not characters as defined in chapter 2 of Unicode 5.0. >>
> > Well, this is it in a nut-shell. > > I don't think that Perl characters (that is, things that are components > of Perl strings) are, or should be, defined to be Unicode characters. I > think they should be abstract values -- with a 1-to-1 mapping to/from > 31- or 32- bit unsigned integers. >
This is indeed it in a nut-shell. Perl has a choice: On one hand, it could adopt and conform to Unicode, taking advantage of all the work and expertise put into the foremost international standard for character encoding. On the other hand, Perl could decide that it somehow knows more about character encoding than the Unicode Consortium (and the subject experts that contributed to their standard) and go off and invent something new and inconsistent with the constraints the Unicode Consortium found it necessary to impose.
CC: perlbug-followup [...] perl.org
Subject: Re: [perl #51936] Inconsistent handling of characters with value > 0x7FFF_FFFF and other issues
Date: Tue, 25 Mar 2008 20:59:20 -0700
To: John Gardiner Myers <jgmyers [...] proofpoint.com>
From: Glenn Linderman <perl [...] NevCal.com>
Download (untitled) / with headers
text/plain 2.5k
On approximately 3/25/2008 3:27 PM, came the following characters from the keyboard of John Gardiner Myers: Show quoted text
> Chris Hall via RT wrote:
>> Well, this is it in a nut-shell. >> >> I don't think that Perl characters (that is, things that are >> components of Perl strings) are, or should be, defined to be Unicode >> characters. I think they should be abstract values -- with a 1-to-1 >> mapping to/from 31- or 32- bit unsigned integers.
> This is indeed it in a nut-shell. Perl has a choice: On one hand, it > could adopt and conform to Unicode, taking advantage of all the work > and expertise put into the foremost international standard for > character encoding. On the other hand, Perl could decide that it > somehow knows more about character encoding than the Unicode > Consortium (and the subject experts that contributed to their > standard) and go off and invent something new and inconsistent with > the constraints the Unicode Consortium found it necessary to impose.
Perl seems to have already made that choice... and chose TMTOWTDI. The language implements an extension of UTF-8 encoding rules for 70** bit integers (which is very space inefficient above 31 bit integers, even more so that UTF-8 itself) which it calls utf8. The language has certain string operations* that implement certain Unicode semantics for strings stored in utf8 format. Module Encode implements (as best as Dan and kibitzers can) UTF-8 encoding and decoding and validity checking. So people that want to use utf8 strings as containers for 16-bit integers are welcome to, as Chris suggests. And people that want to conform to strict Unicode interpretations have the tools to do so. And people that choose to use utf8 strings only for Unicode codepoints can restrict themselves to doing so. It appears that reported bugs get fixed, as time permits. It appears that the goal is to conform toUnicode semantics in Module Encode, and certain string operations* within the language. * This list is fairly well known, including "\l\L\u\U" uc ucfirst lc lcfirst and certain regexp operations, all of which have different semantics when applied to utf8 strings vs non-utf8 strings. This is considered a bug by some, and a deficiency by most of the rest. ** maybe it is 72? Larger than 64, apparently, and such values higher than the platform's native integer size (usually 32 or 64) are hard to access... chr and ord can't deal with them. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
CC: jgmyers [...] proofpoint.com
Subject: Re: [perl #51936] Inconsistent handling of characters with value > 0x7FFF_FFFF and other issues
Date: Thu, 27 Mar 2008 13:34:51 +0000
To: perlbug-followup [...] perl.org
From: Chris Hall <chris.hall [...] highwayman.com>
Download (untitled) / with headers
text/plain 5.6k
On Tue, 25 Mar 2008 you wrote Show quoted text
>On approximately 3/25/2008 3:27 PM, came the following characters from >the keyboard of John Gardiner Myers:
>> Chris Hall via RT wrote:
>>> Well, this is it in a nut-shell. >>> >>> I don't think that Perl characters (that is, things that are >>> components of Perl strings) are, or should be, defined to be Unicode >>> characters. I think they should be abstract values -- with a 1-to-1 >>> mapping to/from 31- or 32- bit unsigned integers.
Show quoted text
>> This is indeed it in a nut-shell. Perl has a choice: On one hand, it >> could adopt and conform to Unicode, taking advantage of all the work >> and expertise put into the foremost international standard for >> character encoding. On the other hand, Perl could decide that it >> somehow knows more about character encoding than the Unicode >> Consortium (and the subject experts that contributed to their >> standard) and go off and invent something new and inconsistent with >> the constraints the Unicode Consortium found it necessary to impose.
Show quoted text
>Perl seems to have already made that choice... and chose TMTOWTDI. > >The language implements an extension of UTF-8 encoding rules for 70** >bit integers (which is very space inefficient above 31 bit integers, >even more so that UTF-8 itself) which it calls utf8.
As reported: the 7 and 13 byte extended sequences are not properly handled everywhere. The documentation is coy about characters greater than 31 bits. IMO things are so broken (not even utf8::valid() likes the 7- and 13-byte sequences !) that it's not too late to row back from this.... - stopping at 31 bit integers is at least consistent with well-known 4, 5 and 6 byte "UTF-8" sequences. - 32 bit integers could be supported in 6 byte sequences (by treating 0xFC..0xFF prefixes as containing the MS 2 bits) and would be portable (and remains reasonably practical, space-wise). ....the extent of brokenness recalls the guiding mantra: KISS. Show quoted text
>The language has certain string operations* that implement certain >Unicode semantics for strings stored in utf8 format. > >Module Encode implements (as best as Dan and kibitzers can) UTF-8 >encoding and decoding and validity checking. > >So people that want to use utf8 strings as containers for 16-bit >integers are welcome to, as Chris suggests. And people that want to >conform to strict Unicode interpretations have the tools to do so. And >people that choose to use utf8 strings only for Unicode codepoints can >restrict themselves to doing so.
The separation between the content of strings and Unicode is unclear. The name utf8 doesn't help ! A good example of this is chr(n) which: - issues warnings if 'n' is a Unicode surrogate or non-character. These warnings are a nuisance for people using strings as containers for n-bit integers. Those wanting help with strict Unicode aren't materially helped by this behaviour. - accepts characters beyond the Unicode range without warning. So isn't consistent in its "Unicode support". - generates a chr(0xFFFD) in response to chr(-1). Which makes no sense where strings are used as containers for n-bit integers ! Show quoted text
>It appears that reported bugs get fixed, as time permits. It appears >that the goal is to conform toUnicode semantics in Module Encode, and >certain string operations* within the language.
There are plenty of bugs to go round :-} I hope that has not been lost in the discussion. I'm not sure that the Encode Module is the right place for all support for Unicode. It seems to me that Encode is to do with mapping between Perl characters (interpreted as Unicode code-points) and a variety of Character Encoding Schemes, including UTF-8. On input this concerns itself with ill-formed stuff. On output it concerns itself with things that cannot be output. In both cases there is mapping between different Coded Character Sets, and coping with impossible mappings. With Unicode there are additional, specific options required either to allow or do something else with: * the non-characters -- all of them. * the Replacement Character -- perhaps should not send these, or perhaps do not wish to receive these. * private use characters -- which may or may not be suitable for exchange. * perhaps more general support for sub-sets of Unicode. * dealing with canonical equivalences. Now I suppose that a lot can be done by regular expressions and other such processing. This looks like hard work. And might not be terribly efficient ? (Certainly John Gardiner Myers wants utf8::valid to do very strict UTF-8 checking as an efficient test for whether other processing is required.) Show quoted text
>* This list is fairly well known, including "\l\L\u\U" uc ucfirst lc >lcfirst and certain regexp operations, all of which have different >semantics when applied to utf8 strings vs non-utf8 strings. This is >considered a bug by some, and a deficiency by most of the rest. > >** maybe it is 72? Larger than 64, apparently, and such values higher >than the platform's native integer size (usually 32 or 64) are hard to >access... chr and ord can't deal with them.
It's 72: 13 byte sequence, starting 0xFF followed by 12 bytes carrying 6 bits of the value each. FWIW, on 64-bit integer machine: $v = 0xFFFF_FFFF_FFFF_FFFD ; if ($v != ord(chr($v))) { die ; } ; work just fine. Though Perl whimpers: Hexadecimal number > 0xffffffff non-portable at .... (compile time) While: $v = 0xFFFF_FFFF_FFFF_FFFF ; if ($v != ord(chr($v))) { die ; } ; whimpers: Hexadecimal number > 0xffffffff non-portable at .... (compile time) Unicode character 0xffffffffffffffff is illegal at .... (run time) where the second whinge is baroque. Chris -- Chris Hall highwayman.com +44 7970 277 383
Download signature.asc
application/pgp-signature 470b

Message body not shown because it is not plain text.

Subject: Re: [perl #51936] Inconsistent handling of characters with value > 0x7FFF_FFFF and other issues
Date: Thu, 27 Mar 2008 13:44:08 +0000
To: perlbug-followup [...] perl.org
From: Chris Hall <chris.hall [...] highwayman.com>
Download (untitled) / with headers
text/plain 11.4k
On Tue, 25 Mar 2008 John Gardiner Myers wrote Show quoted text
>Chris Hall via RT wrote:
>> It says they are ill-formed. It doesn't mandate what your application >> might do with them when they appear.
Show quoted text
>Unicode 5.0 conformance requirement C10 does mandate a restriction on >what an application might do with ill-formed sequences. It states >"When a process interprets a code unit sequence which purports to be in >a Unicode character encoding form, it shall treat ill-formed code unit >sequences as an error condition and shall not interpret such sequences >as characters."
Sure. But the real point is that this doesn't specify how the error condition must be dealt with. And, as previously discussed, the issue of ill-formed UTF-8 is only part of the problem. Show quoted text
>> A quick and dirty application might just throw rubbish away, and might >> get away with it. >> >> Another application might convert rubbish to U+FFFD and later whinge >> about unspecified errors in the input.
Show quoted text
>Unicode permits such behavior.
Sure. But the point is that there isn't a single correct approach, it depends on the application. Show quoted text
>> Sequences between 0x10_FFFF and 0x7FFF_FFFF are well understood, though >> UTF-8 declares them ill-formed (while noting that ISO-10646 accepts >> them).
Show quoted text
>ISO-10646 Amendment 2 no longer accepts characters above U+10FFFF.
OK. I was going by what Unicode 5.0 says: "The definition of UTF-8 in Annex D of ISO/IEC 10646:2003 also allows for the use of five and six-byte sequences to encode characters that are outside the range of the Unicode character set; those five- and six-byte sequences are illegal for the use of UTF-8 as an encoding form of Unicode characters." Show quoted text
>> Should an entire 4, 5 or 6 byte sequence beyond 0x10_FFFF be >> treated as ill-formed, or each individual byte as being part of an >> ill-formed sequence ?
Show quoted text
>Unicode permits either behavior.
And any given application may wish to do one or the other. Show quoted text
>> Similarly, the redundant longer forms, which UTF-8 says are ill-formed, >> different applications may wish to handle differently.
Show quoted text
>Again, Unicode conformance requirement C10 prohibits applications from >interpreting such sequence of characters. To interpret such sequences >as characters leaves applications vulnerable to serious security holes. >See Unicode 5.0 section 5.19, Unicode Security, which addresses this >very issue.
It takes a narrow view of this. Obviously it is good to encourage strict encoding ! If one wanted to be "generous in what one accepts" one might accept and recode the redundant but longer forms -- which deals with the security issue. But this is not a big requirement. .... Show quoted text
>>> By allowing values that are not permitted by Unicode, you are laying a >>> trap for developers not wary of getting get such illegal input.
Show quoted text
>> No, I'm suggesting removing all the clutter from simple character >> handling, which gets in the way of some applications. >> >> Applications that don't trust their input have their own issues, which I >> think need to be treated separately, with facilities for different >> applications to specify (a) what is invalid for them and (b) how to deal >> with invalid input.
Show quoted text
>Ill-formed sequences are invalid for everybody. By pushing the >responsibility for handling such non-obvious character handling issues >from the Perl core to individual applications, you would be >significantly increasing the number of applications that fail to handle >such issues as needed. This is laying traps.
The existing handling is in a mess. I suggest that this is partly because the problem is not straightforward, and there is no single, universal solution. The problem is not simply ill-formed sequences. IMO the solution is (a) to simplify the base string and character data structures -- so that they are not confused by the conflicting requirements, and (b) to beef up support for strict Unicode character and UTF handling, with sufficient flexibility to allow for different applications to do different things, and with a sensible set of defaults for straightforward use. Show quoted text
>Even you seem to have been unaware of the seriously adverse security >impact of handling the redundant longer forms as characters.
As above. I grant that handling the redundant longer forms is not a big requirement, but if handled correctly the security issue is dealt with. Show quoted text
> How do you >expect a run-of-the-mill Perl script writer to even know that they might >have to run extra Unicode-specific validity checks? The current >distinction that Perl makes between "utf8" and "utf-8" is quite obscure.
Yes, it doesn't help clarify things. Show quoted text
>>> The requirements with respect to noncharacters are admittedly complex >>> and obscure. Because of the U+FFFE issue, my experience has been that >>> it is best to simply disallow them all.
Show quoted text
>> Except that non-characters are entirely legal, and may be essential to >> some applications.
Show quoted text
>Please provide an example of a reasonable application to which a >non-character is essential. There is no shortage of private use >characters--I find it hard to believe that the loss of 66 potential >characters is quite so catastrophic.
Except that it would no longer be Unicode conformant. If you want to argue that non-characters are a Bad Thing, that's a separate topic. Using private use characters instead simply moves the problem. If I use non-characters as delimiters in my application, I should remove them before sending the text to somebody who does not expect them. If I were to use some private-use characters for the same thing, I should still remove them, shouldn't I ? Show quoted text
>> Then there's what to do with (a) unassigned characters, (b) private use >> characters when exchanging data between unconnected parties, (c) >> characters not known to the recipient, (d) control characters, etc. etc.
Show quoted text
>There are no such problems with any of these categories.
Well... if you're troubled by the exchange of 66 non-character values I'm surprised you're not troubled by the huge number of private use characters ! If my system were to place some internal significance on some private use characters, it might be a security issue if these were not filtered out on exchange with third parties -- much like the non-characters. An application that was *really* worried about what it was being sent might wish to filter any or all of these things. It might wish to filter down to some supported sub-set. Not to mention the reduction to canonical form(s). Show quoted text
>> So, not only is it (a) more general and (b) conceptually simpler to >> treat strings as sequences of abstract entities, but we can see that as >> soon as we try to do more, we run into (i) definition issues and (ii) >> application-dependent issues.
Show quoted text
>No, it's the converse. When you fail to provide a consistent definition >across the language, you run into issues with mismatched and >inconsistent definitions within and across applications.
I agree that without a consistent definition you get a mess. Show quoted text
>> I don't see why handling an IPv6 address as a short sequence of 16 bit >> "characters" (that is, things that go in strings) is any less reasonable >> than handling IPv4 addresses as short sequences of 8 bit "characters".
Show quoted text
>In neither case are they characters.
Looking at the "Character Encoding Model", where I said "characters" a little loosely, the jargon suggests 'code units'. But in any case, if a thing that is an element of a string is not a "character" what would you recommend I call it ? Show quoted text
>> In the old 8-bit character world what one did with characters was not >> limited by any given character set interpretation. The new world of 31 >> (or more) bit characters should not be limited either.
Show quoted text
>The old 8-bit character world is hardly a model of reasonableness. One >didn't necessarily know what the character encoding scheme was, so one >was quite likely to give the data the wrong interpretation. Some >schemes, such as ISO 2022, were an absolute nightmare.
Granted that character encodings in the 8-bit world were tricky. But chr() didn't get upset about, for example, DEL (0x7F) or DLE (0x10) despite the obvious issues of interpretation. Core Perl does not attempt to intervene here. I realise that this may appear trivial, but it illustrates the difference between treating strings as sequences of generic 'code units' and treating them as characters according to some specific 'coded character set'. ..... Show quoted text
>> Well, this is it in a nut-shell. >> >> I don't think that Perl characters (that is, things that are components >> of Perl strings) are, or should be, defined to be Unicode characters. I >> think they should be abstract values -- with a 1-to-1 mapping to/from >> 31- or 32- bit unsigned integers.
Show quoted text
>This is indeed it in a nut-shell. Perl has a choice: On one hand, it >could adopt and conform to Unicode, taking advantage of all the work and >expertise put into the foremost international standard for character >encoding.
Leaving to one side any questions about ill-formed sequences. What should be done with: * non-characters -- allow, filter out, replace, ... ? * private-use characters -- allow, filter out, replace, ... ? * unassigned characters -- allow, filter out, replace, ... ? * canonical equivalences -- allow, filter out, replace, ... ? The standard acknowledges a security issue here, but punts it: "However, another level of alternate representation has raised other security questions: the canonical equivalences between precomposed characters and combining character sequences that represent the same abstract characters. .... The conformance requirement, however, is that conforming implementations cannot be required to make an interpretation distinction between canonically equivalent representations. The way for a security- conscious application to guarantee this is to carefully observe the normalization specifications (see Unicode Standard Annex #15, “Unicode Normalization Forms”) so that data is handled consistently in a normalized form." * requirements to handle only sub-sets of characters. * other things, perhaps ? Even surrogates are potentially tricky... ... in UTF-8 surrogate values are explicitly ill-formed. ... in UTF-16 they should travel in pairs, but I guess decoders need to do something with poorly formed or possibly incomplete input. ... but it appears that some code will combine surrogate code points even after decoding the UTF -- I suspect that this is a hangover from older systems where a 16-bit internal character form looked like a reasonable compromise. ... so banning these values from Perl strings is problematic. With ill-formed sequences the question is how to deal with the error condition(s). The point here is that the requirements are not simple and not universal. There is, absolutely, a crying need for clear and effective support for handling Unicode and the UTFs -- especially UTF-8, given its increasing dominance. Show quoted text
> On the other hand, Perl could decide that it somehow knows >more about character encoding than the Unicode Consortium (and the >subject experts that contributed to their standard) and go off and >invent something new and inconsistent with the constraints the Unicode >Consortium found it necessary to impose.
This is a false alternative. Supporting generic "character" and string primitives does not preclude layering strong and flexible UTF and Unicode handling on top, allowing different applications to take more or less control over the various options/ambiguities. At present Perl is achieving neither. Chris PS: big-endian integers are sinful. -- Chris Hall highwayman.com +44 7970 277 383
Download signature.asc
application/pgp-signature 470b

Message body not shown because it is not plain text.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 2.1k
After much further discussion and gnashing of teeth, this has been resolved. The output of the original program in this ticket on current blead is: Hexadecimal number > 0xffffffff non-portable at 51936.pl line 21. __Runtime__ at 51936.pl line 4. Valid(4): 'Hello World !' $r=\x{FFFFFFFD}\x{7FFFFFFF}\x{E0}\x{80000000} @w=\x{FFFFFFFD}\x{7FFFFFFF}\x{E0}\x{80000000} $r=\x{FFFFFFFD}\x{7FFFFFFF}\x{E0}\x{80000000}\x{112345678} @w=\x{FFFFFFFD}\x{7FFFFFFF}\x{E0}\x{80000000}\x{112345678} The decision was made to allow any unsigned values be stored as strings internally in Perl. The non-character code points are all recognized, and allowed. When printing any of the surrogates, non-char code points, or above-legal-Unicode code points, a warning is raised. All are prohibited under strict UTF-8 input. There has been discussion and some work on making the :utf8 layer more strict. I believe this will happen. When doing an operation that requires Unicode semantics on an above-Unicode code point, a warning is raised. An example is changing the case, and this is a no-op. Unicode doesn't actually forbid the use of isolated surrogates in strings inside languages, even though a non-lawyer, such as myself, reading the standard would think that it did. There is some text that allows it. I posted to p5p a portion of an email from the president of Unicode that reiterated this (sent to someone on another project). The clincher is that ICU, the semi-official Unicode implementation does allow isolated surrogates in strings. And, Unicode as of version 5.1 does give property values for every property for the surrogates. At this time, we are warning on surrogates if a case change (including /i regular expression matching) is done on them. I'm not sure that this is correct, as Unicode does furnish casing semantics for them, but it is easier to remove a warning later than to add one. The portion of the original ticket involving chr(-1) has not been resolved. I submitted a bug report for just that, but have not gotten a reply back as to the number assigned to it. In any event, I believe much of the inglorious handling of this whole situation is now fixed --Karl Williamson


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org