From @cognominal

On a macintel 10.5 I have some problem with unicode. unicode
characters are not recognized as such. See the rakudo test below

The configuring phase gives :

Determining whether ICU is installed...................................yes.

The compiling phase finish with an error but it apprently causes no
problems except I can't run 'make test' because of
the dependance on a successful compilation.

ar​: blib/lib/libparrot.a is a fat file (use libtool(1) or lipo(1) and
ar(1) on it)
ar​: blib/lib/libparrot.a​: Inappropriate file type or format
make​: *** [blib/lib/libparrot.a] Error 1

rakudo is generated without problem

But the following test fails. I pasted the content of the literal
string with a character that emacs says to be #x8a0

my $s = " "; say $s.chars # $s == "\x8a0"
2

I expected one.

--
cognominal stef

From @cognominal

On Mon May 19 10​:29​:29 2008, cognominal wrote​:

On a macintel 10.5 I have some problem with unicode. unicode
characters are not recognized as such. See the rakudo test below

The configuring phase gives :

Determining whether ICU is installed...................................yes.

The compiling phase finish with an error but it apprently causes no
problems except I can't run 'make test' because of
the dependance on a successful compilation.

ar​: blib/lib/libparrot.a is a fat file (use libtool(1) or lipo(1) and
ar(1) on it)
ar​: blib/lib/libparrot.a​: Inappropriate file type or format
make​: *** [blib/lib/libparrot.a] Error 1

rakudo is generated without problem

But the following test fails. I pasted the content of the literal
string with a character that emacs says to be #x8a0

my $s = " "; say $s.chars # $s == "\x8a0"
2

I expected one.

parrot compiles correctly with the correct icu fetched by the correct
configuration : perl Configure.pl --icu-config=/usr/local/bin/icu-config
like I said in #​52898

But the problem mentionned "one unicode character listed as two" remains.

@cognominal - Status changed from 'new' to 'open'

From @pmichaud

On Mon, May 19, 2008 at 10​:29​:29AM -0700, Stephane Payrard wrote​:

But the following test fails. I pasted the content of the literal
string with a character that emacs says to be #x8a0

my $s = " "; say $s.chars # $s == "\x8a0"
2

I expected one.

Because Parrot's primary support for unicode is utf-8 encoding,
and because utf-8 greatly slows down parsing of long strings
(such as program source code), we've elected for the time being
to have rakudo use "fixed8" for its default input encoding. When
Parrot becomes faster at processing unicode strings, we'll likely
switch the default to utf8.(*)

This doesn't mean that unicode can't be used in rakudo programs,
though. One can always encode the character explicitly​:

  $ ./parrot perl6.pbc
  > my $s = "€"; say $s.chars; # doesn't work
  3
  > my $s = "\x20ac"; say $s.chars; # works
  1

Also, rakudo understands the --encoding=utf8 option to specify that
the source code is coming in as UTF-8​:

  $ ./parrot perl6.pbc --encoding=utf8
  > my $s = "€"; say $s.chars; # works
  1

For now I'll mark this ticket as "stalled", awaiting faster Parrot
unicode support or a decision that we're going to live with
slower parsing of source code.

Thanks!

Pm

(*) Another option we might have could be to default to utf8 and
transcode to ucs2 on platforms that have ICU present (which can be
faster), but stay at a fixed8 default for systems without ICU.
But at this stage I think consistency and explicit options are
better, otherwise people will be confused as to why a particular
program works on some systems but not others.

@pmichaud - Status changed from 'open' to 'stalled'

From @pmichaud

Some transcoding options have been added to HLLCompiler and Rakudo since
this ticket was last addressed... could we determine if this is still an
issue for macosx? If not, I'd like to close the ticket.

Thanks!

Pm

The RT System itself - Status changed from 'stalled' to 'open'

From @cognominal

my $s = " "; say $s.chars # now returns 1

Note : the bug was reported on macintel 32 bits which died. I am now
testing on a macintel 64 bits.
I don't know if it can affect the test.

On Mon, May 19, 2008 at 6​:28 PM, Stéphane Payrard <cognominal@​gmail.com> wrote​:

On a macintel 10.5 I have some problem with unicode. unicode
characters are not recognized as such. See the rakudo test below

The configuring phase gives :

Determining whether ICU is installed...................................yes.

The compiling phase finish with an error but it apprently causes no
problems except I can't run 'make test' because of
the dependance on a successful compilation.

ar​: blib/lib/libparrot.a is a fat file (use libtool(1) or lipo(1) and
ar(1) on it)
ar​: blib/lib/libparrot.a​: Inappropriate file type or format
make​: *** [blib/lib/libparrot.a] Error 1

rakudo is generated without problem

But the following test fails. I pasted the content of the literal
string with a character that emacs says to be #x8a0

my $s = " "; say $s.chars # $s == "\x8a0"
2

I expected one.

--
cognominal stef

--
cognominal stef

From @pmichaud

Since the original bug no longer appears to be present, I'm marking this
ticket as resolved.

Thanks!

Pm

@pmichaud - Status changed from 'open' to 'resolved'