From @titsuki

I'm creating a Perl 6 bindings for MeCab ( http://taku910.github.io/mecab/ \)

See the following codes and documentations(translations).

My repository is​:
https://github.com/titsuki/p6-MeCab/tree/for-RT

The following subtest in the t/02-lattice.t in the above repository is hung up.

subtest {
    my MeCab​::Model $model .= new;
    my @​texts = (("私","僕") xx 3).flat;

    my @​actual = (@​texts.hyper(​:batch(1))\
                  .map(
                         {
                             my MeCab​::Tagger $tagger = $model.create-tagger;
                             my MeCab​::Lattice $lattice = $model.create-lattice;
                             $lattice.sentence($_);
                             $lattice.tostr if $tagger.parse($lattice);
                             $tagger.DESTROY;
                             $lattice.DESTROY;
                         }
                     ).list);

    my Str $r1 = ("私\t名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ\nEOS\n");
    my Str $r2 = ("僕\t名詞,代名詞,一般,*,*,*,僕,ボク,ボク\nEOS\n");
    my @​expected = (($r1, $r2) xx 3).flat;
    is @​actual, @​expected;
}, "MeCab​::Tagger should work in the multithread environment";

In the MeCab documentation page(Japanese) https://taku910.github.io/mecab/libmecab.html
"C++ サンプルコード" (en​: C++ sample code) section says that​:
In the multithread environment (MeCab​::Tagger, MeCab​::Model, MeCab​::Lattice)

* Call MeCab​::createModel() and create the Model object.

* Call model->createTagger and create the Tagger object. The Tagger objects share the same model, even though you create the multiple models under the one model per thread constraint.

* Call model->createLattice or MeCab​::createLattice() and crate the Lattice object. The Lattice objects include all local variables for the morpheme analyze. Must keep the constraint that you can create only one object per thread.

* model->swap(antoher_model) function replaces the Tagger object models created from the invocant model by another_model. This operation is thread-safe.

I think I properly call the MeCab functions from Perl 6(e.g. $model.create-tagger, $model.create-lattice) according to the above instructions.
So I think something is wrong in the NativeCall.    

From @geekosaur

On Tue, Nov 1, 2016 at 12​:01 PM, Itsuki Toyota <perl6-bugs-followup@​perl.org

wrote​:

I think I properly call the MeCab functions from Perl 6(e.g.
$model.create-tagger, $model.create-lattice) according to the above
instructions.
So I think something is wrong in the NativeCall.

Not sure NativeCall even knows about TLS as yet. Probably needs a new
attribute for functions that use TLS, and code to ensure those always run
on the same thread. (This also needs to group functions so they will all
use the same thread... which, given the usage for this particular library,
may need to be dynamic where the init call picks an available thread and
"pins" it for subsequent calls. Which will be tricky, because it needs to
be attached to the object returned by init. Urgh.)

--
brandon s allbery kf8nh sine nomine associates
allbery.b@​gmail.com ballbery@​sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net

The RT System itself - Status changed from 'new' to 'open'

From @jnthn

On Tue Nov 01 09​:01​:04 2016, cookbook_000@​yahoo.co.jp wrote​:

I'm creating a Perl 6 bindings for
MeCab ( http://taku910.github.io/mecab/ \)

See the following codes and documentations(translations).

My repository is​:
https://github.com/titsuki/p6-MeCab/tree/for-RT

The following subtest in the t/02-lattice.t in the above repository is
hung up.
----
subtest {
    my MeCab​::Model $model .= new;
    my @​texts = (("私","僕") xx 3).flat;

    my @​actual = (@​texts.hyper(​:batch(1))\
                  .map(
                         {
                             my MeCab​::Tagger $tagger = $model.create-
tagger;
                             my MeCab​::Lattice $lattice =
$model.create-lattice;
                             $lattice.sentence($_);
                             $lattice.tostr if
$tagger.parse($lattice);
                             $tagger.DESTROY;
                             $lattice.DESTROY;
                         }
                     ).list);

    my Str $r1 = ("私\t名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ\nEOS\n");
    my Str $r2 = ("僕\t名詞,代名詞,一般,*,*,*,僕,ボク,ボク\nEOS\n");
    my @​expected = (($r1, $r2) xx 3).flat;
    is @​actual, @​expected;
}, "MeCab​::Tagger should work in the multithread environment";
---

In the MeCab documentation
page(Japanese) https://taku910.github.io/mecab/libmecab.html
"C++ サンプルコード" (en​: C++ sample code) section says that​:
In the multithread environment (MeCab​::Tagger, MeCab​::Model,
MeCab​::Lattice)

* Call MeCab​::createModel() and create the Model object.

* Call model->createTagger and create the Tagger object. The Tagger
objects share the same model, even though you create the multiple
models under the one model per thread constraint.

* Call model->createLattice or MeCab​::createLattice() and crate the
Lattice object. The Lattice objects include all local variables for
the morpheme analyze. Must keep the constraint that you can create
only one object per thread.

* model->swap(antoher_model) function replaces the Tagger object
models created from the invocant model by another_model. This
operation is thread-safe.

I think I properly call the MeCab functions from Perl 6(e.g.
$model.create-tagger, $model.create-lattice) according to the above
instructions.
So I think something is wrong in the NativeCall.

I think the hang was due to long-running native functions called on one thread blocking GC (and thus progress) in all other threads. That is addressed in MoarVM/MoarVM@5c65917 and MoarVM/MoarVM@f769569 and tested in rakudo/rakudo@5ba75f4.

If you still have problems, please re-open this issue. Also, if possible, see if you can find a smaller reproduction (for example, try just running the code in a number of start blocks, or threads, instead of using `.hyper`).

Thanks,

/jnthn

@jnthn - Status changed from 'open' to 'resolved'

From @titsuki

On 2016-11月-02 水 06​:30​:25, jnthn@​jnthn.net wrote​:

On Tue Nov 01 09​:01​:04 2016, cookbook_000@​yahoo.co.jp wrote​:

I'm creating a Perl 6 bindings for
MeCab ( http://taku910.github.io/mecab/ \)

See the following codes and documentations(translations).

My repository is​:
https://github.com/titsuki/p6-MeCab/tree/for-RT

The following subtest in the t/02-lattice.t in the above repository
is
hung up.
----
subtest {
    my MeCab​::Model $model .= new;
    my @​texts = (("私","僕") xx 3).flat;

    my @​actual = (@​texts.hyper(​:batch(1))\
                  .map(
                         {
                             my MeCab​::Tagger $tagger =
$model.create-
tagger;
                             my MeCab​::Lattice $lattice =
$model.create-lattice;
                             $lattice.sentence($_);
                             $lattice.tostr if
$tagger.parse($lattice);
                             $tagger.DESTROY;
                             $lattice.DESTROY;
                         }
                     ).list);

    my Str $r1 = ("私\t名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ\nEOS\n");
    my Str $r2 = ("僕\t名詞,代名詞,一般,*,*,*,僕,ボク,ボク\nEOS\n");
    my @​expected = (($r1, $r2) xx 3).flat;
    is @​actual, @​expected;
}, "MeCab​::Tagger should work in the multithread environment";
---

In the MeCab documentation
page(Japanese) https://taku910.github.io/mecab/libmecab.html
"C++ サンプルコード" (en​: C++ sample code) section says that​:
In the multithread environment (MeCab​::Tagger, MeCab​::Model,
MeCab​::Lattice)

* Call MeCab​::createModel() and create the Model object.

* Call model->createTagger and create the Tagger object. The Tagger
objects share the same model, even though you create the multiple
models under the one model per thread constraint.

* Call model->createLattice or MeCab​::createLattice() and crate the
Lattice object. The Lattice objects include all local variables for
the morpheme analyze. Must keep the constraint that you can create
only one object per thread.

* model->swap(antoher_model) function replaces the Tagger object
models created from the invocant model by another_model. This
operation is thread-safe.

I think I properly call the MeCab functions from Perl 6(e.g.
$model.create-tagger, $model.create-lattice) according to the above
instructions.
So I think something is wrong in the NativeCall.

I think the hang was due to long-running native functions called on
one thread blocking GC (and thus progress) in all other threads. That
is addressed in
MoarVM/MoarVM@5c65917
and
MoarVM/MoarVM@f769569
and tested in
rakudo/rakudo@5ba75f4.

If you still have problems, please re-open this issue. Also, if
possible, see if you can find a smaller reproduction (for example, try
just running the code in a number of start blocks, or threads, instead
of using `.hyper`).

Thanks,

/jnthn

Thanks a lot !
I confirmed that this problem about the MeCab bindings has been fixed.