switching locales no longer invalidates gettext translation cache since 5.28 #17081
Comments
From @ntyniCreated by @ntyniAs discussed in https://bugs.debian.org/931139 and Starting with Perl 5.28, Perl uses POSIX 2008 thread-safe locales, so This makes gettext(3) think that a translation for the new locale is While this has resulted in a 5.28 regression / behaviour change for Perl, I'm attaching a test case in Perl using Locale::gettext that outputs three I'm also attaching a test case in C that demonstrates that the difference Obviously the relevant locales (fi_FI, fr_FR, en_US) need to be installed Please let me know if I'm not explaining this well enough. Perl Info |
From @ntyni#include <stdio.h>
#include <stdlib.h>
#include <libintl.h>
#include <locale.h>
/* https://bugs.debian.org/931139 */
int main(int argc, char **argv)
{
locale_t loc;
int i=0;
/* The C locale is special cased in glibc to not look at LANGUAGE
so we set C.UTF-8 as the base locale instead */
setenv("LANG", "C.UTF-8", 1);
char *locales[] = { "fi_FI", "fr_FR", "en_US", NULL };
for (i=0; locales[i] != NULL; i++) {
setenv("LANGUAGE", locales[i], 1);
if (argc > 1) {
/* "Old" behaviour, no bug */
setlocale(LC_MESSAGES, "");
} else {
/* "New" behaviour, first translation stays cached */
loc = newlocale(LC_MESSAGES_MASK, "", (locale_t) 0);
if (loc == (locale_t) 0)
perror("newlocale");
uselocale(loc);
}
printf("%s\n", dgettext("bash", "syntax error"));
}
return EXIT_SUCCESS;
} |
From @khwilliamsonOn 7/4/19 12:01 PM, Niko Tyni (via RT) wrote:
This looks like an oversight in glibc to me, hence their bug. No matter In looking at the links you provided, I didn't see how to read the We should look for workarounds in case glibc refuses to change (I don't
|
|
The RT System itself - Status changed from 'new' to 'open' |
From @ntyniOn Thu, Jul 04, 2019 at 12:39:05PM -0700, karl williamson via RT wrote:
Thanks. I suppose I need to look at reporting this there next.
It just adds extra calls to bindtextdomain() and textdomain() to Hope these links work for you: https://salsa.debian.org/installer-team/console-setup/merge_requests/2/diffs
I had a go with the attached crude patch against 5.28.1, and it does Obviously the textdomain() calls and the inclusion of <libintl.h> still Are there thread safety concerns here? Do we need to worry about one |
From @ntynigettext-caching.diffFrom 2a410d595ef74bf36c0ea5d06a7afbf59e5d485f Mon Sep 17 00:00:00 2001
From: Niko Tyni <ntyni@debian.org>
Date: Thu, 4 Jul 2019 23:57:59 +0300
Subject: Invalidate glibc translation cache when switching locales
Bug: https://rt.perl.org/Public/Bug/Display.html?id=134264
Bug-Debian: https://bugs.debian.org/931139
Patch-Name: fixes/gettext-caching.diff
---
locale.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/locale.c b/locale.c
index aedd7da0c..183fd9bdf 100644
--- a/locale.c
+++ b/locale.c
@@ -54,6 +54,8 @@
# include <wchar.h>
#endif
+#include <libintl.h>
+
/* If the environment says to, we can output debugging information during
* initialization. This is done before option parsing, and before any thread
* creation, so can be a file-level static */
@@ -1128,6 +1130,9 @@ S_emulate_setlocale(const int category,
# endif
+ /* Invalidate glibc cache of loaded translations, see [perl #134264] */
+ textdomain(textdomain(NULL));
+
return locale;
}
|
From @khwilliamsonOn 7/5/19 1:00 AM, Niko Tyni wrote:
Thanks
I don't know. The man pages mention nothing about threads. That could Maybe someone in glibc land would know, or someone could look at the source. |
From @khwilliamsonOn 7/5/19 10:18 PM, Karl Williamson wrote:
And we do have a locale semaphore to solve races with these. |
From @khwilliamsonOn Sat, 06 Jul 2019 08:44:13 -0700, public@khwilliamson.com wrote:
Ticket against glibc filed: https://sourceware.org/bugzilla/show_bug.cgi?id=24936 |
|
@ntyni Do you want perl to change to invalidate the cache under gcc |
|
On Mon, Nov 25, 2019 at 04:47:45PM -0800, Karl Williamson wrote:
@ntyni Do you want perl to change to invalidate the cache under gcc
I guess you mean under glibc.
The perl behaviour regressed in 5.28 due to this, and glibc
doesn't look like it's getting fixed any time soon.
So yes, having a workaround in perl would be nice.
Thanks for looking at this,
--
Niko
|
Please see the ticket for a full explanation. This bug has been submitted to glibc, without any real action forthcoming. This invalidates the message cache each time the locale of LC_MESSAGES is changed, as glibc should be doing this when uselocale changes that, but glibc fails to do so.
|
Confirming that your workaround works for me as well, and passes the test suite here. |
Please see the ticket for a full explanation. This bug has been
submitted to glibc, without any real action forthcoming so far.
This invalidates the message cache each time the locale of LC_MESSAGES
is changed, as glibc should be doing this when uselocale changes that,
but glibc fails to do so.
This patch is an extension to the one submitted by Niko Tyni++.
I don't know how to test it, since a test would rely on several
different locales in different languages being available, and that
depends on what's installed on the platform. I suppose that one could
go through the available locales, and try to find three with different
wording for the same message. Doing so however would trigger the bug,
and at the end, if we didn't get three that differed, we wouldn't know
we wouldn't know if it is because of the bug, or that they just didn't
exist on the system.
However, below is a perl program that demonstrated the patch worked.
You could adjust it to the available locales. The buggy code shows the
same text for all locales. The fixed shows three different languages.
use strict;
use Locale::gettext;
use POSIX;
$ENV{LANG} = 'C.UTF-8';
for my $lang (qw(fi_FI fr_FR en_US)) {
$ENV{LANGUAGE} = $lang;
setlocale(LC_MESSAGES, '');
my $d = Locale::gettext->domain("bash");
print $d->get('syntax error'), "\n";
}
|
Note that you can Configure with I have created a branch for your perusal which I believe fixes this bug, smoke-me/khw-locale. There is already a symbol My patch basically just fancies up yours, the one you called crude, and makes sure that it doesn't invalidate the cache if some other category instead of I think we'll know that there is a thread race if there start to be segfault reports. Those can be solved, I believe by use of the existing semaphore. We won't know if there is such a race if no reports come in, unless we get some assurance from the glibc folks. (Don't laugh so hard.) |
Migrated from rt.perl.org#134264 (status was 'open')
Searchable as RT134264$
The text was updated successfully, but these errors were encountered: