Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use encoding and "eq" cause memory leak #7393

Closed
p5pRT opened this issue Jun 28, 2004 · 4 comments
Closed

use encoding and "eq" cause memory leak #7393

p5pRT opened this issue Jun 28, 2004 · 4 comments
Labels
distro-All type-core

Comments

@p5pRT
Copy link

p5pRT commented Jun 28, 2004

Migrated from rt.perl.org#30509 (status was 'resolved')

Searchable as RT30509$
@p5pRT
Copy link
Author

p5pRT commented Jun 28, 2004

From pajas@ufal.ms.mff.cuni.cz

Created by pajas@ufal.mff.cuni.cz

If a subroutine defined before "use encoding" compares strings one of
which is flagged UTF8 (typically, this would be a string read from a
filehandle with an IO encoding layer), the comparison leaks terribly.
This doesn't happen if "use encoding" is used before the defining
subroutine or if it is replaced with "use utf8". On the other hand,
encoding mapage says, that

The best way to avoid ... oddity is to use this pragma RIGHT AFTER
other modules are loaded.

But this approach will leak, if the "other" modules compare UTF8
encoded strings (which is how I found this bug).

On my box the test case below uses >13Mb within 2 seconds and
grows until the system kills it.

__________
#!/usr/bin/perl -w
use strict;
#use encoding "iso-8859-2"; #ok if here

use Encode;
sub leak {
  my $foo = "foo";
  Encode​::_utf8_on($foo); # or read utf8 string from a file
  while (1) {
  $foo eq "x" ? 1 : 0; # eq leaks
  }
}

#use utf8; # this would be ok

use encoding "iso-8859-2"; # leak if here

leak();
__END__

Perl Info 

Flags:
    category=core
    severity=high

Site configuration information for perl v5.8.2:

Configured by pajas at Mon Dec  1 16:06:41 CET 2003.

Summary of my perl5 (revision 5.0 version 8 subversion 2) configuration:
  Platform:
    osname=linux, osvers=2.4.20-20.9, archname=i386-linux-thread-multi
    uname='linux sup.ms.mff.cuni.cz 2.4.20-20.9 #1 mon aug 18 11:45:58 edt 2003 i686 i686 i386 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -march=i386 -mcpu=i686 -Dversion=5.8.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5.8.2 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dinc_version_list=5.8.0/i386-linux-thread-multi 5.8.0 -Dpager=/usr/bin/less -isr'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -I/opt/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -g -pipe -march=i386 -mcpu=i686',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -I/opt/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.2.2 20030222 (Red Hat Linux 3.2.2-5)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib -L/opt/local/lib'
    libpth=/usr/local/lib /opt/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.2/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -L/opt/local/lib'

Locally applied patches:
    


@INC for perl v5.8.2:
    /net/su/h/local2-rh8/lib/perl5/site_perl/5.8.2
    /net/su/h/local2-rh8/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/5.8.2
    /usr/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.2
    /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.2
    /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/5.8.2/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/5.8.2/5.8.2
    /usr/lib/perl5/5.8.2/i386-linux-thread-multi
    /usr/lib/perl5/5.8.2
    .


Environment for perl v5.8.2:
    HOME=/home/pajas
    LANG=cs_CZ.UTF-8
    LANGUAGE (unset)
    LC_CTYPE=cs_CZ.UTF-8
    LD_LIBRARY_PATH=/lib:/usr/lib:/home/pajas/local2/lib:/home/pajas/lib:
    LOGDIR (unset)
    PATH=/home/pajas/bin:/net/su/h/local2-rh8/bin:/usr/ssh2/bin:/usr/bin:/bin:/usr/kerberos/bin:/usr/X11R6/bin:/home/pajas/bin:/usr/local/bin:/usr/local/exec:/home/pajas/treebank/perl:/home/pajas/treebank/rev:/home/pajas/jdk/bin
    PERLLIB=/net/su/h/local2-rh8/lib/perl5/site_perl/5.8.2:/net/su/h/local2-rh8/lib/perl5/site_perl/5.8.2/i386-linux-thread-multi
    PERL_BADLANG (unset)
    PERL_RL=Perl
    SHELL=/bin/bash

@p5pRT
Copy link
Author

p5pRT commented Jun 28, 2004

From @iabyn

On Mon, Jun 28, 2004 at 03​:47​:26PM -0000, pajas @​ ufal. ms. mff. cuni. cz wrote​:

If a subroutine defined before "use encoding" compares strings one of
which is flagged UTF8 (typically, this would be a string read from a
filehandle with an IO encoding layer), the comparison leaks terribly.
This doesn't happen if "use encoding" is used before the defining
subroutine or if it is replaced with "use utf8". On the other hand,
encoding mapage says, that

The best way to avoid ... oddity is to use this pragma RIGHT AFTER
other modules are loaded.

But this approach will leak, if the "other" modules compare UTF8
encoded strings (which is how I found this bug).

On my box the test case below uses >13Mb within 2 seconds and
grows until the system kills it.

__________
#!/usr/bin/perl -w
use strict;
#use encoding "iso-8859-2"; #ok if here

use Encode;
sub leak {
my $foo = "foo";
Encode​::_utf8_on($foo); # or read utf8 string from a file
while (1) {
$foo eq "x" ? 1 : 0; # eq leaks
}
}

#use utf8; # this would be ok

use encoding "iso-8859-2"; # leak if here

leak();
__END__

Thanks for the concise report. I've fixed it in the development version
of Perl by the patch below; it may get included in the imminent 5.8.5
release, or failing that it should be in 5.8.6.

(Whoo hoo​: an easy bug to diagnose and fix! Makes a pleasant change from
all those $%&* threading bugs!)

Dave.

--
This is a great day for France!
  -- Nixon at Charles De Gaulle's funeral

Change 23006 by davem@​davem-percy on 2004/06/28 22​:50​:01

  [perl #30509] use encoding and "eq" cause memory leak
  Perl_sv_eq() was creating a temp and not always freeing it

Affected files ...

... //depot/perl/sv.c#749 edit

Differences ...

==== //depot/perl/sv.c#749 (text) ====

@​@​ -6639,8 +6639,10 @​@​
  pv1 = SvPV(svrecode, cur1);
  }
  /* Now both are in UTF-8. */
- if (cur1 != cur2)
+ if (cur1 != cur2) {
+ SvREFCNT_dec(svrecode);
  return FALSE;
+ }
  }
  else {
  bool is_utf8 = TRUE;

@p5pRT
Copy link
Author

p5pRT commented Jun 28, 2004

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Jun 28, 2004

@iabyn - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this as completed Jun 28, 2004
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
distro-All type-core
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@p5pRT