Skip Menu |
Report information
Id: 39358
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: johnh [at] isi.edu
Cc:
AdminCc:

Operating System: Linux
PatchStatus: (no value)
Severity: High
Type:
Perl Version:
  • 5.8.8
  • 5.10.0
  • 5.25.2
Fixed In: (no value)



Subject: sort with custom subname and prototype ($$) segfaults intermittently
Date: Thu, 8 Jun 2006 21:22:30 -0700
To: perlbug [...] perl.org
From: John Heidemann <johnh [...] isi.edu>
Download (untitled) / with headers
text/plain 8.9k
This is a bug report for perl from johnh@isi.edu, generated with the help of perlbug 1.35 running under perl v5.8.8. ----------------------------------------------------------------- [Please enter your report here] For a long time (more than a year), perl has segfaulted for me in a program that does some nasty sorting. Recently I finally narrowed it down to a modest-size test case: ---------------------------------------- #!/usr/bin/perl -w @scores = ( '0000.0000.00039:0000.0008.0031.', '0000.0000.00032:0000.0008.0024.', '0002.0002.00033:0002.0011.0020.', '0028.0028.00190:0077.0085.0028.'); @match_indices = (0,1,2,3); sub sort_by_index($$) { my($A,$B) = @_; return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; } @match_indices = sort sort_by_index @match_indices; ---------------------------------------- It segfaults for me on the last line, both with perl 5.8.8 on fedora core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes back to at least perl 5.6, but I don't have good notes. Here's the backtrace on 5.8.8 built with debugging: (gdb) run -Ilib /tmp/t.pl Starting program: /usr/src/redhat/BUILD/perl-5.8.8/perl -Ilib /tmp/t.pl Reading symbols from shared object read from target memory...done. Loaded system supplied DSO at 0xd88000 [Thread debugging using libthread_db enabled] [New Thread -1208506688 (LWP 31509)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1208506688 (LWP 31509)] 0x0069c8f0 in Perl_pp_aelem (my_perl=0x89b1008) at pp_hot.c:3007 3007 IV elem = SvIV(elemsv); (gdb) bt #0 0x0069c8f0 in Perl_pp_aelem (my_perl=0x89b1008) at pp_hot.c:3007 #1 0x00697937 in Perl_runops_standard (my_perl=0x89b1008) at run.c:37 #2 0x00712860 in sortcv_stacked (my_perl=0x89b1008, a=0x89cd5c0, b=0x89cd5cc) at pp_sort.c:1791 #3 0x0071065e in S_mergesortsv (my_perl=0x89b1008, base=Variable "base" is not available. ) at pp_sort.c:429 #4 0x007119d2 in Perl_sortsv (my_perl=0x89b1008, array=0x89d1a10, nmemb=4, cmp=0x7127f1 <sortcv_stacked>) at pp_sort.c:1458 #5 0x007120b7 in Perl_pp_sort (my_perl=0x89b1008) at pp_sort.c:1685 #6 0x00697937 in Perl_runops_standard (my_perl=0x89b1008) at run.c:37 #7 0x0063d989 in perl_run (my_perl=0x89b1008) at perl.c:2367 #8 0x080492f4 in main (argc=3, argv=0xbfe9a544, env=0xbfe9a554) at perlmain.c:99 (gdb) q The test program is 100% repeatable. I extracted it from a larger program where I get intermittent problems depending on the input (but always in the sort routine). For a different input I got a different traceback: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1208793408 (LWP 31157)] 0x008c6ccd in Perl_mg_get (my_perl=0x81f7008, sv=0x8506ac0) at mg.c:164 164 newmg = cur = head = mg = SvMAGIC(sv); (gdb) bt #0 0x008c6ccd in Perl_mg_get (my_perl=0x81f7008, sv=0x8506ac0) at mg.c:164 #1 0x008e8695 in Perl_sv_setsv_flags (my_perl=0x81f7008, dstr=0x821e200, sstr=Variable "sstr" is not available. ) at sv.c:3856 #2 0x008e9d2b in Perl_sv_mortalcopy (my_perl=0x81f7008, oldstr=0x8506ac0) at sv.c:6814 #3 0x008d8b81 in Perl_pp_aelem (my_perl=0x81f7008) at pp_hot.c:3055 #4 0x008d3937 in Perl_runops_standard (my_perl=0x81f7008) at run.c:37 #5 0x0094e860 in sortcv_stacked (my_perl=0x81f7008, a=0x876a934, b=0x876a868) at pp_sort.c:1791 #6 0x0094c65e in S_mergesortsv (my_perl=0x81f7008, base=Variable "base" is not available. ) at pp_sort.c:429 #7 0x0094d9d2 in Perl_sortsv (my_perl=0x81f7008, array=0x8506ab0, nmemb=4, cmp=0x94e7f1 <sortcv_stacked>) at pp_sort.c:1458 #8 0x0094e0b7 in Perl_pp_sort (my_perl=0x81f7008) at pp_sort.c:1685 #9 0x008d3937 in Perl_runops_standard (my_perl=0x81f7008) at run.c:37 #10 0x00879989 in perl_run (my_perl=0x81f7008) at perl.c:2367 #11 0x080492f4 in main (argc=6, argv=0xbfe54dd4, env=0xbfe54df0) at perlmain.c:99 (I didn't isolate this to a test kernel.) Any suggestions? Thanks, -John Heidemann [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=core severity=high --- This perlbug was built using Perl v5.8.8 in the Red Hat build system. It is being executed now by Perl v5.8.8 - Wed Mar 1 18:29:13 EST 2006. Site configuration information for perl v5.8.8: Configured by Red Hat, Inc. at Wed Mar 1 18:29:13 EST 2006. Summary of my perl5 (revision 5 version 8 subversion 8) configuration: Platform: osname=linux, osvers=2.6.9-22.18.bz155725.elsmp, archname=i386-linux-thread-multi uname='linux hs20-bc1-6.build.redhat.com 2.6.9-22.18.bz155725.elsmp #1 smp thu nov 17 15:34:08 est 2005 i686 i686 i386 gnulinux ' config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Dversion=5.8.8 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dinc_version_list=5.8.7 5.8.6 5.8.5 5.8.4 5.8.3 -Dscriptdir=/usr/bin' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm', optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm' ccversion='', gccversion='4.1.0 20060228 (Red Hat 4.1.0-1)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='gcc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.3.90.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='2.3.90' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib' Locally applied patches: --- @INC for perl v5.8.8: /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl/5.8.4 /usr/lib/perl5/site_perl/5.8.3 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl/5.8.4 /usr/lib/perl5/vendor_perl/5.8.3 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 . --- Environment for perl v5.8.8: HOME=/home/johnh LANG=en_US.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH=/usr/local/lib LOGDIR (unset) PATH=/home/johnh/BIN:/home/johnh/BIN/LINUX:/usr/local/bin:/bin:/usr/bin:/home/johnh/BIN/WP:/home/johnh/BIN/HOSTS:/home/johnh/BIN/DB:/usr/X11R6/bin:/usr/local/etc:/usr/local/sbin:/etc:/usr/etc:/sbin:/usr/sbin:/usr/hosts:/usr/local/games:/usr/games PERL_BADLANG (unset) SHELL=/bin/bash
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Fri, 9 Jun 2006 14:51:32 +0200
To: perl5-porters [...] perl.org
From: "H.Merijn Brand" <h.m.brand [...] xs4all.nl>
Download (untitled) / with headers
text/plain 10.1k
On Thu, 08 Jun 2006 21:23:41 -0700, "johnh@isi.edu (via RT)" <perlbug-followup@perl.org> wrote: Show quoted text
> # New Ticket Created by johnh@isi.edu > # Please include the string: [perl #39358] > # in the subject line of all future correspondence about this issue. > # <URL: https://rt.perl.org/rt3/Ticket/Display.html?id=39358 > > > > > This is a bug report for perl from johnh@isi.edu, > generated with the help of perlbug 1.35 running under perl v5.8.8. > > > ----------------------------------------------------------------- > [Please enter your report here] > > For a long time (more than a year), perl has segfaulted for me in a > program that does some nasty sorting. Recently I finally narrowed it > down to a modest-size test case: > > ---------------------------------------- > #!/usr/bin/perl -w > > @scores = ( > '0000.0000.00039:0000.0008.0031.', > '0000.0000.00032:0000.0008.0024.', > '0002.0002.00033:0002.0011.0020.', > '0028.0028.00190:0077.0085.0028.'); > > @match_indices = (0,1,2,3); > sub sort_by_index($$) { > my($A,$B) = @_; > return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; > } > @match_indices = sort sort_by_index @match_indices; > ----------------------------------------
It has changed in devel: pc09:/pro/3gl/CPAN/perl-current 105 > ./perl -Ilib ~/test.pl Bizarre copy of UNKNOWN in aelem at /home/merijn/test.pl line 16. Show quoted text
> It segfaults for me on the last line, both with perl 5.8.8 on fedora > core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes > back to at least perl 5.6, but I don't have good notes. > > Here's the backtrace on 5.8.8 built with debugging: > > (gdb) run -Ilib /tmp/t.pl > Starting program: /usr/src/redhat/BUILD/perl-5.8.8/perl -Ilib > /tmp/t.pl > Reading symbols from shared object read from target memory...done. > Loaded system supplied DSO at 0xd88000 > [Thread debugging using libthread_db enabled] > [New Thread -1208506688 (LWP 31509)] > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread -1208506688 (LWP 31509)] > 0x0069c8f0 in Perl_pp_aelem (my_perl=0x89b1008) at pp_hot.c:3007 > 3007 IV elem = SvIV(elemsv); > (gdb) bt > #0 0x0069c8f0 in Perl_pp_aelem (my_perl=0x89b1008) at pp_hot.c:3007 > #1 0x00697937 in Perl_runops_standard (my_perl=0x89b1008) > at run.c:37 > #2 0x00712860 in sortcv_stacked (my_perl=0x89b1008, a=0x89cd5c0, > b=0x89cd5cc) at pp_sort.c:1791 > #3 0x0071065e in S_mergesortsv (my_perl=0x89b1008, base=Variable > "base" is not available. > ) > at pp_sort.c:429 > #4 0x007119d2 in Perl_sortsv (my_perl=0x89b1008, array=0x89d1a10, > nmemb=4, cmp=0x7127f1 <sortcv_stacked>) at pp_sort.c:1458 > #5 0x007120b7 in Perl_pp_sort (my_perl=0x89b1008) at pp_sort.c:1685 > #6 0x00697937 in Perl_runops_standard (my_perl=0x89b1008) > at run.c:37 > #7 0x0063d989 in perl_run (my_perl=0x89b1008) at perl.c:2367 > #8 0x080492f4 in main (argc=3, argv=0xbfe9a544, env=0xbfe9a554) > at perlmain.c:99 > (gdb) q > > > The test program is 100% repeatable. I extracted it from a larger > program where I get intermittent problems depending on the input (but > always in the sort routine). > > For a different input I got a different traceback: > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread -1208793408 (LWP 31157)] > 0x008c6ccd in Perl_mg_get (my_perl=0x81f7008, sv=0x8506ac0) > at mg.c:164 > 164 newmg = cur = head = mg = SvMAGIC(sv); > (gdb) bt > #0 0x008c6ccd in Perl_mg_get (my_perl=0x81f7008, sv=0x8506ac0) > at mg.c:164 > #1 0x008e8695 in Perl_sv_setsv_flags (my_perl=0x81f7008, > dstr=0x821e200, sstr=Variable "sstr" is not available. > ) at sv.c:3856 > #2 0x008e9d2b in Perl_sv_mortalcopy (my_perl=0x81f7008, > oldstr=0x8506ac0) at sv.c:6814 > #3 0x008d8b81 in Perl_pp_aelem (my_perl=0x81f7008) at pp_hot.c:3055 > #4 0x008d3937 in Perl_runops_standard (my_perl=0x81f7008) > at run.c:37 > #5 0x0094e860 in sortcv_stacked (my_perl=0x81f7008, a=0x876a934, > b=0x876a868) at pp_sort.c:1791 > #6 0x0094c65e in S_mergesortsv (my_perl=0x81f7008, base=Variable > "base" is not available. > ) > at pp_sort.c:429 > #7 0x0094d9d2 in Perl_sortsv (my_perl=0x81f7008, array=0x8506ab0, > nmemb=4, cmp=0x94e7f1 <sortcv_stacked>) at pp_sort.c:1458 > #8 0x0094e0b7 in Perl_pp_sort (my_perl=0x81f7008) at pp_sort.c:1685 > #9 0x008d3937 in Perl_runops_standard (my_perl=0x81f7008) > at run.c:37 > #10 0x00879989 in perl_run (my_perl=0x81f7008) at perl.c:2367 > #11 0x080492f4 in main (argc=6, argv=0xbfe54dd4, env=0xbfe54df0) > at perlmain.c:99 > > (I didn't isolate this to a test kernel.) > > Any suggestions? > > Thanks, > -John Heidemann > > > [Please do not change anything below this line] > ----------------------------------------------------------------- > --- > Flags: > category=core > severity=high > --- > This perlbug was built using Perl v5.8.8 in the Red Hat build system. > It is being executed now by Perl v5.8.8 - Wed Mar 1 18:29:13 EST 2006. > > Site configuration information for perl v5.8.8: > > Configured by Red Hat, Inc. at Wed Mar 1 18:29:13 EST 2006. > > Summary of my perl5 (revision 5 version 8 subversion 8) configuration: > Platform: > osname=linux, osvers=2.6.9-22.18.bz155725.elsmp, archname=i386-linux-thread-multi > uname='linux hs20-bc1-6.build.redhat.com 2.6.9-22.18.bz155725.elsmp #1 smp thu nov 17 15:34:08 est 2005 i686 i686 i386 gnulinux ' > config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -Dversion=5.8.8 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dinc_version_list=5.8.7 5.8.6 5.8.5 5.8.4 5.8.3 -Dscriptdir=/usr/bin' > hint=recommended, useposix=true, d_sigaction=define > usethreads=define use5005threads=undef useithreads=define usemultiplicity=define > useperlio=define d_sfio=undef uselargefiles=define usesocks=undef > use64bitint=undef use64bitall=undef uselongdouble=undef > usemymalloc=n, bincompat5005=undef > Compiler: > cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm', > optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables', > cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include -I/usr/include/gdbm' > ccversion='', gccversion='4.1.0 20060228 (Red Hat 4.1.0-1)', gccosandvers='' > intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 > d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 > ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 > alignbytes=4, prototype=define > Linker and Libraries: > ld='gcc', ldflags =' -L/usr/local/lib' > libpth=/usr/local/lib /lib /usr/lib > libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc > perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc > libc=/lib/libc-2.3.90.so, so=so, useshrplib=true, libperl=libperl.so > gnulibc_version='2.3.90' > Dynamic Linking: > dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE' > cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib' > > Locally applied patches: > > > --- > @INC for perl v5.8.8: > /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.4/i386-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.3/i386-linux-thread-multi > /usr/lib/perl5/site_perl/5.8.8 > /usr/lib/perl5/site_perl/5.8.7 > /usr/lib/perl5/site_perl/5.8.6 > /usr/lib/perl5/site_perl/5.8.5 > /usr/lib/perl5/site_perl/5.8.4 > /usr/lib/perl5/site_perl/5.8.3 > /usr/lib/perl5/site_perl > /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.4/i386-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.3/i386-linux-thread-multi > /usr/lib/perl5/vendor_perl/5.8.8 > /usr/lib/perl5/vendor_perl/5.8.7 > /usr/lib/perl5/vendor_perl/5.8.6 > /usr/lib/perl5/vendor_perl/5.8.5 > /usr/lib/perl5/vendor_perl/5.8.4 > /usr/lib/perl5/vendor_perl/5.8.3 > /usr/lib/perl5/vendor_perl > /usr/lib/perl5/5.8.8/i386-linux-thread-multi > /usr/lib/perl5/5.8.8 > . > > --- > Environment for perl v5.8.8: > HOME=/home/johnh > LANG=en_US.UTF-8 > LANGUAGE (unset) > LD_LIBRARY_PATH=/usr/local/lib > LOGDIR (unset) > PATH=/home/johnh/BIN:/home/johnh/BIN/LINUX:/usr/local/bin:/bin:/usr/bin:/home/johnh/BIN/WP:/home/johnh/BIN/HOSTS:/home/johnh/BIN/DB:/usr/X11R6/bin:/usr/local/etc:/usr/local/sbin:/etc:/usr/etc:/sbin:/usr/sbin:/usr/hosts:/usr/local/games:/usr/games > PERL_BADLANG (unset) > SHELL=/bin/bash > >
-- H.Merijn Brand Amsterdam Perl Mongers (http://amsterdam.pm.org/) using & porting perl 5.6.2, 5.8.x, 5.9.x on HP-UX 10.20, 11.00, 11.11, & 11.23, SuSE 10.0, AIX 4.3 & 5.2, and Cygwin. http://qa.perl.org http://mirrors.develooper.com/hpux/ http://www.test-smoke.org http://www.goldmark.org/jeff/stupid-disclaimers/
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Fri, 9 Jun 2006 11:49:08 -0400
To: perl5-porters [...] perl.org
From: Ronald J Kimball <rjk-perl-p5p [...] tamias.net>
Download (untitled) / with headers
text/plain 830b
On Thu, Jun 08, 2006 at 09:23:41PM -0700, johnh @ isi. edu wrote: Show quoted text
> ---------------------------------------- > #!/usr/bin/perl -w > > @scores = ( > '0000.0000.00039:0000.0008.0031.', > '0000.0000.00032:0000.0008.0024.', > '0002.0002.00033:0002.0011.0020.', > '0028.0028.00190:0077.0085.0028.'); > > @match_indices = (0,1,2,3); > sub sort_by_index($$) { > my($A,$B) = @_; > return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; > } > @match_indices = sort sort_by_index @match_indices; > ---------------------------------------- > > It segfaults for me on the last line, both with perl 5.8.8 on fedora > core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes > back to at least perl 5.6, but I don't have good notes.
I get a segfault in 5.8.7, but it works fine in 5.8.3. Ronald
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Fri, 9 Jun 2006 17:59:02 +0200
To: perl5-porters [...] perl.org
From: Rafael Garcia-Suarez <rgarciasuarez [...] mandriva.com>
Download (untitled) / with headers
text/plain 688b
johnh@isi.edu (via RT) wrote: Show quoted text
> @scores = ( > '0000.0000.00039:0000.0008.0031.', > '0000.0000.00032:0000.0008.0024.', > '0002.0002.00033:0002.0011.0020.', > '0028.0028.00190:0077.0085.0028.'); > > @match_indices = (0,1,2,3); > sub sort_by_index($$) { > my($A,$B) = @_; > return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; > } > @match_indices = sort sort_by_index @match_indices;
I don't understand the algorithm, but making a copy of @match_indices and using that in sort_by_index() solves the problem. Also, the $$ is necessary for this problem. Might be related to some unclean things perl does to pass arguments to the sort routine rapidly... --
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Fri, 09 Jun 2006 09:34:41 -0700
To: perlbug-followup [...] perl.org
From: John Heidemann <johnh [...] isi.edu>
Download (untitled) / with headers
text/plain 980b
On Fri, 09 Jun 2006 08:59:10 PDT, "Rafael Garcia-Suarez via RT" wrote: Show quoted text
>johnh@isi.edu (via RT) wrote:
>> @scores = ( >> '0000.0000.00039:0000.0008.0031.', >> '0000.0000.00032:0000.0008.0024.', >> '0002.0002.00033:0002.0011.0020.', >> '0028.0028.00190:0077.0085.0028.'); >> >> @match_indices = (0,1,2,3); >> sub sort_by_index($$) { >> my($A,$B) = @_; >> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; >> } >> @match_indices = sort sort_by_index @match_indices;
> >I don't understand the algorithm, but making a copy of @match_indices >and using that in sort_by_index() solves the problem. > >Also, the $$ is necessary for this problem. Might be related to some >unclean things perl does to pass arguments to the sort routine >rapidly... >-- >
the ($$) is not required. I get the same segfault with: sub sort_by_index { return $scores[$match_indices[$a]] cmp $scores[$match_indices[$b]]; } (on perl-5.8.8 / fedora core 5). -John
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Fri, 09 Jun 2006 09:32:51 -0700
To: perlbug-followup [...] perl.org
From: John Heidemann <johnh [...] isi.edu>
Download (untitled) / with headers
text/plain 1.5k
On Fri, 09 Jun 2006 08:49:56 PDT, "Ronald J Kimball via RT" wrote: Show quoted text
>On Thu, Jun 08, 2006 at 09:23:41PM -0700, johnh @ isi. edu wrote: >
>> ---------------------------------------- >> #!/usr/bin/perl -w >> >> @scores = ( >> '0000.0000.00039:0000.0008.0031.', >> '0000.0000.00032:0000.0008.0024.', >> '0002.0002.00033:0002.0011.0020.', >> '0028.0028.00190:0077.0085.0028.'); >> >> @match_indices = (0,1,2,3); >> sub sort_by_index($$) { >> my($A,$B) = @_; >> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; >> } >> @match_indices = sort sort_by_index @match_indices; >> ---------------------------------------- >> >> It segfaults for me on the last line, both with perl 5.8.8 on fedora >> core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes >> back to at least perl 5.6, but I don't have good notes.
> >I get a segfault in 5.8.7, but it works fine in 5.8.3.
For me, the bug was inconsistent. Depending on the arguments passed to the larger program, sometimes it was triggered and sometimes not. A colleague of mine pointed out that there are two bugs here: 1. perl shouldn't (ever) segfault 2. the code doesn't do quite what I thought it should It actually has a mistaken double indirection. The subroutine should just be: sub sort_by_index($$) { my($A,$B) = @_; return $scores[$A] cmp $scores[$B]; } Fixing bug #2, which conincidently makes the segfault go away for me. I'll leave the original code and bug #1 to you folks, since I don't know perl internals :-) -John
CC: perl5-porters [...] perl.org
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Fri, 9 Jun 2006 15:45:52 -0500 (CDT)
To: "Rafael Garcia-Suarez" <rgarciasuarez [...] mandriva.com>
From: "Graham Barr" <gbarr [...] pobox.com>
Download (untitled) / with headers
text/plain 1010b
On Fri, June 9, 2006 10:59 am, Rafael Garcia-Suarez wrote: Show quoted text
> johnh@isi.edu (via RT) wrote:
>> @scores = ( >> '0000.0000.00039:0000.0008.0031.', >> '0000.0000.00032:0000.0008.0024.', >> '0002.0002.00033:0002.0011.0020.', >> '0028.0028.00190:0077.0085.0028.'); >> >> @match_indices = (0,1,2,3); >> sub sort_by_index($$) { >> my($A,$B) = @_; >> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; >> } >> @match_indices = sort sort_by_index @match_indices;
> > I don't understand the algorithm, but making a copy of @match_indices > and using that in sort_by_index() solves the problem.
That because sort is modifying @match_indices as it goes because it detected that the result is being assigned to the same array being sorted. Simply assigning to a different array also solves it. Try adding warn join(",",@match_indices); into sort_by_index and see what you get. It would seem that when doing sort in-place that the array is at times in a non-consistent state. Graham.
Subject: Re: [perl #39358] sort with custom subname and prototype ($$)segfaults intermittently
Date: Sat, 10 Jun 2006 09:33:08 +0200
To: perl5-porters [...] perl.org, Rafael Garcia-Suarez <rgarciasuarez [...] mandriva.com>
From: Salvador Fandiño <sfandino [...] yahoo.com>
Download (untitled) / with headers
text/plain 1.1k
Rafael Garcia-Suarez wrote: Show quoted text
> johnh@isi.edu (via RT) wrote:
>> @scores = ( >> '0000.0000.00039:0000.0008.0031.', >> '0000.0000.00032:0000.0008.0024.', >> '0002.0002.00033:0002.0011.0020.', >> '0028.0028.00190:0077.0085.0028.'); >> >> @match_indices = (0,1,2,3); >> sub sort_by_index($$) { >> my($A,$B) = @_; >> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; >> } >> @match_indices = sort sort_by_index @match_indices;
> > I don't understand the algorithm, but making a copy of @match_indices > and using that in sort_by_index() solves the problem. > > Also, the $$ is necessary for this problem. Might be related to some > unclean things perl does to pass arguments to the sort routine > rapidly...
This is probably related to @data = sort compare @data; being optimized into sort_inplace compare @data; that causes @data to be in an inconsistent state when accessed from &compare. The solution may be to remove the optimization or to disallow access to @data from inside the sort in some way (using magic or localizing it to something that croaks when accessed). Cheers, - Salva
CC: perl5-porters [...] perl.org, Rafael Garcia-Suarez <rgarciasuarez [...] mandriva.com>
Subject: Re: [perl #39358] sort with custom subname and prototype ($$)segfaults intermittently
Date: Sun, 11 Jun 2006 21:34:38 +0100
To: Salvador Fandiño <sfandino [...] yahoo.com>
From: Dave Mitchell <davem [...] iabyn.com>
On Sat, Jun 10, 2006 at 09:33:08AM +0200, Salvador Fandiño wrote: Show quoted text
> This is probably related to > > @data = sort compare @data; > > being optimized into > > sort_inplace compare @data; > > that causes @data to be in an inconsistent state when accessed from > &compare. > > The solution may be to remove the optimization or to disallow access to > @data from inside the sort in some way (using magic or localizing it to > something that croaks when accessed).
Yes, the mergesort stores temporary (non-SV) pointers in the SV** array while sorting. (Hmm, that should nicely mess up -Dsv output too). I think the fix is to, in the presence of a sort function, memcopy the AvARRAY onto the stack and then sort that, then copy back to AvARRAY. Note that this is still a lot more efficient than the unoptimised @a = sort @a, which created a copy of every element, then deleted each original element. I'll fix this when I get some time. -- The optimist believes that he lives in the best of all possible worlds. As does the pessimist.
CC: perlbug-followup [...] perl.org
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Sun, 11 Jun 2006 22:26:23 +0200
To: John Heidemann <johnh [...] isi.edu>
From: David Landgren <david [...] landgren.net>
Download (untitled) / with headers
text/plain 1.8k
John Heidemann wrote: Show quoted text
> On Fri, 09 Jun 2006 08:49:56 PDT, "Ronald J Kimball via RT" wrote:
>> On Thu, Jun 08, 2006 at 09:23:41PM -0700, johnh @ isi. edu wrote: >>
>>> ---------------------------------------- >>> #!/usr/bin/perl -w >>> >>> @scores = ( >>> '0000.0000.00039:0000.0008.0031.', >>> '0000.0000.00032:0000.0008.0024.', >>> '0002.0002.00033:0002.0011.0020.', >>> '0028.0028.00190:0077.0085.0028.'); >>> >>> @match_indices = (0,1,2,3); >>> sub sort_by_index($$) { >>> my($A,$B) = @_; >>> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; >>> } >>> @match_indices = sort sort_by_index @match_indices; >>> ---------------------------------------- >>> >>> It segfaults for me on the last line, both with perl 5.8.8 on fedora >>> core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes >>> back to at least perl 5.6, but I don't have good notes.
>> I get a segfault in 5.8.7, but it works fine in 5.8.3.
> > For me, the bug was inconsistent. Depending on the arguments passed > to the larger program, sometimes it was triggered and sometimes not. > > A colleague of mine pointed out that there are two bugs here: > 1. perl shouldn't (ever) segfault
Agreed. Show quoted text
> 2. the code doesn't do quite what I thought it should > > It actually has a mistaken double indirection. The subroutine should > just be: > > sub sort_by_index($$) { > my($A,$B) = @_; > return $scores[$A] cmp $scores[$B]; > }
I would also lose the prototypes, and the assignment from @_ to $A and $B. You are introducing an unnecessary slowdown, since a sort comparator is considered to be so expensive that perl already arranges to have an $a and $b set up when the routine is called, to lessen the cost. I can't think of any reason off-hand why one would want to remove this optimisation. David -- hope still, a little resistance always maybe stubborn tiny lights vs. clustering darkness forever ok?
CC: John Heidemann <johnh [...] isi.edu>, perlbug-followup [...] perl.org
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Sun, 11 Jun 2006 22:20:19 -0700
To: David Landgren <david [...] landgren.net>
From: Yitzchak Scott-Thoennes <sthoenna [...] efn.org>
Download (untitled) / with headers
text/plain 1.9k
On Sun, Jun 11, 2006 at 10:26:23PM +0200, David Landgren wrote: Show quoted text
> John Heidemann wrote:
> >On Fri, 09 Jun 2006 08:49:56 PDT, "Ronald J Kimball via RT" wrote:
> >>On Thu, Jun 08, 2006 at 09:23:41PM -0700, johnh @ isi. edu wrote: > >>
> >>>---------------------------------------- > >>>#!/usr/bin/perl -w > >>> > >>>@scores = ( > >>> '0000.0000.00039:0000.0008.0031.', > >>> '0000.0000.00032:0000.0008.0024.', > >>> '0002.0002.00033:0002.0011.0020.', > >>> '0028.0028.00190:0077.0085.0028.'); > >>> > >>>@match_indices = (0,1,2,3); > >>>sub sort_by_index($$) { > >>> my($A,$B) = @_; > >>> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; > >>>} > >>>@match_indices = sort sort_by_index @match_indices; > >>>---------------------------------------- > >>> > >>>It segfaults for me on the last line, both with perl 5.8.8 on fedora > >>>core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes > >>>back to at least perl 5.6, but I don't have good notes.
> >>I get a segfault in 5.8.7, but it works fine in 5.8.3.
> > > >For me, the bug was inconsistent. Depending on the arguments passed > >to the larger program, sometimes it was triggered and sometimes not. > > > >A colleague of mine pointed out that there are two bugs here: > >1. perl shouldn't (ever) segfault
> > Agreed. >
> >2. the code doesn't do quite what I thought it should > > > >It actually has a mistaken double indirection. The subroutine should > >just be: > > > >sub sort_by_index($$) { > > my($A,$B) = @_; > > return $scores[$A] cmp $scores[$B]; > >}
> > I would also lose the prototypes, and the assignment from @_ to $A and > $B. You are introducing an unnecessary slowdown, since a sort comparator > is considered to be so expensive that perl already arranges to have an > $a and $b set up when the routine is called, to lessen the cost. > > I can't think of any reason off-hand why one would want to remove this > optimisation.
Where the sort_by_index resides in a different package.
CC: johnh [...] isi.edu, david [...] landgren.net
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Mon, 12 Jun 2006 06:48:25 -0400
To: perl5-porters [...] perl.org
From: "John P. Linderman" <jpl [...] research.att.com>
Download (untitled) / with headers
text/plain 2.9k
David Landgren <david@landgren.net> replied to John Heidemann <johnh@isi.edu>: Show quoted text
>>>> #!/usr/bin/perl -w >>>> >>>> @scores = ( >>>> '0000.0000.00039:0000.0008.0031.', >>>> '0000.0000.00032:0000.0008.0024.', >>>> '0002.0002.00033:0002.0011.0020.', >>>> '0028.0028.00190:0077.0085.0028.'); >>>> >>>> @match_indices = (0,1,2,3); >>>> sub sort_by_index($$) { >>>> my($A,$B) = @_; >>>> return $scores[$match_indices[$A]] cmp $scores[$match_indices[$B]]; >>>> } >>>> @match_indices = sort sort_by_index @match_indices; >>>> ---------------------------------------- >>>> >>>> It segfaults for me on the last line, both with perl 5.8.8 on fedora >>>> core 5, and with perl 5.8.6 on fedora core 4. I believe the bug goes >>>> back to at least perl 5.6, but I don't have good notes.
>>> I get a segfault in 5.8.7, but it works fine in 5.8.3.
>> >> For me, the bug was inconsistent. Depending on the arguments passed >> to the larger program, sometimes it was triggered and sometimes not. >> >> A colleague of mine pointed out that there are two bugs here: >> 1. perl shouldn't (ever) segfault
> >Agreed. >
>> 2. the code doesn't do quite what I thought it should >> >> It actually has a mistaken double indirection. The subroutine should >> just be: >> >> sub sort_by_index($$) { >> my($A,$B) = @_; >> return $scores[$A] cmp $scores[$B]; >> }
Bug 1 trumps bug 2. During the sort, there's a second, parallel, array allocated for the merge sort. The original array and the parallel array swap roles as the merging takes place. The array that doesn't currently hold pointers to the original array elements holds pointers into the other array, delimiting the runs of sorted elements. Any attempt to use these pointers to runs as pointers to array elements is likely to come to grief. Grief will be intermittent: if you go into the original array when it is holding the sorted runs, you'll be ok. If you go into it when it holds pointers to the runs, Segfault City (although even this will be intermittent, since there's only one pointer per run, with other pointers still referring to original array elements). This may have crept in with the in-place optimization, although it would be possible any time the comparison routine had access to the original array. Such a comparison routine is almost certainly in error, because it violates the mandate that the comparison routine be consistent, and, as elements move around in the array during the sort, maintaining consistent results would be nearly impossible. It might be possible to fix this by keeping the array pointer pointing to the "correct array", but changing the identity of the array (as might be seen by stringifying it) during the sort operation seems wrong. If possible, it would seem best to disallow references to the array while the sort was ongoing, particularly since they are almost certainly in error as above, but I don't know if this is possible. Otherwise, the merge sort implementation would need some major changes. -- jpl
CC: perl5-porters [...] perl.org
Subject: Re: [perl #39358] sort with custom subname and prototype ($$)segfaults intermittently
Date: Mon, 12 Jun 2006 07:27:26 -0400 (EDT)
To: davem [...] iabyn.com
From: "John P. Linderman" <jpl [...] research.att.com>
Download (untitled) / with headers
text/plain 942b
Dave Mitchell <davem@iabyn.com> offered Show quoted text
> Yes, the mergesort stores temporary (non-SV) pointers in the SV** array > while sorting. (Hmm, that should nicely mess up -Dsv output too). > > I think the fix is to, in the presence of a sort function, memcopy the > AvARRAY onto the stack and then sort that, then copy back to AvARRAY. > > Note that this is still a lot more efficient than the unoptimised > @a = sort @a, which created a copy of every element, then deleted each > original element.
Would it be possible to use magic to disable access to the array (as opposed to elements of the array, which will obviously be needed for all comparisons) while the sort is in progress? It's hard to imagine a valid (in the sense of consistent results) comparison routine referring to the array itself. If true, we'd be doing people a favor to alert them to what is almost certainly an error (as was the case in the original bug report). -- jpl
CC: perl5-porters [...] perl.org, johnh [...] isi.edu
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Mon, 12 Jun 2006 17:38:31 +0100
To: "John P. Linderman" <jpl [...] research.att.com>
From: hv [...] crypt.org
Download (untitled) / with headers
text/plain 1.3k
"John P. Linderman" <jpl@research.att.com> wrote: :This may have crept in with the in-place optimization, :although it would be possible any time the comparison :routine had access to the original array. Such a :comparison routine is almost certainly in error, :because it violates the mandate that the comparison :routine be consistent, and, as elements move around :in the array during the sort, maintaining consistent :results would be nearly impossible. But "in-place sorting with @x = sort @x" is an optimisation, and as such it is surely our responsibility to ensure it acts the same way as without the optimisation, ie to act as if the assignment to @x happens atomically after the sort has completed? :It might be possible to fix this by keeping the :array pointer pointing to the "correct array", :but changing the identity of the array :(as might be seen by stringifying it) :during the sort operation seems wrong. : :If possible, it would seem best to disallow references :to the array while the sort was ongoing, particularly :since they are almost certainly in error as above, but I don't :know if this is possible. Otherwise, the merge sort :implementation would need some major changes. -- jpl I doubt we can detect them (since they may occur via aliasing etc), but if we can it would be better to disable the in-place optimisation in that case rather than complain. Hugo
CC: "John P. Linderman" <jpl [...] research.att.com>, perl5-porters [...] perl.org, johnh [...] isi.edu
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Date: Mon, 12 Jun 2006 17:42:47 +0100
To: hv [...] crypt.org
From: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 567b
On Mon, Jun 12, 2006 at 05:38:31PM +0100, hv@crypt.org wrote: Show quoted text
> I doubt we can detect them (since they may occur via aliasing etc), > but if we can it would be better to disable the in-place optimisation > in that case rather than complain.
I think my suggestion of the partial pessimistion of memcopying AvARRAY onto the stack then back again should fix this. During the course of the sort, any sort sub sees the original array. -- "There's something wrong with our bloody ships today, Chatfield." -- Admiral Beatty at the Battle of Jutland, 31st May 1916.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1010b
On Thu Jun 08 21:23:40 2006, johnh@isi.edu wrote: Show quoted text
> #!/usr/bin/perl -w > > @scores = ( > '0000.0000.00039:0000.0008.0031.', > '0000.0000.00032:0000.0008.0024.', > '0002.0002.00033:0002.0011.0020.', > '0028.0028.00190:0077.0085.0028.'); > > @match_indices = (0,1,2,3); > sub sort_by_index($$) { > my($A,$B) = @_; > return $scores[$match_indices[$A]] cmp > $scores[$match_indices[$B]]; > } > @match_indices = sort sort_by_index @match_indices; > ----------------------------------------
This gives me: Bizarre copy of UNKNOWN in aelem at - line 12. in commit 42bd538f820268331bc55a66fa9df0673de69250 d963bf01c4c4db296760b1148f98bf668efcaf58, three commits later (the intervening commits affecting only documentation), gives Use of uninitialized value $match_indices[2] in array element at - line 12. Use of uninitialized value $match_indices[2] in array element at - line 12. Use of uninitialized value $match_indices[0] in array element at - line 12. but I can’t see how it fixes it.
RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 18.9k

Message body is not shown because it is too large.

Date: Wed, 10 Aug 2016 16:36:39 +0100
CC: perl5-porters [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
To: Father Chrysostomos via RT <perlbug-comment [...] perl.org>
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Download (untitled) / with headers
text/plain 4.1k
On Sun, Sep 26, 2010 at 02:39:49PM -0700, Father Chrysostomos via RT wrote: Show quoted text
> On Thu Jun 08 21:23:40 2006, johnh@isi.edu wrote:
> > #!/usr/bin/perl -w > > > > @scores = ( > > '0000.0000.00039:0000.0008.0031.', > > '0000.0000.00032:0000.0008.0024.', > > '0002.0002.00033:0002.0011.0020.', > > '0028.0028.00190:0077.0085.0028.'); > > > > @match_indices = (0,1,2,3); > > sub sort_by_index($$) { > > my($A,$B) = @_; > > return $scores[$match_indices[$A]] cmp > > $scores[$match_indices[$B]]; > > } > > @match_indices = sort sort_by_index @match_indices; > > ----------------------------------------
> > This gives me: > Bizarre copy of UNKNOWN in aelem at - line 12. > in commit 42bd538f820268331bc55a66fa9df0673de69250 > > d963bf01c4c4db296760b1148f98bf668efcaf58, three commits later (the > intervening commits affecting only documentation), gives > Use of uninitialized value $match_indices[2] in array element at - line 12. > Use of uninitialized value $match_indices[2] in array element at - line 12. > Use of uninitialized value $match_indices[0] in array element at - line 12. > > but I can’t see how it fixes it.
Fixed now in blead by the following. The issue was as the bit below about mergesort leaving random pointers lying about. commit 84721d614eb7d9835d9a09505b0001c7be40a865 Author: David Mitchell <davem@iabyn.com> AuthorDate: Wed Aug 10 15:12:56 2016 +0100 Commit: David Mitchell <davem@iabyn.com> CommitDate: Wed Aug 10 16:34:04 2016 +0100 Partially pessimise in-place sorting There's currently an optimisation that converts at compile-time @a = sort { .... } @a into (approximately) sort { ... } \@a Then at run time, rather than passing an svp pointer to the appropriate sort routine which points to a list of SV*'s on the stack, pp_sort() passes a pointer to @a's AvARRAY. This allows the array to be sorted in-place, which is more efficient. However, it has some issues. First, the @a visible to the sort routine will be varying, whereas logically it should still hold the original list of values until after the '@a = ...' assignment. Secondly, the mergesort algorithm cureently used internally, when in mid-sort, temporarily stores pointers in the array which aren't pointers to SVs - this means that if @a elements are accessed mid-sort, it can crash. The solution to both these problems is for pp_sort() to push the elements of @a onto the stack at the beginning, sort the stack (like normal sorts do), then copy back to @a at the end. This is less efficient than before, but is still a lot more efficient than executing separate padav and aassign ops. Here are benchmark results in raw instruction counts etc (lower is better) for the sort line in this code block: my (@a, @b); @a = reverse 1..10; @b = sort { $a <=> $b } @a; A is for a non-in-place sort, i.e. @b = sort ... @a; B and C are for an inline sort, i.e. as above, but @a = sort ... @a; where B is blead before this commit and C is this commit. A B C ------ ------ ------ Ir 5238.0 2324.0 2772.0 Dr 1464.0 649.0 765.0 Dw 919.0 298.0 370.0 COND 782.0 320.0 405.0 IND 25.0 25.0 26.0 COND_m 14.9 13.0 17.0 IND_m 8.0 5.0 5.0 As can be seen, this partial pessimisation slows down in-place sorting by round 20%, but overall in-place is still nearly twice the speed as without the optimisation. These are the figures for a plain numeric sort (which is optimised to use a C comparison function); for other types of sort, the cost of the comparator dominates, and so the slowdown is much less marked. -- "But Sidley Park is already a picture, and a most amiable picture too. The slopes are green and gentle. The trees are companionably grouped at intervals that show them to advantage. The rill is a serpentine ribbon unwound from the lake peaceably contained by meadows on which the right amount of sheep are tastefully arranged." -- Lady Croom, "Arcadia"
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
To: Dave Mitchell <davem [...] iabyn.com>, Father Chrysostomos via RT <perlbug-comment [...] perl.org>
CC: perl5-porters [...] perl.org
From: Karl Williamson <public [...] khwilliamson.com>
Date: Wed, 10 Aug 2016 11:48:54 -0600
Download (untitled) / with headers
text/plain 1.7k
On 08/10/2016 09:36 AM, Dave Mitchell wrote: Show quoted text
> On Sun, Sep 26, 2010 at 02:39:49PM -0700, Father Chrysostomos via RT wrote:
>> On Thu Jun 08 21:23:40 2006, johnh@isi.edu wrote:
>>> #!/usr/bin/perl -w >>> >>> @scores = ( >>> '0000.0000.00039:0000.0008.0031.', >>> '0000.0000.00032:0000.0008.0024.', >>> '0002.0002.00033:0002.0011.0020.', >>> '0028.0028.00190:0077.0085.0028.'); >>> >>> @match_indices = (0,1,2,3); >>> sub sort_by_index($$) { >>> my($A,$B) = @_; >>> return $scores[$match_indices[$A]] cmp >>> $scores[$match_indices[$B]]; >>> } >>> @match_indices = sort sort_by_index @match_indices; >>> ----------------------------------------
>> >> This gives me: >> Bizarre copy of UNKNOWN in aelem at - line 12. >> in commit 42bd538f820268331bc55a66fa9df0673de69250 >> >> d963bf01c4c4db296760b1148f98bf668efcaf58, three commits later (the >> intervening commits affecting only documentation), gives >> Use of uninitialized value $match_indices[2] in array element at - line 12. >> Use of uninitialized value $match_indices[2] in array element at - line 12. >> Use of uninitialized value $match_indices[0] in array element at - line 12. >> >> but I can’t see how it fixes it.
> > Fixed now in blead by the following. The issue was as the bit below about > mergesort leaving random pointers lying about. > > commit 84721d614eb7d9835d9a09505b0001c7be40a865 > Author: David Mitchell <davem@iabyn.com> > AuthorDate: Wed Aug 10 15:12:56 2016 +0100 > Commit: David Mitchell <davem@iabyn.com>
blead is failing to compile for me unless I revert this, with the message: pp_sort.c: In function ‘OP* Perl_pp_sort(PerlInterpreter*)’: pp_sort.c:1599:26: error: invalid conversion from ‘OP* {aka op*}’ to ‘char’ [-fpermissive] copytmps = PL_sortcop;
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
To: Dave Mitchell <davem [...] iabyn.com>, Father Chrysostomos via RT <perlbug-comment [...] perl.org>
CC: perl5-porters [...] perl.org
From: Karl Williamson <public [...] khwilliamson.com>
Date: Wed, 10 Aug 2016 12:27:58 -0600
Download (untitled) / with headers
text/plain 1.9k
On 08/10/2016 11:48 AM, Karl Williamson wrote: Show quoted text
> On 08/10/2016 09:36 AM, Dave Mitchell wrote:
>> On Sun, Sep 26, 2010 at 02:39:49PM -0700, Father Chrysostomos via RT >> wrote:
>>> On Thu Jun 08 21:23:40 2006, johnh@isi.edu wrote:
>>>> #!/usr/bin/perl -w >>>> >>>> @scores = ( >>>> '0000.0000.00039:0000.0008.0031.', >>>> '0000.0000.00032:0000.0008.0024.', >>>> '0002.0002.00033:0002.0011.0020.', >>>> '0028.0028.00190:0077.0085.0028.'); >>>> >>>> @match_indices = (0,1,2,3); >>>> sub sort_by_index($$) { >>>> my($A,$B) = @_; >>>> return $scores[$match_indices[$A]] cmp >>>> $scores[$match_indices[$B]]; >>>> } >>>> @match_indices = sort sort_by_index @match_indices; >>>> ----------------------------------------
>>> >>> This gives me: >>> Bizarre copy of UNKNOWN in aelem at - line 12. >>> in commit 42bd538f820268331bc55a66fa9df0673de69250 >>> >>> d963bf01c4c4db296760b1148f98bf668efcaf58, three commits later (the >>> intervening commits affecting only documentation), gives >>> Use of uninitialized value $match_indices[2] in array element at - >>> line 12. >>> Use of uninitialized value $match_indices[2] in array element at - >>> line 12. >>> Use of uninitialized value $match_indices[0] in array element at - >>> line 12. >>> >>> but I can’t see how it fixes it.
>> >> Fixed now in blead by the following. The issue was as the bit below about >> mergesort leaving random pointers lying about. >> >> commit 84721d614eb7d9835d9a09505b0001c7be40a865 >> Author: David Mitchell <davem@iabyn.com> >> AuthorDate: Wed Aug 10 15:12:56 2016 +0100 >> Commit: David Mitchell <davem@iabyn.com>
> > > blead is failing to compile for me unless I revert this, with the message: > > pp_sort.c: In function ‘OP* Perl_pp_sort(PerlInterpreter*)’: > pp_sort.c:1599:26: error: invalid conversion from ‘OP* {aka op*}’ to > ‘char’ [-fpermissive] > copytmps = PL_sortcop; > > > > >
It does compile with some options. Attached is a diff with, the options it works with marked by '<', and those it doesn't, by '>'
Download diff.out
text/plain 2.6k

Message body is not shown because sender requested not to inline it.

RT-Send-CC: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 394b
On Wed Aug 10 10:49:29 2016, public@khwilliamson.com wrote: Show quoted text
> blead is failing to compile for me unless I revert this, with the > message: > > pp_sort.c: In function ‘OP* Perl_pp_sort(PerlInterpreter*)’: > pp_sort.c:1599:26: error: invalid conversion from ‘OP* {aka op*}’ to > ‘char’ [-fpermissive] > copytmps = PL_sortcop;
Fixed in ff859a7f357d. -- Father Chrysostomos
From: "John P. Linderman" <jpl.jpl [...] gmail.com>
Date: Wed, 10 Aug 2016 17:03:28 -0400
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
To: Perl5 Porters <perl5-porters [...] perl.org>
Show quoted text
#!/usr/bin/perl -w
@scores = (
  '0000.0000.00039:0000.0008.0031.',
  '0000.0000.00032:0000.0008.0024.',
  '0002.0002.00033:0002.0011.0020.',
  '0028.0028.00190:0077.0085.0028.');
@match_indices = (0,1,2,3);
sub sort_by_index($$) {
    my($A,$B) = @_;
    return $scores[$match_indices[$A]] cmp
   $scores[$match_indices[$B]];
}
@match_indices = sort sort_by_index @match_indices;
Show quoted text

This could be "fixed" at the expense of allocating two extra arrays instead of just one (which is already more than quicksort, which sorts "in place"). But I wonder why people think that the contents of any array in the process of being sorted can participate in any meaningful way in a comparison. Must all sort implementations guarantee that the intermediate contents of the array to be sorted are some permutation of the original array? I don't like the idea of penalizing all users for the dubious intent of some users. Segfaults are evil, of course. A better solution would be to somehow mark an array as "off limits" while a sort is in progress, but that's way beyond my pay grade.
To: "John P. Linderman" <jpl.jpl [...] gmail.com>
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
From: Dave Mitchell <davem [...] iabyn.com>
CC: Perl5 Porters <perl5-porters [...] perl.org>
Date: Thu, 11 Aug 2016 09:11:29 +0100
Download (untitled) / with headers
text/plain 2.6k
On Wed, Aug 10, 2016 at 05:03:28PM -0400, John P. Linderman wrote: Show quoted text
> #!/usr/bin/perl -w > @scores = ( > '0000.0000.00039:0000.0008.0031.', > '0000.0000.00032:0000.0008.0024.', > '0002.0002.00033:0002.0011.0020.', > '0028.0028.00190:0077.0085.0028.'); > @match_indices = (0,1,2,3); > sub sort_by_index($$) { > my($A,$B) = @_; > return $scores[$match_indices[$A]] cmp > $scores[$match_indices[$B]]; > } > @match_indices = sort sort_by_index @match_indices; > > > This could be "fixed" at the expense of allocating *two* extra arrays > instead of just one (which is already more than quicksort, which sorts "in > place").
I don't understand any of the above sentence. Its just been fixed, so why is it still "could be"? No extra arrays are created, and I don't understand why creating an extra array would be of benefit. Show quoted text
> But I wonder why people think that the contents of any array in > the process of being sorted can participate in any meaningful way in a > comparison. Must all sort implementations guarantee that the intermediate > contents of the array to be sorted are some permutation of the original > array?
In this code: @a = sort {...} @a at the logical perl level, a list is created from the contents of @a, sort returns a sorted version of that list, then that list is assigned to @a. If @a gets modified during the sort itself (and before the assignment), then that's internal optimisation details leaking out. The internal quicksort and mergesort implementations sort a (C) array of SV pointers - usually that array is just a chunk of perl's argument stack. Before my commit, they someones also worked directly on an AvARRAY array, which turned out to be a bad idea. Show quoted text
> I don't like the idea of penalizing *all* users for the dubious > intent of *some* users. Segfaults are evil, of course. A better solution > would be to somehow mark an array as "off limits" while a sort is in > progress, but that's way beyond my pay grade.
The previous code temporarily marked the array as readonly, which stopped modification, but didn't stop access. The only way to stop reading would be to temporarily tie the array. But this wuld be semantically wrong: there's nothing to imply in the expression something = sort { ... } @a that accessing @a during the sort would be an error. -- "But Sidley Park is already a picture, and a most amiable picture too. The slopes are green and gentle. The trees are companionably grouped at intervals that show them to advantage. The rill is a serpentine ribbon unwound from the lake peaceably contained by meadows on which the right amount of sheep are tastefully arranged." -- Lady Croom, "Arcadia"
Date: Thu, 11 Aug 2016 12:07:55 -0400
From: "John P. Linderman" <jpl.jpl [...] gmail.com>
CC: Perl5 Porters <perl5-porters [...] perl.org>
To: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #39358] sort with custom subname and prototype ($$) segfaults intermittently
Download (untitled) / with headers
text/plain 4.9k
I think we are in violent agreement. As you noted, long ago the internal sort worked directly on an AvARRAY, and, when the sort algorithm was quicksort, that was not a bad idea. The contents of that array could be rearranged, but they always contained SV pointers. When I ported Peter McIlroy's mergesort to Perl, I allocated another array of the same size, and, during the initial pass, it held pointers into the original array to delimit runs. On subsequent passes, however, the roles of the two arrays were reversed, one holding the original SV pointers, the other pointers into that array. So in every other pass, the original array held pointers to things that were most assuredly not SV's. Accessing them as though they were SV pointers generally lead to segfaults.

What I meant to say is that changes could have been made to the sort routine so that an AvARRAY could continue to be the argument. The sort could have allocated two extra arrays, and copied the original into one of the arrays, so the contents of the original would never be anything other than SV pointers. This would have cost extra copying and extra space, but this is arguably better than risking segfaults. It appears you have paid the copying and allocating expenses at a higher level, and I certainly have no problem with that. It has the effect of making the copy of the array that is participating in the sort "off limits".

The other point I was trying to make is that access to the SV pointers that are in the process of being rearranged (now impossible) was always asking for trouble. In the code in question

$match_indices[$A]

is simply $A, and would have been better written as such. If that expression referenced a partially rearranged version of the original array, then the comparison routine would probably "violate the contract". For example, if the first two elements of @match_indices were reversed, then the sense of the comparison for elements 0 and 1 would also be reversed, and the contract demands that the comparison routine be consistent when the same two elements are compared.

In (not so very) short, comparison routines that reference the array that is being sorted are probably not doing what the user expects.

On Thu, Aug 11, 2016 at 4:11 AM, Dave Mitchell <davem@iabyn.com> wrote:
Show quoted text
On Wed, Aug 10, 2016 at 05:03:28PM -0400, John P. Linderman wrote:
> #!/usr/bin/perl -w
> @scores = (
>   '0000.0000.00039:0000.0008.0031.',
>   '0000.0000.00032:0000.0008.0024.',
>   '0002.0002.00033:0002.0011.0020.',
>   '0028.0028.00190:0077.0085.0028.');
> @match_indices = (0,1,2,3);
> sub sort_by_index($$) {
>     my($A,$B) = @_;
>     return $scores[$match_indices[$A]] cmp
>    $scores[$match_indices[$B]];
> }
> @match_indices = sort sort_by_index @match_indices;
>
>
> This could be "fixed" at the expense of allocating *two* extra arrays
> instead of just one (which is already more than quicksort, which sorts "in
> place").

I don't understand any of the above sentence. Its just been fixed, so why
is it still "could be"?  No extra arrays are created, and I don't
understand why creating an extra array would be of benefit.

> But I wonder why people think that the contents of any array in
> the process of being sorted can participate in any meaningful way in a
> comparison. Must all sort implementations guarantee that the intermediate
> contents of the array to be sorted are some permutation of the original
> array?

In this code:

    @a = sort {...} @a

at the logical perl level, a list is created from the contents of @a,
sort returns a sorted version of that list, then that list is assigned
to @a. If @a gets modified during the sort itself (and before the
assignment), then that's internal optimisation details leaking out.

The internal quicksort and mergesort implementations sort a (C) array
of SV pointers - usually that array is just a chunk of perl's argument
stack. Before my commit, they someones also worked directly on an AvARRAY
array, which turned out to be a bad idea.

> I don't like the idea of penalizing *all* users for the dubious
> intent of *some* users. Segfaults are evil, of course. A better solution
> would be to somehow mark an array as "off limits" while a sort is in
> progress, but that's way beyond my pay grade.

The previous code temporarily marked the array as readonly, which stopped
modification, but didn't stop access. The only way to stop reading would
be to temporarily tie the array. But this wuld be semantically wrong:
there's nothing to imply in the expression

    something = sort { ... } @a

that accessing @a during the sort would be an error.

--
"But Sidley Park is already a picture, and a most amiable picture too.
The slopes are green and gentle. The trees are companionably grouped at
intervals that show them to advantage. The rill is a serpentine ribbon
unwound from the lake peaceably contained by meadows on which the right
amount of sheep are tastefully arranged." -- Lady Croom, "Arcadia"

Download (untitled) / with headers
text/plain 313b
Thank you for filing this report. You have helped make Perl better. With the release today of Perl 5.26.0, this and 210 other issues have been resolved. Perl 5.26.0 may be downloaded via: https://metacpan.org/release/XSAWYERX/perl-5.26.0 If you find that the problem persists, feel free to reopen this ticket.


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org