endless loop debugging DosGlob #12785

p5pRT · 2013-02-18T03:27:48Z

Migrated from rt.perl.org#116819 (status was 'open')

Searchable as RT116819$

p5pRT · 2013-02-18T03:27:48Z

From @craigberry

Recently when running ext/File-DosGlob/t/DosGlob.t under the Perl debugger, I noticed that it got stuck in an endless loop. I whittled it down to:

$ cat debug_dosglob.t
use File::DosGlob 'glob';

my $previous_dollar_underscore = '';
while (<m*.c>) {
if ($_ eq $previous_dollar_underscore) {
print STDERR "Bzzzt ... don't repeat yourself.\n";
exit 1;
}
$previous_dollar_underscore = $_;
print "# $_ <";
while (<pp*.c>) {
print " $_";
}
print " >\n";
}

exit 0;
[end]

where we know that something went wrong if the outer loop gives us the same result we got on the previous iteration. The script succeeds when run without -d, but with -d it would loop endlessly without the code to detect that we're repeating ourselves and bail out.

After putting the following debugger initialization in my home directory:

$ cat ~/.perldb
&parse_options("NonStop=0 ReadLine=0 TTY=db.out LineInfo=db.out");

sub afterinit {
push(@DB::typeahead,
'c',
'q',
);
}

I was able to bisect it with:

$ git clone perl perl2
$ cd perl2
$ ../perl/Porting/bisect.pl --start=v5.16.0 -- ./perl -Ilib -d ../debug_dosglob.t

...

HEAD is now at c619428 DosGlob: Don’t use the magic 2nd arg to glob
bad - non-zero exit from ./perl -Ilib -d ../debug_dosglob.t
c619428 is the first bad commit
commit c619428
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Apr 27 16:48:36 2012 -0700

DosGlob: Don’t use the magic 2nd arg to glob

Use the address of the glob op instead.

This argument is going away, because it is undocumented, unused on
CPAN outside of the core, and may get in the way of allowing glob() to
be overridden properly.

Another reason is that File::DosGlob leaks memory, because a glob op
freed before iteration has finished will leave File::DosGlob still
holding on to the remainder of the list of files. The easiest way to
fix that will involve using an op address instead of a special index,
so there will be no reason to keep it.

:100644 100644 ac3aac4c4f1b2ee67991d925176b6268fdb5c842 a8ff0e670284a2449d7cb9598e804765c95b3175 M MANIFEST
:040000 040000 936fae86bb0174e55e2ecc48c58d001371cf04f0 ad05e50b1af9eaf181455d7775b3696c1f873642 M ext
bisect run success
That took 2527 seconds

The commit in question adds an XS routine _callsite that consists entirely of:

SV *
_callsite(...)
CODE:
RETVAL = newSVpvn(
(char *)&cxstack[cxstack_ix].blk_sub.retop, sizeof(OP *)
);
OUTPUT:
RETVAL

I'm wondering why it's taking the address of retop since that is already of type OP*, but I may be misparsing this and don't have the time at the moment to dig deeper.

I think the only other necessary bit to log the ticket is:

$ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 17 subversion 9) configuration:
Local Commit: 480c672
Ancestor: 139a355
Platform:
osname=darwin, osvers=12.2.0, archname=darwin-2level
uname='darwin triamond.local 12.2.0 darwin kernel version 12.2.0: sat aug 25 00:48:52 pdt 2012; root:xnu-2050.18.24~1release_x86_64 x86_64 '
config_args='-Dusedevel -Dcc=clang -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='clang', ccflags ='-fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include',
optimize='-O3',
cppflags='-no-cpp-precomp -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include'
ccversion='', gccversion='4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector -L/usr/local/lib -L/opt/local/lib'
libpth=/usr/local/lib /opt/local/lib /usr/lib
libs=-ldbm -ldl -lm -lutil -lc
perllibs=-ldl -lm -lutil -lc
libc=, so=dylib, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib -fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE
PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
Locally applied patches:
480c672
Built under darwin
Compiled at Feb 16 2013 17:46:51
@INC:
lib
/usr/local/lib/perl5/site_perl/5.17.9/darwin-2level
/usr/local/lib/perl5/site_perl/5.17.9
/usr/local/lib/perl5/5.17.9/darwin-2level
/usr/local/lib/perl5/5.17.9
/usr/local/lib/perl5/site_perl
.

________________________________________
Craig A. Berry
mailto:craigberry@mac.com

"... getting out of a sonnet is much more
difficult than getting in."
Brad Leithauser

p5pRT · 2013-02-18T05:04:26Z

From @rocky

Although this might be a not be debugger bug, I wondered if
Devel::Trepan<https://metacpan.org/module/Devel::Trepan>exhibits the
same problem. Possibly not:

$ trepan.pl /tmp/bug3.pl
-- main::(/tmp/bug3.pl:3)
my $previous_dollar_underscore = '';
(trepanpl): list
/tmp/bug3.pl [1-10]
1 use File::DosGlob 'glob';
2
3 -> my $previous_dollar_underscore = '';
4 while (<m*.c>) {
5 if ($_ eq $previous_dollar_underscore) {
6 print STDERR "Bzzzt ... don't repeat yourself.\n";
7 exit 1;
8 }
9 $previous_dollar_underscore = $_;
10 print "# $_ <";
(trepanpl): c
Debugged program terminated. Use 'q' to quit or 'R' to restart.
(trepanpl): quit
trepan.pl: That's all, folks...
$ perl -v

This is perl 5, version 17, subversion 7 (v5.17.7) built for i686-linux ...

On Sun, Feb 17, 2013 at 10:27 PM, Craig A. Berry
<perlbug-followup@perl.org>wrote:

# New Ticket Created by Craig A. Berry
# Please include the string: [perl #116819]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=116819 >

Recently when running ext/File-DosGlob/t/DosGlob.t under the Perl
debugger, I noticed that it got stuck in an endless loop. I whittled it
down to:

$ cat debug_dosglob.t
use File::DosGlob 'glob';

my $previous_dollar_underscore = '';
while (<m*.c>) {
if ($_ eq $previous_dollar_underscore) {
print STDERR "Bzzzt ... don't repeat yourself.\n";
exit 1;
}
$previous_dollar_underscore = $_;
print "# $_ <";
while (<pp*.c>) {
print " $_";
}
print " >\n";
}

exit 0;
[end]

where we know that something went wrong if the outer loop gives us the
same result we got on the previous iteration. The script succeeds when run
without -d, but with -d it would loop endlessly without the code to detect
that we're repeating ourselves and bail out.

After putting the following debugger initialization in my home directory:

$ cat ~/.perldb
&parse_options("NonStop=0 ReadLine=0 TTY=db.out LineInfo=db.out");

sub afterinit {
push(@DB::typeahead,
'c',
'q',
);
}

I was able to bisect it with:

$ git clone perl perl2
$ cd perl2
$ ../perl/Porting/bisect.pl --start=v5.16.0 -- ./perl -Ilib -d
../debug_dosglob.t

...

HEAD is now at c619428 DosGlob: Don’t use the magic 2nd arg to glob
bad - non-zero exit from ./perl -Ilib -d ../debug_dosglob.t
c619428 is the first bad commit
commit c619428
Author: Father Chrysostomos <sprout@cpan.org>
Date: Fri Apr 27 16:48:36 2012 -0700
DosGlob&#8203;: Don’t use the magic 2nd arg to glob

Use the address of the glob op instead\.

This argument is going away\, because it is undocumented\, unused on
CPAN outside of the core\, and may get in the way of allowing glob to
be overridden properly\.

Another reason is that File&#8203;::DosGlob leaks memory\, because a glob op
freed before iteration has finished will leave File&#8203;::DosGlob still
holding on to the remainder of the list of files\.  The easiest way to
fix that will involve using an op address instead of a special index\,
so there will be no reason to keep it\.
:100644 100644 ac3aac4c4f1b2ee67991d925176b6268fdb5c842
a8ff0e670284a2449d7cb9598e804765c95b3175 M MANIFEST
:040000 040000 936fae86bb0174e55e2ecc48c58d001371cf04f0
ad05e50b1af9eaf181455d7775b3696c1f873642 M ext
bisect run success
That took 2527 seconds

The commit in question adds an XS routine _callsite that consists entirely
of:

SV *
_callsite(...)
CODE:
RETVAL = newSVpvn(
(char *)&cxstack[cxstack_ix].blk_sub.retop, sizeof(OP *)
);
OUTPUT:
RETVAL

I'm wondering why it's taking the address of retop since that is already
of type OP*, but I may be misparsing this and don't have the time at the
moment to dig deeper.

I think the only other necessary bit to log the ticket is:

$ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 17 subversion 9) configuration:
Local Commit: 480c672
Ancestor: 139a355
Platform:
osname=darwin, osvers=12.2.0, archname=darwin-2level
uname='darwin triamond.local 12.2.0 darwin kernel version 12.2.0: sat
aug 25 00:48:52 pdt 2012; root:xnu-2050.18.24~1release_x86_64 x86_64 '
config_args='-Dusedevel -Dcc=clang -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='clang', ccflags ='-fno-common -DPERL_DARWIN -no-cpp-precomp
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include',
optimize='-O3',
cppflags='-no-cpp-precomp -fno-common -DPERL_DARWIN -no-cpp-precomp
-fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include
-I/opt/local/include'
ccversion='', gccversion='4.2.1 Compatible Apple Clang 4.1
((tags/Apple/clang-421.11.66))', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='
-fstack-protector -L/usr/local/lib -L/opt/local/lib'
libpth=/usr/local/lib /opt/local/lib /usr/lib
libs=-ldbm -ldl -lm -lutil -lc
perllibs=-ldl -lm -lutil -lc
libc=, so=dylib, useshrplib=false, libperl=libperl.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup
-L/usr/local/lib -L/opt/local/lib -fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE
PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
Locally applied patches:
480c672
Built under darwin
Compiled at Feb 16 2013 17:46:51
@INC:
lib
/usr/local/lib/perl5/site_perl/5.17.9/darwin-2level
/usr/local/lib/perl5/site_perl/5.17.9
/usr/local/lib/perl5/5.17.9/darwin-2level
/usr/local/lib/perl5/5.17.9
/usr/local/lib/perl5/site_perl
.

________________________________________
Craig A. Berry
mailto:craigberry@mac.com

"... getting out of a sonnet is much more
difficult than getting in."
Brad Leithauser

p5pRT · 2013-02-18T05:04:27Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2013-02-19T01:01:09Z

From @craigberry

On Sun, Feb 17, 2013 at 11:03 PM, Rocky Bernstein <rocky@cpan.org> wrote:

Although this might be a not be debugger bug, I wondered if Devel::Trepan
exhibits the same problem. Possibly not:

$ trepan.pl /tmp/bug3.pl
-- main::(/tmp/bug3.pl:3)
my $previous_dollar_underscore = '';
(trepanpl): list
/tmp/bug3.pl [1-10]
1 use File::DosGlob 'glob';
2
3 -> my $previous_dollar_underscore = '';
4 while (<m*.c>) {
5 if ($_ eq $previous_dollar_underscore) {
6 print STDERR "Bzzzt ... don't repeat yourself.\n";
7 exit 1;
8 }
9 $previous_dollar_underscore = $_;
10 print "# $_ <";
(trepanpl): c
Debugged program terminated. Use 'q' to quit or 'R' to restart.
(trepanpl): quit
trepan.pl: That's all, folks...
$ perl -v

I forgot to mention that you need to have the current working
directory be the top-level of a Perl source tree, or at least
something that has multiple files matching the patterns m*.c and
pp*.c. If you don't see some output when running it without the
debugger, then you're not going to get a valid test because you will
never start the loop and thus never get stuck in it.

I suspect the debugger is an innocent victim in this case, but don't
know that for sure yet.

p5pRT · 2013-02-19T09:12:16Z

From @rocky

On Mon, Feb 18, 2013 at 8:00 PM, Craig A. Berry <craig.a.berry@gmail.com>wrote:

On Sun, Feb 17, 2013 at 11:03 PM, Rocky Bernstein <rocky@cpan.org> wrote:

Although this might be a not be debugger bug, I wondered if Devel::Trepan
exhibits the same problem. Possibly not:

$ trepan.pl /tmp/bug3.pl
-- main::(/tmp/bug3.pl:3)
my $previous_dollar_underscore = '';
(trepanpl): list
/tmp/bug3.pl [1-10]
1 use File::DosGlob 'glob';
2
3 -> my $previous_dollar_underscore = '';
4 while (<m*.c>) {
5 if ($_ eq $previous_dollar_underscore) {
6 print STDERR "Bzzzt ... don't repeat yourself.\n";
7 exit 1;
8 }
9 $previous_dollar_underscore = $_;
10 print "# $_ <";
(trepanpl): c
Debugged program terminated. Use 'q' to quit or 'R' to restart.
(trepanpl): quit
trepan.pl: That's all, folks...
$ perl -v

I forgot to mention that you need to have the current working
directory be the top-level of a Perl source tree, or at least
something that has multiple files matching the patterns m*.c and
pp*.c. If you don't see some output when running it without the
debugger, then you're not going to get a valid test because you will
never start the loop and thus never get stuck in it.

Still works for me:

$ trepan.pl /tmp/bug3.pl
trepan.pl /tmp/bug3.pl
-- main::(/tmp/bug3.pl:3)
my $previous_dollar_underscore = '';
set auto eval is on.
(trepanpl): c
c
# miniperlmain.c < pp_sort.c pp_hot.c pp_sys.c pp_ctl.c pp.c pp_pack.c >
# mathoms.c < pp_sort.c pp_hot.c pp_sys.c pp_ctl.c pp.c pp_pack.c >
# madly.c < pp_sort.c pp_hot.c pp_sys.c pp_ctl.c pp.c pp_pack.c >
# mg.c < pp_sort.c pp_hot.c pp_sys.c pp_ctl.c pp.c pp_pack.c >
# mro.c < pp_sort.c pp_hot.c pp_sys.c pp_ctl.c pp.c pp_pack.c >
# malloc.c < pp_sort.c pp_hot.c pp_sys.c pp_ctl.c pp.c pp_pack.c >
Debugged program terminated. Use 'q' to quit or 'R' to restart.
(trepanpl):

And for you?

I suspect the debugger is an innocent victim in this case, but don't
know that for sure yet.

Possibly. But if it is reproducible in Devel::Trepan:
1. I have a lot more interest in it
2. It would be easier for me to isolate what the bad interaction is

p5pRT · 2013-07-16T07:06:12Z

From @cpansprout

On Sun Feb 17 19:27:48 2013, craigberry wrote:

The commit in question adds an XS routine _callsite that consists
entirely of:

SV *
_callsite(...)
CODE:
RETVAL = newSVpvn(
(char *)&cxstack[cxstack_ix].blk_sub.retop,
sizeof(OP *)
);
OUTPUT:
RETVAL

I'm wondering why it's taking the address of retop since that is
already of type OP*, but I may be misparsing this and don't have
the time at the moment to dig deeper.

It’s reading the pointer as a string, to get the memory address of the op.

So presumably it’s getting confused because it is not stepping over the
debugger’s call frames. Or something like that.

I wonder whether Devel::Callsite has the same bug.

--

Father Chrysostomos

p5pRT · 2013-07-17T02:01:54Z

From @rocky

On Tue, Jul 16, 2013 at 3:06 AM, Father Chrysostomos via RT <
perlbug-followup@perl.org> wrote:

On Sun Feb 17 19:27:48 2013, craigberry wrote:

The commit in question adds an XS routine _callsite that consists
entirely of:

SV *
_callsite(...)
CODE:
RETVAL = newSVpvn(
(char *)&cxstack[cxstack_ix].blk_sub.retop,
sizeof(OP *)
);
OUTPUT:
RETVAL

I'm wondering why it's taking the address of retop since that is
already of type OP*, but I may be misparsing this and don't have
the time at the moment to dig deeper.

It’s reading the pointer as a string, to get the memory address of the op.

So presumably it’s getting confused because it is not stepping over the
debugger’s call frames. Or something like that.

I wonder whether Devel::Callsite has the same bug.

If it does and you want to suggest a fix, I'd be happy to apply it.

--

Father Chrysostomos

---
via perlbug: queue: perl5 status: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=116819

p5pRT · 2013-07-18T01:00:12Z

From @cpansprout

On Tue Jul 16 00:06:12 2013, sprout wrote:

On Sun Feb 17 19:27:48 2013, craigberry wrote:

The commit in question adds an XS routine _callsite that consists
entirely of:

SV *
_callsite(...)
CODE:
RETVAL = newSVpvn(
(char *)&cxstack[cxstack_ix].blk_sub.retop,
sizeof(OP *)
);
OUTPUT:
RETVAL

I'm wondering why it's taking the address of retop since that is
already of type OP*, but I may be misparsing this and don't have
the time at the moment to dig deeper.

It’s reading the pointer as a string, to get the memory address of the op.

So presumably it’s getting confused because it is not stepping over the
debugger’s call frames. Or something like that.

Without actually confirming (or even looking at the code), I think it is
getting the address of the line in DB::sub that calls glob.

I wonder whether Devel::Callsite has the same bug.

So yes, it probably does.

I don’t know offhand which is the best way to determine whether it
should be skipped. I think pp_caller has something we can copy.

--

Father Chrysostomos

p5pRT added Severity Low type-debugger labels Oct 19, 2019

xenu removed the Severity Low label Dec 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

endless loop debugging DosGlob #12785

endless loop debugging DosGlob #12785

p5pRT commented Feb 18, 2013

p5pRT commented Feb 18, 2013

p5pRT commented Feb 18, 2013

p5pRT commented Feb 18, 2013

p5pRT commented Feb 19, 2013

p5pRT commented Feb 19, 2013

p5pRT commented Jul 16, 2013

p5pRT commented Jul 17, 2013

p5pRT commented Jul 18, 2013

endless loop debugging DosGlob #12785

endless loop debugging DosGlob #12785

Comments

p5pRT commented Feb 18, 2013

p5pRT commented Feb 18, 2013

From @craigberry

p5pRT commented Feb 18, 2013

From @rocky

p5pRT commented Feb 18, 2013

p5pRT commented Feb 19, 2013

From @craigberry

p5pRT commented Feb 19, 2013

From @rocky

p5pRT commented Jul 16, 2013

From @cpansprout

p5pRT commented Jul 17, 2013

From @rocky

p5pRT commented Jul 18, 2013

From @cpansprout