Skip Menu |
Report information
Id: 131811
Status: resolved
Priority: 0/
Queue: perl5

Owner: Nobody
Requestors: craigberry <craigberry [at] mac.com>
Cc:
AdminCc:

Operating System: (no value)
PatchStatus: (no value)
Severity: low
Type: unknown
Perl Version: (no value)
Fixed In: 5.27.3



Date: Fri, 28 Jul 2017 08:13:16 -0500
To: perlbug [...] perl.org
From: "Craig A. Berry" <craigberry [...] mac.com>
Subject: panic: previous op failed to extend arg stack
Download (untitled) / with headers
text/plain 4.5k
On VMS, blead as of: $ type .patch blead 2017-07-27.19:28:14 a8cb1947aad7d9fcd7bb6fdc7fe8bf92f699d59d v5.27.2-77-ga8cb1947aa has several new test failures that have appeared in the last few days, all with the panic in my subject line. Here's the shortest, simplest one run under the VMS debugger: $ dbgperl -T dist/Locale-Maketext/t/09_compile.t %DEBUG-W-DWNOT1PROC, the 1 process debugger cannot be run in DECwindows mode OpenVMS I64 Debug64 Version V8.4-001 %DEBUG-I-INITIAL, Language: C, Module: PERLMAIN %DEBUG-I-NOTATMAIN, Type GO to reach MAIN program Show quoted text
DBG> go
break at routine PERLMAIN\main 182985: PL_use_safe_putenv = FALSE; Show quoted text
DBG> set image dbgperlshr DBG> set module/all DBG> go
1..2 panic: previous op failed to extend arg stack: base=1a6f670, sp=1a6fe88, hwm=1a6f678 # Looks like your test exited with 1 before it could output anything. %SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=0000000000000001, PC=0000000000370820, PS=0000001B break on unhandled exception at HV\Perl_hv_common\%LINE 182678+32 182678: if (HeHASH(entry) != hash) /* strings can't be equal */ Show quoted text
DBG> call Perl_sv_dump(entry)
SV = NULL(0x0) at 0x463c9e8 REFCNT = 1 FLAGS = (PADTMP) value returned is 70647408 Show quoted text
DBG> call Perl_sv_dump(hash)
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=00000000ED060719, PC=00000000004D0E81, PS=0000001B break on unhandled exception at DUMP\Perl_sv_dump\%LINE 186350+31 186350: if (sv && SvROK(sv)) Show quoted text
DBG>
The simplest reproducer I could come up with is: $ perl -MTest::More -e "my @ENV_values = sort keys %ENV;" panic: previous op failed to extend arg stack: base=834790, sp=834fac, hwm=834798 assert error: expression = (CxTYPE(cx) == CXt_SUB && CxMULTICALL(cx)) || PL_savestack_ix == cx->blk_oldsaveix, in file D0:[craig.blead]inline.h;1 at line 1379 Removing Test::More makes the panic go away. I'm not really sure where to begin debugging this. There is a fair amount of VMS-only code in hv.c (delineated by DYNAMIC_ENV_FETCH), so my first hunch would be that the magic there disagrees with something new. Here's my configuration: $ perl -V Summary of my perl5 (revision 5 version 27 subversion 3) configuration: Snapshot of: a8cb1947aad7d9fcd7bb6fdc7fe8bf92f699d59d Platform: osname=VMS osvers=V8.4 archname=VMS_IA64 uname='VMS alma V8.4 HP rx2660 (1.67GHz/9.0MB)' config_args='-"Dusedevel" -"DDEBUGGING" -"Duse64bitint" -"Dusevmsdebug" -"des"' hint=none useposix=false d_sigaction=define useithreads=undef usemultiplicity=undef use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=undef default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='CC/DECC' ccflags ='/Include=[]/Standard=Relaxed_ANSI/Prefix=All/Obj=.obj /NOANSI_ALIAS/float=ieee/ieee=denorm/NAMES=(SHORTENED)/Define=_USE_STD_STAT=1' optimize='/List/Debug/NoOpt' cppflags='undef' ccversion='70390020' gccversion='' gccosandvers='undef' intsize=4 longsize=4 ptrsize=4 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=1 ivtype='long long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='Link/nodebug' ldflags ='/Debug/Trace/Map' libpth=/sys$share /sys$library libs= perllibs= libc=(DECCRTL) so=exe useshrplib=true libperl=undef gnulibc_version='undef' Dynamic Linking: dlsrc=dl_vms.xs dlext=exe d_dlsymun=undef ccdlflags='' cccdlflags='' lddlflags='/Share' Characteristics of this PERLSHR image: Compile-time options: DEBUGGING HAS_TIMES HAVE_INTERP_INTERN PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_EXTERNAL_GLOB PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_INT USE_IEEE USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF VMS_DO_SOCKETS VMS_SHORTEN_LONG_SYMBOLS Built under VMS Compiled at Jul 27 2017 22:02:42 @INC: /perl_root/lib/site_perl/VMS_IA64 /perl_root/lib/site_perl /perl_root/lib/VMS_IA64/5_27_3 /perl_root/lib Show quoted text
________________________________________ Craig A. Berry "... getting out of a sonnet is much more difficult than getting in." Brad Leithauser
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #131811] panic: previous op failed to extend arg stack
Date: Fri, 28 Jul 2017 20:28:19 +0100
To: perl5-porters [...] perl.org
Download (untitled) / with headers
text/plain 1.7k
On Fri, Jul 28, 2017 at 07:13:35AM -0700, Craig A.Berry wrote: Show quoted text
> panic: previous op failed to extend arg stack: base=1a6f670, sp=1a6fe88, hwm=1a6f678
This panic was added by me a month ago with 87058c3, and has had a couple of tweaks since, most recently with 978b185 2017-07-16 PL_curstackinfo->si_stack_hwm: gently restore Its quite possible that its a false positive, given that its a new mechanism, or it may be detecting a real issue. Its mechanism is as follows: In DEBUGGING builds, in Perl_runops_debug, just before calling the pp function, it sets PL_curstackinfo->si_stack_hwm to the current stack position. During execution of the op, macros like EXTEND() update si_stack_hwm. On return from the pp function, Perl_runops_debug checks that the new stack position isn't greater than si_stack_hwm, i.e. that the op didn't return more things on the stack than it requested space for. The complicated bit comes when the pp function triggers runops loop entries and exits (e.g. tie, die): here si_stack_hwm needs to be correctly saved and restored to avoid false positives, and that's where I've had to tweak things. To debug this, I would set a breakpoint on the croak("panic...") call, than work backwards and find out what op had just been called in what runoops loop, then set a break on that op with a suitable ignore count, then see what the op does and how it messes with the stack and si_stack_hwm (and if it causes inner runops loops to be started, how si_stack_hwm is saved and restored). Eventually you'll either find some code which is pushing things on the stack without extending it first, or si_stack_hwm is in some way getting messed up. If the latter, I may need to fix it. -- It's not that I'm afraid to die, I just don't want to be there when it happens. -- Woody Allen
To: perlbug-followup [...] perl.org
Subject: Re: [perl #131811] panic: previous op failed to extend arg stack
From: "Craig A. Berry" <craigberry [...] mac.com>
Date: Sun, 30 Jul 2017 17:44:18 -0500
Download (untitled) / with headers
text/plain 4.9k
Show quoted text
> On Jul 28, 2017, at 2:28 PM, Dave Mitchell via RT <perlbug-followup@perl.org> wrote: > > On Fri, Jul 28, 2017 at 07:13:35AM -0700, Craig A.Berry wrote:
>> panic: previous op failed to extend arg stack: base=1a6f670, sp=1a6fe88, hwm=1a6f678
> > This panic was added by me a month ago with 87058c3, and has had a couple > of tweaks since, most recently with > > 978b185 2017-07-16 PL_curstackinfo->si_stack_hwm: gently restore > > Its quite possible that its a false positive, given that its a new > mechanism, or it may be detecting a real issue. > > Its mechanism is as follows: > > In DEBUGGING builds, in Perl_runops_debug, just before calling the pp > function, it sets PL_curstackinfo->si_stack_hwm to the current stack > position. During execution of the op, macros like EXTEND() update > si_stack_hwm. On return from the pp function, Perl_runops_debug checks > that the new stack position isn't greater than si_stack_hwm, i.e. that the > op didn't return more things on the stack than it requested space for. > > The complicated bit comes when the pp function triggers runops loop > entries and exits (e.g. tie, die): here si_stack_hwm needs to be correctly > saved and restored to avoid false positives, and that's where I've had to > tweak things. > > To debug this, I would set a breakpoint on the croak("panic...") call, > than work backwards and find out what op had just been called in what > runoops loop, then set a break on that op with a suitable ignore count, > then see what the op does and how it messes with the stack and si_stack_hwm > (and if it causes inner runops loops to be started, how si_stack_hwm is > saved and restored). > > Eventually you'll either find some code which is pushing things on the stack > without extending it first, or si_stack_hwm is in some way getting messed > up. If the latter, I may need to fix it.
Thanks. I tried to follow this advice and found it difficult to identify which instance of S_cx_popblock (where the panic check lives) was firing. Since it's a static function from inline.h, each module has its own copy. By the time the panic hits, the C stack is hosed and there are no active call frames. It appears to be the one from pp_ctl.c, but when I set a breakpoint on that I get stuck in a loop of endless "Out of memory!" messages until I kill it. I also noticed that there are often (but not always) access violations in non-debugging builds, so I don't think this panic is a false positive. I eventually bisected the problem to: commit 8dc9003ff3914e78971f561eaece965e9ceeb49e Author: David Mitchell <davem@iabyn.com> Date: Fri Jul 21 14:32:57 2017 +0100 hv_pushkv(): handle keys() and values() too The newish function hv_pushkv() currently just pushes all key/value pairs on the stack. i.e. it does the equivalent of the perl code '() = %h'. Extend it so that it can handle 'keys %h' and values %h' too. This is basically moving the remaining list-context functionality out of do_kv() and into hv_pushkv(). The rationale for this is that hv_pushkv() is a pure HV-related function, while do_kv() is a pp function for several ops including OP_KEYS/VALUES, and expects PL_op->op_flags/op_private to be valid. The main difference I can see is that before the commit above, we always extended the stack by 2 regardless of whether we were iterating over keys, values, or both. Now we only extend by 2 when iterating both keys and values, and otherwise only extend by 1. I've further simplified the reproducer to: $ perl -"Dst" -e "$ENV{foo}='bar'; my @E = keys %ENV;" EXECUTING... => (-e:0) enter => (-e:0) nextstate => (-e:1) const(PV("bar"\0)) => PV("bar"\0) (-e:1) multideref($ENV{"foo"}) => PV("bar"\0) PVMG("bar"\0) (-e:1) sassign => PVMG("bar"\0) (-e:1) nextstate => (-e:1) pushmark => * (-e:1) gv(main::ENV) => * GV() (-e:1) rv2hv => * HV() (-e:1) keys panic: previous op failed to extend arg stack: base=834790, sp=834f9c, hwm=834794 assert error: expression = (CxTYPE(cx) == CXt_SUB && CxMULTICALL(cx)) || PL_savestack_ix == cx->blk_oldsaveix, in file D0:[craig.blead]inline.h;1 at line 1379 When I run the same thing on macOS, the line => PV("bar"\0) PVMG("bar"\0) looks like: => PV("bar"\0) PVMG() but otherwise (except for the panic) looks the same. My debugging time for the weekend is just about up, but it strikes me as plausible that the panic is telling the truth and the multideref($ENV{"foo"}) only extended the stack by one while placing two items on it and the "keys" op is an innocent victim that used to paper things over by reserving two slots on the stack where it really only needs one for its own purposes. But I'm not even sure I have read the debugging output correctly or know what I'm talking about and will have to revisit this later. Show quoted text
________________________________________ Craig A. Berry "... getting out of a sonnet is much more difficult than getting in." Brad Leithauser
CC: perlbug-followup [...] perl.org
Date: Mon, 31 Jul 2017 16:04:51 +0100
Subject: Re: [perl #131811] panic: previous op failed to extend arg stack
From: Dave Mitchell <davem [...] iabyn.com>
To: "Craig A. Berry" <craigberry [...] mac.com>
Download (untitled) / with headers
text/plain 2.4k
On Sun, Jul 30, 2017 at 05:44:18PM -0500, Craig A. Berry wrote: Show quoted text
> Thanks. I tried to follow this advice and found it difficult to > identify which instance of S_cx_popblock (where the panic check lives) > was firing.
I think you're getting confused. The panic lives in Perl_runops_debug(). Show quoted text
> I've further simplified the reproducer to: > > $ perl -"Dst" -e "$ENV{foo}='bar'; my @E = keys %ENV;" > > EXECUTING... > > => > (-e:0) enter > => > (-e:0) nextstate > => > (-e:1) const(PV("bar"\0)) > => PV("bar"\0) > (-e:1) multideref($ENV{"foo"}) > => PV("bar"\0) PVMG("bar"\0) > (-e:1) sassign > => PVMG("bar"\0) > (-e:1) nextstate > => > (-e:1) pushmark > => * > (-e:1) gv(main::ENV) > => * GV() > (-e:1) rv2hv > => * HV() > (-e:1) keys > panic: previous op failed to extend arg stack: base=834790, sp=834f9c, hwm=834794 > assert error: expression = (CxTYPE(cx) == CXt_SUB && CxMULTICALL(cx)) || PL_savestack_ix == cx->blk_oldsaveix, in file D0:[craig.blead]inline.h;1 at > line 1379 > > When I run the same thing on macOS, the line > > => PV("bar"\0) PVMG("bar"\0) > > looks like: > > => PV("bar"\0) PVMG() > > but otherwise (except for the panic) looks the same.
That's distinctly odd. Since the string 'bar' at that point is on the stack, but has not yet been used to assign a value to $ENV{foo}, $ENV{foo} can't possibly have the value 'bar'. So the magic SV (whose set magic causes the process's env to be updated) shouldn't be PVMG("bar"\0). Show quoted text
> My debugging time for the weekend is just about up, but it strikes me as > plausible that the panic is telling the truth and the > multideref($ENV{"foo"}) only extended the stack by one while placing two > items on it and the "keys" op is an innocent victim that used to paper > things over by reserving two slots on the stack where it really only > needs one for its own purposes. But I'm not even sure I have read the > debugging output correctly or know what I'm talking about and will have > to revisit this later.
The panic means exactly that in Perl_runops_debug(), on return from pp_keys, PL_stack_sp was at 0x834f9c, while EXTEND() called during the run of pp_keys only extended the stack to 0x834794. I would suggest: set a breakpoint at the start of Perl_pp_keys, then set watchpoints on PL_curstackinfo->si_stack_hwm and PL_stack_sp and see how they change during the execution of pp_keys(). -- "You're so sadly neglected, and often ignored. A poor second to Belgium, When going abroad." -- Monty Python, "Finland"
Date: Sat, 05 Aug 2017 09:56:56 -0500
CC: perlbug-followup [...] perl.org
From: "Craig A. Berry" <craigberry [...] mac.com>
Subject: Re: [perl #131811] panic: previous op failed to extend arg stack
To: Dave Mitchell <davem [...] iabyn.com>
Download (untitled) / with headers
text/plain 5.8k
[at long last getting back to this] Show quoted text
> On Jul 31, 2017, at 10:04 AM, Dave Mitchell <davem@iabyn.com> wrote: > > On Sun, Jul 30, 2017 at 05:44:18PM -0500, Craig A. Berry wrote:
>> Thanks. I tried to follow this advice and found it difficult to >> identify which instance of S_cx_popblock (where the panic check lives) >> was firing.
> > I think you're getting confused. The panic lives in Perl_runops_debug().
Yup. For some reason I was thinking the assert and the panic happened in the same place. Show quoted text
>
>> I've further simplified the reproducer to: >> >> $ perl -"Dst" -e "$ENV{foo}='bar'; my @E = keys %ENV;" >> >> EXECUTING... >> >> => >> (-e:0) enter >> => >> (-e:0) nextstate >> => >> (-e:1) const(PV("bar"\0)) >> => PV("bar"\0) >> (-e:1) multideref($ENV{"foo"}) >> => PV("bar"\0) PVMG("bar"\0) >> (-e:1) sassign >> => PVMG("bar"\0) >> (-e:1) nextstate >> => >> (-e:1) pushmark >> => * >> (-e:1) gv(main::ENV) >> => * GV() >> (-e:1) rv2hv >> => * HV() >> (-e:1) keys >> panic: previous op failed to extend arg stack: base=834790, sp=834f9c, hwm=834794 >> assert error: expression = (CxTYPE(cx) == CXt_SUB && CxMULTICALL(cx)) || PL_savestack_ix == cx->blk_oldsaveix, in file D0:[craig.blead]inline.h;1 at >> line 1379 >> >> When I run the same thing on macOS, the line >> >> => PV("bar"\0) PVMG("bar"\0) >> >> looks like: >> >> => PV("bar"\0) PVMG() >> >> but otherwise (except for the panic) looks the same.
> > That's distinctly odd. Since the string 'bar' at that point is on the > stack, but has not yet been used to assign a value to $ENV{foo}, $ENV{foo} > can't possibly have the value 'bar'. So the magic SV (whose set magic > causes the process's env to be updated) shouldn't be PVMG("bar"\0). >
>> My debugging time for the weekend is just about up, but it strikes me as >> plausible that the panic is telling the truth and the >> multideref($ENV{"foo"}) only extended the stack by one while placing two >> items on it and the "keys" op is an innocent victim that used to paper >> things over by reserving two slots on the stack where it really only >> needs one for its own purposes. But I'm not even sure I have read the >> debugging output correctly or know what I'm talking about and will have >> to revisit this later.
> > The panic means exactly that in Perl_runops_debug(), on return from > pp_keys, PL_stack_sp was at 0x834f9c, while EXTEND() called during the > run of pp_keys only extended the stack to 0x834794. > > I would suggest: set a breakpoint at the start of Perl_pp_keys, then set > watchpoints on PL_curstackinfo->si_stack_hwm and PL_stack_sp and see how > they change during the execution of pp_keys(). >
We use a bunch more stack than we have allocated in Perl_hv_pushkv. The basic problem is that Perl_hv_pushkv extends stack space once based on how many keys are in the HV. But on VMS, the keys don't exist yet until the first call to Perl_hv_iternext_flags triggers a call to Perl_prime_env_iter in vms/vms.c. The following patch (not meant to be applied) shows how far off we are: --- hv.c;-0 2017-07-27 05:49:23 -0500 +++ hv.c 2017-08-05 07:04:59 -0500 @@ -1008,6 +1008,7 @@ Perl_hv_pushkv(pTHX_ HV *hv, U32 flags) } else { Size_t nkeys = HvUSEDKEYS(hv); + Size_t actual_nkeys = 0; SSize_t ext; if (!nkeys) @@ -1021,6 +1022,7 @@ Perl_hv_pushkv(pTHX_ HV *hv, U32 flags) EXTEND(SP, ext); while ((entry = hv_iternext(hv))) { + actual_nkeys++; if (flags & 1) { SV *keysv = newSVhek(HeKEY_hek(entry)); SvTEMP_on(keysv); @@ -1030,6 +1032,9 @@ Perl_hv_pushkv(pTHX_ HV *hv, U32 flags) if (flags & 2) PUSHs(HeVAL(entry)); } + if (actual_nkeys != nkeys) + Perl_croak(aTHX_ "panic: extended stack for %" UVuf " keys but iterated over %" UVuf, + (UV)nkeys, (UV)actual_nkeys); } PUTBACK; [end] which dies like so: $ perl -e "$ENV{'foo'}='bar'; my @e = keys %ENV;" panic: extended stack for 1 keys but iterated over 514 at -e line 1. assert error: expression = (CxTYPE(cx) == CXt_SUB && CxMULTICALL(cx)) || PL_savestack_ix == cx->blk_oldsaveix, in file D0:[craig.blead]inline.h;1 at line 1379 The safest thing to do would be to extend the stack inside the loop in Perl_hv_pushkv that iterates over keys, but we already have a code path that does that when the HV is a tied hash. So it seems to me the best solution is to piggyback PERL_MAGIC_env on the PERL_MAGIC_tied logic and consider both a form of being tied where we don't know the number of keys we'll get. The following solves the problem and I will push soonish unless someone tells me there is a better way to do it: commit 61084eefd823e097255934e8ab599c0e418e5253 Author: Craig A. Berry <craigberry@mac.com> Date: Sat Aug 5 08:00:32 2017 -0500 Consider magic %ENV as tied in hv_pushkv. For the DYNAMIC_ENV_FETCH case, we don't know the number of keys until the first iteration triggers a call to prime_env_iter(), so piggyback on the tied magic case, which already handles extending the stack for each iteration rather than all at once beforehand. diff --git a/hv.c b/hv.c index 20b4eceb98..7029e28387 100644 --- a/hv.c +++ b/hv.c @@ -988,7 +988,11 @@ void Perl_hv_pushkv(pTHX_ HV *hv, U32 flags) { HE *entry; - bool tied = SvRMAGICAL(hv) && mg_find(MUTABLE_SV(hv), PERL_MAGIC_tied); + bool tied = SvRMAGICAL(hv) && (mg_find(MUTABLE_SV(hv), PERL_MAGIC_tied) +#ifdef DYNAMIC_ENV_FETCH /* might not know number of keys yet */ + || mg_find(MUTABLE_SV(hv), PERL_MAGIC_env) +#endif + ); dSP; PERL_ARGS_ASSERT_HV_PUSHKV; [end] Show quoted text
________________________________________ Craig A. Berry "... getting out of a sonnet is much more difficult than getting in." Brad Leithauser
Date: Sat, 5 Aug 2017 16:13:11 +0100
CC: perlbug-followup [...] perl.org
From: Dave Mitchell <davem [...] iabyn.com>
Subject: Re: [perl #131811] panic: previous op failed to extend arg stack
To: "Craig A. Berry" <craigberry [...] mac.com>
Download (untitled) / with headers
text/plain 1.1k
On Sat, Aug 05, 2017 at 09:56:56AM -0500, Craig A. Berry wrote: Show quoted text
> Author: Craig A. Berry <craigberry@mac.com> > Date: Sat Aug 5 08:00:32 2017 -0500 > > Consider magic %ENV as tied in hv_pushkv. > > For the DYNAMIC_ENV_FETCH case, we don't know the number of keys > until the first iteration triggers a call to prime_env_iter(), so > piggyback on the tied magic case, which already handles extending > the stack for each iteration rather than all at once beforehand. > > diff --git a/hv.c b/hv.c > index 20b4eceb98..7029e28387 100644 > --- a/hv.c > +++ b/hv.c > @@ -988,7 +988,11 @@ void > Perl_hv_pushkv(pTHX_ HV *hv, U32 flags) > { > HE *entry; > - bool tied = SvRMAGICAL(hv) && mg_find(MUTABLE_SV(hv), PERL_MAGIC_tied); > + bool tied = SvRMAGICAL(hv) && (mg_find(MUTABLE_SV(hv), PERL_MAGIC_tied) > +#ifdef DYNAMIC_ENV_FETCH /* might not know number of keys yet */ > + || mg_find(MUTABLE_SV(hv), PERL_MAGIC_env) > +#endif > + ); > dSP;
Looks good to me. -- But Pity stayed his hand. "It's a pity I've run out of bullets", he thought. -- "Bored of the Rings"
Subject: Re: [perl #131811] panic: previous op failed to extend arg stack
From: "Craig A. Berry" <craigberry [...] mac.com>
Date: Sat, 05 Aug 2017 10:24:31 -0500
To: perlbug-followup [...] perl.org
Download (untitled) / with headers
text/plain 325b
Show quoted text
> On Aug 5, 2017, at 10:13 AM, Dave Mitchell via RT <perlbug-followup@perl.org> wrote: > > > Looks good to me.
Thanks, now pushed. This ticket can be closed. Show quoted text
________________________________________ Craig A. Berry "... getting out of a sonnet is much more difficult than getting in." Brad Leithauser


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

For issues related to this RT instance (aka "perlbug"), please contact perlbug-admin at perl.org