Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PATCH: -DPERL_TRACE_OPS builds fail make test, for all other config options. #15270

Closed
p5pRT opened this issue Apr 11, 2016 · 8 comments
Closed

Comments

@p5pRT
Copy link

p5pRT commented Apr 11, 2016

Migrated from rt.perl.org#127880 (status was 'resolved')

Searchable as RT127880$

@p5pRT
Copy link
Author

p5pRT commented Apr 11, 2016

From @jimc

Created by @jimc

-DPERL_TRACE_OPS builds fail make test, for all other config options.

This is due to extra STDERR output from perl_destruct(),
which the test harnesses reject.

I have a patch, sending once I have the ticket#
Please disregard [Patch 1/7] numbering, others are not "bugs"

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.22.1:

Configured by Red Hat, Inc. at Wed Mar  2 13:26:59 UTC 2016.

Summary of my perl5 (revision 5 version 22 subversion 1) configuration:

  Platform:
    osname=linux, osvers=4.3.5-300.fc23.x86_64,
archname=i386-linux-thread-multi
    uname='linux buildvm-07.phx2.fedoraproject.org 4.3.5-300.fc23.x86_64 #1
smp mon feb 1 03:18:41 utc 2016 i686 i686 i386 gnulinux '
    config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstac
k-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m32
-march=i686 -mtune=atom -fasynchronous-unwind-tables -Dldflag
s=-Wl,-z,relro  -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro
 -Dlddlflags=-shared -Wl,-z,relro  -Dshrpdir=/usr/lib -DDEBUGGING=-g -
Dversion=5.22.1 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc
-Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Ds
iteprefix=/usr/local -Dsitelib=/usr/local/share/perl5
-Dsitearch=/usr/local/lib/perl5 -Dprivlib=/usr/share/perl5
-Dvendorlib=/usr/shar
e/perl5/vendor_perl -Darchlib=/usr/lib/perl5
-Dvendorarch=/usr/lib/perl5/vendor_perl -Darchname=i386-linux-thread-multi
-Duseshrplib -
Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles
-Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog
-Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005
-Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_en
dhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto
-Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto
-Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fs
tack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m32
-march=i686 -mtune=atom -fasynchronous-unwind-tables -fwrap
v -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
    optimize='  -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-prote
ctor-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m32
-march=i686 -mtune=atom -fasynchronous-unwind-tables -fwrapv -fno-str
ict-aliasing -I/usr/local/include'
    ccversion='', gccversion='5.3.1 20151207 (Red Hat 5.3.1-2)',
gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234,
doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12,
longdblkind=3
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-Wl,-z,relro  -fstack-protector-strong
-L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib /lib
    libs=-lpthread -lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
-lgdbm_compat
    perllibs=-lpthread -lresolv -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.22'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro '
    cccdlflags='-fPIC', lddlflags='-shared -Wl,-z,relro  -L/usr/local/lib
-fstack-protector-strong'

Locally applied patches:
    Fedora Patch1: Removes date check, Fedora/RHEL specific
    Fedora Patch4: use libresolv instead of libbind
    Fedora Patch5: USE_MM_LD_RUN_PATH
    Fedora Patch6: Skip hostname tests, due to builders not being network
capable
    Fedora Patch7: Dont run one io test due to random builder failures
    Fedora Patch15: Define SONAME for libperl.so
    Fedora Patch16: Install libperl.so to -Dshrpdir value
    Fedora Patch22: Document Math::BigInt::CalcEmu requires Math::BigInt
(CPAN RT#85015)
    Fedora Patch26: Make *DBM_File desctructors thread-safe (RT#61912)
    Fedora Patch27: Make PadlistNAMES() lvalue again (CPAN RT#101063)
    Fedora Patch28: Make magic vtable writable as a work-around for Coro
(CPAN RT#101063)
    Fedora Patch29: Fix CVE-2016-2381 (ambiguous environment variables
handling)
    Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on
Linux
    Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux


@INC for perl 5.22.1:
    /home/jimc/perl5/lib/perl5/5.22.1/i386-linux-thread-multi
    /home/jimc/perl5/lib/perl5/5.22.1
    /home/jimc/perl5/lib/perl5/i386-linux-thread-multi
    /home/jimc/perl5/lib/perl5
    /home/jimc/perl5/lib/perl5/5.22.1/i386-linux-thread-multi
    /home/jimc/perl5/lib/perl5/5.22.1
    /home/jimc/perl5/lib/perl5/i386-linux-thread-multi
    /home/jimc/perl5/lib/perl5
    /usr/local/lib/perl5
    /usr/local/share/perl5
    /usr/lib/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib/perl5
    /usr/share/perl5

@p5pRT
Copy link
Author

p5pRT commented Apr 11, 2016

From @jimc

0001-fix-make-test-failures-in-Accflags-DPERL_TRACE_OPS.patch
From dc3c76a43b488d1a3f673ccac505d74f115c9efb Mon Sep 17 00:00:00 2001
From: Jim Cromie <jim.cromie@gmail.com>
Date: Wed, 20 Jan 2016 16:11:34 -0700
Subject: [PATCH 1/7] fix make test failures in -Accflags=-DPERL_TRACE_OPS

builds with -DPERL_TRACE_OPS currently fail make test, on ~124 files,
for both normal & -DDEBUGGING, and plain & threaded configs.  These
tests fail because they check STDERR content, and reject the extra
output issued by perl_destruct() in these builds.  It turns out that
"Trace of all OPs executed:" by itself is enough to cause the failures.

The easiest fix is to silence -DPERL_TRACE_OPS printing by default,
and enable it if $ENV{PERL_TRACE_OPS} > 0.  Due to various aspects of
the design, this is also the best fix.

 - OP-counting is only done in runops_debug(). runops_standard()
   avoids all counting overhead, so the only 'cost' then is the extra
   space in the interpreter structure.

 - use of runops_debug() vs runops_standard() varies at build, with -DDEBUGGING,
   and can be changed at runtime, with -MDevel::Peek -e 'BEGIN{runops_debug(0 or 1)}'

 - when runops_standard is used, perl_destruct() reports ALL-0s
   OP-exec-counts.  This is wrong, silence is better.

 - when runops_* is toggled, OP-counts will only reflect part of the work done.

Given the flexibilty of use, perl_destruct() cannot really know the
counts are good, and shouldnt just spew them to STDERR.  With this
patch, the user asks for the output at runtime.

 - with patch, the build is not noisy, more usable, no stderr to deal with.
   miniperl is also silent, reducing `make` output too.

 - via simple XS, data can be read, cleared, at any compilation phase.
   with this, data to stderr is just a convienience, not "the way" to get data.
   Ive got Devel::TraceOps on-deck.

I also took the liberty of suppressing "<optype>: 0" prints entirely.

 - no real info provided (0 count is inferrable by silence)
 - 0 is usually wrong, unless other OP-counts (from same interp) are non-0
 - `make test` on a non-debug threaded build emits 5 million of them

And now './perl -e 'print "foo\n"' becomes readably small:
foo
Trace of all OPs executed:
  pushmark: 1
  gvsv: 1
  nextstate: 1
  enter: 1
  leave: 1
  print: 1

NOTES:

Since the PERL_TRACE_OPs counting overhead is runtime selectable, it
seemed useful to measure and report it:

 function run_trace_overhead() {
    for i in seq 1 2; do
	perf stat -r5 -- ./perl -Ilib -MDevel::Peek -e 'BEGIN{runops_debug(0)}' -e 'for (1..9_999_999){$i++}'
	perf stat -r5 -- ./perl -Ilib -MDevel::Peek -e 'BEGIN{runops_debug(1)}' -e 'for (1..9_999_999){$i++}'
    done 2>&1 | grep -E 'Performance|instructions'
 }

bench.pl couldnt do this, not without different --perlargs per contestant.

[jimc@popeye perl]$ run_trace_overhead
 Performance counter stats for './perl -MDevel::Peek -e BEGIN{runops_debug(0)} -e for (1..9_999_999){$i++}' (5 runs):
     3,460,280,502      instructions              #    1.57  insns per cycle          (25.01%)
 Performance counter stats for './perl -MDevel::Peek -e BEGIN{runops_debug(1)} -e for (1..9_999_999){$i++}' (5 runs):
     3,857,253,945      instructions              #    2.00  insns per cycle          (25.07%)
 Performance counter stats for './perl -MDevel::Peek -e BEGIN{runops_debug(0)} -e for (1..9_999_999){$i++}' (5 runs):
     3,437,504,999      instructions              #    1.57  insns per cycle          (25.07%)
 Performance counter stats for './perl -MDevel::Peek -e BEGIN{runops_debug(1)} -e for (1..9_999_999){$i++}' (5 runs):
     3,859,640,318      instructions              #    2.00  insns per cycle          (25.09%)

IE: when swapping runops_debug for runops_standard, the instruction
count predictably goes up: by 3.8/3.4 = 1.12 in these results.
Subject to many caveats, this is a reasonable estimate of the
worst-case cost overhead of OP-counting.

NB: the insns per cycle above are garbage, on this older 32bit box.
but the instruction counts look plausible (I didnt look at the
assembly code), and the per-cycle numbers are sane on a 64bit cpu
(with a newer PMU).
---
 perl.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/perl.c b/perl.c
index 228a0d8..1d8876b 100644
--- a/perl.c
+++ b/perl.c
@@ -602,16 +602,25 @@ perl_destruct(pTHXx)
     my_fflush_all();
 
 #ifdef PERL_TRACE_OPS
-    /* If we traced all Perl OP usage, report and clean up */
+    /* dump OP-counts if $ENV{PERL_TRACE_OPS} > 0 */
+    {
+        const char * const ptoenv = PerlEnv_getenv("PERL_TRACE_OPS");
+        UV uv;
+
+        if (!ptoenv || !Perl_grok_atoUV(ptoenv, &uv, NULL)
+            || !(uv > 0))
+        goto no_trace_out;
+    }
     PerlIO_printf(Perl_debug_log, "Trace of all OPs executed:\n");
     for (i = 0; i <= OP_max; ++i) {
-        PerlIO_printf(Perl_debug_log, "  %s: %"UVuf"\n", PL_op_name[i], PL_op_exec_cnt[i]);
-        PL_op_exec_cnt[i] = 0;
+        if (PL_op_exec_cnt[i])
+            PerlIO_printf(Perl_debug_log, "  %s: %"UVuf"\n", PL_op_name[i], PL_op_exec_cnt[i]);
     }
     /* Utility slot for easily doing little tracing experiments in the runloop: */
     if (PL_op_exec_cnt[OP_max+1] != 0)
         PerlIO_printf(Perl_debug_log, "  SPECIAL: %"UVuf"\n", PL_op_exec_cnt[OP_max+1]);
     PerlIO_printf(Perl_debug_log, "\n");
+ no_trace_out:
 #endif
 
 
-- 
2.5.5

@p5pRT
Copy link
Author

p5pRT commented Apr 12, 2016

From @tonycoz

On Mon Apr 11 10​:34​:49 2016, yoduh wrote​:

I'll apply this once 5.25 starts.

Tony

@p5pRT
Copy link
Author

p5pRT commented Apr 12, 2016

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented May 10, 2016

From @tonycoz

On Mon Apr 11 10​:28​:23 2016, yoduh wrote​:

-DPERL_TRACE_OPS builds fail make test, for all other config options.

This is due to extra STDERR output from perl_destruct(),
which the test harnesses reject.

I have a patch, sending once I have the ticket#
Please disregard [Patch 1/7] numbering, others are not "bugs"

Thanks, applied as e71f25b.

Tony

@p5pRT
Copy link
Author

p5pRT commented May 10, 2016

@tonycoz - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

p5pRT commented May 30, 2017

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Author

p5pRT commented May 30, 2017

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant