From rt.perl.org@mavit.org.uk

This is a bug report for perl from rt.perl.org@​mavit.org.uk,
generated with the help of perlbug 1.40 running under perl 5.26.2.

The perlop man page says that the qx/STRING/ operator returns "undef if the command failed". It seems to me that this is misleading, because it does not go on to explain when the command is considered to have failed. The reader might reasonably expect that qx/STRING/ will return undef if the command exits non-zero, but this is not the case.

I suspect that qx/STRING/ only returns undef if forking and execing /bin/sh fails, which isn't what I'd think of as "the command" in this context.

Flags​:
  category=docs
  severity=low

Site configuration information for perl 5.26.2​:

Configured by Red Hat, Inc. at Mon Apr 16 12​:45​:12 UTC 2018.

Summary of my perl5 (revision 5 version 26 subversion 2) configuration​:
 
  Platform​:
  osname=linux
  osvers=4.15.14-300.fc27.x86_64
  archname=x86_64-linux-thread-multi
  uname='linux buildhw-03.phx2.fedoraproject.org 4.15.14-300.fc27.x86_64 #1 smp thu mar 29 16​:13​:44 utc 2018 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -Dldflags=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dlddlflags=-shared -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.26.2 -Dmyhostname=localhost -Dperladmin=root@​localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5
-Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=define
  usemultiplicity=define
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='gcc'
  ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
  optimize=' -g'
  cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fwrapv -fno-strict-aliasing -I/usr/local/include'
  ccversion=''
  gccversion='7.3.1 20180303 (Red Hat 7.3.1-5)'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='gcc'
  ldflags ='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib
  libs=-lpthread -lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
  perllibs=-lpthread -lresolv -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.26.so
  so=so
  useshrplib=true
  libperl=libperl.so
  gnulibc_version='2.26'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
  cccdlflags='-fPIC'
  lddlflags='-lpthread -shared -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -fstack-protector-strong'

Locally applied patches​:
  Fedora Patch1​: Removes date check, Fedora/RHEL specific
  Fedora Patch3​: support for libdir64
  Fedora Patch4​: use libresolv instead of libbind
  Fedora Patch5​: USE_MM_LD_RUN_PATH
  Fedora Patch6​: Provide MM​::maybe_command independently (bug #1129443)
  Fedora Patch7​: Dont run one io test due to random builder failures
  Fedora Patch15​: Define SONAME for libperl.so
  Fedora Patch16​: Install libperl.so to -Dshrpdir value
  Fedora Patch22​: Document Math​::BigInt​::CalcEmu requires Math​::BigInt (CPAN RT#85015)
  Fedora Patch26​: Make *DBM_File desctructors thread-safe (RT#61912)
  Fedora Patch27​: Make PadlistNAMES() lvalue again (CPAN RT#101063)
  Fedora Patch30​: Replace EU​::MakeMaker dependency with EU​::MM​::Utils in IPC​::Cmd (bug #1129443)
  Fedora Patch31​: Make File​::Glob more resistant against degenerative matching (RT#131211)
  Fedora Patch36​: Fix glob UTF-8 flag on a glob reassignment (RT#131263)
  Fedora Patch38​: Fix handling backslashes in PATH environment variable when executing "perl -S" (RT#129183)
  Fedora Patch45​: Fix File​::Glob rt131211.t test random failures
  Fedora Patch46​: Fix t/op/hash.t test random failures
  Fedora Patch47​: Parse caret variables with subscripts as normal variables inside ${...} escaping (RT#131664)
  Fedora Patch49​: Do not display too many bytes when reporting malformed UTF-8 character
  Fedora Patch51​: Fix error message for "our sub foo​::bar" (RT#131679)
  Fedora Patch52​: Fix executing arybase​::_tie_it() in Safe compartement (RT#131588)
  Fedora Patch54​: Fix splitting non-ASCII strings if unicode_strings feature is enabled (RT#130907)
  Fedora Patch55​: Fix compiler warnings in code generated by ExtUtils​::Constant (CPAN RT#63832)
  Fedora Patch56​: Fix compiler warnings in code generated by ExtUtils​::Constant (CPAN RT#101487)
  Fedora Patch58​: Fix unreliable Time-HiRes tests (CPAN RT#122819)
  Fedora Patch61​: Fix Term​::ReadLine not to create spurious &STDERR files (RT#132008)
  Fedora Patch64​: Fix an overflow when parsing a character range with no preceding character (RT#132245)
  Fedora Patch65​: Fix walking symbol table for ISA in Carp
  Fedora Patch66​: Fix handling file names with null bytes in stat and lstat functions (RT#131895)
  Fedora Patch67​: Fix a crash when untying an object witout a stash
  Fedora Patch68​: Fix deparsing of transliterations with unprintable characters (RT#132405)
  Fedora Patch69​: Fix error reporting on do() on a directory (RT#125774)
  Fedora Patch70​: Fix stack manipulation when a lexical subroutine is defined in a do block in a member of an iteration list (RT#132442)
  Fedora Patch71​: Fix setting $! when statting a closed filehandle (RT#108288)
  Fedora Patch72​: Fix tainting of s/// with overloaded replacement (RT#115266)
  Fedora Patch73​: Expand system() arguments before a fork (RT#121105)
  Fedora Patch76​: Avoid undefined behavior when copying memory in Glob and pp_caller (RT#131746)
  Fedora Patch78​: Fix compatibility with libxcrypt (bug #1536752)
  Fedora Patch79​: Link XS modules to pthread library to fix linking with -z defs
  Fedora Patch80​: Fix parsing braced subscript after parentheses (RT#8045)
  Fedora Patch200​: Link XS modules to libperl.so with EU​::CBuilder on Linux
  Fedora Patch201​: Link XS modules to libperl.so with EU​::MM on Linux

@​INC for perl 5.26.2​:
  /usr/local/lib64/perl5
  /usr/local/share/perl5
  /usr/lib64/perl5/vendor_perl
  /usr/share/perl5/vendor_perl
  /usr/lib64/perl5
  /usr/share/perl5

Environment for perl 5.26.2​:
  HOME=/home/mavit
  LANG=en_GB.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/usr/local/bin​:/usr/local/sbin​:/usr/bin​:/usr/sbin​:/sbin
  PERL_BADLANG (unset)
  PERL_LOCAL_LIB_ROOT=
  SHELL=/bin/tcsh

From @jkeenan

On Tue, 29 May 2018 12​:08​:55 GMT, cpan@​mavit.org.uk wrote​:

This is a bug report for perl from rt.perl.org@​mavit.org.uk,
generated with the help of perlbug 1.40 running under perl 5.26.2.

-----------------------------------------------------------------
The perlop man page says that the qx/STRING/ operator returns "undef
if the command failed". It seems to me that this is misleading,
because it does not go on to explain when the command is considered to
have failed. The reader might reasonably expect that qx/STRING/ will
return undef if the command exits non-zero, but this is not the case.

I suspect that qx/STRING/ only returns undef if forking and execing
/bin/sh fails, which isn't what I'd think of as "the command" in this
context.

I believe the documentation in pod/perlop.pod for
'qx/STRING' is misleading -- but not quite in the way you
describe. Here is the relevant documentation​:

#####
"qx/*STRING*/"
`*STRING*`
A string which is (possibly) interpolated and then executed
as a system command with /bin/sh or its equivalent. Shell
wildcards, pipes, and redirections will be honored. The
collected standard output of the command is returned;
standard error is unaffected. In scalar context, it comes
back as a single (potentially multi-line) string, or "undef"
if the command failed. In list context, returns a list of
lines (however you've defined lines with $/ or
$INPUT_RECORD_SEPARATOR), or an empty list if the command
failed.
#####

The part that is correct is that the return value from
qx/<command>/ is the collected STDOUT from that command.
Whether that command fully succeeded or not is irrelevant to
the return value. If part of the shell command succeeded
and generated STDOUT before the command ultimately failed as
a whole, then that STDOUT is captured in the return value.
The command's exit status is in no way reflected in the
return value.

As far as I can tell from a couple of hours of focus on the
problem (see attachment 133235-qx.t), the return value from
qx/STRING/ will never be undefined. In scalar context it
will be a (potentially multi-line) string -- perhaps the
empty string. In list context it will be a list of 0 or
more elements.

If I'm correct then we need a patch for the documentation and possibly some tests like those in the attachment which drive the point home.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

From @jkeenan

#!/usr/bin/env perl
use strict;
use warnings;
use 5.10.1;
use Cwd;
use File​::Spec;
use File​::Temp qw(tempdir);
use File​::Path 2.15 qw(make_path);
use Test​::More;
use Data​::Dump qw(dd);

my ($rv, @​lines, $cmd);
my $cwd = cwd();
my $hw = 'hello world';

{
  my $tdir = tempdir(CLEANUP => 1);
  chdir $tdir or die "Unable to chdir to tempdir​: $!";

  my $sdir = File​::Spec->catdir($tdir, qw(foo bar));
  ok(! -d $sdir, "$sdir does not yet exist");

  $cmd = "echo -n $hw";
  $rv = qx/$cmd/;
  is($rv, $hw, "qx captured STDOUT '$hw' rom shell command, as expected");

  $cmd = "echo -n $hw && cd $sdir";
  $rv = qx/$cmd/;
  is($rv, $hw,
  "qx captured STDOUT '$hw' from shell command even though command did not fully succeed");

  $cmd = "cd $sdir && echo -n $hw";
  $rv = qx/$cmd/;
  ok(defined $rv, "qx returned a defined value even though shell command failed");
  is(length $rv, 0, "... but that defined value was an empty string");

  $cmd = "echo $hw && cd $sdir && echo 'Who do you love?'";
  chomp(@​lines = qx/$cmd/);
  is(scalar(@​lines), 1, "Only 1 line captured before failure of shell command within qx");
  is($lines[0], $hw, "qx captured '$hw' before failure of shell command");

  $cmd = "$sdir";
  @​lines = qx/$cmd/;
  is(scalar(@​lines), 0,
  "Command (which failed) sent nothing to STDOUT; hence return value in scalar context is empty list");

  make_path($sdir) or die "Unable to create $sdir for testing";
  ok(-d $sdir, "Created $sdir for testing");

  $cmd = "echo -n $hw && cd $sdir && echo -n ' ' && pwd";
  $rv = qx/$cmd/;
  like($rv, qr/$hw/, "'$hw' found in STDOUT captured by qx over from shell command");
  like($rv, qr/$sdir/, "Directory '$sdir' to which shell changed also captured by qx");
  chdir $tdir or die "Unable to change back";

  $cmd = "echo $hw && cd $sdir && echo '' && pwd";
  chomp(@​lines = qx/$cmd/);
  is($lines[0], $hw, "'$hw' found in STDOUT captured by qx over from shell command");
  is($lines[1], '', "Second line is empty string");
  like($lines[2], qr/$sdir/, "Directory '$sdir' to which shell changed also captured by qx");
  chdir $tdir or die "Unable to change back";

  chdir $cwd or die "Unable to chdir back to starting point​: $!";
}

done_testing();

The RT System itself - Status changed from 'new' to 'open'

From @tonycoz

On Mon, 03 Sep 2018 17​:08​:33 -0700, jkeenan wrote​:

As far as I can tell from a couple of hours of focus on the
problem (see attachment 133235-qx.t), the return value from
qx/STRING/ will never be undefined. In scalar context it
will be a (potentially multi-line) string -- perhaps the
empty string. In list context it will be a list of 0 or
more elements.

$ ./perl -Ilib -lE '$x = `lsx`; print $x // "undef";'
undef

From looking at the code it "fails" if the fork or exec fails, or if other system calls fail (eg. pipe())

If your command includes shell metacharacters then the shell executes successfully and so a defined value is returned (and the shell complains)​:

$ ./perl -Ilib -lE '$x = `lsx;`; print $x // "undef";'
sh​: 1​: lsx​: not found

Tony