From rob.navarro@appdynamics.com

Created by rob.navarro@appdynamics.com

Subject​: paragraph input mode ($/="") prevents readline from returning
latest paragraph from pipe
Message-Id​: <5.10.1_6992_1553291784@​centos67S>
Reply-To​: rob.navarro@​appdynamics.com
To​: perlbug@​perl.org

This is a bug report for perl from rob.navarro@​appdynamics.com,
generated with the help of perlbug 1.39 running under perl 5.10.1.

-----------------------------------------------------------------
Noticed whilst trying to filter live and long-running "iostat -tmzx 3" in
time sensitive application.

The very last "paragraph" written into a pipe is not found by readline
until the next block of pipe inputs are emitted or EOF is sent.
There are no issues when reading from a file.
This is only a problem with readline in paragraph mode reading from a pipe
i.e. perl -000 ... or perl -e '$/=""; ...'

[terminal output follows]
# Firstly to prove I can read line buffered rows on right hand side of pipe
i.e. reading when they are sent without delay​:

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) { print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'| {
while read -r L; do echo "$(date +%FT%T)>> $L"; done; }
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 1
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 2
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 3
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 1
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 2
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 3
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 1
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 2
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 3
2019-03-22T15​:22​:37>>

# Notice that in above the pipe reader's timestamp matches the timestamp
sent through the pipe for each of the three "paragraphs".
# Now with Perl reading from pipe with -000 or equally problematic with
$/=""

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) { print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }' |
perl -MPOSIX -000 -e 'while (<>) { print "## ".strftime( "%FT%T", localtime
)." ## >$_<"; }'
## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 1

<## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 2

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:44 3

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 1

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 2

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:50 3

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 1

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 2

<## 2019-03-22T15​:23​:00 ## >## 2019-03-22T15​:22​:55 3

# Notice that only two of the three paragraphs are output before 5 second
sleep. Pipe reader timestamp is always 5 seconds behind for third
"paragraph".

[end of terminal output]

This issue seems to occur regardless of how big or small each paragraph is
and seems to be just related to the last paragraph not being found by
readline.
There is no problem with $/="\n" or sysread from a pipe, only a problem
when paragraph mode for readline() has been enabled i.e. -000 or $/="";

Also a problem on v5.18.2

Flags:
    category=core
    severity=medium

This perlbug was built using Perl 5.10.1 in the Fedora build system.
It is being executed now by Perl 5.10.1 - Wed Mar 22 11:01:16 UTC 2017.

Site configuration information for perl 5.10.1:

Configured by Red Hat, Inc. at Wed Mar 22 11:01:16 UTC 2017.

Summary of my perl5 (revision 5 version 10 subversion 1) configuration:

  Platform:
    osname=linux, osvers=3.10.0-514.10.2.el7.x86_64,
archname=x86_64-linux-thread-multi
    uname='linux c1bm.rdu2.centos.org 3.10.0-514.10.2.el7.x86_64 #1 smp fri
mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64
-mtune=generic -DDEBUGGING=-g -Dversion=5.10.1 -Dmyhostname=localhost
-Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr
-Dvendorprefix=/usr -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5
-Dprivlib=/usr/share/perl5 -Darchlib=/usr/lib64/perl5
-Dvendorlib=/usr/share/perl5/vendor_perl
-Dvendorarch=/usr/lib64/perl5/vendor_perl -Dinc_version_list=5.10.0
-Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64
/usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles
-Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog
-Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005
-Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto
-Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto
-Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto
-Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing
-pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.4.7 20120313 (Red Hat 4.4.7-18)',
gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -fstack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.12'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/usr/lib64/perl5/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic'

Locally applied patches:



@INC for perl 5.10.1:
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5
    .


Environment for perl 5.10.1:
    HOME=/home/appdyn
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)

PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/appdyn/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash


-- 
*Rob Navarro* | Principal Software Engineer
Office +1 415-442-3004

*AppDynamics*
The Application Intelligence Company
Watch our video
<https://player.vimeo.com/external/137403850.hd.mp4?s=14c2030b6f8bc436aebba39b1b89ef03&profile_id=119>
  |  Try our FREE Trial
<https://portal.appdynamics.com/account/signup/signupForm>  |  Twitter
<https://twitter.com/appdynamics>  |  LinkedIn
<https://www.linkedin.com/company/appdynamics>  |  appdynamics.com
<https://www.appdynamics.com>

<http://www.appdynamics.com/appsphere>AppDynamics, Inc. is a Delaware
corporation with its principal place of business located at 303 Second
Street, North Tower, 8th Floor, San Francisco, CA 94107.
Tel: +1 415.442.8400 website: appdynamics.com <https://www.appdynamics.com>

From @jkeenan

On Fri, 22 Mar 2019 22​:48​:18 GMT, rob.navarro@​appdynamics.com wrote​:

Subject​: paragraph input mode ($/="") prevents readline from returning
latest paragraph from pipe
Message-Id​: <5.10.1_6992_1553291784@​centos67S>
Reply-To​: rob.navarro@​appdynamics.com
To​: perlbug@​perl.org

This is a bug report for perl from rob.navarro@​appdynamics.com,
generated with the help of perlbug 1.39 running under perl 5.10.1.

-----------------------------------------------------------------
Noticed whilst trying to filter live and long-running "iostat -tmzx 3"
in
time sensitive application.

The very last "paragraph" written into a pipe is not found by readline
until the next block of pipe inputs are emitted or EOF is sent.
There are no issues when reading from a file.
This is only a problem with readline in paragraph mode reading from a
pipe
i.e. perl -000 ... or perl -e '$/=""; ...'

[terminal output follows]
# Firstly to prove I can read line buffered rows on right hand side of
pipe
i.e. reading when they are sent without delay​:

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) {
print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'|
{
while read -r L; do echo "$(date +%FT%T)>> $L"; done; }
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 1
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 2
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 3
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 1
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 2
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 3
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 1
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 2
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 3
2019-03-22T15​:22​:37>>

# Notice that in above the pipe reader's timestamp matches the
timestamp
sent through the pipe for each of the three "paragraphs".
# Now with Perl reading from pipe with -000 or equally problematic
with
$/=""

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) {
print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'
|
perl -MPOSIX -000 -e 'while (<>) { print "## ".strftime( "%FT%T",
localtime
)." ## >$_<"; }'
## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 1

<## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 2

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:44 3

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 1

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 2

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:50 3

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 1

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 2

<## 2019-03-22T15​:23​:00 ## >## 2019-03-22T15​:22​:55 3

# Notice that only two of the three paragraphs are output before 5
second
sleep. Pipe reader timestamp is always 5 seconds behind for third
"paragraph".

[end of terminal output]

This issue seems to occur regardless of how big or small each
paragraph is
and seems to be just related to the last paragraph not being found by
readline.
There is no problem with $/="\n" or sysread from a pipe, only a
problem
when paragraph mode for readline() has been enabled i.e. -000 or
$/="";

Also a problem on v5.18.2

Confirmed in perl-5.28.0.

--
James E Keenan (jkeenan@​cpan.org)

The RT System itself - Status changed from 'new' to 'open'

From @tonycoz

On Fri, 22 Mar 2019 15​:48​:18 -0700, rob.navarro@​appdynamics.com wrote​:

Noticed whilst trying to filter live and long-running "iostat -tmzx 3"
in
time sensitive application.

The very last "paragraph" written into a pipe is not found by readline
until the next block of pipe inputs are emitted or EOF is sent.
There are no issues when reading from a file.
This is only a problem with readline in paragraph mode reading from a
pipe
i.e. perl -000 ... or perl -e '$/=""; ...'

[terminal output follows]
# Firstly to prove I can read line buffered rows on right hand side of
pipe
i.e. reading when they are sent without delay​:

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) {
print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'|
{
while read -r L; do echo "$(date +%FT%T)>> $L"; done; }
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 1
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 2
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 3
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 1
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 2
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 3
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 1
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 2
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 3
2019-03-22T15​:22​:37>>

# Notice that in above the pipe reader's timestamp matches the
timestamp
sent through the pipe for each of the three "paragraphs".
# Now with Perl reading from pipe with -000 or equally problematic
with
$/=""

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) {
print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'
|
perl -MPOSIX -000 -e 'while (<>) { print "## ".strftime( "%FT%T",
localtime
)." ## >$_<"; }'
## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 1

<## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 2

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:44 3

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 1

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 2

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:50 3

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 1

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 2

<## 2019-03-22T15​:23​:00 ## >## 2019-03-22T15​:22​:55 3

# Notice that only two of the three paragraphs are output before 5
second
sleep. Pipe reader timestamp is always 5 seconds behind for third
"paragraph".

[end of terminal output]

This issue seems to occur regardless of how big or small each
paragraph is
and seems to be just related to the last paragraph not being found by
readline.
There is no problem with $/="\n" or sysread from a pipe, only a
problem
when paragraph mode for readline() has been enabled i.e. -000 or
$/="";

This happens because paragraph mode tries to consume *all* of the newlines following the paragraph.

So in your case, readline() is blocking on trying to read that next character to check if it's a newline.

I expect you'd be better off using $/ = "\n\n"

Tony

From rob.navarro@appdynamics.com

On Sun, 24 Mar 2019 21​:50​:29 -0700, tonyc wrote​:

On Fri, 22 Mar 2019 15​:48​:18 -0700, rob.navarro@​appdynamics.com wrote​:

Noticed whilst trying to filter live and long-running "iostat -tmzx
3"
in
time sensitive application.

The very last "paragraph" written into a pipe is not found by
readline
until the next block of pipe inputs are emitted or EOF is sent.
There are no issues when reading from a file.
This is only a problem with readline in paragraph mode reading from a
pipe
i.e. perl -000 ... or perl -e '$/=""; ...'

Thank you for the clear response (tried responding to email but test did not show up here).
I understand your point about readline() needing to consume all inter-paragraph newlines, but this should not mean that *after* a valid delimited paragraph is found that that paragraph should not be returned immediately.
Surely it makes better sense to return the correctly delimited paragraph as soon as it is recognised and *then* keep consuming new lines. After all, the final paragraph has been correctly delimited and we should now just be awaiting the next one - if any.

From rob.navarro@appdynamics.com

Thank you for the clear response.
I understand your point about readline() needing to consume all
inter-paragraph newlines, but this should not mean that after a valid
delimited paragraph is found that that paragraph should not be returned
immediately.
Surely it makes better sense to return the correctly delimited paragraph as
soon as it is recognised and *then* keep consuming new lines. After all,
the final paragraph has been correctly delimited and we should now just be
awaiting the next one - if any.

On Sun, Mar 24, 2019 at 9​:50 PM Tony Cook via RT <perlbug-followup@​perl.org>
wrote​:

On Fri, 22 Mar 2019 15​:48​:18 -0700, rob.navarro@​appdynamics.com wrote​:

Noticed whilst trying to filter live and long-running "iostat -tmzx 3"
in
time sensitive application.

The very last "paragraph" written into a pipe is not found by readline
until the next block of pipe inputs are emitted or EOF is sent.
There are no issues when reading from a file.
This is only a problem with readline in paragraph mode reading from a
pipe
i.e. perl -000 ... or perl -e '$/=""; ...'

[terminal output follows]
# Firstly to prove I can read line buffered rows on right hand side of
pipe
i.e. reading when they are sent without delay​:

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) {
print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'|
{
while read -r L; do echo "$(date +%FT%T)>> $L"; done; }
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 1
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 2
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:27>> ## 2019-03-22T15​:22​:27 3
2019-03-22T15​:22​:27>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 1
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 2
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:32>> ## 2019-03-22T15​:22​:32 3
2019-03-22T15​:22​:32>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 1
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 2
2019-03-22T15​:22​:37>>
2019-03-22T15​:22​:37>> ## 2019-03-22T15​:22​:37 3
2019-03-22T15​:22​:37>>

# Notice that in above the pipe reader's timestamp matches the
timestamp
sent through the pipe for each of the three "paragraphs".
# Now with Perl reading from pipe with -000 or equally problematic
with
$/=""

$ perl -MPOSIX -e 'select((select( STDOUT ), $|=1)[0]);for (1..3) {
print
"## ".strftime( "%FT%T", localtime )." $_\n\n" for (1..3); sleep 5 }'
|
perl -MPOSIX -000 -e 'while (<>) { print "## ".strftime( "%FT%T",
localtime
)." ## >$_<"; }'
## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 1

<## 2019-03-22T15​:22​:44 ## >## 2019-03-22T15​:22​:44 2

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:44 3

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 1

<## 2019-03-22T15​:22​:50 ## >## 2019-03-22T15​:22​:50 2

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:50 3

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 1

<## 2019-03-22T15​:22​:55 ## >## 2019-03-22T15​:22​:55 2

<## 2019-03-22T15​:23​:00 ## >## 2019-03-22T15​:22​:55 3

# Notice that only two of the three paragraphs are output before 5
second
sleep. Pipe reader timestamp is always 5 seconds behind for third
"paragraph".

[end of terminal output]

This issue seems to occur regardless of how big or small each
paragraph is
and seems to be just related to the last paragraph not being found by
readline.
There is no problem with $/="\n" or sysread from a pipe, only a
problem
when paragraph mode for readline() has been enabled i.e. -000 or
$/="";

This happens because paragraph mode tries to consume *all* of the newlines
following the paragraph.

So in your case, readline() is blocking on trying to read that next
character to check if it's a newline.

I expect you'd be better off using $/ = "\n\n"

Tony

--
*Rob Navarro* | Principal Software Engineer
Office +1 415-442-3004

*AppDynamics*
The Application Intelligence Company
Watch our video
<https://player.vimeo.com/external/137403850.hd.mp4?s=14c2030b6f8bc436aebba39b1b89ef03&profile_id=119>
  | Try our FREE Trial
<https://portal.appdynamics.com/account/signup/signupForm>  | Twitter
<https://twitter.com/appdynamics>  | LinkedIn
<https://www.linkedin.com/company/appdynamics>  | appdynamics.com
<https://www.appdynamics.com>

<http​://www.appdynamics.com/appsphere>AppDynamics, Inc. is a Delaware
corporation with its principal place of business located at 303 Second
Street, North Tower, 8th Floor, San Francisco, CA 94107.
Tel​: +1 415.442.8400 website​: appdynamics.com <https://www.appdynamics.com>

From @tonycoz

On Wed, 27 Mar 2019 21​:42​:21 -0700, rob.navarro@​appdynamics.com wrote​:

Thank you for the clear response.
I understand your point about readline() needing to consume all
inter-paragraph newlines, but this should not mean that after a valid
delimited paragraph is found that that paragraph should not be returned
immediately.
Surely it makes better sense to return the correctly delimited paragraph as
soon as it is recognised and *then* keep consuming new lines. After all,
the final paragraph has been correctly delimited and we should now just be
awaiting the next one - if any.

The problem with not consuming the extra newlines is the next read from the stream might not be from readline, existing code might expect paragraph mode to consume them and then switch $/ = "\n" to consume a line at a time, or use read().

It would be a backwardly incompatible change.

Tony

From rob.navarro@appdynamics.com

Thank you for your response.

By analogy with the single newline separate case, I now understand your point and agree. When each row is returned it is also assumed to have *already* consumed the 'newline' separator that comes next.

Hence my observation of delayed readline (which I assumed was a bug) is a simple consequence of​:
1 inputs are not returned until their next separator is fully consumed/observed
2 paragraph mode promises to consume an unlimited number of contiguous newlines as a single separator

therefore paragraph mode should always hold on to an input until it sees either EOF or the next non-delimiter - which makes it rubbish for pipe processing !

Please go ahead and close this issue as a non-bug (surprising consequence).

On Sun, 31 Mar 2019 16​:34​:54 -0700, tonyc wrote​:

On Wed, 27 Mar 2019 21​:42​:21 -0700, rob.navarro@​appdynamics.com wrote​:

Thank you for the clear response.
I understand your point about readline() needing to consume all
inter-paragraph newlines, but this should not mean that after a valid
delimited paragraph is found that that paragraph should not be
returned
immediately.
Surely it makes better sense to return the correctly delimited
paragraph as
soon as it is recognised and *then* keep consuming new lines. After
all,
the final paragraph has been correctly delimited and we should now
just be
awaiting the next one - if any.

The problem with not consuming the extra newlines is the next read
from the stream might not be from readline, existing code might expect
paragraph mode to consume them and then switch $/ = "\n" to consume a
line at a time, or use read().

It would be a backwardly incompatible change.

Tony