Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-compliant Syslog UDP packets #7200

Closed
p5pRT opened this issue Mar 28, 2004 · 12 comments
Closed

Non-compliant Syslog UDP packets #7200

p5pRT opened this issue Mar 28, 2004 · 12 comments

Comments

@p5pRT
Copy link

p5pRT commented Mar 28, 2004

Migrated from rt.perl.org#28019 (status was 'resolved')

Searchable as RT28019$

@p5pRT
Copy link
Author

p5pRT commented Mar 28, 2004

From lool@via.ecp.fr

Created by lool@via.ecp.fr

  Hi,

I experienced troubles with Perl-generated UDP Syslog messages and
it turns out Perl sends non RFC compliant datagrams.

Here's a sample program doing UDP Syslog-ing​:

| #!/usr/bin/perl -w
|
| use Sys​::Syslog qw(​:DEFAULT setlogsock);
|
| setlogsock([ 'udp' ]);
| openlog("testprg", 'ndelay', 'local7');
|
| syslog('info', 'test');
|
| closelog;
|

When tcpdumping the above (or strace-ing), one can see the datagrams
end in 0a 00 ("\n\000")​:
0x0000 4500 0030 a2f7 4000 4011 99c1 7f00 0002 E..0..@​.@​.......
0x0010 7f00 0002 804e 0202 001c 5346 3c31 3930 .....N....SF<190
0x0020 3e74 6573 7470 7267 3a20 7465 7374 0a00 >testprg​:.test..

Packets generated by a Linux or Solaris standard syslogd end in 0a
("\n") and packets generated by a standard BSD syslogd have no
terminator at all.

The RFC 3164 doesn't recommend the use of a terminator, it seems
no terminator are required at all.

While I understand it is preferable for the receiving party to handle
all kinds of terminators, I think it would be best if Perl would
generate standard UDP Syslog messages.

  Regards,

--
Loïc Minier <lool@​via.ecp.fr>

Perl Info

Flags:
    category=library
    severity=low

Site configuration information for perl v5.8.3:

Configured by Debian Project at Sun Feb 15 17:22:09 EST 2004.

Summary of my perl5 (revision 5.0 version 8 subversion 3) configuration:
  Platform:
    osname=linux, osvers=2.4.22-xfs+ti1211, archname=i386-linux-thread-multi
    uname='linux kosh 2.4.22-xfs+ti1211 #1 sat oct 25 10:11:37 est 2003 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i386-linux -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.3 -Dsitearch=/usr/local/lib/perl/5.8.3 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.3 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O3',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='3.3.3 20040125 (prerelease) (Debian)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so.5.8.3
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.3:
    /etc/perl
    /usr/local/lib/perl/5.8.3
    /usr/local/share/perl/5.8.3
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8
    /usr/share/perl/5.8
    /usr/local/lib/site_perl
    .


Environment for perl v5.8.3:
    HOME=/home/lool
    LANG=C
    LANGUAGE (unset)
    LC_ADDRESS=fr_FR@euro
    LC_ALL=
    LC_COLLATE=fr_FR@euro
    LC_CTYPE=fr_FR@euro
    LC_IDENTIFICATION=fr_FR@euro
    LC_MEASUREMENT=fr_FR@euro
    LC_MESSAGES=C
    LC_MONETARY=fr_FR@euro
    LC_NAME=fr_FR@euro
    LC_NUMERIC=fr_FR@euro
    LC_PAPER=fr_FR@euro
    LC_TELEPHONE=fr_FR@euro
    LC_TIME=fr_FR@euro
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/lool/bin/:/home/lool/bin/kernel/:/home/lool/bin/zsh/:/usr/local/sbin:/sbin:/usr/sbin:/usr/local/bin:/bin:/usr/bin:/usr/bin/X11:/usr/games:/usr/bin/X11:/usr/games
    PERL_BADLANG (unset)
    SHELL=/bin/zsh

@p5pRT
Copy link
Author

p5pRT commented Mar 31, 2004

From @iabyn

On Sun, Mar 28, 2004 at 01​:47​:26PM -0000, Loc Minier wrote​:

I experienced troubles with Perl-generated UDP Syslog messages and
it turns out Perl sends non RFC compliant datagrams.

Here's a sample program doing UDP Syslog-ing​:

| #!/usr/bin/perl -w
|
| use Sys​::Syslog qw(​:DEFAULT setlogsock);
|
| setlogsock([ 'udp' ]);
| openlog("testprg", 'ndelay', 'local7');
|
| syslog('info', 'test');
|
| closelog;
|

When tcpdumping the above (or strace-ing), one can see the datagrams
end in 0a 00 ("\n\000")​:
0x0000 4500 0030 a2f7 4000 4011 99c1 7f00 0002 E..0..@​.@​.......
0x0010 7f00 0002 804e 0202 001c 5346 3c31 3930 .....N....SF<190
0x0020 3e74 6573 7470 7267 3a20 7465 7374 0a00 >testprg​:.test..

Packets generated by a Linux or Solaris standard syslogd end in 0a
("\n") and packets generated by a standard BSD syslogd have no
terminator at all.

The RFC 3164 doesn't recommend the use of a terminator, it seems
no terminator are required at all.

While I understand it is preferable for the receiving party to handle
all kinds of terminators, I think it would be best if Perl would
generate standard UDP Syslog messages.

Looking at the history of this, perl Syslog has included the \n since at
least the perl 4.036 syslog.pl days; however the \0 was deliberatley
added between 5.003_22 and 5.004_05, by change #81. That change includes
(amongst many others),

  Title​: "Sys​::Syslog patch to allow unix domain sockets"
  From​: Sean Robinson <robinson_s@​sc.maricopa.edu>
  Msg-ID​: <33B31342.7EB16A44@​sc.maricopa.edu>
  Files​: lib/Sys/Syslog.pm

which I guess must be the change in question, but I can't find anything
more about it. It's possible that the \0 is required for the UNIX domain
stuff but not for UDP. Anyway, I'm reluctant to just change it back
without further info.

Dave.

--
A power surge on the Bridge is rapidly and correctly diagnosed as a faulty
capacitor by the highly-trained and competent engineering staff.
  -- Things That Never Happen in "Star Trek" #9

@p5pRT
Copy link
Author

p5pRT commented Mar 31, 2004

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Apr 1, 2004

From lool@via.ecp.fr

Dave Mitchell via RT <perlbug-followup@​perl.org> - Wed, Mar 31, 2004​:

Title&#8203;:  "Sys&#8203;::Syslog patch to allow unix domain sockets"
From&#8203;:  Sean Robinson \<robinson\_s@&#8203;sc\.maricopa\.edu>

which I guess must be the change in question, but I can't find anything
more about it. It's possible that the \0 is required for the UNIX domain
stuff but not for UDP. Anyway, I'm reluctant to just change it back
without further info.

I thik the \n is ok, it's included in most packets, only the BSD
syslogd I have had access to did not include any terminator. Linux
syslogd and syslog-ng both add a \n.

I understand you can be reluctant to change a possibly widely-used
package, but how could I further clear things up? Should I contact
Sean Robinson?

  Regards,

--
Loïc Minier <lool@​via.ecp.fr>

@p5pRT
Copy link
Author

p5pRT commented Apr 1, 2004

From @iabyn

On Thu, Apr 01, 2004 at 09​:56​:48AM +0200, Loic Minier wrote​:

Dave Mitchell via RT <perlbug-followup@​perl.org> - Wed, Mar 31, 2004​:

Title&#8203;:  "Sys&#8203;::Syslog patch to allow unix domain sockets"
From&#8203;:  Sean Robinson \<robinson\_s@&#8203;sc\.maricopa\.edu>

which I guess must be the change in question, but I can't find anything
more about it. It's possible that the \0 is required for the UNIX domain
stuff but not for UDP. Anyway, I'm reluctant to just change it back
without further info.

I thik the \n is ok, it's included in most packets, only the BSD
syslogd I have had access to did not include any terminator. Linux
syslogd and syslog-ng both add a \n.

I understand you can be reluctant to change a possibly widely-used
package, but how could I further clear things up? Should I contact
Sean Robinson?

My request for "further info" was aimed mainly at the old-timers on the
perl5-porters mailing this that these emails get sent to; I'm hoping
someone may remember something about this (it was before my time).

Dave.

--
"You're so sadly neglected, and often ignored.
A poor second to Belgium, When going abroad."
  -- Monty Python - "Finland"

@p5pRT
Copy link
Author

p5pRT commented Mar 14, 2006

From julian@mehnle.net

package libmail-spf-query-perl
retitle 356700 syslog-ng​: Inappropriately expects \n (and \0?) in syslog messages
reassign 356700 syslog-ng 1.9.9-1
thanks

John A. Martin wrote​:

The same libmail-spf-query-perl-1.999.1-1 was used before and after
the log lines were run together. The problem arose when the perl and
perl-base packages were upgraded from 5.8.8-2 to 5.8.8-3.

Now when downgrading perl and perl-base to 5.8.8-2 the problem
disappears. When upgrading perl and perl-base to 5.8.8-3 the problem
appears again. The problem in reproducible.

Julian Mehnle wrote​:

There is a line in Sys​::Syslog that adds a newline to the message before
sending it to the syslog socket, but it hasn't changed in Perl 5.8.8

Oh, it _has_ changed! See the attached diff of the Sys​::Syslog changes in
perl 5.8.8-3 vs. perl 5.8.8-2. Apparently the Sys​::Syslog in perl 5.8.8-3
does no longer append a newline (\n) to syslog messages. Since RFC 3164[1],
"The BSD syslog Protocol", does not require a newline terminator in syslog
messages, I think this change is valid and appropriate.

However it seems syslog-ng does not add its own newline when writing the
received messages to the log file, so it relies on the generator of the
message to add the newline. This is a bug in syslog-ng, so I'll reassign
the bug there.

There's probably another, similar issue with regard to adding NULL byte
(\0) terminators to log messages. RFC 3164 does not require such \0
terminators either, and the changed Sys​::Syslog in perl 5.8.8-3, which also
stops appending \0 bytes to syslog messages, seems not to cause problems
with sysklogd.

Although there seems to have been some reluctance by the Perl folks to omit
the \0 terminators due to potential problems with UNIX domain sockets[2], I
don't see why such terminators would be required with UNIX domain sockets
but not with UDP sockets. I am not a sockets expert, though.

If \0 terminators are indeed required with UNIX domain sockets (which I
don't believe to be true), then their omittance in the Sys​::Syslog in perl
5.8.8-3 is inappropriate and should be reverted. Otherwise, this is
another bug in syslog-ng.

In any case, those are not bugs in libmail-spf-query-perl AKA Mail​::SPF​::
Query.

Julian.

References​:
1. http​://www.rfc-editor.org/rfc/rfc3164.txt
2. http​://rt.perl.org/rt3/Ticket/Display.html?id=28019#txn-83160

@p5pRT
Copy link
Author

p5pRT commented Mar 14, 2006

From julian@mehnle.net

06_fix_syslog_null.diff
Remove spurious \n and \0 terminators.

diff -Naur --exclude=debian perl-5.8.8.orig/ext/Sys/Syslog/Syslog.pm perl-5.8.8/ext/Sys/Syslog/Syslog.pm
--- perl-5.8.8.orig/ext/Sys/Syslog/Syslog.pm	2006-01-11 23:22:47.000000000 +1100
+++ perl-5.8.8/ext/Sys/Syslog/Syslog.pm	2006-02-11 11:59:14.000000000 +1100
@@ -676,15 +676,16 @@
 	$mask =~ s/(?<!%)((?:%%)*)%m/$1$err/g;
     }
 
-    $mask .= "\n" unless $mask =~ /\n$/;
     $message = @_ ? sprintf($mask, @_) : $mask;
+    $message =~ s/[\r\n]+/ /g;
+    $message =~ s/ +$//;
 
     $sum = $numpri + $numfac;
     my $oldlocale = setlocale(LC_TIME);
     setlocale(LC_TIME, 'C');
     my $timestamp = strftime "%b %e %T", localtime;
     setlocale(LC_TIME, $oldlocale);
-    my $buf = "<$sum>$timestamp $whoami: $message\0";
+    my $buf = "<$sum>$timestamp $whoami: $message";
 
     # it's possible that we'll get an error from sending
     # (e.g. if method is UDP and there is no UDP listener,
@@ -724,7 +725,6 @@
 
 sub _syslog_send_console {
     my ($buf) = @_;
-    chop($buf); # delete the NUL from the end
     # The console print is a method which could block
     # so we do it in a child process and always return success
     # to the caller.
@@ -743,7 +743,7 @@
 	}
     } else {
         if (open(CONS, ">/dev/console")) {
-	    my $ret = print CONS $buf . "\r";
+	    my $ret = print CONS $buf . "\r\n";
 	    exit ($ret) if defined $pid;
 	    close CONS;
 	}

@p5pRT
Copy link
Author

p5pRT commented Mar 14, 2006

From maddingue@free.fr

Hello,

Julian Mehnle wrote​:

There is a line in Sys​::Syslog that adds a newline to the message
before
sending it to the syslog socket, but it hasn't changed in Perl 5.8.8

Oh, it _has_ changed! See the attached diff of the Sys​::Syslog
changes in
perl 5.8.8-3 vs. perl 5.8.8-2. Apparently the Sys​::Syslog in perl
5.8.8-3
does no longer append a newline (\n) to syslog messages. Since RFC
3164[1], "The BSD syslog Protocol", does not require a newline
terminator in syslog messages, I think this change is valid and
appropriate.

However it seems syslog-ng does not add its own newline when writing
the
received messages to the log file, so it relies on the generator of the
message to add the newline. This is a bug in syslog-ng, so I'll
reassign
the bug there.

There's probably another, similar issue with regard to adding NULL byte
(\0) terminators to log messages. RFC 3164 does not require such \0
terminators either, and the changed Sys​::Syslog in perl 5.8.8-3, which
also
stops appending \0 bytes to syslog messages, seems not to cause
problems
with sysklogd.

Just for the records, and as I see you still referring to RFC3164, I'll
restate here what I privately wrote to Brendan O'Dea​:

I discussed this issue with an experienced Unix sysadmin. We found
the following facts​:

1. As I said in a previous mail, RFC3164 was written in august 2001,
while syslog daemons have been here since probably 20 years or so.
Hence my question, how valid can this RFC be, which leads to the
second point.

2. The RFC 3164 is in the category "informational", which indicates
that it does not defines a standard, but provides informational,
non-normative documentation.

3. The title of the RFC 3164 is "The BSD syslog Protocol". Not "The
unix syslog Protocol" or "The syslog Protocol".

We therefore reached the conclusion that this RFC was written by
someone at Cisco for documenting the way the syslog daemon present in
BSD systems works and expects data, but is in no way a normative
reference.

Hence, in the strict literal meaning of the terms, no software can
claim RFC compliance as there is no normative RFC that officially
defines the syslog protocol. The conclusion is that I feel now
allowed to blame syslog-ng for not accepting Sys​::Syslog messages. I
can probably even remark that syslog-ng being most probably younger
than the version of Sys​::Syslog with UDP support by a few years, it
should be compatible with Sys​::Syslog and not the other way around >> :-)

Although there seems to have been some reluctance by the Perl folks to
omit
the \0 terminators due to potential problems with UNIX domain
sockets[2], I
don't see why such terminators would be required with UNIX domain
sockets
but not with UDP sockets. I am not a sockets expert, though.

If \0 terminators are indeed required with UNIX domain sockets (which I
don't believe to be true), then their omittance in the Sys​::Syslog in
perl
5.8.8-3 is inappropriate and should be reverted. Otherwise, this is
another bug in syslog-ng.

This NULL byte was most probably added to be compatible with some given
commercial Unix syslog daemon (I think there are people here who can
speak about broken implementations ;-). Even if we could test a version
Sys​::Syslog without that NULL byte on all current Unix systems and
check that it works as expected, this won't help for already installed
systems, the kind of "works, won't touch, won't upgrade!"

Best Regards,

Sébastien Aperghis-Tramoni
  -- - --- -- - -- - --- -- - --- -- - --[ http​://maddingue.org ]
Close the world, txEn eht nepO

@p5pRT
Copy link
Author

p5pRT commented Apr 20, 2006

From julian@mehnle.net

package syslog-ng
retitle 345157 syslog-ng​: Should strip \0 terminators from syslog messages
tags 345157 - patch
retitle 356700 perl (Sys​::Syslog)​: \0 terminator required in syslog messages
reassign 356700 perl 5.8.8-3
package perl
close 356700 5.8.8-4
thanks

Cleaning up bugs #356700 and #345157 in order to avoid any further
confusion for those who encounter them in the BTS after us. I reassigned
#356700 to perl before closing it as a reminder that \0 terminators are
required with stream sockets[1]. This fact should be documented in the
Sys​::Syslog code! (Perhaps the Debian perl maintainers could submit a
documentation patch upstream?)

Julian Mehnle.

References​:
1. http​://bugs.debian.org/cgi-bin/bugreport.cgi?bug=356700#msg64

@p5pRT
Copy link
Author

p5pRT commented Mar 29, 2016

From odc@cpan.org

This issue has been resolved since 2011 with Sys​::Syslog v0.29 (Added options noeol and nonul).

@p5pRT
Copy link
Author

p5pRT commented Mar 31, 2016

From @rjbs

Thanks, closed!

--
rjbs

@p5pRT p5pRT closed this as completed Mar 31, 2016
@p5pRT
Copy link
Author

p5pRT commented Mar 31, 2016

@rjbs - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant