Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO::Socket::INET performance PROBLEMS! #14948

Closed
p5pRT opened this issue Sep 30, 2015 · 9 comments
Closed

IO::Socket::INET performance PROBLEMS! #14948

p5pRT opened this issue Sep 30, 2015 · 9 comments

Comments

@p5pRT
Copy link

p5pRT commented Sep 30, 2015

Migrated from rt.perl.org#126224 (status was 'rejected')

Searchable as RT126224$

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2015

From konstantin.kokin@gmail.com

Hello Perl Developers!

I suppose I found strange bug. I started small web server and discovered
that when I'm defining LocalAddr, then I get significant performance impact
in multiple times. The results of the test you can find below. The same
scenario was reproduced on others FreeBSD systems with same perl version.

Perl version v5.14.4

Bencmark utility and command.

[root]# ab -c 10 -n 3000 http​://94.140.102.10​:1080/monitor.html
This is ApacheBench, Version 2.0.40-dev <$Revision​: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http​://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http​://www.apache.org/

Code of the programm

#!/usr/bin/perl
# use bytes;
use HTTP​::Daemon;
use HTTP​::Status;
$SIG{PIPE} = 'IGNORE';

my $res = HTTP​::Response->new(RC_OK);
$res->content("Hello World\n");
$res->content_type('text/plain');

$|++;
my $d = HTTP​::Daemon->new( LocalPort => 1080, ReuseAddr => 1 , LocalAddr =>
'94.140.102.8' ) || die;
print "Please contact me at​: <URL​:", $d->url, ">\n";

for ( 1 .. 4 ) {

  my $pid = fork;
  next if $pid;
  next unless defined $pid;
  do {
  flock $d, 2;
  my $c = $d->accept;
  flock $d, 8;
  my $oldfh = select($c);
  $|++;
  select($oldfh);

  while ( my $r = $c->get_request ) {

  if ( $r->method eq 'GET' ) { #and $r->url->path eq "/monitor.html" ) {
  $c->send_response($res);
  }
  else {
  $c->send_error(RC_FORBIDDEN);
  }
  }
  $c->close;
  undef($c);

  } while (1);
  exit 0;
}
while (1) { waitpid( -1, 0 ) }

====================================================

No LocalAddr declaration


Connection Times (ms)
  min mean[+/-sd] median max
Connect​: 26 26 1.0 26 29
Processing​: 27 28 6.1 28 260
Waiting​: 26 27 6.1 27 259
Total​: 53 54 6.3 54 288

Percentage of the requests served within a certain time (ms)
  50% 54
  66% 54
  75% 57
  80% 57
  90% 57
  95% 57
  98% 58
  99% 58
100% 288 (longest request)


With LocalAddr declaration.


Connection Times (ms)
  min mean[+/-sd] median max
Connect​: 26 26 1.0 26 33
Processing​: 28 93 547.2 29 5035
Waiting​: 28 92 547.2 28 5034
Total​: 54 119 547.2 55 5063

Percentage of the requests served within a certain time (ms)
  50% 55
  66% 56
  75% 58
  80% 59
  90% 59
  95% 59
  98% 63
  99% 4654
100% 5063 (longest request)


Try it by yourself it's very easy to reproduce.

Regards,
K.

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2015

From @leonerd

On Wed, 30 Sep 2015 07​:17​:19 -0700
Konstantin Kokin (via RT) <perlbug-followup@​perl.org> wrote​:

With LocalAddr declaration.
...
Percentage of the requests served within a certain time (ms)
50% 55
66% 56
75% 58
80% 59
90% 59
95% 59
98% 63
99% 4654
100% 5063 (longest request)

Surely that one single request that took much longer, was the name
resolver trying to resolve the name you passed in, and timing out after
5 seconds? Subsequent requests would be cached by the C library.

You might want to see if similar happens with IO​::Socket​::IP; as that
uses getaddrinfo(), this might be better behaved when presented with
strings that already look like numeric addresses, and not involve a DNS
roundtrip.

--
Paul "LeoNerd" Evans

leonerd@​leonerd.org.uk
http​://www.leonerd.org.uk/ | https://metacpan.org/author/PEVANS

@p5pRT
Copy link
Author

p5pRT commented Sep 30, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2015

From konstantin.kokin@gmail.com

Hello LeoNerd!

Probably, you get little bit wrong. Let me describe more details to clarify
situation.

1) I don't use dns record for this test. You can see it from benchmarks
command line.
2) The server act in little bit different way​:
Servers lags(wait) for 5sec each 300-400 replied packets.
Yes really!!! I didn't beleaved my eyes. When I remove LocalAddr line from
the script then it works fine.

I'm working with networking on BSD and Linux more than 15 year and I really
do understand the meaning of DNS

If you want, we can appoint a meeting in Skype or whatever and will
demonstrate the problem online. It takes roughly 5-15mins

Regards,
K.
On Wed, 30 Sep 2015 07​:17​:19 -0700
Konstantin Kokin (via RT) <perlbug-followup@​perl.org> wrote​:

With LocalAddr declaration.
...
Percentage of the requests served within a certain time (ms)
50% 55
66% 56
75% 58
80% 59
90% 59
95% 59
98% 63
99% 4654
100% 5063 (longest request)

Surely that one single request that took much longer, was the name
resolver trying to resolve the name you passed in, and timing out after
5 seconds? Subsequent requests would be cached by the C library.

You might want to see if similar happens with IO​::Socket​::IP; as that
uses getaddrinfo(), this might be better behaved when presented with
strings that already look like numeric addresses, and not involve a DNS
roundtrip.

--
Paul "LeoNerd" Evans

leonerd@​leonerd.org.uk
http​://www.leonerd.org.uk/ | https://metacpan.org/author/PEVANS

@p5pRT
Copy link
Author

p5pRT commented Oct 1, 2015

From Mark@Overmeer.net

* Konstantin Kokin (konstantin.kokin@​gmail.com) [151001 03​:24]​:

1) I don't use dns record for this test. You can see it from benchmarks
command line.
2) The server act in little bit different way​:
Servers lags(wait) for 5sec each 300-400 replied packets.
Yes really!!! I didn't beleaved my eyes. When I remove LocalAddr line from
the script then it works fine.

I tried your script on Linux​: there is only 1 elapse very slow, it does
not relate to the number of requests sent, it does not relate to LocalAddr

It does very much depend on the value of parameter '-c' of ab (in my case
ab2 = ApacheBench, Version 2.3) The number of parallel clients.

I'm working with networking on BSD and Linux more than 15 year and I really
do understand the meaning of DNS

LeoNerd is experienced as well.
I have seen cases where you really do not expect DNS to come into play,
but it still did. To get a reverse name-lookup, for instance. Only if
you log your DNS server, you may find out.

If I make the '-c' small, the extra-ordinary slow is fast. When I make
it much higher, requests get lost. So, it seems to have something to
do with N clients speaking to M servers. It may be a problem of 'ab'.
--
Regards,
  MarkOv


  Mark Overmeer MSc MARKOV Solutions
  Mark@​Overmeer.net solutions@​overmeer.net
http​://Mark.Overmeer.net http​://solutions.overmeer.net

@p5pRT
Copy link
Author

p5pRT commented Oct 6, 2015

From tsibley@cpan.org

On Thu Oct 01 00​:12​:26 2015, Mark@​Overmeer.net wrote​:

* Konstantin Kokin (konstantin.kokin@​gmail.com) [151001 03​:24]​:

I'm working with networking on BSD and Linux more than 15 year and I
really
do understand the meaning of DNS

LeoNerd is experienced as well.
I have seen cases where you really do not expect DNS to come into
play, but it still did. To get a reverse name-lookup, for instance. Only
if you log your DNS server, you may find out.

Indeed, every call to $c->get_request() calls $d->url, which calls gethostbyaddr() in
the case of your usage of LocalAddr​:

  https://github.com/gisle/http-daemon/blob/master/lib/HTTP/Daemon.pm#L51

strace is also a good tool for finding DNS queries where you don't expect it.

Thomas

@p5pRT
Copy link
Author

p5pRT commented Dec 3, 2015

From @iabyn

On Mon, Oct 05, 2015 at 10​:45​:42PM -0700, Thomas Sibley via RT wrote​:

On Thu Oct 01 00​:12​:26 2015, Mark@​Overmeer.net wrote​:

* Konstantin Kokin (konstantin.kokin@​gmail.com) [151001 03​:24]​:

I'm working with networking on BSD and Linux more than 15 year and I
really
do understand the meaning of DNS

LeoNerd is experienced as well.
I have seen cases where you really do not expect DNS to come into
play, but it still did. To get a reverse name-lookup, for instance. Only
if you log your DNS server, you may find out.

Indeed, every call to $c->get_request() calls $d->url, which calls gethostbyaddr() in
the case of your usage of LocalAddr​:

https://github.com/gisle/http-daemon/blob/master/lib/HTTP/Daemon.pm#L51

strace is also a good tool for finding DNS queries where you don't expect it.

Given that the 5000ms delays are almost certainly due to to HTTP​::Daemon
doing a DNS reverse lookup, I'm going to mark this ticket as rejected.

--
If life gives you lemons, you'll probably develop a citric acid allergy.

@p5pRT
Copy link
Author

p5pRT commented Dec 5, 2015

From @jkeenan

Dave indicated he was rejecting the ticket, but didn't actually change the status. Doing so now.
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT p5pRT closed this as completed Dec 5, 2015
@p5pRT
Copy link
Author

p5pRT commented Dec 5, 2015

@jkeenan - Status changed from 'open' to 'rejected'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant