fast scan

[Rainer Weikusat]

...

The solution is really simply to rate-limit requests being sent

[...]

This 'parallelism' idea is inherently broken because replies aren't
guaranteed to arrive.

[...]

Maybe, I'm confused but my understanding is that each boxcar on the
"POE-Zug" fires off its icmp ping, waits for the user-configurable
timeout, and gathers any/all responses.

This description assumes a full-fledged 'virtual thread' per ping but
it is generally correct: The 'parallelism' thing requires an
individual timeout per request in flight in order to keep going in
face of replies which may never arrive.

Overall send-rate is throttled with the configurable parallelism
setting.

'Send 1000 requests as fast as you can, then, do nothing for ten
seconds' is not the same as 'continue sending a request every 0.01s
for 10 seconds': Again simplifying things, 'an ethernet is binary': At
any given time, it is either 'in use' or 'not in use'. The bulk send
means it is 'in use' for a relatively long period of time at the
beginning and will be 'in use' for a similarly long period of time as
soon as the replies start arriving. Otherwise, it will be 'in use' for
many short time periods and 'be available' in between (the same is
true for resources on the sending/ receiving host where it means 'be
available to deal with replies').

If that rate is properly tuned to avoid saturating the network,
where does the train run off the tracks? (I'd guess even an
alternative micro-sleep between requests might potentially bog down
a network without tuning)

Even the slowest isosychronous injection of 'packets' into a
network might be the straw breaking the camel's back. That's why
'serious' general purpose algorithms for congestion avoidance are
adaptive (and usually not exactly simple). But it's getting late here,
I still have some (household) work to do and I'm not entirely
sober. Because of this, I'm now terminating this with another 'vulgar
appeal to common sense': Hearing works better when one stops shouting
more frequently.

 

[Charles DeRykus]

'Send 1000 requests as fast as you can, then, do nothing for ten
seconds' is not the same as 'continue sending a request every 0.01s
for 10 seconds': Again simplifying things, 'an ethernet is binary': At
any given time, it is either 'in use' or 'not in use'. The bulk send
means it is 'in use' for a relatively long period of time at the
beginning and will be 'in use' for a similarly long period of time as
soon as the replies start arriving. Otherwise, it will be 'in use' for
many short time periods and 'be available' in between (the same is
true for resources on the sending/ receiving host where it means 'be
available to deal with replies').

Yes, thanks for the clarification. POE auto-sizes various default
settings including 'Parallelism' based on OS and other factors. I notice
there's also a FAQ strategy to introduce pauses within its pseudo-kernel
if the requirement is there. Clearly any approach will require tuning.
....

 

[George Mpouras]

Στις 7/8/2013 08:17, ο/η Charles DeRykus έγÏαψε:

Yes, thanks for the clarification. POE auto-sizes various default
settings including 'Parallelism' based on OS and other factors. I notice
there's also a FAQ strategy to introduce pauses within its pseudo-kernel
if the requirement is there. Clearly any approach will require tuning.

...

can you run the POE stuff while monitor cpu and network and reply your
findings ;

 

[Rainer Weikusat]

'Send 1000 requests as fast as you can, then, do nothing for ten
seconds' is not the same as 'continue sending a request every 0.01s
for 10 seconds': Again simplifying things, 'an ethernet is binary': At
any given time, it is either 'in use' or 'not in use'. The bulk send
means it is 'in use' for a relatively long period of time at the
beginning and will be 'in use' for a similarly long period of time as
soon as the replies start arriving. Otherwise, it will be 'in use' for
many short time periods and 'be available' in between (the same is
true for resources on the sending/ receiving host where it means 'be
available to deal with replies').

[...]

POE auto-sizes various default settings including 'Parallelism'
based on OS and other factors.

According to what I read yesterday, it sets it to the number which
results from dividing the socket buffer size by the 'assumed' size of
an ICMP echo request message. Naively, this can be thought of as the
number to send calls the application could make before being blocked
until space in the socket buffer is again available but this isn't
really true because as soons as there's data in the socket buffer, the
process of sending it and thus removing it from the buffer begins in
the background and the application can neither observe that (if it
isn't monitoring all outgoing ethernet frames in real-time) nor
predict how long the execution of its send calls will actually take:
The way to avoid blocking is to keep sending in non-blocking mode
until an EAGAIN error occurs and then continue sending once 'a
suitable system call' (for Perl, that would be select) indicated that
buffer space is again available. There's really no reason for using
this particular number except "I need some number. This is a
number. Therefore, I need that."

In an ideal scenario, this 'parallellism' idea would work such that
one starts by sending as many message without any delay as someone
guesses that network will be capable of handling and afterwards, the
algorithm auto-adjusts because each reception of a reply will trigger
the next send. Even this can already be considered as problematic
because its the opposite of the TCP 'slow start' behaviour: TCP is
supposed to start transmitting data slowly and increase the send rate
as information regarding what 'the network' can actually handle
becomes available'. This algorithm 'starts fast' and is supposed to
slow down if fast was actually too fast. Also, this ignores the fact
that 'sending requests as fast as we can' means 'replies arriving as
fast as they can' and they are more and more powerful than us: There's
no attempt at 'reply flow control'.

And real conditions are not ideal: Let's assume the local computer
starts with sending a batch of 100 requests within a millisecond. Now
what? TCP says than one should wait for twice the estimated RTT before
assuming that a message was lost in transit but what is 'the estimated
RTT'? There's no data available yet. Also, what if the batch send
happened at an unfortunate time and 95% of the message got dropped? Or
if the 'host list segment' in question happens to be a black hole in
the sense that the IP addresses it is composed of are unused? In both
cases, the application will now uselessly sit idle for some 'best
random guess' for a sensible RTO which may not have the slighest
correlation to the actual RTT.

I could go on for this for some paragraphs because this idea is
really less-than-brilliant in so many ways to mentioning them all is
likely to put off a reader to a sufficient degree that he rather
ignores the issue than works through half of a library ...

 

[Charles DeRykus]

Στις 7/8/2013 08:17, ο/η Charles DeRykus έγÏαψε:


can you run the POE stuff while monitor cpu and network and reply your
findings ;

I think at this point there are too many moving parts to draw definitive
conclusions about speed, cpu, network congestion, etc.
Benchmarks are always hard to get right. FUD can immobilize you if you
let it. More often, just applying common sense measures to mitigate any
potential impact is the best course.

Network congestion should always be a concern though so Rainer's
suggestion of the 0.01s delay is well advised. Especially if you pinging
tens of thousands of hosts across the public internet instead of a few
hundred hosts in a private, less trafficked area. The POE pinger
mentioned could be tweaked with the .01s delay (or even written to take
advantage of POE's own kernel delay call I think). The other potential
impact would be consuming your own host resources too greedily with lots
of rapid forking which is why I suggested POE... as well as the
advantage of POE's large base of robust templates for more complicated
network app's.

 

[Charles DeRykus]

Στις 7/8/2013 08:17, ο/η Charles DeRykus έγÏαψε:


can you run the POE stuff while monitor cpu and network and reply your
findings ;

I think at this point there are too many moving parts to draw definitive
conclusions about speed, cpu, network congestion, etc.
Benchmarks are always hard to get right. FUD can immobilize you if you
let it. More often, just applying common sense measures to mitigate any
potential impact is the best course.

Network congestion should always be a concern though so Rainer's
suggestion of the 0.01s delay is well advised. Especially if you pinging
tens of thousands of hosts across the public internet instead of a few
hundred hosts in a private, less trafficked area. The POE pinger
mentioned could be tweaked with the .01s delay (or even written to take
advantage of POE's own kernel delay call I think). The other potential
impact would be consuming your own host resources too greedily with lots
of rapid forking which is why I suggested POE... as well as the
advantage of POE's large base of robust templates for more complicated
network app's.