Non-deterministic behavior of multithreaded software

[itt ium]

Group,

I have recently modified a software from single threaded to
multithreaded design. In single threaded version, a connection fd was
read by single thread, so event were always delivered in order. Now
since multiple threads handle different events (on the same fd), the
events are sometimes delivered out of order. Please note, on the
connection fd, events are received in proper order and this non-
deterministic behavior is introduced by my software only.

Following are my doubt

1. How to handle such a situation

2. Is it a bad idea to have multiple threads for the same fd. If yes
than how to increase capacity of a software that receives large amount
of data on a single fd only.

Because of the relevance I am posting the message to following groups.

comp.unix.programmer

comp.programming.threads

thanks
Ittium

 

[Victor Bazarov]

I have recently modified a software from single threaded to
multithreaded design. In single threaded version, a connection fd was
read by single thread, so event were always delivered in order. Now
since multiple threads handle different events (on the same fd), the
events are sometimes delivered out of order.

"Delivered" by whom, to where? Handling of events is one thing,
delivering them is another. Do you need them processed in order, then
don't parallelise their processing. If your events can be processed out
of order but then need to be re-queued, then you need to mark each event
(see below) for reassembling of the queue.

Please note, on the
connection fd, events are received in proper order and this non-
deterministic behavior is introduced by my software only.

Following are my doubt

1. How to handle such a situation

Every event, when read from your 'fd' (whatever that is), needs to be
wrapped in a package with the assigned ordinal number so that after
processing it can be again placed in another queue _sorted_.

2. Is it a bad idea to have multiple threads for the same fd. If yes
than how to increase capacity of a software that receives large amount
of data on a single fd only.

Are you guessing what can and cannot be parallelized? It's better to do
a systematic analysis of your own software to see what improvement you
can make by adding multithreading to it. There are many good/decent
books on thread and designing for parallel execution, why don't you
start with them?

Because of the relevance I am posting the message to following groups.

comp.unix.programmer

comp.programming.threads

No, you're not. I'm reading this in 'comp.lang.c++'. Perhaps you
multiposted, but that's a BAD IDEA(tm). If you have to use more than
one newsgroup, *cross-post*.

V

 

[Jorgen Grahn]

.

2. Is it a bad idea to have multiple threads for the same fd.

For doing the actual *reading*, or for processing the data once read?

If yes
than how to increase capacity of a software that receives large amount
of data on a single fd only.

Impossible to say anything without knowing the specifics. We know (or
can guess) you do some kind of IPC, but that's all. There is no
single best design for all kinds of servers.

And, it's completely offtopic here.

/Jorgen

 

[ittium]

"Delivered" by whom, to where? Handling of events is one thing,
delivering them is another. Do you need them processed in order, then
don't parallelise their processing. If your events can be processed out
of order but then need to be re-queued, then you need to mark each event
(see below) for reassembling of the queue.

There is another module that receives the processed data and that data
is delivered to the module out of order. Moreover We want to increase
the message processing capacity per second for the software.with a
single threaded design, we have reached the maximum capacity (in terms
of message processed per second). Please note we are running the
software on a hexacore machine.

Every event, when read from your 'fd' (whatever that is), needs to be
wrapped in a package with the assigned ordinal number so that after
processing it can be again placed in another queue _sorted_.

There are few more inputs here. My software is handing the call session.
The problem occur, if event for a call are delivered out of order. I
have modified the software such that message belonging to a call are
processed by same thread, in this way ordering of events for the call
state machine is maintained. But this has introduced a pinner thread
that assign the messages to different worker threads based on a a hash
function that take input as call ID.

1. Thread ID may not be uniformly distributed, any logic of work
devision might load few thread.

2. A context switch is introduced because of pinner. Earlier a single
thread was doing all the work (from reading from socket to delivering it
to other modules.

Are you guessing what can and cannot be parallelized? It's better to do
a systematic analysis of your own software to see what improvement you
can make by adding multithreading to it. There are many good/decent
books on thread and designing for parallel execution, why don't you
start with them?

I think, this is very generic comment, You can safely assume that I
know, what can be executed in parallel and I have been using pthread for
quite sometime.

No, you're not. I'm reading this in 'comp.lang.c++'. Perhaps you
multiposted, but that's a BAD IDEA(tm). If you have to use more than one
newsgroup, *cross-post*.

My apologies and thanks for pointing this out. I will definitely try
multiplying in future

 

[ittium]

.

For doing the actual *reading*, or for processing the data once read?


Impossible to say anything without knowing the specifics. We know (or
can guess) you do some kind of IPC, but that's all. There is no
single best design for all kinds of servers.

And, it's completely offtopic here.

My apologies, this was posted by mistake. You can ignore it.