Socket is still connected after Server-Side socket termination.

[Gordon Beaton]

When the client first sends the message the server immediately
closes the socket (I also terminate the server to make sure). The
client, on the other hand, never gets notified that the socket is
closed and everything seems running normal (it still prints out
"true" every socket.isConnect() method call).
[...]

I heard that the way we wrote it isn't the problem. We think the
problem relies on the implementation of sockets in Java or Windows.

No, the problem is that you are relying on isConnected() to tell you
the status of the connection.

The way to determine when the socket is closed is to attempt to read
from it or write to it. Both of these operations will tell you when
you have reached EOF on the correspoding socket stream, indicating
that the remote has closed his end.

/gordon

--

 

[pek]

Basic Idea
So I have this client-server project where the client uses Java's
Socket class to communicate with the server using JSON. Let's say that
I have a fake message that the client sends to the server and the
server closes the socket with the client (it doesn't send any response
to the client). In the client side I have a Thread that loops through
the socket's inputstream and prints out any server response. I also
print out the socket's connection status (socket.isConnected()).

Problem
When the client first sends the message the server immediately closes
the socket (I also terminate the server to make sure). The client, on
the other hand, never gets notified that the socket is closed and
everything seems running normal (it still prints out "true" every
socket.isConnect() method call). Only after a few attempts of sending
a message will the client understand that the socket is closed and
tries to reconnect.

Structure
SockeEngine is a class that handles the thread and has a
BlockingQueue<String> for outgoing messages. The thread loops forever.
The loop looks at the queue, the inputstream and outputstream. When
the queue has a message(s) to send it writes it to the outputstream of
the socket. After that it reads the inputstream and finnaly sleeps for
100ms.

I heard that the way we wrote it isn't the problem. We think the
problem relies on the implementation of sockets in Java or Windows. We
also think that it most likely not. Our temporary solution is sending
a termination message from the server to the client and then closing.
Any ideas/suggestions?

Thank you very much for your time.
-pek

 

[derek]

from your description it sounds like you are not using nio. if you only have one thread that handles all of the input and output streams, and it only reads from the inputstream after it does a write, then i would think that of course you wouldnt see it close until after you did a write. i have never found socket.isClosed() to be reliable if the other end of the communication is shut down. i have found that doing a continuous read on the inputstream is reliable. it will let you know when the socket has been shut down on the other end. at least it has always worked for me. i typically use a background thread for each socket input stream. that is if i am not using nio.

 

[pek]

from your description it sounds like you are not using nio. if you only have one thread that handles all of the input and output streams, and it only reads from the inputstream after it does a write, then i would think that of course you wouldnt see it close until after you did a write. i have never found socket.isClosed() to be reliable if the other end of the communication is shut down. i have found that doing a continuous read on the inputstream is reliable. it will let you know when the socket has been shut down on the other end. at least it has always worked for me. i typically use a background thread for each socket input stream. that is if i am not using nio.

First off, I don not use isConnected() to determine if the socket has
terminated or not. I just print it out to see if it works. And it
doens't. So nothing changes in the code.
The Thread writes data only when it is available but it ALWAYS reads
the inputstream. Even when the socket at the server terminates the
read still works! I print out the bytes and I see gibberish. Here are
my imports in regard to the nio. (No, I don't use it. I just now have
heard of it :S)

import java.io.IOException;
import java.io.InputStream;
import java.iutputStream;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.net.SocketException;
import java.net.UnknownHostException;
import java.util.List;
import java.util.Queue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

Thank you again for your time.

 

[derek]

I think it would behoove you to include a snippet of your code.
From your description its difficult to get a good grasp of what
you are actually doing.
So, the best response your going to get is just some random
guesses.
Maybe just post the portion where you are doing the actual reading
to start with.

 

[pek]

To make things easier here is the code of the method run of the
looping thread :


StringBuffer outgoingBuffer;
List<Message> incomming;
while (running) {
try {
// Count the number of
messages the outgoing queue has
int messageCount = outgoing.size();
outgoingBuffer = new StringBuffer();
for (int i=0;i<messageCount;i++) {
outgoingBuffer.append(new
String(parser.serialize(outgoing.poll())));
}
out.write(outgoingBuffer.toString().getBytes());

int availableBytes = in.available();
byte[] toDeserialize = new byte[availableBytes];
in.read(toDeserialize);
// Use parser to serialize/
deserialize message (from/to JSON)
incomming = parser.deserialize(toDeserialize);
if (incomming.size()>0) {
System.out.println("server: "+new String(toDeserialize));
for (Message message:incomming) {
// Send all
registered messageListeners the message that has been received.
eventManager.fireEvent(message);
}
}
Thread.sleep(100);
} catch(SocketException e) {
for (int i=0;i<CONNECTION_ATTEMPTS;i++) {
try {
System.out.println("Attempt to connect...");
disconnect();
parser.reset();
connect(hostname,port);
out.write(parser.serialize(createRecoveryMessage()));
System.out.println("Connected!");
break;
} catch(Exception e1) {
e1.printStackTrace();
try { Thread.sleep(1000); } catch(InterruptedException e2) {}
}
}
} catch(IOException e) {
} catch(InterruptedException e) {

}
}
try {
socket.close();
} catch(IOException e) {e.printStackTrace();}
}

 

[Roedy Green]

When the client first sends the message the server immediately closes
the socket (I also terminate the server to make sure).

here is how I would tackle this problem.

1. read up on TCP/IP protocol to learn just how it handles close, and
if both ends are even supposed to be immediately notified of the other
end's close. Keep in mind TCP/IP does not send polling packets when
there is no traffic.

2. use Wireshark or other protocol sniffer
http://mindprod.com/jgloss/sniffer.html
to watch packets during an close on some socket you consider to behave
acceptably.

3. watch packets in your case.

4. read the fine print of what isConnected in supposed to tell you.

5. If it turns out TCP/IP is not designed to tell you of disconnect
sufficiently quickly, introduce some packets into your private higher
level protocol.

1.SHUT: other end, please shut down.
2.SHUTTING: other end, I am shutting down, this is the last you will
hear from me.

So the usual sequence in one end issues SHUT the waits for SHUTTING,
then shuts down.

The other end on hearing a SHUT, issues a shutting, then shuts down.

This then works independently of anything in TCP/IP.

IIRC this is what I resorted to in one project, then Sun fixed
something so it was no longer necessary, but I left it in as belt and
suspenders. It fits in with the ISO idea of protocol layering.

The protocol also had heartbeat "are you still alive" packets. It may
be that Sun added that support to basic TCP/IP, not EOF. Sorry, it
was a while ago.

 

[pek]

I was just about to do that.. Here it is..


StringBuffer outgoingBuffer;
List<Message> incomming;
while (running) {
try {
// Count the number of
messages the outgoing queue has
int messageCount = outgoing.size();
outgoingBuffer = new StringBuffer();
for (int i=0;i<messageCount;i++) {
outgoingBuffer.append(new
String(parser.serialize(outgoing.poll())));
}
out.write(outgoingBuffer.toString().getBytes());

int availableBytes = in.available();
byte[] toDeserialize = new byte[availableBytes];
in.read(toDeserialize);
// Use parser to serialize/
deserialize message (from/to JSON)
incomming = parser.deserialize(toDeserialize);
if (incomming.size()>0) {
System.out.println("server: "+new String(toDeserialize));
for (Message message:incomming) {
// Send all
registered messageListeners the message that has been received.
eventManager.fireEvent(message);
}
}
Thread.sleep(100);
} catch(SocketException e) {
for (int i=0;i<CONNECTION_ATTEMPTS;i++) {
try {
System.out.println("Attempt to connect...");
disconnect();
parser.reset();
connect(hostname,port);
out.write(parser.serialize(createRecoveryMessage()));
System.out.println("Connected!");
break;
} catch(Exception e1) {
e1.printStackTrace();
try { Thread.sleep(1000); } catch(InterruptedException e2) {}
}
}
} catch(IOException e) {
} catch(InterruptedException e) {

}
}
try {
socket.close();
} catch(IOException e) {e.printStackTrace();}
}

 

[derek]

Without running it, and just scanning it, it looks ok.
I would split out the reading though into a separate thread from
the writing.
Then just put that thread into a loop on the in.read(buffer).
I've never had any problems doing it that way.

 

[pek]

here is how I would tackle this problem.

1. read up on TCP/IP protocol to learn just how it handles close, and
if both ends are even supposed to be immediately notified of the other
end's close. Keep in mind TCP/IP does not send polling packets when
there is no traffic.

I'm not quite sure. The server properly closes the socket, so, if this
was true, the client would have listened to that.

2. use Wireshark or other protocol snifferhttp://mindprod.com/jgloss/sniffer.html
to watch packets during an close on some socket you consider to behave
acceptably.

3. watch packets in your case.

I am currently developing on localhost. Windows (as wireshock makes it
clear) doesn't support monitoring localhost since there is no physical
interface (modem etc.)

4. read the fine print of what isConnected in supposed to tell you.

After reading some other posts, I am now sure that isConnected()
doesn't tell you if the server has closed the socket.

5. If it turns out TCP/IP is not designed to tell you of disconnect
sufficiently quickly, introduce some packets into your private higher
level protocol.

1.SHUT: other end, please shut down.
2.SHUTTING: other end, I am shutting down, this is the last you will
hear from me.

So the usual sequence in one end issues SHUT the waits for SHUTTING,
then shuts down.

The other end on hearing a SHUT, issues a shutting, then shuts down.

This then works independently of anything in TCP/IP.

This was our immediate "coding reaction". The thing is, we are on a
tight budget, and using network traffic costs us. We thought that the
simplest way was that server should send a "I closed your socket and
don't care what will you do" message and close the socket immediately
after that.

IIRC this is what I resorted to in one project, then Sun fixed
something so it was no longer necessary, but I left it in as belt and
suspenders. It fits in with the ISO idea of protocol layering.

The protocol also had heartbeat "are you still alive" packets. It may
be that Sun added that support to basic TCP/IP, not EOF. Sorry, it
was a while ago.

This greatens the traffic (and our budget). But of course, if this is
the only solution, then we don't have any other choice.

 

[derek]

Also, the inputstream.available() only checks how many bytes
are available to be called and not block.
I dont believe it will tell you when the other side of a
socket is closed.
I believe that is why just putting it into a in.read(buffer) loop
works for me. It will return -1 if the end of the stream has
been reached.

from the javadocs:
read() returns:
the total number of bytes read into the buffer, or -1 is there is no more data because the end of the stream has been reached.

 

[pek]

Also, the inputstream.available() only checks how many bytes
are available to be called and not block.
I dont believe it will tell you when the other side of a
socket is closed.
I believe that is why just putting it into a in.read(buffer) loop
works for me. It will return -1 if the end of the stream has
been reached.

from the javadocs:
read() returns:
the total number of bytes read into the buffer, or -1 is there is no more data because the end of the stream has been reached.

Yes, but read() blocks while read(buffer) doesn't. And I don't want to
use a second thread for concurrency reasons (I am not sure if there
are any, but since there is no harm using one thread, I don't think it
is necessary).

read(buffer) adds bytes to the buffer. Two questions:
what is -1 in bytes..?
the end of stream is reached when the socket is closed or when there
are no data to read? (server isn't sending anything)

 

[Gordon Beaton]

I was just about to do that.. Here it is..

You are throwing away the information you need to determine when the
remote has closed the socket.

Look at the return value of in.read(), it will be -1 when the remote
has closed. In other cases, it will tell you how much data it actually
read, which may at times be less than you expect.

Your failure to look at this value is the reason you think you think
you are receiving garbage after the remote has closed the socket.

/gordon

--

 

[pek]

You are throwing away the information you need to determine when the
remote has closed the socket.

Look at the return value of in.read(), it will be -1 when the remote
has closed. In other cases, it will tell you how much data it actually
read, which may at times be less than you expect.

Well, read(buffer) returns bytes. And -1 is represented in bytes. So
what is -1 in bytes? Messages that come from the server are strings.
So I use String(theReadBytes) to convert them and read them. How do I
do that with -1?
Also, I use read(buffer) and not read() because the latest blocks.

Your failure to look at this value is the reason you think you think
you are receiving garbage after the remote has closed the socket.

I get the same "garbage" even if the server didn't close the socket.
They are probably empty bytes of some kind. The funny is that I get
the same stream of "garbage" even if I close the socket from the
server!

This is an example of the bytes I get when socket opens (I print them
out in each loop)
[B@19bb25a
[B@da6bf4
[B@1e58cb8
[B@179935d
[B@b9e45a
[B@3ef810

And this when the socket on the server has been closed.
[B@17200b4
[B@18c3679
[B@4c47db
[B@1ac3379
[B@6779e6
[B@174219d

Obviously they need to be converted to a human-readable string or
something..

 

[derek]

Yes, but read() blocks while read(buffer) doesn't.

false, the javadocs specifically say that both read() and read(byte[]) block.

read(buffer) adds bytes to the buffer. Two questions:
what is -1 in bytes..?
the end of stream is reached when the socket is closed or when > there
are no data to read? (server isn't sending anything)

-1 means the socket has been closed on the other side.

0 means the actual number of bytes that were read into the

byte[]
i dont think a 0 is ever returned.

btw, all of this is in the javadocs.
http://java.sun.com/j2se/1.5.0/docs/api/java/io/InputStream.html

 

[Gordon Beaton]

Well, read(buffer) returns bytes. And -1 is represented in bytes. So
what is -1 in bytes? Messages that come from the server are strings.
So I use String(theReadBytes) to convert them and read them. How do
I do that with -1? Also, I use read(buffer) and not read() because
the latest blocks.

Both can block.

-1 is the return value of the method at EOF, not the buffer contents.
Do you understand the difference? So:

byte[] buffer = new byte[someNumber];
int n = in.read(buffer);

if (n < 0) {
System.out.println("Remote has closed");
}
else {
System.out.println("Received " + n " bytes");
}

I get the same "garbage" even if the server didn't close the socket.
They are probably empty bytes of some kind. The funny is that I get
the same stream of "garbage" even if I close the socket from the
server!

This is an example of the bytes I get when socket opens (I print them
out in each loop)
[B@19bb25a
[B@da6bf4
[B@1e58cb8
[B@179935d
[B@b9e45a
[B@3ef810

There are no "empty bytes", and the above is not the received data, it
is a representation of the references to your byte arrays. If you want
to display the actual data, you need to iterate over the array
contents (observing n, the return value from in.read()) and display
each byte, perhaps in hex:

for (int i=0; i<n; i++) {
System.out.print(" " + Integer.toHexString(ff & buffer));
}
System.out.println();

/gordon

--

 

[Martin Gregorie]

Basic Idea
So I have this client-server project where the client uses Java's
Socket class to communicate with the server using JSON. Let's say that
I have a fake message that the client sends to the server and the
server closes the socket with the client (it doesn't send any response
to the client).
>

I think this is a poor design. The server should:
- wait for incoming connections from clients
- when a connection is received set up context (if needed)
for the client
- process requests from the client, generating at least one response
per request
- when the connection closes discard any client-specific
context.
- wait for the next connection.

The client should:
- open a connection to the server
- send requests to the server
- read server response(s) to each request
- close the connection when it's done.

IMO the server should NEVER intentionally close a connection: as you've
seen this can cause problems. The method I outlined is much cleaner:
- the client knows when it has finished talking to the server and so
can close the connection.
- in this scheme any connection closures seen by the client are
ALWAYS an error.
- the logic of this scheme means that the server will be waiting for
a new request from the client when the connection is closed and so
will be ready to handle it or close the connection without needing
to disentangle incomplete processing.
- designing the protocol so that every client request generates at
least one server response makes error checking easy (the client
always gets a response or sees the connection close due to an error.
If a simple ACK response is short, the overheads are minimal.

If you design the protocol so that the messages contain text then use of
a packet sniffer is a lot easier. If you add debugging code that prints
all messages sent and received then you don't need a packet sniffer and
debugging processrocess connections within a single computer is
simple. Specifying the protocol in terms of formatted records and using
record buffers to handle the messages rather than raw
streams also simplifies the protocol logic and helps a lot with
debugging. I normally design messages as length-delimited records
containing comma separated fields, but ymmv.

Lastly, the read/write loop on your client is probably a bad idea.
Unless your client is quite unusual this just adds complexity without
improving throughput. It also chews up CPU with its polling loop. To me
it smacks of inappropriate optimization: queues and scan loops should
only be introduced if monitoring code in the client shows that simple
"write - wait for response - read" logic is positively identified as a
bottleneck.

When the client first sends the message the server immediately closes
the socket (I also terminate the server to make sure).
>

Why? A server should be written to service multiple clients which can
connect and disconnect while the server continues to run. If you want to
stop it, use a dead simple client that connects, sends STOP, waits for
OK and then disconnects. Besides, such a client is often useful for
seeing what the server is doing, getting statistics, etc.

I heard that the way we wrote it isn't the problem.
>

Disagree! How many client/server designs had your adviser implemented
successfully?

We think the
problem relies on the implementation of sockets in Java or Windows.
>

Sockets work fine for both C and Java if your message exchange protocol
is a clean design.

 

[Lew]

false, the javadocs [sic] specifically say that both read() and read(byte[]) block.

read(buffer) adds bytes to the buffer. Two questions:
what is -1 in bytes..?

What does that question even mean?

It's not relevant to InputStream.read().

You don't put the -1 in the buffer. You use it to end the loop. Just like
you don't ever use the value returned by the method in the buffer, even when
greater than or equal to zero. The return value of read() is not the data
read, but the amount of data read, or -1 if the socket has closed. As others
have stated. As the Javadocs state.

-1 means the socket has been closed on the other side.
i [sic] dont [sic] think a 0 is ever returned.

btw, all of this is in the javadocs [sic].
http://java.sun.com/j2se/1.5.0/docs/api/java/io/InputStream.html

Make that
<http://java.sun.com/javase/6/docs/api/java/io/InputStream.html>

You might as well be current.

Do read one or the other of those links. Both links answer all the questions
you've been asking.

 

[Lew]

To make things easier here is the code of the method run of the
looping thread :

Please do not embed TABs in Usenet source listings.

 

[Lew]

To make things easier here is the code of the method run of the
looping thread :


StringBuffer outgoingBuffer;

StringBuilder is the preferred class these days.

List<Message> incomming;

Consider spelling the variable the same as the highly similar natural-language
word to avoid later maintenance errors.

while (running) {
try {

int messageCount = outgoing.size();
outgoingBuffer = new StringBuffer();
for (int i=0;i<messageCount;i++) {
outgoingBuffer.append( new
String(parser.serialize(outgoing.poll())));
}
out.write(outgoingBuffer.toString().getBytes());

int availableBytes = in.available();
byte[] toDeserialize = new byte[availableBytes];

You don't really need to allocate a new buffer each time. You can re-use a
single buffer.

in.read(toDeserialize);

Others have told you of the value of checking the return value of read() to
make sure that you have received any data at all, and if so, all that you expect.

You should do that. If it returns -1, the socket is closed.

incomming = parser.deserialize( toDeserialize );

But you have no certainty that the toDeserialize has everything you want in
it, or anything at all!

You need to check the return value of read().