Lost data when reading socket

[Mark Bratcher]

I have a java client program that opens a socket to a web site (port 80)
and reads binary files from the site. Most of the time this works fine.

However, in some cases, I don't get the first 2K bytes (exactly that mount)
of the file. Even though the file is in the same directory structure as
the rest of the files. I can download the file fine using any browser's
file download capability. But my Java program won't see the first 2K bytes.
It works on most other files, just not on a few.

The test program I wrote looks like this:


mport java.net.*;
import java.io.*;
import java.util.*;

public class DumpData extends Object
{
public DumpData( String site, String filePath ) throws IOException
{
Socket sockSite;

try
{
sockSite = new Socket( site, 80 );
}
catch ( IOException e )
{
if ( e.getMessage() == null )
System.out.println("Unable to open " + site );
else
System.out.println("Unable to open " + site + ": " + e.getMessage());
return;
}

BufferedReader in = new BufferedReader( new InputStreamReader(sockSite.getInputStream()) );

BufferedWriter out = new BufferedWriter( new OutputStreamWriter(sockSite.getOutputStream()) );
out.write( "GET " + filePath + "\n\n" );
out.flush();

int c;
int count = 0;

while ( (c = in.read()) >= 0 )
{
count = count + 1;

System.out.print( Integer.toHexString(c) + " " );

if ( (count & 0xF) == 0 )
{
System.out.println( "" );
count = 0;
}
}

in.close();
out.close();
}
}

Any thoughts on this?

Thanks.
Mark

 

[Karel Suikers]

I have a java client program that opens a socket to a web site (port 80)
and reads binary files from the site. Most of the time this works fine.

However, in some cases, I don't get the first 2K bytes (exactly that mount)
of the file.

Any thoughts on this?

I also discovered the problem (but with BufferedInputStream) and changed
my loop to:

BufferedInputStream in = Blabla
byte[] b = new byte[1024];
int red = 1;
while (red > 0) {
red = in.read(b);
if (red > 0) { do anything }
}

 

[DP]

Im guessing your connecting to a UDP socket on the other end so just
continue to read until you get an error.

 

[Mark Bratcher]

I'm reading normally, without errors. The first data I get is data that's already
2K within the file (skips past the first 2K bytes), so not sure how reading until
error would get me that first 2K in the right order

 

[Steve Horsley]

I have a java client program that opens a socket to a web site (port 80)
and reads binary files from the site. Most of the time this works fine.

However, in some cases, I don't get the first 2K bytes (exactly that mount)
of the file. Even though the file is in the same directory structure as
the rest of the files. I can download the file fine using any browser's
file download capability. But my Java program won't see the first 2K bytes.
It works on most other files, just not on a few.

The test program I wrote looks like this:

<snip>

I know it's not much help, but as a sanity check, I can't see anything
wrong with your code. Can I suggest that you try tracing the connection
with Ethereal (free protocol analyser) and check that the server is
actually sending it all? I wonder if your rather minimalist HTTP GET
header is upsetting the server.

Steve

 

[Mark Bratcher]

I know it's not much help, but as a sanity check, I can't see anything
wrong with your code. Can I suggest that you try tracing the connection
with Ethereal (free protocol analyser) and check that the server is
actually sending it all? I wonder if your rather minimalist HTTP GET
header is upsetting the server.

Steve

Thanks for the tip.

Another thing I tried was a more elaborate HTTP GET header. I appended
" HTTP/1.0" onto the GET command as I had seen in some examples. It behaved
even worse. No files came back intact, although they all came back with
more bytes out of the file than my initial failing case.

Mark

 

[Mark Bratcher]

Thanks for the tip.

Another thing I tried was a more elaborate HTTP GET header. I appended
" HTTP/1.0" onto the GET command as I had seen in some examples. It behaved
even worse. No files came back intact, although they all came back with
more bytes out of the file than my initial failing case.

Mark

My mistake: appending HTTP/1.0 appears to get the data but prepends a couple
hundred bytes of info in front of the file.

Mark

 

[Mark Bratcher]

My mistake: appending HTTP/1.0 appears to get the data but prepends a couple
hundred bytes of info in front of the file.

Mark

OK, so what I did was go ahead and put the HTTP/1.0 on the end of the GET
request, then peel off the header that came back with the data I wanted.
That seems to be working OK.

Must be something about the simple "GET file" approach that the server
doesn't always like (?). But the "GET file HTTP/1.0" is fine if I take
the returned header off.

Mark

 

[Esmond Pitt]

I have a java client program that opens a socket to a web site (port 80)
and reads binary files from the site.

If you are reading binary files you should be using Input/OutputStreams,
not Readers and Writers.

 

[Steve Horsley]

OK, so what I did was go ahead and put the HTTP/1.0 on the end of the GET
request, then peel off the header that came back with the data I wanted.
That seems to be working OK.

Must be something about the simple "GET file" approach that the server
doesn't always like (?). But the "GET file HTTP/1.0" is fine if I take
the returned header off.

Mark

Strange - I've never heard of a server doing that before - it really was just a guess. I'm glad you have it sorted.

Steve

 

[Lothar Kimmeringer]

However, in some cases, I don't get the first 2K bytes (exactly that mount)
of the file. Even though the file is in the same directory structure as
the rest of the files. I can download the file fine using any browser's
file download capability. But my Java program won't see the first 2K bytes.
It works on most other files, just not on a few.

The test program I wrote looks like this:

[Program using Socket]

Without looking into your program too much, what's wrong with
using HttpUrlConnection for getting the data?

URL url = new URL(size);
URLConnection conn = url.openConnection();
InputStream is = conn.getInputStream();

// read the data

alternatively you can call
InputStream is = url.openStream()

With HTTP 1.1 the server can answer with content-type "chunked"
where the data is sent in a "strange" way.

BTW:
out.write( "GET " + filePath + "\n\n" );
This is no correct HTTP you're talking, so the fact, that it's
working is a pure coincidence.


Regards, Lothar
--
Lothar Kimmeringer E-Mail: (e-mail address removed)
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)

Always remember: The answer is forty-two, there can only be wrong
questions!

 

[Thomas Weidenfeller]

My mistake: appending HTTP/1.0 appears to get the data but prepends a couple
hundred bytes of info in front of the file.

I think you badly want to read about the HTTP protocol. You are
completely butchering it. You also want to rethink the way you read the
data. You say it is binary, but then you wrap the read into Readers,
which are intended for text data only. Oh, and giving URLConnection and
its subclasses a good look is also a very good idea.

/Thomas