HttpWebRequest/Response and executionTimeout

R

Rich

I created an asp.net code to scrape data off a website. It calls many
pages on a website.

Coded like this:

for(int i=0;i<2000;i++)
{
HttpWebRequest myHttpWebRequest =
(HttpWebRequest)WebRequest.Create(websiteUrl+pageUrl);
HttpWebResponse myHttpWebResponse =
(HttpWebResponse)myHttpWebRequest.GetResponse();
using (StreamReader sr = new
StreamReader(myHttpWebResponse.GetResponseStream()))
{
strResponse = sr.ReadToEnd();
}
myHttpWebResponse.Close();
}


I encountered a Request timed out error:

System.Web.HttpException: Request timed out.

Exception message: Thread was being aborted.
Method that threw the exception: System.Net.Sockets.SocketError
setsockopt(System.Net.SafeCloseSocket,
System.Net.Sockets.SocketOptionLevel,
System.Net.Sockets.SocketOptionName, Int32 ByRef, Int32)


The timer that used the executionTimeOut value specified in web.config
is not reset after each loop. I cannot set a executionTimeOut value
because I do not have knowledge of how many calls to be made to the
website.
Does anyone know how to solve this?
Possibly force a timer reset or close socket or find the thread that
used the socket and kill it? How to do it?

Thanks in advance,

Rich
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,995
Messages
2,570,226
Members
46,815
Latest member
treekmostly22

Latest Threads

Top