Using HttpWebRequest to post form data with cookies

[Luis Esteban Valencia Muñoz]

I'm working on a web scraping application that needs to log into a website
before it can get the data to scrape. I've always been confused about how
the HttpWebRequest and HttpWebResponse objects work together with cookies,
and was hoping that someone here could clear it up for me!

Here's the steps I need to accomplish:

1) Set two cookies containing information I already have
2) Request the login page URL, and save a third cookie that is given to me
from that page
3) Submit a form via POST, along with the 3 cookies
4) Save a fourth cookie that is set after the form is submitted with valid
information

I don't need the entire code to do this, but rather some help with the order
to do it. Do I create a request and THEN set the cookies? Can I use the
same CookieContainer for each request? If someone could point me in the
right direction it would be greatly appreciated!

 

[Joerg Jooss]

I'm working on a web scraping application that needs to log into a
website before it can get the data to scrape. I've always been
confused about how the HttpWebRequest and HttpWebResponse objects
work together with cookies, and was hoping that someone here could
clear it up for me!

Here's the steps I need to accomplish:

1) Set two cookies containing information I already have
2) Request the login page URL, and save a third cookie that is given
to me from that page
3) Submit a form via POST, along with the 3 cookies
4) Save a fourth cookie that is set after the form is submitted with
valid information

I don't need the entire code to do this, but rather some help with
the order to do it. Do I create a request and THEN set the cookies?
Can I use the same CookieContainer for each request? If someone
could point me in the right direction it would be greatly appreciated!

This topic has been beaten to death by now -- just search for
WebRequest and CookieContainer on Google: http://tinyurl.com/9yp6r

Just make sure to use one CookieContainer instance throughout your web
conversation. It will pick up all cookies, and you don't need to copy
cookies from responses back to it -- the CookieContainer picks them up
automatically.

You can set any request header (including cookies) as long as you
haven't written to the request stream or read from the response stream
yet.

Cheers,

 

[Buddy Ackerman]

This example might be helpful:
http://msdn.microsoft.com/library/d...etHttpWebRequestClassCookieContainerTopic.asp

The key is that you need to create and assign a cookiecontainer object to
the request object's CookieContainer property first. Then and only then
will you receive the response cookies.


--Buddy