How to crawl a web page/file in a .net application

Crawling refers to request the web page data and get the reponse data programmatically. You can easily do this in .net. Here is the daily code for this

// Be sure to include these namespaces
using System.Net;
using System.IO;
...
...
WebRequest request = WebRequest.Create("http://www.dailycoding.com/");
using (WebResponse response = request.GetResponse())
{
  using (StreamReader responseReader = 
    new StreamReader(response.GetResponseStream()))
  {
    string responseData = responseReader.ReadToEnd();
    using (StreamWriter writer =
      new StreamWriter(@"C:\\DailyCoding.html"))
    {
      writer.Write(responseData);
    }
  }
}

7 comment(S)


himanshu on Nov 11, 2008 09:57 AM

Hey Thank You so much, gr8 help.

himanshu on Nov 11, 2008 09:57 AM

Hey Thank You so much, gr8 help.

himanshu on Nov 11, 2008 09:57 AM

Hey Thank You so much, gr8 help.

Visual C# Kicks on Jan 13, 2009 12:16 AM

Something similar but data is downloaded in chunks (makes it possible to determine download speed and such):

http://www.vcskicks.com/download_file_http.html

pradeep on Apr 27, 2012 12:07 AM

using System.Net;
using System.IO;
...
...
WebRequest request = WebRequest.Create("http://www.dailycoding.com/");
using (WebResponse response = request.GetResponse())
{
using (StreamReader responseReader =
new StreamReader(response.GetResponseStream()))
{
string responseData = responseReader.ReadToEnd();
using (StreamWriter writer =
new StreamWriter(@"C:\\DailyCoding.html"))
{
writer.Write(responseData);
}
}
}

pradeep on Apr 27, 2012 12:09 AM

using System.Net;
using System.IO;
...
...
WebRequest request = WebRequest.Create("http://www.dailycoding.com/");
using (WebResponse response = request.GetResponse())
{
using (StreamReader responseReader =
new StreamReader(response.GetResponseStream()))
{
string responseData = responseReader.ReadToEnd();
using (StreamWriter writer =
new StreamWriter(@"C:\\DailyCoding.html"))
{
writer.Write(responseData);
}
}
}
can anyone explain me this code

Deependra Kushwaha on May 29, 2013 04:45 AM

Hi,

it is helpful,but can you please describe in more details.

Leave a comment