How to crawl a web page/file in a .net application

Crawling refers to request the web page data and get the reponse data programmatically. You can easily do this in .net. Here is the daily code for this

// Be sure to include these namespaces
using System.Net;
using System.IO;
...
...
WebRequest request = WebRequest.Create("http://www.dailycoding.com/");
using (WebResponse response = request.GetResponse())
{
  using (StreamReader responseReader = 
    new StreamReader(response.GetResponseStream()))
  {
    string responseData = responseReader.ReadToEnd();
    using (StreamWriter writer =
      new StreamWriter(@"C:\DailyCoding.html"))
    {
      writer.Write(responseData);
    }
  }
}

Discussion