Crawling refers to request the web page data and get the reponse data programmatically.
You can easily do this in .net. Here is the daily code for this
// Be sure to include these namespaces
WebRequest request = WebRequest.Create("http://www.dailycoding.com/");
using (WebResponse response = request.GetResponse())
using (StreamReader responseReader =
string responseData = responseReader.ReadToEnd();
using (StreamWriter writer =