Crawling refers to request the web page data and get the reponse data programmatically. You can easily do this in .net. Here is the daily code for this
// Be sure to include these namespaces using System.Net; using System.IO; ... ... WebRequest request = WebRequest.Create("http://www.dailycoding.com/"); using (WebResponse response = request.GetResponse()) { using (StreamReader responseReader = new StreamReader(response.GetResponseStream())) { string responseData = responseReader.ReadToEnd(); using (StreamWriter writer = new StreamWriter(@"C:\DailyCoding.html")) { writer.Write(responseData); } } }
Hey Thank You so much, gr8 help.
Something similar but data is downloaded in chunks (makes it possible to determine download speed and such):http://www.vcskicks.com/download_file_http.html