How to read a web page using C#

Ever wondered How does search engine spider crawl web pages? On this article, I will try to show you How you can achieve similar task using a C# .NET. The following code will read a web page using a virtual path and get the response then store it locally.


First, include the following namespace to use the WebRequest and StreamReader classes.

using System.Net;
using System.IO;

Then use the following code to read or crawl a web page

WebRequest request = WebRequest.Create("http://www.fryan0911.com/2009/08/how-to-read-web-page-using-c.html");
using (WebResponse response = request.GetResponse())
{
  using (StreamReader responseReader = 
    new StreamReader(response.GetResponseStream()))
  {
    string responseData = responseReader.ReadToEnd();
    using (StreamWriter writer =
      new StreamWriter(@"C:\SaveResponse\sample.html"))
    {
      writer.Write(responseData);
    }
  }
}

For more C# coding tips & tricks, subscribe now.