Scrape information from asp.net gridview with paging - asp.net

Question

I'm trying to scrape some schedules off of a website. the information is displayed in a GridView with paging.
The url is:
http://www.landmarkworldwide.com/when-and-where/register/search-results.aspx?prgid=0&pgID=270&crid=0&ctid=&sdt=0
My Issue is when I want to scrape pages other then #1 in the grid view.
The best post I found so far was This One, but it doesn't work and that topic is not complete. I tried to use Fiddler and Chrome to get the post data and use it, but I can't get it to work for me. Can you guys see what's missing?
Here's the code I am using. it's in VB, but you can answer in C# and I'll translate -) (sorry)
Protected Sub Page_Load(sender As Object, e As System.EventArgs) Handles Me.Load
Dim lcUrl As String = "http://www.landmarkworldwide.com/when-and-where/register/search-results.aspx?prgid=0&pgID=270&crid=0&ctid=&sdt=0"
' first, request the login form to get the viewstate value
Dim webRequest__1 As HttpWebRequest = TryCast(WebRequest.Create(lcUrl), HttpWebRequest)
Dim responseReader As New StreamReader(webRequest__1.GetResponse().GetResponseStream())
Dim responseData As String = responseReader.ReadToEnd()
responseReader.Close()
' extract the viewstate value and build out POST data
Dim viewState As String = ExtractViewState(responseData)
Dim loHttp As HttpWebRequest = DirectCast(WebRequest.Create(lcUrl), HttpWebRequest)
' *** Send any POST data
Dim lcPostData As String = [String].Format("__VIEWSTATE={0}&__EVENTTARGET={1}&__EVENTARGUMENT={2}", viewState, HttpUtility.UrlEncode("contentwrapper_0$maincontent_0$maincontentfullwidth_0$ucSearchResults$gvPrograms"), HttpUtility.UrlEncode("Page$3"))
loHttp.Method = "POST"
Dim lbPostBuffer As Byte() = System.Text.Encoding.GetEncoding(1252).GetBytes(lcPostData)
loHttp.ContentLength = lbPostBuffer.Length
Dim loPostData As Stream = loHttp.GetRequestStream()
loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length)
loPostData.Close()
Dim loWebResponse As HttpWebResponse = DirectCast(loHttp.GetResponse(), HttpWebResponse)
Dim enc As Encoding = System.Text.Encoding.GetEncoding(1252)
Dim loResponseStream As New StreamReader(loWebResponse.GetResponseStream(), enc)
Dim lcHtml As String = loResponseStream.ReadToEnd()
loWebResponse.Close()
loResponseStream.Close()
Response.Write(lcHtml)
End Sub
Private Function ExtractViewState(s As String) As String
Dim viewStateNameDelimiter As String = "__VIEWSTATE"
Dim valueDelimiter As String = "value="""
Dim viewStateNamePosition As Integer = s.IndexOf(viewStateNameDelimiter)
Dim viewStateValuePosition As Integer = s.IndexOf(valueDelimiter, viewStateNamePosition)
Dim viewStateStartPosition As Integer = viewStateValuePosition + valueDelimiter.Length
Dim viewStateEndPosition As Integer = s.IndexOf("""", viewStateStartPosition)
Return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition))
End Function

Answer 1

To make it work you need to send all input fields to the page, not only viewstate. Other critical data is the __EVENTVALIDATION for example that you do not handle it. So:
First you need to make scrape on the #1 page. So load it and use the Html Agility Pack to convert it to a usable struct.
Then extract from that struct the input data that you need to post. From this answer HTML Agility Pack get all input fields here is a code sniped on how you can do that.
foreach (HtmlNode input in doc.DocumentNode.SelectNodes("//input"))
{
// use this to create the post string
// input.Attributes["value"];
}
Then when you have the post data that is needed to be a valid post, you move to the next step. Here is an example How to pass POST parameters to ASP.Net web request?
You can also read: How to use HTML Agility pack