Mechanize Traverse & Parse - mechanize-ruby

Question

I'm trying to walk through a hitlist of pages found by Mechanize. Upon a search, which is working just fine, I get a hitlist with 10 records per page. A bottom navigation system takes me to the record count > 10. Display 10 per page. So 53 records = 6 "group pages" as I'm calling them.
What I want to do is use the top search results page to do the following:
Grab the html behind every record on this list. I can do that through an iteration.
Follow through the '[Next]' link at the bottom and repeat both 1 and 2 until there is no more 2. Essentially this will get every record.
I'm having an issue with moving off the first page into the second page. I'm grabbing the html behind the first 10 records, but then the system bonks on me. Not sure why. I thought I was iterating through the group pages, but it isn't advancing past the first group page.
counter = 1
puts "Counter: #{counter}"
while agent.page.links_with(:text => '[Next]').count == 1
page = agent.page.link_with(:text => '[Next]').click
puts "Counter/Next: #{counter} / #{agent.page.links_with(:text => '[Next]').count}"
agent.page.links_with(text: '[complete profile]').each do |link|
a = link.click # link.click goes to each company page
r = a.body.to_s
r = r.gsub(/^\s+\n/, "")
# enter company into db
class Company < ActiveRecord::Base
end
company = Company.new
company.cdate = DateTime.now
company.status = 'new'
company.requestID = req_id
company.html = r
company.save
end
counter += 1
end
Any insight appreciated. I know I'm close.
Cheers

Answer 1

Solution: move the click method behind page:
page = agent.click(page.link_with(:text => "[Next]"))