I am scraping an ASP.net page with Scrapy which displays a paginated list of items. To navigate through the items, there are next and previous buttons in the following format:
<form name="aspnetForm" method="POST" action="search_active_main.aspx" id="aspnetForm">
<!-- other content here -->
<!-- previous button -->
<input type="image" name="ctl00$gvMain$ctl01$btnPrevious" id="ctl00_gvMain_ctl01_btnPrevious" src="/image/previous.gif" />
<!-- next button -->
<input type="image" name="ctl00$gvMain$ctl01$btnNext" id="ctl00_gvMain_ctl01_btnNext" src="/image/next.gif" />
<!-- other content here -->
</form>
When you click one of the buttons, something like this is sent as part of the POST:
ctl00$gvMain$ctl01$btnNext.x:37
ctl00$gvMain$ctl01$btnNext.y:10
What do these numbers represent / how can I crawl through them without using something like Selenium?
As Obsidian Phoenix suggests, the numbers represent the coordinates of the button that is clicked. To crawl the page, you just have to POST the following as formdata in FormRequest to simulate a next button click:
FormRequest.from_response(
response,
formdata={
'ctl00$gvMain$ctl01$btnNext.x':'1'
'ctl00$gvMain$ctl01$btnNext.y':'1'
},
dont_click=True,
dont_filter=True,
callback=self.your_callback_function
)
One thing to investigate is the URL that is present on each page. You may find subsequent pages have /2 /3 etc in their URL but otherwise be identical.
If that's the case, then you can bypass the need to click any buttons and just reload the page with the new page number.
Related
Our company uses PinPointe for email marketing and we have a Drupal 6 site with several language domains. I have created a web form except I did not create any fields in Drupal. Instead in the node Edit NOT THE NODE WEBFORM EDIT....in the node edit for the body section I added the HTML and the javascript for form. Everything works well and the data is captured to pinpointe. The problem lies in the fact that the page..upon clicking submit..actually redirects to PinPointe where I get a friendly message saying. Thanks for joining. Well I don't want this. I would like to just pop an alert saying thanks and leave the user on the page they were on. I tried this code for using jquery to do the post but it isn't loading and I suspect that's because I need it in the header not the body.
So all I want is to submit the data to pinpointe and not redirect the user. So here is where my limited Drupal knowledge runs out:
If I create the input fields in the node webform then how do I get the form to post to pinpointe?
If I create the fields dynamically in the node body (not node>>webform) I can direct the submission to PinPointe but then how do I stop the redirect?
FWIW here is the jquery I was trying to use but suspect has to go in the header http://jsfiddle.net/4xDFK/4/
FWIW here is the code for the dynamic creation:
<form action="http://na04.mypinpointe.com/...." id="webform-client-form-1375" method="post" onsubmit="return CheckForm257(this);">
<div>
<div id="webform-component-UsrEmail">
<div id="edit-submitted-UsrEmail-wrapper">
<input id="edit-submitted-UsrEmail" name="email" size="30" type="email" />
</div>
</div>
<div class="form-actions form-wrapper" id="edit-actions">
<input class="form-submit" id="edit-submit" name="op" type="submit" value=" " />
</div>
</div>
</form>
Create your custom confirmation page in Drupal. Then set up PinPointe to redirect to this confirmation page.
In the PinPointe form manager, there is a option under the 'Thank you page options' to send a signup user to a custom URL.
I'm working on making some changes to a Dot Net Nuke website with a customized skin. I found out that the header to the skins file was located in 'Default.aspx' here.
The form has some very strange behavior. I have had to disable the enter button because pressing within the form causes the webpage to go to "/HOME.aspx" however that action is never specified within the Default.aspx.
The code is as follows.
<dnn:Form id="Form" runat="server" ENCTYPE="multipart/form-data" >
<asp:Label ID="SkinError" runat="server" CssClass="NormalRed" Visible="False"></asp:Label>
<asp:PlaceHolder ID="SkinPlaceHolder" runat="server" />
<input id="ScrollTop" runat="server" name="ScrollTop" type="hidden" />
<input id="__dnnVariable" runat="server" name="__dnnVariable" type="hidden" />
</dnn:Form>
The form after being processed displays in the browser as.
<form name="Form" method="post" action="/HOME.aspx" onsubmit="javascript:return WebForm_OnSubmit();" id="Form" enctype="multipart/form-data">
What I want the code to display as is simply.
<form name="Form" method="get" action="/SearchResults.aspx" id="Form">
I tried removing the dnn code with the html directly but removing the dnn form causes the website to crash.
EDIT
What I'm trying to do can be seen at http://www.ontariosheep.org
Notice if you press the button the search works but pressing enter causes the page to refresh.
You can use some Javascript to do this:
jQuery('#SearchBox').keypress(function(e){
if(e.which == 13){
e.preventDefault();CallSearchPage('http://www.ontariosheep.org/SearchResults.aspx');
}
});
You would need to put that in script tags and also in a jQuery document ready area... like
<script>
jQuery(document).ready(function(){
//code above here
});
</script>
Changing the behavior of the form in DNN is not something you are going to do easily. DNN uses the ASP.NET Web Forms model, so the action for the page is always the current page.
If you want to customize this the only real way is to modify the form action via JavaScript on a specific page, but note that doing that prior to a button click or similar trigger WILL break all administration functions on the page that require a postback to the server.
What are you trying to accomplish?
Good morning, I am trying and failing miserably to implement a captcha, mostly because I have no asp experience, could anyone assist?
Multiple html pages have a contact form in footer, this has form validation in place and fires off an asp page which sends an email & redirects to thanku.asp.
I wanted to add the honeypot captcha, so added another field [body] and hid it using css. I then added the code below that to check its null. When I click submit the form is processed even when hidden field has content inside, can anyone spot where I am going wrong?
<div id="captchafield">
<input type="text" name="body" value="" />
</div>
<script language=javascript>
if(!String.IsNullOrEmpty(Request.Form["body"]))
IgnoreComment();
</script>
You're going wrong by checking the field with only JavaScript; you should check that it's blank server-side and if not, then ignore / reject submission. E.g.
HTML
<div id="captchafield">
<noscript>Security field; please leave this blank</noscript>
<input type="text" name="captcha" value="" />
</div>
ASP
Dim captcha
captcha = Request.Form("captcha")
If captcha <> "" Then
Response.Redirect("badcontent.asp")
End If
' continue
What you can use JavaScript for is to hide the captcha field and / or display a message which lets users know not to fill it in if their JavaScript is off (see <noscript> in example code above). Bots / scripts will ignore the warning and fill in the field and trigger your trap.
I need to display a submission form inside a lightbox of an ASP.NET page.
If there is an error in the submission form such as user name not being unique, the postback then renders the ASP.NET page outside the lightbox.
How can I solve that problem?
Here is a code snippet of the .aspx page that includes the lightbox:
<...>
<p>
QunatumMotors is located in Detroit. Please use the link below to contact us.</p>
<p>
<!--START CONTACT FORM OVERLAY-->
<!-- first overlay. id attribute matches the selector -->
<a href="**../informational/contactform.aspx"** rel="#overlay" style="text-decoration:none">
> Click here to contact us
</a>
<div class="simple_overlay" id="form_contact">
<!-- overlayed element -->
<div class="apple_overlay" id="overlay">
<!-- the external content is loaded inside this tag -->
<div class="contentWrap"></div>
</div>
</div>
<!--END CONTACT FORM OVERLAY-->
<p> </p><p> </p>
<...>
contactform.aspx is just a standard .aspx page with form fields, field validators, label to display errors (e.g. username not unique) and submit button.
When a postback occurs on contactform.aspx then (of course) it is rendered outside the lightbox. How can I display the postback of contactform.aspx inside the lightbox?
Thanks for your help!
The lightbox content isn't like another tab or window: it becomes part of the host page.
In other words, it incorporates the HTML generated by contactform.aspx into the host page's document object model (DOM). Activating the lightbox adds a form to the host page that posts to the contact page:
<html><body>
<!-- host page content here -->
<!-- contact form content -->
<form action="contactform.aspx">
<!-- text boxes, buttons, etc. -->
</form>
</body></html>
When the user submits, their browser issues a new request: a POST to contactform.aspx. That request returns the contact form's HTML.
There are several ways you might work around that:
Use ajax to perform the update asynchronously. (You might even be able to do this by using an UpdatePanel in contactform.aspx, but I don't use them much anymore and haven't though that through).
Convert contactform.aspx into a control (if you're using it in several places) and embed it in the host page.
At the end of the contact form's submit handler, redirect to the host page with a flag that instructs the page to immediately activate the lightbox. (This has numerous issues and sounds pretty fragile ... but it's plausible.)
I have a webpage, in that page I have a button. How can I refresh the page when clicking on that button?
<input type="button" value="Reload" onclick="window.location.reload(true);" />
However, if the page is created from a postback, you would need to use an asp:Button control instead, and let another postback refresh the page. You also have to make sure that the correct code is executed in the code behind to recreate the correct result.
are you using an
<asp:button />
tag?
If so the page should refresh when the button is clicked by default.
<form>
<input TYPE="submit" VALUE="Submit Information Now" />
</form>
SUBMIT is a TYPE attribute value to the INPUT element for FORMs. It specifies a button that, when activated, submits the information entered to a processing script. If there are multiple SUBMIT buttons in a form, only the one activated should be sent to the form processing script.
When you press button your page must refresh, only not refresh that controls which are in AJAX Update Panel