I searched google for a good number of hours. Maybe I searched for the wrong keywords.
Here is what I want to do.
I'm posting data to a website which then makes a HTTP POST request and returns a .CGI webpage. I want to know the parameters the web page uses to send that HTTP POST request so that I can directly link a page from my Webpage to the final .CGI webpage by making the user enter the data on my own webpage.
How do I achieve it?
Usually the POST body is piped into STDIN, just read it as a normal file
Related
I received the task to make some kind of redirection or forwarding to an url containing a form.
The thing is:
The form will be autofilled with the info that i send to it in the request body as a json object.
I'm not supossed to call post with a form formatted body because the form has more inputs that will be requested during the visit to that form.
When I test that url through an api client (for convenience on testing the endpoint) the webpage text is correctly downloaded although I must do this through a proper browser.
The techs through I must work this are the tagged ones (spring,jsp or thymeleaf) but I would appreciate generic advice.
UPDATE:
I've been reading about general GET and POST redirects.
Spring, redirect to external url using POST
https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect
RequestDispatcher.forward() vs HttpServletResponse.sendRedirect()
I have a server log and it shows POST and GET
So, if a page is showing POST /ping and GET /xyz
Does this mean that the user agent is Requesting a page is GET and POST is the response from the server?
Because in my server logs, it's showing a lot of POST with million of /ping while the other pages have been GET is a smaller amount of number.
Which should I focus on? Get the POST pages get index if the server shows this to Search engines?
I would suggest you learn the difference between HTTP GET and POSTS.
This answer is quite good.
In summary, the GET requests are pages/data being requested by clients. POSTs are clients sending data to the server, usually expecting data as a response.
In their comment, Sylwit pretty much explains what this has to do with search engines. I'm going to just describe the differences between GET and POST
GET and POST are two different types of requests to the server. A GET request is normally used to retrieve information from the server and usually has a series of GET parameters. When you search something on Google you're making a GET request.
https://google.com/?q="how+do+i+get"
In this case, the GET parameter is the q after the ?, and has a value of "how do i get". It should be noted that a GET request doesn't need these additional parameters (http://google.com) is still a GET request
POST requests, on the other hand, are normally used to send data to the server. You'll see this anytime you send a message, submit a form etc. When I click submit on this answer, I'll be making a POST request to stackoverflow's servers. The parameters for these aren't immediately visible in the browser. POST requests can also return a HTTP response, with a message.
Hope that shows the differences between the two.
I am developing a web scraper and I need to download a .pdf file from a page. I can get the file name from the html tag, but can't find the complete url (or request body) that downloads the file.
I have tried to sniff the traffic with the chrome and firefox network traffic tool and with wireshark, with no success. I can see it make a post request to the exact same url as the page itself, and so I can't understand why this happens. My guess is that the filename is being sent inside the POST request body, but I also can't find that information in those tools. If I could see the variable name in the body, I could create a copy of the request and then get the file.
How can I get that information?
Here is the website I am talking about: http://www2.trt8.jus.br/consultaprocesso/formulario/ProcessoConjulgado.aspx?sDsTelaOrigem=ListarProcessos.aspx&iNrInstancia=1&sFlTipo=T&iNrProcessoVaraUnica=126&iNrProcessoUnica=1267&iNrProcessoAnoUnica=2010&iNrRegiaoUnica=8&iNrJusticaUnica=5&iNrDigitoUnica=24&iNrProcesso=1267&iNrProcessoAno=2010&iNrProcesso2a=0&iNrProcessoAno2a=0
EDIT: for those seeking to do something similar, take a look at this website: http://curl.trillworks.com/
It converts a cURL to a python requests code. Very useful
The POST data used for the request is encoded content generated by ASP.NET. It contains various state/session information of the page that the link is on. This makes it difficult to directly scrape for the URL.
You can examine the HAR by exporting it from the Network tab in Chrome DevTools:
The __EVENTVALIDATION data is used to ensure events raised on the client originate from the controls rendered on the page from the server.
You might be able to achieve what you want by requesting the page the link is on first, then extract the required POST data from the response (containing the page state and embedded request for file), and then make a new request with this information. This assumes the server doesn't expire any sessions in the meantime.
When I send a one-off document to RightSignature via their API, I'm specifying a callback location in the XML document as specified in RightSignature's schema definition. I then get a signer-link value back from their API for the document. I display the HTML response from the signer-link URL in an iFrame on our website. When our user signs the document in this iFrame, which is rendering the responses from their website, I want their website to post to our callback location.
Can I do this with the RightSignature API and does it make sense?
So far, I'm only getting content in the iFrame that indicates that the signing was successful. The callback location does not seem to be getting called.
I got it solved just now. Basically, i was doing two things wrong first you have to go in RightSignature Account and set it there the CallBack url
Account > Settings > Advanced Settings
But the thing which RS is unable to mention to us that this url can not be of localhost, but it should be of https i mean like Live URL of your site like
https://stagingmysite.azurewebsites.net/User/CallBackFunction
And then in your CallBack just write these two lines and you will receive complete XML which would have the GUID and document status as well.
byte[] data = Request.BinaryRead(Request.TotalBytes);
string callBackXML = System.Text.Encoding.UTF8.GetString(data);
I found the answer with some help from the API team at RightSignature. I was using callback_location but what I really wanted is redirect_location. Their online documentation was difficult to follow and did not clearly point out the difference.
I got this working after a lot of trial and error.
I'm trying to manipulate a .net ASP form on a site that's using AJAX Control Toolkit. The site is only accessible to valid logins, and I do have a valid account. It consists of a search page with a form. Each time a submit button is clicked on the form, the server is updated using the values of some text fields on the form, and then the VIEWSTATE and EVENTVALIDATION tokens will be updated based on the response from the server, ready for the next request.
I'm using HttpClient in Java to do this. I suspect there's something I'm not doing correctly with regard to interacting with ASPX forms in general.
When I hit the main search page for the first time (cookies are validating my login with the server), I get the HTML for the search page back. I extract the VIEWSTATE and EVENTVALIDATION tokens for the next request. I've examined the exact form fields and their values that need to be sent to the server in a POST by looking at the Chrome debugger utility after making a request on the site manually. I've replicated them exactly as they should be, inserting the VIEWSTATE and EVENTVALIDATION appropriately.
But the response I get back from the server is not what it should be. What I get back is just the same HTML for the main search page that I get the first time I hit the webpage. The form data I'm using looks like this:
ctl00$ScriptManager1:ctl00$ContentPlaceHolder1$UpdatePanel1|ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$acceptButton
ctl00_ContentPlaceHolder1_TabContainer1_ClientState:{"ActiveTabIndex":0,"TabState":[true,true]}
__EVENTTARGET:
__EVENTARGUMENT:
__LASTFOCUS:
__VIEWSTATE:<token extracted from first page hit>
__VIEWSTATEENCRYPTED:
__EVENTVALIDATION:<token extracted from first page hit>
ctl00$ContentPlaceHolder1$LabelFee:0
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$RadioButtonList1:Person
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$snameText:aSurname
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$HiddenField1:
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$fnameText:aFirstname
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$dayFromTextBox:01
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$monthFromTextBox:January
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$yearFromTextBox:2001
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$dayToTextBox:01
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$monthToTextBox:January
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$yearToTextBox:2008
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$DropDownList1:aCity
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$PropText:
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel2$RefText:
__ASYNCPOST:true
ctl00$ContentPlaceHolder1$TabContainer1$TabPanel1$acceptButton:Accept
I've also tried replicating the headers that the Chrome debugger shows, so my request is including the same Content-Type, Host, Origin, Referer, User-Agent (for my browser) and every other header, including this header X-MicrosoftAjax: Delta=true.
I know there's a lot of moving parts here, but I intentionally haven't mentioned how I'm actually making the POST request with the HttpClient lib because I'd don't want to complicate the question anymore or alienate anyone who doesn't know Java but knows ASP. I'd like to know if there's an ASP issue I'm not addressing, but I can post the Java code is necessary.
Edit:
I've checked the debugging info that HttpClient is outputting just before sending the request, and the form data is being added properly as multi-part form data. The headers are all there too.
This answer is a long shot, but I've seen weirder things.
You mention this header:
X-MicrosoftAjax: Delta=true
I did some deep googling and found that this is often shown as all lower case in dumps of Ajax and UpdatePanel POST requests:
x-microsoftajax: Delta=true
See here and here.
Could it be as simple as not casing the header correctly?
I eventually got this working. The problem was not specific to ASP in general, it was actually a problem with how Java (specifically HttpClient) was sending the request. I was using HttpClient to compile the request using multi-part form, but after using Fiddler to analyse and compare the requests (see the edited part of this question for more details on that) sent from both my application and the actual webpage, my app request was structured very differently.
The real website request had the form options embedded in the request body in what looked like a URL encoded query string. My request was a series of entries in the request body where each option was wrapped in the Content-Type and Content-Disposition headers. The requests succeeded after changing the POST to add the parameters like:
request.setEntity(new UrlEncodedFormEntity(paramList));