I have a server log and it shows POST and GET
So, if a page is showing POST /ping and GET /xyz
Does this mean that the user agent is Requesting a page is GET and POST is the response from the server?
Because in my server logs, it's showing a lot of POST with million of /ping while the other pages have been GET is a smaller amount of number.
Which should I focus on? Get the POST pages get index if the server shows this to Search engines?
I would suggest you learn the difference between HTTP GET and POSTS.
This answer is quite good.
In summary, the GET requests are pages/data being requested by clients. POSTs are clients sending data to the server, usually expecting data as a response.
In their comment, Sylwit pretty much explains what this has to do with search engines. I'm going to just describe the differences between GET and POST
GET and POST are two different types of requests to the server. A GET request is normally used to retrieve information from the server and usually has a series of GET parameters. When you search something on Google you're making a GET request.
https://google.com/?q="how+do+i+get"
In this case, the GET parameter is the q after the ?, and has a value of "how do i get". It should be noted that a GET request doesn't need these additional parameters (http://google.com) is still a GET request
POST requests, on the other hand, are normally used to send data to the server. You'll see this anytime you send a message, submit a form etc. When I click submit on this answer, I'll be making a POST request to stackoverflow's servers. The parameters for these aren't immediately visible in the browser. POST requests can also return a HTTP response, with a message.
Hope that shows the differences between the two.
Related
I'm trying to fetch data from a website (https://gesetze.berlin.de/bsbe/search). Using Mozilla, I've taken a look at the network analysis. Usually, I'm just messing around with the parameters of the POST-Request to see how I might influence the response of the server. But when I simply re-send the request (making no changes at all), I'm getting HTTP-response 500. The server answer states as message: security_notAuthenticated.
Can anyone explain that behaviour? The request is done by the same PC, the same browser in the same session, and there is no login function on that website. Pictures shown below.
Picture 1 - Code 200
Picture 2 - Code 500
The response security_notAuthenticated indicates, that your way of repeating the request omits some authentication-related information.
When I repeat the request, using Mozilla Firefox's "Resend" or "Edit and resend" function, the Cookie header is not sent with the request. Although it occurs in the editable header list when using "Edit and resend" it's missing when watching the actual sent request. I'm not sure whether this is a feature or a bug.
When using Firefox's "Use as Fetch in Console" function, the header will automatically be included and you still have the ability to change the headers and the body. The fetch API is a web standard and some introductory material about fetch can be found on MDN.
If you want to do custom requests, in the browser, fetch is a good option.
In other environments and languages you usually use some HTTP client (just search the web for "...your language... http request" or similar, you will find something).
I've been reading a bit about post and get methods in http and I'm a bit confused
from the words themselves it sounds like GET is to get something/a resource from somewhere and POST is to send something somewhere
so I could do something like:
axios.get('https://jsonplaceholder.typicode.com/users) and I have received the list of users. that makes perfect sense
and I could do a POST request once a user has submitted my form to a database. easy
the bit that confuses me is when people say to use a POST instead of a GET. like how would I return the users from the url above using a POST? also the verbs sound weird if you are using POST to receive data??
also I read that GET requests can only take stuff over the url whereas POST has a response body. but in my example above with the users I got all the users in an array back to my app using GET and nothing changed in the url?
can someone help clarify this and explain how to use a POST when doing a GET
It really depends on the specific endpoint you are accessing. You shouldn't use a POST in lieu of a GET request if your goal is simply to read a list of users from the server. Also, if an endpoint is created as only accepting GET requests and you send a POST request then you will get a 405 "Method not allowed" in response. However, if the endpoint allows both POST and GET in order to read a list of users from the server then it depends on the use case for this POST method. The reasons can vary from security to specifying (in the body) the format in which you would like the data returned.
I'm trying to scrape some pages on a website that uses ASPX forms. The forms involve adding details of people by updating the server (one person at a time) and then proceeding to a results page that shows information regarding the specified people. There are 5 steps to the process:
Hit the login page (the site is HTTPS) by sending a POST request with my credentials. The response will contain cookies that will be used to validate all subsequent requests.
Hit the search criteria page by sending a GET request (no parameters). The only purpose of this is to discover the __VIEWSTATE and __EVENTVALIDATION tokens in the HTML response to be used in the next step.
Update the server with a person. This involves hitting the same webpage in step 2 but using a POST request with form parameters that correspond to the form controls on the page for adding person details and their values. The form parameters will include the __VIEWSTATE and __EVENTVALIDATION tokens gained from the previous step. The server response will include a new __VIEWSTATE and __EVENTVALIDATION. This step can be repeated using the new __VIEWSTATE and __EVENTVALIDATION, or can proceed to the next step.
Signal to the server that all people have been added. This involves hitting the same page as the previous 2 steps by sending a POST request with form parameters that correspond to the form controls on the page for signalling that all people have been added. The server response will simply be 25|pageRedirect||/path/to/results.aspx|.
Hit the search results page specified in the redirect response from the previous step by sending a GET request (no parameters - cookies are enough). The server response will be the HTML that I need to scrape.
If I follow the process manually with any browser, filling in the form controls and clicking the buttons etc. (testing with just one person) I get to the results page and the results are fine. If I do this programmatically from an application running on my machine, then ultimately the search results HTML is wrong (the page returns valid HTML, but there are no results compared with the browser version and some null values were there should not be).
I've run this using a Java application with Apache HttpClient handling the requests. I've also tried it using a Ruby script with Mechanize handling the requests. I've setup a proxy server using Charles to intercept and examine all 5 HTTPS requests. Using Charles, I've scrutinized the raw requests (headers and body) and made comparisons between requests made using a browser and requests made using the application(s). They are all identical (except for the VIEWSTATE / EVENTVALIDATION values and session cookie values, which I would expect to differ).
A few additional points about the programmatic attempts:
The login step returns successful data, and the cookies are valid (otherwise the subsequent requests would all fail)
Updating the server with a person (step 3) returns successful responses, in that they are the same as would be returned from interaction using a browser. I can only assume this must mean the server is updating successfully with the person added.
A custom header is being added to requests in step 3 X-MicrosoftAjax: Delta=true (just like the browser requests are doing)
I don't own or have access to the server I'm scraping
Given that my application requests are identical to the browser requests that succeed, it baffles me that the server is treating them differently somehow. I can't help but feel that this is an ASP.net issue with forms that I'm overlooking. I'd appreciate any help.
Update:
I went over the raw requests again a bit more methodically, and it turns out I was missing something in the form parameters of the requests. Unfortunately, I don't think it will be of much use to anyone else, because it would seem to be specific to this particular ASP servers logic.
The POST request that notifies the server that all people have been added (step 4) requires two form parameters specifying the county and address of the last person that was added to the search. I was including these form parameters in my request, but the values were empty strings. I figured the browser request was just snagging these values because when the user hits the Continue button on the form, those controls would have the values of the last person added. I figured they wouldn't matter and forgot about them, but I was wrong.
It's a peculiar issue that I should have caught the first time. I can't complain though, I am scraping a site after all.
Review Charles logs again. It is possible that the search results and other content may be coming over via Ajax, and that your Java/Ruby apps are not actually doing all of the requests/responses that happen with the browser. Look for any POST or GET requests in between the requests you are already duplicating. If search results are populated via Javascript your client app may not be able to handle this?
When i forward a request from within a post method, a confirmation alert appear
with a message "page cannot be refreshed without resending the information".
But this alert box doesn't appear when the forward is done from a get method.
What is the reason ?
Please help.
Because a POST, in the HTTP specifications, is intended for requests that are non-idempotent, because they modify state on the server (for example, by adding a new product to a category), that would be modified again if the request was resubmlited (it would create yet a new product in the category, for example).
A GET, on the other hand, is intended for requests that are idempotent. For example, a google search is idempotent. Searching twice for the same thing doesn't modify anything on the server, and resubmitting the same request doesn't have any unwanted effect.
The browser expects web applications to respect this convention, and thus warns the user about this unwanted side-effect before re-submitting a POST request.
The usual practice is to follow the post-redirect-get pattern to let the user refresh after a post without this annoying popup, and without unwanted side-effect.
Because a GET request includes those parameters in the URL (e.g. the URL ends with ?param1=foo¶m2=bar). GET requests usually don't involve sensitive data or actions that change state of the server. From the URL, you know what you're sending.
With a POST, the parameters are "hidden", submitted in the background as part of your HTTP request, and you can't see them by looking at the URL. Those params cause the server to change state, and it could cause issues if that same data was transmitted twice (e.g. you'd accidentally purchase something twice from a web store). The browser lets you know in case you don't realize you'd be resending it.
I'm a bit confused or maybe I don't fully understand http requests.
There is a website on which the search results are fetched through a GET request. I can see the whole parameter list in Firebug and if I click "search" the results are displayed as you would expect. What I don't understand is if I take this request URL (with the same parameters) and copy it in a new browser tab it doesn't return results anymore. Instead I see a 500 - Internal server error.
Can someone explain why is this happening or what can I do to see the results when accessing the URL?
As robert_b_clark suggested the solution is to send the referrer header when making the request.