I am programming an HTTP request parser and I would like to see the string Chrome sends whenever a resource is requested (that may be a website or content on the website). I know that you can go into the Chrome developer tools under the network tab and then see all the details about the requests sent. However, these details are already parsed into categories and I would like a raw string representation of the HTTP request. Is there any way to get just that and, if not, could you recommend any tool or website to get a bunch of dummy HTTP requests to debug my parser?
There is no built in function to acquire the raw http request string in Chrome. Request parameters can either be optained by using the network tab in the developer tools or by using chrome://net-export.
However, external tools such as Fiddler can obtain the raw http requests send and responses recieved by your computer. Just select a request on the left and the raw output format.
Related
I tried to reproduce a bunch of SOAP requests sent by an .exe file in Postman but the API end point does not send back the correct result.
I constructed a request exactly like the one captured with Wireshark, but the response is not correct.
What seems to be the problem? What am I missing?
Update:
I just tried to send these request using SoapUI instead of Postman and with SoapUI the response is a SOAP response so it seems more correct, the endpoint still doesn't send back correct result, but at least the response is a SOAP response now.
Apparently Postman messes up the SOAP request in some way.
Solution:
I created a soap service with SoapUI and used the wsdl.xml file provided by the web service
SoapUI automatically generated all operations/requests defined in the wsdl.xml
I sent these auto generated requests made by SoapUI and they worked.
So I compared these requests with the ones I was sending and realized the syntax (xml body of the request) of these auto generated requests was different than the ones described in the web service doc.
but there are still some weird things that I don't understand, like the requests I captured with Wireshark had the same xml body as in the Doc described but they were successful responses.
I have a web server which contains an API to upload files to Amazon's S3 storage. Since I do not want to waste resources on streaming the files through my server, when an upload request comes in, I generate a pre-signed URL for the client and then redirect that client to this URL using HTTP 307 - Temporary redirect.
In practice, the flow looks like this:
Client issues a PUT request to my server, requesting a file upload
My server inspects the request and generates a pre-signed URL for S3
My server responds to client with 307 redirection to the pre-signed URL
Client repeats the PUT request to the pre-signed URL
Upload commences
The challenge
My server uses the Authorization header for... well, authorisation. Incidentally, Amazon also accepts this header for authorisation, although the values expected by both parties are completely different.
The problem is, that since my upload API requires this header to be present during file upload request, when my server issues the 307 redirect back to the client, the client takes all the headers in the original request and sends them along to the pre-signed S3 URL, which causes the request to be rejected by Amazon due to authorisation error.
The question
Can I somehow instruct the client (via HTTP response header) to not include the Authorization header when following the redirection?
Current solution
Right now we "fixed" this by returning the pre-signed URL to the client in the response body. The client then manually issues a new PUT request to that URL without the Authorization header. This works fine. I would like to know if there is a way to achieve this behaviour without this extra manual work.
What is the client? In the above statements, when I mention the "client", right now it could be either a modern web browser or a native iOS or Android app. On iOS, we use Alamofire for HTTP communication. I am unsure of what library or components are used on Android.
Note: I have seen this question and its answers, but it does not contain the answers I seek.
I use the SoundCloud API to retrieve the stream URL for a streamable track.
I follow the redirect and I end up with an URL that looks like:
http://ec-media.soundcloud.com/eodihgiuh.128.mp3?<a string>
AWSAccessKeyId=<access key>
&Expires=<timestamp>
&Signature=<signature>
or
http://ak-media.soundcloud.com/euieuieie.128.mp3?
AWSAccessKeyId=<access key>
&Expires=<timestamp>
&Signature=<signature>
&__gda__=<a string>
Then I start streaming the MP3 data at this URL.
First I send a HEAD request to read the Content-Length header, so that I know how many GET requests I will have to send in order to play the whole song.
Then I send several partial GET requests, each one with a different Range header.
The problem is that sometimes the HEAD request returns a 403 status code, even though a GET request to the exact same URL returns with a 200 status code. It seems that this happens if and only if the host is ak-media.soundcloud.com.
Is this supposed to happen? I expected the HEAD request to return exactly the same headers as the GET request, only without the body response.
Cheers,
PB
P.S: I should probably mention that my code is not running on a computer, but on an audio device with a tiny 8-bit processor which has extremely limited resources.
Unfortunately, currently we only offer guaranteed proper response for GET requests.
As a hack, you could try to do requests with very short ranges.
I am setting up a back end API in a script of mine that contacts one of my sites by sending XML to my web server in the form of POST data. This script will be used by many and I want to limit the bandwidth waste for people that accidentally turn the feature on without a proper access key.
I will be denying requests that do not have the correct access key by maybe generating a 403 access code.
Lets say the POST data is ~500kb of data. Does the server receive all 500kb of data when this attempt is made regardless of the status code?
How about if I made the url contain the key mydomain/api/123456789 and generate 403 status on all bad access keys.
Does the POST data still get sent/received regardless or is it negotiated before the data is finally sent.
Thanks in advance!
Generally speaking, the entire request will be sent, including post data. There is often no way for the application layer to return a response like a 403 until it has received the entire request.
In reality, it will depend on the language/framework used and how closely it is linked to the HTTP server. Section 8.2.2 of RFC2616 HTTP/1.1 specification has this to say
An HTTP/1.1 (or later) client sending
a message-body SHOULD monitor the
network connection for an error status
while it is transmitting the request.
If the client sees an error status, it
SHOULD immediately cease transmitting
the body. If the body is being sent
using a "chunked" encoding (section
3.6), a zero length chunk and empty trailer MAY be used to prematurely
mark the end of the message. If the
body was preceded by a Content-Length
header, the client MUST close the
connection.
So, if you can find a language environemnt closely linked with the HTTP server (for example, mod_perl), you could do this in a way which does comply with standards.
An alternative approach you could take is to make an initial, smaller request to obtain a URL to use for the larger POST. The application can then deny providing the URL to clients without an appropriate key.
Here is great book about RESTful Web Services, where it's explained how HTTP works: http://oreilly.com/catalog/9780596529260
You can consider any request as envelope, where on top of it it's written address (URL), some properties (HTTP Headers) and inside it there's some data (if request is initiated by post method). So as you might guess you can't receive envelope partially.
Oh I forgot, it's when you are using HTTP Post with standard HTTP header "application/x-www-form-urlencoded" but if you are uploading files (correspondingly using ""multipart/form-data") Django gives you control over streamed chunks of files using Middleware classes: http://docs.djangoproject.com/en/dev/topics/http/middleware/
In the header exchange below I see that the server is returning the page Gzipped but I don't see where my browser ever indicated that it could accept GZip. How did the server know?
The content you have reproduced here is not what was sent by your browser; the "general" part is a mix of some of the request data and some of the response data. If you want to see the actual request an response, use something like wireshark.
Coincidentally, it is worth noting that some so-called security products will interfere with your browsers request - a common "enhancement" is to remove or mangle the header asking for compression. Your webserver will honour such requests in the absence of specific configuration to force compression. Google delivers a compressed JavaScript to the client when it sees such behaviour - if it runs on the client then Google start sending compressed content. There are Apache config snippets on the web which can detect and override some such tampering.
But there's no evidence here to suggest that is the case with your setup. You're just not seeing the request headers.