We've spotted (in our error tracking tool) some http 400 issues while fetching some static files.
Also, there are logs in our API gateway regarding that it redirected the request to S3 which responded 400.
It's not our CDN neither our API gateway.
Why would S3 respond 400 to a static file?
We couldn't find anything exactly about it anywhere so far, but some general resources about 400 were pointing to issues with some HTTP header:
Malformed request syntax
If-Modified-Since
Amazon ALB 400 Request header or cookie too large
Kong 400 Request header or cookie too large
Since we couldn't manually reproduce that accessing our system, we started trying to fetch the static files with some unexpected headers using Postman.
We finally could quickly reproduce the issue by sending huge Cookies.
Our hypothesis is that, probably, Amazon S3's static website has some configuration that blocks requests with some specific long headers/cookies.
Once we couldn't find a way to configure that on S3, we've added a plugin to our API Gateway that when it was a static file request, it would remove the cookie before redirecting it to S3 (it wasn't required).
After that change, we've monitored our error tracking tool, and the occurrences of http 400 on static website dropped to 0.
Today I just got kind same case, my case reason is the cookie side is too big.
Related
I was doing a automation testing on my web application with SOAPUI, I have found a bug which is http method fuzzing basically it means "HTTP Method Fuzzing
An HTTP Method Fuzzing Scan attempts to use other HTTP verbs (methods) than those defined in an API. For instance, if you have defined GET and POST, it will send requests using the DELETE and PUT verbs, expecting an appropriate HTTP error response and reporting alerts if it doesn't receive it.
Sometimes, unexpected HTTP verbs can overwrite data on a server or get data that shouldn't be revealed to clients."
Can anyone knows how I can solve this issue or how I block the HTTP request other than GET or POST which may remove this bug.
I am using Node.js and express for my web application.
Please check the images:
Image 1
Image 2
Status code 400 Bad Request is used when the client has sent a request that cannot be processed due to being malformed.
Status code 404 Not Found is used when the requested resource does not exist / cannot be found.
My question is, when a client sends a request to an endpoint my API does not serve, which of these status codes is more appropriate?
Should an endpoint be considered a "resource", and thus a 404 be returned? My issue with this is that if the client only checks the status code, they cannot tell the difference between a 404 indicating that they got to the correct endpoint, but there was no result matching their query, versus a 404 indicating that they queried a non-existing endpoint.
Alternatively, should we expect that a client has prior knowledge of all available API endpoints, and thus treat their request as malformed and return a 400 if the endpoint they are trying to reach does not exist?
Maybe this depends on whether the endpoints are REST or not. If they are REST endpoints, the client should not need prior API knowledge, but be able to learn about all relevant API endpoints by navigating the API from a single root endpoint. In such a case, I guess 404 would be more appropriate.
In my specific case right now, this is an internal (non-REST) HTTP API, where I expect the client to have prior knowledge of all API endpoints, so I am leaning towards 400, to avoid issues where 404 from accessing the wrong endpoint could be misconstrued as a 404 indicating that what they sought from the correct endpoint could not be found.
Thoughts?
As a convenience, many modern APIs provide human-readable endpoints for developer convenience. The intent of REST, however, is that URLs are treated as opaque - they may happen to contain semantic content, but can't be relied upon to do so. There's no such thing as a "malformed" URL. There's only a URL that points to something and a URL that doesn't.
Now, that's the REST dogma (and arguably also the HTTP 1.1 spec). That doesn't mean it's what you should do. If you have a single internal client for your API, and that's not going to change, you have a lot of flexibility in designing your own standards. Just make sure to document them, especially those that might confuse the guy straight out of college that they hire to replace you when you move on.
I'm trying to retrieve 3 response headers (Rails Devise Auth Headers: uid, client, access-token) in every request to a Rails Server.
Using Postman (http client) it's working.
With OkHttp (java http client) the headers just don't show up in the client (i've checked using Wireshark).
When i'm in debug mode it just work...
The additional headers with postman are due to postman sending an Origin header and the server is replying with CORS headers, i.e. Access-Control-.... These headers are send within the normal HTTP header, i.e. not after the response.
But these access control headers are only relevant when the access is done from a browser because they control the cross origin behavior of XHR. Since you are not inside a browser they should be irrelevant for what you are doing. What is relevant are the body of the response and some of the other headers and here you'll find no differences. Also irrelevant should be if multiple requests are send within the same TCP connection (HTTP keep-alive done by postman) or with multiple connections (OkHttp) because each request is independent from the other and using the same TCP connection is only a performance optimization.
If you really want to get these special headers you should add an Origin header within you OkHttp request. See the OkHttp examples on how to add your own headers. But like I said: these access control headers should be irrelevant for the real task and there should be no need to get to these headers.
There is a property "config.batch_request_buffer_throttle" in the file "config/initializers/devise_token_auth.rb" of the Rails Project. We changed it from 5 seconds to 0 seconds.
It is a property to keep the current token available for that amount of time to the following requests.
As the original documentation: "Sometimes it's necessary to make several requests to the API at the same time. In this case, each request in the batch will need to share the same auth token. This setting determines how far apart the requests can be while still using the same auth token."
So when we did the request using Postman or in Java Debug the 5 seconds was running allowing Devise to generate new tokens then retrieve them to the client.
I have a web server which contains an API to upload files to Amazon's S3 storage. Since I do not want to waste resources on streaming the files through my server, when an upload request comes in, I generate a pre-signed URL for the client and then redirect that client to this URL using HTTP 307 - Temporary redirect.
In practice, the flow looks like this:
Client issues a PUT request to my server, requesting a file upload
My server inspects the request and generates a pre-signed URL for S3
My server responds to client with 307 redirection to the pre-signed URL
Client repeats the PUT request to the pre-signed URL
Upload commences
The challenge
My server uses the Authorization header for... well, authorisation. Incidentally, Amazon also accepts this header for authorisation, although the values expected by both parties are completely different.
The problem is, that since my upload API requires this header to be present during file upload request, when my server issues the 307 redirect back to the client, the client takes all the headers in the original request and sends them along to the pre-signed S3 URL, which causes the request to be rejected by Amazon due to authorisation error.
The question
Can I somehow instruct the client (via HTTP response header) to not include the Authorization header when following the redirection?
Current solution
Right now we "fixed" this by returning the pre-signed URL to the client in the response body. The client then manually issues a new PUT request to that URL without the Authorization header. This works fine. I would like to know if there is a way to achieve this behaviour without this extra manual work.
What is the client? In the above statements, when I mention the "client", right now it could be either a modern web browser or a native iOS or Android app. On iOS, we use Alamofire for HTTP communication. I am unsure of what library or components are used on Android.
Note: I have seen this question and its answers, but it does not contain the answers I seek.
It's my understanding that caching is one of the main utilities of a proxy server. I'm currently trying to develop a simple one and I would like to know exactly how caching works.
Intuitively I think that it's basically an association between a request and a response. For example: for the following request: "GET google.com" you have the following response: "HTTP/1.0 200 OK..."
That way, whenever the proxy server receives a request for that URL he can reply with the cached response (I'm not really worried right now about when to serve the cached response and when to actually send the request to the real destination).
What I don't understand is how to establish the association between a request and a response since the HTTP response doesn't have any field saying "hey this is the response you get when you request the X URL" (or does it?).
Should I get this information by analyzing the underlying protocols? If so, how?
Your cache proxy server is already putted into play when a request arrives. Therefore you have the requested resource URL. Then you look in your cache and try to find the cached resource for the requested resource URL, if you cannot find the resource in your cache (or the cache is outdated), you fetch the data from the source. Keep in Mind, that you have to invalidate the cached resource if you receive a PUT, POST or DELETE request.