I want to raise an error in my code if the status code of a request response is 429, which error should I use?
Exception feels too broad and ValueError doesn't seem right at all.
Does requests have errors built-in I can use?
Or are there standard ways of handling things like this that I am not aware of?
Pseudo-code example
import requests
response = requests.request('GET', url)
if response.status_code == 200:
continue
elif response.status_code == 429:
raise Exception('''You made too many requests, please wait a
minute and then try again''')
Related
I have a spider that is getting cookies from a site in the first few steps. I would like to get the cookies, start the scrape, and if the HTTP status of the current request == 302, I want to loop back to the cookies part to refresh them. How can I log the HTTP status as a variable in scrapy shell, to add in an "if http_status ==302, break and go back to step 1"? Thank you!
I'm an idiot. If anyone comes across this, all you have to do it set your variable (in my case http_response) to response.status. so http_response = response.status returns '200' or whatever depending on the status of the current request. lol solved.
I'm trying to send a grpc request and I'm expected to receive stream of message back. Instead I'm receiving a response as <_MultiThreadedRendezvous object>. Can anyone help me to understand why I'm receiving this and what should I do to extract the expected message from this object. The server is C++ and client is python in this case.
Please take a look at gRPC's example of streaming RPC. The _MultiThreadedRendezvous object is the library's representation of an RPC result. It is an iterable, you can use for to fetch all responses, or you can use list() to get all messages.
Using e.code() and e.details() helped me parsing it:
try:
response = stub.SayHello(...)
except _MultiThreadedRendezvous as e:
if e.code() == grpc.StatusCode.CANCELLED:
pass
# process cancelled status
elif e.code() == grpc.StatusCode.UNAVAILABLE and 'Connection reset by peer' in e.details():
pass
# process unavailable status
else:
raise
Looking at the source code of that error helped me understand how to parse it.
Credit to this answer on another question.
This is part of code of my project.
URL="http://www.amazon.com",
HTTPOpts = [{autoredirect, false}],
case httpc:request(get, {URL, [{"User-Agent", "Mozilla"}]}, HTTPOpts, []) of
{ok, {{_, Code, _}, Headers, Body}} when Code == 200 ->
%%code for process code=200 %%
{ok, {{_, Code, _}, Headers, _}} when Code < 310 , Code >= 300 ->
%% redirection
{ok, {{_, Code, _}, Headers, _}} when Code ==503 ->
%%service unavailable
The problem is when I perform http request, it returns different status Code.
In case of URL above I'm getting two responses, Code = 200 and Code = 503, how do I handle this, so that I always get Code = 200
I also tried it using wget "www.amazon.com", it gives same result.
My idea: re-request in case of Code = 503, but problem with this it may go into loop and may never return Code = 200 or return after several iterations, which produce delay in client request.
How to resolve it?
As developers, we do not have control over the responses of third-party systems we try to talk to. With the example you provided, it seems like amazon is deliberately denying you access because they suspect you as a bot or scraper. You can prove this by looking at the response body whenever you get a 503.
What you can do as a developer, is to adapt to every possible situation that can occur when connecting to a certain system.
For HTTP, when you encounter 5xx error codes, normally you will need to retry your request. To prevent yourself being stuck on a loop, implement an exponential backoff with a limit on how much you allow your code to retry.
HTTP 4xx error codes usually means there's something wrong with your request. You don't want to retry here, just take a look at what could be wrong with your request.
For your particular case, since Amazon thinks you're an automated visitor, try to mimic a normal web browser. Start with the User-agent header, cookies, etc.
I'm wondering if it is correct to return HTTP 200 OK when an error occurred on the server side (the error details would be contained inside the response body).
Example:
We're sending HTTP GET
Something unexpected happened on the server side.
Server returns HTTP 200 OK status code with error inside a response (e.g. {"status":"some error occurred"})
Is this the correct behavior or not? Should we change the status code to something else than 200?
No, it's very incorrect to send 200 with a error body
HTTP is an application protocol. 200 implies that the response contains a payload that represents the status of the requested resource. An error message usually is not a representation of that resource.
If something goes wrong while processing GET, the right status code is 4xx ("you messed up") or 5xx ("I messed up").
HTTP status codes say something about the HTTP protocol. HTTP 200 means transmission is OK on the HTTP level (i.e request was technically OK and server was able to respond properly). See this wiki page for a list of all codes and their meaning.
HTTP 200 has nothing to do with success or failure of your "business code". In your example the HTTP 200 is an acceptable status to indicate that your "business code error message" was successfully transferred, provided that no technical issues prevented the business logic to run properly.
Alternatively you could let your server respond with HTTP 5xx if technical or unrecoverable problems happened on the server. Or HTTP 4xx if the incoming request had issues (e.g. wrong parameters, unexpected HTTP method...) Again, these all indicate technical errors, whereas HTTP 200 indicates NO technical errors, but makes no guarantee about business logic errors.
To summarize: YES it is valid to send error messages (for non-technical issues) in your http response together with HTTP status 200. Whether this applies to your case is up to you. If for instance the client is asking for a file that isn't there, that would be more like a 404. If there is a misconfiguration on the server that might be a 500. If client asks for a seat on a plane that is booked full, that would be 200 and your "implementation" will dictate how to recognise/handle this (e.g. JSON block with a { "booking": "failed" })
I think these kinds of problems are solved if we think about real life.
Bad Practice:
Example 1:
Darling everything is FINE/OK (HTTP CODE 200) - (Success):
{
...but I don't want us to be together anymore!!!... (Error)
// Then everything isn't OK???
}
Example 2:
You are the best employee (HTTP CODE 200) - (Success):
{
...But we cannot continue your contract!!!... (Error)
// Then everything isn't OK???
}
Good Practices:
Darling I don't feel good (HTTP CODE 400) - (Error):
{
...I no longer feel anything for you, I think the best thing is to separate... (Error)
// In this case, you are alerting me from the beginning that something is wrong ...
}
This is only my personal opinion, each one can implement it as it is most comfortable or needs.
Note: The idea for this explanation was drawn from a great friend #diosney
Even if I want to return a business logic error as HTTP code there is no such
acceptable HTTP error code for that errors rather than using HTTP 200 because it will misrepresent the actual error.
So, HTTP 200 will be good for business logic errors. But all errors which are covered by HTTP error codes should use them.
Basically HTTP 200 means what server correctly processes user request (in case of there is no seats on the plane it is no matter because user request was correctly processed, it can even return just a number of seats available on the plane, so there will be no business logic errors at all or that business logic can be on client side. Business logic error is an abstract meaning, but HTTP error is more definite).
To clarify, you should use HTTP error codes where they fit with the protocol, and not use HTTP status codes to send business logic errors.
Errors like insufficient balance, no cabs available, bad user/password qualify for HTTP status 200 with application specific error handling in the response body.
See this software engineering answer:
I would say it is better to be explicit about the separation of protocols. Let the HTTP server and the web browser do their own thing, and let the app do its own thing. The app needs to be able to make requests, and it needs the responses--and its logic as to how to request, how to interpret the responses, can be more (or less) complex than the HTTP perspective.
I think people have put too much weight into the application logic versus protocol matter. The important thing is that the response should make sense. What if you have an API that serves a dynamic resource and a request is made for X which is derived from template Y with data Z and either Y or Z isn't currently available? Is that a business logic error or a technical error? The correct answer is, "who cares?"
Your API and your responses need to be intelligible and consistent. It should conform to some kind of spec, and that spec should define what a valid response is. Something that conforms to a valid response should yield a 200 code. Something that does not conform to a valid response should yield a 4xx or 5xx code indicative of why a valid response couldn't be generated.
If your spec's definition of a valid response permits { "error": "invalid ID" }, then it's a successful response. If your spec doesn't make that accommodation, it would be a poor decision to return that response with a 200 code.
I'd draw an analogy to calling a function parseFoo. What happens when you call parseFoo("invalid data")? Does it return an error result (maybe null)? Or does it throw an exception? Many will take a near-religious position on whether one approach or the other is correct, but ultimately it's up to the API specification.
"The status-code element is a three-digit integer code giving the result of the attempt to understand and satisfy the request"
Obviously there's a difference of opinion with regards to whether "successfully returning an error" constitutes an HTTP success or error. I see different people interpreting the same specs different ways. So pick a side, sure, but also accept that either way the whole world isn't going to agree with you. Me? I find myself somewhere in the middle, but I'll offer some commonsense considerations.
If your server-side code catches an unexpected exception when dispatching a request, that sounds like the very definition of a 500 Internal Server Error. This seems to be OP's situation. The application should not return a 200 for unexpected errors, but also see point 3.
If your server-side code should be able to gracefully handle a given invalid input, and it doesn't constitute an "exceptional" error condition, your spec should accommodate HTTP 200 responses that provide meaningful diagnostic information.
Above all: Have a spec. Make it consistent. Stick to it.
In OP's situation, it sounds like you have a de-facto standard that unhandled exceptions yield a 200 with a distinguishable response body. It's not ideal, but if it's not breaking things and actively causing problems, you probably have bigger, more important problems to solve.
HTTP Is the Protocol handling the transmission of data over the internet.
If that transmission breaks for whatever reason the HTTP error codes tell you why it can't be sent to you.
The data being transmitted is not handled by HTTP Error codes. Only the method of transmission.
HTTP can't say 'Ok, this answer is gobbledigook, but here it is'. it just says 200 OK.
i.e : I've completed my job of getting it to you, the rest is up to you.
I know this has been answered already but I put it in words I can understand. sorry for any repetition.
What HTTP status should I return if my script throws an exception?
200 OK
or
500 Internal Server Error
Let's say user request parameters are correct but there is a bug in my script which causes an error message to appear instead of a proper response (XML, JSON or other format). What should be the HTTP status?
500 Internal Server Error is the correct status if the error can't be fixed by the client changing their request.
Use any of the 4XX statuses if the client might be able to fix their request to avoid the error (or 404 if the resource wasn't found).
200 OK is not the appropriate status in almost any error situation, because then the client thinks things are running normally (which they are not) and may continue to make the same error-causing requests.
Familiarize yourself with the available status codes in RFC2616 and find one that most appropriately fits the situation.
It depends on why the exception is thrown since they can be used for almost any error. If it's thrown because some id in the URI is not found in the database I'd say 404. On the other hand if it's because the database is down I would throw a 500. If an exception is thrown but the resulting page would still be useful to the user I would say return 200.
Review the Status Code Definitions. 500 or 400 should do for general issues, however, the more detailed you can be then the more useful the returned status will be.