I'm a noob when it comes to ASP.NET. I know few basic commands such as Response.Redirect("URL") to redirect my application web page to a different location.
However i receive HTTP Error 400 - Bad Request, whenever i try to use the code shown below
Response.Redirect(Server.UrlEncode(this.Downloadlink));
where this.Downloadlink is a user defined property which returns something like this
http://mdn.vatsag.net/fp;files/DOWNLOAD/VTSetup.exe
If i post this link in the browser, the .exe file pops up (means the link is good)
However this error comes when i use the ASP.NET code.
Any form of response on this issue/reason is deeply appreciated.
See here: http://www.kirit.com/Response.Redirect%20and%20encoded%20URIs
In short: if you quickly want to fix the issue, remove the part of your code that is UrlEncoding the URL!
Related
I am trying to crawl the website https://www.rightmove.co.uk/properties/105717104#/?channel=RES_NEW
but I get (410) error
INFO: Ignoring response <410 https://www.rightmove.co.uk/properties/105717104>: HTTP status code is not handled or not allowed
I am just trying to find the properties that have been sold using the notification on the page "This property has been removed by the agent."
I know the website has not blocked me because I am able to use the scrappy shell to get the data and also view(response) works fine too, I can directly go to the same URL using web browser so the 410 doesn't make sense I can also crawl pages from the same domain,
(ie) the pages without the notification "This property has been removed by the agent."
Any help would be much appreciated.
Seem's the when a listing has been marked as removed by and agent on Rightmove then the website will return status code 410 Gone (Which is quite weird). But to solve this, simply do something like this in your request:
def start_requests(self):
yield scrapy.Request(
url='https://www.rightmove.co.uk/properties/105717104#/?channel=RES_NEW',
meta={
'handle_httpstatus_list': [410],
}
)
EDIT
Explanation: Basically, Scrapy will only handle the status code from the response is in the range 200-299, since 2XX means that it was a successful response. In your case, you got a 4XX status code which means that some error happened. By passing handle_httpstatus_list = [410] we tell Scrapy that we want it to also handle 410 responses and not only 200-299.
Here is the docs: https://docs.scrapy.org/en/latest/topics/spider-middleware.html#std-reqmeta-handle_httpstatus_list
Consider the following situation:
I have a MVC setup where the view invokes a mapper to get a specific URL for a href attribute. The mapper holds keys to make the referencing easy.
Now the view requests an invalid key, so the mapper responds with say /invalid_url. But the dispatcher knows this URL and when a user clicks the link an error message will be displayed about how bad we feel.
But what if the Google Bot visits this invalid URL?
What would a search engine friendly status code be? I feel like 500 would be appropriate because it is a server side failure. But then this has the feeling of being a temporary error or somehow not related to the URL but to the internal mechanics. The other option that comes to mind is 404. This is also valid because the requested page does not exist. However 4xx errors are client side errors ("You requested the wrong URL. So it's basically your error"). And it just doesn't feel like a client side error to me.
Am I overthinking things? Should I just go with 404?
When I have a question like this, I refer to a site like http://www.restapitutorial.com/httpstatuscodes.html
5xx error codes imply that the server made a mistake. But you're saying the view (client-side) requested the wrong key. That sounds like a client-side error to me. While it's not the user's fault, the server doesn't know this, and the requested URL really doesn't exist. So a 404 would be appropriate.
But this is a weird case, and you should still want to fix the underlying issue of the client consistently(?) requesting a bad url.
I am currently building a website and I want to add the HTTP 404 not found Error page. I designed the bad request page and its link is this, for example:
www.mysite.com/badr.cshtml
The problem is, I want to show the contents of this page on every invalid request without redirecting to that link. For example, if I type www.mysite.com/noexistingpage, the contents of badr.cshtml should load without redirecting to it.
And for your information, I am using Webmatrix2.
you do this via the web.config and the custom errors section.
http://www.localwisdom.com/blog/2010/08/how-to-setup-custom-404s-for-iis-and-asp-net-through-web-config/
For example, day, in a somewhat REST-oriented environment, a request comes in for an object that doesn't exist, like:
GET http://example.com/thing/5
Is there anything wrong with sending back a 404 response who's body is the same as a a different page? For example, responding like:
404 body: [content from "http://example.com/thing/" which is a list of things]
Does it make any sense to do this? Will this cause any problems with certain browsers? Is it confusing to the user? Or is this perfectly fine to do?
Along these same lines, I would have the content of the 404 response match the request's accept headers as best I could. (ie. abide by content negotiation with the user agent)
For example, a xml or json request would get something along the lines of a simple error message and something that says "look here for similar things", while an html request would get an HTML page that has the error message as well as the content of the list page (as I indicated above)
I think it depends on how the Restful web services are being consumed. If I'm programmatically consuming the web service from a different application, then I would want the status code together and a plain text message instead of a message decorated with HTML tags. I mean, say for example, it doesn't make sense to return a bloated 404 content if your user makes the web service call using Curl because the message will not be readable to them.
You could have different "consumes" for each restful webservice. If it's an XML request, then you return 404 and a plain text message. Otherwise, you return the error page content.
I don't see anything wrong with it. In our webservice we always send back a json error object which includes stacktraces and other details about the response. Even on a regular web server, you get at least text which can be displayed in a browser saying that you got a 404 response.
So, from here...
In ASP.NET, you have a choice about how to respond to that - it's in the web.config as CustomErrors. Turn that on, then redirect to a fancy 404 page (maybe you already do). The fancy 404 page, then, could be checking the requested querystring (which gets passed over to the custom error page as yet another querystring) to see if it's a valid redirect, lives in your database, etc. Just do a Response.Redirect() from there.
Then schooner writes:
Thanks, we do have a 404 now but we would prefer this not to be detected as a 404 in the process. We would like ot handle it directly and seperately if possible.
..and I'd like to know just how bad a practice this is. I don't expect to put my "pretty" URLs on the internet (just business cards) and I have a sample of 404-redirecting-to-a-helpful-site code working, but I don't want to get to production and have an issue with a browser that takes the initial 404 too seriously. Can anyone help me understand more about why I wouldn't want to use customErrors / 404 to flow users to the page they actually wanted?
The main problem with using customeErrors as your 404 error handler is that every time customErrors picks up an errored request rather than throwing a 404 error back to your browser and letting your browser know there was a bad request, it instead returns a 302 which indicates that a page has been relocated to whatever your customErrors page is. This isn't bad for most users because they don't know or even notice the difference, the problem comes from the fact that web crawlers DO know the difference and the status code they receive directly affects how their indexing works.
Consider the scenario where you have a page at http://mysite.com/MyAwesomePageAboutStuff.aspx for some period of time and then one day you decide you no longer need it and delete the file. If Google or some other crawler has already indexed that URL and goes back to it after you delete it the crawler will get a 302 status code instead of a 404 error and because of this status code the crawler will update the page's url to point to your error page rather deleting the non-existent link. Now, whenever someone finds that url by way of a search engine they'll end up at your error page.
It's not really a huge issue, but you can definitely see the headaches this can create for your users in the long run.
Look here for some corroborating data.
I created a vanity url system using the 404 as the handler. There's no need for a 302 on my side as the 404 dynamically loads the content and returns that. I am fully able to handle any and all POST / GET and SERVER data.
Works great. If you are interested TarantulaHawk is up on SourceForge.