I'm using a luminati (newly named brightdata) proxy on one of my scrapy spider and I can't find anything to prevent cookies/headers from being discarded when sending a request.
I need to pass cookies and custom headers but when I'm checking what's being sent (via requestbin) I see that my custom headers and cookies have disappeared.
Any idea on how I could prevent this behavior ?
I have created a WebAPI controller as below
[EnableCors("http://localhost:1234", "*", "*"]
public class DummyController : ApiController
{
public string GetDummy()
{
return "Iam not DUMMY";
}
}
When I hit the service using ajax from my application which is hosted on locahost:5678 It throws error since it is not allowed but when I hit the same API from restclient like PostMan it returns data.
Questions
1) CORS restricts only ajax requests and not the normal HTTP requests because I believe postman sends normal http requests.
2) How does EnableCors restrict to provided origins? Consider if I modify the origin and referrer params in the ajax request I can fish the values. What strategy does CORS use to identify the referrer URL.
As W3C states HttpReferrer can be easily modified, one should not depend on its value to authorize the access. If that is the case What does EnableCors checking in behind to authorize the origin.
I could just change my origin in ajax request also. Please help me with this Iam pretty much confused
CORS restricts only ajax requests and not the normal HTTP requests because I believe postman sends normal http requests.
Yes, specifically browsers restrict Ajax requests — that is, browsers by default don’t allow frontend JavaScript code to access responses from cross-origin requests made with XMLHttpRequest, the Fetch API, or with Ajax methods from JavaScript libraries.
Servers don’t themselves enforce any restrictions on cross-origin requests; instead, servers send responses to any clients that make requests to them, including postman — and including browsers.
Browsers themselves always get the responses that any other client would; but just because the browser gets a response doesn’t mean the browser will allow frontend JavaScript code to access that response. Browsers will only expose a response for a cross-origin request to frontend code if the response includes the Access-Control-Allow-Origin header.
How does EnableCors restrict to provided origins?
It doesn’t. When you CORS-enable a server, the only effect that has is to cause the server to send additional response headers, based on the values of particular request headers it receives — in particular, the Origin request header.
Consider if I modify the origin and referrer params in the ajax request I can fish the values. What strategy does CORS use to identify the referrer URL.
Servers don’t (and can’t) do any validation of the Origin value to confirm it hasn’t been spoofed or whatever. But the CORS protocol doesn’t require servers to do that — because all CORS enforcement is done by browsers.
As W3C states HttpReferrer can be easily modified, one should not depend on its value to authorize the access. If that is the case What does EnableCors checking in behind to authorize the origin.
I could just change my origin in ajax request also. Please help me with this Iam pretty much confused
Browsers know the real origin of any frontend code that sends a cross-origin request, and browsers do CORS checks against what they know to be the real origin of the request — and not against the value of the Origin header.
Browsers are what set the Origin request header and send it over the network to begin with; they set the Origin value based on what they know to be the real origin, and not for their own use — because they already know what the origin is and that value is what they use internally.
So even if you manage to change an Origin header for a request, that won’t matter to the browser — it’s going to ignore that value and continue checking against the real origin.
cf. the answer at
In the respective of security, is it meaningful to allow CORS for specific domains?
What is the conceptual difference between forward() and sendRedirect()?
In the web development world, the term "redirect" is the act of sending the client an empty HTTP response with just a Location header containing the new URL to which the client has to send a brand new GET request. So basically:
Client sends a HTTP request to some.jsp.
Server sends a HTTP response back with Location: other.jsp header
Client sends a HTTP request to other.jsp (this get reflected in browser address bar!)
Server sends a HTTP response back with content of other.jsp.
You can track it with the web browser's builtin/addon developer toolset. Press F12 in Chrome/IE9/Firebug and check the "Network" section to see it.
Exactly the above is achieved by sendRedirect("other.jsp"). The RequestDispatcher#forward() doesn't send a redirect. Instead, it uses the content of the target page as HTTP response.
Client sends a HTTP request to some.jsp.
Server sends a HTTP response back with content of other.jsp.
However, as the original HTTP request was to some.jsp, the URL in browser address bar remains unchanged. Also, any request attributes set in the controller behind some.jsp will be available in other.jsp. This does not happen during a redirect because you're basically forcing the client to create a new HTTP request on other.jsp, hereby throwing away the original request on some.jsp including all of its attribtues.
The RequestDispatcher is extremely useful in the MVC paradigm and/or when you want to hide JSP's from direct access. You can put JSP's in the /WEB-INF folder and use a Servlet which controls, preprocesses and postprocesses the requests. The JSPs in the /WEB-INF folder are not directly accessible by URL, but the Servlet can access them using RequestDispatcher#forward().
You can for example have a JSP file in /WEB-INF/login.jsp and a LoginServlet which is mapped on an url-pattern of /login. When you invoke http://example.com/context/login, then the servlet's doGet() will be invoked. You can do any preprocessing stuff in there and finally forward the request like:
request.getRequestDispatcher("/WEB-INF/login.jsp").forward(request, response);
When you submit a form, you normally want to use POST:
<form action="login" method="post">
This way the servlet's doPost() will be invoked and you can do any postprocessing stuff in there (e.g. validation, business logic, login the user, etc).
If there are any errors, then you normally want to forward the request back to the same page and display the errors there next to the input fields and so on. You can use the RequestDispatcher for this.
If a POST is successful, you normally want to redirect the request, so that the request won't be resubmitted when the user refreshes the request (e.g. pressing F5 or navigating back in history).
User user = userDAO.find(username, password);
if (user != null) {
request.getSession().setAttribute("user", user); // Login user.
response.sendRedirect("home"); // Redirects to http://example.com/context/home after succesful login.
} else {
request.setAttribute("error", "Unknown login, please try again."); // Set error.
request.getRequestDispatcher("/WEB-INF/login.jsp").forward(request, response); // Forward to same page so that you can display error.
}
A redirect thus instructs the client to fire a new GET request on the given URL. Refreshing the request would then only refresh the redirected request and not the initial request. This will avoid "double submits" and confusion and bad user experiences. This is also called the POST-Redirect-GET pattern.
See also:
How do servlets work? Instantiation, sessions, shared variables and multithreading
doGet and doPost in Servlets
How perform validation and display error message in same form in JSP?
HttpServletResponse sendRedirect permanent
requestDispatcher - forward() method
When we use the forward method, the request is transferred to another resource within the same server for further processing.
In the case of forward, the web container handles all processing internally and the client or browser is not involved.
When forward is called on the requestDispatcherobject, we pass the request and response objects, so our old request object is present on the new resource which is going to process our request.
Visually, we are not able to see the forwarded address, it is transparent.
Using the forward() method is faster than sendRedirect.
When we redirect using forward, and we want to use the same data in a new resource, we can use request.setAttribute() as we have a request object available.
SendRedirect
In case of sendRedirect, the request is transferred to another resource, to a different domain, or to a
different server for further processing.
When you use sendRedirect, the container transfers the request to the client or browser, so the URL given inside the sendRedirect method is visible as a new request to the client.
In case of sendRedirect call, the old request and response objects are lost because it’s treated as new request by the browser.
In the address bar, we are able to see the new redirected address. It’s not transparent.
sendRedirect is slower because one extra round trip is required, because a completely new request is created and the old request object is lost. Two browser request are required.
But in sendRedirect, if we want to use the same data for a new resource we have to store the data in session or pass along with the URL.
Which one is good?
Its depends upon the scenario for which method is more useful.
If you want control is transfer to new server or context, and it is treated as completely new task, then we go for sendRedirect.
Generally, a forward should be used if the operation can be safely repeated upon a browser reload of the web page and will not affect the result.
Source
The RequestDispatcher interface allows you to do a server side forward/include whereas sendRedirect() does a client side redirect. In a client side redirect, the server will send back an HTTP status code of 302 (temporary redirect) which causes the web browser to issue a brand new HTTP GET request for the content at the redirected location. In contrast, when using the RequestDispatcher interface, the include/forward to the new resource is handled entirely on the server side.
The main important difference between the forward() and sendRedirect() method is that in case of forward(), redirect happens
at server end and not visible to client, but in case of
sendRedirect(), redirection happens at client end and it's visible to
client.
Either of these methods may be "better", i.e. more suitable, depending on what you want to do.
A server-side redirect is faster insofar as you get the data from a different page without making a round trip to the browser. But the URL seen in the browser is still the original address, so you're creating a little inconsistency there.
A client-side redirect is more versatile insofar as it can send you to a completely different server, or change the protocol (e.g. from HTTP to HTTPS), or both. And the browser is aware of the new URL. But it takes an extra back-and-forth between server and client.
SendRedirect() will search the content between the servers. it is slow because it has to intimate the browser by sending the URL of the content. then browser will create a new request for the content within the same server or in another one.
RquestDispatcher is for searching the content within the server i think. its the server side process and it is faster compare to the SendRedirect() method. but the thing is that it will not intimate the browser in which server it is searching the required date or content, neither it will not ask the browser to change the URL in URL tab. so it causes little inconvenience to the user.
Technically redirect should be used either if we need to transfer control to different domain or to achieve separation of task.
For example in the payment application
we do the PaymentProcess first and then redirect to displayPaymentInfo. If the client refreshes the browser only the displayPaymentInfo will be done again and PaymentProcess will not be repeated. But if we use forward in this scenario, both PaymentProcess and displayPaymentInfo will be re-executed sequentially, which may result in incosistent data.
For other scenarios, forward is efficient to use since as it is faster than sendRedirect
Request Dispatcher is an Interface which is used to dispatch the request or response from web resource to the another web resource. It contains mainly two methods.
request.forward(req,res): This method is used forward the request from one web resource to another resource. i.e from one servlet to another servlet or from one web application to another web appliacation.
response.include(req,res): This method is used include the response of one servlet to another servlet
NOTE: BY using Request Dispatcher we can forward or include the request or responses with in the same server.
request.sendRedirect(): BY using this we can forward or include the request or responses across the different servers. In this the client gets a intimation while redirecting the page but in the above process the client will not get intimation
Simply difference between Forward(ServletRequest request, ServletResponse response) and sendRedirect(String url) is
forward():
The forward() method is executed in the server side.
The request is transfer to other resource within same server.
It does not depend on the client’s request protocol since the forward () method is provided by the servlet container.
The request is shared by the target resource.
Only one call is consumed in this method.
It can be used within server.
We cannot see forwarded message, it is transparent.
The forward() method is faster than sendRedirect() method.
It is declared in RequestDispatcher interface.
sendRedirect():
The sendRedirect() method is executed in the client side.
The request is transfer to other resource to different server.
The sendRedirect() method is provided under HTTP so it can be used only with HTTP clients.
New request is created for the destination resource.
Two request and response calls are consumed.
It can be used within and outside the server.
We can see redirected address, it is not transparent.
The sendRedirect() method is slower because when new request is created old request object is lost.
It is declared in HttpServletResponse.
I'm successfully setting up an HTTP Digest Authorization between the web browser and the server. But some of the resources on the same page to the same host are failing because the browser isn't sending the authorization for them.
For example,
Page https://myhost/A/B/C/D/E/ is loaded, browser sends Authorization header.
Page contains IMG ref to https://myhost/A/B/C/D/E/F.JPG, browser sends Authorization header.
Page also contains IMG ref to https://myhost/A/B.JPG, but for some reason browser does not send Authorization header. Server returns 401 Unauthorized but browser does not retry with authorization or pop up a username/password field, it simply displays the "broken image" icon.
I have looked a bit at how HTTP Authorization and I don't see anything mentioned regarding the scope of a request. Nevertheless, because I am explicitly sending back a 401 if the browser doesn't send Authorization for a request, I would expect it should work.
How can I fix this problem?
HTML authorization is governed by RFC 2617, which in section 1.2 says:
The realm value (case-sensitive), in combination with the canonical
root URL (the absoluteURI for the server whose abs_path is empty; see
section 5.1.2 of [2]) of the server being accessed, defines the
protection space.”. Later in the same section it says: “The protection
space determines the domain over which credentials can be
automatically applied. If a prior request has been authorized, the
same credentials MAY be reused for all other requests within that
protection space for a period of time determined by the authentication
scheme, parameters, and/or user preference.
So as long as the two URLS are in the same "protection space" the browser is supposed to resend the same credentials. However in this case the problem is that they are not. If authorization occurs in the https://myhost/A/B/C/D/E/ space, then the browser may not see a need to send authorization for https://myhost/A/B.JPG.
Section 2 mentions:
A client SHOULD assume that all paths at or deeper than the depth of
the last symbolic element in the path field of the Request-URI also
are within the protection space specified by the Basic realm value of
the current challenge. A client MAY preemptively send the
corresponding Authorization header with requests for resources in that
space without receipt of another challenge from the server.
So the solution is to make sure that the WWW-Authenticate header sent by the server sends a domain=/ entry, so that everything under that will be considered in the same protection space.
I have a Jquery Ajax request which goes to Facebook server. (jsonP)
However , Im not satisfy with the current response and I would like to "edit" the response before it goes back to the ajax call.
In order to do it , I was thinking about building a proxy by ashx. ( it will do the request , and when the response is back - he will edit it , and it will return to the ajax call).
jQuery ajax ---> myHandler.ashx ----> Facebook +--->
|
|
jQuery ajax <---- myHandler.ashx(+edit) <-----------+
the problem is that Facebook requires its cookies , And IMHO -only facebook can access its cookies.
Is there any way for the ashx handler to be able to TRANSFER the cookie along with his request ( as if jQuery ajax would do ) ?
Technically, you may simulate a cookie-aware user agent and relay them to user.
Request 1:
Browser sends the first request to your ashx handler.
Your handler relays it to facebook server.
FB responds with a content + a set-cookie header(s) (fb-cookie=xxx on facebook.com)
Your handler edits the content and returns the set-cookie header with some kind of prefix (your-fb-cookie=xxx on yoursite.com)
Request 2:
Next time the browser sends a request to your ashx handler, you will receive the the "your-fb-cookie=xxx" (as it is defined on yoursite.com)
Your handler relays the request to FB with the cookie "fb-cookie=xxx"
...
However, be aware that facebook (or any other service) may (and should) use some kind of anti-flood detection.
It may block your server's IP adress as it receives lots of requests from it.
This can even be against FB licensing terms.