So, I've noticed that using Dart's built in HttpServer class tends to make the client request for every file every time.
On Apache, it is possible to tell the client to cache the file for a maximum of a certain length of time -- does Dart support this feature to lighten the load on HttpSever?
Thanks for the question! You can set any HTTP header in an HTTP response.
For instance:
onRequest(HttpRequest request, HttpResponse response) {
...
response.headers.add("Cache-Control", "max-age=3600");
...
}
If you want more sophisticated handling, such as respecting Etags or If-Modified-Since, you'll probably have to add them yourself. In general, it makes sense to proxy the Dart HTTP Server behind a server such as Nginx or Apache, and then have that server take care serving all of your static files.
Related
Does/should server code run when the server return status code 304?
I understand that the server should not return anything (client should use the cache), but I cant find any info on whether the server will executed the code in an api endpoint for example.
RFC 2616 section 10.3.5 describes the 304 Not Modified response:
If the client has performed a conditional GET request and access is
allowed, but the document has not been modified, the server SHOULD
respond with this status code. The 304 response MUST NOT contain a message-body
The server will send Date, ETag and/or Content-Location (200 only), or Expires, Cache-Control, and/or Vary if the respond may vary.
What is a Conditional GET?
An cliet application, browser, or proxy with a retained Last-Modified or Etag value will issue a Conditional GET as an initial header only. This allows the client to determine if the resource has been updated.
How does the client know if the resource changed?
Well it depends on how the server is configured.
The Origin Server May:
Ignore caching, serve every request new.
Similar to ignoring, you may develop the application to change query strings. This prevents caching at Proxy Servers and invalidates the client cache.
If configured to do so, issue a Last-Modified or Etag value. Often done for static content. Proxy Servers and Client Caches use to invalidate their version.
A Web application could issue a Last-Modified far
into the future then change the URL to invalidate stale content. This requires the application be developed with this feature in mind.
Resources may also be issued version numbers. This allows them to preserve Proxy Caches but invalidate Client Caches.
Does server code run when the server return status code 304?
Almost Never. Disregarding that it is technically possible for an incorrectly configured application to respond with a 304 Not Modified code instead of a 200 code.
With a ETag and/or Content-Location value, a server (nginx for example) can confirm nothing has changed without issued a call to the application. This also neatly handles resources with version numbers the same way.
For query strings (image.jpg?version=12), the client cache will invalidate the content. A Proxy Server will also invalidate, and the query will be requested fresh.
I understand that the server should not return anything (client should use the cache), but I cant find any info on whether the server will executed the code in an api endpoint for example.
I'm a fan of nginx, here is a good resource on how caching applies to it.
In short, as much as you can do to support various caches between your client and the application the more requests you can support per day.
Considering that the server responds with the following headers:
Cache-Control: public
Expires: <EXPIRATION DATE>
ETag: <HASH VALUE>
Both <EXPIRATION DATE> and the <HASH VALUE> are not changed if the underlying resource is not actually updated.
I'm I correct to expect the following:
all intermediate proxy-servers (including the CDN) will consider this resource public and safe to cache.
all intermediate proxy-servers (including the CDN) as well as browsers will consider this resource fresh until the <EXPIRATION DATE> and will return it from the cache without accessing the network. However, after the <EXPIRATION DATE> they all will use HTTP validation mechanism with every request to check if the resource is outdated.
So, if the resource is updated after the <EXPIRATION DATE> I can safely expect that all the clients will receive the fresh version of the resource with the next request (because HTTP validation will fail due to the ETag's change)?
I'm interested both from the standards perspective (RFCs) as well as from the real life perspective (e.g. known browser and proxy quirks).
I would like my resource to be fresh for e.g. one day from the time the file is actually updated on the server and to be always returned from the cache. However, after one day, I would like all the client to receive the fresh copy only if the file was actually changed (using HTTP validation mechanism).
As Kevin's comment says:
in terms of the standard your analysis is correct
It's difficult to answer in terms of "known browser and proxy quirks" without knowing your engineering requirements. It sounds like you may be serving static content; consider services like S3 and CloudFront.
For this design, from your expectations:
browsers will consider this resource fresh until the and will return it from the cache without accessing the network
Most browsers will still reach out to the network when a resource is directly referenced, even when it's still fresh in their cache. That should be a conditional request, but it's still network traffic.(immutable may help.)
Any cache may evict the resource; for one CDN:
If a file in an edge location isn't frequently requested, CloudFront might evict the file
If your intention is to reduce load on your origin server, it's a good strategy. You're making correct use of Expires, Cache-Control: public and ETag, assuming you're also handling conditional requests correctly. In practice, you should:
be ready for browsers to make more than a single request in a 24 hour period
be ready to tune your CDN and confirm that it's respecting those headers, and that all requests lead to the same cache key
expect more than a single request per day to your origin server
The Prosody server has several modules that allow it to respond to HTTP requests.
For example, mod_http_rest allows me to make a POST request on port 5280 with an XMPP stanza as the payload, and the Prosody server responds by sending that stanza on its way.
However, I am looking for a module that would do the opposite: i.e. take an XMPP message received by the usual means, and make an HTTP POST request to a specified server with that message as the payload.
I can't seem to find any module that will do this. Is there such a thing?
If not, is there any functionality available in the Prosody API that would allow such a module to be written?
Alternatively, are there good reasons why this might be a bad idea?
To answer your question, yes, it's possible for Prosody to make HTTP requests to external services.
The module you describe (take XMPP and forward it over HTTP) did not exist, but there's little reason for it not to. So I just published a module that provides this feature: https://modules.prosody.im/mod_component_http.html
I dont known if there is a module already for that but there is a way to achieve it yourself. I made my own module some months back for specific purposes and I used net.http which as the Docs say:
Is an API for making non-blocking HTTP requests to remote servers.
So you should check the Docs from here so you can use it like:
local req = require "net.http"
req.request( URL, opts, callBack )
Where opts are your options in json, and callBack is a function that defines what you want to do with the response.
I hope it helps.
I'm using playframework and nginx. playframework may add following cookies to http response: PLAY_SESSION, PLAY_FLASH, PLAY_LANG.
I want to make sure that only above cookies (PLAY_*) are allowed in nginx level. If there are other cookies (let's say they're added accidentally) they should be removed by nginx.
How can I allow only predefined cookies in http response in nginx?
PS: If it's not possible to solve this issue in nginx, I need to fix by using playframework.
How cookies work?
First, let's establish what's cookies — they're little pieces of "sticky" hidden information that lets you keep state on your web-site for a given User-Agent. These cookies are often used for tracking users, keeping session and storing minor preference information for the site.
Set-Cookie HTTP response header (from server to client)
Cookies can be set by the server through the Set-Cookie response header (with a separate header for each cookie), or, after the page has already been transferred from the server to the client, through JavaScript.
Note that setting cookies is a pretty complex job — they have expiration dates, http/https settings, path etc — hence the apparent necessity to use a separate Set-Cookie header for each cookie.
This requirement to have a separate header is usually not an issue, since cookies aren't supposed to be modified all that often, as they usually store very minimal information, like a session identifier, with the heavy-duty information being stored in an associated database on the server.
Cookie HTTP request header (from client to server)
Regardless how they were first set, cookies would then included in eligible subsequent requests to the server by the client, using the Cookie request header, with a whole list of eligible cookies in one header.
Note that, as such, these cookies that are sent by the client back to the server is a simple list of name and attribute pairs, without any extra information about the underlying cookies that store these attributes on the client side (e.g., the expiration dates, http/https setting and paths are saved by the client internally, but without being revealed in subsequent requests to the server).
This conciseness of the Cookie request header field is important, because, once set, eligible cookies will be subsequently included in all forthcoming requests for all resources with the eligible scheme / domain / path combination.
Caching issues with cookies.
The normal issue of using cookies, especially in the context of acceleration and nginx, is that:
cookies invalidate the cache by default (e.g., unless you use proxy_ignore_headers Set-Cookie;),
or, if you do sloppy configuration, cookies could possibly spoil your cache
e.g., through the client being able to pass cookies to the upstream in the absence of proxy_set_header Cookie "";,
or, through the server insisting on setting a cookie through the absence of proxy_hide_header Set-Cookie;.
How nginx handles cookies?
Cookie from the client
Note that nginx does support looking through the cookies that the client sends to it (in the Cookie request header) through the $cookie_name scheme.
If you want to limit the client to only be sending certain cookies, you could easily re-construct the Cookie header based on these variables, and send only whichever ones you want to the upstream (using proxy_set_header as above).
Or, you could even make decisions based on the cookie to decide which upstream to send the request to, or to have a per-user/per-session proxy_cache_key, or make access control decisions based on the cookies.
Set-Cookie from the backend
As for the upstream sending back the cookies, you can, of course, decide to block it all to considerably improve the caching characteristics (if applicable to your application, or parts thereof), or fix up the domain and/or path with proxy_cookie_domain and/or proxy_cookie_path, respectively.
Otherwise, it's generally too late to make any other routing decision — the request has already been processed by the selected upstream server, and the response is ready to be served — so, naturally, there doesn't seem to be a way to look into these individual Set-Cookie cookies through normal means in nginx (unless you want to go third-party modules, or lua, or perl), since it'd already be too late to make any important routing decisions for a completed request.
Basically, these Set-Cookie cookies have more to do with the content than with the way it is served or routed, so, it doesn't seem appropriate to have integrated functionality to look into them through nginx.
(If you do need to make routing decisions after the completion of the request, then nginx does support X-Accel-Redirect, as well as some other special headers.)
If your issue is security, then, as I've pointed out above, the upstream developer can already use JavaScript to set ANY extra cookies however they want, so, effectively, trying to use nginx to limit some, but not all, Set-Cookie responses from the server is kind of a pointless endeavour in the real world (as there is hardly any difference between the cookies set through JavaScript compared to Set-Cookie).
In summary:
you can easily examine and reconstruct the Cookie header sent by the client to the server before passing it over to the backend, and only include the sanctioned cookies in the request to upstream backend,
but, unless you want to use lua/perl, or have your own nginx module (as well as possibly quarantine the JavaScript from the pages you serve), then you cannot pass only certain Set-Cookie headers back from the upstream backend to the client with a stock nginx.conf — with the Set-Cookie headers, it's an all-or-nothing situation, and there doesn't seem to be a good-enough use-case for a distinct approach.
For an Nginx solution it might be worth asking over at serverfault. Here is a potential solution via Play Framework.
package filters
import javax.inject._
import play.api.mvc._
import scala.concurrent.ExecutionContext
#Singleton
class ExampleFilter #Inject()(implicit ec: ExecutionContext) extends EssentialFilter {
override def apply(next: EssentialAction) = EssentialAction { request =>
next(request).map { result =>
val cookieWhitelist = List("PLAY_SESSION", "PLAY_FLASH", "PLAY_LANG")
val allCookies = result.newCookies.map(c => DiscardingCookie(c.name))
val onlyWhitelistedCookies = result.newCookies.filter(c => cookieWhitelist.contains(c.name))
result.discardingCookies(allCookies: _*).withCookies(onlyWhitelistedCookies: _*)
}
}
}
This solution utilizes Filters and Result manipulation. Do test for adverse effects on performance.
In how many ways can an HTTP request be generated?
There are endless ways how you can create and from where you can send HTTP requests to a server. Actually your server has no idea, what the origin of such a request is (if it's AJAX or "regular" request, or sent from a console application or ...)
But there are HTTP methods (HTTP verbs) that (can) tell the server about the intent of the request: http://en.wikipedia.org/wiki/HTTP_Verbs#Request_methods
Also you can set headers in a request, for example the content-type or the accepted encoding: http://en.wikipedia.org/wiki/List_of_HTTP_header_fields
Most JavaScript libraries for example set the (non-standard) HTTP header X-Requested-With, so your application can differentiate between regular and ajax requests.
You see, it's even possible to set your own, non-standard headers. There are endless possible combinations...
HttpRequest is a C# class that wraps a petition sent by a client during a Web request.
There are many ways to generate it. The most usual one happens when your browser connects to an ASP.NET website.
You can, for example, create your own custom HttpRequest to petition a specific web page from a C# console application.
Are you trying to achieve something more specific?