I am using the (free) version of varnsih (version 6).
Here is my configuration :
2 varnish hosts (varnish_A, varnish_B)
2 webs servers (node_A, node_B)
I only cache all the pages /app/api/client (GET) from my web servers and it's working very well
But I have a problem, when there is an update on the page (from a PUT) /app/api/client/test_data I added a BAN in the sub vcl_recv part
if (req.url ~ "^app/api/client($|/.*)" && ( req.method == "POST" || req.method == "PUT" )) {
ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
std.syslog(180, "[debug]["+req.method+"] - Starting Purge of the cache for: " + req.http.host + req.url );
return (pass);
set req.backend_hint = web_node.backend();
}
The ban is working, but only on the current varnish host (for example varnish_B), so the other one (varnish_A) still has the old version of the object...
Does any one know how to "share" the ban on all host ?
Thank you !
Send a ban to both Varnish servers
It's just a matter of running the ban on both Varnish servers, rather than on one. You can probably build that into your content management flow where the 2 endpoints are called.
Varnish Broadcaster
Alternatively you could sign up for Varnish Enterprise and use the Varnish Broadcaster as your banning endpoint. It will distribute the request to all or to specific Varnish nodes. See https://docs.varnish-software.com/varnish-broadcaster/examples/ for examples.
Optimizing your bans
Just a sidenote, using req.url and req.http.host is not ideal for bans. When the ban is executed, the ban lurker doesn't have access to these variables, which means the banning is the responsibility of the next request.
A way to circumvent this limitation is by adding the following VCL code:
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
You could then run the following ban:
ban("obj.http.x-host ~ " + req.http.host + " && obj.http.x-url == " + req.url);
See https://www.varnish-software.com/developers/tutorials/ban/ for a full tutorial on Varnish bans.
Related
Just new to Varnish. It's getting hard, more than expected :-(
I'm trying to improve some php code, developed some time ago, using varnish.
This code uses just two cookies: PHPSESSID and LANGUAGE
All pages set PHPSESSID cookie if it's not defined. However this cookie for anonymous sessions it's only used in one page.
Let's say I have Page1, Page2, Page3 and Page4. My configuration should be as follows:
Page1, Page2 and Page3 need LANGUAGE cookie and should be cached with that cookie: one cache for each language and page.
Page4 needs PHPSESSID and LANGUAGE cookies, and shouldn't be cached as it's specific for each user.
My default.vlc is not working properly, so any orientation would be really apreciated. Maybe I've missunderstood some concepts.
sub vcl_init {
# When requests come to Varnish I need to remove PHPSESSID so it's not used for the hash in caching. Page4 doesn't need caching as it's specific for each user:
if (req.http.host ~ "Page4") {
return(pass);
}
# remove PHPSESSID so pages1, 2, and 3 get cached just once for everyuser but in all languages.
if ((req.url !~ "page4")) {
set req.http.Cookie = regsuball(req.http.Cookie, "PHPSESSID=[^;]+(; )?", "");
}
return (hash);
}
I need to cache webpages with LANGUAGE cookie so I include it in vcl_hash:
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# hash cookies for requests that have them
if (req.http.Cookie) {
hash_data(req.http.Cookie);
}
}
How can I remove just PHPSESSIONID?
sub vcl_backend_response {
# Called after the response headers has been successfully retrieved from the backend.
if (!(bereq.url ~ "Page4")) {
unset beresp.http.set-cookie;
}
return (deliver);
}
You are on the right track. If I understand your question, the thing that does not work is that you remove all cookies server response (if it's not Page4) instead of just removing PHPSESSID.
You can do a regexp in your sub vcl_backend_response to remove only phpsessionid if url is not Page4.
beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "PHPSESSID=[^;]+(; )?", "")
Alternatively, if you use varnish 4 or above, you should use the vmod cookie which makes cookie handling much easier (no more need for regexp).
I wonder if I can get some help. I am running a VPS with Varnish. This VPS is predominantly WordPress but does have a Joomla site running too. For some reason my Varnish config file decided it was going to go back to default so I have been having a range of problems as I was silly enough not to save my custom config file!
Here is my current config file:
# This is a basic VCL configuration file for varnish. See the vcl(7)
# man page for details on VCL syntax and semantics.
#
# Default backend definition. Set this to point to your content
# server.
#
backend default {
.host = "LIVE IP";
.port = "8080";
.max_connections = 800;
}
acl purge { "localhost"; "127.0.0.1"; }
sub vcl_recv {
set req.grace = 2m;
# Set X-Forwarded-For header for logging in nginx
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
# Remove has_js and CloudFlare/Google Analytics __* cookies and statcounter is_unique
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js|is_unique)=[^;]*", "");
# Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
# Either the admin pages or the login
if (req.url ~ "/wp-(login|admin|cron)") {
# Don't cache, pass to backend
return (pass);
}
# Remove the wp-settings-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");
# Remove the wp-settings-time-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-time-1=[^;]+(; )?", "");
# Remove the wp test cookie
set req.http.Cookie = regsuball(req.http.Cookie,
"wordpress_test_cookie=[^;]+(;)?", "");
# Static content unique to the theme can be cached (so no user uploaded images)
# The reason I don't take the wp-content/uploads is because of cache size on bigger blogs
# that would fill up with all those files getting pushed into cache
if (req.url ~ "wp-content/themes/" && req.url ~
"\.(css|js|png|gif|jp(e)?g)") {
unset req.http.cookie;
}
# Even if no cookies are present, I don't want my "uploads" to be cached due to their potential size
if (req.url ~ "/wp-content/uploads/") {
return (pass);
}
# any pages with captchas need to be excluded
if (req.url ~ "^/contact/" || req.url ~ "^/links/domains-for-sale/")
{
return(pass);
}
# Check the cookies for wordpress-specific items
if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
# A wordpress specific cookie has been set
return (pass);
}
# allow PURGE from localhost
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
# Force lookup if the request is a no-cache request from the client
if (req.http.Cache-Control ~ "no-cache") {
return (pass);
}
# Try a cache-lookup
return (lookup);
}
sub vcl_fetch {
#set obj.grace = 5m;
set beresp.grace = 2m;
}
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
}
The issue I am having is that my WP core files are moved to a subfolder called 'core' and I change my login URL to /administrator instead of wp-login.php. so to login, I'd go to either domain.com/administrator or domain.com/core/administrator. If I type the domain sans /core/ it would direct to */core/administrator.
The way that Varnish works, it doesn't allow me to log into the WP-admin side of the site which I think is due to the cookies. I added /administrator to the VCL-config text but it didn't seem to work.
Is anyone able to please help me out with this as I'd rather not revert all my installations back to wp-login.php.
I also don't seem to be able to login to SSH and purge the cache like I could before. I honestly can't remember the config I had before but I am sure I am missing something, so if anyone is able to improve on this code to make it work as best as it can (or whether I have missed important things out entirely, specifically with Joomla) then help would be greatly appreciated.
I don't make theme on the fly but I do have some users logging into various sites hosted on the VPS to add/change posts so getting the cache to purge properly would be fantastic as I'm beginning to pull my hair out!
Thanks everyone.
To make PURGE work, you can try to add your VPS hostname in "acl purge" section. I know I had to do that on my VPS and Varnish v4.
And for wp admin part, it should not be caches at all, so try changeing this part:
# Either the admin pages or the login
if (req.url ~ "/wp-(login|admin|cron)") {
# Don't cache, pass to backend
return (pass);
}
into this:
# Either the admin pages or the login
if (req.url ~ "/core/administrator" || req.url ~ "/administrator") {
# Don't cache, pass to backend
return (pass);
}
I'm attempting to get Varnish to cache two different domains with blogs, but upon adding the second one, the previous one stops working,
The basic setup is as following:
backend default {
.host = "127.0.0.1";
.port = "81";
}
backend onedomain {
.host = "127.0.0.1";
.port = "81";
}
backend newdomain {
.host = "127.0.0.1";
.port = "81";
}
acl purge {
"localhost";
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
#Bypass large files
if (req.http.x-pipe-mark && req.restarts > 0) {
return(pipe);
}
# all domains in here will return a "pass" which means they won't be cached
if (req.http.host ~ "(www\.)?(domain1.com|domain2.com)") {
return (pass);
}
#else check if something we're going to cache
else if(req.http.host ~ "(www\.)?(onedomain.nu)") {
set req.http.host = "onedomain.com";
set req.backend_hint = onedomain;
}
else if(req.http.host ~ "(www\.)?(newdomain.com)") {
set req.http.host = "newdomain.com";
set req.backend_hint = newdomain;
}
else {
return (pass);
}
Newdomain loads fine while domain4 just sends me to an infinite redirect loop (according to the chrome error)
I added the full config in a pastebin: http://pastebin.com/J1Hb76dZ
I realize Varnish doesn't send any redirect commands itself, the site works on the old configuration, it's only when I try this that the redirect issue arises on one of the websites.
Is there anyone that has experience with this happening and can suggest what to do?
Old question, but try modifying the vcl_hash subroutine. This worked for me on a single site, that included multiple redirects for http -> https and domain.com -> www.domain.com. It should also configure the hash to tell your different domains apart, as that was necessary for me to store all the redirects separate from the site data that caused the dreaded "too many redirects" errors. You may need to adjust/remove the X-Forwarded-Proto header as I am behind a load balancer.
sub vcl_hash {
hash_data(req.http.host);
hash_data(req.url);
hash_data(req.http.X-Forwarded-Proto);
return(hash);
}
I'm very lost on this problem, and I don't know where could be the problem, so, I hope that you could help me.
I have an HTTP BASIC authentification with symfony, and I'm trying to reach an url which is protected by this auth, with an tag in a Drupal page. Every requests are send to a Varnish
I give username and password in the url like that :
<esi:include src="http://admin:adminpass#api.dev:8081/app.php/next"/>
In my varnish configuration file, I have only that lines for auth.http:
if (req.http.Authorization) {
return (pass);
}
My backend for Symfony is working well without http authentification, and the http authentification is working well when there's not Varnish and esi tag.
If anyone have an idea of the problem, please, tell me, even if it's wrong =)
ESI in varnish doesn't work like an iframe or link tag in a browser in that it doesn't connect to whatever url you give it. ESI just starts a new request within varnish and goes through the workflow (vcl_recv, etc).
You are expecting varnish to act like an http client, parsing the url, setting the authorization header, setting a host header to api.dev:8081 and initiating a new http connection/request which it will not. In this case, my guess is it starts a new req with req.url set to /app.php/next inheriting the headers from the request for the parent resource (containing the esi tag) or possibly just ignores the esi tag completely.
The way to accomplish what you want to do is (in vcl_recv):
if (req.esi_level > 0 && req.url == "/app.php/next") {
set req.http.Authorization = "BASIC [base64 encoded admin:adminpass]"
return (pass);
}
and then the esi tag should look like <esi:include src="/app.php/next" />
If you need the ESI request to hit a different backend server, you need to add that server as a different named backend:
backend authorization_needed {
.host = "api.dev";
.port = "8081";
}
and in vcl_recv, tell varnish to use it for esi requests:
if (req.esi_level > 0 && req.url == "/app.php/next") {
set req.http.Authorization = "BASIC [base64 encoded admin:adminpass]"
set req.backend = authorization_needed;
return (pass);
}
you may also need to set req.http.Host in that if block if the backend responds to a different virtual host than "api.dev".
Update:
Since basic authorization is coming from the client, and you are calling return (pass) when req.http.Authorization is present, varnish will not ESI process those pages. You must explicitly enable esi in vcl_fetch() which is not called when you pass.
So to pass authorization for the ESI fragments but not for the parent page, change in vcl_rev:
if (req.http.Authorization && req.esi_level == 0) {
set req.http.X-Esi-Authorization = req.http.Authorization;
unset req.http.Authorization;
}
else if (req.http.X-Esi-Authorization && req.esi_level > 0 ) {
set req.http.Authorization = req.http.X-Esi-Authorization;
return (pass);
}
And add to vcl_fetch:
if (req.http.X-Esi-Authorization) {
set beresp.do_esi = true;
}
The net effect is the parent response is cacheable and will process esi, the esi fragments themselves will always be passed to the backend with the client's authorization header.
I am trying to reverse proxy several socket.io websockets and also rewrite the URLs so that I can have a front end server that can connect to several backend tty.js instances.
For example:
http://mysite.com/server1 reverse proxies to http://server1/ which is running tty.js
http://mysite.com/server2 reverse proxies to http://server2/ which is running tty.js
For HTTP any old proxy works fine. However when I rewrite the URLs instead of the websocket looking like:
http://mysite.com/server1/socket.io/1/?t=1354135029745 -> http://server1/socket.io/1/?t=1354135029745
it looks like:
http://mysite.com/socket.io/1/?t=1354135029745
This of course goes unhandled. I assume the rewrite happens and so the socket assumes it's "connection" is at /socket.io/1/?t=1354135029745 instead of /server1/socket.io/1/?t=1354135029745
Is there some way to do URL rewriting as well as proxy a websocket?
So far I have tried both Varnish, node-http-proxy and Nginx using the TCP proxy module and I haven't been able to figure out how to get this to work.
Right now my Varnish config looks like:
backend backend1 {
.host = "1.1.1.1";
.port = "8080";
.connect_timeout = 1s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
backend backend2 {
.host = "2.2.2.2";
.port = "8080";
.connect_timeout = 1s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
sub vcl_recv {
set req.grace = 120s
if(req.url ~ "^/server1/") {
set req.url = regsub(req.url, "^/server1/", "/");
set req.backend = backend1;
return(pipe);
}
if(req.url ~ "^/server2/") {
set req.url = regsub(req.url, "^/server2/", "/");
set req.backend = backend2;
return(pipe);
}
}
sub vcl_pipe {
if(req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
} else {
set bereq.http.connection = "close";
}
return(pipe);
}
I am forced to leave out the obligatory:
if(req.url ~ "^/socket.io/") {
set req.backend = backend1;
return(pipe);
}
since I have multiple backends. Is there a way to get this to work using node-http-proxy, Nginx or Varnish? Or if none of those have the ability, would something like haproxy or really anything else be able to?
My first inclination would be to set a custom header so when the URL does get rewritten it knows which backend it should hit, but I'm not sure if/how that would work.
Edit 1: I was able to sort of fix this issues by also pulling the req.http.referer and adding that to the socket.io statement:
if(req.url ~ "^/socket.io/" && req.http.referer ~ "server1"){
set req.backend = backend1;
return(pipe);
}
However it appears to be a little wonky and seems to revert socket.io to xhr polling which is not ideal but better than nothing.
Edit 2: After testing it out with a few backends what happens is once long polling starts it breaks the rest of the proxy. As soon as I refresh the page now the reverse proxy instead of going:
http://mysite.com/server1 -> http://server1
goes:
http://mysite.com/ -> http://server1
until I reload varnish and then goes back.
Most likely you also need to change the Host: header as well in your vcl_recv():
sub vcl_recv {
set req.grace = 120s
if(req.url ~ "^/server1/") {
set req.url = regsub(req.url, "^/server1/", "/");
set req.http.host = "server1";
set req.backend = backend1;
return(pipe);
}
if(req.url ~ "^/server2/") {
set req.url = regsub(req.url, "^/server2/", "/");
set req.http.host = "server2";
set req.backend = backend2;
return(pipe);
}
}
That should take care of your initial problem.
However using pipe in Varnish has it's pitfalls which you might want to consider. The most important of which is the lack of X-Forwarded-For header after the first request.