I need to know how many traffic each customer use in our CDN. As I undersand, varnishstat command return statistic for all backend but i need to know traffic for each backend.
I don't think you can.
But with Varnish 3, you can customize log format and then parse/split your log file.
Related
When working with the PageSpeed product from Huckabuy which uses Cloudflare workers to implement some page speed boosters I want to be able to bypass the behavior of the boosters without having to reconfigure the settings. Are there any ways to accomplish this that are exposed by Huckabuy or perhaps a generic way using a URL query string parameter or an HTTP header to bypass any given workers in Cloudflare?
Below is an example of what I'd like to be able to do.
https://www.example.com?huckabuy_pagespeed=false
If that's not possible then perhaps something specific to Cloudflare like the example below.
https://www.example.com?disable_cf_workers=true
Or potentially the following example HTTP header.
DISABLE_CF_WORKER: true
I don't know anything about Huckabuy specifically, but there is no general way to bypass Cloudflare Workers with special request properties. Workers are often used to implement security policies, so it's important that they cannot be bypassed.
Of course, any particular Worker is free to implement its own bypass mechanism if it so chooses.
I'm new to Jenkins and I don't know if with it I can do what I want. I need to send a file from one point (in another PC) to an endpoint in a server where Jenkins exist, I believe the best way to send the file is with a POST Request because I can send metadata that could help me to process the file. The thing is I don't know if I can configure this endpoint with Jenkins so it could recieve the file or I have to use another thing to create this endpoint and later it should call Jenkins. I don't know if a Jenkins' Plugin could help me here or something.
What I was considering was maybe using Kafka to handle the requests (I believe it can do that) with a queue and it later could call Jenkins to execute the actions I really want with the file that arrived and the details that came in the metadata.
BTW, I'm using Ubuntu 20, but this should be able to run in Windows too, but for now I only want to test this in Ubuntu.
Also, if anyone could recomend me some software that is like Jenkins only to see other options. The main feature that I found in Jenkins that I want to use for now besides what I'm asking in this post is the capability of process the emails that arrives in an specific mail, this is done with a Plugin, the CI/CD features are welcome, tho, I haven't use them yet.
Thanks.
Need some advice before starting develop some things.. I've 15 WordPress websites on different installs, and I've remote server which gets data 24/7 from those websites.
I've reached a point that I want the server to modify the websites based on his calculated data.
The things are this:
Should I allow the server the access the WP DB remotely and modify things without using WP on the circle?
Or, use WP REST API and supply some secured routes which provide data and accept data and make those changes?
My instinct is to use the WP API, but. After all its a PHP (nginx+apache) which have some limits (timeout for example) and I find it hard to run hard and long process on the WP itself.
I can divide the tasks to different levels, for example:
fetching data (simple get)
make some process on the remote server
loop and modify in small batches to another route
My concerns are that this circle require perfect match between remote server and WP API, and any change or fix on WP side brings plugins update on the websites which is not much fun.
Hope for any ideas and suggests to make it forward.
"use WP REST API and supply some secured routes which provide data and accept data and make those changes", indeed.
i don't know why timeout or another limits may cause a problem - but using API is the best way for such kind of cases. You can avoid timeout problems with some adjustments on web servers side.
Or you can increase memory, timeout limit exclusively for requested server.
f.e.
if ($_SERVER["remote_attr"]=='YOUR_MAIN_SERVER_IP') {
ini_set('max_execution_time',1000);
ini_set('memory_limit','1024M');
}
Parse.com has a very useful tool in which it graphs the number of requests per second made to your application across a given time. I was wondering for an Nginx configuration, is there any tool that does the same time?
Using Nginx Plus would be another option to parsing the logs.
You can use the ngx_http_stub_status module (http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) to export basic information, combined with collectd's nginx plugin (https://collectd.org/wiki/index.php/Plugin:nginx).
I want to let my clients speak directly with ElasticSearch REST API, obviously preventing them from performing any data or configuration change.
I had a look at ElasticSearch REST interface and I noticed the pattern: HTTP GET requests are pretty safe (harmless queries and status of cluster).
So I thought I can use Cloudfront as a CDN/Proxy that only allows GET/HEAD methods (you can impose such restrict it in the main configuration).
So far so good, all is set up. But things don't work because I would need to open my EC2 security group to the world in order to be reachable from Cloudfront! I don't want this, really!
When I use EC2 with RDS, I can simply allow access to my EC2 security group in RDS security groups. Why can't I do this with CloudFront? Or can I?
Ideas?
edit: It's not documented, but ES accepts facets query, which involve a (JSON) body, not only with POST, but also with GET. This simply breaks HTTP recommendation (as for RFC3616) by not ignoring the body for GET request (source).
This relates because, as pointed out, exposing ES REST interface directly can lead to easy DOS attacks using complex queries. I'm still convinced though, having one less proxy is still worth it.
edit: Other option for me would be to skip CloudFront and adding a security layer as an ElasticSearch plugin as shown here
I ended coding with my own plugin. Surprisingly there was nothing quite like this around.
No proxies, no Jetty, no Tomcat.
Just a the original ES rest module and my RestFilter. Using a minimum of reflection to obtain the remote address of the requests.
enjoy:
https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin
Note that even a GET request can be harmful in Elasticsearch. A query which simply takes up too much resources to compute will bring down your cluster. Facets are a good way to do this.
I'd recommend writing a simple REST API you place in front of ES so you get much more control over what hits your search cluster. If that's not an option you could consider running Nginx on your ES boxes to act as a local reverse proxy, which will give you the same control (and a whole lot more) as CloudFront does. Then you'd only have to open up Nginx to the world, instead of ES.
A way to do this in AWS would be:
Set up an Application Load Balancer in front of your ES cluster. Create a TLS cert for the ALB and serve https. Open the ES security group to the ALB.
Set up CloudFront and use the ALB as origin. Pass a custom header with a secret value (for WAF, see next point).
Set up WAF on your ALB to only allow requests that contain the custom header with the secret value. Now all requests have to go through CloudFront.
Set up a Lambda#Edge function on your CloudFront distribution to either remove the body from GET requests, or DENY such requests.
It’s quite some work, but there’s advantages over the plugin, e.g.:
CloudFront comes with free network DDOS protection
CloudFront gives your users lower latency to ES because of the fast CloudFront network and global PoP’s.
Opens many options to use CloudFront, WAF and Lamba#Edge to further protect your ES cluster.
I’m working on sample code in CDK to set all of this up. Will report back when that’s ready.