Here is our current infrastructure:
2 web servers behind a shared load balancer
dns is pointing to the load balancer
web app is done in asp.net, with wcf services
My question is how to set up the SSL certificate to support https connection.
Here are 2 ideas that I have:
SSL certificate terminates at the load balancer. secure/unsecure communication behind the load balancer will be forwarded to 2 different ports.
pro: only need 1 certificate as I scale horizontally
cons: I have to check secure or not secure by checking which port the request is
coming from. doesn't quite feel right to me
WCF by design will not work when IIS is binded 2 different ports
(according to this)
SSL certificate terminates on each of the server?
cons: need to add more certificates to scale horizontally
thanks
Definitely terminate SSL at the load balancer!!! Anything behind that should NOT be visible outside. Why wouldn't two ports for secure/insecure work just fine?
You don't actually need more certificates at all. Because the externally seen FQDN is the same you use the same certificate on each machine.
This means that WCF (if you're using it) will work. WCF with the SSL terminating on the external load balancer is painful if you're signing/encrypting at a message level rather than a transport level.
You don't need two ports, most likely. Just have the SSL virtual server on the load balancer add an HTTP header to the request and check for that. It's what we do with our Zeus ZXTM 5.1.
You don't have to get a cert for every site there are such things as wildcard certs. But it would have to be installed on every server. (assuming you are using subdomains, if not then you can reuse the same cert across machines)
But I would probably put the cert on the load balancer if not just for the sake of easy configuration.
Related
I use Nginx to manage a lot of my web services. They listens different port, but all accessed by the reverse proxy of Nginx within one domain. Such as to access a RESTful-API server I can use http://my-domain/api/, and to access a video server I can use http://my-domain/video.
I have generated a SSL certificate for my-domain and added it into my Nginx conf so my Nginx server is HTTPS now -- But those original servers are still using HTTP.
What will happen when I visit https://my-domain/<path>? Is this as safe as configuring SSL on the original servers?
One of the goals of making sites be HTTPS is to prevent the transmitted data between two endpoints from being intercepted by outside parties to either be modified, as in a man-in-the-middle attack, or for the data to be stolen and used for bad purposes. On the public Internet, any data transmitted between two endpoints needs to be secured.
On private networks, this need isn't quite so great. Many services do run on just HTTP on private networks just fine. However, there are a couple points to take into consideration:
Make sure unused ports are blocked:
While you may have an NGINX reverse proxy listening on port 443, is port 80 blocked, or can the sites still be accessed via HTTP?
Are the other ports to the services blocked as well? Let's say your web server runs on port 8080, and the NGINX reverse proxy forwards certain traffic to localhost:8080, can the site still be accessed at http://example.com:8080 or https://example.com:8080? One way to prevent this is to use a firewall and block all incoming traffic on any ports you don't intend to accept traffic on. You can always unblock them later, if you add a service that requires that port be opened.
Internal services are accessible by other services on the same server
The next consideration relates to other software that may be running on the server. While it's within a private ecosystem, any service running on the server can access localhost:8080. Since the traffic between the reverse proxy and the web server are not encrypted, that traffic can also be sniffed, even if authorisation is required in order to authenticate localhost:8080. All a rogue service would need to do is monitor the port and wait for a user to login. Then that service can capture everything between the two endpoints.
One strategy to mitigate the dangers created by spyware is to either use virtualisation to separate a single server into logical servers, or use different hardware for things that are not related. This at least keeps things separate so that the people responsible for application A don't think that service X might be something the team running application B is using. Anything out of place will more likely stand out.
For instance, a company website and an internal wiki probably don't belong on the same server.
The simpler we can keep the setup and configuration on the server by limiting what that server's job is, the more easily we can keep tabs on what's happening on the server and prevent data leaks.
Use good security practices
Use good security best practices on the server. For instance, don't run as root. Use a non-root user for administrative tasks. For any services that run which are long lived, don't run them as root.
For instance, NGINX is capable of running as the user www-data. With specific users for different services, we can create groups and assign the different users to them and then modify the file ownership and permissions, using chown and chmod, to ensure that those services only have access to what they need and nothing more. As an example, I've often wondered why NGINX needs read access to logs. It really should, in theory, only need write access to them. If this service were to somehow get compromised, the worst it could do is write a bunch of garbage to the logs, but an attacker might find their hands are tied when it comes to retrieving sensitive information from them.
localhost SSL certs are generally for development only
While I don't recommend this for production, there are ways to make localhost use HTTPS. One is with a self signed certificate. The other uses a tool called mkcert which lets you be your own CA (certificate authority) for issuing SSL certificates. The latter is a great solution, since the browser and other services will implicitly trust the generated certificates, but the general consensus, even by the author of mkcert, is that this is only recommended for development purposes, not production purposes. I've yet to find a good solution for localhost in production. I don't think it exists, and in my experience, I've never seen anyone worry about it.
I have got a setup like this
Load balancer
Machine 1 - haproxy load balancer
Machine 2 - haproxy load balancer
Web servers
Machine 1 - nginx with app
Machine 2 - nginx with app
Now where should I set up SSL certificate. On loadbalancers or web servers or on both?
What is the correct way of doing it?
The "correct way" to do this depends on your setup. If your load balancers are on the same machines as your webservers, it doesn't matter which you choose to put the cert on. If they are on different servers, encryption depends on how important security is for these particular web apps. If you put the certs on the load balancers you will have unencrypted traffic visible to anyone in your network (as it goes from load balancer to server). If you put certs on your nginx server you will have encryption all the way through to the local server, but you will have to change your haproxy a little to have it route encrypted traffic properly. You also will not be able to route off the url path. You can also put certs on both to be able to route off the url path, but that is a little more to manage (two certs vs one). Overall it's probably best to put the cert on nginx server, assuming your don't need to do any routing in the load balancer off of the url. Also definitely do your own research.
I'm now reading design of Instagram and I found such a description of their load balancing system.
Every request to Instagram servers goes through load balancing machines; we used to run 2 nginx machines and DNS Round-Robin between them. The downside of this approach is the time it takes for DNS to update in case one of the machines needs to get decomissioned. Recently, we moved to using Amazon’s Elastic Load Balancer, with 3 NGINX instances behind it that can be swapped in and out (and are automatically taken out of rotation if they fail a health check). We also terminate our SSL at the ELB level, which lessens the CPU load on nginx. We use Amazon’s Route53 for DNS, which they’ve recently added a pretty good GUI tool for in the AWS console.
The question is. Am I right that for now they have a DNS Server which uses RR to decide on which nginx server to send the request. And each of this nginx servers at their turn resends the request to a cluster?
And the second question is. What the difference between nginx and load balancer. Why cannot we use nginx instead?
For your first question, I believe the answer seems to be that Instagram now uses Route53 to map DNS to an Elastic Load Balancer, which does two things: It routes traffic fairly evenly to three NGINX load balancers, and it provides SSL for all traffic. The NGINX servers then act as load balancers to content/application servers further down the stack. Using an ELB instead of round-robin DNS means they can add/remove/update instances attached to the ELB without ever having to worry about DNS updates or TTL.
As for the second question, you can use NGINX just as easily as HAproxy or other services to do load balancing. I am sure that part of the appeal to Instagram in choosing NGINX is its incredible speed and that it's asynchronous and "event-driven" instead of threaded like Apache2. When set up properly, that can mean less headaches under heavy loads.
What is the better approach for load balancing on web servers? My services run in .NET and Mono, so they could be hosted on IIS or Apache2, and the will have to provide SSL connection.
I've read two main approaches, store the state in a common server and use sticky sessions, there is any other else?
I've read 3 diffent things about sticky sessions:
1)the load balancing device will know with which server did you start the connection and all the further connections from that host will be routed to the same server.
2)the load balancing devide read a cookie named: JSESSIONID
3)the load balancing devide read a cookie named: ASPSESSIONID
I'm a little bit confused, what will happen exactly? As the connections will be SSL there is not a chance for the load balancing devide of read the cookies, so then what?
About store the estate in a common server, what solutions do you know? I've read memcache is a good solution but is there any other else?
Cheers.
When using SSL with a load balancer, it is common to put the SSL certificate on the load balancing server, and not on the back end servers. In this way you only need 1 certificate on 1 server. The load balancer then talks to the back end servers using plain HTTP. This obviously requires that your back end servers are not directly accessible from the internet.
So, if the load balancer is responsible for decrypting the request, it will also be able to inspect the request for a jsessionid.
Sticky sessions work well with Apache as load balancer. You should check out the Apache modules mod_proxy and mod_proxy_balancer.
Generally SSL load balancing means that the client is talking to the load balancer over HTTPS, and the load balancer is talking to the web server via HTTP.
Some load balancers are smart enough to establish an SSL session with the web server (so it can read cookies) and maintain a separate SSL session with the client.
And, some load balancers can maintain stickiness without using web server cookies. My load balancers are able to send their own cookies to the client (they have a bunch of other stickiness settings as well).
Is it possible to have a FTP server using SSL on a application server that does not use SSL?
How would you setup an ASP.NET 2.0 to consume a SSL certificate?
This certianly sounds possible but is it advisable, is it good practice?
The choice on using SSL us made on a per connection level, usually determined by the IP port being used (i.e. will be set up between client and server before any application code involved).
The same service/content could be set up on multiple ports each with a different choice for SSL.
The certificate is per host name, but servers can generally support appearing under different names.
To use SSL with ASP.NET takes nothing special, it just works once the IIS web site is configured to support SSL (or to require it: when connections to port 80 for HTTP are redirected to the SSL port); this choice can be made on a per folder basis.
FTP is at the application layer, and SSL is lower, at the presentation layer. The SSL sessions are on a per connection basis. Take a look at the Wikipedia page. The SSL connection is established before anything happens with your application. Your FTP server probably isn't running inside your .NET application server, is it? You should be able to setup an SSH server listening for SCP connections separately. If it really does run from inside your app server, you should be able to listen on a separate port for the SSL connection.
Short of any of that, heres a good link for configuring SSL in IIS. You don't have to make the certificates mandatory. That way you can allow unsecured traffic and secure traffic if that fits in with your application model.
While protecting an application with SSL is always a good idea, it is technically not trivial.
Having a web application protected with SSL requires the webserver to be reachable on a new port (443/https instead of 80/http). This has to be configured "system-wide". Also, there may be only one certificate per IP-address, which is often a problem when hosting multiple domains on the same server.