I've been building Azure Bicep files with the goal of having our infrastructure codified.
All is going well except for AKS. Reading and experimenting I think I have two options.
AKS has pods with Nodejs or .net services running in them which need environment variables like database connection strings. These can be passed in at the deploy stage of each node/.net or they can be 'included' in each AKS instance.
Am I on the right track and does one have advantages over the other?
The IaC Code for your AKS should not be mixed with deployment code of workload (your Nodejs or .Net Pods).
I would also not recommend to use ENV variables for secrets and connections strings. Kubernetes upstream decided that CSI (Container Storage Interface) is the way to go.
With that being said, you can write a Bicep deployment that deploys AKS & Azure Key Vault. Enable the azureKeyvaultSecretsProvider add-on for the AKS to sync secrets from Azure KeyVault to Kubernetes secrets or directly as files into pods.
After this you write you workload deployment of Nodejs and .Net Pods and refer the AZURE KEY VAULT PROVIDER FOR SECRETS STORE CSI DRIVER. This also make you more independent if you create more cluster etc.
Related
I'm currently in the process of migrating 3 applications from Elastic Kubernetes Service (EKS) to ECS Fargate. Each application is built with Node JS .The current setup seems to be only 1 load balancer in front of one application and the other two applications are accessed through that one load balancer. This is currently how all three applications is accessed:
first_app.example.com
first_app.example.com/second_app
first_app.example.com/third_app
The front end of each application is being powered by an nginx proxy in EKS. I'm not entirely sure if I need nginx to be in ECS Fargate because the application load balancer I'm planning on to use will have an SSL cert integrated with it for redirects to HTTPS from HTTP. I'm a little unclear how to approach moving these applications to Fargate. Additionally, the third app has 3 additional functions:
Apollo GraphQL (abstraction layer between the front end & back end)
CSV
File Manager
This functionality also needs to be implemented on the Fargate side.
Currently I have setup one ECS Fargate cluster, one ECS Service, and one task definition. The task definition currently has the following 7 ECR images:
app_one_front_end
app_two_front_end
app_three_front_end
app_three_csv_job
app_three_file_manager_job
app_three_graphql
nginx ??
All of these images are stored in ECR. However I don't believe I need nginx in this Fargate cluster.
I'm a little unsure how to approach the architecture for this set of applications. It seems I can only have one task definition running on a service, that's why all containers were implemented into one task definition. The service can then be associated with an application load balancer where I set path based routing to access each application.
Any advice on how to approach this migration would be appreciated.
Thanks!
Each Kubernetes Replica Set should be converted to an ECS Service. Each Kubernetes Pod would be converted to an ECS Task.
Kubernetes Replica Set == ECS Service
Kubernetes Pod == ECS Task
If you had multiple Replica Sets in Kubernetes, in order to scale your pods independently, then in order to have the same scalability in ECS you would configure them as separate services with independent scaling configurations.
You are correct in that you probably don't need the Nginx container in ECS.
It seems I can only have one task definition running on a service, that's why all containers were implemented into one task definition.
Services can communicate with each other. You would enable ECS Service Discovery to facilitate that. However it is fine to have them all in the same Task/Service if they don't need to be scaled out independently.
Also, multiple services can be associated with a single Application Load Balancer by creating different listener rules in the load balancer that map to different Target Groups, if that is something you need. You might need to have multiple Target Groups even if you only have a single ECS Service, because you will need to map different load balancer listeners to different containers in your task. That basically allows the Application Load Balancer to perform the job that Nginx was doing in Kubernetes.
I would like to know if I can make the airflow UI accessible to all people who have a user, web page type. For this, I would have to connect it to a server, no? Which server do you recommend for this? I was looking around and some were using Amazon EC2.
If your goal is just making the airflow UI visible to public, there is a lot of solutions, where you can do it even in your local computer (of course it is not a good idea).
Before choosing the cloud provider and the service, you need to think about the requirements:
in your team, do you have the skills and the time to manage the server? if no you need a managed service like GCP cloud composer or AWS MWAA.
which executor yow want to use? KubernetesExecutor? CeleryExecutor on K8S? if yes you need a K8S service and not just a VM.
do you have a huge loading? do you need a HA mode? what about the scalability?
After defining the requirements, you can choose between the options:
Small server with LocalExecutor or CeleryExecutor on a VM -> AWS EC2 with a static IP and Route 53 for DNS name
A scalable server in HA mode on a K8S cluser -> AWS EKS or google GKE
A managed service and focusing only on the development part -> google cloud composer
Hello I have worked on API gateway for a identity service. I used HttpClient and currently call the identity service with localhost. My worry is when we deploy them we plan to have identity service in a cluster in azure there will be a need to use DNS name then. Will the calls still work like they do now using the localhost just by switching to production it will use the DNS name of the cluster? Or are there any other configurations that need to be done?
I have deployed API manager 4.0.0 All-in-one on 2 VMs. I am using Mysql as DB which is on a separate VM and I am sharing the databases as mentioned in the document. Now I am trying to cluster these 2 nodes as mentioned in this document. There are a few things that are not clear to me from this document.
If I give the https_endpoint with domain "gw.am.wso2.com" behind the gateway. environments, I need to add 8243 in my reverse proxy server, or not, it fails when trying out API in the console. Why?
I have successfully deployed a Flask application to AWS Elastic Beanstalk. The application uses an SQLAlchemy database, and I am using Flask-Security to handle login/registration, etc. I am using Flask-Migrate to handle database migrations.
The problem here is that whenever I use git aws.push it will push my local database to AWS and overwrite the live one. I guess what I'd like to do is only ever "pull" the live one from AWS EB, and only push in rare circumstances.
Will I be able to access the SQLAlchemy database which I have pushed to AWS? Or, is this not possible? Perhaps there is some combination of .gitignore and .elasticbeanstalk settings which could work?
I am using SQLite.
Yes, your database needs to not be in version control, it should live on persistent storage (most likely the Elastic Block Storage service (EBS)), and you should handle schema changes (migrations) using something like Flask-Migrate.
The AWS help article on EBS should get you started, but at a high level, what you are going to do is:
Create an EBS volume
Attach the volume to a running instance
Mount the volume on the instance
Expose the volume to other instances using a Network File System (NFS)
Ensure that when new EBS instances launch, they mount the NFS
Alternatively, you can:
Wait until Elastic File System (EFS) is out of preview (or request access) and mount all of your EB-started instances on the EFS once EB supports EFS.
Switch to the Relational Database Service (RDS) (or run your own database server on EC2) and run an instance of (PostgreSQL|MySQL|Whatever you choose) locally for testing.
The key is hosting your database outside of your Elastic Beanstalk environment. If not, as the load increases different instances of your Flask app will be writing to their own local DB. There won't a "master" database that will contain all the commits.
The easiest solution is using the AWS Relational Database Service (RDS) to host your DB as an outside service. A good tutorial that walks through this exact scenario:
Deploying a Flask Application on AWS using Elastic Beanstalk and RDS
SQLAlchemy/Flask/AWS is definitely not a waste of time! Good luck.