I am trying to create MWAA as root user and I have all AWS services (s3 and EMR )in North California. MWAA doesn't exist in North California. Hence created this in Oregon.
I am creating this in a private network, it also required a new s3 bucket in that region for my dags folder.
I see that it also needed a new vpc and private subnet as we dont have anything in that region created by clicking on "Create VPC ".
Now when I click on airflow UI. It says
"This site can’t be reached". Do I need to add my Ip to the security group here to access Airflow UI?
Someone, please guide.
Thanks,
Xi
From AWS MWAA documentation:
3. Enable network access. You'll need to create a mechanism in your Amazon VPC to connect to the VPC endpoint (AWS PrivateLink) for your Apache Airflow Web server. For example, by creating a VPN tunnel from your computer using an AWS Client VPN.
Apache Airflow access modes (AWS)
The AWS documentation suggests 3 different approaches for accomplishing this (tutorials are linked in the documentation).
Using an AWS Client VPN
Using a Linux Bastion Host
Using a Load Balancer (advanced)
Accessing the VPC endpoint for your Apache Airflow Web server (private network access)
Related
I would like to know if I can make the airflow UI accessible to all people who have a user, web page type. For this, I would have to connect it to a server, no? Which server do you recommend for this? I was looking around and some were using Amazon EC2.
If your goal is just making the airflow UI visible to public, there is a lot of solutions, where you can do it even in your local computer (of course it is not a good idea).
Before choosing the cloud provider and the service, you need to think about the requirements:
in your team, do you have the skills and the time to manage the server? if no you need a managed service like GCP cloud composer or AWS MWAA.
which executor yow want to use? KubernetesExecutor? CeleryExecutor on K8S? if yes you need a K8S service and not just a VM.
do you have a huge loading? do you need a HA mode? what about the scalability?
After defining the requirements, you can choose between the options:
Small server with LocalExecutor or CeleryExecutor on a VM -> AWS EC2 with a static IP and Route 53 for DNS name
A scalable server in HA mode on a K8S cluser -> AWS EKS or google GKE
A managed service and focusing only on the development part -> google cloud composer
I have an application running in an AWS EKS cluster, the application is previously serving with a public-facing load balancer thus it can easily AWS SNS service, but due to some security reasons we are asked to move that to serve via the internal load balancer, now after moving to the internal load balancer, the application is working but was not able to access SNS service.
How can we configure the application from the internal network to access the AWS SNS service?
You might be after these documents
https://aws.plainenglish.io/publishing-private-amazon-sns-messages-348d38ffc351
https://docs.aws.amazon.com/sns/latest/dg/sns-vpc-tutorial.html
In Short, as you are within a private network, you need to create VPC Endpoints in order to access the AWS services that are outside of a VPC.
I'm trying to run Wordpress on my Ec2 instance with RDS MySQL database.
Here's the tutorial I've followed https://cristianocastro.net/instalando-wordpress-em-um-servidor-amazon-aws-pt-2-3/
But when I try to access the Ec2 instance that connect to the RDS on the browser it shows this message
Error establishing a database connection
I put in the security group inbound and outbound all traffic anywhere.
Here's the print from my security group rule.
Appreciate the help
PS: Saw this question, with no anwser Can't Connect to RDS mysql DB from Wordpress on Amazon linux EC2 Instance
Edit: Rules from security group of the ec2 instance that will access RDS.
Based on the info you provided it seems your RDS is publicly accessible via the internet and connections are going to be accepted from any host (you might want to restrict these later)
Assuming that your software setup (OS, php, MySQL libraries, Wordpress) is okay and you’re using the right parameters in the WP configuration (host, port, user, password, db).
One of the remaining issues might be that the Security Group attached to your EC2 instance doesn’t allow connectivity on port 3306 (MySQL) towards the IP Address of your RDS instance.
It appears that you have:
An Amazon EC2 instance
An Amazon RDS instance
The typical security configuration would be:
The Amazon RDS database configured as Publicly Accessible = No
A Security Group (EC2-SG) on the Amazon EC2 instance that permits inbound traffic from your desired locations (eg port 22 for SSH), and default Outbound rules permitting all traffic outbound
A Security Group (RDS-SG) on the Amazon RDS database instance that permits inbound traffic on port 3306 from EC2-SG
That is, RDS-SG should specifically reference EC2-SG to permit the inbound connection.
The app running on the EC2 instance should reference the RDS database via the DNS Name shown in the RDS console.
Anybody knows is it possible to associate Elastic IP with scheduled data pipeline? I have configured data pipeline to run every day. During data pipeline execution, I need access to Google DB. To have access to Google DB I should add IP (CIDR) in DB authorization settings. But without known public IP of EC2 instance created by data pipeline I cannot configure it.
So I need to have a possibility to setup Elastic IP once to be used for EC2 instance which is creating automatically by data pipeline each time when data pipeline is runned by scheduler.
I am not aware how you can associate a EIP, however, you can create a VPC with with a NAT gateway. When you create your EC2 put it in your subnet that you've created and if everything is setup properly then your public IP will always be the same.
A second option would be to run your pipeline on a Task Runner.
I am fairly new to the NFV+SDN. I have downloaded the OpenDayLight and OpenStack in one Fedora 20 VM. I have mininet network as underlying physical topology in a separate VM. I want to run services like VPN, L3 routing and NAT, Loadbalancing etc on OpenStack, but I don't have a very clear image on how to start. As far as I have understood I have to run these services on OpenStack nodes (through VM instances) and route the traffic through mininet topology with OpenDayLight as the controller in the middle.
My confusions are:
How to start writing the applications (Firewall, VPN, NAT, etc) on OpenStack?
Do I have to write a code for such services or is it command line configuration?
I came across Neutron API, Is that of any help?
Came across this: http://docs.openstack.org/api/openstack-network/2.0/content/API_extensions.html
I have looked at the other questions regarding writing "Hello World" on OpenStack but could not find anything. I shall be grateful to you for any information that could get me started on this project.
I would suggest you to check OpenBaton.
Nowadays I'm working with it which can be used NFV MANO. In addition it's ETSI compliant and their solutions are easy to implement and configure.
For your confusions- You do NOT need to write code explicitly for Firewall / VPN / LB. You need to configure the Openstack Neutron to allow these services directly. The code is already present. You need to configure them to use them. For NAT there is L3 agent already running in the default setup ( al least via packstack )
Neutron API is of any use??? I assume you are refering to REST API and NOT CLI.
Well everything that you do on Dashboard is actualy represented as a REST API to Neutron Server ( not just Neutron but all the other components of Openstack ). All the components of Openstack ( Neutron, Nova, Glance, Keystone, etc ) interact via REST API with each other and RPC mechanism within each component. All the clicks on the Dashboard are actually thrown as a REST API call to the component servers!