I am using Terraform to provision emr . In order to do so i am calling
resource "aws_cloudformation_stack" and then attaching cloudformation template to launch an EMR. its working now i want my EMR to have 22 inbound port open for ssh connection.
please see reference
https://www.terraform.io/docs/providers/aws/r/cloudformation_stack.html
i can do this by attaching a security group . could somebody please let me know how can i do this?
You should probably be able to handle this by creating the security group via Terraform, referencing it as a dependency to make sure it's created before the cloudformation stack and then providing the security group ID to the cloudformation template as a variable.
Related
Overview
I tried creating a VPC network, having a subnet and adding a Serverless VPC connector with terraform in GCP. I was following the official guide ( https://cloud.google.com/vpc/docs/configure-serverless-vpc-access#terraform ) and initially everything was working well. After that I accidently commited my JSON key to github, someone stole it and used it for crypto, the project was disabled but shortly after that reinstated
After that my terraform VPC connector creations started to fail. I tried a lot of different things but nothing seems to work(running destroy, changine service accounts, changing names, deleting all of the terraform subfolders, deleting EVERY resource and restarting the process)
The errors I am getting are:
│ Error: Error waiting to create Connector: Error waiting for Creating Connector: Error code 13, message: An internal error occurred: Failed to create a VPC Access connector. Please delete the connector manually.
│
or
│ Error: Error creating Connector: googleapi: Error 409: Requested entity already exists
Today I tried to create VPC connector from the command line(gcloud) and from the UI tool. The errors persisted
Unknown error. Original error message: Operation failed: Insufficient CPU quota in region.
Max throughput of the connector per day over last seven days.
or
An internal error occurred: Failed to create a VPC Access connector. Please delete the connector manually.
errors while deleting:
│ Error: Error waiting for Deleting Network: The network resource 'projects/static-emblem-327016/global/networks/sun-serverless-network' is already being used by 'projects/static-emblem-327016/global/routes/default-route-5cbc9de02e21bb35'
│
I was lookint at this issue https://issuetracker.google.com/issues/164378672 In it I was problems with us-central1 but I tried a couple of different regions and still I have the same issue
Questions:
I am running out of ideas, I was wondering if this is an infrastructural issue, maybe I should dump the project and create a new one ? Where can I check if there are infra issues ? How can I resolve my issue?
I recently get this error Error: Error creating Connector: googleapi: Error 409: Requested entity already exists. So I can explain the root cause and it's fix.
What I was doing is like trying to create a GCP resource (Create PubSub topic) using terraform (plan and then apply).
But before executing the terraform apply, I created the resource manually long time back with the same name. I expected that the terraform plan or terraform apply will not try to create it again since the resource name is same. But instead of Refreshing state, I found it was trying to Creating the resource. The reason it that, terraform does not know about your resource history. Either you need to import your resource history using terraform import command or else delete the manually created resource and then run the terraform apply command.
The message “An internal error occurred: Failed to create a VPC Access connector. Please delete the connector manually” can indicate that you don't have enough resources in your project to create the connector. Please make sure you have enough Resource Quota available in your GCP project.
The message “googleapi: Error 409: Requested entity already exists” indicates that The resource that a client tried to create already exists.
If you want to know what the root cause is, you can check the logs of the VPC Connector creation in the System Event Audit Logs.
System Event audit logs contain log entries for Google Cloud actions that modify the configuration of resources. System Event audit logs are generated by Google systems; they aren't driven by direct user action. System Event audit logs are always written; you can't configure, exclude, or disable them. The instructions to access them are here.
On the other hand, generating and distributing service account keys poses severe security risks to your organization. They are long-lived credentials that are not automatically rotated. These keys can be leaked accidentally or maliciously allow attackers to gain access to your sensitive GCP resources. If you accidentally compromised your JSON Key, please read the recommendations in this link.
If you want to know more about the risk and alternatives to download Service Account, Key please follow this link. Please note that this is not GCP official documentation, so I cannot vouch for its accuracy.
I was able to resolve my issue. It turns out that I had deleted my default compute engine service account in panic. I was able to recover it and everything worked out from there. For more info go here: https://cloud.google.com/iam/docs/creating-managing-service-accounts#undeleting_a_service_account
you have to identify the default service acc for compute engine and undelete it:
gcloud beta iam service-accounts undelete ACCOUNT_ID
I have been trying to create vpc infrastructure in AWS through terraform I am unable to perform the "Terraform apply" command anyone has a similar problem while using a free trial account.
Error: Error creating VPC: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 4HZVo3-eWCS-YLhRy55P_0T13F_fPtA29TYrJrSe5_dyPxIcqRbh7_wCcrCZr2cpmb-B5--_fxVaOngBfHD_7yfnPH7NLf1rrqpb7ge1mvQrK8P0Ltfpgpm37nZXezZUoYf1t4peB25aCxnbfeboHpgJjcFnHvqvf5so5G2PufnGZSB4FUZMfdaqppnJ-sNT7b36TonHUDNbLhBVUl5Fwd8d02R-6ZraRYvDx-o4lDfP9xSWs6PMUFXNr1qzruYaeMYMxIe-9kGOQptgBLYZXsxr966ajor-p6aLJAKlIwPGN7Iz7v893oGpGgz_8wxTv4oEb5GnfYOuPOqSyEMLKI69b2JUvVU1m4tCcjKBaHJARP5sIiFSGhh4lb_E0_cKkmmFfKzyET2h8YkSD8U9Lm4rRtGbAEJvIoDZYDkNxlW7W2XvsccmLnQFeSxpLolVhguExkP7DT9uXffJzFEjQn-VkhqKnWlwv0vxIcOcoLP04Li5WAqRRr3l7yK2bYznfg
│ status code: 403, request id: 5c297a4d-7bcf-4bb4-b311-37480e1f26b8
make sure you have properly setup aws credentials and permissions.
check these two files
~/.aws/credentials
~/.aws/config
this docs can help you.
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
Did you configure your access keys?
provider "aws" {
region = "us-west-2"
access_key = "my-access-key"
secret_key = "my-secret-key"
}
There are multiple ways to do it (described here).
My above example can be a good start but you don't want to commit those so I recommend to configure the keys in ~/.aws/credentials (like you need them for AWS CLI). The aws provider will pick them up automatically and so you don't need to define them somewhere in your terraform code.
I need to keep track of Instance creation in openstack Nova.
That is I need to perform some special operations on creation of new instance in openstack.
So that I need to know where all the details are getting stored (In Log file).
Please some one guide me regarding the Log file for tracking instance creation or some other way to track the same.
As I am aware you have to look in the following services' log files
nova-scheduler (oftenly installed on controller node). This will show which 'server' will host the newly created Virtual Machine.
The logs of nova-compute service running on the host that the Virtual Machine was instantiated.
You can additionally check the logs of qemu and libvirt (again on the host that the Virtual Machine was instantiated)
Have in mind that the info you will find there, depends on the 'logging level' you have set in each service configuration files. For more information about how you can configure the OpenStack Components logging refer to the official documentation "Logging and Monitoring".
I'm running a .net server on Amazon's Elastic Beanstalk and when I try to instantiate AmazonDynamoDBClient it fails. According to the information I've been following from these two pages:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.iam.roles.apps.html
http://docs.aws.amazon.com/AWSSdkDocsNET/latest/DeveloperGuide/net-dg-roles.html
It should retrieve the credentials from the IAM role assigned to the EC2 instance.
It has the DynamoDBFullAccess template as a policy.
If I supply the credentials in the code itself through the web.config file it works, but only when debugging locally and not when deployed to elastic beanstalk.
Here is the code that breaks it:
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
or
AmazonDynamoDBClient client = new AmazonDynamoDBClient(new InstanceProfileAWSCredentials());
Neither works. And I can't seem to retrieve crash logs from elastic beanstalk, so it is hard to debug. I'm pretty sure that I'm following the descriptions in those two links to the letter and am confused.
I've been trying to solve this for the better part of a day and would really appreciate some help. If there is any information missing, please let me know.
Thank you.
If you go to aws console and select ec2 view instances you will see the ec2 instance for your elastic beanstalk. The name of that instance will be either "Default-Environment" or whatever name you chose for your elastic beanstalk environment. Then you can ssh to that ec2 instance and view logs. For example my tomcat logs are stored at /var/log/tomcat7
You can also scp your credential file to your ec2 host just for debug and run the app. It should work because you already have it working on your local machine. but I guess this approach is not recommended and may has security concern.
This should at least get you started. may be an EC2 expert can help you solve the real problem :)
I am using .NET SDK for AWS and and trying to create a service that can create/mange instances. As part of this I want to create an EC2 instance in a specific VPC (non-default). There may have more then one VPC in a zone and I want to programatically be able to create/manage instances in any of the VPC rather than just the default VPC.
Is this possible? If yes how? I looked through the API docs and could not find a way to specify the VPC at the time of creation of EC2 isntance.
The VPC appears to be implied by the subnet-id that you specify. If this doesn't get you there, it might at least get you an error message explaining what you've missed.
http://docs.aws.amazon.com/sdkfornet/latest/apidocs/items/PEC2Instance_SubnetId_NET4_5.html
http://docs.aws.amazon.com/sdkfornet/latest/apidocs/items/TEC2RunInstancesRequest_NET4_5.html
http://docs.aws.amazon.com/AWSSdkDocsNET/latest/DeveloperGuide/run-instance.html
From the underlying REST API:
SubnetId
[EC2-VPC] The ID of the subnet to launch the instance into.
Type: String
Default: None
Required: No
http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-RunInstances.html