does anyone know how I can Encrypt System Disk in Alibaba ECS through Terraform? Support says it is possible but I can't find any source for that. Thank you in advance!
it seems only possible for data disk
https://registry.terraform.io/providers/aliyun/alicloud/latest/docs/resources/ess_scaling_configuration
Related
I am trying to enable SSE with a Customer-Managed CMK in my production Redshift cluster to follow certain security protocols.
For POC purposes, I spun up a 1 Node dc2.large Redshift cluster and following this doc, I was able to enable SSE.
However, my question is, does enabling SSE encrypt the existing data in the cluster? If not, what steps should be taken?
Overall what are the downsides, if any, of enabling encryption at rest in a production Redshift cluster and what are the best practices?
There is no need to change anything in your code or existing pipelines/process. This is Disk encryption. Its nothing to do with your database connections or code.
To know more about the process then read these links.
https://aws.amazon.com/about-aws/whats-new/2018/10/encrypt-amazon-redshift-1-click/
https://docs.aws.amazon.com/redshift/latest/mgmt/changing-cluster-encryption.html
I’m looking to offer potential customers a free trial of a product by way of offering a VM for download.
However we could like to control the licensing/trial by remote disk encryption/decryption.
Basically I'm looking for a way to get an full disk encrypted VM to make a call to remote host and check if the license is still valid and then pass it a decryption key.
So question is, has anyone heard/seen/done this? Is it possible?
Thanks in advance for any response.
red_brick
Consider the Risk associated:
Users download the trial VM.
You decrypt within trial period.
Some Users will copy the files and won't buy the product.
The Cloudera documentation says that Hadoop does not support on disk encryption. Would it be possible to use hardware encrypted hard drives with Hadoop?
eCryptfs can be used to do per-file encryption on each individual Hadoop node. It's rather tedious to setup, but it certainly can be done.
Gazzang offers a turnkey commercial solution built on top of eCryptfs to secure "big data" through encryption, and partners with several of the Hadoop and NoSQL vendors.
Gazzang's cloud-based Encryption Platform for Big Data helps
organizations transparently encrypt data stored in the cloud or on
premises, using advanced key management and process-based access control
lists, and helping meet security and compliance requirements.
Full disclosure: I am one of authors and current maintainers of eCryptfs. I am also Gazzang's Chief Architect and a lead developer.
If you have mounted a file system on the drive then Hadoop can use the drive. HDFS stores its data in the normal OS file system. Hadoop will not know whether the drive is encrypted or not and it will not care.
Hadoop doesn't directly support encryption, though a compression codec can be used used for encryption/decryption. Here are more details about encryption and HDFS.
Regarding h/w based encryption, I think Hadoop should be able to work on it. As Spike mentioned, HDFS is like any other Java application and stores it's data in the normal OS file systems. FYI, MapR uses Direct I/O for better HDFS performance.
See also Intel's Rhino. Not open source yet...
https://github.com/intel-hadoop/project-rhino/
https://hadoop.intel.com/pdfs/IntelEncryptionforHadoopSolutionBrief.pdf
To handle high traffic, I'm planning to scale out, run my web application (WordPress based) on some EC2 instances (I'm very new to AWS). The instances need to work on the same data (images, videos...).
I am thinking about using S3 as the storage for this shared data.
My questions are:
If I use S3, do I need to write extra codes for my application to upload and get data to/from S3? Or there is a magic way to mount EC2 instances to S3, and after that EC2 instances can access S3 as accessing the local storage?
I've heard that S3 is a bit slow since it is accessed through web services (if users upload files and it takes time to upload the files to S3). So is there any better way for storing shared data?
I've read some documents about the ability of scaling of Amazon EC2. But none of them mentions about how to handle shared data. Any help is highly appreciated. Thanks.
There is no native facility to 'mount' an S3 bucket as storage to an EC2 instance, although there are several 3rd-party apps which offer mechanisms to make S3 storage available as a virtual drive or repository. Most of them offer a preset amount of free storage and then a tiered charging mechanism for larger amounts - Google for 'S3 storage interface' and take a look.
Whether you write code to use S3 through the API or use an interface layer, there will always be some latency between your app and the storage. That's a fact of physics and there's nothing you can do to eliminate the delay, because the S3 repository is not local to the EC2 cluster - so you will never achieve 'local' storage access speeds.
An alternative might be to use EBS which is local to your EC2 instance - it has different properties to S3 (for example, it does not offer edge locations for regionally-localised access) but is much faster for application use because it is inside the EC2 cluster and mounted as local storage.
You could mount S3 bucket onto all your EC2 instances. It's a 2-way mount so all your files will be synchronized. You could use s3f3 to do the mounting.
I used this guide and set up mine pretty quick: Mount S3 onto EC2
If you are then concerned about speed, you can use Amazon ElastiCache or even use EBS as a cache drive.
For starts you question lacks some details about your application architecture, but there are some possibilities.
First, if you project is medium-sized you could use GlusterFS on your main nodes as servers and clients at the same time (using native or NFS protocol), RDS *Multi-AZ* MySQL instance for DataBase. CloudFront as CDN with CDN linker or W3TC plugins. Also, put an ELB in front.
In this particular case I would recommend a couple c3.large instances at least.
Second, when your project would grow - you should make instance AMI and created an auto-scaling group that would just connect to your main storage and compute instances. (Also consider lifting compute load from these rather small instances).
Things to consider additionally:
Great WordPress article about WordPress clusters for is http://harish11g.blogspot.ru/2012/01/scaling-wordpress-aws-amazon-ec2-high.html
Alternative to GlusterFS solution could be Ceph File System
You also could (or maybe should) you SSD cache (for example flashcache) for that GlusterFS volume.
In relation to my earlier similar SO question , I tried using snow/snowfall on AWS for parallel computing.
What I did was:
In the sfInit() function, I provided the public DNS to socketHosts parameter like so
sfInit(parallel=TRUE,socketHosts =list("ec2-00-00-00-000.compute-1.amazonaws.com"))
The error returned was Permission denied (publickey)
I then followed the instructions (I presume correctly!) on http://www.imbi.uni-freiburg.de/parallel/ in the 'Passwordless Secure Shell (SSH) login' section
I just cat the contents of the .pem file that I created on AWS into the ~/.ssh/authorized_keys of the AWS instance I want to connect to from my master AWS instance and for the master AWS instance as well
Is there anything I am missing out ?
I would be very grateful if users can share their experiences in the use of snow on AWS.
Thank you very much for your suggestions.
UPDATE:
I just wanted to update the solution I found to my specific problem:
I used StarCluster to setup my AWS cluster : StarCluster
Installed package snowfall on all the nodes of the cluster
From the master node issued the following commands
hostslist <- list("ec2-xxx-xx-xxx-xxx.compute-1.amazonaws.com","ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com")
sfInit(parallel=TRUE, cpus=2, type="SOCK",socketHosts=hostslist)
l <- sfLapply(1:2,function(x)system("ifconfig",intern=T))
lapply(l,function(x)x[2])
sfStop()
The ip information confirmed that the AWS nodes were being utilized
Looks not that bad but the pem file is wrong. But it is sometimes not that simple and many people have to fight with this issues. A lot of tips you can find in this post:
https://forums.aws.amazon.com/message.jspa?messageID=241341
Or check google for other posts.
From my experience most people have problems in these steps:
Can you log onto the machines via ssh? (ssh ec2-00-00-00-000.compute-1.amazonaws.com). Try to use the public DNS, not the public IP to connect.
You should check your "Security groups" in AWS if the 22 port is open for all machines!
If you plan to start more than 10 worker machines you should work on a MPI installation on your machines (much better performance!)
Markus from cloudnumbers.com :-)
I believe #Anatoliy is correct: you're using an X.509 certificate. For the precise steps to take to add the SSH keys, look at the "Types of credentials" section of the EC2 Starters Guide.
To upload your own SSH keys, take a look at this page from Alestic.
It is a little confusing at first, but you'll want to keep clear which are your access keys, your certificates, and your key pairs, which may appear in text files with DSA or RSA.