I want to share a persistence volumen from my Google Container Engine instance with my FTP Server instance in other to access to the file that upload to FTP server.
If I've use a Computer Engine Disk I can't do that because the disk became readonly.
I see that I can use a Cloud Storage but I don't know If I can used as persistence volumen in Container Engine.
Anybody can help me?
Regards
First of all you need a FTP server container image for that. I'm not sure if anything off-the-shelf exists, you may need to build it yourself.
Then you can simply use GCE Persistent Disks as you said (however I'm not sure what you mean by "because the disk became readonly", you're supposed to explain your problem in detail at StackOverflow, people here simply can't guess it correctly all the time).
So you can follow this tutorial to learn about how to use GCE Persistent Disks on GKE clusters: https://cloud.google.com/kubernetes-engine/docs/tutorials/persistent-disk then you can modify the mysql part with your FTP server docker image.
Google Cloud Storage won't give you the filesystem semantics (such as permission bits, user/group on Linux) so it probably won't suit for FTP. Also you can't mount GCS to your compute instances.
Related
I mounted the bucket to compute the engine VM via gcsfuse and then ran the Nginx program in the VM. Normally the Nginx program writes the cache to the directory, i.e. the mounted bucket, using the proxy_cache directive.
However, I have a problem that the Nginx can create cache files in the filesystem under the bucket directory, but the size of the cache file is always 0B. client requests keep getting "Miss cache" status.
So it seems that after mounting with gcsfuse, the Nginx application can only create cache files, but cannot write to them.
My VM environment is.
Machine type: n2-standard-2
CPU platform: Intel Cascade Lake
Architecture: x86/64
System: ubuntu-1804
In addition, gcsfuse has specified a service account with owner privileges via the --key-file directive, and the Nginx program has been run with the root user.
For example, the following debug log is an empty file (8/2b/0a7ed71cddee71b9276a0d72a030f2b8) created in the bucket after a client request and not written to the cache. What could be the cause of this possibility?
https://storage.googleapis.com/cloud_cdn_cache_bucket/debug_log.txt
Here is the debug log obtained by the command --debug_fuse --debug_fs --debug_gcs --debug_http -foreground.
You can't use Cloud Storage Fuse for cache.
There is a technical reason to that: GCSFuse is a Cloud Storage API call wrapper that transforms system calls to API calls. However, all the system calls aren't supported, especially those related to "database" format with stream write, seek, and rewrite partial content of the file. All common operations for a database (or cache) but not compliant with Cloud Storage: you can only write/read/delete a file. Update/partial write aren't supported. It's not a file system!
In addition, because you now know that GCSFuse is a wrapper (of system calls to API calls), you should feel that using that file system type is not a good idea: the latency is terrible!! It's API calls! Absolutely not recommended for cache and low latency operations.
The best solution is to use a local file system dedicated to cache. But if you scale out (more servers in parallel) you could have issues (cache is not shared between instances):
Use sticky session mechanism to always route the same user session on the same NGINX server and therefore always use the same cache context
Use the Filestore service that offers a NFS (Network File Share) system to mount the same storage space on different servers, with an interesting latency (not as good as a local file system)
You talked about a key file in your question also. I recommend you to avoid as much as you can to use the service account key files; especially if your app runs on Google Cloud. Let me know what's your key file usage in detail if you want more guidance
I have a bucket in Google Cloud which I have upload Angular template i.e. http://digitaldevnet.appspot.com
then I have VM instance IP where I have WordPress website i.e.
http://35.200.194.201
I found different tutorials where you can connect domain to Google Cloud hosting
but I want to connect appspot link i.e. http://digitaldevnet.appspot.com to WordPress site
can be connected and once we browse it should work as VM may be offline sometime.
Any recommendation and tutorial, please let me know
You would need to set up a Load balancer in order to direct your traffic between the GCE instance and the bucket, you can find the instructions for this over here.
Nevertheless, it would be a bit complicated as you would need to set up the configuration and some health checks so that the GCE instance is not accessible when it's down.
Hope you find this useful.
/* I'm new here and I've done days of research. */
What is the best practice to share files with multiple autoscaling Google Compute Engine VMs?
I'm planning to set up an Instance group of VMs with NGINX for serving static files for multiple domains. These VMs would autoscale to n (multiply itself) and the files would change a lot. I need a storage for the files these VMs will serve.
So far I've found these solutions:
1) Persistent disk + rsync -> This should have the smallest latency, but when I reach GBs of files, autoscaled VMs would be syncing for a long time after they spawn, thus throwing 404s.
2) Master VM without web server + nfs/smb -> Small latency, but no redundancy.
3) Cloud Storage + FUSE -> Big latency, great redundancy, no SLA.
4) Shared Persistent disk -> Small latency, but read-only.
5) NGINX + Cloud SQL/Bigtable/Spanner/Datastore -> Mehish latency and I don't feel well about connecting webserver to a DB.
Are there any other better solutions?
Thanks
EDIT: The static files are multiple index.html files -> homepages of multiple domains.
There is also:
6) Firebase Hosting - https://firebase.google.com/docs/hosting
or
7) another way - I would personally go with Cloud Storage but not FUSE. Or at least not for serving. You can still use FUSE for writing to Bucket(s). Of course the best way would be to just use Cloud Storage API from within the application.
For serving files:
I would create a Load balancer with a Backend bucket the same bucket where the application writes. Also be careful to enable Cloud CDN on that Load balancer.
More details at:
Load balancer - https://cloud.google.com/load-balancing/
Cloud CDN - https://cloud.google.com/cdn/
or just be bold and create now a load balancer at https://console.cloud.google.com/networking/loadbalancing/loadBalancers/list?project=
For serving static files the best is definitely to use a load balancer and backend buckets with Cloud CDN enabled.
The load balancer have rules to forward traffic. For example it can intercept all requests by hosts, subdomain, or path.
*.mysite1.com => bucket1
demo.mysite2.net => bucket1
test.mysite3.com => bucket2
Because files are served with Cloud CDN, the latency becomes minimal.
In order to write your files to a bucket you could use FUSE, or create files locally and use gsutil cp.
Persistent disks can only be shared within several Compute Engines in read mode. If you need write mode it won't work.
The last option Cloud SQL + Nginx is actually pretty good. Cloud SQL is way more fast than Mysql servers. And connection between cloud sql and gce is easy and reliable.
But it is more a matter of preferences here... if you feel comfortable writing the scripts that will read and write to it.
Say I have a docker image, and I deployed it on some server. But I don't want other user to access this image. Is there a good way to encrypt the docker image ?
Realistically no, if a user has permission to run the docker daemon then they are going to have access to all of the images - this is due to the elevated permissions docker requires in order to run.
See the extract from the docker security guide for more info on why this is.
Docker daemon attack surface
Running containers (and applications)
with Docker implies running the Docker daemon. This daemon currently
requires root privileges, and you should therefore be aware of some
important details.
First of all, only trusted users should be allowed to control your
Docker daemon. This is a direct consequence of some powerful Docker
features. Specifically, Docker allows you to share a directory between
the Docker host and a guest container; and it allows you to do so
without limiting the access rights of the container. This means that
you can start a container where the /host directory will be the /
directory on your host; and the container will be able to alter your
host filesystem without any restriction. This is similar to how
virtualization systems allow filesystem resource sharing. Nothing
prevents you from sharing your root filesystem (or even your root
block device) with a virtual machine.
This has a strong security implication: for example, if you instrument
Docker from a web server to provision containers through an API, you
should be even more careful than usual with parameter checking, to
make sure that a malicious user cannot pass crafted parameters causing
Docker to create arbitrary containers.
For this reason, the REST API endpoint (used by the Docker CLI to
communicate with the Docker daemon) changed in Docker 0.5.2, and now
uses a UNIX socket instead of a TCP socket bound on 127.0.0.1 (the
latter being prone to cross-site request forgery attacks if you happen
to run Docker directly on your local machine, outside of a VM). You
can then use traditional UNIX permission checks to limit access to the
control socket.
You can also expose the REST API over HTTP if you explicitly decide to
do so. However, if you do that, being aware of the above mentioned
security implication, you should ensure that it will be reachable only
from a trusted network or VPN; or protected with e.g., stunnel and
client SSL certificates. You can also secure them with HTTPS and
certificates.
The daemon is also potentially vulnerable to other inputs, such as
image loading from either disk with ‘docker load’, or from the network
with ‘docker pull’. This has been a focus of improvement in the
community, especially for ‘pull’ security. While these overlap, it
should be noted that ‘docker load’ is a mechanism for backup and
restore and is not currently considered a secure mechanism for loading
images. As of Docker 1.3.2, images are now extracted in a chrooted
subprocess on Linux/Unix platforms, being the first-step in a wider
effort toward privilege separation.
Eventually, it is expected that the Docker daemon will run restricted
privileges, delegating operations well-audited sub-processes, each
with its own (very limited) scope of Linux capabilities, virtual
network setup, filesystem management, etc. That is, most likely,
pieces of the Docker engine itself will run inside of containers.
Finally, if you run Docker on a server, it is recommended to run
exclusively Docker in the server, and move all other services within
containers controlled by Docker. Of course, it is fine to keep your
favorite admin tools (probably at least an SSH server), as well as
existing monitoring/supervision processes (e.g., NRPE, collectd, etc).
Say if only some strings need to be encrypted. Could possibly encrypt this data using openssl or an alternative solution. Encryption solution should be setup inside the docker container. When building container - data is encrypted. When container is run - data is decrypted (possibly with the help of an entry using a passphrase passed from .env file). This way container can be stored safely.
I am going to play with it this week as time permits, as I am pretty curious myself.
I'm thinking about configuring the remind calendar program so that I can use the same .reminders file from my Ubuntu box at home and from my Windows box at work. What I'm going to try to do is to make the directory on my home machine that contains the file externally visible through webdav on Apache. (Security doesn't really concern me, because my home firewall only forwards ssh, to hit port 80 my my home box, you need to use ssh tunneling.)
Now my understanding is that webdav was designed to arbitrate simultaneous access attempts. My question is whether this is compatible with direct file access from the host machine. That is, I understand that if I have two or more remote webdav clients trying to edit the same file, the webdav protocol is supposed to provide locking, so that only one client can have access, and hence the file will not be corrupted.
My question is whether these protections will also protect against local edits going through the filesystem, rather than through webdav. Should I mount the webdav directory, on the host machine, and direct all local edits through the webdav mount? Or is this unnecessary?
(In this case, with only me accessing the file, it's exceedingly unlikely that I'd get simultaneous edits, but I like to understand how systems are supposed to work ;)
If you're not accessing the files under the WebDAV protocol, you're not honoring locks set via LOCK and UNLOCK methods and therefore will open to potential to overwrite changes made by another client. This situation is described in the WebDAV RFC here: https://www.rfc-editor.org/rfc/rfc4918#section-7.2