Hi accidentally deleted a jupyter notebook that I was running in DataProc
I can't see a checkpoints folder in my GCS bucket. Any suggestions of recovery?
if you never saved the checkpoints before or you explicitly deleted from the dataproc web UI, then it's very likely your data won't be able to recovered.
Things you can check to ensure GCS is able to save your checkpoints:
Make sure the GCS is set up correctly to save checkpoints, you can check /etc/jupyter/jupyter_notebook_config.py and make sure c.GCSContentsManager.bucket_name is present and set with right bucket. If not present, you should set it and then from dataproc Web UI, Kernel > Restart to restart Jupyter from the Jupyter menu.
Make sure your account have admin/write access to your bucket. How - https://cloud.google.com/storage/docs/access-control/
Related
I do most of my dev work locally but ocassionally I have to switch over to using a preconfigured jupyterlab instance on GCP. The way things are set up now, I'm unable to ssh into these notebook servers and the only way for me to interact with them is through the jupyterlab integrated terminal.
I have custom save hook functions set up in my local environment for testing, linting, etc.--comes in really handy for keeping everything in a production-ready state and I'd like to be able to set up a sort of "environment as code" system where when I pull updated code into a new environment, the customized configuration would move with it and take effect automatically. I suppose the proper way to do this would be to use a Docker image and rebuild the cloud instance from scratch every time it needs an update but it kind of seems like overkill for such minor changes. (Also the Google docker images don't work that well on my M1 MacBook ).
I am currently working on an personal application where, I upload documents/pictures/videos from my phone to Cloud Storage. During this time my computer sitting at home is consistently running a shell script waiting for a new document to be uploaded to Cloud Storage, after it finds an uploaded file, it downloads it does some work to it, and then deletes it.
I can figure out how to upload and connect my application to Firebase, but I am not sure if its possible for a shell script to do the remaining work.
Should I look into some other service to do this, or another method?
thank you for your help!
You can use a command line program called gsutil to upload and download files from a Cloud Storage bucket. This should be easy to use from a shell script.
Can someone please advise how to setup a backup for Files in Firebase storage. I am able to make a backup of Database but not sure how to setup a regular backup for files (I have images) in firebase storage.
How to make local backup of Firebase Storage
There is no built-in method via Firebase. However, since Firebase uses Google Cloud Storage behind the scenes for Firebase Storage it's possible to use the gutils Tool.
Prerequisites
Make sure Python (2.7.9+) is installed on your machine python -V
Go to the Google Cloud SDK page and follow the directions to download and install Google Cloud SKD on your OS.
Steps
At the end of the Google SDK installation you should have run gcloud init. This will ask you to select your project and authenticate you. Since Firebase uses Google Cloud Platform behind the scenes your Firebase project should be available as a choice.
In order for Google Cloud Utils to download the files that were uploaded with Firebase permissions you need to give your account Firebase Privileges. Go to the IAM page and select your email address you signed into cloud init with. In the list of available permissions you need to select Firebase Rules System from the Other category.
Get your Google Storage URL from the Firebase Storage Page in the dashboard (Towards the top) Should look something like this: gs://<bucket_name>
In command line on your local machine navigate to the folder you want to do a local backup to. Make sure you are in the folder you want as the following command will download all files right there in current folder
Run the gutil command gsutil -m cp -R gs://<bucket_name> .
-m enables multithreading for faster downloads if you have many files.
cp is the copy command
-R is recursive. If enabled it will download all files and folders in the specified tree.
You're done! This will run for some time depending on the size of your storage.
This can be used to also make a copy(backup) to another Google Cloud Storage Bucket or AWS etc.
Use Google Cloud Transfer Service.
Select your current project
Create Transfer Job
Select source (storage bucket url)
Select destination (click browse and create new bucket)
Use created bucket URL as destination
Configure transfer settings (This is where you can schedule how often the backup runs.)
Click "Create"
If you follow the wizard in the link it will guide you through pretty easily.
There is no built-in backup feature in Cloud Storage for Firebase.
But since it is built on top of Google Cloud Storage, any backup solution for GCS can work for Firebase too. Typically this will involve creating a separate bucket that is the target of the regular bucket where you store/read files.
To upload manifest file at the time of deployment, I want that there should be prompt every time for uploading manifest file, the previous manifest file should not be picked up automatically.
For this, is there any option in ROC? With CLI, since I dont have the access to the main server can I install CLI on Agent? It would be great if I can have a clear process for installation on Agent, I am confused with documents available.
I have one running Agent.
Thanks in Advance.
Can you throw some points on how it is a best way, best practice
to install web application on Unixes?
Like:
where to place app and its bases and so for,
how to configure to be secure and easy to backup,
etc
For example I know such suggestion -- to set uniq user for each app.
App in question is Jira on FreeBSD, but more general suggestions are also welcomed.
Here's what I did for my JIRA install on Fedora Linux:
Create a separate user to run JIRA
Install JIRA under the JIRA user's home directory
Made a soft link "/home/jira/jira" pointing to the JIRA installation directory (the directory as installed contains the version number, something like /home/jira/atlassian-jira-enterprise-4.0-standalone)
Created an /etc/init.d script to run JIRA as a service, and added it to chkconfig so that it runs at system startup - see these instructions
Created a MySQL database for JIRA on a separate data volume
Set up scheduled XML backups via the JIRA admin interface
Set up a remote backup script to dump the MySQL database and copy the DB dump and XML backups to a separate backup server
In order to avoid having to open extra firewall ports, set up an Apache virtual host "jira.myhost.com" and used mod_proxy to forward requests to the JIRA URL.
I set everything up on a virtual machine (an Amazon EC2 instance in my case) and cloned the machine image so that I can easily restart a new instance if the current one goes down.