My Google Drive storage is 2TB and I have only used 44GB, but the disk in Google Colabarotary is stuck at 60GB and when I run my piece of code in Google Colaboratory notebook that works with a video dataset and is supposed to extract each video to its frames, after some time of running, it stops and gives a message that there is not enough storage to write on the disk. I know it is not possible, because the size of the entire dataset as frames would be 288GB and I have empty storage way more than that.
I would appreciate it if someone tells me how I can extend the Google Colaboratory disk and make it to actually see the disk storage in my Google Drive, even though I have mounted it.
Thanks!
Colab has a Pro version that doubles the size of local disks. That won't quite bring you to 288G presently, however.
Related
My research lab has a lab Google account to which we are expected to save/upload all of our files. To avoid the messy challenge of maintaining local files that can be read by RStudio as well as Cloud versions on the shared Google Drive account, I have recently began using Drive for Desktop to stream files from the Drive to my desktop to open in RStudio. (More specifically, I share my folder in the lab Drive with my personal Drive which in turn is synced/streamed to my Mac desktop.)
However, I believe this creates conflicts when Drive tries to sync/access files while RStudio is also updating those files (see: zero, one, two, three, four, five, six, seven). I do get a pop-up every so often that I have to close, but that is manageable. A bigger problem for now is that I have noticed that the Drive is now filled with random empty folders (see screenshot below). These folders are located within .Rproj.user. Does anyone know how to prevent these random folders from being generated when I use RStudio in this fashion? It is annoying because they show up in the Google Drive 'home page' and 'Recents' for everyone who accesses the shared Drive.
The other posts linked above describe the need to exclude some file types from sync in order to prevent conflicts, but the directions were for the old Google Drive Back Up and Sync system. I cannot find these settings in the new Drive for Desktop system, and I do not know if this would solve the random folder generation problem.
Has anyone encountered this problem before, know what causes it, or how to fix it?
Many thanks in advance.
I have around 8GB of data on Firestore. How can I download it. I want to backup it on my Google Drive. If there is any way to do it directly, that'd be great.
You can access your project's backend with Google Cloud, you can select the respective project in the top corner as needed. On the left hand side is a Import/Export option where you can choose to export all of your database or only partial and follow the prompts.
This does affect your project's usage however as 1 read still counts as 1 read.
Source: https://firebase.google.com/docs/firestore/manage-data/export-import#export_data
GOTO https://console.cloud.google.com/firestore/
choose import export
choose your project from the status bar
make sure billing is enabled
Download the data
It seems that snapshots and instances are very similar (e.g. https://serverfault.com/questions/527449/why-does-openstack-distinguish-images-from-snapshots).
However, I've been unable to share snapshots publicly globally (i.e. across all projects). Note, I'm a user of the OpenStack installation, not an administrator of the installation.
Assuming that Images don't suffer the same limitation of Snapshots, is there a procedure for convert a snapshot to an image? If not, maybe I should ask a separate question, but my cloud admin tells me it needs to be an image.
for download
glance-image download
Initially I tried this:
openstack image save --file NixOS.qcow2 5426bbcf-06b3-42f3-b117-37395e7dde83
However, the reported size of NixOS.qcow2 was always 0 bytes. Not good. Well, the issue was apparently related to the fact that is is also what was reported in OpenStack and Horizon for the size of the Snapshot. So something weird was going on, but functionally, I could still use the snapshot to create instances without issue.
I then created a volume of a snapshot in horizon (when shut off, couldn't create a volume when shelved), then used this command to create an image from the newly created volume (NixSnapVol):
openstack image create --volume NixSnapVol NixSnapVol-img
Interestingly the reported size went from 41GB to 45GB, maybe that was part of the issue. Anyway, seems to work now, and bonus is that it is now RAW type instead of qcow2, so I don't need to do the conversion (our system largely benefits from using RAW as we have a ceph backend).
I have looked all over (spent about 7 hours). I have found numerous articles on how to map a drive (google drive, onedrive etc). What I cannot seem to find an answer to is this: Once I have mapped the drive can I use the files on that drive just like I use files on a server. Open the file, read a record, write a record. I have created a file, mapped a network drive, wrote records to the file and retrieved records from the file. I have a home grown database that is implemented with a large binary (as opposed to text) file. I have to go to a byte position and read a fixed number of bytes. If WebDAV is copying the file to my computer and then writing it back this would make my file access way to slow and I cannot seem to find an answer. Some programmers I have talked to say I cannot even do that, yet I can. Any direction would be very much appreciated.
Charlie
That's likely because standard WebDAV doesn't allow updating ranges of resources only, so the whole thing needs to be written back.
Often in "restricted security" situations in which programs can't be installed on a computer I run R from a flash drive. Works like a charm. I've recently started using dropbox and was thinking it could be used in a similar fashion to the flash drive. For anyone who has tried this does it work?
I can test it myself but don't want to go to the bother if it's a dead end.
Thanks in advance.
PS this has the advantage of storing an .Rprofile that people whom you share the dropbox folder with can then run your R code. This is particularly nice for people unfamiliar with R.
It should just work.
R is set up in such a way that all its files are relative to a given top-level directory. Whether that is a F:\ or Z:\ drive from your flashdrive, or your Dropbox folder should not matter.
By the same token, R can run happily off a shared folder, be it via Samba, NFS or another mechanism.
It is fine if you want to share .Rprofile or .Rhistory. However, I see a problem with .Rdata, because it can be big (for example 4Gb). For me to save 100 Mb file on Dropbox takes minutes and .RData can be far bigger.
An alternative would be a remote server, where you could connect through ssh.