1) We upload a python wheel to the storage account associated with the workspace successfully.
2) In the second step we submit an experiment which runs in the cluster and needs to download and run the package from step 1.
The experiment is able to download the package and run when the storage account is not associated with any VNet. However, when we associate the storage account in Vnets the experiment hangs and eventually fails. This storage account is in two Vnet’s, a, and b. The cluster is also in the Vnet, a.
I don’t know why the cluster cannot download the wheel package when the storage account is in a Vnet. It is our policy to have storage accounts in Vnet’s.
It there something else we are missing? I also checked the container registry setting and its set to ‘allow from everywhere’ (we are using std SKU).
Let me know if any further information is required. Thanks.
Related
I'm looking for options for failover in the case of a region outage that doesn't involve incurring the cost of having a hot second region and geo-redundancy. Is it possible to use the backup feature to manually create a new account in a new region and then upload the related documents?
I was looking at the restore feature and noticed these limitations:
"The restored account is created in the same region where your source account exists. You can't restore an account into a region where the source account did not exist."
I also saw this limitation which makes me wonder if backups work in case of region failure? Even with the geo-redundant backup storage option picked?:
"The backups are not automatically geo-disaster resistant. You have to explicitly add another region to have resiliency for the account and the backup."
Does this mean that the backups will also go down if geo-redundancy and multiple regions aren't set up?
Thanks.
Backups can only be restored into the same region in which the account was created. Backups are not intended or able to provide resiliency to regional outages. And certainly cannot provide anything approaching 99.999% availability. Regional replication is the only means to provide resiliency to regional events for mission critical workloads.
Backup is by default is geo-dr into the pair for that region. These are only restored into the source region where the regional event impacted the storage for that region which prevents it from recovering normally.
When I use googlesheets4 in R, I use sheets_auth() in the console and it works fine. But when I try to run it in R markdown, and when I try to knit, I cannot seem to get the credentials. Can someone walk me through the process? I've gone to the vignettes for googlesheets4 but cannot seem to understand it.
This is working for me
gs4_auth(path = "xxxxxxxxxxxxxxxx.json")
It doesn't return anything, but after that I'm able to write data in my sheet with sheet_write()
To get the credentials in a json file you have to follow these steps:
From the Developers Console, in the target GCP Project, go to IAM & Admin > Service accounts.
Give it a decent name and description.
For example, the service account used to create the googledrive docs has name “googledrive-docs” and description “Used when generating
googledrive documentation”.
Service account permissions. Whether you need to do anything here depends on the API(s) you are targetting. You can also modify roles
later and iteratively sort this out.
For example, the service account used to create the googledrive docs does not have any explicit roles.
The service account used to test bigrquery has roles BigQuery Admin and Storage Admin.
Grant users access to this service account? So far, I have not done this, so feel free to do nothing here. Or if you know this is useful
to you, then by all means do so.
Do Create key and download as JSON. This file is what we mean when we talk about a “service account token” in the documentation of gargle
and packages that use gargle. gargle::credentials_service_account()
expects the path to this file.
Appreciate that this JSON file holds sensitive information. Treat it like a username & password combo! This file holds credentials that
potentially have a lot of power and that don’t expire.
Consider storing this file in such a way that it will be automatically discovered by the Application Default Credentials search
strategy. See credentials_app_default() for details.
You will notice the downloaded JSON file has an awful name, so sometimes I create a symlink that uses the service account’s name, to
make it easier to tell what this file is.
Remember to grant this service account the necessary permissions on any resources you plan to access, e.g., read or write permission on a
specific Google Sheet. The service account has no formal relationship
to you as a Google user and won’t automatically inherit permissions.
(copied from here https://gargle.r-lib.org/articles/get-api-credentials.html#service-account-token)
I use Firebase for my web apps, not Google Cloud Platform -- though I'm aware that behind the scenes, every Firebase project is also a GCP project.
I just received an email from GCP, saying (excerpted):
[Action required]: Enable the Cloud Build API for your projects before April 20, 2020, to ensure Cloud Function is built and deployed correctly ... You must enable the Cloud Build API for your project(s) to ensure your Cloud Function is built and deployed correctly. Once you enable the API, you may incur charges if your Cloud Build, Container Registry, or Cloud Storage usage exceeds the free tier limits for these products.
Does this impact the deployment or development process for Firebase at all? (e.g., right now I use firebase deploy )
As far as I can tell, the answer is no -- it just allows for a greater maximum build-time-per-day. But the email didn't mention Firebase at all, so I want to be sure I understand any other implications well before April 20.
I had the same questions and got in touch with the firebase support team, here are my questions and the answers I received:
"Is that mandatory?"
Only if you are going to deploy Cloud Functions in the future, already
deployed functions will not be affected.
"How do I know if my Cloud Build, Container Registry, or Cloud Storage usage will exceed the free tier limits for these products? is there any way how can check what I've been using so I'll be prepared?"
Cloud Build provides a free tier where only usage above 120
build-minutes/day will be charged. When your usage is within the free
tier, you will not be charged for the Cloud Build portion of Cloud
Function deployments. See Cloud Build pricing for more information.
Similarly, Cloud Storage and Container Registry share a free tier
where only usage above 5GB-months will be charged. ( * Note : free
tier only limited to US regions - US-WEST1, US-CENTRAL1, and US-EAST1
and aggregated over all 3 regions )? For example, if you have a large
deployment that uses 100GB of storage, you will only be charged an
additional $2.47 for storage/month (based on these particular U.S.
regional storage prices). You can monitor your usage and see whether
you are getting close to hitting the free quotas.
"Lastly, what does all this stuff do for my app? Why do I need to add this other features? If you were to sell me this, how would you present it for my use-case?"
This is a best effort from Google to communicate information that is
necessary to the user’s continued use of the product or that is
considered a necessary legal update and keep customers away from
having future issues. Using Cloud Build, Container Registry and Cloud
Storage provides the following benefits: Detailed function build logs
will be available in the GCP Console, aiding in debugging and
increasing visibility. The ability to get build time that exceeds the
current build quota of 120 build-mins/day. The ability to view a built
container image for your function in Container Registry.
I received the following response from Firebase Support:
You will not be able to use cloud functions if you do not enable the Cloud Build API after this date [April 20, 2020].
And, in a follow-up exchange:
as far as I know in terms of deployment everything will be the same
So, my interpretation of those responses with respect to the original question (will this affect the mechanics of development or deployment) is:
Deployment process: no change necessary
Development process: no change necessary*
*if your build time exceeds 120 minutes per day, and you wish to avoid associated charges, you could voluntarily investigate changing your deployment/development process to reduce build minutes.
Thanks also to Fran for his answer, which sheds light on other aspects of the change.
I have linked GA360 to Big Query. I do have a service account added to GCP as per documentation. The account I used has Project Owner permissions as required to link to said project.
Can I remove the Project Owner permissions from the GCP account once the link has been established in GA360? I do not want that account to have such a high access level to the project.
I did run a test on a small scale and it worked but I am not willing to risk a transfer failure on all of the data in production.
Yes, you can remove the permissions from the account you used to link GA360 to BQ.
The permission is only required for the time of setting this up.
It is not being checked whether the account which set up a connection is still active or has the same rights.
We have had multiple views linked by different accounts, of which most are not in the team anymore and therefore do not have "owner" rights anymore. The exports still work though (which makes sense, given that a company might keep using GA and the exports but part ways with the internal/external employee who sat it up).
Since echo backs of typing character is very slow for me in Japan, it seems that instances of Google Cloud Shell sessions are in some us region.
How do I change the zone of an instance?
Cloud Shell is globally distributed across multiple GCP regions. When a user connects for the first time, the system assigns them to the geographically closest region that can accommodate new users. While the users cannot manually chose their region, the system does its best to pick the closest region Cloud Shell operates in. If Cloud Shell does not initially pick the closest region or if the user later connects from a location that's geographically closer to a different region, Cloud Shell will migrate the user to a closer region on session end.
Since Cloud Shell runs as a GCP VM, you can view your current region by running
curl -H "Metadata-Flavor: Google" metadata/computeMetadata/v1/instance/zone
inside the Cloud Shell session. This is documented in https://cloud.google.com/compute/docs/storing-retrieving-metadata.