Google Cloud Build can't pull Container Registry images after 1 day - firebase

We have CICD enabled, GCP Cloud Build has all the permissions (including Editor, Storage Admin and many many other permissions)
When we follow the steps for getting the image for the first time using instructions here: https://cloud.google.com/cloud-build/docs/deploying-builds/deploy-firebase, cloud build works perfectly.
However, after some time (say 1 day), it starts giving error "unable to pull the image"
Step #0: latest: Pulling from <project name>/firebase
Step #0: 81fc19181915: Pulling fs layer
Step #0: 828510924538: Pulling fs layer
...
Step #0: fe190282039f: Waiting
Step #0: error pulling image configuration: unknown blob
CANCELLED
ERROR: context canceled
ERROR: failed to pull because we ran out of retries.
If we upload the image again, it starts working again only for some time.
Note: We checked and the image is there in Container Registry, but still for some reason Google Cloud Build can't find it.
How can we fix this? Any suggestions on what we need to look at?
Uploading image every single time makes the whole CICD process redundant as we might as well deploy manually.
Thank you,

I had the exact same issue - after pulling images successfully for a period I would start getting the error "unable to pull the image". Permissions were not the issue.
I found my problem. Google GCP Container Registry stores the Docker artefacts in Google GCP Storage in artifacts.<PROJECT_ID>.appspot.com. I set 2 week expiry times on all my Cloud Storage locations to save money. Therefore whenever a Docker artefact was deleted in Cloud Storage all the related Docker images in GCP Container Registry would report an unknown blob when trying to download the deleted slice and the image download would fail. Maybe Google could have explained the link to Cloud Storage better but I suppose should not have set expiry times on storage locations for which I didn't know the exact purpose.

As you mentioned that the Cloud Build only works for one day, you need to re-upload the image to make sure it is working. I think the issue may be caused by the cloud build configuration, maybe be the trigger or tags. Since it is not appropriate to provide your cloud build configuration file here. I recommend you to open a support ticket in the Cloud Console. You can refer to this link to get the instruction of how to File a support case.

Related

Cannot access Firebase Storage after creating storage bucket in GCP Console

I noticed that Firebase wouldn't let me create a storage bucket asia-southeast1 so I upgraded my plan to blaze, went to the GCP console and created a bucket from there.
I did the same for Firestore and it's working fine.
However, after I created the storage bucket in GCP, and I came back to the Firebase console to access Firebase Cloud Storage, I keep receiving this error.
I don't see how creating a bucket in the GCP console may affect your Firebase project (unless you created a bucket named <your-project-name>.appspot.com).
These kinds of errors normally happen when a certain quota has been reached, and is usually solved by upgrading your project to a paid plan.
As you have already upgraded your project, please bear into account that billing updates usually take some time to be processed. Waiting a few hours, or maybe checking the next day whether your project has been successfully upgraded should do the trick.
If that does not help, verify whether you have actually upgraded your project - You can check that here. The Blaze Plan button should be grayed out with the message Current Plan.
The answer provided to this question may also be of help:
Try to activate this feature using an incognito window/private window
and disable all browser extensions
Try using a different browser (and make sure browser is supported)
Verify if there are any other programs that might be blocking your
connection such as antivirus, proxy or firewall. If yes, try to
disable them.
Connect using a different network.
Connect using a different device.
Check the Developer Tools Console for any errors.
For Safari: Safari > Preferences > Advanced > Check "Show Develop menu
in menu bar" Develop > Show Web Inspector > Console Tab
If none of these approaches work for you, consider contacting Firebase Support here.
I fixed mine by using the command to work on it using
firebase init storage
This would create the storage after that it would add a storage.rule in which you can update to your liking in your project .
Then you run this command
firebase deploy --only storage
This would initialize the Storage
If you are still having issues check out this link https://stackoverflow.com/a/74750337 which was also provided by firebase for me to fix the error

Reduce latency when rebuilding/updating Google Cloud Composer?

I am working with Google Cloud Composer, and whenever I change an environment variable or or any setting in airflow, it will trigger a rebuild of the airflow environment.
I was thinking there may be a way to resolve package dependencies (ie. upload python packages) to cut back on the rebuild latency.
I ask because the rebuild can take anywhere from 2-15 mins.
Has anyone had any luck with reducing this build time (with or without increasing costs)?
Environment updates can generally take between 5-30 minutes in Cloud Composer and it works as intended at this moment. Please check this public issue tracker for more insight. You can click on +1 to make it more visible to the Cloud Composer engineering team.
Please note, that the Composer needs to take care of a lot of resources, i.e. deploys Airflow within the Google Kubernetes Engine and App Engine, which forces rebuilding images of the container, updating the Airflow Webserver and so on.
I suggest you to take a look in the following Cloud Composer architecture documentation, where you can find all the components that needs to be updated with each change.
You can check if any of your PODs are in Evicted state, which means that your node is on low resource, so you should consider using higher resource machine instead of the standard one n1-standard-2.
I hope you find the above pieces of information useful.

AWS Amplify: Resource is not in the state stackUpdateComplete

I'm setting up aws-amplify to my project. I am facing a problem in amplify push when I configured for the first time it worked fine. now i changed the repository since i had to do sub-tree from the old repo.
Now when i do amplify push i get
Resource is not in the state stackUpdateComplete
⠸ Updating resources in the cloud. This may take a few minutes...Error updating cloudformation stack
⠸ Updating resources in the cloud. This may take a few minutes...
Following resources failed
✖ An error occurred when pushing the resources to the cloud
Resource is not in the state stackUpdateComplete
An error occured during the push operation: Resource is not in the state stackUpdateComplete
Just to give some background about this error - what does Resource is not in the state stackUpdateComplete actually mean?
Well basically Amplify is telling you that one of the stacks in your app did not deploy correctly, but it doesn't know why (which is remarkably unhelpful, but in fairness it's deploying a lot of potentially complex resources).
This can make diagnosing and fixing the issue really problematic, so I've compiled this kind of mental checklist that I go through to fix it. Each of the techniques will work some of the time, but I don't think there are any that will work all of the time. This list is not intended to help you diagnose what causes this issue, it's literally just designed to get you back up and running.
The fast options (will solve most problems)
Try running amplify push --iterative-rollback. It's supposed to roll your environment back to the last successful deployment, but tbh it rarely works.
Try running amplify push --force. Although counter-intuitive, this is actually a rollback method. It basically does what you think --iterative-rollback will do, but works more frequently.
In the AWS console, go to the deployment bucket for your environment (the bucket will be named amplify-${project_name}-${environment_name}-${some_random_numbers}-deployment). If there is a file called deployment-state.json, delete it and try amplify push again from the CLI.
If you are working in a team of more than one developer, or have your environment in several different repos locally, or across multiple different machines, your amplify/team-provider-info.json file might be out of sync. Usually this is caused by the environment variable(s) in an Amplify Lambda function being set in one of the files but not in another. The resolution will depend on how out of sync these files are, but you can normally just copy the contents of the last working team-provider-info.json file across to the other repo (from where the deployment is failing) and run the deployment again. However, if you've got multiple devs/machines/repos, you might be better off diffing the files and checking where the differences are.
The slow option (production-friendly)
Hopefully you haven't got this far, but at this point I'd recommend you open a ticket in the amplify-cli GitHub with as much info as you can. They tend to respond in 1-2 working days.
If you're pre-production, or you're having issues with a non-production environment, you could also try cloning the backend environment in the Amplify console, and seeing if you can get the stack working from there. If so, then you can push the fixed deployment back to the previous env (if you want to) using amplify env checkout ${your_old_env_name} and then amplify push.
The complex option (solves more intricate problems with your stack)
If none of the above work (or you don't have time to wait for a response on a GitHub issue), head over to CloudFormation in the AWS console and search for the part of your stack that is erroring. There's a few different ways to do this:
Check the CLI output for your last push and find the item whose status is something other than UPDATE_COMPLETE. You can copy the name of the stack and search for it in CloudFormation.
Search CloudFormation for your environment name, click on any of the resulting stacks, click the link under Parent stack, repeat until you find a stack with no parent. You are now in the root stack of your deployment, there are two ways to find your erroring stack from here:
Click on the Resources tab and find one with something red in the status column. Select the stack from this row.
Click on the Events tab and find one with something red in the status column. Select the stack from this row.
Once you've found the broken stack, click the Stack actions button and select Detect drift from the dropdown menu.
Click the Stack actions button again and select View drift results from the dropdown menu.
In the Resource drift results page, you'll see a list of resources in the stack. If any of them show DRIFTED in the Drift status column, select the radio button to the left of that item and then click the View drift details button. The drift details will be displayed side by side, git-style, on the next page. You can also click the checkbox(es) in the list above to highlight the drift change(s). Keep the current page open, you'll need it later.
Fixing the drift will depend on what it is - it's usually something in an IAM policy that's changed, you can fix this directly in the console. Sometimes it's a missing environment variable on a Lambda function, which you're better off fixing in the CLI (in which case you would need to run amplify push again and wait for the build to complete in order for the fix to be deployed to your environment).
Once you've fixed the drift, you can click the orange Detect stack drift button at the top of the page and it will update. Hopefully you've solved the problem.
GraphQL bonus round (completely bananas DDB drift)
Another fun thing that Amplify does from time-to-time is to (seemingly spontaneously) change the server-side encryption setting on the definition of some or all of your DynamoDB tables without you even touching it. This is by far and away the most bizarre Amplify error I've encountered (and that's saying something)!
I have a sort-of fix for this, which is to open amplify/backend/api/${your_api_name}/parameters.json and change the DynamoDBEnableServerSideEncryption setting from false to true, save it, then run amplify push. This will fail. But it's fine, because then you just reverse the change (set it back to false), save it, push again and voila! I still cannot for the life of me understand how or why this happens.
I said it's a sort-of fix, and that's because you'll still see drift for the stacks that deploy the affected tables in CloudFormation. This goes away after a while. Again, I have no idea how or why.
The nuclear option (DO NOT USE IN PRODUCTION)
Obviously this one comes with a huge disclaimer: don't do this in production. If working with any kind of DB, you will lose the data.
You can make backups of everything and then start to remove the problematic resources one at a time, with an amplify push in between each one, until the stack build successfully. Once it's built, you can start adding your resources back in.
Hopefully this helps someone, please feel free to suggest edits or other solutions.
This worked for me:
$ amplify update auth
Choose the option “Yes, use default configuration” (uses the Cognito Identitypool).
Then:
$ amplify push
Another reason can be this
The issue is tied to the selection of this option - Select the authentication/authorization services that you want to use: User Sign-Up & Sign-In only (Best used with a cloud API only) which creates just the UserPool and not the IdentityPool which the rootstack is looking for. It's a bug and we'll fix that.
To unblock, for just the first question, you could select - ❯ User
Sign-Up, Sign-In, connected with AWS IAM controls (Enables per-user
Storage features for images or other content, Analytics, and more)
which would create a user pool as well as as the identity pool and
then choose any of the other configurations that you've mentioned
above.
I debugged my AWS Amplify CLI push error by doing the following:
Open CloudFormation
Find parent stack with name such as: amplify-companyName-envName-123456
Click Events tab
Scroll down until you find UPDATE_FAILED, which should give you a detailed description of why it failed. e.g. The following resource(s) failed to create: ...
Alternatively (to find parent stack):
Navigate to environment in AWS Amplify site, Overview tab
Click View in CloudFormation
Under Stack info tab, click link for Parent stack
On the parent page, click Events tab
You can try as below
First do
amplify env checkout {environment} and then
amplify push
The solution is:
a. Go to the s3 bucket containing project settings.
b. locate deployment-state.json file in root folder and delete it.
c. amplify push
I got this after making some modifications to my GraphQL schema. I adjusted the way I was making #connection directives on a few tables. I was able to fix this by following these steps:-
Make a backup copy of your new schema that you're trying to push
Run amplify pull to restore your local to be in sync with your backend in the cloud.
Once that completes, you should have the local synced to the cloud and amplify push should work without flaws because it is synced to the cloud and there should be no updates.
Copy over the new schema onto the pulled schema and try running the amplify push once more to see if it works.
If it doesn't work, undo the overwrite to the pulled schema and compare what is different between the pulled schema and the updated schema that you backed up. Do a line by line diffcheck and see what has changed and try to push the changes one by one to see where it is failing. I think it is wiser to not push too many changes to the schema at once. Do it one by one so that you can troubleshoot more easily. If you do have other issues, then it should be unrelated to the one highlighted in this question, because the pulling should solve this particular issue.
In my case the issue was due to multiple #connections referring to GSI, which were not getting removed and added correctly when I do the amplify push api.
I was able to resolve this by amplify pull then, comment off the #connection then the GSI linked to connection then, add each new changes manually, but there was trouble in GSI getting linked again because the local update considered the GSI already removed but in cloud it seems to be retained, and I got error that a GSI is being added which was already in cloud. So I renamed the model name, so it got recreated to new tables in dynamoDB then I reverted it back to the correct name. This is ideal for dev environment which has no much impact.
But of course it ate up most of my time, but it did fix my issue.
In my case it was an issue when switching between amplify env (checkout), the error was not clear but this is what I did to fix it without having to "clear" api and lose the whole database :
Delete the existing API Key by setting the "CreateAPIKey" to "0" in the "amplify/backend/api//parameters.json" then save file and execute "amplify push".
once done, do the same process with "CreateAPIKey" to "1" then "amplify push".
This fixed my issue.
This worked for me
amplify remove storage
And, then
amplify add storage
Then, again
amplify push
As after amplify add storage I mistakenly choose Y to Do you want to add a Lambda Trigger for your S3 Bucket?
I didn't have any Lamda function and also I didn't have anything in my bucket.
In my opinion, these kind of problems always related to 3rd party auth.
Amplify update auth,
then update auth flow the id and secret of 3rd party.
Then push.
It will fix the problem
It's look like a conflict between backend and local
The only thing that work for me is backing up the local schema and initiating the amplify pulling command.
Then use the back up schema file and initial the amplify push.
In most of case updates in the following file must be set manually (for Android):
app/src/main/res/raw/amplifyconfiguration.json
As mentioned by others in this thread - the issue comes from one of the resources that you updated locally.
Check which ones did you modify:
$ amplify status
Then remove and add it again, followed by push. The Api is known not to work with updates right now, so you must remove it if you've changed it locally:
$ amplify api remove YourAPIName
$ amplify api add
$ amplify push

Firebase back-up

I use Firebase for an Android project and I have some data into a database. I want to know if there is any possibility to save/make a back up to the structure of data. I ask this because somehow, I deleted the project and I had to do all again.
If you just deleted your project, you may still be able to restore it from the Resources pending deletion page in the Google Cloud Platform console with the process shown here.
If that doesn't work, reach out to Firebase support to see if they can help you recover it.
Note that both of these are dependent on the time that has passed since you deleted the project, so time is of the essence.

Authenticating Firebase Database with Google API via Firebase Functions

here's the task I have and the resources I'm following to try to get it done.
Ultimately what I want is for users to be able to take pictures with my app (made with Ionic) and then those pictures get sorted and end up on the company's Google Team Drive.
What I originally started with was uploading the images to Firebase Storage, and then I was told they must be accessible to Team Drive.
Some of the people taking pictures aren't part of the company's G Suite, so I was thinking it would be easier to just have the database itself authenticate with the Team Drive and upload from there.
Here is an example from github that shows the same thing I want, except it uploads from Firebase Database to Google Sheets:
https://github.com/firebase/functions-samples/tree/master/google-sheet-sync
My plan was to follow this guide exactly to get it working, and then figure out how I can change it for Storage -> Team Drive.
However in following the instructions in the readme (they're pretty old) I've come across an error and now I'm stuck.
For Step #7 in that link where it says to run $ firebase functions:config:set googleapi.client_id="<id>" and the rest of the command, I get in terminal:
Error: HTTP Error: 403, The caller does not have permission
What should be the steps I'm looking at to get this working? I am also unsure if my current implementation makes sense anymore. It wouldn't be a big deal to make everyone using the app sign-in with a Google account so that they can be a member of the Team Drive and just upload directly from the app to Drive. Would that still include the sort of safety behaviour like if the connection drops the file upload will resume independently?
The answer to my first question is to install googleapis and google-auth-library. Run the command in the actual /functions/ directory of the sample.
There were a few other errors created by old code, but they aren't in the scope of this question and I raised them in the github Issues sections.

Resources