AWS Amplify: Resource is not in the state stackUpdateComplete - aws-amplify

I'm setting up aws-amplify to my project. I am facing a problem in amplify push when I configured for the first time it worked fine. now i changed the repository since i had to do sub-tree from the old repo.
Now when i do amplify push i get
Resource is not in the state stackUpdateComplete
⠸ Updating resources in the cloud. This may take a few minutes...Error updating cloudformation stack
⠸ Updating resources in the cloud. This may take a few minutes...
Following resources failed
✖ An error occurred when pushing the resources to the cloud
Resource is not in the state stackUpdateComplete
An error occured during the push operation: Resource is not in the state stackUpdateComplete

Just to give some background about this error - what does Resource is not in the state stackUpdateComplete actually mean?
Well basically Amplify is telling you that one of the stacks in your app did not deploy correctly, but it doesn't know why (which is remarkably unhelpful, but in fairness it's deploying a lot of potentially complex resources).
This can make diagnosing and fixing the issue really problematic, so I've compiled this kind of mental checklist that I go through to fix it. Each of the techniques will work some of the time, but I don't think there are any that will work all of the time. This list is not intended to help you diagnose what causes this issue, it's literally just designed to get you back up and running.
The fast options (will solve most problems)
Try running amplify push --iterative-rollback. It's supposed to roll your environment back to the last successful deployment, but tbh it rarely works.
Try running amplify push --force. Although counter-intuitive, this is actually a rollback method. It basically does what you think --iterative-rollback will do, but works more frequently.
In the AWS console, go to the deployment bucket for your environment (the bucket will be named amplify-${project_name}-${environment_name}-${some_random_numbers}-deployment). If there is a file called deployment-state.json, delete it and try amplify push again from the CLI.
If you are working in a team of more than one developer, or have your environment in several different repos locally, or across multiple different machines, your amplify/team-provider-info.json file might be out of sync. Usually this is caused by the environment variable(s) in an Amplify Lambda function being set in one of the files but not in another. The resolution will depend on how out of sync these files are, but you can normally just copy the contents of the last working team-provider-info.json file across to the other repo (from where the deployment is failing) and run the deployment again. However, if you've got multiple devs/machines/repos, you might be better off diffing the files and checking where the differences are.
The slow option (production-friendly)
Hopefully you haven't got this far, but at this point I'd recommend you open a ticket in the amplify-cli GitHub with as much info as you can. They tend to respond in 1-2 working days.
If you're pre-production, or you're having issues with a non-production environment, you could also try cloning the backend environment in the Amplify console, and seeing if you can get the stack working from there. If so, then you can push the fixed deployment back to the previous env (if you want to) using amplify env checkout ${your_old_env_name} and then amplify push.
The complex option (solves more intricate problems with your stack)
If none of the above work (or you don't have time to wait for a response on a GitHub issue), head over to CloudFormation in the AWS console and search for the part of your stack that is erroring. There's a few different ways to do this:
Check the CLI output for your last push and find the item whose status is something other than UPDATE_COMPLETE. You can copy the name of the stack and search for it in CloudFormation.
Search CloudFormation for your environment name, click on any of the resulting stacks, click the link under Parent stack, repeat until you find a stack with no parent. You are now in the root stack of your deployment, there are two ways to find your erroring stack from here:
Click on the Resources tab and find one with something red in the status column. Select the stack from this row.
Click on the Events tab and find one with something red in the status column. Select the stack from this row.
Once you've found the broken stack, click the Stack actions button and select Detect drift from the dropdown menu.
Click the Stack actions button again and select View drift results from the dropdown menu.
In the Resource drift results page, you'll see a list of resources in the stack. If any of them show DRIFTED in the Drift status column, select the radio button to the left of that item and then click the View drift details button. The drift details will be displayed side by side, git-style, on the next page. You can also click the checkbox(es) in the list above to highlight the drift change(s). Keep the current page open, you'll need it later.
Fixing the drift will depend on what it is - it's usually something in an IAM policy that's changed, you can fix this directly in the console. Sometimes it's a missing environment variable on a Lambda function, which you're better off fixing in the CLI (in which case you would need to run amplify push again and wait for the build to complete in order for the fix to be deployed to your environment).
Once you've fixed the drift, you can click the orange Detect stack drift button at the top of the page and it will update. Hopefully you've solved the problem.
GraphQL bonus round (completely bananas DDB drift)
Another fun thing that Amplify does from time-to-time is to (seemingly spontaneously) change the server-side encryption setting on the definition of some or all of your DynamoDB tables without you even touching it. This is by far and away the most bizarre Amplify error I've encountered (and that's saying something)!
I have a sort-of fix for this, which is to open amplify/backend/api/${your_api_name}/parameters.json and change the DynamoDBEnableServerSideEncryption setting from false to true, save it, then run amplify push. This will fail. But it's fine, because then you just reverse the change (set it back to false), save it, push again and voila! I still cannot for the life of me understand how or why this happens.
I said it's a sort-of fix, and that's because you'll still see drift for the stacks that deploy the affected tables in CloudFormation. This goes away after a while. Again, I have no idea how or why.
The nuclear option (DO NOT USE IN PRODUCTION)
Obviously this one comes with a huge disclaimer: don't do this in production. If working with any kind of DB, you will lose the data.
You can make backups of everything and then start to remove the problematic resources one at a time, with an amplify push in between each one, until the stack build successfully. Once it's built, you can start adding your resources back in.
Hopefully this helps someone, please feel free to suggest edits or other solutions.

This worked for me:
$ amplify update auth
Choose the option “Yes, use default configuration” (uses the Cognito Identitypool).
Then:
$ amplify push
Another reason can be this
The issue is tied to the selection of this option - Select the authentication/authorization services that you want to use: User Sign-Up & Sign-In only (Best used with a cloud API only) which creates just the UserPool and not the IdentityPool which the rootstack is looking for. It's a bug and we'll fix that.
To unblock, for just the first question, you could select - ❯ User
Sign-Up, Sign-In, connected with AWS IAM controls (Enables per-user
Storage features for images or other content, Analytics, and more)
which would create a user pool as well as as the identity pool and
then choose any of the other configurations that you've mentioned
above.

I debugged my AWS Amplify CLI push error by doing the following:
Open CloudFormation
Find parent stack with name such as: amplify-companyName-envName-123456
Click Events tab
Scroll down until you find UPDATE_FAILED, which should give you a detailed description of why it failed. e.g. The following resource(s) failed to create: ...
Alternatively (to find parent stack):
Navigate to environment in AWS Amplify site, Overview tab
Click View in CloudFormation
Under Stack info tab, click link for Parent stack
On the parent page, click Events tab

You can try as below
First do
amplify env checkout {environment} and then
amplify push

The solution is:
a. Go to the s3 bucket containing project settings.
b. locate deployment-state.json file in root folder and delete it.
c. amplify push

I got this after making some modifications to my GraphQL schema. I adjusted the way I was making #connection directives on a few tables. I was able to fix this by following these steps:-
Make a backup copy of your new schema that you're trying to push
Run amplify pull to restore your local to be in sync with your backend in the cloud.
Once that completes, you should have the local synced to the cloud and amplify push should work without flaws because it is synced to the cloud and there should be no updates.
Copy over the new schema onto the pulled schema and try running the amplify push once more to see if it works.
If it doesn't work, undo the overwrite to the pulled schema and compare what is different between the pulled schema and the updated schema that you backed up. Do a line by line diffcheck and see what has changed and try to push the changes one by one to see where it is failing. I think it is wiser to not push too many changes to the schema at once. Do it one by one so that you can troubleshoot more easily. If you do have other issues, then it should be unrelated to the one highlighted in this question, because the pulling should solve this particular issue.

In my case the issue was due to multiple #connections referring to GSI, which were not getting removed and added correctly when I do the amplify push api.
I was able to resolve this by amplify pull then, comment off the #connection then the GSI linked to connection then, add each new changes manually, but there was trouble in GSI getting linked again because the local update considered the GSI already removed but in cloud it seems to be retained, and I got error that a GSI is being added which was already in cloud. So I renamed the model name, so it got recreated to new tables in dynamoDB then I reverted it back to the correct name. This is ideal for dev environment which has no much impact.
But of course it ate up most of my time, but it did fix my issue.

In my case it was an issue when switching between amplify env (checkout), the error was not clear but this is what I did to fix it without having to "clear" api and lose the whole database :
Delete the existing API Key by setting the "CreateAPIKey" to "0" in the "amplify/backend/api//parameters.json" then save file and execute "amplify push".
once done, do the same process with "CreateAPIKey" to "1" then "amplify push".
This fixed my issue.

This worked for me
amplify remove storage
And, then
amplify add storage
Then, again
amplify push
As after amplify add storage I mistakenly choose Y to Do you want to add a Lambda Trigger for your S3 Bucket?
I didn't have any Lamda function and also I didn't have anything in my bucket.

In my opinion, these kind of problems always related to 3rd party auth.
Amplify update auth,
then update auth flow the id and secret of 3rd party.
Then push.
It will fix the problem

It's look like a conflict between backend and local
The only thing that work for me is backing up the local schema and initiating the amplify pulling command.
Then use the back up schema file and initial the amplify push.
In most of case updates in the following file must be set manually (for Android):
app/src/main/res/raw/amplifyconfiguration.json

As mentioned by others in this thread - the issue comes from one of the resources that you updated locally.
Check which ones did you modify:
$ amplify status
Then remove and add it again, followed by push. The Api is known not to work with updates right now, so you must remove it if you've changed it locally:
$ amplify api remove YourAPIName
$ amplify api add
$ amplify push

Related

How can I update dynamodb GSI when projection type is changed?

I am using serverless framework to manage AWS infra and I have defined a dynamodb table with a GSI. Now I need to update the projection type of the GSi and I know that it is not allowed. So I manually delete the GSI from dynamodb table but I still get below error when running sls deploy:
An error occurred: DeviceTable - Cannot update GSI's properties other than Provisioned Throughput. You can create a new GSI with a different name..
What I don't understand is that the GSI has been removed but why it still complains about the GSI's properties. Is there any place remember the GIS? What is the right way to do that?
I had the same error. I work with the CDK but I think the problem is the same with the Serverless Framework. I solved it as follows:
Delete the affected indexes from your configuration file (serverless.yaml or CDK files).
Deploy the complete stack, not only one function
Add the new or changed index again
Deploy the complete stack, not only one function
This is how it worked for me.
ps: you can't replace the index with a new one, as mentioned above, you need to delete the old one and then add the new one. Or you will end up with an error message like this one: "Cannot perform more than one GSI creation or deletion in a single update"
I have faced the same issue. I tried to changed my ProjectionType but after that my serverless deployment started failing with the same error.
As mentioned in the question, I also deleted my GSI from DynamoDb, but that did not help me as well.
After Googling a lot, I did not get a proper answer. But this is how I resolved it finally.
When you get this error, you also get a link to the CloudFormation
stack which you are trying to update with your GSI changes.
Follow this link and go to your CloudFormation stack, there click on Template tab.
After that click on View in Designer button or link whatever you call it.
It will load your stack in Designer template.
At the botton window, it will ask you to Choose template Language, chose YAML as the option.
Locate your DynamoDb table where you are trying to apply GSI. Here you will see the old previous GSI settings.
Now please remove only the GSI settings carefully without removing anything related to DynamoDb table settings.
Now from the top left, you will see a couple of options to Redo, Undo, and Create Stack etc. There you will also find a link Save.
Save the template as a Serverless file to your local computer.
Now go back to Template tab and this time click on the button Update.
Chose the option Replace current template and the Update a template file. Here upload the Serverless file which you saved earlier, and click on next to finally upload it.
Now wait for sometime to let the CloudFormation update the template for you.
That's it, now if you run your yarn sls command again, it won't stop you to let your new DynamoDb GSI projection settings get deployed.
One more tip, it takes some time to create a new GSI on Cloud, so please be patient and look for the Status Active on your DynamoDb table under the tab Indexes.

Azure Devops publishing to own feed suddenly results in 403 forbidden

I have been using Azure DevOps for a project for quite some time, but suddenly publishing to my own organisation/collection feed results in a 403.
I created a feed and I can select it on the nuget push build step, but it does not work. I created a new feed to publish the NuGet packages to and this works perfectly again. It seems to me like a token expired, but I never created one or used it to authenticate. I also do not want to change my NuGet feed to the new one, as I want to use older packages as well.
This is the buildpipeline:
And this is the stack trace:
Active code page: 65001 SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true SYSTEMVSSCONNECTION exists true
[warning]Could not create provenance session: {"statusCode":500,"result":{"$id":"1","innerException":null,"message":"User
'a831bb9f-aef5-4b63-91cd-4027b16710cf' lacks permission to complete
this action. You need to have
'ReadPackages'.","typeName":"Microsoft.VisualStudio.Services.Feed.WebApi.FeedNeedsPermissionsException,
Microsoft.VisualStudio.Services.Feed.WebApi","typeKey":"FeedNeedsPermissionsException","errorCode":0,"eventId":3000}}
Saving NuGet.config to a temporary config file. Saving NuGet.config to
a temporary config file. [command]"C:\Program Files\dotnet\dotnet.exe"
nuget push d:\a\1\a\Microwave.0.13.3.2019072215-beta.nupkg --source
https://simonheiss87.pkgs.visualstudio.com/_packaging/5f0802e1-99c5-450f-b02d-6d5f1c946cff/nuget/v3/index.json
--api-key VSTS error: Unable to load the service index for source https://simonheiss87.pkgs.visualstudio.com/_packaging/5f0802e1-99c5-450f-b02d-6d5f1c946cff/nuget/v3/index.json.
error: Response status code does not indicate success: 403
(Forbidden - User 'a831bb9f-aef5-4b63-91cd-4027b16710cf' lacks
permission to complete this action. You need to have 'ReadPackages'.
(DevOps Activity ID: 2D81C262-96A3-457B-B792-0B73514AAB5E)).
[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1
[error]Packages failed to publish
[section]Finishing: dotnet push to own feed
Is there an option I am overlooking where I have to authenticate myself somehow? It is just so weird.
"message":"User 'a831bb9f-aef5-4b63-91cd-4027b16710cf' lacks
permission to complete this action. You need to have 'ReadPackages'.
According to this error message, the error you received caused by the user(a831bb9f-aef5-4b63-91cd-4027b16710cf) does not have the access permission to your feed.
And also, as I checked from backend, a831bb9f-aef5-4b63-91cd-4027b16710cf is the VSID of your Build Service account. So, please try with adding this user(Micxxxave Build Service (sixxxxss87)) into your target feed, and assign this user the role of Contributor or higher permissions on the feed.
In addition, here has the doc you can refer:
There is a new UI in the Feed Permissions:
To further expand on Merlin's solution & related links (specifically this one about scope), if your solution has only ONE project within it, Azure Pipelines seems to automatically restrict the scope of the job agent to the agent itself. As a result, it has no visibility of any services outside of it, including your own private NuGet repos held in Pipelines.
Solutions with multiple projects automatically have their scope unlocked, giving build agents visibility of your private NuGet feeds held in Pipelines.
I've found the easiest way to remove the scope restrictions on single project builds is to:
In the pipelines project, click the "Settings" cog at the bottom left of the screen.
Go to Pipelines > Settings
Uncheck "Limit job authorization scope to current project"
Hey presto, your 403 error during your builds involving private NuGet feeds should now disappear!
I want to add a bit more information just in case somebody ends up having the same kind of problem. All information shared by the other users is correct, there is one more caveat to keep into consideration.
The policies settings are superseded by the organization settings. If you find yourself unable to modify the settings or they are grayed out click on "Azure DevOps" logo at the left top of the screen.
Click on Organization Settings at the bottom left.
Go to Pipeline --> Settings and verify the current configuration.
When I created my organization it was limiting the scope at the organization level. It took me a while to realize it was superseding the project.
Still wondering where that "Limit job authorization scope to current project" setting is, took me a while to find it, its in the project settings, below screenshot should help
It may not be immediately obvious or intuitive, but this error will also occur when the project your pipeline is running under is public, but the feed it is accessing is not. That might be the case, for instance, when accessing an organization-level feed.
In that scenario, there are three possible resolutions:
Make the feed public, in which case authentication isn't required; or
Make the project private, thus forcing the service to authenticate; or
Include the Allow project-scoped builds under your feed permissions.
The instructions for the last option are included in #Merlin Liang - MSFT's excellent answer, but the other options might be preferable depending on your requirements.
At minimum, this hopefully provides additional insight into the types of circumstances that can lead to this error.
Another thing to check, if using a yaml file for the Pipelines, is if the feed name is correct.
I know this might seem like a moot point, but I spent a long time debugging the ..lacks permission to complete this action. You need to have 'AddPackage'. error only to find I had referenced the wrong feed in my azure-pipelines.yaml file.
If you don't want to/cannot change Project-level settings like here
You can set this per feed by clicking 'Allow Project-scoped builds' (for me greyed out as it's already enabled).
That's different from the accepted answer, as you don't have to explicitly add the user and set the permissions.
Adding these two permissions solved my issue.
Project Collection Build Service (PROJECT_NAME)
[PROJECT_NAME]\Project Collection Build Service Accounts
https://learn.microsoft.com/en-us/answers/questions/723164/granting-read-privileges-to-azure-artifact-feed.html
If I clone an existing pipeline that works and modify it for a new project the build works fine.
But if I try to create a new pipeline I get the 403 forbidden error.
This may not be a solution but I have tried everything else suggest here and elsewhere but I still cannot get it to work.
Cloning worked for me.

Previewing failed, but deployment works

Background:
I'm quite new to App Maker, but have been involved in programming/IT for over 2 decades.
I have created an App Maker app, which works fine. It is deployed, and functions internally in our organization.
It accesses a Team Drive spreadsheet, makes modifications to it based on input criteria, and sends an email out to a hardcoded user. It uses no external GCP database or other resource.
The OAuth scopes it requires are:
admin.directory.user.readonly
drive.readonly
script.send_mail
spreadsheets
userinfo.email
Problem:
I can no longer preview the app.
When I click on "Preview" at the top right, a new tab opens and a spinning wheels seems to indicate that the preview is loading. Within about 4 seconds, the tab closes and the original tab (with the scripts, UI etc) gives a "Previewing failed. Dismiss" error in the bottom centre.
I am both able to deploy the exact same code/UI/etc, as well as run it without issue.
I do not know what I changed, since being able to preview the app, but cannot seem to regress to that state.
What I've tried:
Admittedly not much, as I don't know where to look. I'm rather certain that there must be some setting somewhere, but for all my googling, I've come up empty.
This can't be a client/server script or other syntax issue, as otherwise the deployment also wouldn't work.
With a more meaningful error, I would know where to look.
Expected Result:
Obviously, I should be able to preview the app if it is deployable.
Following #Morfinismo's comment, I contacted G Suite Support; my matter was escalated to the API Team.
I was asked by Google Cloud Support ("Support") to provide network traffic info using FiddlerCap. As I am on a linux machine and FiddlerCap is a windows application, I suggested alternatives (eg:Wireshark). It was eventually not required and never provided.
I noticed that on the Google scripts page, when accessing the project in question selecting "Preview", it was missing the following OAuth:
https://www.googleapis.com/auth/admin.directory.user.readonly
The functioning deployed version did not have this missing.
Still in the Preview, I selected "Stackdriver (logs)", which gave me an error that the project had been deleted. The actual wording was:
Access forbidden
Project XXXX is shut down and scheduled to be deleted. A project owner can cancel the shutdown on the projects list page.
Clicking on the link in the error "Go to projects list page" brought me to a page with the title "Resources pending deletion", which did not load a list of projects (but otherwise fully loaded) and would display the spinning wheel in perpetuity. I attempted this multiple times, including leaving it overnight once.
Support presumed that I had deleted the GCP project, although I honestly don't/didn't think I had. I also confirmed that creating new previews did not work, but creating new deployments did. I also confirm(ed) that this particular App Maker app did not require (eg) a GCP SQL database.
Support pointed me toward the following website: Google undelete project and I was asked to follow these steps (copy-pasted here):
a. For projectId, enter your project ID. From the screenshot you provided, this is "XXX (redacted)" (the quotes are just to emphasise the project ID, you shouldn't enter them.
b. Click EXECUTE.
c. You'll be prompted to grant authorisation, which may be preceded by a prompt to choose your admin account. Please do so.
d. You should receive a 200 response, with an empty body, that is {}.
e. Attempt to access the project via Project link (with the actual project id redacted here).
The above yielded some strange behaviour:
a. undeleteing the project gave it a different name that the App Maker app;
b. I notice that I had 3 other projects all called correctly (the App Maker app name).
c. When asked to reauthorize, I was provided with yet another project name ("Untitled project"), which was different from the correct one and different from the one in para. 7a, above.
d. I then also obtained another error in a new window which read:
That's an error
Request Details
(a bunch of stuff)
That's all we know.
Support advised that there may be a propagation issue, and that I should wait up to 30 minutes. I did, and it then worked! The only weird thing was the project name was wrong, but it was only for the preview, so I didn't really care.
If anyone needs additional information, I can PM screenshots I took along the way.
Hope this helps someone!
SJL

Firebase back-up

I use Firebase for an Android project and I have some data into a database. I want to know if there is any possibility to save/make a back up to the structure of data. I ask this because somehow, I deleted the project and I had to do all again.
If you just deleted your project, you may still be able to restore it from the Resources pending deletion page in the Google Cloud Platform console with the process shown here.
If that doesn't work, reach out to Firebase support to see if they can help you recover it.
Note that both of these are dependent on the time that has passed since you deleted the project, so time is of the essence.

Custom entities not shown in default solution screen - Dynamics CRM 2013

I've come across this really strange scenario that I'm hoping that someone else has come across (best scenario) or perhaps that someone with a lot of CRM experience can perhaps point me in the direction of where to look for a possible solution. I cant seem to find anything online thats close to this issue.
We have 4 environments : DEV, DEV-IST, UAT and PreProd. We progress CRM deployments in the following order: DEV -> DEV-IST -> UAT -> PreProd. We havent gone live with our development and hence I've not included Production.
We progress CRM Deployments using the following steps.
Go to DEV.
Export any unmanaged solutions to managed solutions. We then copy those managed solutions into the 'PkgFolder' of the package deployer.
We then take a backup of the DEV-IST CRM (Organisation A) database and restore it to the DEV database server as (Organisation A). We then run the package deployer on Organisation A so that it now has all the changes that we'd made in development. Before we do anything further, we login to Organisation A and go to the following screen: Home -> Settings -> Customisations -> Customize the System. I can see under the Entities node all the custom entities.
I now backup Organisation A (which now includes my Dev updates) and copy the backup file to the DEV-IST database server.
I first go to Deployment Manager on DEV-IST and disable the organisation and then delete it.
Now I go to the SQL Database for DEV-IST and restore the backup from step 4 above and overwrite the Organisation A database.
I go back into Deployment Manager on DEV-IST and use the import organisation to reimport the organisation A and thus now have a organisation A now updated with my dev updates. The above process ensures that we deploy updates with very little downtime.
However, my issue is that after step 7 when I login to the organisation and go to Home -> Settings -> Customisations -> Customize the System... I cant see any of the custom entities!
Any ideas what might be causing this. Strangely I've noticed that I cant see custom entities in DEV-IST and PreProd but can see them in the DEV and UAT environments.
PS. I've already tried clicking the 'Publish all customizations' and it didnt have any effect.
Your deployment process is really complicated and I can't see any reason why are you doing customization transfer in so complex way. You should be simply transferring Solutions with your customizations from DEV environment to DEV-IST environment and then to UAT and PreProd.
Backing up your database on one environment and restoring it on another is for sure not faster way (because you stated that such process gives you short downtime of environment).
Also if you insist of doing deployment this way, you should not only remove your organization from Deployment Manager, but also database from the server (and restore the backed up database).
If you are simply customizing CRM for your client/company, you should not use managed solutions. Managed solutions are used to share some reusable customizations that you want to share or sell to the others. Please read this great post by Shan McArthur, that will make you better understand the problem:
https://community.adxstudio.com/blogs/shan/2014-01-17-converting-crm-solutions-from-managed-to-unmanaged/
And if you must use managed solutions - you should only do that on your final PROD environment, not on your staging environments (where you should have full control over customizations to provide hotfixes)
EDIT:
I would add it as a comment, but I cannot comment yet, so I have to edit my response instead. I replicated your process and I can see all the entities on all environments, so there must be something specific that you are doing which is causing this behaviour. Have you checked that after you restored the database all the custom entities exist in your restored database? (every entity is a separate table which has the same name as the entity). Also I would compare all the solution-related tables in database where you can see the custom entities vs the database where you cannot see custom entities (so that would be tables SolutionBase, SolutionComponenBase, MetadataSchema.Entity and all other MetadataSchema.* tables). Last experiment you can do is to save somewhere the db where you cannot see the custom entities, remove the managed solution, import the managed solution (standard way) and check if the entities are there. I bet that they would be there, so the next step would be to again compare the databases (all the metadata/solution tables) with some red-gate comparison tool, maybe that would point you to the right direction.
Basically the metadata in CRM is built this way - take everything from the default solution, then apply all the managed solutions in order they were imported and build the resulting metadata. Your case looks like CRM somehow "forgot" to apply the managed solutions over the default solution ( I believe that this might be an unknown bug of deployment manager, that should set something during organization import). I hope that you will discover this when you compare your database data between environments.
I understand that this is not a valid response and normally I would not write it as an answer but as a comment, but I cannot comment yet, unfortunately :(
I finally got to the bottom of this issue and thought I'd update it here for anyone else who comes along with the same issue. 'Someone' put a registry edit on the CRM server to get CRM to return only the top 250 records on any query. As custom entities are above this number, they weren't being shown. I've since reverted this change and I'm able to see the custom entities now.

Resources