Can't access Cosmos Db that has firewall restrictions from Azure Synapse - azure-cosmosdb

I am attempting to setup Synapse to access a Cosmos Db that has firewall rules set to only allow whitelisted IPs.
After a bit of research, I came across this article:
Securing Azure Synapse Workspaces? Beware of One Inescapable Networking Blocker | by Moussa Taifi PhD | Towards Dev
According to that post, the only option is to whitelist the entire range of IPs that might be used by the pool. Can someone let me know if this is indeed the case? I started looking at private endpoints as that seems like a perfect solution, but I can't get it to work. I tried the following multiple times:
Create new CosmosDb with Azure Synapse Link enabled
Restrict to Selected networks
Create a new DB and Container
Verify that I can’t add a new item
Add my IP
Add new item
Create a new Synapse Workspace, choosing Managed VNet
After creation, verify that the Integration Runtime is in the Managed VNet.
Create two new private endpoints for my Cosmos db. One for type Sql, and one for Analytical (I’m not sure which I need yet)
Go to the Private Link center and approve both end points
Data > Connect to External Data
Ensure that my runtime is in the Manage VNet
Select my DB
I waited 10 min, but the managed endpoint list is stuck at “Refreshing.” I continued to save anyway, but when I try to make a SQL call (after creating the credential), I get:
Resolving CosmosDB path has failed with error 'Access to the database account '*******' is forbidden.'.
The endpoints are permanently "Refreshing" in both the properties of the connection and also in the Manage Private Endpoints. The end point links are "approved" and show as such in Cosmos DB.
Can anyone let me know:
Are Private endpoints a method that I can use to connect my Synapse Workspace to my locked down Cosmos DB?
If so, what might I be doing wrong?
Thanks!

You should consider opening a support case in the case of misconfiguration in your settings.
There should be an option to allow access to Azure IPs. This is contained in documentation located here:
Add a managed private endpoint for Azure Cosmos DB analytical store
Sign into the Azure portal.
From the Azure portal, navigate to your Synapse Analytics workspace and open the Overview pane.
Launch Synapse Studio by navigating to Getting Started pane and select Open under Open Synapse Studio.
In the Synapse Studio, open the Manage tab.
Navigate to Managed private endpoints and select New
Create a new private endpoint for analytical store.
Select Azure Cosmos DB(SQL API) account type > Continue.
Select Azure Cosmos DB SQL API to create a private endpoint.
Fill out the New managed private endpoint form with the following details:
Name - Name for your managed private endpoint. This name cannot be updated after it's created.
Description - Provide a friendly description to identify your private endpoint.
Azure subscription - Select an Azure Cosmos DB account from the list of available accounts in your Azure subscriptions.
Azure Cosmos DB account name - Select an existing Azure Cosmos DB account of type SQL or MongoDB.
Target sub-resouce - Select one of the following options: Analytical: If you want to add the private endpoint for Azure Cosmos DB analytical store. Sql (or MongoDB): If you want to add OLTP or transactional account endpoint.
Note
You can add both transactional store and analytical store private endpoints to the same Azure Cosmos DB account in an Azure Synapse Analytics workspace. If you only want to run analytical queries, you may only want to map the analytical private endpoint.
Choose analytical for the target subresource.
After creating, go to the private endpoint name and select Manage approvals in Azure portal.
Navigate to your Azure Cosmos DB account, select the private endpoint, and select Approve.
Navigate back to Synapse Analytics workspace and click Refresh on the Managed private endpoints pane. Verify that private endpoint is in Approved state.

Related

How to copy data from Cosmos DB API for MongoDB to another Cosmos DB account

How to copy data, one collection, from one Cosmos DB API for MongoDB account to another Cosmos DB API for MongoDB account, in another subscription, placed in another Azure region.
Preferably do it periodically.
You can use Azure Data Factory to easily copy a collection from one Cosmos DB API for MongoDB account to another Cosmos DB API for MongoDB account, in any other subscription, placed in any other Azure region simply using Azure Portal.
You need to deploy some required components like Linked Services, Datasets and Pipeline with Copy data activity in order to accomplish this task.
Use Azure Cosmos DB (MongoDB API) Linked Service to connect the Azure Data Factory with your Cosmos DB Mongo API account. Refer Create a linked service to Azure Cosmos DB's API for MongoDB using UI for more details and step to deploy.
Note: You need to deploy two Azure Cosmos DB (MongoDB API) Linked Service, one for source account from where you need to copy the collection, and another for destination account where the data will be copied.
Create Datasets by using Linked service created in above step. Your dataset will connect you to the collection. Again you need to deploy two datasets, one for source collection and another for destination collection. It will look like as shown below.
Now create a pipeline using Copy data activity
In Source and Sink tab in copy data activity settings, select the source dataset and sink dataset respectively which you have created in step 2.
Now just Publish the changes and click on Debug option to run the pipeline once. The pipeline will run and collection will be copied at destination.
If you want to run the pipeline periodically, you can create Trigger based on event or any specific time. Check Create a trigger that runs a pipeline on a schedule for more details.

Create database inside Azure Cosmos DB account with RBAC

I use java version 4 SDK for azure cosmos db. I want to create database inside azure cosmos db account with service principal, not with masterkey.
I assigned to service principal DocumentDB Account Contributor and Cosmos DB Operator built-in-role definitions, according to this documentation:
https://learn.microsoft.com/pl-pl/azure/role-based-access-control/built-in-roles#cosmos-db-operator
I was not able to create CosmosAsyncClient, until I added new custom role, which just contains reading metadata. Above mentioned built-in-role definitions do not contain it...
TokenCredential ServicePrincipal = new ClientSecretCredentialBuilder()
.authorityHost("https://login.microsoftonline.com")
.tenantId(tenant_here)
.clientId(clientid_here)
.clientSecret(secret_from_above_client)
.build();
client = new CosmosClientBuilder()
.endpoint(AccountSettings.HOST)
.credential(ServicePrincipal)
.buildAsyncClient();
After I added this role, client was created, but I am not able to create database instance and also container inside it as next step. In access control I can see that roles are assigned so service principal is correct here.
What is more, when firstly I create database and container with master key and then I want to read/write data using service principal, it works (obviously after adding custom role for writting also).
Then I do not know why DocumentDB Account Contributor and Cosmos DB Operator does not work for creation database.
Looks it is a bug in java SDK, the DocumentDB Account Contributor role is enough to create the database and container as it has the Microsoft.DocumentDb/databaseAccounts/* permission(* is a wildcard, it also includes the Microsoft.DocumentDB/databaseAccounts/readMetadata you mentioned).
When I test to use a service principal with this role to create the database with the powershell New-AzCosmosDBSqlDatabase, it works fine. When using the service principal to run this command, it essentially uses the Azure AD client credential flow to get the token, then uses the token to call the REST API - PUT https://management.azure.com/subscriptions/xxxx/resourceGroups/xxxx/providers/Microsoft.DocumentDB/databaseAccounts/xxxx/sqlDatabases/testdb1?api-version=2020-04-01 to create the database, the java SDK essentially also does the same thing, so it should also work.

Cosmos Graph DB auto-failover when using region-specific Gremlin API endpoints?

Following the advice in this article, I have configured my applications to use region-specific Gremlin endpoints so that reads and writes are always directed to the master replica in the same data centre (the Cosmos DB account is multi-master and the applications are deployed to every region on AKS). My question is this: in the event of a regional Cosmos DB outage, what will the behaviour be when using region-specific Gremlin connection strings? Will applications that reference a regional endpoint that is affected by an outage be automatically redirected to a region where the Cosmos replica is healthy?
This depends upon the client SDK the application is using to connect and the connection string logic. If the application connection string is pointed to the .NET SDK URI, then you will want to implement either the .NET SDK v2 or .NET SDK v3 multi-master functionality. If you are using the Gremlin Endpoint, please follow the specific guidance: Regional endpoints for Cosmos DB for Graph Accounts
Once that is configured correctly, in the event of an outage, the routing will automatically be redirected to an available write region.

AppMaker error say "The instance must be a 2nd generation SQL database" but it is a 2nd gen

AppMaker gives this error: "The default Google Cloud SQL instance is not setup properly. Please ask a G Suite administrator to check the Google Cloud SQL configuration for your domain. (Reason: App Maker is unable to verify the default Google Cloud SQL instance. The instance must be a 2nd generation SQL database.)"
But the SQL database is a 2nd generation
ALF-experiments instances:
Instance ID Type High availability Location Labels
sgialfmysql MySQL 2nd Gen 5.7 us-central1-a
Connected as the default in admin console
Google Cloud SQL instance setting
Enter the Google Cloud SQL instance connection name to use with App Maker:
alf-experiments:us-central1:sgialfmysql
I have made multiple Cloud instances under multiple project but none seem to be able to connect. I have reviewed the documentation several times to verify that GCP and Cloud SQL are set correctly and they are.

How to connect appmaker to a Google SQL over private IP?

when trying to connect to my org SQL cloud from app maker, defined to use only private IP it fails:
The default Google Cloud SQL instance is not setup properly. Please ask a G Suite administrator to check the Google Cloud SQL configuration for your domain. Cannot create new database in the default Google Cloud SQL instance.
any specific service needs to be given to appmaker-maestro to make this work ?
if I connect to a custom instance which has public IP enabled it works all well.
thanks for any guidance you might have.

Resources