A vendor provided access to their server via ODBC connector that created a SYSTEM DSN on my local machine. I would like to replicate the data in Azure Synapse data lake for analysis.
The Azure Synapse Linked Service configuration does not allow me to reference the newly linked server shown in image 1.
There are possibly a few ways to connect the two
Add connection parameters in Azure Synapse to reference the linked server
Setup the integration as ODBC connector instead of SQL server
(Last resort) Create a duplicated image of the data to connect to Azure Synapse
Guidance and preference would be greatly appreciated on the best method
Local SSMS
Azure Synapse Linked Service
AFAIK we can't connect to the linked servers of SQL server. If we want to copy the data of linked server, we can copy the tables into database and copy the data to blob storage.
For that I followed below procedure:
I created ODBC linked server in SQL server. I retrieved some data.
Image for reference:
I copied the linked server table to database using below code:
SELECT *
INTO db.dbo.product
FROM linkedserver.AZURE SYNAPSE.dbo.product
The table is copied successfully. I created synapse workspace and created selfhosted IR and connected successfully.
I created linked service for SQL server through selfhosted IR:
I am having data lake storage linked service. I create new pipeline:
Performed copy data activity following below procedure:
I created sql server dataset using sql server linked service and selected dbo.product table as source:
Source data preview:
I selected synapse default storage, selected delimited text and selected the folder as sink.
I run the Copy activity. It run successfully.
Delimited file is stored storage.
In this way you can copy data from linked server of sql server.
Related
How to copy data, one collection, from one Cosmos DB API for MongoDB account to another Cosmos DB API for MongoDB account, in another subscription, placed in another Azure region.
Preferably do it periodically.
You can use Azure Data Factory to easily copy a collection from one Cosmos DB API for MongoDB account to another Cosmos DB API for MongoDB account, in any other subscription, placed in any other Azure region simply using Azure Portal.
You need to deploy some required components like Linked Services, Datasets and Pipeline with Copy data activity in order to accomplish this task.
Use Azure Cosmos DB (MongoDB API) Linked Service to connect the Azure Data Factory with your Cosmos DB Mongo API account. Refer Create a linked service to Azure Cosmos DB's API for MongoDB using UI for more details and step to deploy.
Note: You need to deploy two Azure Cosmos DB (MongoDB API) Linked Service, one for source account from where you need to copy the collection, and another for destination account where the data will be copied.
Create Datasets by using Linked service created in above step. Your dataset will connect you to the collection. Again you need to deploy two datasets, one for source collection and another for destination collection. It will look like as shown below.
Now create a pipeline using Copy data activity
In Source and Sink tab in copy data activity settings, select the source dataset and sink dataset respectively which you have created in step 2.
Now just Publish the changes and click on Debug option to run the pipeline once. The pipeline will run and collection will be copied at destination.
If you want to run the pipeline periodically, you can create Trigger based on event or any specific time. Check Create a trigger that runs a pipeline on a schedule for more details.
I created Azure CosmosDb database and container for my documents.
I use NiFi as a main data ingestion tool and want to feed my container with documents from NiFi flow files.
Can anybody please share a way to post flowfile content to Azure Cosmos Db from NiFi?
Thanks in advance
UPDATE(2019.05.26):
In the end I used Python script and called it from NiFi to post messages. I passed a message as a parameter. The reason I chose python is because it has some examples on official Microsoft site with all the required connection settings and libraries, so it was easy to connect to Cosmos.
I tried Mongo component, but couldn't connect to Azure (security config didn't work), didn't really go too far with it as Python script worked just fine.
Azure CosmosDB exposes MongoDB API so you can use the following MongoDB processors which are available in NiFi to read/query/write to & from Azure CosmosDB using Apache NiFi.
DeleteMongo
GetMongo
PutMongo
PutMongoRecord
RunMonogAggregation
Useful Links
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-feature-support
Valeria. According to the components list supported by Apache Nifi related to Azure, you could only get Azure Blob Storage, Queue Storage, Event Hub etc,not including Cosmos DB.
So,I suggest you using PutAzureBlobStorage to feed azure blob container with documents from NiFi flow files. Then please create a copy activity pipeline in Azure Data Factory to transfer data from Azure Blob Storage into Azure Cosmos DB.
I'm new to CosmosDB and have used Data Factory to import some test data from a BLOB into a CosmoDB container. The Monitor screen tells me it was successful. I then went to the Azure portal, opened my container and clicked 'Documents' but this does not show me any data. I then clicked the refresh button in the sub-pane (the one on the 'load more' section) and it gave me the error:
Error while fetching page of documents:
{"code":400,"body":"Command find failed: Unknown server error occurred when processing this request.."}
I also could not find any good tutorials online or on Youtube that shoes step by step how to import a CSV from BLOB storage into ComosDB Document store via DataFactory, so unable to tell if I am doing it correctly.
I contacted Microsoft. Response was: "The Azure Data Factory loads data using the SQL API SDK, and does not support mMngo yet. The data loaded using the SQL API SDK would have to be in MongoDB BSON schema. Also, the Mongo DB Native driver expect the data in JSON schema and fails to deserialize triggerin the 400 error."
The MongoBulkExecutor API was recommended as an alternative but from what I can tell this really requires json too.
I also could not find any good tutorials online or on Youtube that
shoes step by step how to import a CSV from BLOB storage into ComosDB
Document store via DataFactory, so unable to tell if I am doing it
correctly.
In fact, you should check the below components when you import csv from blob storage into cosmos db.
1.You already have created cosmos db Linked Service and DataSet.
2.You already have created cosmos db Linked Service and DataSet.
You could do the above steps on the portal.
3.Create copy activity and fill the blob storage input and cosmos db output into the activity.
4.In addition,you need to know cosmos db sql api and cosmos db mongo api are different api,though they call named cosmos db. Based on Supported capabilities in document: Copy data to or from Azure Cosmos DB by using Azure Data Factory, Azure Cosmos DB connector only supports Copying data from and to the Azure Cosmos DB SQL API. So,please don't confuse.
If you do want to use Mongo api, you could choose mongo connector to do your jobs which is mentioned in this case:https://social.msdn.microsoft.com/Forums/security/en-US/52cddbf7-c132-490c-9088-65a38f9b7200/copy-activity-to-cosmosdb-with-mongo-api?forum=AzureDataFactory.
I created a new AppMaker app and selected the "default" (as opposed to "custom") Cloud SQL database backend.
While I'm prototyping my app, I'd like to be able to inspect the contents of my database periodically as an admin to debug issues. With a custom Cloud SQL database this is easy because you can acess a custom Cloud SQL database from the cloud console, but I don't see how to manually query a default Cloud SQL database.
I know that I can export my database to a Google Sheet, but that's inconvenient to do frequently.
How do I inspect the contents of my AppMaker default Cloud SQL database for debugging (eg. via a SQL command line, UI tool, etc)?
I believe it would be the same as with the custom one. The documentation explains:
A G Suite administrator can set up a Google Cloud SQL instance that is shared among App Maker apps in an organization. When this is enabled, a new database is automatically created for your app when you add at least one Cloud SQL data model. Choose this option if your app needs a database that is easy to use and requires no set up.
This means that you had set up correctly the instance information in the G Suite Admin console:
So to connect to your SQL instance, you just need to follow the instructions here. Then simply use the instance connection name where required. You will also need the database name and you can get that from the appsettings or deployment settings in appmaker.
For the preview mode it will be in the app settings. For any deployed version, it will be in the deployment settings:
Can I use a SQL Server Express database as my local database for an occasionally connected application (OCA) written using Visual Studio? Would that require SQL Server to be installed on the client machine? It looks like the default architecture for OCAs in .NET is to use SQL Server Compact. However, SQL Server Compact doesn't permit the use of stored procedures. I use stored procedures for all the data access in my application so, I am confused about the best way to create an occasionally connected client to extend the functionality.
I currently have an ASP.NET web application that connects to a web service (WCF). The web service connects to the DB and calls stored procedures to get data and submit changes to data. Now, I am trying to write a desktop application that can connect to the web service when a connection is available, and work locally when a connection is not available, using the MS Sync Framework. I don't quite understand how to do the architecture for this bit.
Yes, local data cache works with SQL CE 3.5 and you cannot use stored procedures on the cache. Once you add local data cache item to your project it automatically prepares all necessary MS Sync Framework code for data synchronization with the main data source + all necessary SQL scripts for local database and it will also offer you to create either typed datasets or entity data model to access the cache from your application.
Item doesn't work with SQL Server Express - it doesn't offer any other data provider then SQL Compact 3.5. Anyway if you want to use SQL Server Express you will have to either install it on the client machine or use another machine as DB server which breaks whole purpose of Local data cache.
Btw. I think Local data cache works only against database as the main data source so you cannot use it if you want to have WCF services as data source and you will have to write the store and synchronization yourselves.