I have a project that uses CloudTableClient and query the cosmos db in this way:
var table = cloudTableClient.GetTableReference(tableName);
var cosmosResult = await table.ExecuteQuerySegmentedAsync(GetTableQuery<DynamicTableEntity>(queryOptions), tableContinuationToken, GetTableRequestOptions(requestOptions), operationContext);
If I use CosmosClient, I can set the ConnetionMode
CosmosClient client = new CosmosClient(connectionString,
new CosmosClientOptions
{
ConnectionMode = ConnectionMode.Gateway // ConnectionMode.Direct is the default
});
However, with CouldTableClient, seems I can't find an option to set this. Is it possible to use Direct Mode with CouldTableClient or I actually need to move everything to CosmosClient in order to do it.
Related
I manage to ingest data successfully using below code
var kcsbDM = new KustoConnectionStringBuilder(
"https://test123.southeastasia.kusto.windows.net",
"testdb")
.WithAadApplicationTokenAuthentication(acquireTokenTask.AccessToken);
using (var ingestClient = KustoIngestFactory.CreateDirectIngestClient(kcsbDM))
{
var ingestProps = new KustoQueuedIngestionProperties("testdb", "TraceLog");
ingestProps.ReportLevel = IngestionReportLevel.FailuresOnly;
ingestProps.ReportMethod = IngestionReportMethod.Queue;
ingestProps.Format = DataSourceFormat.json;
//generate datastream and columnmapping
ingestProps.IngestionMapping = new IngestionMapping() {
IngestionMappings = columnMappings };
var ingestionResult = ingestClient.IngestFromStream(memStream, ingestProps);
}
when I try to use QueuedClient and IngestFromStreamAsync, the code is executed successfully but no any data is ingested into database even after 30 minutes
var kcsbDM = new KustoConnectionStringBuilder(
"https://ingest-test123.southeastasia.kusto.windows.net",
"testdb")
.WithAadApplicationTokenAuthentication(acquireTokenTask.AccessToken);
using (var ingestClient = KustoIngestFactory.CreateQueuedIngestClient(kcsbDM))
{
var ingestProps = new KustoQueuedIngestionProperties("testdb", "TraceLog");
ingestProps.ReportLevel = IngestionReportLevel.FailuresOnly;
ingestProps.ReportMethod = IngestionReportMethod.Queue;
ingestProps.Format = DataSourceFormat.json;
//generate datastream and columnmapping
ingestProps.IngestionMapping = new IngestionMapping() {
IngestionMappings = columnMappings };
var ingestionResult = ingestClient.IngestFromStreamAsync(memStream, ingestProps);
}
Try running .show ingestion failures on "https://test123.southeastasia.kusto.windows.net" endpoint, see if there are ingestion error.
Also, you set Queue reporting method, you can get the detailed result by reading from the queue.
ingestProps.ReportLevel = IngestionReportLevel.FailuresOnly;
ingestProps.ReportMethod = IngestionReportMethod.Queue;
(On the first example you used KustoQueuedIngestionProperties, you should use KustoIngestionProperties. KustoQueuedIngestionProperties has additional properties that will be ignored by the ingest client, ReportLevel and ReportMethod for example)
Could you please change the line to:
var ingestionResult = await ingestClient.IngestFromStreamAsync(memStream, ingestProps);
Also please note that queued ingestion has a batching stage of up to 5 minutes before the data is actually ingested:
IngestionBatching policy
.show table ingestion batching policy
I find the reason finally, need to enable stream ingestion in the table:
.alter table TraceLog policy streamingingestion enable
See the Azure documentation for details.
enable streamingestion policy is actually only needed if
stream ingestion is turned on in the cluster (azure portal)
the code is using CreateManagedStreamingIngestClient
the ManagedStreamingIngestClient will first try stream ingesting the data, if it fails a few times, then it will use the QueuedClient
if the ingesting data is smaller, under 4MB, it's recommended to use this client.
if using QueuedClient, you can try
.show commands-and-queries | | where StartedOn > ago(20m) and Text contains "{YourTableName}" and CommandType =="DataIngestPull"
This can give you the command executed; however it could have latency > 5 mins
Finally, you can check the status with any client you use, do this
StreamDescription description = new StreamDescription
{
SourceId = Guid.NewGuid(),
Stream = dataStream
};
then you have the source id
ingesting by calling this:
var checker = await client.IngestFromStreamAsync(description, ingestProps);
after that, call
var statusCheck = checker.GetIngestionStatusBySourceId(description.sourceId.Value);
You can figure out the status of this ingestion job. It's better wrapped in a separate thread, so you can keep checking once a few seconds, for example.
I am trying to get metadata information of my Cosmos Graph Database. There are a number of Graphs created in this database and I want to list those Graph names.
In the Gremlin API, we have support to connect to any Graph DB container and then we can submit the query as I mentioned in the below code sample. But here we need a {collection} that is our GraphName as well. So somehow we are bound to a particular graph here.
var gremlinServer = new GremlinServer(hostname, port, enableSsl: true,
username: "/dbs/" + database + "/colls/" + collection,
password: authKey);
using (var gremlinClient = new GremlinClient(gremlinServer, new GraphSON2Reader(), new GraphSON2Writer(), GremlinClient.GraphSON2MimeType))
{
gremlinClient.SubmitAsync(query);
}
Is there any way so that we can connect to GraphDB only and get some metadata information ? Such as, in my case, list of available Graphs.
It looks like the Gremlin Client is implemented at a Collection level (i.e. graph) so it won't be possible to enumerate Graphs from one account / database using the gremlin connection.
You can always use the CosmosDB SDK to connect to the account and enumerate the databases/collections and then use the Gremlin Clients to connect to each of them separately.
Install-Package Microsoft.Azure.Cosmos
using (var client = new CosmosClient(endpoint, authKey))
{
var dbIterator = client.GetDatabaseQueryIterator<DatabaseProperties>();
while(dbIterator.HasMoreResults)
{
foreach (var database in await dbIterator.ReadNextAsync())
{
var containerIterator = database.GetContainerQueryIterator<ContainerProperties>();
while (containerIterator.HasMoreResults)
{
foreach (var container in await containerIterator.ReadNextAsync())
{
Console.WriteLine($"{database.Id} - {container.Id}");
}
}
}
}
}
I have an error when I'm trying to use BulkExecutor to update one of the properties in CosmosDb. The error message is "Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index"
Important point- I don't have partition key defined on my collection.
Here is my code:
SetUpdateOperation<string> player1NameUpdateOperation = new SetUpdateOperation<string>("Player1Name", name);
var updateOperations = new List<UpdateOperation>();
updateOperations.Add(player1NameUpdateOperation);
var updateItems = new List<UpdateItem>();
foreach (var match in list)
{
string id = match.id;
updateItems.Add(new UpdateItem(id, null, updateOperations));
}
var executor = new Microsoft.Azure.CosmosDB.BulkExecutor.BulkExecutor(_client, _collection);
await executor.InitializeAsync();
var executeResult = await executor.BulkUpdateAsync(updateItems);
var count = executeResult.NumberOfDocumentsUpdated;
What am I missing?
If I run the bulk executor on a collection without a partition key, I get the same error. If I run it with a collection that does have it and i specify it, the bulk executor works fine.
Pretty sure they just don't support it right now through the bulk executor api, just use the normal cosmos api for updating the doc as a workaround for now.
I have created two app maker apps for different purposes. However, there are some data which is common between two apps.
How can I access/connect data between those app maker apps?
You will need to setup a Custom Google Cloud SQL instance. Then you can point both apps at the instance in Settings > Database > Switch to a Custom Cloud SQL Database
EDIT:
The other option is setting up a calculated reference to a model in another app.
Grant access to App2 in App1's Google Cloud SQL Instance Authorization Settings (see above link).
Create a Calculated model in App2.
In the Datasource query:
var conn = Jdbc.getCloudSqlConnection('jdbc:google:mysql://INSTANCE_CONNECTION_NAME/DATABASE_NAME', 'USERNAME', 'PASSWORD');
var stmt = conn.prepareStatement("SELECT * from TABLE_NAME");
var res = stmt.executeQuery();
var records = [];
while(res.next()) {
var record = app.models.MODEL_NAME.newRecord();
record.FIELD_1 = res.getString(1);
record.FIELD_2 = res.getString(2);
record.FIELD_3 = res.getString(3);
records.push(record);
}
res.close();
stmt.close();
conn.close();
return records;
We use the following code to communicate with DynamoDB:
using (var client = AWSClientFactory.CreateAmazonDynamoDBClient(RegionEndpoint.USEast1))
{
var table = Table.LoadTable(client, "Users");
var item = await table.GetItemAsync(id);
}
How can I force AWS.SDK for .NET to always use SSL connections? This is very important as we want to store user data including passwords in DynamoDB.
Here is a code snippet that will allow you to set the endpoint explicitly:
AmazonDynamoDBConfig config = new AmazonDynamoDBConfig();
config.ServiceURL = "https://dynamodb.us-east-1.amazonaws.com";
var client = new AmazonDynamoDBClient(config);
Feel free to replace the region (us-east-1) with any other AWS region.
This example is based on the documentation available here on setting the endpoint: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TestingDotNetApiSamples.html