I have created Azure Durable Functions to read a large volume of data from Azure CosmosDB. It consists of performing some Actions and storing the result back in a new Container of CosmosDB. I'm following Fan-Out/Fan-In Approach as shown below.
HTTP request (HttpTrigger Function) will be triggered by the client
via the Function URL and In turn, it calls an Orchestration
(OrchestrationTrigger Function)
The Orchestration calls an Activity Function (GetStoresToBeProcessed) to get all the Distinct Stores to be Processed.
The Orchestration makes multiple parallel function calls to Activity Function(DirectToStoreWac) to process all the transactional data of each Store
Waits for all Calculations to complete
This approach is working fine. I need suggestions for the below points:
Activity Function (GetStoresToBeProcessed) returns 3000+ stores back to Orchestration Function and from there 3000+ tasks will be created and awaited. Is there any way I can batch 3000+ stores? Instead of sending 3000+ tasks in one go here.
// Get all the Stores to be Processed.
int[] stores = await context.CallActivityAsync<int[ (FuncConstants.GetStoresToBeProcessed, null);
var parallelTasks = new List<Task<int>>();
foreach (var store in stores)
{
Task<int> task = context.CallActivityAsync<int>(FuncConstants.DirectToStoreWac, store);
parallelTasks.Add(task);
}
//wait for all results to come back
await Task.WhenAll(parallelTasks);
Each Activity Function (GetStoresToBeProcessed) will operate on Single Store's data (approximately 80-90K records) and internally the _calculationService.Calculate(storeId) functioncalls the Cosmos Bulk Feature await _wacDirectToStoreCosmosRepository.AddBulkAsync(list)
Here, is it correct approach to call Task.Await(cosmos bulk tasks of a store) inside Task.Await(3000 stores call)
[FunctionName(FuncConstants.DirectToStoreWac)]
public async Task<int> RunAsync([ActivityTrigger] int storeId, ILogger log)
{
log.LogInformation($"The Store ({storeId}) has been Queued for Processing at {DateTime.UtcNow}");
int totalRecordsProcessed = 0;
await _calculationService.Calculate(storeId).ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
totalRecordsProcessed += t.Result;
}
else
{
log.LogError(t.Exception, $"[{FuncConstants.DirectToStoreWac}] - Error occured while processing Function");
}
});
return totalRecordsProcessed;
}
I want to use sqflite in my app. To do this, I'm trying to follow this tutorial: https://flutter.dev/docs/cookbook/persistence/sqlite. However, I don't know where to place the code inside my application. In the tutorial, the code seems to be placed at the main() function - but, if I do that, how can I call the insert, update and delete methods at other files?
Update:
As suggested by #Madhavam Shahi, I created a file databaseServices.dart. Now, at the other file, I'm importing databaseServices.dart and trying to use it as below:
import 'databaseServices.dart';
DataBaseServices db=DataBaseServices();
db.delete() //example
However, it is not working. I think the databaseServices.dart is not structured the right way, but I can't spot the error. I know I must be making a very newbie mistake. Here is the code for the databaseServices.dart:
import 'dart:async';
import 'package:path/path.dart';
import 'package:sqflite/sqflite.dart';
import 'counter.dart';
class DatabaseServices {
void whatever() async {
// Open the database and store the reference.
final Future<Database> database = openDatabase(
// Set the path to the database.
join(await getDatabasesPath(), 'counter_database.db'),
// When the database is first created, create a table to store counters;
onCreate: (db, version) {
// Run the CREATE TABLE statement on the database.
return db.execute(
"CREATE TABLE counters(id INTEGER PRIMARY KEY, name TEXT, value INTEGER)",
);
},
// Set the version. This executes the onCreate function and provides a
// path to perform database upgrades and downgrades.
version: 1,
);
// Define a function that inserts counters into the database.
Future<void> insertCounter(Counter counter) async {
// Get a reference to the database.
final Database db = await database;
// Insert the Counter into the correct table. Here, if a counter is inserted twice,
// it replace any previous data.
await db.insert(
'counters',
counter.toMap(),
conflictAlgorithm: ConflictAlgorithm.replace,
);
}
// A method that retrieves all the counters from the counters table.
Future<List<Counter>> counters() async {
// Get a reference to the database.
final Database db = await database;
// Query the table for all the Counters.
final List<Map<String, dynamic>> maps = await db.query('counters');
// Counvert the List<Map<String, dynamic>> into a List<Counter>
return List.generate(maps.length, (i) {
return Counter(
id: maps[i]['id'],
name: maps[i]['name'],
value: maps[i]['value'],
);
});
}
// Method to update a Counter in the database
Future<void> updateCounter(Counter counter) async {
final db = await database;
await db.update(
'counters',
counter.toMap(),
where: "id = ?",
whereArgs: [counter.id],
);
}
//Delete a Counter from the database
Future<void> deleteCounter(int id) async {
final db = await database;
await db.delete(
'counters',
where: "id = ?",
whereArgs: [id],
);
}
}
}
No, it doesn't matter wherever you create the database.
You can create a file databaseServices.dart which manages the database service. It'll be easy for you to manage your code then.
In the cookbook, they are just showing an example, how you can use sqlflite.
But, however you should place this line WidgetsFlutterBinding.ensureInitialized(); in your main() method before anything, IF you are doing an asynchronous task in the main() method.
Update:
To perform CRUD in other files,
import your databaseServices.dart file that file, in which you want to perform the CRUD.
import 'databaseServices.dart';
DataBaseServices db=DataBaseServices();// create an object (DataBaseServices is the name of the class)
//Now, you can access all the methods,
db.delete()//example
Alternatively, if you don't want to create a class in the databaseServices.dart file, and want to keep every function a top level function, then you can do the following.
import 'databaseServices.dart' as db;
//Now, you can access all the top level functions or variables.
db.delete()//example.
Update 2:-
To make database accessible to every function,
move Future database outside the whatever () method, and place it just below the class name. (Making it global so that every function can access it), notice i removed the "final" keyword, because we are going to initialise it later, in the whatever method. Now, in the whatever method do what you were, but instead of final Future database = //yoir code, do this, database =//your code.. by doing this, you will be initialising the database variable, and as database variable is a global variable(declared outside any function, inside a class), any function can access it. But you have to keep in mind that, database has to be initialised before you call any other method which requires database, because if you will not call whatever () method before any other function, then the database won't be initialised, and hence your other functions won't work.
Example,
import 'dart:async';
import 'package:path/path.dart';
import 'package:sqflite/sqflite.dart';
import 'counter.dart';
class DatabaseServices {
Future<Database> database;//making database global so that every function inside the class can access it.
void whatever() async {
// Open the database and store the reference.
database = openDatabase(
// Set the path to the database.
join(await getDatabasesPath(), 'counter_database.db'),
// When the database is first created, create a table to store counters;
onCreate: (db, version) {
// Run the CREATE TABLE statement on the database.
return db.execute(
"CREATE TABLE counters(id INTEGER PRIMARY KEY, name TEXT, value INTEGER)",
);
},
// Set the version. This executes the onCreate function and provides a
// path to perform database upgrades and downgrades.
version: 1,
);
}//Function whatever () ends here
// Define a function that inserts counters into the database.
Future<void> insertCounter(Counter counter) async {
// Get a reference to the database.
final Database db = await database;
// Insert the Counter into the correct table. Here, if a counter is inserted twice,
// it replace any previous data.
await db.insert(
'counters',
counter.toMap(),
conflictAlgorithm: ConflictAlgorithm.replace,
);
}
// A method that retrieves all the counters from the counters table.
Future<List<Counter>> counters() async {
// Get a reference to the database.
final Database db = await database;
// Query the table for all the Counters.
final List<Map<String, dynamic>> maps = await db.query('counters');
// Counvert the List<Map<String, dynamic>> into a List<Counter>
return List.generate(maps.length, (i) {
return Counter(
id: maps[i]['id'],
name: maps[i]['name'],
value: maps[i]['value'],
);
});
}
// Method to update a Counter in the database
Future<void> updateCounter(Counter counter) async {
final db = await database;
await db.update(
'counters',
counter.toMap(),
where: "id = ?",
whereArgs: [counter.id],
);
}
//Delete a Counter from the database
Future<void> deleteCounter(int id) async {
final db = await database;
await db.delete(
'counters',
where: "id = ?",
whereArgs: [id],
);
}
}
Now, since there are no nested functions, you can easily create an object of the class, and call the functions as you need easily :)
When I create a database I want to initialize it with a ton of data.
I have the following initialization service.
// This needs to be a top-level method because it's run on a background isolate
DatabaseConnection _backgroundConnection() {
// construct the database. You can also wrap the VmDatabase in a "LazyDatabase" if you need to run
// work before the database opens.
final database = VmDatabase.memory();
return DatabaseConnection.fromExecutor(database);
}
Future<void> _initDatabase(Map<String, dynamic> args) async {
var moorIsolate = await MoorIsolate.spawn(_backgroundConnection);
var connection = await moorIsolate.connect();
var db = BillingDatabase.connect(connection);
_initBillingSpecialties(db, args["specialties"]);
}
Future<void> _initBillingSpecialties(BillingDatabase db, String specialtiesJson) async {
var json = jsonDecode(specialtiesJson);
var jsonSpecialties = json["specialties"] as List<dynamic>;
var specialities = jsonSpecialties.map((s) =>
DbSpecialtiesCompanion(name: Value(s["specialty_name"]),
mohNumber: Value(s["moh_specialty"]))).toList();
return db.specialtyDao.saveAllSpecialties(specialities);
}
#injectable
class InitDbService {
Future<void> initDatabase() async {
WidgetsFlutterBinding.ensureInitialized();
var specialties = await rootBundle.loadString("lib/assets/billing_specialties.json");
compute(_initDatabase, {"specialties": specialties});
//initDbSync(specialties);
}
Future<void> initDbSync(String specialtiesJson) async {
var json = jsonDecode(specialtiesJson);
var jsonSpecialties = json["specialties"] as List<dynamic>;
var specialities = jsonSpecialties.map((s) =>
DbSpecialtiesCompanion(name: Value(s["specialty_name"]),
mohNumber: Value(s["moh_specialty"]))).toList();
var dao = GetIt.instance.get<SpecialtyDao>();
return dao.saveAllSpecialties(specialities);
}
}
initDbSync runs and inserts just fine. While db.specialtyDao.saveAllSpecialties(specialities); never actually exectues any SQL. I have it printing log statements for the moment so I can see what it's doing.
Update: I found out that VmDatabase.memory(logStatements: true); was needed to see the SQL. I can see it printing the statements.
I'm running on a simulator so I can look at the raw db file. And there's nothing there. When I query in the app there's also nothing there.
So what's not really clear in the documentation is that VmDatabase.memory(); opens up a new database in memory. Not takes the database from memory.
You want to take your reference to the file that you pass in the constructor, and use
VmDatabase(File(dbFile));
then it will actually run on your sql.
I was trying to simulate a situation where two users (on seperate devices) both run a Transaction at the same time. To imitate this, I made a List<String> of strings which would be added to the database without a delay between them.
However, only the first item in the List was added to the database, the second never arrived. What am I doing wrong? I am trying to have both items added to the database.
The call to the Transaction happens in the code below, along with the creation of the list:
List<String> items = new List<String>();
items.add("A test String 1");
items.add("A test String 2");
for (String q in questions)
{
database.updateDoc( q );
}
The code I use for updating the data in my database:
void updateDoc( String item ) async
{
var data = new Map<String, dynamic>();
data['item'] = item;
Firestore.instance.runTransaction((Transaction transaction) async {
/// Generate a unique ID
String uniqueID = await _generateUniqueQuestionCode();
/// Update existing list
DocumentReference docRef = Firestore.instance
.collection("questions")
.document("questionList");
List<String> questions;
await transaction.get(docRef)
.then (
(document) {
/// Convert List<dynamic> to List<String>
List<dynamic> existing = document.data['questions'];
questions = existing.cast<String>().toList();
}
);
if ( ! questions.contains(uniqueID) )
{
questions.add( uniqueID );
var newData = new Map<String, dynamic>();
newData['questions'] = questions;
transaction.set(docRef, newData );
}
/// Save question
docRef = Firestore.instance
.collection("questions")
.document( uniqueID );
transaction.set(docRef, data);
});
}
In reality, I have a few fields in the document I'm saving but they would only complicate the code.
I keep track of a list of documents because I need to be able to retreive a random document from the database.
When executing the first code snippet, only the first item in the list will be added to the database and to the list that keeps track of the documents.
No error is thrown in the debug screen.
What am I missing here?
As explained by Doug Stevenson in the comments under my question:
That's not a typical use case for a single app instance. If you're
trying to find out if transactions work, be assured that they do.
They're meant to defend against cases where multiple apps or processes
are making changes to a document, not a single app instance.
And also:
The way the Firestore SDK works is that it keeps a single connection
open and pipelines each request through that connection. When there
are multiple clients, you have multiple connection, and each request
can hit the service at a different time. I'd suspect that what you're
trying to simulate isn't really close to the real thing.
I have a desire to add a property with a default value to a set of documents that I retrieve via a SELECT query if they contain no value.
I was thinking of this in two parts:
SELECT * FROM c article WHERE article.details.locale = 'en-us'
I'd like to find all articles where article.details.x does not exist.
Add the property, article.details.x = true
I was hoping this EXEC command could be supported via the Azure Portal so I don't have to create a migration tool to run this command once but I couldn't find this option in the portal. Is this possible?
You can use Azure Document DB Studio as a front end to creating and executing a stored procedure. It can be found here. It's pretty easy to setup and use.
I've mocked up a stored procedure based on your example:
function updateArticlesDetailsX() {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
var response = getContext().getResponse();
var docCount = 0;
var counter = 0;
tryQueryAndUpdate();
function tryQueryAndUpdate(continuation) {
var query = {
query: "select * from root r where IS_DEFINED(r.details.x) != true"
};
var requestOptions = {
continuation: continuation
};
var isAccepted =
collection
.queryDocuments(collectionLink,
query,
requestOptions,
function queryCallback(err, documents, responseOptions) {
if (err) throw err;
if (documents.length > 0) {
// If at least one document is found, update it.
docCount = documents.length;
for (var i=0; i<docCount; i++){
tryUpdate(documents[i]);
}
response.setBody("Updated " + docCount + " documents");
}
else if (responseOptions.continuation) {
// Else if the query came back empty, but with a continuation token;
// repeat the query w/ the token.
tryQueryAndUpdate(responseOptions.continuation);
} else {
throw new Error("Document not found.");
}
});
if (!isAccepted) {
throw new Error("The stored procedure timed out");
}
}
function tryUpdate(document) {
//Optimistic concurrency control via HTTP ETag.
var requestOptions = { etag: document._etag };
//Update statement goes here:
document.details.x = "some new value";
var isAccepted = collection
.replaceDocument(document._self,
document,
requestOptions,
function replaceCallback(err, updatedDocument, responseOptions) {
if (err) throw err;
counter++;
});
// If we hit execution bounds - throw an exception.
if (!isAccepted) {
throw new Error("The stored procedure timed out");
}
}
}
I got the rough outline for this code from Andrew Liu on GitHub.
This outline should be close to what you need to do.
DocumentDB has no way in a single query to update a bunch of documents. However, the portal does have a Script Explorer that allows you to write and execute a stored procedure against a single collection. Here is an example sproc that combines a query with a replaceDocument command to update some documents that you could use as a starting point for writing your own. The one gotcha to keep in mind is that DocumentDB will not allow sprocs to run longer than 5 seconds (with some buffer). So you may have to run your sproc multiple times and keep track of what you've already done if it can't complete in one 5 second run. The use of IS_DEFINED(collection.field.subfield) != true (thanks #cnaegle) in your query followed up by a document replacement that defines that field (or removes that document) should allow you to run the sproc as many times as necessary.
If you didn't want to write a sproc, the easiest thing to do would be to export the database using the DocumentDB Data Migration tool. Import that into Excel to manipulate or write a script to do the manipulation. Then upload it again using the Data Migration tool.