Truncate Equivalent command for Azure DocumentDB - azure-cosmosdb

I have a collection with over 1000 json documents. I want to now delete all the documents in that collection.
Is there any way of doing it from the portal using a query?
PS: I know I can delete the collection and re-create it or use a c# application to do it.

No, there is no equivalent to the SQL DELETE expression in DocumentDB so there is no way to do it with a query. Here is a stored procedure (CoffeeScript) that does this for me when I want to delete based upon a query.
deleteSomeDocuments = (memo) ->
collection = getContext().getCollection()
unless memo?
memo = {}
if memo.returnDeleted
unless memo.deleted?
memo.deleted = []
else
memo.returnDeleted = false
stillQueuingOperations = true
query = () ->
if stillQueuingOperations
responseOptions =
pageSize: memo.remaining
setBody()
if memo.filterQuery?
memo.stillQueueing = collection.queryDocuments(collection.getSelfLink(), memo.filterQuery, responseOptions, onReadDocuments)
else
memo.stillQueueing = collection.readDocuments(collection.getSelfLink(), responseOptions, onReadDocuments)
onReadDocuments = (err, resources, options) ->
if err
throw err
if resources.length isnt memo.remaining
throw new Error("Expected memo.remaining (#{memo.remaining}) and the number of rows returned (#{resources.length}) to match. They don't.")
memo.stillQueueing = true
while memo.remaining > 0 and memo.stillQueueing
oldDocument = resources[memo.remaining - 1]
documentLink = oldDocument._self
etag = oldDocument._etag
options = {etag} # Sending the etag per best practice, but not handling it if there is conflict.
getContext().getResponse().setBody(memo)
memo.stillQueueing = collection.deleteDocument(documentLink, options)
if memo.stillQueueing
if memo.returnDeleted
memo.deleted.push(oldDocument)
memo.remaining--
setBody = () ->
getContext().getResponse().setBody(memo)
query()
return memo
exports.deleteSomeDocuments = deleteSomeDocuments

Related

Android: Why is Room so slow?

I am working on a simple database procedure in Kotlin using Room, and I can't explain why the process is so slow, mostly on the Android Studio emulator.
The table I am working on is this:
#Entity(tableName = "folders_items_table", indices = arrayOf(Index(value = ["folder_name"]), Index(value = ["item_id"])))
data class FoldersItems(
#PrimaryKey(autoGenerate = true)
var uid: Long = 0L,
#ColumnInfo(name = "folder_name")
var folder_name: String = "",
#ColumnInfo(name = "item_id")
var item_id: String = ""
)
And what I am just trying to do is this: checking if a combination folder/item is already present, insert a new record. If not, ignore it. on the emulator, it takes up to 7-8 seconds to insert 100 records. On a real device, it is much faster, but still, it takes around 3-4 seconds which is not acceptable for just 100 records. It looks like the "insert" query is particularly slow.
Here is the procedure that makes what I have just described (inside a coroutine):
val vsmFoldersItems = FoldersItems()
items.forEach{
val itmCk = database.checkFolderItem(item.folder_name, it)
if (itmCk == 0L) {
val newFolderItemHere = vsmFoldersItems.copy(
folder_name = item.folder_name,
item_id = it
)
database.insertFolderItems(newFolderItemHere)
}
}
the variable "items" is an array of Strings.
Here is the DAO definitions of the above-called functions:
#Query("SELECT uid FROM folders_items_table WHERE folder_name = :folder AND item_id = :item")
fun checkFolderItem(folder: String, item: String): Long
#Insert
suspend fun insertFolderItems(item: FoldersItems)
Placing the loop inside a single transaction should significantly reduce the time taken.
The reason is that each transaction (by default each SQL statement that makes a change to the database) will result in a disk write. So that's 100 disk writes for your loop.
If you begin a transaction before the loop and then set the transaction successful when the loop is completed and then end the transaction a single disk write is required.
What I am unsure of is exactly how to do this when using a suspended function (not that familiar with Kotlin).
As such I'd suggest either dropping the suspend or having another Dao for use within loops.
Then have something like :-
val vsmFoldersItems = FoldersItems()
your_RoomDatabase.beginTransaction()
items.forEach{
val itmCk = database.checkFolderItem(item.folder_name, it)
if (itmCk == 0L) {
val newFolderItemHere = vsmFoldersItems.copy(
folder_name = item.folder_name,
item_id = it
)
database.insertFolderItems(newFolderItemHere)
}
}
your_RoomDatabase.setTransactionSuccessful() //<<<<<<< IF NOT set then ALL updates will be rolled back
your_RoomDatabase.endTransaction()
You may wish to refer to:-
https://developer.android.com/reference/androidx/room/RoomDatabase
You may wish to especially refer to runInTransaction

Limit with getting info about audience from Analytics API

I'm trying to get audienc name aduience id etc we' ve created on our google analytics account. We have around 2,4k audiences list but I can just get 999 of them. I can't find any soultions. Code is below
function main() {
var spreadsheet = SpreadsheetApp.openByUrl('https://docs.google.com/spreadshe');
var sheet = spreadsheet.getSheetByName('Sh');
function listRemarketingAudiences(accountId, propertyId) {
var request = Analytics.Management.RemarketingAudience.list(
accountId,
propertyId
);
var leno = Object.keys(request).length
console.log(leno);
sheet.getRange(1,1).setValue("audianceName");
sheet.getRange(1,2).setValue("audianceId");
sheet.getRange(1,3).setValue("audianceDefinition");
sheet.getRange(1,4).setValue("audianceDescription");
for ( var i = 2; i <3000; i++) {
var audianceName = request.items[i+154].name ;
Logger.log(audianceName);
console.log(i);
sheet.getRange(i,1).setValue("elo")
var audianceId = request.items[i].id ;
sheet.getRange(i,2).setValue(audianceId);
// var audianceId = request.items[i].
var audienceDefinition = request.items[i].audienceDefinition ;
sheet.getRange(i,3).setValue(audienceDefinition);
var audienceDescription = request.items[i].description ;
sheet.getRange(i,4).setValue(audienceDescription);
};
}
listRemarketingAudiences('xxxxx', 'UA-xxxxx-1');
}
Currently you are supplying only the required parameters: accountId and webPropertyId. These are necessary to identify the Analytics property, where you are looking for the data.
Based on the documentation, optional parameters can be passed, which are actually in connection with the pagination, which you are trying to achieve.
As the developer guide is not mentioning the absolute limit of the result, you could experiment with higher limits, with a code something like this:
request = gapi.client.analytics.management.remarketingAudience.list(
{
'accountId': accountId,
'webPropertyId': propertyId,
'max-results': 5000
}
If you can't get all the data at once, you need to implement paging yourself, where an other paramerer, start-index will be necessary. You need to call the function several times, preferably from a loop, where start index is continuously increased.
request = gapi.client.analytics.management.remarketingAudience.list(
{
'accountId': accountId,
'webPropertyId': propertyId,
'start-index': 999,
'max-results': 1000
}
I wrote sth like this:
var optional = {'startIndex': 12,
'maxresults': 212};
function listRemarketingAudiences (accountId, propertyId, optional){
var request = Analytics.Management.RemarketingAudience.list(
accountId,
propertyId,
optional.maxresults
);
and an error occure:
We're sorry, a server error occurred. Please wait a bit and try again. (line 9, file "Code")

Titan db no documents getting created on elastic search even index is configuration options enabled

Here is the configuration options i am using.
storage.backend=cassandra
storage.hostname=192.168.56.121
storage.cassandra.keyspace=graphs
cache.db-cache = false
cache.db-cache-clean-wait = 20
index.search.backend=elasticsearch
index.search.hostname=192.168.56.122
index.search.elasticsearch.client-only=true
index.search.index-name=graphs
TitanGraph graph = GraphFactory.getInstance().getGraph();
TitanManagement mgmt = null;
try {
mgmt = graph.openManagement();
PropertyKey name = mgmt.getPropertyKey(Schema.NAME);
if (name == null) {
name = mgmt.makePropertyKey(Schema.NAME).dataType(String.class).make();
}
TitanGraphIndex graphIndex = mgmt.getGraphIndex("byName");
if (graphIndex == null) {
IndexBuilder builder = mgmt.buildIndex("byName", Vertex.class).addKey(name);
builder.buildCompositeIndex();
}
PropertyKey id = mgmt.getPropertyKey(Schema.ID);
if (id == null) {
id = mgmt.makePropertyKey(Schema.ID).dataType(Long.class).make();
}
PropertyKey sourceType = mgmt.getPropertyKey(Schema.SOURCE_TYPE);
if (sourceType == null) {
sourceType = mgmt.makePropertyKey(Schema.SOURCE_TYPE).dataType(String.class).make();
}
TitanGraphIndex uniqueIndex = mgmt.getGraphIndex("uniqueIndex");
if (uniqueIndex == null) {
IndexBuilder builder = mgmt.buildIndex("uniqueIndex", Vertex.class).addKey(id).addKey(sourceType);
builder.unique().buildCompositeIndex();
}
// Edges
EdgeLabel deps = mgmt.getEdgeLabel("deps");
if (deps == null) {
deps = mgmt.makeEdgeLabel("deps").multiplicity(Multiplicity.SIMPLE).make();
}
RelationTypeIndex depsIndex = mgmt.getRelationIndex(deps, "depsIndex");
if(depsIndex == null) {
depsIndex = mgmt.buildEdgeIndex(deps, "depsIndex", Direction.BOTH, Order.decr);
}
mgmt.commit();
// Re index the existing data
if (reIndexData) {
mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("uniqueIndex"), SchemaAction.REINDEX).get();
mgmt.updateIndex(mgmt.getGraphIndex("byName"), SchemaAction.REINDEX).get();
deps = mgmt.getEdgeLabel("deps");
mgmt.updateIndex(mgmt.getRelationIndex(deps,"depsIndex"), SchemaAction.REINDEX).get();
mgmt.commit();
}
} catch (Throwable e) {
log.error(e.getMessage(), e);
if (mgmt != null) {
mgmt.rollback();
}
}
I have created lots of documents and every thing is working fine. But when i observed the number document available in the elastic search is 0.
I am wondered whether titan db really using the elastic search or not.
Any idea what i am missing here ? And why documents are not getting created in elastic search.
And i also tried the belown configuration as well but no luck
storage.backend=cassandra
storage.hostname=192.168.56.121
storage.cassandra.keyspace=graphs
cache.db-cache = false
cache.db-cache-clean-wait = 20
index.graphs.backend=elasticsearch
index.graphs.hostname=192.168.56.122
index.graphs.elasticsearch.client-only=true
index.graphs.index-name=graphs
Titan uses storage backend (cassandra/hbase) for Composite Index and index backend (Solr/Elastic Search) for Mixed Index
Mixed indexes retrieve vertices or edges by any combination of previously added property keys. Mixed indexes provide more flexibility than composite indexes and support additional condition predicates beyond equality. On the other hand, mixed indexes are slower for most equality queries than composite indexes.
Unlike composite indexes, mixed indexes require the configuration of an indexing backend and use that indexing backend to execute lookup operations. Titan can support multiple indexing backends in a single installation. Each indexing backend must be uniquely identified by name in the Titan configuration which is called the indexing backend name.
In you schema you are creating only composite index. That's why there is not data in ElasticSearch.
Here is a example how to create a mixed index :
IndexBuilder builder = mgmt.buildIndex('byName', Vertex.class).addKey(name);
builder.buildMixedIndex("search");
mgmt.commit();
Read More
Source : http://s3.thinkaurelius.com/docs/titan/1.0.0/indexes.html

Initial sync and duplicate objects with same primary key inside Realm Mobile Platform

I'm trying to turn on realm sync on device which already contains some data, that already persists on server. When new user connects to realm, it should merge local realm data with synced realm data. But this code is launched before initial sync happens. Since there is no data from server is received yet, app creates some records in synchronized realm. When sync finishes I see same data twice. Records I've just created and data fetched from server. With same primary key.
See code below for an example:
RLMRealmConfiguration *config = [[RLMRealmConfiguration alloc] init];
config.syncConfiguration = [[RLMSyncConfiguration alloc] initWithUser:user realmURL:self.realmURL];
NSError *error = nil;
RLMRealm *newRealm = [RLMRealm realmWithConfiguration:config error:&error];
if(newRealm != nil && error == nil)
{
[newRealm beginWriteTransaction];
for(ModelFolder *folder in [ModelFolder allObjectsInRealm:curRealm])
{
ModelFolder *newFolder = [ModelFolder objectInRealm:newRealm forPrimaryKey:folder.uuid];
if(newFolder == nil)
[ModelFolder createInRealm:newRealm withValue:folder];
}
[newRealm commitWriteTransaction];
}
Is there a way to detect, that realm is completed initial sync?
UPD: Few more details.
ModelFolder contains #property RLMArray<ModelBookmark *><ModelBookmark> *bookmarks; And when I create Folder, that equals some folder that will be fetched in a few seconds they merged correctly. But. Bookmarks inside Folder object is not deduplicated and we get something like this:
RLMResults <0x802082d0> (
[0] ModelFolder {
uuid = 2615AB34-1C08-4E7B-8D49-6E02EDBCDF89;
name = (null);
descr = (null);
shareURL = (null);
date = 1484566331137;
bookmarks = RLMArray <0x806c78d0> (
[0] ModelBookmark {
uuid = C752FCEB-65CB-47C8-8CF4-6CA44C119ECC;
name = (null);
descr = (null);
shareURL = (null);
date = 1484566331137;
folderUuid = 2615AB34-1C08-4E7B-8D49-6E02EDBCDF89;
longitude = 27.54834598813616;
latitude = 53.91333128839566;
mapZoom = 11.73785983313041;
category = 0;
visible = 1;
},
[1] ModelBookmark {
uuid = C752FCEB-65CB-47C8-8CF4-6CA44C119ECC;
name = (null);
descr = (null);
shareURL = (null);
date = 1484566331137;
folderUuid = 2615AB34-1C08-4E7B-8D49-6E02EDBCDF89;
longitude = 27.54834598813616;
latitude = 53.91333128839566;
mapZoom = 11.73785983313041;
category = 0;
visible = 1;
}
);
tracks = RLMArray <0x806fb120> (
);
opened = 1;
}
)
Unfortunately merging of the ordered lists is not supported currently (until https://github.com/realm/realm-core/issues/1206 is implemented). For now you have to manually deduplicate list items, you can use the same workaround we use in RealmTasks app, see https://github.com/realm/RealmTasks/pull/180 for implementation details.

DynamoDB Xcode6 Swift using three columns as key

I am trying to use a DynamoDB table to store this data:
DartsPlayerInsultTable
CustomerId String
PlayerId String
PlayerInsult String
Using the method (concept, not code) described here:
https://java.awsblog.com/post/Tx3GYZEVGO924K4/The-DynamoDBMapper-Local-Secondary-Indexes-and-You
here:
http://mobile.awsblog.com/post/TxTCW7KW8BGZAF/Amazon-DynamoDB-on-Mobile-Part-4-Local-Secondary-Indexes
and here:
http://labs.journwe.com/2013/12/15/dynamodb-secondary-indexes/comment-page-1/#comment-116
I want to have multiple insult records per customer-player.
CustomerId is my Hash Key
PlayerId is my Range Key
and I a trying to use PlayerInsult in a key so that
a second PlayerInsult value inserts a second record
rather than replacing the existing one.
Have tried both Global and Secondary indexes for this,
but if I try to add a row with a new insult, it still
replaces the insult with the same customer-player key
rather than adding a new one.
Any suggestions on the best approach to use for this is
DynanoDB? Do I need to create a hybrid column for a range-key?
Trying to keep this simple...
class func createDartsPlayerInsultTable() -> BFTask {
let dynamoDB = AWSDynamoDB.defaultDynamoDB()
let hashKeyAttributeDefinition = AWSDynamoDBAttributeDefinition()
hashKeyAttributeDefinition.attributeName = "CustomerId"
hashKeyAttributeDefinition.attributeType = AWSDynamoDBScalarAttributeType.S
let hashKeySchemaElement = AWSDynamoDBKeySchemaElement()
hashKeySchemaElement.attributeName = "CustomerId"
hashKeySchemaElement.keyType = AWSDynamoDBKeyType.Hash
let rangeKeyAttributeDefinition = AWSDynamoDBAttributeDefinition()
rangeKeyAttributeDefinition.attributeName = "PlayerId"
rangeKeyAttributeDefinition.attributeType = AWSDynamoDBScalarAttributeType.S
let rangeKeySchemaElement = AWSDynamoDBKeySchemaElement()
rangeKeySchemaElement.attributeName = "PlayerId"
rangeKeySchemaElement.keyType = AWSDynamoDBKeyType.Range
/*
let indexRangeKeyAttributeDefinition = AWSDynamoDBAttributeDefinition()
indexRangeKeyAttributeDefinition.attributeName = "PlayerInsult"
indexRangeKeyAttributeDefinition.attributeType = AWSDynamoDBScalarAttributeType.S
let rangeKeySchemaElement = AWSDynamoDBKeySchemaElement()
rangeKeySchemaElement.attributeName = "PlayerId"
rangeKeySchemaElement.keyType = AWSDynamoDBKeyType.Range
let indexRangeKeyElement = AWSDynamoDBKeySchemaElement()
indexRangeKeyElement.attributeName = "PlayerInsult"
indexRangeKeyElement.keyType = AWSDynamoDBIndexRangeKeyType.
*/
//Add non-key attributes
let playerInsultAttrDef = AWSDynamoDBAttributeDefinition()
playerInsultAttrDef.attributeName = "PlayerInsult"
playerInsultAttrDef.attributeType = AWSDynamoDBScalarAttributeType.S
let provisionedThroughput = AWSDynamoDBProvisionedThroughput()
provisionedThroughput.readCapacityUnits = 5
provisionedThroughput.writeCapacityUnits = 5
// CREATE GLOBAL SECONDARY INDEX
/*
let gsi = AWSDynamoDBGlobalSecondaryIndex()
let gsiArray = NSMutableArray()
let gsiHashKeySchema = AWSDynamoDBKeySchemaElement()
gsiHashKeySchema.attributeName = "PlayerId"
gsiHashKeySchema.keyType = AWSDynamoDBKeyType.Hash
let gsiRangeKeySchema = AWSDynamoDBKeySchemaElement()
gsiRangeKeySchema.attributeName = "PlayerInsult"
gsiRangeKeySchema.keyType = AWSDynamoDBKeyType.Range
let gsiProjection = AWSDynamoDBProjection()
gsiProjection.projectionType = AWSDynamoDBProjectionType.All;
gsi.keySchema = [gsiHashKeySchema,gsiRangeKeySchema];
gsi.indexName = "PlayerInsult";
gsi.projection = gsiProjection;
gsi.provisionedThroughput = provisionedThroughput;
gsiArray .addObject(gsi)
*/
// CREATE LOCAL SECONDARY INDEX
let lsi = AWSDynamoDBLocalSecondaryIndex()
let lsiArray = NSMutableArray()
let lsiHashKeySchema = AWSDynamoDBKeySchemaElement()
lsiHashKeySchema.attributeName = "CustomerId"
lsiHashKeySchema.keyType = AWSDynamoDBKeyType.Hash
let lsiRangeKeySchema = AWSDynamoDBKeySchemaElement()
lsiRangeKeySchema.attributeName = "PlayerInsult"
lsiRangeKeySchema.keyType = AWSDynamoDBKeyType.Range
let lsiProjection = AWSDynamoDBProjection()
lsiProjection.projectionType = AWSDynamoDBProjectionType.All;
lsi.keySchema = [lsiHashKeySchema,lsiRangeKeySchema];
lsi.indexName = "PlayerInsult";
lsi.projection = lsiProjection;
//lsi.provisionedThroughput = provisionedThroughput;
lsiArray .addObject(lsi)
//Create TableInput
let createTableInput = AWSDynamoDBCreateTableInput()
createTableInput.tableName = DartsPlayerInsultTableName;
createTableInput.attributeDefinitions = [hashKeyAttributeDefinition, rangeKeyAttributeDefinition, playerInsultAttrDef]
//createTableInput.attributeDefinitions = [hashKeyAttributeDefinition, rangeKeyAttributeDefinition]
createTableInput.keySchema = [hashKeySchemaElement, rangeKeySchemaElement]
createTableInput.provisionedThroughput = provisionedThroughput
//createTableInput.globalSecondaryIndexes = gsiArray as [AnyObject]
createTableInput.localSecondaryIndexes = lsiArray as [AnyObject]
return dynamoDB.createTable(createTableInput).continueWithSuccessBlock({ (var task:BFTask!) -> AnyObject! in
if ((task.result) != nil) {
// Wait for up to 4 minutes until the table becomes ACTIVE.
let describeTableInput = AWSDynamoDBDescribeTableInput()
describeTableInput.tableName = DartsPlayerInsultTableName;
task = dynamoDB.describeTable(describeTableInput)
for var i = 0; i < 16; i++ {
task = task.continueWithSuccessBlock({ (task:BFTask!) -> AnyObject! in
let describeTableOutput:AWSDynamoDBDescribeTableOutput = task.result as! AWSDynamoDBDescribeTableOutput
let tableStatus = describeTableOutput.table.tableStatus
if tableStatus == AWSDynamoDBTableStatus.Active {
return task
}
sleep(15)
return dynamoDB .describeTable(describeTableInput)
})
}
}
return task
})
}
Putting this as an answer and not another comment in case it gets long...
It sounds like the average user's insults might fit into a single record. With the disclaimer that I know absolutely nothing about swift, this might at least be something relatively simple. Keep your customer and player keys. Before you persist the insults, turn the whole list into one big string using whatever version of join("|") swift has. When you fetch the record, do a split("|") to get your list back. (Just be a little judicious with your choice of separators, I'm only using "|" as an example, you don't want to choose something that might appear in an insult...)
There's going to be that one user with enough insults to take you over the 400kb object limit. Set a max list size constant in your code -- when you turn your lists into strings to persist them to dynamo, check the player's list length against that limit. If you exceed it, break your list into chunks of that size and use hash and range keys like ("foo", "bar"), ("foo", "bar1"), ("foo", "bar2"), etc. Yes, the first one does not have a bucket number at the end...
When you query for the data, just do a straight query first and assume you'll be in the good case (just "foo" and "bar", no other buckets). When you unpack that first list, check its length. If it's equal to your max list size constant, you know that you got a "bad" user and need to do a range query. That second one can use the hash key "foo" and the range "bar" to "bar9999". You will fetch back all those buckets with that range query. Unpack and concatenate all the lists.
This is a little gory, but it should also ultimately be straight ahead to code up. Hopefully it's still simple enough to hook into the patterns you mentioned.
What I decided to do was make a conventional dynamodb table with just one hash key, but the new hash key is a combined string of:
CustomerId + "|" + PlayerId
It is not too hard to maintain synchrony between players and insults tables because once a player is inserted into the player table, modifying the player name results in a new row being inserted. Thus, insults do not need to be modified if the player name changes. You only need to cleanup insults if a player is deleted.
This update behavior is just the way dynamodb works if you make Player name a hash key, which I did to insure they were unique.

Resources