im using BulkWriteOperation in order to insert bulk Documents ,this is my code
val builder: casbah.BulkWriteOperation = collection.initializeOrderedBulkOperation
for(p<-posts){
if (p._1.\("object_id").asOpt[String].getOrElse("") != "" && p._1.\("message").asOpt[String].getOrElse("") !="")
builder.insert(MongoDBObject("id" -> p._1.\("id").asOpt[String], "object_id" -> p._1.\("object_id").asOpt[String], "message" -> p._1.\("message").asOpt[String],"shares"->p._1.\("shares").\("count").asOpt[Long].getOrElse(0),"likes"->p._2.\("summary").\("total_count").asOpt[Long].getOrElse(0)))
}
val result = builder.execute()
Is there any way to create Bulk Upsert ?
thanks,
miki
Yes the find operation returns a BulkWriteRequestBuilder which allows you to replace, remove, update and mark it as an upsert request. So to perform an upsert you can do:
val builder: BulkWriteOperation = collection.initializeOrderedBulkOperation
builder.find(MongoDBObject("_id" -> 1)).upsert().updateOne($set("x" -> 2))
val result = builder.execute()
Related
Update:
add my Cargo.toml in case needed:
serde = { version = "1.0.117", default-features = false }
serde_json = "1.0.66"
sql-builder = "3.1"
sqlite = "0.26.0"
Update:
i did #cdhowie says on comment, add println!("{:?}", row[2].kind());, it's print String.
it's the way i store blob wrong, shouldn't use serde_json::to_string or something?
OP:
i writed a demo to learn sqlite and rust, here the code:
use std::{fs::File, io::{Read, Write}};
use sql_builder::{quote, SqlBuilder};
use sqlite::Connection;
fn main() {
// create sqlite databases on ./tmp/sqlite.db
let conn = Connection::open("./tmp/sqlite.db").unwrap();
// create table
conn.execute(
"CREATE TABLE IF NOT EXISTS icon (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT,
content BLOB,
used STRING
)",
)
.unwrap();
// read image file from disk and store to sqlite as blob
let mut file = File::open("./tmp/in.jpg").unwrap();
let mut contents = Vec::new();
file.read_to_end(&mut contents).unwrap();
println!("{:?}", contents);
// build sql query
let sql = SqlBuilder::insert_into("icon")
.fields(&["name", "content", "used"])
.values(&[
quote(serde_json::to_string("in").unwrap()),
quote(serde_json::to_string(&contents).unwrap()),
quote(serde_json::to_string("1").unwrap()),
])
.sql().unwrap();
// execute query
conn.execute(&sql).unwrap();
// read image file from sqlite and store to disk
let mut builder = SqlBuilder::select_from("icon");
builder.field("id");
builder.field("name");
builder.field("content");
builder.field("used");
let stmt = conn.prepare(&builder.sql().unwrap()).unwrap();
let mut cursor = stmt.into_cursor();
let row = cursor.next().unwrap().unwrap();
let id = row[0].as_integer().unwrap();
let name = row[1].as_string().unwrap();
let content = row[2].as_binary().unwrap(); // src/main.rs:51:38
let used = row[3].as_string().unwrap();
println!("{} {} {}", id, name, used);
let mut file = File::create("./tmp/out.jpg").unwrap();
file.write_all(content).unwrap();
}
when i run this code, will got error:
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/main.rs:51:38
but where i check the sqlite.db file, it shows:
seems file stored success. but how can i fix the code make read form sqlite and store to disk work?
if you guys need more info, please let me know. :)
I can fix the issue now, but this may not be the best solution.
// ...
let content = row[2].as_string().unwrap();
// ...
// ...
let content: Vec<u8> = serde_json::from_str(&content).unwrap();
file.write_all(&content).unwrap();
// ...
also, I may have found a better one SQL generation lib, called diesel.
Maybe as I keep learning, I might find a better, more elegant way. Anyway, it's working now. :)
I am working on a simple database procedure in Kotlin using Room, and I can't explain why the process is so slow, mostly on the Android Studio emulator.
The table I am working on is this:
#Entity(tableName = "folders_items_table", indices = arrayOf(Index(value = ["folder_name"]), Index(value = ["item_id"])))
data class FoldersItems(
#PrimaryKey(autoGenerate = true)
var uid: Long = 0L,
#ColumnInfo(name = "folder_name")
var folder_name: String = "",
#ColumnInfo(name = "item_id")
var item_id: String = ""
)
And what I am just trying to do is this: checking if a combination folder/item is already present, insert a new record. If not, ignore it. on the emulator, it takes up to 7-8 seconds to insert 100 records. On a real device, it is much faster, but still, it takes around 3-4 seconds which is not acceptable for just 100 records. It looks like the "insert" query is particularly slow.
Here is the procedure that makes what I have just described (inside a coroutine):
val vsmFoldersItems = FoldersItems()
items.forEach{
val itmCk = database.checkFolderItem(item.folder_name, it)
if (itmCk == 0L) {
val newFolderItemHere = vsmFoldersItems.copy(
folder_name = item.folder_name,
item_id = it
)
database.insertFolderItems(newFolderItemHere)
}
}
the variable "items" is an array of Strings.
Here is the DAO definitions of the above-called functions:
#Query("SELECT uid FROM folders_items_table WHERE folder_name = :folder AND item_id = :item")
fun checkFolderItem(folder: String, item: String): Long
#Insert
suspend fun insertFolderItems(item: FoldersItems)
Placing the loop inside a single transaction should significantly reduce the time taken.
The reason is that each transaction (by default each SQL statement that makes a change to the database) will result in a disk write. So that's 100 disk writes for your loop.
If you begin a transaction before the loop and then set the transaction successful when the loop is completed and then end the transaction a single disk write is required.
What I am unsure of is exactly how to do this when using a suspended function (not that familiar with Kotlin).
As such I'd suggest either dropping the suspend or having another Dao for use within loops.
Then have something like :-
val vsmFoldersItems = FoldersItems()
your_RoomDatabase.beginTransaction()
items.forEach{
val itmCk = database.checkFolderItem(item.folder_name, it)
if (itmCk == 0L) {
val newFolderItemHere = vsmFoldersItems.copy(
folder_name = item.folder_name,
item_id = it
)
database.insertFolderItems(newFolderItemHere)
}
}
your_RoomDatabase.setTransactionSuccessful() //<<<<<<< IF NOT set then ALL updates will be rolled back
your_RoomDatabase.endTransaction()
You may wish to refer to:-
https://developer.android.com/reference/androidx/room/RoomDatabase
You may wish to especially refer to runInTransaction
I have a collection with over 1000 json documents. I want to now delete all the documents in that collection.
Is there any way of doing it from the portal using a query?
PS: I know I can delete the collection and re-create it or use a c# application to do it.
No, there is no equivalent to the SQL DELETE expression in DocumentDB so there is no way to do it with a query. Here is a stored procedure (CoffeeScript) that does this for me when I want to delete based upon a query.
deleteSomeDocuments = (memo) ->
collection = getContext().getCollection()
unless memo?
memo = {}
if memo.returnDeleted
unless memo.deleted?
memo.deleted = []
else
memo.returnDeleted = false
stillQueuingOperations = true
query = () ->
if stillQueuingOperations
responseOptions =
pageSize: memo.remaining
setBody()
if memo.filterQuery?
memo.stillQueueing = collection.queryDocuments(collection.getSelfLink(), memo.filterQuery, responseOptions, onReadDocuments)
else
memo.stillQueueing = collection.readDocuments(collection.getSelfLink(), responseOptions, onReadDocuments)
onReadDocuments = (err, resources, options) ->
if err
throw err
if resources.length isnt memo.remaining
throw new Error("Expected memo.remaining (#{memo.remaining}) and the number of rows returned (#{resources.length}) to match. They don't.")
memo.stillQueueing = true
while memo.remaining > 0 and memo.stillQueueing
oldDocument = resources[memo.remaining - 1]
documentLink = oldDocument._self
etag = oldDocument._etag
options = {etag} # Sending the etag per best practice, but not handling it if there is conflict.
getContext().getResponse().setBody(memo)
memo.stillQueueing = collection.deleteDocument(documentLink, options)
if memo.stillQueueing
if memo.returnDeleted
memo.deleted.push(oldDocument)
memo.remaining--
setBody = () ->
getContext().getResponse().setBody(memo)
query()
return memo
exports.deleteSomeDocuments = deleteSomeDocuments
I have a documentDb instance which I can query using the azure portal tools:
When I write the same query in code like this:
let valuationCollection = client.CreateDocumentCollectionQuery(database.CollectionsLink).Where(fun dc -> dc.Id = "taxinformation").ToArray().FirstOrDefault()
let valuationDocumentLink = valuationCollection.SelfLink
let valuationQueryString = "SELECT * FROM ti WHERE ti.index = 1"
let valuationQuery = client.CreateDocumentQuery(valuationQueryString,valuationQueryString)
let valuationValue = valuationQuery |> Seq.head
let valuation = HouseValuation.Parse(valuationValue.ToString())
I am getting this error:
System.AggregateException: One or more errors occurred. --->
Microsoft.Azure.Documents.NotFoundException: The value 'SELECT * FROM
TI WHERE TI.INDEX = 1' specified for query '$resolveFor' is invalid.
at
Microsoft.Azure.Documents.BackoffRetryUtility`1.d__0.MoveNext()
I have other queries that work fine. I am wondering if "index" should not be used?
Thanks in advance
No, index is not reserved. You should be able to use this.
The problem was with the code itself on line 4 above:
let valuationQuery = client.CreateDocumentQuery(valuationQueryString,valuationQueryString)
should be
let valuationQuery = client.CreateDocumentQuery(valuationDocumentLink,valuationQueryString)
hi guys i am stuck converting below sql to LINQ query.
all i want is to have maximum number from list of (FA-00001 ,FA-00059)
SELECT MAX(CAST(SUBSTRING(ReferenceId, PATINDEX('%-%', ReferenceId) + 1, LEN(ReferenceId) - PATINDEX('%-%', ReferenceId)) AS int)) AS MaxReferenceId FROM [ClientRC].[dbo].[EHO_Action]
is this possible to convert to LINQ? thanks
An alternative approach using anonymous projection:
var y = (from record in
(from record in db.ClientRC
select new
{
Group = "x",
ReferenceNumber = Convert.ToInt32(record.ReferenceId.Split('-')[1])
})
group record by new { record.Group } into g
select new
{
MaxReferenceId = g.Max(p => p.ReferenceNumber)
});
http://msdn.microsoft.com/en-us/library/bb386972.aspx
var myvar = (from v in db.object where v!=null select v.id).Max();
MSDN has lots of examples for stuff like this.
Or, you can execute queries directly against a datacontext if you're using entity framework. Just make sure if you're doing anything with parameters you're parameterizing the query and not taking user input directly into it.
http://msdn.microsoft.com/en-us/library/ee358769.aspx
Try this..
var list = DBContext.EHO_Action
.Where(x => x.YourListColumn != null)
.Select(x => x.YourListColumn).ToList(); // Take the list (FA-00001 ,FA-00059) from db to a list
var maxNo = list.Max(x => Convert.ToInt32(x.Split('-')[1]));
Please change the context and column names according to your Linq context.
If you want to use sql you can do it this way..
var list = DBContext.ExecuteQuery<string>("select yourreqrdcolumn from [ClientRC].[dbo].[EHO_Action]");