I have a list of files that should be inserted or updated in dynamodb, so I'm doing in this way:
var batch = _dynamoDbContext.CreateBatchWrite<MyEntity>();
batch.AddPutItems(myEntityList);
batch.ExecuteAsync();
This works fine if DynamoDB table is empty, but sometimes I should update instead insert, but I got the following error:
An item with the same key has already been added. Key: Amazon.DynamoDBv2.DocumentModel.Key
How can I solve it ? I need to use batch, because of performance.
You can use transactions to do insert or updates but they are double the cost, otherwise you will need to update one by one
Here's some more info on a previous post
DynamoDB Batch Update
Related
I'm working on a symfony application, and i need to insert multiple raws at once, Doctrine ORM is not a good option because for each raw it will open a connection to execute the query, to avoid this and have one connection inserting all the raws i used prepared statement of doctrine dbal and it works fine, except i need to get the ids of the inserted raws, it seems the only available function is lastinsertedid which returns only the last id not all the last inserted ones, how can i achieve this?
any help would be appreciated!
This is actually not related to doctrine at all. If you want all inserted id's it must be possible in MySQL. "It's unlikely that if doctrine don't have batch insert it will support returning list of ids after batch insert :)"
Check answers related to MYSQL:
How can I Insert many rows into a MySQL table and return the new IDs?
MySQL LAST_INSERT_ID() used with multiple records INSERT statement
But it's possible in postgresql (since you didn't mention you DB):
Retrieving serial id from batch inserted rows in postgresql
You can actually generate IDs before inserting content into database. For example, using random UUIDs.
This library might be of use: https://github.com/ramsey/uuid
use Ramsey\Uuid\Uuid;
$uuid4 = Uuid::uuid4();
echo $uuid4->toString()
I refactored a table that stored both metadata and data into two tables, one for metadata and one for data. This allows metadata to be queried efficiently.
I also created an updatable view with the original table's columns, using sqlite's insert, update and delete triggers. This allows calling code that needs both data and metadata to remain unchanged.
The insert and update triggers write each incoming row as two rows - one in the metadata table and one in the data table, like this:
// View
CREATE VIEW IF NOT EXISTS Item as select n.Id, n.Title, n.Author, c.Content
FROM ItemMetadata n, ItemData c where n.id = c.Id
// Trigger
CREATE TRIGGER IF NOT EXISTS item_update
INSTEAD OF UPDATE OF id, Title, Author, Content ON Item
BEGIN
UPDATE ItemMetadata
SET Title=NEW.Title, Author=NEW.Author
WHERE Id=old.Id;
UPDATE ItemData SET Content=NEW.Content
WHERE Id=old.Id;
END;
Questions:
Are the updates to the ItemMetadata and ItemData tables atomic? Is there a chance that a reader can see the result of the first update before the second update has completed?
Originally I had the WHERE clauses be WHERE rowid=old.rowid but that seemed to cause random problems so I changed them to WHERE Id=old.Id. The original version was based on tutorial code I found. But after thinking about it I wonder how sqlite even comes up with an old rowid - after all, this is a view across multiple tables. What rowid does sqlite pass to an update trigger, and is the WHERE clause the way I first coded it problematic?
The documentation says:
No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect.
Commands in a trigger are considered part of the command that triggered the trigger.
So all commands in a trigger are part of a transaction, and atomic.
Views do not have a (usable) rowid.
I know we can do update by two operations, first get the primary key by querying the db, and then update it by put operation. But does DynamoDB support update by one operation as the relational db (such as mysql)? Since two operations will cost more time in network transferring.
My situation is as:
I have a table A with fields ID, Name, Location, Value.
And name+location can uniquely define a row.
So now I want to update the field "Value" when Name and Location satisfied some condition, but I don't know the ID. So if I use mysql, then I can update it by "Update A set value = XXX where name = "abc" and location="123"".
But when I use dynamoDB, I have to first get the primary key ID.
Then use the Key to update the item. So my question is that does DynamoDB also support similar update operation as mysql does.
Thanks!
Chen hit it on the nose. Joey, the situation you described (Get followed by a Put) is equivalent to 2 mysql functions
SELECT *
FROM TABLE
WHERE key = x
UPDATE TABLE
SET var = param
WHERE key = x
Do you see how the Select/PutItem aren't part of the update process? As long as you have the keys, you don't need to perform a query. I'm assuming you're performing the GetItem before the PutItem request because the PutItem replaces the entire item/row (i.e. deletes all attributes not specified in the Put request).
So if the original item looked like: < key-id=1, first-name=John, last-name=Doe, age=22>
and you perform a PutItem of: < key-id=1,location=NY>
The final item looks like: < key-id=1,location=NY>
If you perform an UpdateItem in place of PutItem then you would instead get:
< key-id=1, first-name=John, last-name=Doe, age=22, location=NY>
Here's a link for using the UpdateItem with Java. There also examples using .net and php
UpdateItem for Java
Correct me if I am wrong but Update Item will consume 1 operation only it will get hash key value and update it if exists else will create new Item (up-to 1 kb item)
here is the link for reference : http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.html#CapacityUnitCalculations
Hope that helps
You don't need to get the primary key first. If you know the primary key, you don't need to get anything and you can simply use the UpdateItem API call to update your item.
If that still isn't clear, please edit your question and add some code samples of what you are trying to do.
We have a table(say T1) that is referenced by about 16 other tables with foreign keys in our SQL Server database. The data is accessed through an ASP.NET application with LINQToSQL. When the user tried to delete a record from T1 the statement would time out. So we decided to first delete the records from the tables that reference T1 and only then delete the record in T1. The problem is that deletion from T1 does not work as fast as expected.
My question is: is it normal that deletion from a table referenced by many other tables to be so time-consuming even if the record itself does not have any 'children' records?
EDIT: Apparently the cause for the timeout was not the delete itself but another query that retrieved data from the same DataContext. Thank you for your suggestions, I have marked as answer the suggestion to add indexes for all foreign keys because it improved our script's execution plan.
I suspect that you may need to look into the indexing on your child tables.
It sounds as if you FKs are set to Cascade Deletes, so I would suspect that some of your tables do not have an index that includes the key to the parent as the first in the index.
In this way your delete will be full scanning the child tables - even if you've already deleted the child records it will still check as you've still got the Cascade set.
When you define a relationship in DB, you can set the Delete rule as Cascade in SQL server. In this way, when you delete the record from the parent table, it will be automatically deleted from the child tables.
Please see the image below:
If it taking long time, you may have set other constraint that will slow
down the process of deletion.
Linq does not do bulk deletes if you're having it operate directly on the record set -- instead, it is probably deleting one record at a time.
To improve performance, use a stored procedure instead for any bulk insert, update or delete operations.
I'm unable to retrieve the latest inserted id from my SQL Server 2000 db using a typed dataset in asp.NET
I have created a tableadapter and I ticked the "Refresh datatable" and "Generate Insert, Update and Delete statements". This auto-generates the Fill and GetData methods, and the Insert, Update, Select and Delete statements.
I have tried every possible solution in this thread
http://forums.asp.net/t/990365.aspx
but I'm still unsuccesfull, it always returns 1(=number of affected rows).
I do not want to create a seperate insert method as the auto-generated insertCommand perfectly suits my needs.
As suggested in the thread above, I have tried to update the InsertCommand SQL syntax to add SELECT SCOPY_IDENTITY() or something similar, I have tried to add a parameter of type ReturnValue, but all I get is the number of affected rows.
Does anyone has a different take on this?
Thanks in advance!
Stijn
I decided to give up, I can't afford to waste any more time on this.
I use the Insert statement after which I do a select MAX(id) query to hget the insert ID
If anyone should have a solution, I'll be glad to read it here
Thanks
Stijn
I successfully found a way to get the incremental id after insert using my table adapter.
My approach is a little different, I'm using a Store procedure to make the insert, so my insert command has all the values but the ID, I made the sp return the ID just calling:
SET #ID=SCOPE_IDENTITY()
and then
COMMIT TRAN
and last line will be
RETURN #ID
Then I searched my table adapter parameters for InsertCommand and set the #RETURNVALUE to the column of the incremental ID of the table, so when it's executed automatically put the return value on the id field.
Hope this help
You need to tell your table's table-adapter to refresh the
data-table after update/insert operation.
This is how you can do that.
Open the properties of TableAdapter -> Default Select Query -> Advnaced options. and Check the option of Refresh the data table. Save the adapter now. Now when you call update on table-adapter, the data-table will be updated [refreshed] after the update/insert operation and will reflect the latest values from database table. if the primary-key or any coloumn is set to auto-increment, the data-table will have those latest value post recent update.
Now you can Call the update as TableAdapterObj.Update(ds.dataTable);
Read latest values from the DataTable(ds.dataTable) coloumns and assign respective values into the child table before update/insert. This will work exactly the way you want.
alt text http://ruchitsurati.net/files/tds1.png