When I delete an item using AppSync (or DataStore) on an AWS Amplify app, it makes two changes to the DynamoDB item:
The delete field is set to true
A _ttl field is added, and a timestamp value is provided of 1 month in the future.
According to the AppSync conflict detection documentation, the value is configured on the DeltaSyncTableTTL value, which is configured on the data source:
_ttl
A numeric value that stores the timestamp, in epoch seconds,
when an item should be removed from the Delta table. This value is
determined by adding the DeltaSyncTableTTL value configured on the
data source to the moment when the change occurred. This field should
be configured as the DynamoDB TTL Attribute.
If I go to my AppSync console, and navigate to 'Data Sources' in the left panel, I'm provided with links to my DynamoDB data sources. But I can't find any settings anywhere in the AppSync or DynamoDB consoles to update a DeltaSyncTableTTL value.
To update the "Base table time to live", go to your AppSync console and click 'Data Sources' on the panel on the left side of the page.
Then, select the datasource/dynamodb table by clicking the round checkbox to the left of the data source and click the 'Edit' button above.
Scroll down to the following and adjust as required:
Related
In application insights I can click on an exception and find nicely formatted information about it if I click on it.
This could look like this:
I would like to archive the same in an Azure Workbook. Here I can display all my exceptions with the KQL Term "Exceptions".
With the column settings I was hoping to be able to create a link to the Application insights page that shows me the same result like in the picture above.
I use this configuration:
Now I have a link, but when I click it there is just an empty pane:
Is there anything I misunderstand?
How do I need to configure my column settings to get a direct link to the Exception details like in clicking on the same item in Application Insights?
While not super obvious, the info bubble for the link settings shows:
The value in the column is expected to be an itemId of an exception telemetry item.
So as long as your query returns the itemId field, you can map that in the column settings:
(also, iirc, if the query returns a timestamp column, the details view will try to narrow down its search to just use a range around that time instead of trying to query a larger amount of time to find an item with that id)
Edit to add: if you are using thee Log Analytics based schema, AppExceptions the field is there but missing from documentation and the schema, there is an _ItemId field that is this field. I'm working with the App Insights/ Log Analytics teams to get this properly documented there.
Also, even if you're using the log analytics based app insights, you can also always query through the "app insights" resource and use the app insights schema the "old" way as well. (unless you're doing something specific in log analytics that the AI based way doesn't have?)
example:
left is query against app insights schema, with exceptions table and itemId column
right is query against logs schema, with AppExceptions table and _ItemId field, returning the same exact item by that id value. you can seee that the _ItemId field shows red squiggles like it isn't valid, but that is incorrect, you'll get the right row if you run the query.
Thanks in advance for your advice!
Background
I’m creating a database to track orders placed by customers.
An ‘Orders’ table stores general details about an order like the customer’s name, order date, and delivery-required date.
A separate ‘Order_Items’ table stores the specific items that the customer has ordered.
The is a one-to-many relationship between the ‘Orders’ table and ‘Order_Items’ table, i.e. one ‘Order’ can have many ‘Order_Items’, but each ‘Order_Item’ must be associated with only one ‘Order’.
Current State
Currently, I have a page where the user creates a new ‘Order’ record. The user is then taken to another page where they can create as many ‘Order_Item’ records as are needed for the order.
Desired State
What I would like to achieve is: When a user creates new ‘Order_Item’ records, it automatically allocates the current ‘Order’ record as the foreign key for the new ‘Order_Item’ record.
What I've Tried So Far
Manual Action By The User: One way of establishing the link between an 'Order' and all of its 'Order_Items' would be to add a drop-down widget which which effectively asks the user something like "Which order number do all of these items belong to"? The user's action would then establish the link between the two tables and associate one 'Order' with many 'Order_Items'. However, my goal is for this step to be handled programatically instead.
Official Documentation: I’ve referred to the offical documentation which was useful, but as I'm still learning I don’t really know exactly what to search for. The prefetch feature appeared promising but does not actually establish a link; it just loads associated records more efficiently.
App Maker Tutorials: I found an App Maker tutorial which creates an HR App where a user can create a list of ‘Departments’, then create a list of ‘Employees’, and then link an ‘Employee’ to a ‘Department’. However, in the example app this connection is established manually by the user. In my desired state I would like the link to be established programatically.
Manual Save Mode:
I’ve also tried switching to manual save mode so that the user has to create a draft ‘Orders’ record and then several draft ‘Order Items’ records and then save them all at once. However, I haven’t managed to make this work. I’m not sure whether the failure of this approach is because 1) I’m try to create draft records on more than one table, 2) I’m just not doing it correctly, or 3) I thought I read somewhere that draft records are deprecated.
Other Ideas
I'm very new to this field and am may be wrong, but I have a feeling I may need to use some scripting to establish the link. For example, maybe I could use a global variable to remember which 'Order' the user creates. Then, for each 'Order_Item' I could use the onBeforeCreate event to trigger a script that establishes the link between the 'Order_Item' and the 'Order' that was remembered from the previously established global variable.
Updated Question
Thanks Markus and Morfinismo for your answers. I have been using both answers with some success.
Morfinismo: I've successfully used the code you directed me to on existing records but cannot seem to get it to work for newly created records.
For example:
widget.datasource.createItem(); // This creates a new record
var managerRecord = app.datasources.Manager.item; // This sets the Manager of the currently selected parent record as a variable successfully.
var teamRecord = app.datasources.Teams.item; // This attempts to set the Manager of the currently selected record as a variable. However, the record that was created in line 1 is not selected. Therefore, App Maker does not seem to know which record this line of code relates to and returns the error Cannot set property ‘Manager’ of null.
// Assign the manager to the team.
teamRecord.Manager = managerRecord; // This successfully assigns the manager but only in cases where the previous line of code was successful (i.e. existing records and not newly created ones).
Do you have any suggestions or comments on how to apply this code to records that are created by the initial line of code in line 1?
I have found the easiest way to create related items for situations such as yours is to actually import a form with the datasource set to Parent: Child (relation) or Parent: Child (relation) (create). So in your case the datasource would need to be set to Order: Order_Items (relation).
You can get this accomplished in two different ways using the form widget wizard:
Option 1:
If your page datasource is set to Order_Items, drag your form on your page.
In the datasource selection section, your datasource in the form widget should default to `Inherited: Order_Items'. Click the 'Advanced' button in the bottom left corner, then from the datasources category find Order as your datasource, then select relations in the next field, and then Order_Items in the next field, choose 'Insert only' or 'Edit' form and then the appropriate fields you want in the form.
Now every item that gets created in that form will automatically be a child record of the currently selected record in your Order datasource.
Option 2:
If your page datasource is set to Order, drag your form on your page.
In the datasource selection section, your datasource in the form widget should default to Inherited: Order. Scroll down in your datasource selection section until you find Order: Order_Items (relation), then choose 'Insert only' or 'Edit' form and then the appropriate fields you want in the form.
Now every item that gets created in that form will automatically be a child record of the currently selected record in your Order datasource.
In your Order model, make sure that the security setting is set appropriately that a user is allowed to create relations of Order_Items in Order. That is the simplest approach in my opinion since you don't have to hard code the parent into your form or client/server scripts. It is automatically based on the currently selected parent, and is essentially doing the same thing that #Morfinismo explained in the client script section.
The comment I placed under your question included a link to the official documentation that explains what you need. Anyways, your question is not clear enough to determine whether you are creating the records via client script or server script, hence this is a very general answer.
To manage relations via client script:
var managerRecord = app.datasources.Manager.item;
var teamRecord = app.datasources.Teams.item;
// Assign the manager to the team.
teamRecord.Manager = managerRecord;
// Changes are saved automatically if the datasource in auto-save mode
// Add a team member to a Manager's team.
// Note: Retrieve Members on the client before proceeding, such as by using prefetch option in datasource - datasources Team -> Members)
var engineerRecord = app.datasources.TeamMember.item;
teamRecord.Members.push(engineerRecord);
To manage relations via server script:
// Get the record for the Team to modify.
var teamRecord = app.models.Teams.getRecord("team1");
// Assign a manager to the Team.
var managerRecord = app.models.EmployeeDB.getRecord("manager1");
teamRecord.Manager = managerRecord;
// Note: The new association is not saved yet
// Assign a team member to the Team.
var engineerRecord = app.models.EmployeeDB.getRecord("engineer1");
teamRecord.Members.push(engineerRecord);
// Save both changes to the database.
app.saveRecords([teamRecord]);
The above information is taken directly from the official documentation, which like I said, I referred to in the comment I placed under your question.
I need to query a collection and return all documents that are new or updated since the last query. The collection is partitioned by userId. I am looking for a value that I can use (or create and use) that would help facilitate this query. I considered using _ts:
SELECT * FROM collection WHERE userId=[some-user-id] AND _ts > [some-value]
The problem with _ts is that it is not granular enough and the query could miss updates made in the same second by another client.
In SQL Server I could accomplish this using an IDENTITY column in another table. Let's call the table version. In a transaction I would create a new row in the version table, do the updates to the other table (including updating the version column with the new value. To query for new and updated rows I would use a query like this:
SELECT * FROM table WHERE userId=[some-user-id] and version > [some-value]
How could I do something like this in Cosmos DB? The Change Feed seems like the right option, but without the ability to query the Change Feed, I'm not sure how I would go about this.
In case it matters, the (web/mobile) clients connect to data in Cosmos DB via a web api. I have control of the entire stack - from client to back-end.
As the statements in this link:
Today, you see all operations in the change feed. The functionality
where you can control change feed, for specific operations such as
updates only and not inserts is not yet available. You can add a “soft
marker” on the item for updates and filter based on that when
processing items in the change feed. Currently change feed doesn’t log
deletes. Similar to the previous example, you can add a soft marker on
the items that are being deleted, for example, you can add an
attribute in the item called "deleted" and set it to "true" and set a
TTL on the item, so that it can be automatically deleted. You can read
the change feed for historic items, for example, items that were added
five years ago. If the item is not deleted you can read the change
feed as far as the origin of your container.
Change feed is not available for your requirements.
My idea:
Use Azure Function Cosmos DB Trigger to collect all the operations in your specific cosmos collection. Follow this document to configure the input of azure function as cosmos db, then follow this document to configure the output as azure queue storage.
Get the ids of changed items and send them into queue storage as messages.When you want to query the changed item,just query the messages from the queue to consume them at a specific unit time and after that just clear the entire queue. No items will be missed.
With your approach, you can get added/updated documents and save reference value (_ts and id field) somewhere (like blob)
SELECT * FROM collection WHERE userId=[some-user-id] AND _ts > [some-value] and id !='guid' order by _ts desc
This is a similar approach we use to read data from Eventhub and store checkpointing information (epoch number, sequence number and offset value) in blob. And at a time only one function can take a lease of that blob.
If you go with ChangeFeed, you can create listener (Function or Job) to listen all add/update data from collection and you can store those value in some collection, while saving data you can add Identity/version field on every document. This approach may increase your cosmos DB bill.
This is what the transaction consistency levels are for: https://learn.microsoft.com/en-us/azure/cosmos-db/consistency-levels
Choose strong consistency and your queries will always return the latest write.
Strong: Strong consistency offers a linearizability guarantee. The
reads are guaranteed to return the most recent committed version of an
item. A client never sees an uncommitted or partial write. Users are
always guaranteed to read the latest committed write.
I have a question
If I am using dynamodb and I want to put item inside table where I am making hash key similar and for range key I am using current timestamp and if two or more event putitem at the same time then what would be the result. And I want all items to be updated. What can I do here.????
PutItem API - Two items having same hash key and range key:-
The first request will create the item in the table
Second request will update (i.e. overwrite) the item on the table
Update the items having the same hash key but different range key :-
Firstly, DynamoDB doesn't allow to update multiple items in one API call
Second, you need both Hash and Range key to update the item
Steps - Get all the items for the hash key
Update the each item sequentially using update item API (or) use batchWrite item API to update all the items in one API operation. You can use PutRequest on BatchWriteItem to update the item. The API documentation says that you can't update the item. It actually means that it can't be used to update the specific attribute in the item. It will replace the entire item. You should be ok as long as you have the full item data that needs to be replaced. As per the step 5 above, you should have the full item details (i.e. all the attributes in the item) that need to be updated (i.e. replace in case of batch write item)
I'm using Dexi.io to scrape some data that outputs to Google Drive as a CSV, that gets parsed (through a Google sheets script) and added to a Native sheet (all automatically).
I'd like to push my data (automatically) to a "database Visualizer" of some sort (using knack.com currently) that allows me to display the data (in Table format) with some options to filter, sort and dig deeper; all protected by login creds that I manage.
I tried using Zapier to automate the Google Sheets to Knack integration, but Knack only has an option to "Create New Records" through Zapier and not "Update Records". (Updating records exists as an API endpoint)
I need help proceeding as I'm not a developer and am starting to hit the limits of my capabilities.
Could someone please recommend a tool (that integrates with Sheets, updates data periodically and lets me control the domain and login creds) or the optimal way to proceed with this? (I'd gladly hire a freelancer to help me build this out optimally)
Some more, potentially relevant, info: Dexi.io can output through FTP, Drive, Box or Amazon S3 (remember, not a dev :$)
kintone is a "database Visualizer" similar to Knack, and they have actions to update records.
https://zapier.com/zapbook/kintone/
There are two options to update records as there are two ways in which the unique key can be defined.
Each record in kintone has a "Record ID" associated with it - this is an autonumber made by kintone. You can specify this as the key to update, in which case you would use the "Update Record By Record ID" action.
If you would prefer though to set your own unique key and use that as the key to update, you can define that unique key in your database (I guess the data you are scraping has its own ID for each record). In this case, you can place a "Text (single-line)" field in your database, open up the options and select "Prohibit duplicate values" which will make this field into a unique field - meaning that no two records can hold the same value.
Once you set that field up (and update your kintone App settings), you can select this field to be the unique key to update for the "Update Record By Update Key" Action (the "Update Key" in the action name is referring to the unique key that you just made).
And yes, kintone gives you control over login creds, and each login cred can have different view/add/edit permissions over each record you have in your app.
You can also set a custom subdomain name, but the domain name will have to be kintone.com i.e. you can have a https:/ /mycustomname.kintone.com sort of name.
Hope this helps.