I'm learning meteor and reading the todos example:
$ curl https://install.meteor.com | sh
$ meteor create --example todos
In todos/server/bootstrap.js around line 50 there are lines as follows.
var timestamp = (new Date()).getTime();
for (var i = 0; i < data.length; i++) {
var list_id = Lists.insert({name: data[i].name});
for (var j = 0; j < data[i].contents.length; j++) {
var info = data[i].contents[j];
Todos.insert({list_id: list_id,
text: info[0],
timestamp: timestamp,
tags: info.slice(1)});
timestamp += 1; // ensure unique timestamp.
}
}
I wonder why do we need to ensure the timestamp is unique? Is it required for meteor/mongodb or just application-specific?
The timestamp isn't required or doesn't have to be unique, its application specific in this case.
The use case here is just so that each todo item has an sequential timestamp so that it displays it in the correct order as can be seen in the client js, oldest at the top and newest at the bottom.
Template.todos.todos = function () {
...
return Todos.find(sel, {sort: {timestamp: 1}});
};
The server js uses adds 1 microsecond to each item in the loop to simulate the sort order that would perhaps be put in with with an actual user. Its only done once to create an example database.
If there was no sort order/sequential timestamps the todos would be sorted by the records _id, which would make their order at best, random. The reason for this is meteor uses Meteor.uuid() to generate its _id fields which are random & without sequentiality.
Related
I have a use case where I have to write 5000 records into dynamoDB table in one shot. I am using batchSave api of DynamoDBMapper Library. it can write upto 25 records in one go.
can I pass the list of 5000 records to it and it will internally convert them into batch of 25 records and write to dynamodb table or I will have to handle this thing in my code using conditional some logic and will pass only 25 records to batchSave?
According to the batchSave documentation, batchSave():
Saves the objects given using one or more calls to the AmazonDynamoDB.batchWriteItem
Indeed, it splits up the items you give it into appropriately-sized batches (25 items) and writes them using the DynamoDB BatchWriteItem operation.
You can see the code that does this in batchWrite() in DynamoDBMapper.java:
/** The max number of items allowed in a BatchWrite request */
static final int MAX_ITEMS_PER_BATCH = 25;
// Break into chunks of 25 items and make service requests to DynamoDB
for (final StringListMap<WriteRequest> batch :
requestItems.subMaps(MAX_ITEMS_PER_BATCH, true)) {
List<FailedBatch> failedBatches = writeOneBatch(batch, config.getBatchWriteRetryStrategy());
...
Here are the methods I use in order to achieve this end. I manage to do it, by first chucking the dataArray into small arrays (of length 25):
const queryChunk = (arr, size) => {
const tempArr = []
for (let i = 0, len = arr.length; i < len; i += size) {
tempArr.push(arr.slice(i, i + size));
}
return tempArr
}
const batchWriteManyItems = async (tableName, itemObjs, chunkSize = 25) => {
return await Promise.all(queryChunk(itemObjs, chunkSize).map(async chunk => {
await dynamoDB.batchWriteItem({RequestItems: {[tableName]: chunk}}).promise()
}))
}
I have table with multiple customerKey values assigned to a numeric value; I wrote a script where foreach row of data I scan whole table to find all values assigned to the current customerKey and return a highest one;
I have a problem with performance - script processes around 10 records per second - any ideas how to improve this or maybe propose an alternative solution plesae?
function getLastest() {
var date = app.models.magicMain.newQuery();
var date_all = date.run();
date_all.forEach(function(e) { // for every row of date_all
var temp = date_all.filter(function(x) {
return x.SubscriberKey === e.SubscriberKey; // find matching records for the current x.SubscriberKey
});
var dates = [];
temp.forEach(function(z) { // get all matching "dates"
dates.push(z.Date);
});
var finalValue = dates.reduce(function(a, b) { // get highest dates value (integer)
return Math.max(a, b);
});
var record = app.models.TempOperatoins.newRecord(); // save results to DB
record.email = e.SubscriberKey.toString() + " " + finalValue.toString();
app.saveRecords([record]);
});
}
The only suggestion I have would be to add:
var recordstosave = [];
At the top of your function.
Then replace app.saveRecords([record]) with recordstosave.push(record).
Finally outside of your foreach function do app.saveRecords(recordstosave).
I saw major processing time improvements doing this rather than saving each record individually inside a loop.
After reading the documentation, I'm having a hard time conceptualizing the change feed. Let's take the code from the documentation below. The second change feed is picking up the changes from the last time it was run via the checkpoints. Let's say it is being used to create summary data and there was an issue and it needed to be re-run from a prior time. I don't understand the following:
How to specify a particular time the checkpoint should start. I understand I can save the checkpoint dictionary and use that for each run, but how do you get the changes from X time to maybe rerun some summary data
Secondly, let's say we are rerunning some summary data and we save the last checkpoint used for each summarized data so we know where that one left off. How does one know that a record is in or before that checkpoint?
Code that runs from collection beginning and then from last checkpoint:
Dictionary < string, string > checkpoints = await GetChanges(client, collection, new Dictionary < string, string > ());
await client.CreateDocumentAsync(collection, new DeviceReading {
DeviceId = "xsensr-201", MetricType = "Temperature", Unit = "Celsius", MetricValue = 1000
});
await client.CreateDocumentAsync(collection, new DeviceReading {
DeviceId = "xsensr-212", MetricType = "Pressure", Unit = "psi", MetricValue = 1000
});
// Returns only the two documents created above.
checkpoints = await GetChanges(client, collection, checkpoints);
//
private async Task < Dictionary < string, string >> GetChanges(
DocumentClient client,
string collection,
Dictionary < string, string > checkpoints) {
List < PartitionKeyRange > partitionKeyRanges = new List < PartitionKeyRange > ();
FeedResponse < PartitionKeyRange > pkRangesResponse;
do {
pkRangesResponse = await client.ReadPartitionKeyRangeFeedAsync(collection);
partitionKeyRanges.AddRange(pkRangesResponse);
}
while (pkRangesResponse.ResponseContinuation != null);
foreach(PartitionKeyRange pkRange in partitionKeyRanges) {
string continuation = null;
checkpoints.TryGetValue(pkRange.Id, out continuation);
IDocumentQuery < Document > query = client.CreateDocumentChangeFeedQuery(
collection,
new ChangeFeedOptions {
PartitionKeyRangeId = pkRange.Id,
StartFromBeginning = true,
RequestContinuation = continuation,
MaxItemCount = 1
});
while (query.HasMoreResults) {
FeedResponse < DeviceReading > readChangesResponse = query.ExecuteNextAsync < DeviceReading > ().Result;
foreach(DeviceReading changedDocument in readChangesResponse) {
Console.WriteLine(changedDocument.Id);
}
checkpoints[pkRange.Id] = readChangesResponse.ResponseContinuation;
}
}
return checkpoints;
}
DocumentDB supports check-pointing only by the logical timestamp returned by the server. If you would like to retrieve all changes from X minutes ago, you would have to "remember" the logical timestamp corresponding to the clock time (ETag returned for the collection in the REST API, ResponseContinuation in the SDK), then use that to retrieve changes.
Change feed uses logical time in place of clock time because it can be different across various servers/partitions. If you would like to see change feed support based on clock time (with some caveats on skew), please propose/upvote at https://feedback.azure.com/forums/263030-documentdb/.
To save the last checkpoint per partition key/document, you can just save the corresponding version of the batch in which it was last seen (ETag returned for the collection in the REST API, ResponseContinuation in the SDK), like Fred suggested in his answer.
How to specify a particular time the checkpoint should start.
You could try to provide a logical version/ETag (such as 95488) instead of providing a null value as RequestContinuation property of ChangeFeedOptions.
What I need to do is use either collection-2 or another package to automatically create a new order number, incremented from the last order number used.
i.e. Starting off with PO123456, when I save this order, the next time I make a new PO, it automatically generates the number PO123457.
I've been looking for a good example or tutorial, but I'm not able to find one.
Using konecty:mongo-counter in conjuntion with aldeed:collection2 and aldeed:simple-schema should be pretty straightforward. In your schema definition try:
POnumber: { type: String, autoValue: function(){
if ( this.isInsert ){ // restrict to when inserting a document
var currentNumber = incrementCounter('purchase order'); // this will use mongo-counter
// WARNING: you can only ever get as rich as 10M POs!!
var zeroPad = "000000" + currentNumber; // pad with 6 zeros
zeroPad = zeroPad.substr(zeroPad.length-7); // restrict to 7 places
return 'PO' + zeroPad; // prefix with 'PO'
} else if ( this.isSet ){
this.unset(); // prevent attempts to change the number
}
}
I have this code
using (var contents = connection.CreateCommand())
{
contents.CommandText = "SELECT [subject],[note] FROM tasks";
var r = contents.ExecuteReader();
int zaehler = 0;
int zielzahl = 5;
while (r.Read())
{
if (zaehler == zielzahl)
{
//access r["subject"].ToString()
}
zaehler++;
}
}
I want to make it faster by accessing zielzahl directly like r[zielzahl] instead of iterating through all entries. But
r[zielzahl]["subject"]
does not work aswell as
r["subject"][zielzahl]
How do I access the column subject of result number zielzahl?
To get only the sixth record, use the OFFSET clause:
SELECT subject, note
FROM tasks
LIMIT 1 OFFSET 5
Please note that the order of returned records is not guaranteed unless you use the ORDER BY clause.