I have a dynamodb table that I declared using dynamoose as below:
const schema = new dynamoose.Schema({
"email": String,
"name": String,
"vehicleMotor": {
"type": Number,
"default": 0
},
"vehicleMotorId": String,
"vehicleMotorImage1File": String,
"vehicleMotorImage2File": String,
}, {
"saveUnknown": true,
"timestamps": true
});
From my understanding, when I have "timestamps": true declared, it should have both createdAt and updatedAt field.
So when I run my code that looks like this
if (new){
const newSeller = new Seller({
"email": email,
"name": name
})
var saveResult = await newSeller.save();
}else{
var updateResult = await Seller.update( { "email": email, sellerType: 1 }, {
"name": name
})
}
and when I checked the inserted/updated data inside Amazon DynamoDB Management Console, there's no createdAt, only updatedAt. By right I should also have createdAt too right? If not, how to make sure createdAt will always be there?
Based on the comment from the original poster. It looks like this is only occurring for the update call.
There isn't quite enough information in the original question for me to give a concrete answer as to what I believe is the best solution. So I'm going to make a few assumptions, and give a lot of high level detail about how Dynamoose handles this situation.
Just a little bit of the behind the scenes which will help make my answer more clear. From Dynamoose's perspective, it has no idea if the document/item already exists in the database or not. That leads to a situation where createdAt is difficult to get 100% accurate. You are running into one of these situations. For the update call, Dynamoose assumes that the document already exists, and therefore doesn't set the createdAt timestamp. This makes sense because createdAt doesn't really match with an update call. However, DynamoDB & Dynamoose technically allows for using update to create a new document/item. But Dynamoose has no way of knowing which it is, so we use the behavior of assuming update means not creating a new document for this context.
As for a possible solution. You have a new variable. I'm curious how you are defining that variable. One option would be to check the table using a get call and see if the document already exists. If you do that as your new variable, it should work fine. Because it will save if it doesn't exist, and if it already exists, it should have the createdAt variable already. Major downside to this is that you have to always do a read operation before writing. Which increases the latency of the application, and slows things down. But it will achieve what you want.
Now. In the event you have documents in your table that don't have the createdAt timestamp (ex. you created it outside of Dynamoose, or you created it before adding the timestamp option), the above solution won't work. This is because even checking to see if it exists, will cause the update method to be run, which Dynamoose assumes to be an update not a creation. In this case, any solution is really dependent on what your application wants to do. The item already exists in the table, so it's impossible to know when the true createdAt timestamp was (unless you keep logs and all that). You could run a one time operation to go through and add the current timestamp to the createdAt field if it doesn't have it for each document (but again that won't be truly accurate). Or of course you could just ignore it, and not rely on that field always.
To summarize, the timestamps feature in Dynamoose is truly a client side feature. Dynamoose has limited insight into the state of the data, and DynamoDB doesn't provide this functionality built in. This means Dynamoose has to make assumptions about how to handle these situations. However, if you follow Dynamoose's patterns for timestamps (ex. update won't add the timestamp and should only be used for updating existing items, creating all items in Dynamoose, etc), it will be totally accurate, and you won't run into any of these pitfalls.
If you have any creative solutions for how to improve Dynamoose's knowledge here, feel free to create a pull request on the repo or create an issue to discuss your ideas.
Related
In my firebase db I have 3 collections:
Users
{user_id}: {name: "John Smith"}
Items
{item_id}: {value: 12345}
Actions
{action_id}: {action: "example", user: {user_id}, items:{item_id}}
Basically, instead of storing the Users and Items under the Actions Collection, I just keep an ID. But now I need a list of all actions and this also needs info from the Users and Items Collection. How can I efficiently query firebase so I can get a result that looks like this:
{
action: "example",
user: {
name: "John Smith"
},
item: {
value: 1234
}
}
Unfortunately, there is no such thing in firebase or a similar database, basically, you are looking for a traditional join, which is no recommended thing to do in a NoSQL database.
If you want to do it in firebase, you will need:
Get the element you are looking for from your main collection Actions in this case.
Then you need to do another call to the Items collections where item_id == action.item_id.
Then assign in the actions["Item"] = item_gotten.
This is not a recommended use as I said, usually, when you are using a NoSQL Database you are expecting a denormalize structure, from your application you need to save the whole Item, in the Action JSON, and also in the Item. Yes, you will have duplicate data but this is fine for this kind of model. also you shouldn't expect too many changes in one specific object within your whole object key If you are managing a big set of changes you could be using the incorrect kind of DB.
For aggregation queries reference, you might check: https://firebase.google.com/docs/firestore/solutions/aggregation
In firestore i have a collection called things.
Each thing is owned by a user.
Each thing can be shared by the owner with other specified users.
the structure of thing looks something like
{
id: "thing01",
sharedWith: {
"user1": true,
"user2": true,
},
dtCreated: 3458973948
}
When I want to retrieve all thing objects that are shared with user1, ordered by dtCreated desc,
i can't do this without having to create an index on things.thing.user1
i.e. for every unique userid i have to create an index on the things collection.
Obviously this is not practical. The docs talk about using full text search for this, but this doesn't seem like a problem we would want to use full text search for.
Is there a different way i should be structuring the data to achieve what i want?
Is firestore just the wrong technology choice for this?
It's working very well for storing the thing objects themselves.
---- update ----
this question is not a real duplicate of Firestore: Working with nested single queries because the answer provided there is very specific to the OP's context.
I've read almost everywhere about structuring one's Firebase Database for efficient querying, but I am still a little confused between two alternatives that I have.
For example, let's say I want to get all of a user's "maxBenchPressSessions" from the past 7 days or so.
I'm stuck between picking between these two structures:
In the first array, I use the user's id as an attribute to index on whether true or false. In the second, I use userId as the attribute NAME whose value would be the user's id.
Is one faster than the other, or would they be indexed a relatively same manner? I kind of new to database design, so I want to make sure that I'm following correct practices.
PROGRESS
I have come up with a solution that will both flatten my database AND allow me to add a ListenerForSingleValueEvent using orderBy ONLY once, but only when I want to check if a user has a session saved for a specific day.
I can have each maxBenchPressSession object have a key in the format of userId_dateString. However, if I want to get all the user's sessions from the last 7 days, I don't know how to do it in one query.
Any ideas?
I recommend to watch the video. It is told about the structuring of the data very well.
References to the playlist on the firebase 3
Firebase 3.0: Data Modelling
Firebase 3.0: Node Client
As I understand the principle firebase to use it effectively. Should be as small as possible to query the data and it does not matter how many requests.
But you will approach such a request. We'll have to add another field to the database "negativeDate".
This field allows you to get the last seven entries. Here's a video -
https://www.youtube.com/watch?v=nMR_JPfL4qg&feature=youtu.be&t=4m36s
.limitToLast(7) - 7 entries
.orderByChild('negativeDate') - sort by date
Example of a request:
const ref = firebase.database().ref('maxBenchPressSession');
ref.orderByChild('negativeDate').limitToLast(7).on('value', function(snap){ })
Then add the user, and it puts all of its sessions.
const ref = firebase.database().ref('maxBenchPressSession/' + userId);
ref.orderByChild('negativeDate').limitToLast(7).on('value', function(snap){ })
I'm creating an order form and a schema defined for an Order (certain required fields such as address, customer info, items selected and their quantities, etc).
a. User visits site.
b. A unique ID is generated for their session as well as a timestamp.
var userSession = {
_id: createId(),
timestamp: new Date(),
};
var sessionId = userSession._id;
c. The userSession is placed in local storage.
storeInLocalStorage('blahblah', sessionObject);
d. An Order object is created with the sessionId as the only field so far.
var newOrder = {
sessionId: sessionId;
};
e. Obviously at this point the Order object won't validate according to the schema so I can't store it in Mongo. BUT I still want to store it in Mongo so I can later retrieve incomplete orders, or orders in progress, using the sessionID generated on the user's initial visit.
This won't work because it fails validation:
Orders.insert(newOrder);
f. When a user revisits the site I want to be able to get the incomplete order from Mongo and resume:
var sessionId = getLocalStorage('blahblah')._id;
var incompleteOrder = Orders.findOne({'sessionId', sessionId});
So I'm not sure how to go about doing this while accomplishing these points.
I want full simpleschema validation on the Orders collection when the user is entering in items on the forms and when the user is intending to submit a full, complete order.
I want to disable simpleschema validation on the Orders collection and still allow storing into the DB so that partial orders can be stored for resumption at a later time.
I can make a field conditionally required using this here but that would mean 50+ fields would be conditionally required just for this scenario and that seems super cumbersome.
It sounds like you want to have your cake, and eat it too!
I think the best approach here would be keep your schema and validation on the Orders collection, but store incomplete orders elsewhere.
You could store them in another collection (with a more relaxed schema) if you want them on the server (possibly for enabling resume on another device for the logged in user) , or more simply in Local Storage, and still enable the resume previous order behaviour you are wanting.
Only write to the Orders collection when the order is complete (and passes validation).
Here's a variation on #JeremyK's answer: add an inProgress key to your order of type [Object]. This object would have no deeper validation. Keep your in progress order data in there until the order is final then copy/move all the relevant data into the permanent keys and remove the inProgress key. This would require that you make all the real keys optional of course. The advantage is that the object would maintain its primary key throughout the life cycle.
I think this particular case has been solved; but just in case, you can skip Simple Schemma validations by accessing MongoDB native API via Collection#rawCollection():
Orders.rawCollection().insert(newOrder);
While this question is very old in the meantime there is a better solution. You probably use simple schema together with collection2. Collection2 has the ability to set multiple schemas based on a selector and then validate against the correct schema based on it.
https://github.com/Meteor-Community-Packages/meteor-collection2#attaching-multiple-schemas-to-the-same-collection
e.g. you could have a selector {state: 'finished'} and only apply the full schema to these documents while having another selctor, e.g. {state: 'in-progress'} for unfinished orders with a schema with optional fields.
i'm migrating data from a rails system, and it would be really convenient to assign the migrated objects IDs like post0000000000001, etc.
i've read here
Creating Meteor-friendly id's in Mongo?
that Meteor creates random 17 character strings from
23456789ABCDEFGHJKLMNPQRSTWXYZabcdefghijkmnopqrstuvwxyz
which looks to be chosen to avoid possibly ambiguous characters (omits 1 and I, etc.)
do the IDs need to be random for some reason? are there security implications to being able to guess a Meteor document's ID?! or it is just an easy way of generating unique IDs?
Mongo seems fine with sequential ids:
http://docs.mongodb.org/manual/core/document/#the-id-field
http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
so i would guess this would have to be a Meteor constraint if it exists.
The IDs just need to be unique.
Typically there is an element of order: Such as using integers, or timestamps, or something with sequentiality.
This can't work in Meteor since inserts can come from the client, they may be disconnected for a period, or clients clocks may be off/have varying latency. Also its not possible to know the previous _id (in the case of a sequential _id) at the time an _id is written owing to latency compensation (instant inserts).
The consequence of the lack of order in the DDP protocol is the decision to use entirely random ids. That is not to say you can't use your own _ids.
while there is a risk of a collision with this strategy it is minimal on the order of [number of docs in your collection]/[55^17] * 100 % or nearly impossible. In the event this occurs the client will temporarily insert it and cancel it once the server confirms the error with a Mongo Duplicate Key error.
Also when it comes to security with the other answer. It is not too much of an issue if the _id of the user is known. It is not possible to log in without a valid hashed login token or retrieve any information with it. This applies to the user collection only of course. If you have your own collection an easily guessable URL containing an id as a reference without publish method checks on the eligibility to read the data is a risk the high entropy random ids generated by Meteor can mitigate.
As long as they are unique it should be ok to use your own ids.
I am not an expert, but I suppose Mongo needs a unique ID so when it updates the document, it in fact creates a new version of the document of that same ID.
The real question is - I too whish to know - if we can change the ID without screwing Mongo mechanism and reliability, or we need to create a secondary attribute? (It can make a smaller index too I suppose)?
But me too, I can imagine that security wise, it is better if document IDs are difficult to guess, especially user IDs! Otherwise, could it be easy or possible to fake a user, knowing the ID? Anybody, correct me if I am wrong.
I don't think it's possible and desirable to change ID from Mongo.
But you can easily create a autoincrement ID with http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
I have created a package that does just that and that is configurable.
https://atmospherejs.com/stivaugoin/fluid-refno
var refNo = generateRefNo({
name: 'invoices', // default: 'counter'
prefix: 'I-', // default: ''
size: 5, // default: 5
filling: '0' // default: '0'
});
console.log(refNo); // output: "I-00001"
you now can use refNo to add in your document on Insert
maybe it will help you