I am creating an application using MeteorJS that allows users to create items (e.g, text, images) and to collaboratively spatially organize these on a canvas. Multiple canvases can be created. In the future, items may be reused in or copied between (I am unsure yet) multiple canvases. I have never designed a collaborative (or even database driven) application before.
I could not find the possibility to create nested MeteorJS collections, and I am unsure about the (dis)advantages (e.g. considering scalability, speed) of using multiple collections vs. using an array of objects inside a collection, so I wonder what a good design pattern would be:
A:
Collection Canvases {
Canvas {
Array Items;
}
Canvas {
Array Items;
}
}
B:
Collection Items {
Item {
_id
}
Item {
_id
}
}
Collection Canvases {
Canvas {
Array ItemIDs;
}
Canvas {
Array ItemIDs;
}
}
Or perhaps something different?
Since Meteor "identifies changes based on the fields of the MongoDB document. But ... does not support nested fields and arrays", I'd use some data structure like you suggested in your proposal B: two collections. This ensures that only new/ updated items get pushed out to clients and not all items of a canvas.
Then make the relation between Canvas and Items as saimeunt pointed out in his comment above: Canvas{_id:"xxx"} Item{_id:"xxx",canvasId:"xxx"}. (I'm using a similar approach in my project minutocash which works fine.)
Furthermore, you can publish all related items of a canvas with the publish-with-relations package as David Weldon pointed out in this answer to a question of mine, about an issue which you might run into later with this data structure.
Related
What is a clean/efficient method for storing the directory Hierarchy/tree in a Key-Value database (in my case MongoDB but any of them)?
For example a tree structure
- Cars
+ Audi
+ BMW
- M5
+ Ford
- Color
+ Red
- Apple
- Cherry
+ Purple
- Funny
The method I am using now, each object links to it's parent
{
dir: "red"
parent-dir: "color"
}
This makes it very efficient/fast to insert and reorder any aspect of the tree (for example if I want to move Red and all it's children to the Cars directory).
But this method sucks when I want to all subdirectories and their children for a given directory recursively. To make it efficient to parse I can have a structure for example
{
dir: "red"
children: "audi, bmw, ford"
}
{
dir: "bmw"
children: "m5"
}
But if I want to modify the tree, a whole bunch of objects need to touched and modified.
Are there any other methods to storing a directory structure in a KV store?
The method you currently use now is called adjacency list model.
Another model to store hierarchical data in a (relational) database is the nested set model. Its implementation in SQL databases is well known. Also see this article for the modified preorder tree traversal algorithm.
A very simple method: you could store a path per object - with those it should be easy to query trees in NOSQL databases:
{ path: "Color", ... }
{ path: "Color.Red", ... }
{ path: "Color.Red.Apple", ... }
{ path: "Color.Red.Cherry", ... }
When nodes will be removed or renamed some paths must be updated. But in general, this method looks promising. You just have to reserve a special character as separator. The storage space overhead should be negligible.
edit: this method is called materialized path
Finally, here is a comparison of different methods for hierarchical data in NOSQL databases.
I don't have a huge amount of NOSQL experience, so this isn't a definitive answer, but here's how I'd approach it:
I would likely use your first approach, where you have:
{
dir: 'dir_name',
parent_dir: 'parent_dir_name'
}
And then set up a map-reduce to quickly query the children of a directory. MongoDB's map-reduce functionality is still only available in the development branch and I haven't worked with it yet, but in CouchDB (and I assume, with a few modification, in MongoDB) you could do something like:
map:
function(doc) {
emit( doc.parent_dir, doc.dir );
}
reduce:
function(key, values) {
return( values );
}
Which would give you the list of sub-directories for each parent directory.
I suggest storing a heap to the the id's of the data items.
I think this is the best plan. If you need lots and lots of stuff any heap element could be an index to another heap.
eg
{ "id:xxx", "id:yyy", "sub-heap-id:zzz"....}
If this is not clear post a comment and I will explain more when I get home.
Make an index!
http://www.mongodb.org/display/DOCS/Indexes
I need to update a field in a nested object with a dynamic key.
the path could look like this: level1.level2.DYNAMIC_KEY : updatedValue
The update-method deletes everything else on level1 instead of only updating the field in the nested object. The update() acts more like a set(). What am I doing wrong?
I tried the following already:
I read the documentation https://firebase.google.com/docs/firestore/manage-data/add-data#update-data
but that way it is a) static and b) still deletes the other fields.
Update fields in nested objects
If your document contains nested objects, you can use "dot notation" to reference nested fields within the document when you call update()
This would be static and result in
update({
'level1.level2.STATIC_KEY' : 'updatedValue'
});
Then I found this answer https://stackoverflow.com/a/47296152/5552695
which helped me to make the updatepath dynamic.
The desired solution after this could look like
field[`level1.level2.${DYNAMIC_KEY}`] = updateValue;
update(field);
But still: it'll delete the other fields in this path.
UPDATE:
The Structure of my Doc is as follows:
So inside this structure i want to update only complexArray > 0 > innerObject > age
Writing the above path into the update() method will delete everything else on the complexArray-level.
A simple update on first-level-fields works fine and lets the other first-level-fields untouched.
Is it possible, that firestore functions like update() can only act on the lowest field-level on an document. And as soon as i put complex objects into an document its not possible to select such inner fields?
I know there would be the solution to extract those "complex" objects into separate collections + documents and put these into my current lowest document level. I think this would be a more accurate way to stick to the FireStore principles. But on Application side it is easier to work with complex object than to always dig deeper in firestore collection + document structure.
So my current solution is to send the whole complex object into the update() method even though I just changed only one field on application side.
Have you tried using the { merge: true } option in your request?
db
.collection("myCollection")
.doc("myDoc")
.set(
{
level1: { level2: { myField: "myValue" } }
},
{ merge: true }
)
In discover meteor,
posts: function() {
return Posts.find();
}
is used, not this:
posts: function() {
return Posts.find().fetch();
}
I tried function below, it also works, and can realtime update.
What is cursor exactly? And What is the different of above two functions?
The Meteor cursor is like a lazy version of the array of documents.
It's designed to iterate through the results of a query while not actually loading each of the documents until they are actually requested or the cursor is at the position containing the document.
Its better to think of the results of the query as a book. If you use .fetch() all the pages are printed even though you're not reading them.
The cursor prints pages as you're reading them.
Additionally the cursor has several enhancements with regards to Blaze. Content is rendered less often as minute details in a document's change are able to change the DOM section on its own, without recreating an entire object. It's easier for Blaze to interact with the cursor than an array of Javascript objects.
Additionally a cursor can be observed, a plain array of Javascript object's can't
TLDR; Cursor is just like an array of objects, but designed to be more efficient & is slightly more extended with features.
I have some kind of UIComponent grouping which may look something like this with the classes "Group" and "Element".
Groups can have children and children may be elements or groups again, basically similar to a file system or the str+g group function in several graphics programs. The simplest form of a group is a group with only children which are also the most low level groups in the tree.
Edit:
The display hierarchy is already existant, i try to persist it to xml.
Group
- element
- Group
- element
- Group
-element
-element
- element
- element
I want to rebuild this structure in an xml-document for persistence.
I know how to build an xml document in Flex but not how to (recursively) traverse this n-tree correctly.
Update:
For getting only the child nodes one could make use of the following algorithm (pseudo code). But somehow i don't understand how to create the xml from this.
walkTree(group) {
children = node.getChildren
if(children != null) {
for(int i=0; i<children.length; i++) {
if(children[i].isGroup()) {
walkTree(group[i]);
} else {
trace(child);
}
}
}
}
As a starter, I'd suggest this : http://www.sephiroth.it/tutorials/flashPHP/E4X/
So, basically, what you are looking for seems like E4X in Actionscript 3
My suggestion would be to have your structure be data driven from the start, so the XML controls the draw of the screen. Then you would have the XML and wouldn't need to make it.
However, if you really want to do this, you'll need to loop through all the children at each level and add some sort of node that describes the child. However, if you don't have a view that is data driven, I don't see what good this will do you (and if you do, and it's not XML, you're better off writing data export from the data side, not the view side).
I am having a problem with the speed of accessing an association property with a large number of records.
I have an XAF app with a parent class called MyParent.
There are 230 records in MyParent.
MyParent has a child class called MyChild.
There are 49,000 records in MyChild.
I have an association defined between MyParent and MyChild in the standard way:
In MyChild:
// MyChild (many) and MyParent (one)
[Association("MyChild-MyParent")]
public MyParent MyParent;
And in MyParent:
[Association("MyChild-MyParent", typeof(MyChild))]
public XPCollection<MyCHild> MyCHildren
{
get { return GetCollection<MyCHild>("MyCHildren"); }
}
There's a specific MyParent record called MyParent1.
For MyParent1, there are 630 MyChild records.
I have a DetailView for a class called MyUI.
The user chooses an item in one drop-down in the MyUI DetailView, and my code has to fill another drop-down with MyChild objects.
The user chooses MyParent1 in the first drop-down.
I created a property in MyUI to return the collection of MyChild objects for the selected value in the first drop-down.
Here is the code for the property:
[NonPersistent]
public XPCollection<MyChild> DisplayedValues
{
get
{
Session theSession;
MyParent theParentValue;
XPCollection<MyCHild> theChildren;
theParentValue = this.DropDownOne;
// get the parent value
if theValue == null)
{
// if none
return null;
// return null
}
theChildren = theParentValue.MyChildren;
// get the child values for the parent
return theChildren;
// return it
}
I marked the DisplayedValues property as NonPersistent because it is only needed for the UI of the DetailVIew. I don't think that persisting it will speed up the creation of the collection the first time, and after it's used to fill the drop-down, I don't need it, so I don't want to spend time storing it.
The problem is that it takes 45 seconds to call theParentValue = this.DropDownOne.
Specs:
Vista Business
8 GB of RAM
2.33 GHz E6550 processor
SQL Server Express 2005
This is too long for users to wait for one of many drop-downs in the DetailView.
I took the time to sketch out the business case because I have two questions:
How can I make the associated values load faster?
Is there another (simple) way to program the drop-downs and DetailView that runs much faster?
Yes, you can say that 630 is too many items to display in a drop-down, but this code is taking so long I suspect that the speed is proportional to the 49,000 and not to the 630. 100 items in the drop-down would not be too many for my app.
I need quite a few of these drop-downs in my app, so it's not appropriate to force the user to enter more complicated filtering criteria for each one. The user needs to pick one value and see the related values.
I would understand if finding a large number of records was slow, but finding a few hundred shouldn't take that long.
Firstly you are right to be sceptical that this operation should take this long, XPO on read operations should add only between 30 - 70% overhead, and on this tiny amount of data we should be talking milli-seconds not seconds.
Some general perf tips are available in the DevExpress forums, and centre around object caching, lazy vs deep loads etc, but I think in your case the issue is something else, unfortunately its very hard to second guess whats going on from your question, only to say, its highly unlikely to be a problem with XPO much more likely to be something else, I would be inclined to look at your session creation (this also creates your object cache) and SQL connection code (the IDataStore stuff), Connections are often slow if hosts cannot not be resolved cleanly and if you are not pooling / re-using connections this problem can be exacerbated.
I'm unsure why you would be doing it the way you are. If you've created an association like this:
public class A : XPObject
{
[Association("a<b", typeof(b))]
public XPCollection<b> bs { get { GetCollection("bs"); } }
}
public class B : XPObject
{
[Association("a<b") Persistent("Aid")]
public A a { get; set; }
}
then when you want to populate a dropdown (like a lookupEdit control)
A myA = GetSomeParticularA();
lupAsBs.Properties.DataSource = myA.Bs;
lupAsBs.Properties.DisplayMember = "WhateverPropertyName";
You don't have to load A's children, XPO will load them as they're needed, and there's no session management necessary for this at all.
Thanks for the answer. I created a separate solution and was able to get good performance, as you suggest.
My SQL connection is OK and works with other features in the app.
Given that I'm using XAF and not doing anything extra/fancy, aren't my sessions managed by XAF?
The session I use is read from the DetailView.
I'm not sure about your case, just want to share some my experiences with XAF.
The first time you click on a dropdown (lookup list) control (in a detail view), there will be two queries sent to the database to populate the list. In my tests, sometimes entire object is loaded into the source collection, not just ID and Name properties as we thought so depends on your objects you may want to use lighter ones for lists. You can also turn on Server Mode of the list then only 128 objects are loaded each time.