I am writing a simulation model in R to track the behavior of a set of interacting agents. For my own sanity, I give each agent its own S4 object, in which I store its trajectory and other parameters. Currently, I pass an object to a function, do some operations, and pass the object back. For example,
#Create a new class and a sample object
setClass("example", slots = list(N="numeric"), prototype = list(N=0))
agentA<-new("example")
#Define a function to change the value in the N slot
add_one<-function(agent){
agent#N<-agent#N + 1
agent
}
#Call the function.
agentA<-add_one(agentA)
I know this works and it's really important to me that the structure is modular and easy to debug, but I'm wondering about the overhead of passing the agent object back and forth. Most of the objects contain arrays with a few thousand numbers, and they will get passed back and forth thousands of times. Is there a more efficient way to do it, or is this close enough to best practice?
I'm not really clear on how many times the object gets copied, versus when only a pointer is passed.
Related
I am experiencing (very) slow page load times that increase proportionately to the number of active users on the system. I have a hunch that this is related to a custom defined thread object:
define stageStoreCache => thread {
parent map
public oncreate() => ..oncreate()
}
This stageStoreCache object simply mimics the behavior of a map whose data available across the entire instance.
Many threads are reading it and very few threads are writing to it. Is this a poorly conceived solution to having a large map of data available across the instance? It's a fairly large map of maps that when exported to map->asstring can exceed 5MB. The objective is to prevent translating data stored as JSON in the database to Lasso types on the fly.
It seems that the large size of the stageStoreCache is not what causes problems. It seems to really be the number of concurrent users on the system.
Thanks for any insight you can offer.
You said that this holds a map of maps and is rather large. If those sub-maps are large, it is possible that the way you are accessing the data is causing the issue. Here's what I mean, if you are doing something like this:
// Potential problem as it copies the sub-map each time
stageStoreCache->find('sub-map')->find('data')
stageStoreCache->find('sub-map')->find('other')
The problem comes in that each time stageStoreCache->find('sub-map') is called it actually has to copy all the map data it finds for "sub-map" out of the thread object and into the thread requesting that data. If those sub-maps are large, this takes time. A better approach would be to do this once and stash it in a local variable:
// Better Approach
local(cache) = stageStoreCache->find('sub-map')
#cache->find('data')
#cache->find('other')
This at least only has to copy the "sub-map" over once. Another approach that might be better (only testing could tell) would be to refactor your code so that each call to stageStoreCache drills down to the data you actually want, and have just that small amount of data copied over.
// Might even be better as it just copies the values you want
stageStoreCache->drill('sub-map', 'data')
stageStoreCache->drill('sub-map', 'other')
Ultimately, I would love for Lasso to improve thread objects so that they never blocked for reads. (I had thought this had been submitted as a feature request, but I'm not finding it on Rhinotrac.) Until that happens, if none of my suggestions help then you may need to investigate using something else to cache this data in such as memcached.
Testing is the only way to tell for sure. But I would go a long way to avoid having a thread object that contains some 5 MB of data.
Take this snippet from the Lasso guide into consideration:
"all parameter values given to a thread object method are copied, as well as any return value of a thread object method"
http://www.lassoguide.com/language/threading.html
Meaning that one of the key features that makes Lasso 9 so fast, the extensive use of reference data, is lost.
Each time you have a call for stageStoreCache all the data it contains will first be copied into the thread that asks for it. That is an awful lot of copying.
I have found that having settings and site wide data contained in smallest possible chunks is convenient and fast. And also, to only actually set it up when it is called for. Unlike the old approach that had a config file that was included on every call, setting up a bunch of variables where the majority maybe never got used on that particular call. Here's a Ke trick that I'm using instead. Consider this:
define mysetting1 => var(__mysetting1) || $__mysetting1 := 'Setting 1 value'
define mysetting2 => var(__mysetting2) || $__mysetting2 := 'Setting 2 value'
define mysetting3 => var(__mysetting3) || $__mysetting3 := 'Setting 3 value'
Have this is a file that is read at startup, either in a LassoApp that's initiated or a file in the startup folder.
These settings can then be called like this:
code blabla
mysetting2
more code blabla
mysetting1
mysetting2
With the beauty that, in this case, there is no wasted processing to initiate mysetting3, since it's not called for. And that mysetting2 is called for several times but is still only initiated once.
This technique can be used for simple things like the above, but also to initiate complex types or methods. Like session management, calling post or get params etc.
PySide and PyQt employ Qt signal-slot mechanism with which we can connect any/multiple signals to any/multiple slots as far as the trasmistted data types match.
The signalling object has some knowledge of the receiving slots, e.g. knows their number via method receivers or the signal can disconnect from the receiving slots via its disconnect method.
My problem relates to the oposite direction - e.g. does a slot know to which signals it is connected? Can a slot disconnect from the signals?
UPDATE: So why I am asking this - I have an object that performs some calculation. The calculation is defined by a user editable Python expression. The expression is parsed and necessary data sources are identified from this... The calculation object (acts as a slot) then connects to these data sources (they act as signals) and once the data sources produce/update a value, this fact is signalled to the slot and the expression is reevaluated. And when the expression is changed by a user, it needs to be parsed again and disconnected from the existing signals (i.e. data sources) and connect to new data sources. You can imagine it is something like a formula in Excel that is connected to other cells.
I know there are a few ways to work around this, e.g. keeping track of connections manually (well, this is extra work) or deleting the expression object and creating a new one everytime it is changed (seems not good enough, because user might want to trace back the calculation data sources and this will not help). But I was curious if this can be solved purely using simple signal-slot mechanism. In other words, I am not interested in any workarounds... I know of them and will use them I signals-slots will not help here. :)
The approach you propose forces a very close relationship between the concrete data widgets and the calculation engine. You mingle UI with the calculations. This makes it much harder than it needs to be.
What you could try instead is the model-view approach. The model would be a simple implementation of QAbstractTableModel. The view would be the individual data-entry widgets mapped to the model's cells using QDataWidgetMapper. The calculation engine would access only the model, completely unaware of how the model is modified by the widgets. This make life easier.
The calculation object can connect to the model using just one dataChanged signal and it will be notified of changes to any of the variables. You can easily pass both the value and the variable name by having two columns in the table.
The implementation of the model can be very simple, you can have a list of strings for the variable names in the first column, and a list of variants for the second column. The model must correctly emit the dataChanged signal whenever setData is called, of course.
I need to construct a priority queue in R where i will put the ordered seed objects (or the index of the objects) for the OPTICS clustering algorithm.
One possibility is to implement it with heap with the array representation, and pass the heap array in each insert and decrease key call, and return the changed array and reassign it in the calling function. In which case, the reassign operation will make the performance very poor and every time one insert or decrease operation is executed the entire array needs to be copied twice, once for calling, and another once for returning and reassigning.
Another possibility is to code the heap operations inside the function instead of calling it. This will result in code repetition and cumbersome code.
Is there any pointer like access as we do in C
Can i declare user defined functions in the S3 or S4 classes in R ? In the the case i think the call to these functions still requires the same reassignment after returning (not like C++/Java classes, operates on the object (am i right?) )
Is there any builtin way with which i can insert and extract an object in a queue in O(log(n)) time in R?
Is there any other way with which i can achieve the goal, that is maintain a priority based insertion and removal of the seeds depending on the reachability distance of an object in the OPTICS algorithm, except explicitly sorting after each insertion.
R5 classes
define mutable objects, and very similar to Java classes:
they should allow you to avoid the copies when the object is modified.
Note that you do not just need a priority queue.
It actually needs to support efficient updates, too. A simple heap is not sufficient, you need to synchronize a hashmap to find objects efficiently for updating their values. Then you need to repair the heap at the changed position.
In my code I define a new MPI user-defined data type.
I was wondering if the MPI_Barrier function must follow the MPI_Commit or must be placed at some point where the first use of the new data type appears so that all the processes acknowledge and agree on the defintion of the new data type.
Thanks.
No - there's no communication within the MPI_Type commands, they're completely local. In particular, processes don't necessarily have to agree on the definition of a new type.
If rank 1 sends a new data type to rank 0, all they have to agree on is the amount of data, not the layout of the type. For instance, imagine rank 1 was sending all of it's (say, 2d) local array to rank 0 - it might just choose to send an MPI_Type_contiguous of NX*NY floats. But rank 0 might be receiving this into a larger global array; it might choose to receive it into a Subarray type of the global type. Even if those data types had the same names, they can describe different final layouts in memory, as long as the total amount of data is the same.
MPI datatypes are the private business of the process that creates them. They do not need to match, in fact it is possible and perfectly legal for a receiving process to use a type map that differs from that of the sending process (as long as it doesn't lead to memory corruption of course). As such, there is no synchornization whatsoever when using Define or Commit.
I'm trying to integrate two Fortran 9x codes which contain data arrays with opposite array ordering. One code (I'll call it the old code) has an established library of subroutines and I am trying to take advantage of these with the other (new) code as efficiently as possible (i.e. not having to create temporary arrays just to reorder an array and pass it to a subroutine and then have to replace the old array with the new reordered result). For example,
Old code:
oldarray(1:n,1) -> variable 1 for n elements
oldarray(1:n,2) -> variable 2 for n elements
.. and so on
new code:
newarray(1,1:n) -> variable 1 for n elements
newarray(1,1:n) -> variable 2 for n elements
.. and so on
The variable indices do not necessarily relate between the two codes. If I only need one variable to pass to a procedure, I just pass newarray(1,1:n) and the procedure doesn't know the difference. However, if a procedure from the old code requires variables 1-6 of oldarray which might correspond to variables 2,6,8,1,4,3 (I just picked arbitrary numbers) of newarray, is it possible to create a pointer that I could pass to the procedure?
On a simpler side, would it be possible to just create pointer for the tranpose of the new array? As an example, pointer(1000,6) points to newarray(6,1000).
Note: It is not possible to rewrite the new code to use the same array ordering because both codes use an array ordering that best suits its loop structures which cannot be changed.
Also, I have very little experience with pointers. I know I can create a derived datatype which consists of an array of pointers but I don't think I would be able to pass that to a procedure in the manner required (I could be wrong as I also have very little experience with derived datatypes). The reference book I have (Fortran 95/2003 for Scientists and Engineers) only explores advanced applications of pointers in terms of linked lists and trees. I have also found little Fortran pointer information outside of what is covered in this book on the internet.
Thank you for your help.
I think the answer is no, you can't do this, and it wouldn't help anyway.
You can do all sorts of super-cool things with array pointers, with strides across arrays, etc, but I don't see on the face of it how you can change the order of the data.
So I could be wrong on this and it is possible, but then the question is: how would it help you? Presumably you want to user pointers to re-arrange the data without copying; but when you're passing such a thing around, the compiler is allowed to do copy-in, copy-out; eg, create a temporary array, copy the data in, pass it to the subroutine, and copy the data out upon return. And in fact that would almost certainly be the right thing to do in this case, performance-wise; that way the old code could be accessing memory in the fast order, and the transpose-copy could be done in a fast way, as well.
So I suspect the right way to treat this problem is to do the copy-in/copy-out approach yourself explicitly.