I'm a new DM's user and I need to transfer data (pixels bright) between Digital Micrograph and R, for processing and modelling an image.
Specifically, I would need to extract the bright pixels from an original image, send it to R for processing, and return to DM for to represent the new image.
I would like to know if it is possible and how to do it from an script in DM.
A lot of thanks. Regards.
There is very little direct connection between DM (scripting) and the outside world, so the best solution is quite likely the following (DM-centric) route:
A script is started in DM, which does:
all the UI needed
extract the intensities etc.
save all required data in a suitable format on disc at specific path. (Raw data/text-data/...)
call an external application ( anything you can call from a command prompt, including .bat files) and waits until that command has finished
Have all your R code been written in a way that it can be called from a command prompt, potentially with command prompt parameters (i.e. a configuration file):
read data from specific path
process as required (without an UI, so do it 'silently')
save results on disc on specific path
close application
At this point, the script in DM continues, reading in the results (and potentially doing some clean-up of files on disc.)
So, in essence, the important thing is that your R-code can work as a "stand-alone" black-box executable fully controlled by command-line parameters.
The command you will need to launch an external application can be found in the help-documentation under "Utility Functions" and is LaunchExternalProcess. It has been introduced with GMS 2.3.1.
You might also want try using the commands ScrapCopy() and ScrapPasteNew()to copy an image (or image subarea) into the clipboard, but I am not sure how the data is handled there exactly.
Related
I am using the cloudy r aws.s3 code to pull down an RDS file from s3. I have my own custom R runtime in a lambda. aws.s3 has a handy method called s3readRDS("s3://pathtoyourfile"). This works well, but has a limitation in that it saves the RDS file to disk, then must read it back in using readRDS. This is fine for smaller files, but for larger files, there is a no-go as we have limited disk storage.
Right now, I'm kind of stuck with these largish data files and the ability to pull them out into a database is just not feasible at the moment due to the target group, so I'm trying to minimize our cost and maximize throughput and this is the last nail in that coffin.
According to the documentation:
"Some users may find the raw vector response format of \code{get_object} unfamiliar. The object will also carry attributes, including \dQuote{content-type}, which may be useful for deciding how to subsequently process the vector. Two common strategies are as follows. For text content types, running \code{\link[base]{charToRaw}} may be the most useful first step to make the response human-readable. Alternatively, converting the raw vector into a connection using \code{\link[base]{rawConnection}} may also be useful, as that can often then be passed to parsing functions just like a file connection would be. "
Based on this(and an example using load()) the below code looks like it should work but it does not.
foo <- readRDS(rawConnection(get_object("s3://whatever/foo.rds")))
Error in readRDS(rawConnection(get_object("s3://whatever/foo.rds", :
unknown input format
I can't seem the datastream right for readRDS or unserialize to make sense of it. I know the file is correct as using the save to disk/load from disk works fine. But I want to know how to make "foo" into an unserialized object without the save/load.
I have a very large simulink input file (*.csv) that is too big to the handled in a single node...
I am wondering if it is possible to not read the whole file once at the beginning of the simulation, but instead stream the data in real time as needed by the simulation.
My first thought was to implement a custom script in JAVA or C# (sender) that reads line by line the input csv file and STREAM the data to simulink via TCP . Simulink would receive the data using a TCP block receiver.
My questions are two:
Is my approach feasible?
Given the problem stated, what would
be your solution?
I suspect it would be easier to the run simulation using sequential chunks of the data, saving the model state at the end of each chunk, and starting the simulation from the state saved at the end of the previous chunk. The doc describing how to do this is Save and Restore Simulation State as SimState.
You might try writing an S-Function in C that opens your file and streams your data line by line. The easiest way to do this would be using the S-Function Builder block. You would nonetheless need to parse your file in C. (which, in the case of a CSV file, shouldn't be hard)
I have code that generates thumbnails from JPEGs. It pulls an image from S3 and then generates the thumbs.
One in about every 3000 files ends up looking like this. It happens in batches. The high res looks like this and they're all resized down to low res. It does not fail on resize. I can go to my S3 bucket and see that the original file is indeed intact.
I had this code written in Ruby and ported it all over to clojure hoping it would just fix my issue but it's still happening.
What would result in a JPEG that looks like this?
I'm using standard image copying code like so
(with-open [in (clojure.java.io/input-stream uri)
out (clojure.java.io/output-stream file)]
(clojure.java.io/copy in out))
Would there be any way to detect the transfer didn't go well in clojure? Imagemagick? Any other command line tool?
My guess is it is one of 2 possible issues (you know your code, so you can probably rule one out quickly):
You are running out of memory. If the whole batch of processing is happening at once, the first few are probably not being released until the whole process is completed.
You are running out of time. You may be reaching your maximum execution time for the script.
Implementing some logging as the batches are processed could tell you when the issue happens and what the overall state is at that moment.
I am using Lidgren networking library to create a real time multiplayer game.
What I am trying to do is, save all incoming packages (including all bytes) to a peer in a binary file. Later when I need to debug some weird behavior of networking, I can load this file and have it load all (or rebuild) the packages that it saved, sequentially. This way, I can find how the weird behavior occurred exactly.
My question is, how do I recreate this package when I load it from the file?
It is a NetIncomingMessage that I need to recreate, I assume, and so far I thought of either creating it anew, or sending an NetOutgoingMessage to self, so it hopefully has the same effect I want to achieve, if the first approach fails.
The way I solved this is by creating an interface (wrapper object) of the NetIncomingMessage which contains a data byte array among other data members, and than have a thread to fill a list of these objects based on the saved incoming time, which is requested and removed (dequeued) from another thread.
See https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
I'm using Adobe InDesign Server CC and I want to do datamerging. It does merging well, the only problem is that Indesgin Server is SO SLOW. Here is my code:
var source = File(app.scriptArgs.getValue("sourceIndd")); //.indd file
var destination = File(app.scriptArgs.getValue("destination"));
var sourceData = File(app.scriptArgs.getValue("sourceData")); //csv file with data be placed into placeholders
var resolution = app.scriptArgs.getValue("resolution");
var doc = app.open(source,OpenOptions.DEFAULT_VALUE);
doc.dataMergeProperties.selectDataSource(sourceData);
doc.dataMergeOptions.linkImages = true;
doc.dataMergeProperties.mergeRecords();
app.jpegExportPreferences.jpegQuality = JPEGOptionsQuality.low;
app.jpegExportPreferences.exportResolution =parseInt(resolution) ;
app.jpegExportPreferences.antiAlias =true;
app.documents.item(0).exportFile(ExportFormat.JPG, destination);
doc.close();
This code is used for preview generating so QUALITY IS NOT IMPORTANT. Does anybody know how I can speed this up? Or maybe there is another away to export previews?
Do you know which of those steps is slow? To find out, measure individual execution times and write them to a log file. $.hiresTimer provides the best way to measure times, it is reset to 0 after each access.
Your operating system may also have means to measure performance, for example OSX has ways to sample processes down to function calls through the Activity Viewer or the Instruments application provided with the developer tools.
There are so many things that can go wrong, so without a testing environment you can only guess. Is the server loaded with slow startup scripts and configured to restart after each job? Is the job initiated locally, or via network?
If it is really in the quoted script, the problem could be a slow shared volume or a defective local one, overflowing directories and so forth. For either the source document, the source data, the linked images, or the output location. The source document might carry an unnecessary history baggage which you can strip with an IDML roundtrip. Do you use bad fonts? There could be a problem with the actual images (super complex PDF, EPSF or Illustrator file). If it is the export command rather than the data merge, you can for example play around with resolutions and other export options, use down-sampled copies of images etc. As I wrote, just guesswork without a look.