Hi all I am developing an application on QT, and I need to run Python scripts from my application. I am trying to allocate some memory using sipMalloc function. Which is provided in python-SIP. I am using PyQt 4 and python v2.7.
I am importing sip.so im my script thich is present in /usr/lib/python2.7/dist-packages/
NameError: name 'sipMalloc' is not defined
if there is any other alternative to allocate memory in python script w.r.t. QT Application plz share. Thanks
******SOLVED******
I dropped the idea of using SIP. Instead I am using SWIG with python for creating library of exported functions. And SWIG also provides option to allocate memory using malloc, which can be used directly in Python Scripts. Thanks for the help.
One more problem I faces with SIP was exporting existing structures. SIP was not able to recognize when I used typedef and bit fields in the structures. But with SWIG, I exported all the structures by adding in .i file to the python scripts.
Related
I am currently trying to incorporate a C++ library into an R package to expose its functionality. The original C++ library used Cmake, where multiple variables were set. I am trying to avoid using Cmake and am trying to make my way through writing a new Makevars file. However, one of the variables defined through the original Cmake setup was the available physical memory, through cmake_host_system_information.
Is there any recommended way to retrieve the available physical memory to then define it in Makevars?
We have successfully gone through all the SparkR tutorials about setting it up and running basic programs in RStudio on an EC2 instance.
What we can't figure out now is how to then create a project with SparkR as a dependency, compile/jar it, and run any of the various R programs within it.
We're coming from Scala and Java, so we may be coming at this with the wrong mindset. Is this even possible in R or is it done differently that Java's build files and jars or do you just have to run each R script individually without a packaged jar?
do you just have to run each R script individually without a packaged jar?
More or less. While you can create a R package(-s) to store reusable parts of your code (see for example devtools::create or R packages) and optionally distribute it over the cluster (since current public API is limited to high level interactions with JVM backend it shouldn't be required), what you pass to spark-submit is simply a single R script which:
creates a SparkContext - SparkR::sparkR.init
creates a SQLContext / HiveContexts - SparkR::sparkRSQL.init / SparkR::sparkRHive.init
executes the rest of your code
stops SparkContext - SparkR::sparkR.stop
Assuming that external dependencies are present on the workers, missing packages can installed on the runtime using if not require pattern, for example:
if(!require("some_package")) install.packages("some_package")
or
if(!require("some_package")) devtools::install_github("some_user/some_package")
As part of a transition from MATLAB to R, I am trying to figure out how to read TDMS files created with National Instruments LabVIEW using R. TDMS is a fairly complex binary file format (http://www.ni.com/white-paper/5696/en/).
Add-ons exist for excel and open-office (http://www.ni.com/white-paper/3727/en/), and I could make something in LabVIEW to make the conversion, but I am looking for a solution that would let me read the TDMS files directly into R. This would allow us to test out the use of R for certain data processing requirements without changing what we do earlier in the data acquisition process. Having a simple process would also reduce the barriers to others trying out R for this purpose.
Does anyone have any experience with reading TDMS files directly into R, that they could share?
This is far from supporting all TDMS specifications but I started a port of a python npTDMS package into R here https://github.com/msuefishlab/tdmsreader and it has been tested out in the context of a shiny app here
You don't say if you need to automate the reading of these files using R, or just convert the data manually. I'm assuming you or your colleagues don't have any access to LabVIEW yourselves otherwise you could just create a LabVIEW tool to do the conversion (and build it as a standalone application or DLL, if you have the professional development system or app builder - you could run the built app from your R code by passing parameters on a command line).
The document on your first link refers to (a) add-ins for OpenOffice Calc and for Excel, which should work for a manual conversion and which you might be able to automate using those programs' respective macro languages, and (b) a C DLL for reading TDMS - would it be possible for you to use one of those?
I would like to read and write from a single netCDF-4 file from R. The file will be accessed by many nodes processes at the same time, (~100's for development, ~1000's for production).
What is the best way to access parallel I/O features in netCDF from within R?
What I have found:
It appears from the Unidata page that all I need to do is compile with parallel features enabled (--enable-parallel). Is that really all I need to do?
I can not find any mention of parallel io in the ncdf4 package description.
Given that I/O is the bottleneck in my computation, any hints on how to optimize my computation - are there circumstances when it would be better to write to multiple files during computation (e.g., locally), and combine the files later (e.g. using nco)?
Information related to using parallel I/O with Unidata NetCDF may be found here:
https://www.unidata.ucar.edu/software/netcdf/docs/parallel_io.html
The --enable-parallel flag is no longer necessary when configuring netCDF; It will check the documentation and update it if need be. The flag is necessary when building the hdf5 library, however.
In order to use parallel I/O with netCDF-4, you need to make sure that it was built against an hdf5 library with parallel I/O enabled. At configure time, netCDF will query the hdf5 library to see whether or not the parallel I/O symbols are present.
If they are, parallel I/O for netCDF-4 is assumed.
If they are not, parallel I/O for netCDF-4 files is turned off.
If you are installing the netCDF library yourself, you can specify the --enable-parallel-tests flag when configuring; when you run make check, parallel tests will be run. You can also scan the output in config.log to see if parallel I/O functionality was found in the hdf5 library; there should be a message notifying you whether or not it was enabled.
Note that there are some limitations to Parallel I/O with netCDF-4, specifically:
NetCDF-4 provides access to HDF5 parallel I/O features for netCDF-4/HDF5 files. NetCDF classic and 64-bit offset format may not be opened or created for use with parallel I/O. (They may be opened and created, but parallel I/O is not available.)
Assuming that the underlying netCDF library has parallel I/O enabled, and you are operating on the correct type of file, the standard API call invoked by ncdf4 should leverage parallel I/O automatically.
There is one more R package dedicated for parallel handling of NetCDF files which is called pbdNCDF4.
This solution is based on standard ncdf4 package, so the syntax is very similar to the "traditional" approach. Further information available on CRAN: https://cran.r-project.org/web/packages/pbdNCDF4/vignettes/pbdNCDF4-guide.pdf
Ward gave a fine answer. I wanted to add that there is another way to get parallel I/O features out of Unidata NetCDF-4.
NetCDF-4 has an architecture that separates the API from the back end storage implementation. Commonly, that's the NetCDF API on an HDF5 back end. But, here's the neat thing: you can also have the NetCDF API on the Northwestern/Argonne "Parallel-NetCDF" (http://cucis.ece.northwestern.edu/projects/PnetCDF/ and http://www.mcs.anl.gov/parallel-netcdf) back end.
This approach will give you a parallel I/O method to classic and 64-bit offset formatted datasets.
Both Ward and Rob gave fine answers! ;-)
But there is yet another way to get parallel I/O on classic and 64-bit offset files, through the standard netCDF API.
When netCDF is built with --enable-pnetcdf, then the parallel-netcdf library is used, behind the scenes, to perform parallel I/O on classic, 64-bit offset, and CDF5 (though I have not tested that last format with parallel I/O).
When opening the file, use the NC_PNETCDF flag to the mode to indicate that you want to use parallel I/O for that file.
Is there a tool for deploying things into exist? if I've got a bundle of, say, schemas and XQuery stored procedures? Is there a way of, say, bundling those into a zip or tar file and uploading them or deploying them into eXist?
Alternatively what is the best way of storing these things in a version controlled way (in a git repo, say) and deploying them to the eXist server? Ideally, it'd be nice to be able to have a simple script in a scripting language so you can simply call "deploy.py" or whatever and it'd take everything from the repository and load it into the XML database.
The EXpath packaging system specifies a format for generating a ZIP file with XQuery procedures (and other content) and deploying it into multiple XQuery databases.
See the specification. You should be able to use the Python zipfile module to generate these if you're inclined to use Python (though personally, I do so from a makefile).
Unfortunately, the process for checking currently installed package versions to upgrade if necessary is not standardized; I have a solution for BaseX, but nothing for eXist immediately at hand. However, eXist's implementation is well-documented, and you should have little trouble working with it.