I have got the following setup:
----------------------------
| C++ Application |
----------------------------
|
dlopen()s
|
v
---------------------------- -------------------
| C++ Plugin with embedded |--links-->| libpython3.8.so |
| Python interpreter | -------------------
----------------------------
|
runs
|
v
----------------------------
| Python script |
----------------------------
|
imports
|
v
----------------------------
| Python C extension |
| module (e.g., numpy) |
----------------------------
Without any further measures, the import of the Python C extension module from within the Python script fails for the reasons detailed in this SO question, i.e. that the Python extension does not "see" the symbols in libpython.
When calling dlopen("libpython3.8.so", RTLD_LAZY | RTLD_GLOBAL) in the C++ plugin as suggested in answers to the aforementioned question, it works. Yet, intermingling build-time dynamic linking (i.e., specifing -lpython3.8) and "manual" dlopen() calls at run-time somehow does not seem right to me.
Therefore, I am looking for a more stringent solution. My first idea was to skip compile-time linking altogether and fully rely on dlopen() to load libpython. As an extra benefit, this would have allowed to make the Python installation user-selectable, by providing an option to specify the path to libpython in the config of my plugin. However, the approach does not work as even making use of RTLD_LAZY, variables in libpython have to be resolvable already at the moment when the C++ plugin is loaded by the application.
Vice versa, it could also be desirable to rely on build-time linking only. To my understanding, this would require a link-time option instructing to load libpython with the same symbol visibility as achieved by dlopen() with RTLD_GLOBAL. However, I do not know of any such option and was so far unable to find one.
My questions are:
Can you think of any better or more stringent solution than both linking libpython at build-time and dlopen()ing at runtime? It would be an extra bonus if an approach would allow to make libpython selectable via plugin config (independent of the LD_LIBRARY_PATH environment variable).
If you cannot think of any better solution either, what are the risks related to mixing build-time linking and dlopen()? In particular,
what would happen if - for any reason - another version of libpython was loaded with dlopen() than the one that was linked against?
are -lpython3.8 and dlopen("libpython3.8.so", ...) equivalent in the sense that they always resolve to the same library file?
Related
I have a shiny application which uses like 4 functions. I would like to test these functions but it's not a package. How am i supposed to structure my code ? and execute these tests without devtools ?
You can execute tests with testthat::test_dir() or testthat::test_file(). Neither relies on the code being in a package, or using devtools, just the testthat package.
There are few requirements on how to structure your code.
If it were me, I would create a tests directory and add my test scripts under there, which would look something like:
|- my_shiny_app
| |- app.R
| |- tests
| |- test_foo.R
| |- test_bar.R
Then you can run your tests with test_dir('tests'), assuming you're in the my_shiny_app directory.
Your test scripts will have they same structure they have for packages but you'd replace the library() call with source() referencing the file where your functions are defined.
If you have few functions without a package structure, it is better to write single test files manually (so with some simple if/error catching system) that you call with Rscript test_file1.R.
If you start to use the package format instead (which would be advisable for further 'safe' developing) and you still do not want to use testthat, I advise you to follow this blog post: here
I have a structure of dirs with a couple of projects. Some of the projects depend on others ones which may in turn depend on others.
Dir
+-Proj1
| +-Debug
| | makefile
| +-Release
| makefile
| <sources>
+-Proj2
| +-Debug
| | makefile
| +-Release
| makefile
| <sources>
...
The makefiles automatically generated (by the Eclipse CDT) so I could not change it because in case of some changes in projects my changes will go away.
Now, every project must be built in the corresponding Debug or Release dirs. Each of the makefiles has a relative refers to the sources that's why I need to build every project in the Debug or Release dirs.
The makefiles which generated by the Eclipse contain a possibility to include a user makefiles. I use this feature to cause the make build projects which others depend on earlier. Suppose the Proj1 depends on the Proj2. Then I added dependency on the ../../Proj2/<Debug or Release>/proj2.so for the result file of the Proj1 and added this rule for the Proj1:
$(USER_DEPS): # (*)
$(MAKE) -C "$(dir $#)" all
Here the USER_DEPS contains a list of a files which this project depends on. For example
USER_DEPS:=../../Proj2/Debug/proj2.so \
../../Proj3/Debug/proj3.so
And all of this works until I change something in the sources in the Proj2. Now if I run make for the Proj1 it see that the ../../Proj2/Debug/proj2.so is here but it does not see that it is not actual. There could be ideal simply to include the makefile of the Proj2 into Proj1 instead of writing the rule marked with the (*), but the makefile of the Proj2 contains a relative references to its sources. Ie if I will try to include them into Proj1 the make will try to find a Proj2's sources in the Proj1's dir.
Has someone any ideas?
In Question 2918898, users discussed how to avoid caching because
modules were changing, and solutions focused on reloading. My question is
somewhat different; I want to avoid caching in the first place.
My application runs on Un*x and lives in /usr/local. It imports a
module with some shared code used by this application and another.
It's normally run as an ordinary user, and Python doesn't cache the
module in that case, because it doesn't have write permission for that
system directory. All good so far.
However, I sometimes need to run the application as superuser, and then
it does have write permission and it does cache it, leaving unsightly
footprints in a system directory. Do not want.
So ... any way to tell CPython 3.2 (or later, I'm willing to upgrade)
not to cache the module? Or some other way to solve the problem?
Changing the directory permissions doesn't work; root can still write,
root is all-powerful.
I looked through PEP 3147 but didn't see a way to prevent caching.
I don't recall any way to import code other than import. I suppose I
could read a simple text file and exec it, but that seems inelegant
and bug-prone.
The run-as-root is accomplished by calling the program with sudo in a
shell script, and I can have the shell script delete the cache after the
run, but I'm hoping for something more elegant that doesn't change the
directory's last-modified timestamp.
Implemented solution, based on Wander Nauta's answer:
Since I run the executable as a plain filename, not as python executablename, I went with the environment variable. First, the
sudoers file needs to be changed to allow setting environment
variables:
tom ALL=(ALL) SETENV: NOPASSWD: /usr/local/bkup/bin/mkbkup
Then, the invocation needs to include the variable:
/usr/bin/sudo PYTHONDONTWRITEBYTECODE=true /usr/local/bkup/bin/mkbkup "$#"
You can start python with the -B command-line flag to prevent it from writing cached bytecode.
$ ls
bar.py foo.py
$ cat foo.py
import bar
$ python -B foo.py; ls
bar.py foo.py
$ python foo.py; ls
bar.py foo.py __pycache__
Setting the PYTHONDONTWRITEBYTECODE environment variable to a non-empty string or the sys.dont_write_bytecode to True will have the same effect.
Of course, I'd say that the benefits in this case (faster loading times for your app, for free) vastly outweigh the perceived unsightliness you were talking about - but if you really want to disable caching, here's how.
Source: man python
I have several test files in different folders (for different issues) and I want to use a separate folder that will contain all the Custom Libraries I use. All of the sub folders, including the custom libraries, will be in one master folder. How do I import the test library from the separated folder?
Here is the folder hierarchy:
Test Library
-Test Suite1
-test1.txt
-test2.txt
-Test Suite2
-test3.txt
-Custom Libraries
-customlibrary.py
Thank you.
There are many ways. For one, just use the path. For example:
*** Settings ***
| Library | ../Custom Libraries/customlibrary.py
Or, you can add Test Library/Custom Libraries to your PYTHONPATH variable and just use the library name itself:
*** Settings ***
| Library | customlibrary
Or, you can set a variable that defines the directory -- either in a variables table or from the command line:
*** Variables ***
| ${LIBRARIES} | Test Library/Custom Libraries
*** Settings ***
| Library | ${LIBRARIES}/customlibrary.py
This is all described in the robot framework user guide, under the section Using Test Libraries.
Build is a module which has been developed in order to build the PDG.
I wrote a script which uses this module Build but when i try to launch this script with:
frama-c -load-script test.ml
I get the mistake: Unbound module Build.
Is there a way to get access to this module. I need it in my project.
Build is an example but there are another modules like Sets which provides functions to read a PDG. However, others modules like PdgTypes don't make mistakes. If anybody could help me...
In my file test.ml,
let compute = Build.compute_pdg
....
let () = Db.Main.extend main
You can't do that. -load-script can only work for script that do not have any dependency outside of Frama-C (or Frama-C's own dependencies such as OCamlgraph). As suggested by Anne, if your code is contained in more than one file, you should write it as a plugin.
For a simple plugin, you basically only have to write a short Makefile in addition to your OCaml code. This Makefile will mainly contain the list of source files of your plugin and a few additional information (such as the plugin's name), as explained in the developer's manual, which contain a small tutorial.
Alternatively, if you have only two files, it should be possible to assemble them manually into a single module that can then be loaded by Frama-C. Supposing you have build.ml and test.ml, you could do (with the Sodium
version)
ocamlopt -I $(frama-c-config -print-libpath) -c build.ml
ocamlopt -I $(frama-c-config -print-libpath) -c test.ml
ocamlopt -shared -o script.cmxs build.cmx test.cmx
frama-c -load-module script.cmxs [other options] [files]
The modules you refer to, Build and Sets, are not considered as being part of the public user interface of Frama-C. Instead, they are internal to the plugin PDG. The modules of PDG you can access from user scripts are those in the directory src/pdgTypes: PdgIndex, PdgMarks and PdgTypes. Then, a second part of the API is available inside Db.Pdg (Db is in src/kernel/db.ml). In particular, most of the functions of the module Sets are re-exported there.
For the functions available inside Build, they have been deemed too low-level to be exported. If you really need to access it, you will have to copy the directory src/pdg and transform it into a plugin (with a new name, to avoid clashes).