Is there a way to avoid Google Performance Tools listing files as "??:?", that is, failing to locate which file contains the function it is reporting on? How can I work out which library contains the function being called?
$ env LD_PRELOAD="/usr/lib/libprofiler.so.0" \
CPUPROFILE=output.prof python script.py
$ google-pprof --text --files /usr/bin/python output.prof
Using local file /usr/bin/python.
Using local file output.prof.
Removing _L_unlock_13 from all stack traces.
Total: 433 samples
362 83.6% 83.6% 362 83.6% dtrsm_ ??:?
58 13.4% 97.0% 58 13.4% dgemm_ ??:?
1 0.2% 97.2% 1 0.2% PyDict_GetItem /.../Objects/dictobject.c
1 0.2% 97.5% 1 0.2% PyParser_AddToken /.../Parser/parser.c
...
I am aiming to be able to profile the C code in a python package that has many compiled C extension modules. In the toy example above, what would I do to track down where "dtrsm_" is defined? If there are multiple loaded libraries that contain functions with that same name, is there any way to tell which version is being called?
C/C++ won't compile if the same pre-processed sourcefile (e.g. with #includes expanded) contains duplicate definitions for the same symbol. (Note that in the case of C++, symbols are mangled, according to compiler-specific schemes, to incorporate the argument signature so as to facilitate overloaded functions, which could not otherwise be differentiated.)
The linker is only concerned with unresolved symbols (so there ought be nothings preventing multiple libraries concurrently calling their own respective internally-defined functions with coincident names). If a file invokes a declared but undefined function, and multiple available libraries implement that symbol, then the linker is free to choose (say by precedence in a search-path) which version gets substituted in. (Incidentally, this is the same mechanism by which profilers such as gperftools or hpctoolkit are able to inject themselves and alter the normal behaviour of another application.)
Since different libraries are mapped to separate pages of memory, it ought to be possible to identify (from memory addresses) which library contains the executing version of a function. Indeed, the GNU debugger can identify the library that code is contained by, even when it fails to name a function.
$ gdb python
(gdb) run -c "from numpy import *; linalg.inv(random.random((1000,1000)))"
CTRL-C
(gdb) backtrace
#0 0x00007ffff5ba9df8 in dtrsm_ () from /usr/lib/libblas.so.3
...
#3 0x00007ffff420df83 in ?? () from /.../numpy/linalg/_umath_linalg.so
Linux (or rather the GNU C library) provides the "backtrace" call (for getting a list of pointers from the call stack), and the "backtrace_symbols" call for automatically converting each of those pointers to a descriptive string such as:
"/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fc429929ec5]"
Gperftools can (judging from a query on the github mirror) call the generic "backtrace", but instead of "backtrace_symbols" it "forks out to pprof to do the actual symbolizing". This is a fairly-epic perl script, and looks likely where the "??" comes from.
Crucially, google-pprof is trying to report on the source-file (and line-number) which defines the function, not the binary-file containing the machine-code (that is typically quoted in stack traces). It invokes the "nm" utility. On my system it appears (by running "nm -l -D") that libblas, unlike libc and the python binary, has been stripped of such debugging symbols (presumably for optimisation), explaining the result.
To answer the original question: the call-stack samples should definitively and explicitly specify which version is being called. These can probably be dumped using an option which was added in google-pprof several months ago, or (for time-intensive functions) can be roughly ascertained by manual resampling using gdb. (It's even conceivable that g-pprof can be adjusted to explicitly identify the binaries paths in its output summaries.) Alternatively one can run "nm" (and grep) on the candidate binaries/libraries (of which a short-list can be obtained by running "strings" on the profiler's raw output, among other methods). If the source is accessible (to grep) or the libraries are popular (on the web) then of course (and per Mike Dunlavey) it may be easiest to just query for the function name. In theory the "??:?" may be addressed by carefully recompiling the offending objects.
Just Google the offending function names. The ones you show above are defined in LAPACK. dtrsm is for solving a matrix equation. dgemm is for multiplying matrices.
What you need to know is 1) why they are being called, and 2) how big the matrices are.
To find out why they are being called, what I do is just examine individual stack samples, as here.
The reason matrix size matters is if they are small, these LAPACK routines can actually spend a relatively large fraction of their time just classifying their inputs, such as by calling a function LSAME.
Related
I'm looking for idea on how to overwrite a function without modify the source. Like if I have foo() in the original source, I want to overwrite it with my own version with the same function name by adding it in a C file, which may also contains other overwrite functions. Sort of like strong/weak compilation. Currently I have to go in the source files and ifdef with __FRAMAC__. I don't want to touch the source files. Is there some kernel option to not use the second instance of foo() function?
Your question does not specify whether you want to replace a function declaration or a function definition. Since they are handled differently by Frama-C, I'm going to detail both.
Duplicate definitions at the kernel level
Currently, at the parsing level, there is no option in Frama-C to completely ignore the definition of a function that is present in one of the files given for parsing. The Frama-C AST will incorporate the definition of all functions it finds.
There is no exact equivalent for strong/weak symbols.
If a second definition for the same function is found, one of the following will happen:
If both definitions occur in the same compilation unit, there is an error.
If each definition happens in a different compilation unit, Frama-C will try to find a plausible solution:
If both occurrences have the same signature, Frama-C will emit a warning such as:
[kernel] b.c:2: Warning:
dropping duplicate def'n of func f at b.c:2 in favor of that at a.c:1
In this case, you just need to ensure the definition you want appears later in the list of sources to be parsed.
If the occurrences have different signatures, but one of the functions is never actually used, you may have a warning such as:
[kernel:linker:drop-conflicting-unused] Warning:
Incompatible declaration for f:
different number of arguments
First declaration was at a.c:1
Current declaration is at b.c:2
Current declaration is unused, silently removing it
However, if both occurrences are used, then you have an error:
[kernel] User Error: Incompatible declaration for f:
different type constructors: int vs. int *
First declaration was at a.c:1
Current declaration is at b.c:2
Duplicate declarations at the kernel level
Considering function declarations, Frama-C will, in accordance with the C standard, accept as many of them as are given, provided they are compatible. If they have ACSL specifications, those specifications will be merged.
Multiple incompatible declarations are handled as before, with warnings or errors depending on whether both versions are used (in which case Frama-C is unable to choose).
Plugin-specific options
Plug-ins may have specific options to override the default behavior of functions in the AST. For instance, Eva has option -eva-use-spec <fns>, which tells the analysis to ignore the definitions of functions <fns>, using only their specifications instead.
Is there a way to access all the variables/arguments passed through the command line or variable file (-V option) during robotframework execution. I know in python the execution can access it with 'sys.args' feature.
The answer for getting the CLI arguments is inside your question - just look at the content of the sys.argv, you'll see everything that was passed to the executor:
${args}= Evaluate sys.argv sys
Log To Console ${args}
That'll return a list, where the executable itself (run.py) is the 1st member, and all arguments and their values present the in the order given during the execution:
['C:/my_directories/rf-venv/Lib/site-packages/robot/run.py', '--outputdir', 'logs', '--variable', 'USE_BROWSERSTACK:true', '--variable', 'IS_DEV_ENVIRONMENT:false', '--include', 'worky', 'suites\\test_file.robot']
You explicitly mention variable files; that one is a little bit trickier - the framework parses the files itself, and creates the variables according to its rules. You naturally can see them in the CLI args up there, and the other possibility is to use the built-in keyword Get Variables, which "Returns a dictionary containing all variables in the current scope." (quote from its documentation). Have in mind though that these are all variables - not only the passed on the command line, but also the ones defined in the suite/imported keywords etc.
You have Log Variables to see their names and values "at current scope".
There is no possibility to see the arguments passed to robot.
I am trying to create a FUSE filesystem called ordered-dirs using the Haskell wrapper over libfuse, HFuse. This filesystem is a "derived filesystem", i.e. it takes an existing directory (the "base directory") and produces a different view of it.
However, when I try to run my FUSE filesystem program, specifying the arguments in the ordinary mount way, I get an error:
$ ordered-dirs /home/robin/tasks/ /home/robin/to
fuse: invalid argument `/home/robin/to'
There is no way in HFuse (or in libfuse, it seems) to get the base directory (the first argument), so I had just written my own code to get it. But it's not this code that's failing - it's code within C libfuse itself - as the error message indicates.
So what is the correct way to pass the base directory to a fuse filesystem executable that uses libfuse to parse its arguments?
Surprisingly, it seems that the way to do this is to simply strip the base directory argument from the command-line arguments that are parsed to the libfuse parser, so that libfuse never sees it.
In the particular case of HFuse, this can be done by calling fuseRun instead, which allows the command-line arguments to be passed in explicitly. You can see how I've done this here - here is the relevant code (in which I've called the base directory source):
main :: IO ()
main = do
args <- getArgs
let (maybeSource, remainder) = extractSource args
source <- maybe (fail "source not specified") return maybeSource
fuseRun "ordered-dirs" remainder (orderedDirOps source) defaultExceptionHandler
I have tried
let _ = Unix.create_process "ls" [||] Unix.stdin Unix.stdout Unix.stderr
in utop, it will crash the whole thing.
If I write that into a .ml and compile and run, it will crash the terminal and my ubuntu will throw a system error.
But why?
The right way to call it is:
let pid = Unix.create_process "ls" [|"ls"|] Unix.stdin Unix.stdout Unix.stderr
The first element of the array must be the "command" name.
On some systems /bin/ls is a link to some bigger executable that will look at argv.(0) to know how to behave (c.f. Busybox); so you really need to provide that info.
(You see more often that with /usr/bin/vi which is now on many systems a sym-link to vim).
Unix.create_process actually calls fork and the does an execvpe, which itself calls the execv primitive (in the OCaml C implementation of the Unix module).
That function then calls cstringvect (a helper function in the C side of the module implementation), which translates the arg parameters into an array of C string, with last entry set to NULL. However, execve and the like expect by convention (see the execve(2) linux man page) the first entry of that array to be the name of the program:
argv is an array of argument strings passed to the new program. By
convention, the first of these strings should contain the filename
associated with the file being executed.
That first entry (or rather, the copy it receives) can actually be changed by the program receiving these args, and is displayed by ls, top, etc.
I am trying to build a Fortran DLL with Absoft Pro Fortran 13.0.3, 64 bits, for use within R, on Windows 7 64 bits.
Here is my file mycalc.f (it's a dumb example, just to test functionality):
subroutine mycalcf(a,b,c)
real*8 a,b,c
dll_export mycalcf
c=a+b*b
end
The statement dll_export is not standard, but is found in some Fortran compilers (AFAIK it's also found in Lahey and CVF and Intel Fortran has a compiler directive instead). It just tells the compiler which symbols are to be exported.
I compile successfuly with:
af90 -m64 -dll -YDLL_NAMES=LCS mycalc.f -o mycalc.dll
The option -YDLL_NAMES=LCS tells the compiler to build a library with lowercase symbols, which seems better for R.
If I run dumpbin /exports mycalc.dll, I can find mycalcf in exported symbols, in lowercase, without any underscore before or after.
Now, from R (64 bit version), the following works:
dyn.load("mycalc.dll")
is.loaded("mycalcf")
.Fortran("mycalcf", a=4, b=5, c=0)
I get c=29 on return, as expected.
BUT, if I restart R, the following does not work (notice I only removed the is.loaded test):
dyn.load("mycalc.dll")
.Fortran("mycalcf", a=4, b=5, c=0)
I get the error: Fortran symbol name "mycalcf" not in load table.
Now my question is: why is this test so important?
For comparison, when I try the same with gfortran instead of Absoft, I have no problem at all. I compile with: gfortran -m64 -shared -o mycalc2.dll mycalc.f (after commenting out the dll_export statement, which is not needed, nor even recognized, by gfortran).
Then in R:
dyn.load("mycalc2.dll")
.Fortran("mycalcf", a=4, b=5, c=0)
And I get c=29, no error.
Now, I suspect there is something the gcc linker does that is not done automatically by the Absoft linker (actually it's Microsoft's link.exe). But I have no hint what it can be.
Any idea is welcome!
Ok, solution after a good question from Vladimir F (see comments).
Actually, one must append underscores to symbol names. Since there is no way to do it by compiler option, one needs CDEC$ directives (see HP or Intel documentation).
Here it's simply:
subroutine mycalcf(a,b,c)
CDEC$ attributes alias:'mycalcf_' :: mycalcf
real*8 a,b,c
dll_export mycalcf
c=a+b*b
end
Second solution, from Absoft forum: actually I was wrong from the very beginning. Contrary to what I thought, one need not use the dll_export statement, and it even introduced the problem: without it, the compiler appends underscores. All symbols are exported by default, as in gfortran. So the correct code is simply:
subroutine mycalcf(a,b,c)
real*8 a,b,c
c=a+b*b
end
There is even no need for any option to get lowercase symbols, it's also the defaults.
However, one question remains: does the R function .Fortran always add underscore (is there a way to tell it not to?), and if it's always added, why does the call work when is.loaded is called beforehand? R seems to be doing something weird here.
I tried to track is.loaded in R source code (up to do_isloaded in src\main\dotcode.c), to no avail.