Fortran symbol not in load table (unable to call loaded symbol in R) - r

I am trying to build a Fortran DLL with Absoft Pro Fortran 13.0.3, 64 bits, for use within R, on Windows 7 64 bits.
Here is my file mycalc.f (it's a dumb example, just to test functionality):
subroutine mycalcf(a,b,c)
real*8 a,b,c
dll_export mycalcf
c=a+b*b
end
The statement dll_export is not standard, but is found in some Fortran compilers (AFAIK it's also found in Lahey and CVF and Intel Fortran has a compiler directive instead). It just tells the compiler which symbols are to be exported.
I compile successfuly with:
af90 -m64 -dll -YDLL_NAMES=LCS mycalc.f -o mycalc.dll
The option -YDLL_NAMES=LCS tells the compiler to build a library with lowercase symbols, which seems better for R.
If I run dumpbin /exports mycalc.dll, I can find mycalcf in exported symbols, in lowercase, without any underscore before or after.
Now, from R (64 bit version), the following works:
dyn.load("mycalc.dll")
is.loaded("mycalcf")
.Fortran("mycalcf", a=4, b=5, c=0)
I get c=29 on return, as expected.
BUT, if I restart R, the following does not work (notice I only removed the is.loaded test):
dyn.load("mycalc.dll")
.Fortran("mycalcf", a=4, b=5, c=0)
I get the error: Fortran symbol name "mycalcf" not in load table.
Now my question is: why is this test so important?
For comparison, when I try the same with gfortran instead of Absoft, I have no problem at all. I compile with: gfortran -m64 -shared -o mycalc2.dll mycalc.f (after commenting out the dll_export statement, which is not needed, nor even recognized, by gfortran).
Then in R:
dyn.load("mycalc2.dll")
.Fortran("mycalcf", a=4, b=5, c=0)
And I get c=29, no error.
Now, I suspect there is something the gcc linker does that is not done automatically by the Absoft linker (actually it's Microsoft's link.exe). But I have no hint what it can be.
Any idea is welcome!
Ok, solution after a good question from Vladimir F (see comments).
Actually, one must append underscores to symbol names. Since there is no way to do it by compiler option, one needs CDEC$ directives (see HP or Intel documentation).
Here it's simply:
subroutine mycalcf(a,b,c)
CDEC$ attributes alias:'mycalcf_' :: mycalcf
real*8 a,b,c
dll_export mycalcf
c=a+b*b
end
Second solution, from Absoft forum: actually I was wrong from the very beginning. Contrary to what I thought, one need not use the dll_export statement, and it even introduced the problem: without it, the compiler appends underscores. All symbols are exported by default, as in gfortran. So the correct code is simply:
subroutine mycalcf(a,b,c)
real*8 a,b,c
c=a+b*b
end
There is even no need for any option to get lowercase symbols, it's also the defaults.
However, one question remains: does the R function .Fortran always add underscore (is there a way to tell it not to?), and if it's always added, why does the call work when is.loaded is called beforehand? R seems to be doing something weird here.
I tried to track is.loaded in R source code (up to do_isloaded in src\main\dotcode.c), to no avail.

Related

Error: Unable to load gfortran compiled DLL in R ("symbol name not in load table")

Well, I recently fell in love with Fortran (f90) and have been trying to understand the "kung-fu" of R and Fortran. I found several relevant and helpful questions here (e.g. this and this).
What I am trying to do:
I am (probably, trying to do something crazy) trying to call the following .f90 subroutines in R (x64) using .Fortran() function. Here is the test.f90 code:
! Computes the square of a number
Subroutine sr1(a,b)
!DEC$ ATTRIBUTES DLLEXPORT::sr1
!DEC$ ATTRIBUTES C, REFERENCE, ALIAS:'sr1' :: sr1
implicit none
integer a,b
b = a*a
End Subroutine sr1
! Computes the cube of a number
Subroutine sr2(x,y)
!DEC$ ATTRIBUTES DLLEXPORT::sr2
!DEC$ ATTRIBUTES C, REFERENCE, ALIAS:'sr2' :: sr2
implicit none
integer x,y
y = x*x*x
End Subroutine sr2
I am compiling the above test.f90 code via gfortran on my Windows 10 machine by:
gfortran -shared -o test.dll test.f90
The compilation works and I get the test.dll. Now, in R. I try to load it:
dyn.load("path_to_file/test.dll")
It works. But, it fails here:
> is.loaded("test")
[1] False
I already found a relevant question here. But, I could not get the clue to fix my problem. Can someone suggest some workaround to fix the issue?
I am not a big user of R, but my tests show that while
is.loaded("test_R")
indeed returns FALSE, both
is.loaded("sr1")
and
is.loaded("sr2")
return TRUE. But I did my tests on Linux and GCC which may interpret the extrnally visible subroutine names differently.

gperftools failing to identify files

Is there a way to avoid Google Performance Tools listing files as "??:?", that is, failing to locate which file contains the function it is reporting on? How can I work out which library contains the function being called?
$ env LD_PRELOAD="/usr/lib/libprofiler.so.0" \
CPUPROFILE=output.prof python script.py
$ google-pprof --text --files /usr/bin/python output.prof
Using local file /usr/bin/python.
Using local file output.prof.
Removing _L_unlock_13 from all stack traces.
Total: 433 samples
362 83.6% 83.6% 362 83.6% dtrsm_ ??:?
58 13.4% 97.0% 58 13.4% dgemm_ ??:?
1 0.2% 97.2% 1 0.2% PyDict_GetItem /.../Objects/dictobject.c
1 0.2% 97.5% 1 0.2% PyParser_AddToken /.../Parser/parser.c
...
I am aiming to be able to profile the C code in a python package that has many compiled C extension modules. In the toy example above, what would I do to track down where "dtrsm_" is defined? If there are multiple loaded libraries that contain functions with that same name, is there any way to tell which version is being called?
C/C++ won't compile if the same pre-processed sourcefile (e.g. with #includes expanded) contains duplicate definitions for the same symbol. (Note that in the case of C++, symbols are mangled, according to compiler-specific schemes, to incorporate the argument signature so as to facilitate overloaded functions, which could not otherwise be differentiated.)
The linker is only concerned with unresolved symbols (so there ought be nothings preventing multiple libraries concurrently calling their own respective internally-defined functions with coincident names). If a file invokes a declared but undefined function, and multiple available libraries implement that symbol, then the linker is free to choose (say by precedence in a search-path) which version gets substituted in. (Incidentally, this is the same mechanism by which profilers such as gperftools or hpctoolkit are able to inject themselves and alter the normal behaviour of another application.)
Since different libraries are mapped to separate pages of memory, it ought to be possible to identify (from memory addresses) which library contains the executing version of a function. Indeed, the GNU debugger can identify the library that code is contained by, even when it fails to name a function.
$ gdb python
(gdb) run -c "from numpy import *; linalg.inv(random.random((1000,1000)))"
CTRL-C
(gdb) backtrace
#0 0x00007ffff5ba9df8 in dtrsm_ () from /usr/lib/libblas.so.3
...
#3 0x00007ffff420df83 in ?? () from /.../numpy/linalg/_umath_linalg.so
Linux (or rather the GNU C library) provides the "backtrace" call (for getting a list of pointers from the call stack), and the "backtrace_symbols" call for automatically converting each of those pointers to a descriptive string such as:
"/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fc429929ec5]"
Gperftools can (judging from a query on the github mirror) call the generic "backtrace", but instead of "backtrace_symbols" it "forks out to pprof to do the actual symbolizing". This is a fairly-epic perl script, and looks likely where the "??" comes from.
Crucially, google-pprof is trying to report on the source-file (and line-number) which defines the function, not the binary-file containing the machine-code (that is typically quoted in stack traces). It invokes the "nm" utility. On my system it appears (by running "nm -l -D") that libblas, unlike libc and the python binary, has been stripped of such debugging symbols (presumably for optimisation), explaining the result.
To answer the original question: the call-stack samples should definitively and explicitly specify which version is being called. These can probably be dumped using an option which was added in google-pprof several months ago, or (for time-intensive functions) can be roughly ascertained by manual resampling using gdb. (It's even conceivable that g-pprof can be adjusted to explicitly identify the binaries paths in its output summaries.) Alternatively one can run "nm" (and grep) on the candidate binaries/libraries (of which a short-list can be obtained by running "strings" on the profiler's raw output, among other methods). If the source is accessible (to grep) or the libraries are popular (on the web) then of course (and per Mike Dunlavey) it may be easiest to just query for the function name. In theory the "??:?" may be addressed by carefully recompiling the offending objects.
Just Google the offending function names. The ones you show above are defined in LAPACK. dtrsm is for solving a matrix equation. dgemm is for multiplying matrices.
What you need to know is 1) why they are being called, and 2) how big the matrices are.
To find out why they are being called, what I do is just examine individual stack samples, as here.
The reason matrix size matters is if they are small, these LAPACK routines can actually spend a relatively large fraction of their time just classifying their inputs, such as by calling a function LSAME.

Compiler messages in Julia

Consider the following code:
File C.jl
module C
export printLength
printLength = function(arr)
println(lentgh(arr))
end
end #module
File Main.jl
using C
main = function()
arr = Array(Int64, 4)
printLength(arr)
end
main()
Let's try to execute it.
$ julia Main.jl
ERROR: lentgh not defined
in include at /usr/bin/../lib64/julia/sys.so
in process_options at /usr/bin/../lib64/julia/sys.so
in _start at /usr/bin/../lib64/julia/sys.so
while loading /home/grzes/julia_sucks/Main.jl, in expression starting on line 8
Obviously, it doesn't compile, because lentgh is misspelled. The problem is the message I received. expression starting on line 8 is simply main(). Julia hopelessly fails to point the invalid code fragment -- it just points to the invocation of main, but the erroneous line is not even in that file! Now imagine a real project where an error hides really deep in the call stack. Julia still wouldn't tell anything more than that the problem started on the entry point of the execution. It is impossible to work like that...
Is there a way to force Julia to give a little more precise messages?
In this case it's almost certainly a consequence of inlining: your printLength function is so short, it's almost certainly inlined into the call site, which is why you get the line number 8.
Eventually, it is expected that inlining won't cause problems for backtraces. At the moment, your best bet---if you're running julia's pre-release 0.4 version---is to start julia as julia --inline=no and run your tests again.

Ada object declarations "Unsigned Not Declared in System"

In some code that I inherited, I get the compile error "Unsigned" not declared in "System".
I'm trying to compile this using GNAT, but ultimately the code must compile with the original tools, which I don't have ready access to. So I'd like to understand how to resolve this from within the development environment (including the project file), and not modify the existing code.
I checked the file system.ads, and Unsigned is not defined there. Am I referring to the wrong libraries? How would I resolve this with the self imposed constraint mentioned above (to compile in the original environment)?
unsigned is the name of a predefined type in C. If what you need it an Ada type that matches the C type, what you need is Interfaces.C.unsigned. An older Ada implementation (before Interfaces.C was introduced by the 1995 standard) might have defined System.Unsigned for this purpose.
It would help to know what Ada implementation the code was originally written for.
You should examine the code to see whether it uses that type to interface to C code. If not (i.e., if it's just being used as a general unsigned integer type), you might instead consider defining your own modular type.
If I understand correctly, you need the code to compile both in the original environment and with GNAT. That might be difficult. One approach would be to define a new package with two different versions, one for the original environment and one for GNAT (or, ideally, for any modern Ada implementation). For example:
-- version for original environment
with System;
package Foo is
subtype Unsigned is System.Unsigned;
end foo;
and:
-- version for GNAT
with Interfaces.C;
package Foo is
subtype Unsigned is Interfaces.C.Unsigned;
end Foo;
Picking a better name than Foo is left as an exercise, as is determining automatically which version to use.
You could rebuild the GNAT runtime system (RTS) with a slightly modified system.ads.
There’s a Makefile.adalib in the system RTS (well, there is in GNAT GPL 2014) which lets you do this. It’s at the last directory indicated in the “Object Search Path” section of the output of gnatls -v.
The RTS source is similarly indicated in the “Source Search Path” section.
Create a directory say unsigned with subdirectories adainclude, adalib.
Copy the RTS source into unsigned/adainclude, and edit system.ads to include
type Unsigned is mod 2 ** 32;
(I’m guessing a bit, but this is probably what you want!)
Then, in unsigned/adalib,
make -f Makefile.adalib ADA_INCLUDE_PATH=../adainclude ROOT=/opt/gnat-gpl-2014
(ROOT is where you have the compiler installed; it will be different on your system, it’s one above the bin directory in which gnatls and friends are installed).
There will be several errors during this, all caused (when I tried it) by units that use System.Unsigned_Types;. Work round this by inserting this immediately after the package body in the .adb:
subtype Unsigned is System.Unsigned_Types.Unsigned;
The files I had to change were
s-expmod.adb
s-expuns.adb
s-imgbiu.adb
s-imgrea.adb
s-imguns.adb
s-imgwiu.adb
s-valint.adb
s-valuns.adb
s-vercon.adb
It may be best at this stage to remove all the .ali and .a files from unsigned/adalib and repeat, to get a clean build.
Now, you should be able to use System.Unsigned by
gnatmake --RTS=/location/of/unsigned t.adb
In my case, t.adb contained
with System;
with Ada.Text_IO; use Ada.Text_IO;
procedure T is
begin
Put_Line ("first: " & System.Unsigned'First'Img);
Put_Line ("last: " & System.Unsigned'Last'Img);
Put_Line ("42: " & System.Unsigned'Value ("42")'Img);
Put_Line ("16#42#:" & System.Unsigned'Value ("16#42#")'Img);
end T;
and the output was
$ ./t
first: 0
last: 4294967295
42: 42
16#42#: 66

How do I define a macro with the same name as its expansion in m4?

I am attempting to replace if with if( using GNU m4 1.4.14 and I am receiving ERROR: end of file in argument list
when trying:
define(`if', `if(')
define(`then', `){')
define(`fi', `}')
if foo then bar() fi
I have tried escaping the parentheses but that caused m4 to error after a brief period of time saying it's out of memory. Scanning through the manual, I found nothing related to this problem.
Upon changing the name of the macro to 'IF' or something other than 'if', it works as expected, which leads me to believe it's evaluating itself repeatedly.
If so, how can I define a macro that is evaluated only once? Otherwise, what should I look into to fix this?
EDIT I found a way around this issue by processing twice, once to convert if to _IF and the next to convert _IF to if(. I assume there's a better way to do this, so this is only a temporary solution in my eyes.
You need to prevent m4 from attempting to re-expanding the replacements. Do so by double quoting:
define(`if', ``if('')
define(`then', `){')
define(`fi', `}')
if foo then bar() fi

Resources