I am trying to marry CPython and .NET DLR. The later has a mechanism for fast dynamic dispatch based on actual parameter types at the call site, which I'd like to take advantage of. For example, suppose I have an overloaded function in C#, that I want to call from Python:
void Foo(int arg) {}
void Foo(string arg) {}
Then in Python I have a script like this:
Foo(42) # line 1
Foo("Hello") # line 2
So in my C extension I am implementing tp_call for Foo. To be able to use the fast dispatch cache provided by DLR I need to get some value, that can be obtained quickly and uniquely identifies the call site (e.g. tells my tp_call which script line is being invoked). Is anything like that available?
The PyFrameObject returned by PyEval_GetFrame seems like it might do the trick, but it is poorly documented, so it is unclear which fields from it can be used. Is PyFrame_GetLineNumber the only supported available value?
Related
I want to do an LLVM compiler for a very old language, PL/M. This has some peculiar features, not least of which is having nested functions with the ability to jump out of an enclosing function. In pseudocode:
toplevel() {
nested() {
if (something)
goto label;
}
nested();
label:
print("finished!");
}
The constraints here are:
you can only jump into the top-level function, luckily
the stack does get unwound (the language does not support destructors, so this is easy)
you do not have to have executed the statement at label before jumping (so the naive setjmp/longjmp method doesn't work).
code at label can be executed normally, i.e. it's not like catch
LLVM has a number of non-local jump mechanisms, such as the exception handling system, but I've never used that. Can this be implemented using LLVM exceptions, or are they not suitable for this? Is there an easier way?
If you want the stack to get unwound, you'll likely want it to be in a separate function, at least a separate LLVM IR function. (The only real exception is if your language does not have a construct like C's "alloca()" and you don't allow calling a nested function by address in which case you could inline it.)
That part of the problem you mentioned, jumping out of an enclosing function, is best handled by having some way for the callee to communicate "how it exited" to the caller, and the caller having a "switch()" on that value. You could stick it in the return value (if it already returns a value, make it a struct of both values), you could add a pointer parameter that it writes to, you could add it a thread-local global variable and fill that in before calling longjmp, or you could use exceptions.
Exceptions, they're complex (I can't describe how to make them work offhand but the docs are here: https://llvm.org/docs/ExceptionHandling.html ) and slow when the exception path is taken, and really intended for exceptional situations, not for normal code. Setjmp/longjmp does the same thing as exceptions except simpler to use and without the performance trade-off when executed, but unfortunately there are miscompiles in LLVM which you need will be the one to fix if you start using them in earnest (see the postscript at the end of the answer).
Those two options cover the ways you can do it without changing the function signature, which may be necessary if your language allows the address to be taken then called later.
If you do need to take the address of nested, then LLVM supports trampolines. See https://llvm.org/docs/LangRef.html#trampoline-intrinsics . Trampolines solve the problem of accessing the local variables of the calling function from the callee, even when the function is called by address.
PS. LLVM miscompiles setjmp/longjmp today. The current model is that a call to setjmp may return twice, and only functions with the returns_twice attribute may return twice. Note that this doesn't affect the whole call stack, only the direct caller of a function that returns twice has to deal with the twice-returning call-- just because function F calls setjmp does not mean that F itself can return twice. So far, so good.
The problem is that in a function with a setjmp, all function calls may themselves call longjmp. I'd say "unless proven otherwise" as with all things in optimizers, but there is no attribute in LLVM doesnotlongjmp or any code within LLVM that attempts to answer the question of whether a function could call longjmp. Adding that would be a good optimization, but it's a separate issue from the miscompile.
If you have code like this pseudo-code:
%entry block:
allocate val
val <- 0
setjmpret <- call setjmp
br i1 setjmpret, %first setjmp return block, %second setjmp return block
%first setjmp return block:
val <- 1;
call foo();
goto after;
%second setjmp return block:
call print(val);
goto after;
%after:
return
The control flow graph shows that is no path from val <- 0 to val <- 1 to print(val). The only path with "print(val)" has "val <- 0" before it therefore constant propagation may turn print(val) into print(0). The problem here is a missing control flow edge from foo() back to the %second setjmp return block. In a function that contains a setjmp, all calls which may call longjmp must have a CFG edge to the second setjmp return block. In LLVM that control flow edge is missing and LLVM miscompiles code because of it.
This problem also manifests in the backend. The first time I heard of this problem it was in the context of the backend losing track of the placement of variables on the stack, and this issue was the underlying root cause.
For the most part setjmp/longjmp seems to work because LLVM isn't usually able to analyze what calling foo() might do and can't perform the optimization. For instance if val was not a fresh allocation but was a pointer, then who's to say that foo() doesn't have access to the same pointer, and then performs "val <- 1" on it? If LLVM can't prove that impossible, that precludes the transform to print(0). Secondly, setjmp/longjmp are just not used often in real code.
I've been wondering how delegated properties ("by"-Keyword) work under-the-hood. I get that by contract the delegate (right side of "by") has to implement a get and setValue(...) method, but how can that be ensured by the compiler and how can those methods be accessed at runtime? My initial thought was that obviously the delegates must me implementing some sort of "SuperDelegate"-Interface, but it appears that is not the case. So the only option left (that I am aware of) would be to use Reflection to access those methods, possibly implemented at a low level inside the language itself. I find that to be somewhat weird, since by my understanding that would be rather inefficient. Also the Reflection API is not even part of the stdlib, which makes it even weirder.
I am assuming that the latter is already (part of) the answer. So let me furthermore ask you the following: Why is there no SuperDelegate-Interface that declare the getter and setter methods that we are forced to use anyway? Wouldn't that be much cleaner?
The following is not essential to the question
The described Interface(s) are even already defined in ReadOnlyProperty and ReadWriteProperty. To decide which one to use could then be made dependable on whether we have a val/var. Or even omit that since calling the setValue Method on val's is being prevented by the compiler and only use the ReadWriteProperty-Interface as the SuperDelegate.
Arguably when requiring a delegate to implement a certain interface the construct would be less flexible. Though that would be assuming that the Class used as a Delegate is possibly unaware of being used as such, which I find to be unlikely given the specific requirements for the necessary methods. And if you still insist, here's a crazy thought: Why not even go as far as to make that class implement the required interface via Extension (I'm aware that's not possible as of now, but heck, why not? Probably there's a good 'why not', please let me know as a side-note).
The delegates convention (getValue + setValue) is implemented at the compiler side and basically none of its resolution logic is executed at runtime: the calls to the corresponding methods of a delegate object are placed directly in the generated bytecode.
Let's take a look at the bytecode generated for a class with a delegated property (you can do that with the bytecode viewing tool built into IntelliJ IDEA):
class C {
val x by lazy { 123 }
}
We can find the following in the generated bytecode:
This is the field of the class C that stores the reference to the delegate object:
// access flags 0x12
private final Lkotlin/Lazy; x$delegate
This is the part of the constructor (<init>) that initialized the delegate field, passing the function to the Lazy constructor:
ALOAD 0
GETSTATIC C$x$2.INSTANCE : LC$x$2;
CHECKCAST kotlin/jvm/functions/Function0
INVOKESTATIC kotlin/LazyKt.lazy (Lkotlin/jvm/functions/Function0;)Lkotlin/Lazy;
PUTFIELD C.x$delegate : Lkotlin/Lazy;
And this is the code of getX():
L0
ALOAD 0
GETFIELD C.x$delegate : Lkotlin/Lazy;
ASTORE 1
ALOAD 0
ASTORE 2
GETSTATIC C.$$delegatedProperties : [Lkotlin/reflect/KProperty;
ICONST_0
AALOAD
ASTORE 3
L1
ALOAD 1
INVOKEINTERFACE kotlin/Lazy.getValue ()Ljava/lang/Object;
L2
CHECKCAST java/lang/Number
INVOKEVIRTUAL java/lang/Number.intValue ()I
IRETURN
You can see the call to the getValue method of Lazy that is placed directly in the bytecode. In fact, the compiler resolves the method with the correct signature for the delegate convention and generates the getter that calls that method.
This convention is not the only one implemented at the compiler side: there are also iterator, compareTo, invoke and the other operators that can be overloaded -- all of them are similar, but the code generation logic for them is simpler than that of delegates.
Note, however, that none of them requires an interface to be implemented: the compareTo operator can be defined for a type not implementing Comparable<T>, and iterator() does not require the type to be an implementation of Iterable<T>, they are anyway resolved at compile-time.
While the interfaces approach could be cleaner than the operators convention, it would allow less flexibility: for example, extension functions could not be used because they cannot be compiled into methods overriding those of an interface.
If you look at the generated Kotlin bytecode, you'll see that a private field is created in the class holding the delegate you're using, and the get and set method for the property just call the corresponding method on that delegate field.
As the class of the delegate is known at compile time, no reflection has to happen, just simple method calls.
Let's say I'm implementing the kill program in Go. I can accept numeric signals and PIDs from the commandline and send them to syscall.Kill no problem.
However, I don't know how to implement the "string" form of signal dispatch, e.g. kill -INT 12345.
The real use case is a part of a larger program that prompts the user to send kill signals; not a replacement for kill.
Question:
How can I convert valid signal names to signal numbers on any supported platform, at runtime (or at least without writing per-platform code to be run at compile time)?
What I've tried:
Keep a static map of signal names to numbers. This doesn't work in a cross-platform way (for example, different signal lists are returned by kill -l on Mac OSX versus a modern Linux versus an older Linux, for example). The only way to make this solution work in general would be to make maps for every OS, which would require me to know the behavior of every OS, and keep up to date as they add new signal support.
Shell out to the GNU kill tool and capture the signal lists from it. This is inelegant and kind of a paradox, and also requires a) being able to find kill, b) having the ability/permission to exec subprocesses, and c) being able to predict/parse the output of kill-the-binary.
Use the various Signal types' String method. This just returns strings containing the signal number, e.g. os.Signal(4).String() == "signal 4", which is not useful.
Call the private function runtime.signame, which does exactly what I want. go://linkname hacks will work, but I'm assuming that this sort of thing is frowned-upon for a reason.
Ideas/Things I Haven't Tried:
Use CGo somehow. I'd rather not venture into CGO territory for a project that is otherwise not low-level/needful of native integration at all. If that's the only option, I will, but have no idea where to start.
Use templating and code generation to build lists of signals based on external sources at compile time. This is not preferable for the same reasons as CGo.
Reflect and parse the members of syscall that start with SIG somehow. I am told that this is not possible because names are compiled away; is it possible that, for something as fundamental as signal names, there's someplace they're not compiled away?
Commit d455e41 added this feature in March 2019 as sys/unix.SignalNum() and is thus available at least since Go 1.13. More details in GitHub issue #28027.
From the documentation of the golang.org/x/sys/unix package:
func SignalNum(s string) syscall.Signal
SignalNum returns the syscall.Signal for signal named s, or 0 if a signal with such name is not found. The signal name should start with "SIG".
To answer a similar question, "how can I list the names of all available signals (on a given Unix-like platform)", we can use the inverse function sys/unix.SignalName():
import "golang.org/x/sys/unix"
// See https://github.com/golang/go/issues/28027#issuecomment-427377759
// for why looping in range 0,255 is enough.
for i := syscall.Signal(0); i < syscall.Signal(255); i++ {
name := unix.SignalName(i)
// Signal numbers are not guaranteed to be contiguous.
if name != "" {
fmt.Println(name)
}
}
Update some time after I posted the below answer, Golang's stdlib acquired this functionality. An answer describing how to use that functionality was posted by #marco.m and accepted; the below is not recommended unless the version of Go you are using pre-dates the availability of the right tool for the job.
Since no answers were posted, I'll post the less-than-ideal solution I was able to use by "breaking into" a private signal-enumeration function inside Go's standard library.
The signame internal function can get a signal name by number on Unix and Windows. To call it, you have to use the linkname/assembler workaround. Basically, make a file in your project called empty.s or similar, with no contents, and then a function declaration like so:
//go:linkname signame runtime.signame
func signame(sig uint32) string
Then, you can get a list of all signals known by the operating system by calling signame on an increasing number until it doesn't return a value, like so:
signum := uint32(0)
signalmap = make(map[uint32]string)
for len(signame(signum)) > 0 {
words := strings.Fields(signame(signum))
if words[0] == "signal" || ! strings.HasPrefix(words[0], "SIG") {
signalmap[signum] = ""
} else {
// Remove leading SIG and trailing colon.
signalmap[signum] = strings.TrimRight(words[0][3:], ":")
}
signum++
}
After that runs, signalmap will have keys for every signal that can be sent on the current operating system. It will have an empty string where Go doesn't think the OS has a name for the signal (the kill(1) may name some signals that Go won't return names for, I've found, but it's usually the higher-numbered/nonstandard ones), or a string name, e.g. "INT" where a name can be found.
This behavior is undocumented, subject to change, and may not hold true on some platforms. It would be nice if this were made public, though.
I'm writing a program in Scala and trying to remain as functionally pure as is possible. The problem I am facing is not Scala specific; it's more to do with trying to code functionally. The logic for the function that I have to code goes something like:
Take some value of type A
Use this value to generate log information
Log this information by calling a function in an external library and evaluate the return status of the logging action (ie was it a successful log or did the log action fail)
Regardless of whether the log succeeded or failed, I have to return the input value.
The reason for returning the input value as the output value is that this function will be composed with another function which requires a value of type A.
Given the above, the function I am trying to code is really of type A => A i.e. it accepts a value of type A and returns a value of type A but in between it does some logging. The fact that I am returning the same value back that I inputted makes this function boil down to an identity function!
This looks like code smell to me and I am wondering what I should do to make this function cleaner. How can I separate out the concerns here? Also the fact that the log function goes away and logs information means that really I should wrap that call in a IO monad and call some unsafePerformIO function on it. Any ideas welcome.
What you're describing sounds more like debugging than logging. For example, Haskell's Debug.Trace.trace does exactly that and its documentation states: "These can be useful for investigating bugs or performance problems. They should not be used in production code."
If you're doing logging, the logging function should only log and have no further return value. As mentioned by #Bartek above, its type would be A -> IO (), i.e. returning no information () and having side-effects (IO). For example Haskell's hslogger library provides such functions.
I am trying to understand better reflection in Smalltalk. I am using the latest version of Squeak (v4.3). I want to intercept every message sent to instances of one of my classes. I assumed that I could override the method ProtoObject>>withArgs:executeMethod but Stéphane Ducasse explained me that for performance reason, this method is not used (this is my own summary of his answer). Which method should I override / how could intercept sent messages?
Here is the code of my attempt:
Object subclass: #C
instanceVariableNames: 'i'
classVariableNames: ''
poolDictionaries: ''
category: 'CSE3009'.
C class compile: 'newWithi: anInt
^(self new) i: anInt ; yourself.'.
C compile: 'withArgs: someArgs executeMethod: aMethod
Transcript show: ''Caught: ''.
^ super withArgs: someArgs executeMethod aMethod.'.
C compile: 'foo: aText
Transcript show: aText.
Transcript show: i.
Transcript cr.'.
C compile: 'i: anInt
i := anInt.'.
o := C newWithi: 42.
o foo: 'This is foo: '.
Executing this entire piece of code yields:
This is foo: 42
When I would like to have:
Caught: This is foo: 42
There's no build-in way to intercept messages to objects like that. There are two ways we commonly use to do this kind of trick.
First, you can create a wrapper object which responds to doesNotUnderstand:. This object usually has nil for the superclass so it doesn't inherit any instance methods from Object. The doesNotUnderstand: handler would delegate all its messages to the target object. It has the option of performing code before and after the call. All references to the original object would now point to the new "proxy" object. Messages to self wouldn't be intercepted and the proxy would need to test for objects that return self and change the returned object to be the proxy instead.
The second approach is to use a mechanism called Method Wrappers. Method Wrappers allows you to replace all of the methods in a set of classes with methods that do some other operations before and after calling the original method. This approach can provide fairly seemless results and intercepts all messages including those send to self.
MethodWrappers is available for VisualWorks and VASmalltalk. I believe it's also available for Squeak and Pharo but I'm not positive.
The three main techniques are:
Dynamic proxies
Method wrapper
Bytecode instrumentation
For a good comparision of all possible approaches, have a look at "Evaluating Message Passing Control Techniques in Smalltalk" by Stephane Ducasse (you already know him, apparently).
Of interest is also "Smalltalk: A Reflective Langauge" by F. Rivard, that shows how to implement pre- and post-conditions using bytecode rewriting. This is also a form of interception.