When generating a not explicitly generated version of a function, #ngenerate runs
eval(quote
local _F_
$localfunc # Definition of _F_ for the requested value of N
_F_
end)
Since eval runs in the scope of the current module, not the function, I wonder what is the effect of local in this context. As far as I know, the languange documentation only mentions the use of local inside function definitions.
To give some background why this question arose: I frequently need to code loops of the form
function foo(n::Int)
s::Int = 0
for i in 1:1000000000
for j in 1:n
s += 1
end
end
return s
end
where n <= 10 (of course, in my actual code the loops are such that they cannot just be reduced to O(1)). Because this code is very simple for the compiler but demanding at runtime, it turns out to be beneficial to simply recompile the loops with the required value of n each time foo is called.
function clever_foo(n::Int)
eval(quote
function clever_foo_impl()
s::Int = 0
for i in 1:1000000000
s += $(Expr(:call,:+,[1 for j in 1:n]...))
end
return s
end
end)
return clever_foo_impl()
end
However, I am not sure whether I am doing this the right way.
It's to prevent _F_ from being visible in the global method cache.
If you'll call clever_foo with the same n repeatedly, you can do even better by saving the compiled function in a Dict. That way you don't have to recompile it each time.
Related
I am coming from Fortran to Julia. I know in Julia want to prevent using global variables.
But the thing is, how to prevent using global variables in a module?
For example, in the following module,
module Mod
global AAA=zeros(1000000000)
function f(x)
change the most up to date AAA with x in some way.
return nothing
end
function g(x)
using the most up to date AAA, then change/update AAA with x in some way.
return nothing
end
end
In the above example, I need to update the very big array AAA, so I put AAA in the global. So f(x) and g(x) can use the most updated AAA and further update them.
Now since Julia discourage the use of global variable, so in my case, what do I do?
I mean, do I put AAA just in the argument of f(x) and g(x) such that they become f(x,AAA) and g(x,AAA)?
Is passing a very big array like AAA in the argument really faster than putting AAA just as a global variable?
It is possible to pass the array in question to the functions.
Here is an updated version of your pseudo code.
module Mod
AAA=zeros(1000000000)
function f!(ba, x) # the ! mark to indicate that ba will be updated
ba[1]=x
return nothing
end
function g!(ba, x)
ba[1] +=x
return nothing
end
function example()
f!(AAA,1)
g!(AAA,2)
#show(AAA[1])
end
end
I am using my phone, so there could be some typos, and I can't benchmark, but you could do it if you want to convince yourself there is no penalty in passing the array as an argument.
It is common practice to add a ! when the function mutates the content of the argument. Also, the arguments changed are put first in the list. Of course, these are only conventions, but it makes it easier for others to understand the intent of your code.
So here's the code I was trying to execute in Julia:
i = begin
i = 5
while(i<=10)
println(i)
i+=1
end
end
It's just basically a simple code to print the value of i from 5 to 10 but it's messing with me
Are you in the REPL? What you are probably running into is that begin does not introduce its own scope, so i = 5 declares i as a global variable. Because while does introduce its own scope, if you do println(i), it only looks for i in its local scope where it is not defined, because i exists only as a global variable. You can add a line global i at the beginning of the body of the while loop to tell all code after that to use the global i, but note that global variables come with their own performance caveats. An arguably better solution would be to use let instead of begin, which does introduce a new scope, but note that then you can of course not access i afterwards, because it is now only local to the let block.
This behavior will actually be changed in the upcoming release of Julia 1.5, so your code should then just work.
Your issue is scope. When you enter into a loop, variables created inside the loop are local to the loop and destroyed after it exits. i is not currently defined inside your while loop, so you get the error. The quick fix is to tell Julia you want the loop to have access to the global i variable you defined at the top by adding global i immediately after the while statement. You also don't need the begin block, and naming the block i is immediately overwritten by the next statement defining i.
This particular example function is in Lua, but I think the main concepts are true for any language with lexical scoping, first-class functions, and iterators.
Code Description (TL;DR -- see code):
The code below defines an iterator constructor which merely defines a local value and returns an iterator function.
This iterator function, when run, uses the local value from the constructor and increases that value by 1. It then runs itself recursively until the value reaches 5, then adds 1 to the value and returns the number 5. If it's run again, it will run itself recursively until the value reaches 20 or higher, then it returns nil, which is the signal for the loop to stop.
I then run an outer loop using the iterator provided by the constructor, which will run the body of the loop when the iterator returns a value (5). In the body of the outer loop I put an inner loop that also uses an iterator provided by the constructor.
The program should run the body of the loop once (when the value in the outer loop iterator hits 5) and then run the inner loop completely (letting the inner loop value hit 20), then return back to the outer loop and run it until it finishes.
function iterConstr()
local indexes = {0}
function iter()
print(indexes[1])
indexes[1] = indexes[1] + 1
if indexes[1] == 5 then
indexes[1] = indexes[1] + 1
return 5
elseif indexes[1]>=21 then
print("finished!")
return nil
else
print("returning next one")
return iter()
end
end
return iter
end
for val in iterConstr() do
for newVal in iterConstr() do
end
print("big switch")
end
The behavior I did not expect is that the value from the inner loop and outer loop seem to be linked together. When focus is returned to the outer loop after running through the inner loop, it runs with the expected next value for the outer loop (6), but then instead of iterating and incrementing up to 20, it instead jumps immediately to 21, which is where the inner loop ended!
Can anyone help explain why this bizarre behavior occurs?
I believe your problem lies on line 3 - you declare iter as a global function instead of a local one, which makes it accessible from any Lua chunk. Changing that line to local function iter() corrects this behaviour. I think therefore that this is more of an issue with the developer applying lexical scoping than with lexical scoping itself :)
I would like to write a macro #unpack t which takes an object t and copies all its fields into local scope. For example, given
immutable Foo
i::Int
x::Float64
end
foo = Foo(42,pi)
the expression #unpack foo should expand into
i = foo.i
x = foo.x
Unfortunately, such a macro cannot exist since it would have to know the type of the passed object. To circumvent this limitation, I introduce a type-specific macro #unpackFoo foo with the same effect, but since I'm lazy I want the compiler to write #unpackFoo for me. So I change the type definition to
#unpackable immutable Foo
i::Int
x::Float64
end
which should expand into
immutable Foo
i::Int
x::Float64
end
macro unpackFoo(t)
return esc(quote
i = $t.i
x = $t.x
end)
end
Writing #unpackable is not too hard:
macro unpackable(expr)
if expr.head != :type
error("#unpackable must be applied on a type definition")
end
name = isa(expr.args[2], Expr) ? expr.args[2].args[1] : expr.args[2]
fields = Symbol[]
for bodyexpr in expr.args[3].args
if isa(bodyexpr,Expr) && bodyexpr.head == :(::)
push!(fields,bodyexpr.args[1])
elseif isa(bodyexpr,Symbol)
push!(fields,bodyexpr)
end
end
return esc(quote
$expr
macro $(symbol("unpack"*string(name)))(t)
return esc(Expr(:block, [:($f = $t.$f) for f in $fields]...))
end
end)
end
In the REPL, this definition works just fine:
julia> #unpackable immutable Foo
i::Int
x::Float64
end
julia> macroexpand(:(#unpackFoo foo))
quote
i = foo.i
x = foo.x
end
Problems arise if I put the #unpackFoo in the same compilation unit as the #unpackable:
julia> #eval begin
#unpackable immutable Foo
i::Int
x::Float64
end
foo = Foo(42,pi)
#unpackFoo foo
end
ERROR: UndefVarError: #unpackFoo not defined
I assume the problem is that the compiler tries to proceed as follows
Expand #unpackable but do not parse it.
Try to expand #unpackFoo which fails because the expansion of #unpackable has not been parsed yet.
If we wouldn't fail already at step 2, the compiler would now parse the expansion of #unpackable.
This circumstance prevents #unpackable from being used in a source file. Is there any way of telling the compiler to swap steps 2. and 3. in the above list?
The background to this question is that I'm working on an iterator-based implementation of iterative solvers in the spirit of https://gist.github.com/jiahao/9240888. Algorithms like MinRes require quite a number of variables in the corresponding state object (8 currently), and I neither want to write state.variable every time I use a variable in e.g. the next() function, nor do I want to copy all of them manually as this bloats up the code and is hard to maintain. In the end, this is mainly an exercise in meta-programming though.
Firstly, I would suggest writing this as:
immutable Foo
...
end
unpackable(Foo)
where unpackable is a function which takes the type, constructs the appropriate expression and evals it. There are a couple of advantages to this, e.g. that you can apply it to any type without it being fixed at definition time, and the fact that you don't have to do a bunch of parsing of the type declaration (you can just call fieldnames(Foo) == [:f, :i] and work with that).
Secondly, while I don't know your use case in detail (and dislike blanket rules) I will warn that this kind of thing is frowned upon. It makes code harder to read because it introduces a non-local dependency; suddenly, in order to know whether x is a local or global variable, you have to look up the definition of a type in a whole different file. A better, and more general, approach is to explicitly unpack variables, and this is available in MacroTools.jl via the #destruct macro:
#destruct _.(x, i) = myfoo
# now we can use x and i
(You can destruct nested data structures and indexable objects too, which is nice.)
To answer your question: you're essentially right about how Julia runs code (s/parse/evaluate). The whole block is parsed, expanded and evaluated together, which means in your example you're trying to expand #unpackFoo before it's been defined.
However, when loading a .jl file, Julia evaluates blocks in the file one at a time, rather than all at once.
This means that you can happily write a file like this:
macro foo()
:(println("hi"))
end
#foo()
and run julia foo.jl or include("foo.jl") and it will run fine. You just can't have a macro definition and its use in the same block, as in your begin block above.
Try having a look at Parameters package by Mauro (https://github.com/mauro3/Parameters.jl). It has an #unpack macro and the accompanying machinery similar to what you suggest you need.
I saw this question:
Fortran dynamic objects
and the accepted answer made me question if I wrote the following function safely (without allowing a memory leak)
function getValues3D(this) result(vals3D)
implicit none
type(allBCs),intent(in) :: this
real(dpn),dimension(:,:,:),pointer :: vals3D
integer,dimension(3) :: s
if (this%TF3D) then
s = shape(this%vals3D)
if (associated(this%vals3D)) then
stop "possible memory leak - p was associated"
endif
allocate(vals3D(s(1),s(2),s(3)))
vals3D = this%vals3D
else; call propertyNotAssigned('vals3D','getValues3D')
endif
end function
This warning shows up when I run my code, but shouldn't my this%vals3D be associated if it was previously (to this function) set? I'm currently running into memory errors, and they started showing up when I introduced a new module with this function in it.
Any help is greatly appreciated.
I think I wasn't specific enough. I would like to make the following class, and know how to implement the class, safely in terms of memory. That is:
module vectorField_mod
use constants_mod
implicit none
type vecField1D
private
real(dpn),dimension(:),pointer :: x
logical :: TFx = .false.
end type
contains
subroutine setX(this,x)
implicit none
type(vecField1D),intent(inout) :: this
real(dpn),dimension(:),target :: x
allocate(this%x(size(x)))
this%x = x
this%TFx = .true.
end subroutine
function getX(this) result(res)
implicit none
real(dpn),dimension(:),pointer :: res
type(vecField1D),intent(in) :: this
nullify(res)
allocate(res(size(this%x)))
if (this%TFx) then
res = this%x
endif
end function
end module
Where the following code tests this module
program testVectorField
use constants_mod
use vectorField_mod
implicit none
integer,parameter :: Nx = 150
real(dpn),parameter :: x_0 = 0.0
real(dpn),parameter :: x_N = 1.0
real(dpn),parameter :: dx = (x_N - x_0)/dble(Nx-1)
real(dpn),dimension(Nx) :: x = (/(x_0+dble(i)*dx,i=0,Nx-1)/)
real(dpn),dimension(Nx) :: f
real(dpn),dimension(:),pointer :: fp
type(vecField1D) :: f1
integer :: i
do i=1,Nx
f(i) = sin(x(i))
enddo
do i=1,10**5
call setX(f1,f) !
f = getX(f1) ! Should I use this?
fp = getX(f1) ! Or this?
fp => getX(f1) ! Or even this?
enddo
end program
Currently, I'm running on windows. When I CTR-ALT-DLT, and view performance, the "physical memory usage histery" increases with every loop iteration. This is why I assume that I have a memory leak.
So I would like to repose my question: Is this a memory leak? (The memory increases with every one of the above cases). If so, is there a way I avoid the memory leak while still using pointers? If not, then what is happening, should I be concerned and is there a way to reduce the severity of this behavior?
Sorry for the initial vague question. I hope this is more to the point.
Are you really restricted to Fortran 90? In Fortran 2003 you would use an allocatable function result for this. This is much safer. Using pointer function results, whether you have a memory leak with this code or not depends on how you reference the function, which you don't show. If you must return a pointer from a procedure, it is much safer to return it via a subroutine argument.
BUT...
This function is pointless. There's no point testing the association status of this%vals3D` after you've referenced it as the argument to SHAPE in the previous line. If the pointer component is disassocated (or has undefined pointer association status), then you are not permitted to reference it.
Further, if the pointer component is associated, all you do is call stop!
Perhaps you have transcribed the code to the question incorrectly?
If you simply delete the entire if construct starting with if (associated(this%vals3D))... then your code may make sense.
BUT...
if this%TF3D is true, then this%vals3D must be associated.
when you reference the function, you must use pointer assignment
array_ptr => getValues3D(foo)
! ^
! |
! + this little character is very important.
Forget that little character and you are using normal assignment. Syntactically valid, difficult to pick the difference when reading code and, in this case, potentially a source of memory corruption or leaks that might go undetected until the worst possible moment, in addition to the usual pitfalls of using pointers (e.g. you need to DEALLOCATE array_ptr before you reuse it or it goes out of scope). This is why functions returning pointer results are considered risky.
Your complete code shows several memory leaks. Every time you allocate something that is a POINTER - you need to pretty much guarantee that there will be a matching DEALLOCATE.
You have a loop in your test code. ALLOCATE gets called a lot - in both the setter and the getter. Where are the matching DEALLOCATE statements?
Every time setX is called, any previously allocated memory for the x component of your type will be leaked. Since you call the function 10^5 times, you will waste 100000-1 copies. If you know that the size of this%x will never change, simply check to see if a previous call had already allocated the memory by checking to see if ASSOCIATED(this%x) is true. If it is, skip the allocation and move directly to the assignment statement. If the size does change, then you will first have to deallocate the old copy before allocating new space.
Two other minor comments on setX: The TARGET attribute of the dummy argument x appears superfluous since you never take a pointer of that argument. Second, the TFx component of your type also seems superfluous since you can instead check if x is allocated.
For the function getX, why not skip the allocation completely, and merely set res => this%x? Admittedly, this will return a direct reference to the underlying data, which maybe you want to avoid.
In your loop,
do i=1,10**5
call setX(f1,f) !
f = getX(f1) ! Should I use this?
fp = getX(f1) ! Or this?
fp => getX(f1) ! Or even this?
enddo
fp => getX(f1) will allow you to obtain a pointer to the underlying x component of your type (if you adopt my change above). The other two use assignment operators and will copy data from the result of getX into either f, or (if it is previously allocated) fp. If fp is not allocated, the code will crash.
If you do not want to grant direct access to the underlying data, then I suggest that the return value of getX should be defined as an automatic array with the size determined by this%x. That is, you can write the function as
function getX(this) result(res)
implicit none
type(vecField1D),intent(in) :: this
real(dpn),dimension(size(this%x,1)) :: res
res = this%x
end function