Cannot convert Cython memoryviewslice to ndarray - multidimensional-array

I am trying to write an explicit Successive Overrelaxation Function over a 2D matrix. In this case for an electrostatic potential.
When trying to optimize this in Cython I seem to get an error that I am not quite sure I understand.
%%cython
cimport cython
import numpy as np
cimport numpy as np
from libc.math cimport pi
#SOR function
#cython.boundscheck(False)
#cython.wraparound(False)
#cython.initializedcheck(False)
#cython.nonecheck(False)
def SOR_potential(np.float64_t[:, :] potential, mask, int max_iter, float error_threshold, float alpha):
#the ints
cdef int height = potential.shape[0]
cdef int width = potential.shape[1] #more general non quadratic
cdef int it = 0
#the floats
cdef float error = 0.0
cdef float sor_adjustment
#the copy array we will iterate over and return
cdef np.ndarray[np.float64_t, ndim=2] input_matrix = potential.copy()
#set the ideal alpha if user input is 0.0
if alpha == 0.0:
alpha = 2/(1+(pi/((height+width)*0.5)))
#start the SOR loop. The for loops omit the 0 and -1 index\
#because they are *shadow points* used for neuman boundary conditions\
cdef int row, col
#iteration loop
while True:
#2-stencil loop
for row in range(1, height-1):
for col in range(1, width-1):
if not(mask[row][col]):
potential[row][col] = 0.25*(input_matrix[row-1][col] + \
input_matrix[row+1][col] + \
input_matrix[row][col-1] + \
input_matrix[row][col+1])
sor_adjustment = alpha * (potential[row][col] - input_matrix[row][col])
input_matrix[row][col] = sor_adjustment + input_matrix[row][col]
error += np.abs(input_matrix[row][col] - potential[row][col])
#by the end of this loop input_matrix and potential have diff values
if error<error_threshold:
break
elif it>max_iter:
break
else:
error = 0
it = it + 1
return input_matrix, error, it
and I used a very simple example for an array to see if it would give an error output.
test = [[True, False], [True, False]]
pot = np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float64)
SOR_potential(pot, test, 50, 0.1, 0.0)
Gives out this error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [30], line 1
----> 1 SOR_potential(pot, test, 50, 0.1, 0.0)
File _cython_magic_6c09a5060df996862b8e35adacc0e25c.pyx:21, in _cython_magic_6c09a5060df996862b8e35adacc0e25c.SOR_potential()
TypeError: Cannot convert _cython_magic_6c09a5060df996862b8e35adacc0e25c._memoryviewslice to numpy.ndarray
But when I delete the np.float64_t[:, :] part from
def SOR_potential(np.float64_t[:, :] potential,...)
the code works. Of course, the simple 2x2 matrix will not converge but it gives no errors. Where is the mistake here?
I also tried importing the modules differently as suggested here
Cython: how to resolve TypeError: Cannot convert memoryviewslice to numpy.ndarray?
but I got 2 errors instead of 1 where there were type mismatches.
Note: I would also like to ask, how would I define a numpy array of booleans to put in front of the "mask" input in the function?

A minimal reproducible example of your error message would look like this:
def foo(np.float64_t[:, :] A):
cdef np.ndarray[np.float64_t, ndim=2] B = A.copy()
# ... do something with B ...
return B
The problem is, that A is a memoryview while B is a np.ndarray. If both A and B are memoryviews, i.e.
def foo(np.float64_t[:, :] A):
cdef np.float64_t[:, :] B = A.copy()
# ... do something with B ...
return np.asarray(B)
your example will compile without errors. Note that you then need to call np.asarray if you want to return a np.ndarray.
Regarding your second question: You could use a memoryview with dtype np.uint8_t
def foo(np.float64_t[:, :] A, np.uint8_t[:, :] mask):
cdef np.float64_t[:, :] B = A.copy()
# ... do something with B and mask ...
return np.asarray(B)
and call it like this from Python:
mask = np.array([[True, True], [False, False]], dtype=bool)
A = np.ones((2,2), dtype=np.float64)
foo(A, mask)
PS: If your array's buffers are guaranteed to be C-Contiguous, you can use contiguous memoryviews for better performance:
def foo(np.float64_t[:, ::1] A, np.uint8_t[:, ::1] mask):
cdef np.float64_t[:, ::1] B = A.copy()
# ... do something with B and mask ...
return np.asarray(B)

Related

RuntimeError: You should not call `__bool__` / `__nonzero__` on `Formula`,

I was trying to write a least time control code, using drake toolbox. But in the middle, I cannot understand the error info: (please ignore things happened in this parentheis, i just don't know how much detail is needed to submit the post, god!)
'''python
from pydrake.all import MathematicalProgram, Solve
import numpy as np
def g(x):
if abs(x)<1e-7:
return 0.
else:
return 1.
mp = MathematicalProgram()
state_initial = np.asarray([1., 0])
position_goal = np.asarray([0, 0])
N=100
dt=0.01
u_over_time=mp.NewContinuousVariables(1,"u_0")
states_over_time = np.asarray([state_initial])
for k in range(1,N):
u = mp.NewContinuousVariables(1, "u_%d" % k)
state =mp.NewContinuousVariables(2,"state_%d" % k)
u_over_time = np.vstack((u_over_time, u))
states_over_time = np.vstack((states_over_time,state))
print "Number of decision vars", mp.num_vars()
for i in range(N-1):
state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
state_next1 = states_over_time[i,1]+ dt*u_over_time[i]
mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1)
mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1)
mp.AddLinearConstraint(u_over_time[i]<=1.)
mp.AddLinearConstraint(u_over_time[i]>=-1.)
And the error info is :
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-2-be1aa565be42> in <module>()
29 state_next1 = states_over_time[i,1]+ dt*u_over_time[i]
30 mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0)
---> 31 mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1)
32 mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0)
33 mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1)
RuntimeError: You should not call `__bool__` / `__nonzero__` on `Formula`. If you are trying to make a map with `Variable`, `Expression`, or `Polynomial` as keys (and then access the map in Python), please use pydrake.common.containers.EqualToDict`.
May I know what's happening here? Thanks
----------------update line-----------------
I modified the code as you told me. Now the code now becomes:
'''python
from pydrake.all import MathematicalProgram, Solve
import numpy as np
def g(x):
if abs(x)<1e-7:
return 0.
else:
return 1.
mp = MathematicalProgram()
state_initial = np.asarray([1., 0])
position_goal = np.asarray([0, 0])
N=100
dt=0.01
u_over_time=mp.NewContinuousVariables(1,"u_0")
states_over_time = np.asarray([state_initial])
for k in range(1,N):
u = mp.NewContinuousVariables(1, "u_%d" % k)
state =mp.NewContinuousVariables(2,"state_%d" % k)
u_over_time = np.vstack((u_over_time, u))
states_over_time = np.vstack((states_over_time,state))
print "Number of decision vars", mp.num_vars()
for i in range(N-1):
state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
state_next1 = states_over_time[i,1]+ dt*u_over_time[i,0]
mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0[0])
mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1[0])
mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0[0])
mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1[0])
mp.AddLinearConstraint(u_over_time[i,0]<=1.)
mp.AddLinearConstraint(u_over_time[i,0]>=-1.)
'''
And the error info is:
TypeError Traceback (most recent call last)
<ipython-input-7-82e68c2ebfaa> in <module>()
27 state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
28 state_next1 = states_over_time[i,1]+ dt*u_over_time[i,0]
---> 29 mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0[0])
30 mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1[0])
31 mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0[0])
TypeError: 'float' object has no attribute '__getitem__'
What's the problem this time? Thanks.
(Btw, one of my complain is that, the error info always not that effective to give the hint of where the problem is...)
-----------------update 2nd time line--------------------
Now a similar problem happened to the g(x), the code:
'''
from pydrake.all import MathematicalProgram, Solve
import numpy as np
def g(x):
print 'x=',x
print 'x[0]=',x[0]
if x[0]*x[0]+x[1]*x[1]<1e-7: # x.dot(x)
return 0.
else:
return 1.
mp = MathematicalProgram()
state_initial = np.asarray([1., 0])
#position_goal = np.asarray([0, 0]) # already in g(x)
N=100
dt=0.01
u_over_time=mp.NewContinuousVariables(1,"u_0")
states_over_time = np.asarray([state_initial])
for k in range(1,N):
u = mp.NewContinuousVariables(1, "u_%d" % k)
state =mp.NewContinuousVariables(2,"state_%d" % k)
u_over_time = np.vstack((u_over_time, u))
states_over_time = np.vstack((states_over_time,state))
print "Number of decision vars", mp.num_vars()
for i in range(N-1):
state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
state_next1 = states_over_time[i,1]+ dt*u_over_time[i,0]
mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1)
mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1)
mp.AddLinearConstraint(u_over_time[i,0]<=1.)
mp.AddLinearConstraint(u_over_time[i,0]>=-1.)
reward=np.zeros((N,1))
for i in range(N):
reward[i]=g(states_over_time[i,:])
mp.AddQuadraticCost(reward.dot(reward))
result=Solve(mp)
'''
This time neither x or x[0] could solve the problem. the output info is :
Number of decision vars 298
x= [1.0 0.0]
x[0]= 1.0
x= [Variable('state_1(0)', Continuous) Variable('state_1(1)', Continuous)]
x[0]= state_1(0)
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-8-08d1cd75397e> in <module>()
37 reward=np.zeros((N,1))
38 for i in range(N):
---> 39 reward[i]=g(states_over_time[i,:])
40
41 mp.AddQuadraticCost(reward.dot(reward))
<ipython-input-8-08d1cd75397e> in g(x)
5 print 'x=',x
6 print 'x[0]=',x[0]
----> 7 if x[0]*x[0]+x[1]*x[1]<1e-7: # x.dot(x)
8 return 0.
9 else:
RuntimeError: You should not call `__bool__` / `__nonzero__` on `Formula`. If you are trying to make a map with `Variable`, `Expression`, or `Polynomial` as keys (and then access the map in Python), please use pydrake.common.containers.EqualToDict`.
What can I do this time? Thanks
Btw, you see in the code i print x or x[0] only once, but i got two different answer? funny, isn't it? why is this?
state_next1 is not a symbolic expression, it is a numpy array of symbolic expression, so you need to do state_next1[0]. Similarly you will need to change u_over_time[i] <= 1 to u_over_time[i, 0] <= 1.
The other way to solve the problem is to compute state_next1 using u_overt_time[i, 0] instead of u_over_time[i]. After modification, the for loop in your code should be
for i in range(N-1):
state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
state_next1 = states_over_time[i,1]+ dt*u_over_time[i, 0]
mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1)
mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1)
mp.AddLinearConstraint(u_over_time[i, 0]<=1.)
mp.AddLinearConstraint(u_over_time[i, 0]>=-1.)
I changed u_over_time[i] to u_over_time[i, 0] where you define state_next1.
The error thrown in the line
if x[0]*x[0]+x[1]*x[1]<1e-7: # x.dot(x)
return 0.
is because you called with AddQuadraticCost, but your cost is not quadratic. Drake tries to parse the symbolic expression as a quadratic expression, and failed. Specifically Drake fails when you check if the expression x[0] * x[0] + x[1] * x[1] < 1e-7. No quadratic cost can have this type of "if" statement.
What is the mathematical formulation of your cost? Do you really want to impose the cost as defined in your g(x) function, that if x'*x < 1e-7, then g(x) = 0, otherwise g(x) = 1? This is a pretty bad cost (it is almost constant everywhere, but have discrete jumps from 1 to 0 near the origin).
Since you want to solve a least time optimal control problem, I would suggest to change your formulation, and make dt a decision variable in your problem. Namely you will have the dynamic constraint
x[n+1] = x[n] + f(x[n], u[n]) * dt[n]
The final state constraint
x[N] = x_desired
The initial state constraint
x[0] = x_initial
And your cost function is to minimize the time
min sum_i dt[i]
Then you will have smooth cost and constraint.
Here is a piece of code that doesn't throw syntax error
from pydrake.all import MathematicalProgram, Solve
import numpy as np
def g(x):
x_squared_norm = np.power(x.reshape((2, -1)), 2)
return np.sum(x_squared_norm > 1e-7)
mp = MathematicalProgram()
state_initial = np.asarray([1., 0])
#position_goal = np.asarray([0, 0]) # already in g(x)
N=100
dt=0.01
u_over_time=mp.NewContinuousVariables(1,"u_0")
states_over_time = np.asarray([state_initial])
for k in range(1,N):
u = mp.NewContinuousVariables(1, "u_%d" % k)
state =mp.NewContinuousVariables(2,"state_%d" % k)
u_over_time = np.vstack((u_over_time, u))
states_over_time = np.vstack((states_over_time,state))
print "Number of decision vars", mp.num_vars()
for i in range(N-1):
state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
state_next1 = states_over_time[i,1]+ dt*u_over_time[i,0]
mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1)
mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1)
mp.AddLinearConstraint(u_over_time[i,0]<=1.)
mp.AddLinearConstraint(u_over_time[i,0]>=-1.)
mp.AddCost(g, vars=states_over_time[1:,:].reshape((1, -1)).squeeze())
result=Solve(mp)
Notice that I changed the definition of g, and called mp.AddCost instead of mp.AddQuadraticCost. mp.AddQuadraticCost expects a quadratic symbolic expression. The expression in your code is not quadratic (it has an if statement in the cost, and quadratic cost doesn't allow if statement.).
This code should run without error, but I don't know if it can find the solution. Again this cost is not differentiable, so any gradient based nonlinear solver will have trouble.
If you really don't want to solve the problem as a nonlinear optimization problem, you can consider to re-formulate the problem as a mixed-integer program. Namely your cost is the summation of a bunch of binary variables b[i], that b[i] = 1 if |x[i, 0]| > epsilon or |x[i, 1]| > epsilon; otherwise b[i] = 0, and your can formulate this as a mixed-integer linear constraints.
I wrote an answer by bisection method, which also recommended by tedrake on class. but I don't like this method. Too many iterations. I just put it here, when i have a mixed integer code, i will back.
god, i just cannot pass the code check...i really hate the code check machanism of stackoverflow...
'''
from pydrake.all import MathematicalProgram, Solve
import numpy as np
import matplotlib.pyplot as plt
'''
def g(x):
print 'x=',x
print 'x[0]=',x[0]
if x[0]*x[0]+x[1]*x[1]<1e-7: # x.dot(x)
return 0.
else:
return 1.
'''
#mp = MathematicalProgram()
state_initial = np.asarray([1., 0])
#position_goal = np.asarray([0, 0]) # already in g(x)
#N=201
dt=0.01
upper=1000; lower=1;
N=upper
while upper-lower>1:
print '---------------------'
print 'N=',N
mp = MathematicalProgram()
u_over_time=mp.NewContinuousVariables(1,"u_0")
states_over_time = mp.NewContinuousVariables(2,"state intial")
mp.AddLinearConstraint(states_over_time[0]==np.asarray([state_initial[0]]))
mp.AddLinearConstraint(states_over_time[1]==np.asarray([state_initial[1]]))
#states_over_time = np.asarray([state_initial])
for k in range(1,N):
u = mp.NewContinuousVariables(1, "u_%d" % k)
state =mp.NewContinuousVariables(2,"state_%d" % k)
u_over_time = np.vstack((u_over_time, u))
states_over_time = np.vstack((states_over_time,state))
print "Number of decision vars", mp.num_vars()
for i in range(N-1):
state_next0 = states_over_time[i,0]+ dt*states_over_time[i,1]
state_next1 = states_over_time[i,1]+ dt*u_over_time[i,0]
mp.AddLinearConstraint(states_over_time[i+1,0]>=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]>=state_next1)
mp.AddLinearConstraint(states_over_time[i+1,0]<=state_next0)
mp.AddLinearConstraint(states_over_time[i+1,1]<=state_next1)
mp.AddLinearConstraint(u_over_time[i,0]<=1.)
mp.AddLinearConstraint(u_over_time[i,0]>=-1.)
'''
reward=np.zeros((N,1))
for i in range(N):
reward[i]=g(states_over_time[i,:])
'''
mp.AddLinearConstraint(states_over_time[-1,0]<=1e-7)
mp.AddLinearConstraint(states_over_time[-1,1]<=1e-7)
mp.AddLinearConstraint(states_over_time[-1,0]>=1e-7)
mp.AddLinearConstraint(states_over_time[-1,1]>=1e-7)
#mp.AddQuadraticCost(reward.dot(reward))
result=Solve(mp)
print result.is_success()
if result.is_success():
upper=N
else:
lower=N
N=lower+int((upper-lower)/2.0)
N=upper
#print result.is_success()
print 'least time=',dt*N
u_over_time=result.GetSolution(u_over_time)
states_over_time=result.GetSolution(states_over_time)
#print 'u=',u_over_time
#print 'last state=',states_over_time[-1,:]
fig, ax = plt.subplots(2, 1)
plt.subplot(2, 1, 1);plt.plot(np.arange(dt, dt*N, dt),u_over_time);
plt.legend(["u against t"])
plt.subplot(2, 1, 2);plt.plot(states_over_time[:,0],states_over_time[:,1]);
plt.legend(["phase portrait"])
'''

How do I refactor this function in ELM?

I am trying to pick up functional programming and decided to start with Problem 1 on Project Euler: basically add all numbers less than 1000 divisible by 3 or 5 (link: a link).
This is the code that I have written. It outputs a list of factors of 3 or 5 (still need to figure out how to sum).
import Html exposing (text)
import Array
main =
text (
toString
[findSum_maxZ 3 5 1000]
)
findSum_maxZ x y max_z =
Array.filter isDivisible_x_or_y (Array.initialize max_z identity)
isDivisible_x_or_y x =
if x % 3 == 0 || x % 5 == 0 then True else False
My issue is that I reference 3 and 5 twice but I cannot call isDivisible with the additional parameters of the more abstract 'x' and'y'. My goal is to determine effective methods of removing these artificially mutable values so the end user only has to modify each input value once. Any advice?
I apologize if this question is dumb, there is not a lot of information on ELM available (especially compared to python, c, c++, java, etc which I have used) and I am still not fully comfortable with the functional programming jargon. Any and all help is appreciated.
The cool thing about ML languages is that you are pretty much free to build your own "dialect" to solve problems.
You can use currying to apply just the x and y arguments to your function, creating a new function where the supplied values are already set.
import Html exposing (text)
import Array
main = [findSum 3 5 1000]
|>toString
|>text
findSum x y maxZ =
let
isDivisibleByX = isDivisible x
isDivisibleByY = isDivisible y
in
Array.initialize maxZ identity
|>Array.filter isDivisibleByX
|>Array.filter isDivisibleByY
--as you can see, it is possible to use a list instead of creating
--new functions, it is up to you to check which abstraction works
--the best
isDivisible a b =
b % a == 0
You can also work with a single function, without resorting to currying:
import Html exposing (text)
import Array
main = [findSum 3 5 1000]
|>toString
|>text
findSum x y maxZ =
Array.initialize maxZ identity
|>Array.filter (\n-> isDivisible x n ) --or just (isDivisible x)
|>Array.filter (\n-> isDivisible y n)
isDivisible a b =
b % a == 0
If you want to filter the array with just one line, you can do this:
import Html exposing (text)
main = findSum 3 5 1000
|>toString
|>text
findSum x y maxZ =
let
divisibles = \n-> isDivisible x n && isDivisible y n
in
List.range 0 maxZ
|>List.filter divisibles
isDivisible a b =
b % a == 0
The most direct answer to your question is that you can have isDivisible_x_or_y take the two factors, and then use currying to pass the partially applied function to Array.filter.
That is, you can define isDivisible_x_or_y like this (I also removed the if True then True else False syntax and just return the expression directly):
isDivisible_x_or_y x y val =
val % x == 0 || val % y == 0
Currying is the ability to only supply some of the parameters to a function, and get back a function that takes the rest of the parameters. So, the type definition of isDivisible_x_or_y is Int -> Int -> Int -> Bool (that is, it takes in three Int values and returns a Bool). If we supply values for the x and y arguments (e.g. isDivisible_x_y 3 5), we now get a function with the type definition of Int -> Bool. This is the type expected by Array.filter.
You can see a working example at https://ellie-app.com/sdxWFL9ynka1
Another couple of notes:
List is much more common than Array in Elm. You would only use Array if you need to get items at specific indexes. Instead of Array.initialize, you can use List.range
Using the pipeline operator |> can often make your code a lot simpler to read. Instead of text (toString (getValue)), you have getValue |> toString |> text, which is now in the order that the operations occur, and doesn't have extra parenthesis. This whole program could be one simple pipeline (in a lot of scenarios putting everything into one pipeline can be excessive, though):
main =
List.range 0 max_z
|> List.filter (isDivisible_x_or_y 3 5)
|> toString
|> text
isDivisible_x_or_y x y val =
val % x == 0 || val % y == 0

cython - determining the number of items in pointer variable

How would I determine the number of elements in a pointer variable in cython? I saw that in C one way seems to be sizeof(ptr)/sizeof(int), if the pointer points to int variables. But that doesn't seem to work in cython. E.g. when I tried to join two memory views into a single pointer like so:
from libc.stdlib cimport malloc, free
cdef int * join(int[:] a, int[:] b):
cdef:
int n_a = a.shape[0]
int n_b = b.shape[0]
int new_size = n_a + n_b
int *joined = <int *> malloc(new_size*sizeof(int))
int i
try:
for i in range(n_a):
joined[i] = a[i]
for i in range(n_b):
joined[n_a+i] = b[i]
return joined
finally:
free(joined)
#cython.cdivision(True)
def join_memviews(int[:] n, int[:] m):
cdef int[:] arr_fst = n
cdef int[:] arr_snd = m
cdef int *arr_new
cdef int new_size
arr_new = join(arr_fst,arr_snd)
new_size = sizeof(arr_new)/sizeof(int)
return [arr_new[i] for i in range(new_size)]
I do not get the desired result when calling join_memviews from a python script, e.g.:
# in python
a = np.array([1,2])
b = np.array([3,4])
a_b = join_memviews(a,b)
I also tried using the types
DTYPE = np.int
ctypedef np.int_t DTYPE_t
as the arguement inside sizeof(), but that didn't work either.
Edit: The handling of the pointer variable was apparently a bit careless of me. I hope the following is fine (even though it might not be a prudent approach):
cdef int * join(int[:] a, int[:] b, int new_size):
cdef:
int *joined = <int *> malloc(new_size*sizeof(int))
int i
for i in range(n_a):
joined[i] = a[i]
for i in range(n_b):
joined[n_a+i] = b[i]
return joined
def join_memviews(int[:] n, int[:] m):
cdef int[:] arr_fst = n
cdef int[:] arr_snd = m
cdef int *arr_new
cdef int new_size = n.shape[0] + m.shape[0]
try:
arr_new = join(arr_fst,arr_snd, new_size)
return [arr_new[i] for i in range(new_size)]
finally:
free(arr_new)
You can't. It doesn't work in C either. sizeof(ptr) returns the amount of memory used to store the pointer (i.e. typically 4 or 8 depending on your system) rather than the length of the array. The lengths of your malloced arrays are something that you need to keep track of manually.
Additionally the following code is a recipe for disaster:
cdef int *joined = <int *> malloc(new_size*sizeof(int))
try:
return joined
finally:
free(joined)
The free happens immediately on function exit so that an invalid pointer is returned to the calling function.
You should be using properly managed Python arrays (either from numpy or the standard library array module) unless you absolutely can't avoid it.

cython static shaped array views

In cython, one can use array views, e.g.
cdef void func(float[:, :] arr)
In my usage the second dimension should always have a shape of 2. Can I tell cython this? I was thinking of something like:
cdef void func(float[:, 2] arr)
but this results in an invalid syntax; Or is it possible to have something more similar to c++, e.g.
cdef void func(tuple<float, float>[:] arr)
Thanks in advance!
You can use a 2D static array instead. Just use the pointer notation. Here is how you achieve it
def pyfunc():
# static 1D array
cdef float *arr1d = [1,-1, 0, 2,-1, -1, 4]
# static 2D array
cdef float[2] *arr2d = [[1,.2.],[3.,4.]]
# pass to a "cdef"ed function
cfunc(arr2d)
# your function signature would now look like this
cdef void cfunc(float[2] *arr2d):
print("my 2D static array")
print(arr2d[0][0],arr2d[0][1],arr2d[1][0],arr2d[1][1])
Calling it you get:
>>> pyfunc()
my 2D static array
1.0, 2.0, 3.0, 4.0
I don't think this is really supported, but if you want to do this then the best way is probably to use memoryviews of structs (which are compatible with numpys custom dtypes):
import numpy as np
cdef packed struct Pair1: # packed ensures it matches custom numpy dtypes
# (but probably doesn't matter here!)
double x
double y
# pair 1 matches arrays of this dtype
pair_1_dtype = [('x',np.float64), ('y',np.float64)]
cdef packed struct Pair2:
double data[2]
pair_2_dtype = [('data',np.float64, (2,))]
def pair_func1(Pair1[::1] x):
# do some very basic work
cdef Pair1 p
cdef Py_ssize_t i
p.x = 0; p.y = 0
for i in range(x.shape[0]):
p.x += x[i].x
p.y += x[i].y
return p # take advantage of auto-conversion to a dict
def pair_func2(Pair2[::1] x):
# do some very basic work
cdef Pair2 p
cdef Py_ssize_t i
p.data[0] = 0; p.data[1] = 0
for i in range(x.shape[0]):
p.data[0] += x[i].data[0]
p.data[1] += x[i].data[1]
return p # take advantage of auto-conversion to a dict
and a function to show you how to call it:
def call_pair_funcs_example():
# generate data of correct dtype
d = np.random.rand(100,2)
d1 = d.view(dtype=pair_1_dtype).reshape(-1)
print(pair_func1(d1))
d2 = d.view(dtype=pair_2_dtype).reshape(-1)
print(pair_func2(d2))
The thing I'd like to have done is:
ctypedef double[2] Pair3
def pair_func3(Pair3[::1] x):
# do some very basic work
cdef Pair3 p
cdef Py_ssize_t i
p[0] = 0; p[1] = 0
for i in range(x.shape[0]):
p[0] += x[i][0]
p[1] += x[i][1]
return p # ???
That compiles successfully, but I couldn't find any way of converting it from numpy. If you could work out how to get this version to work then I think it would be the most elegant solution.
Note that I'm not convinced of the performance advantages of any of these solutions. Your best move is probably to tell Cython that the trailing dimension is contiguous in memory (e.g. double [:,::1]) but let it be any size.

How do you use matrices in Nimrod?

I found this project on GitHub; it was the only search term returned for "nimrod matrix". I took the bare bones of it and changed it a little bit so that it compiled without errors, and then I added the last two lines to build a simple matrix, and then output a value, but the "getter" function isn't working for some reason. I adapted the instructions for adding properties found here, but something isn't right.
Here is my code so far. I'd like to use the GNU Scientific Library from within Nimrod, and I figured that this was the first logical step.
type
TMatrix*[T] = object
transposed: bool
dataRows: int
dataCols: int
data: seq[T]
proc index[T](x: TMatrix[T], r,c: int): int {.inline.} =
if r<0 or r>(x.rows()-1):
raise newException(EInvalidIndex, "matrix index out of range")
if c<0 or c>(x.cols()-1):
raise newException(EInvalidIndex, "matrix index out of range")
result = if x.transposed: c*x.dataCols+r else: r*x.dataCols+c
proc rows*[T](x: TMatrix[T]): int {.inline.} =
## Returns the number of rows in the matrix `x`.
result = if x.transposed: x.dataCols else: x.dataRows
proc cols*[T](x: TMatrix[T]): int {.inline.} =
## Returns the number of columns in the matrix `x`.
result = if x.transposed: x.dataRows else: x.dataCols
proc matrix*[T](rows, cols: int, d: openarray[T]): TMatrix[T] =
## Constructor. Initializes the matrix by allocating memory
## for the data and setting the number of rows and columns
## and sets the data to the values specified in `d`.
result.dataRows = rows
result.dataCols = cols
newSeq(result.data, rows*cols)
if len(d)>0:
if len(d)<(rows*cols):
raise newException(EInvalidIndex, "insufficient data supplied in matrix constructor")
for i in countup(0,rows*cols-1):
result.data[i] = d[i]
proc `[][]`*[T](x: TMatrix[T], r,c: int): T =
## Element access. Returns the element at row `r` column `c`.
result = x.data[x.index(r,c)]
proc `[][]=`*[T](x: var TMatrix[T], r,c: int, a: T) =
## Sets the value of the element at row `r` column `c` to
## the value supplied in `a`.
x.data[x.index(r,c)] = a
var m = matrix( 2, 2, [1,2,3,4] )
echo( $m[0][0] )
This is the error I get:
c:\program files (x86)\nimrod\config\nimrod.cfg(36, 11) Hint: added path: 'C:\Users\H127\.babel\libs\' [Path]
Hint: used config file 'C:\Program Files (x86)\Nimrod\config\nimrod.cfg' [Conf]
Hint: system [Processing]
Hint: mat [Processing]
mat.nim(48, 9) Error: type mismatch: got (TMatrix[int], int literal(0))
but expected one of:
system.[](a: array[Idx, T], x: TSlice[Idx]): seq[T]
system.[](a: array[Idx, T], x: TSlice[int]): seq[T]
system.[](s: string, x: TSlice[int]): string
system.[](s: seq[T], x: TSlice[int]): seq[T]
Thanks you guys!
I'd like to first point out that the matrix library you refer to is three years old. For a programming language in development that's a lot of time due to changes, and it doesn't compile any more with the current Nimrod git version:
$ nimrod c matrix
...
private/tmp/n/matrix/matrix.nim(97, 8) Error: ']' expected
It fails on the double array accessor, which seems to have changed syntax. I guess your attempt to create a double [][] accessor is problematic, it could be ambiguous: are you accessing the double array accessor of the object or are you accessing the nested array returned by the first brackets? I had to change the proc to the following:
proc `[]`*[T](x: TMatrix[T], r,c: int): T =
After that change you also need to change the way to access the matrix. Here's what I got:
for x in 0 .. <2:
for y in 0 .. <2:
echo "x: ", x, " y: ", y, " = ", m[x,y]
Basically, instead of specifying two bracket accesses you pass all the parameters inside a single bracket. That code generates:
x: 0 y: 0 = 1
x: 0 y: 1 = 2
x: 1 y: 0 = 3
x: 1 y: 1 = 4
With regards to finding software for Nimrod, I would like to recommend you using Nimble, Nimrod's package manager. Once you have it installed you can search available and maintained packages. The command nimble search math shows two potential packages: linagl and extmath. Not sure if they are what you are looking for, but at least they seem more fresh.

Resources