I have created a custom python sync block for use in a gnuradio flowgraph. The block tests for invalid input and, if found, raises a ValueError exception. I would like to create a unit test to verify that the exception is raised when the block indeed receives invalid input data.
As part of the python-based qa test for this block, I created a flowgraph such that the block receives invalid data. When I run the test, the block does appear to raise the exception but then hangs.
What is the appropriate way to test for this? Here is a minimal working example:
#!/usr/bin/env python
import numpy as np
from gnuradio import gr, gr_unittest, blocks
class validate_input(gr.sync_block):
def __init__(self):
gr.sync_block.__init__(self,
name="validate_input",
in_sig=[np.float32],
out_sig=[np.float32])
self.max_input = 100
def work(self, input_items, output_items):
in0 = input_items[0]
if (np.max(in0) > self.max_input):
raise ValueError('input exceeds max.')
validated_in = output_items[0]
validated_in[:] = in0
return len(output_items[0])
class qa_validate_input (gr_unittest.TestCase):
def setUp (self):
self.tb = gr.top_block ()
def tearDown (self):
self.tb = None
def test_check_valid_data(self):
src_data = (0, 201, 92)
src = blocks.vector_source_f(src_data)
validate = validate_input()
snk = blocks.vector_sink_f()
self.tb.connect (src, validate)
self.tb.connect (validate, snk)
self.assertRaises(ValueError, self.tb.run)
if __name__ == '__main__':
gr_unittest.run(qa_validate_input, "qa_validate_input.xml")
which produces:
DEPRECATED: Using filename with gr_unittest does no longer have any effect.
handler caught exception: input exceeds max.
Traceback (most recent call last):
File "/home/xxx/devel/gnuradio3_8/lib/python3.6/dist-packages/gnuradio/gr/gateway.py", line 60, in eval
try: self._callback()
File "/home/xxx/devel/gnuradio3_8/lib/python3.6/dist-packages/gnuradio/gr/gateway.py", line 230, in __gr_block_handle
) for i in range(noutputs)],
File "qa_validate_input.py", line 21, in work
raise ValueError('input exceeds max.')
ValueError: input exceeds max.
thread[thread-per-block[1]: <block validate_input(2)>]: SWIG director method error. Error detected when calling 'feval_ll.eval'
^CF
======================================================================
FAIL: test_check_valid_data (__main__.qa_validate_input)
----------------------------------------------------------------------
Traceback (most recent call last):
File "qa_validate_input.py", line 47, in test_check_valid_data
self.assertRaises(ValueError, self.tb.run)
AssertionError: ValueError not raised by run
----------------------------------------------------------------------
Ran 1 test in 1.634s
FAILED (failures=1)
The top_block's run() function does not call the block's work() function directly but starts the internal task scheduler and its threads and waits them to finish.
One way to unit test the error handling in your block is to call the work() function directly
def test_check_valid_data(self):
src_data = [[0, 201, 92]]
output_items = [[]]
validate = validate_input()
self.assertRaises(ValueError, lambda: validate.work(src_data, output_items))
I am working on a traveling salesman problem. Given that all agents traverse the same graph to find their own path separately, i am trying to parallelize the path-finding action of agents. the task is for each iteration, all agents start from a start node to find their paths and collect all the paths to find the best path in the current iteration.
I am using pathos.multiprocessing.
the agent class has a traverse method as,
class Agent:
def find_a_path(self, graph):
# here is the logic to find a path by traversing the graph
return found_path
I create a helper function to wrap up the method
def do_agent_find_a_path(agent, graph):
return agent.find_a_path(graph)
then create a pool and employ amap by passing the helper function, a list of agent instance and the same graph,
pool = ProcessPool(nodes = 10)
res = pool.amap(do_agent_find_a_path, agents, [graph] * len(agents))
but, the processes are created in sequence and it runs very slow. I'd like to have some instructions on a correct/decent way to leverage pathos in this situation.
thank you!
UPDATE:
I am using pathos 0.2.3 on ubuntu,
Name: pathos
Version: 0.2.3
Summary: parallel graph management and execution in heterogeneous computing
Home-page: https://pypi.org/project/pathos
Author: Mike McKerns
i get the following error with the TreadPool sample code:
>import pathos
>pathos.pools.ThreadPool().iumap(lambda x:x*x, [1,2,3,4])
Traceback (most recent call last):
File "/opt/anaconda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-5-f8f5e7774646>", line 1, in <module>
pathos.pools.ThreadPool().iumap(lambda x:x*x, [1,2,3,4])
AttributeError: 'ThreadPool' object has no attribute 'iumap'```
I'm the pathos author. I'm not sure how long your method takes to run, but from your comments, I'm going to assume not very long. I'd suggest that, if the method is "fast", that you use a ThreadPool instead. Also, if you don't need to preserve the order of the results, the fastest map is typically uimap (unordered, iterative map).
>>> class Agent:
... def basepath(self, dirname):
... import os
... return os.path.basename(dirname)
... def slowpath(self, dirname):
... import time
... time.sleep(.2)
... return self.basepath(dirname)
...
>>> a = Agent()
>>> import pathos.pools as pp
>>> dirs = ['/tmp/foo', '/var/path/bar', '/root/bin/bash', '/tmp/foo/bar']
>>> import time
>>> p = pp.ProcessPool()
>>> go = time.time(); tuple(p.uimap(a.basepath, dirs)); print(time.time()-go)
('foo', 'bar', 'bash', 'bar')
0.006751060485839844
>>> p.close(); p.join(); p.clear()
>>> t = pp.ThreadPool(4)
>>> go = time.time(); tuple(t.uimap(a.basepath, dirs)); print(time.time()-go)
('foo', 'bar', 'bash', 'bar')
0.0005156993865966797
>>> t.close(); t.join(); t.clear()
and, just to compare against something that takes a bit longer...
>>> t = pp.ThreadPool(4)
>>> go = time.time(); tuple(t.uimap(a.slowpath, dirs)); print(time.time()-go)
('bar', 'bash', 'bar', 'foo')
0.2055649757385254
>>> t.close(); t.join(); t.clear()
>>> p = pp.ProcessPool()
>>> go = time.time(); tuple(p.uimap(a.slowpath, dirs)); print(time.time()-go)
('foo', 'bar', 'bash', 'bar')
0.2084510326385498
>>> p.close(); p.join(); p.clear()
>>>
I am trying to understand process communication in erlang. Here I have a master process and five friends process. If a friend sends a message to any of the other 5 they have to reply back. But the master should be aware of all this. I am pasting the code below.
-module(prog).
-import(lists,[append/2,concat/1]).
-import(maps,[from_lists/1,find/2,get/2,update/3]).
-import(string,[equal/2]).
-import(file,[consult/1]).
-export([create_process/1,friends/4, master/1, main/0,prnt/1]).
%% CREATE PROCESS
create_process([])->ok;
create_process([H|T])->
{A,B} = H,
Pid = spawn(prog,friends,[B,self(),0,A]),
register(A,Pid),
create_process(T).
%% FRIENDS PROCESS
friends(Msg, M_pid, State, Self_name)->
S = lists:concat([Self_name," state =",State,"\n"]),
io:fwrite(S),
if
State == 0 ->
timer:sleep(500),
io:fwrite("~p~n",[Self_name]),
lists:foreach(fun(X) -> whereis(X)!{Self_name,"intro",self()} end, Msg),
friends(Msg, M_pid, State + 1, Self_name);
State > 0 ->
receive
{Process_name, Process_msg, Process_id} ->
I = equal(Process_msg,"intro"),
R = equal(Process_msg,"reply"),
XxX = lists:concat([Self_name," recieved ",Process_msg," from ",Process_name,"\n"]),
io:fwrite(XxX),
if
I == true ->
io:fwrite("~p~n",[whereis(Process_name)]),
M_pid!{lists:concat([Self_name," received intro message from ", Process_name , "[",Process_id,"]"]), self()},
io:fwrite(I),
whereis(Process_name)!{Self_name, "reply",self()},
friends(Msg, M_pid, State + 1, Self_name);
R == true ->
M_pid!{lists:concat([Self_name," received reply message from ", Process_name , "[",Process_id,"]"]), self()},
io:fwrite(R),
friends(Msg, M_pid, State + 1, Self_name)
end
after
1000->
io:fwrite(lists:concat([Self_name," has received no calls for 1 second, ending..."]))
end
end.
master(State)->
receive
{Process_message, Process_id} ->
io:fwrite(Process_message),
master(State+1)
after
2000->
ok
end.
main() ->
B = [{john, [jill,joe,bob]},
{jill, [bob,joe,bob]},
{sue, [jill,jill,jill,bob,jill]},
{bob, [john]},
{joe, [sue]}],
create_process(B),
io:fwrite("~p~n",[whereis(sue)]),
master(0).
I think the line in friends() function,
M_pid!{lists:concat([Self_name," received intro message from ", Process_name , "[",Process_id,"]"]), self()}
is the cause of error but I cannot understand why. M_pid is known and I am concatenating all the info and sending it to master but I am confused why it isnt working.
The error I am getting is as follows:
Error in process <0.55.0> with exit value: {function_clause,[{lists,thing_to_list,
[<0.54.0>],
[{file,"lists.erl"},{line,603}]},
{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},
{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},
{prog,friends,4,[{file,"prog.erl"},{line,45}]}]}
I dont know what is causing the error. Sorry for asking noob questions and thanks for your help.
An example of what Dogbert discovered:
-module(my).
-compile(export_all).
go() ->
Pid = spawn(my, nothing, []),
lists:concat(["hello", Pid]).
nothing() -> nothing.
In the shell:
2> c(my).
my.erl:2: Warning: export_all flag enabled - all functions will be exported
{ok,my}
3> my:go().
** exception error: no function clause matching
lists:thing_to_list(<0.75.0>) (lists.erl, line 603)
in function lists:flatmap/2 (lists.erl, line 1250)
in call from lists:flatmap/2 (lists.erl, line 1250)
4>
But:
-module(my).
-compile(export_all).
go() ->
Pid = spawn(my, nothing, []),
lists:concat(["hello", pid_to_list(Pid)]).
nothing() -> nothing.
In the shell:
4> c(my).
my.erl:2: Warning: export_all flag enabled - all functions will be exported
{ok,my}
5> my:go().
"hello<0.83.0>"
From the erl docs:
concat(Things) -> string()
Things = [Thing]
Thing = atom() | integer() | float() | string()
The list that you feed concat() must contain either atoms, integers, floats, or strings. A pid is neither an atom, integer, float, nor string, so a pid cannot be used with concat(). However, pid_to_list() returns a string:
pid_to_list(Pid) -> string()
Pid = pid()
As you can see, a pid has its own type: pid().
I ran your code.
Where you went wrong was to pass Process_id(which is of type pid()) to lists:concat/1.
Let us try to understand this error:
{function_clause,[{lists,thing_to_list,
[<0.84.0>],
[{file,"lists.erl"},{line,603}]},
{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},
{lists,flatmap,2,[{file,"lists.erl"},{line,1250}]},
{prog,friends,4,[{file,"prog.erl"},{line,39}]}]}
It states the function lists:thing_to_list/1 has no definition(see the word function_clause in the error log) which accepts an argument of type pid() as denoted here by [<0.84.0>].
Strings are represented as lists in erlang, which is why we use lists:concat/1.
As #7stud pointed out these are the valid types which can be passed to lists:concat/1 as per the documentation:
atom() | integer() | float() | string()
There are 2 occurrences of the following line. Fix them and you are good to go:
Incorrect Code:
M_pid!{lists:concat([Self_name," received intro message from ", Process_name , "[",Process_id,"]"]), self()},
Corrected Code
M_pid!{lists:concat([Self_name," received intro message from ", Process_name , "[",pid_to_list(Process_id),"]"]), self()},
Notice the use of the function erlang:pid_to_list/1. As per the documentation the function accepts type pid() and returns it as string().
I have been trying to use scipy.optimize curve_fit using multiple variables. It works fine with the test code I created but when I try to implement this on my actual data I keep getting the following error
TypeError: only arrays length -1 can be converted to python scalars
The shape of the arrays and the data types of their elements in my test code and actual code are exactly the same so I am confused as to why I get this error.
Test code:
import numpy as np
import scipy
from scipy.optimize import curve_fit
def func(x,a,b,c):
return a+b*x[0]**2+c*x[1]
x_0=np.array([1,2,3,4])
x_1=np.array([5,6,7,8])
X=scipy.array([x_0,x_1])
Y=func(X,3.1,2.2,2.1)
popt, pcov=curve_fit(func,X,Y)
Actual code:
f=open("Exp_Fresnal.csv", 'rb')
reader=csv.reader(f)
for row in reader:
Qz.append(row[0])
Ref.append(row[1])
Ref_F.append(row[2])
Qz_arr,Ref_Farr=scipy.array((Qz)),scipy.array((Ref_F))
x=scipy.array([Qz_arr,Ref_Farr]
def func(x,d,sig_int,sig_cp):
return x[1]*(x[0]*d*(math.exp((-sig_int**2)*(x[0]**2)/2)/(1-cmath.exp(complex(0,1)*x[0]*d)*math.exp((-sig_cp**2)*(x[0]**2)/2))))**2
Y=scipy.array((Ref))
popt, pcov=curve_fit(func,x,Y)
EDIT
Here is the full error message
Traceback (most recent call last):
File "DCM_03.py", line 46, in <module>
popt, pcov=curve_fit(func,x,Y)
File "//anaconda/lib/python2.7/site-packages/scipy/optimize/minpack.py", line 651, in curve_fit
res = leastsq(func, p0, args=args, full_output=1, **kwargs)
File "//anaconda/lib/python2.7/site-packages/scipy/optimize/minpack.py", line 377, in leastsq
shape, dtype = _check_func('leastsq', 'func', func, x0, args, n)
File "//anaconda/lib/python2.7/site-packages/scipy/optimize/minpack.py", line 26, in _check_func
res = atleast_1d(thefunc(*((x0[:numinputs],) + args)))
File "//anaconda/lib/python2.7/site-packages/scipy/optimize/minpack.py", line 453, in _general_function
return function(xdata, *params) - ydata
File "DCM_03.py", line 40, in func
return (0.062/(2*x))**4*(x*d*(math.exp((-sig_int**2)*(x**2)/2)/(1-cmath.exp(complex(0,1)*x*d)*math.exp((-sig_cp**2)*(x**2)/2))))**2
TypeError: only length-1 arrays can be converted to Python scalars
I figured out the issue. The problem for some reason was the use of math.exp and cmath.exp in the fitting function func. In place of these functions I used np.exp(). I am not completely sure the reason why though.
Although this is my first attempt at using pexpect, the python3 script using pexpect is pretty simple; yet it fails.
#!/usr/bin/env python3
import sys
import pexpect
SSH_NEWKEY = r'Are you sure you want to continue connecting \(yes/no\)\?'
child = pexpect.spawn("ssh -i /user/aws/key.pem ec2-user#xxx.xxx.xxx.xxx date")
i = child.expect( [ pexpect.TIMEOUT, SSH_NEWKEY )
if i == 1:
child.sendline('yes')
print(child.before)
The SSH_NEWKEY is the only response I'm expecting, but the example showed a list containing pexpect.TIMEOUT in it so I used it.
$ ./test.py
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/pexpect/spawnbase.py", line 144, in read_nonblocking
s = os.read(self.child_fd, size)
OSError: [Errno 5] Input/output error
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/pexpect/expect.py", line 97, in expect_loop
incoming = spawn.read_nonblocking(spawn.maxread, timeout)
File "/usr/local/lib/python3.4/site-packages/pexpect/pty_spawn.py", line 455, in read_nonblocking
return super(spawn, self).read_nonblocking(size)
File "/usr/local/lib/python3.4/site-packages/pexpect/spawnbase.py", line 149, in read_nonblocking
raise EOF('End Of File (EOF). Exception style platform.')
pexpect.exceptions.EOF: End Of File (EOF). Exception style platform.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./min.py", line 15, in <module>
i = child.expect( [ pexpect.TIMEOUT, SSH_NEWKEY ] )
File "/usr/local/lib/python3.4/site-packages/pexpect/spawnbase.py", line 315, in expect
timeout, searchwindowsize, async)
File "/usr/local/lib/python3.4/site-packages/pexpect/spawnbase.py", line 339, in expect_list
return exp.expect_loop(timeout)
File "/usr/local/lib/python3.4/site-packages/pexpect/expect.py", line 102, in expect_loop
return self.eof(e)
File "/usr/local/lib/python3.4/site-packages/pexpect/expect.py", line 49, in eof
raise EOF(msg)
pexpect.exceptions.EOF: End Of File (EOF). Exception style platform.
<pexpect.pty_spawn.spawn object at 0x7f70ea4fbcf8>
command: /usr/bin/ssh
args: ['/usr/bin/ssh', '-i', '/user/aws/key.pem', 'ec2-user#xxx.xxx.xxx.xxx', 'date']
searcher: None
buffer (last 100 chars): b''
before (last 100 chars): b'Fri May 6 13:50:18 EDT 2016\r\n'
after: <class 'pexpect.exceptions.EOF'>
match: None
match_index: None
exitstatus: 0
flag_eof: True
pid: 31293
child_fd: 5
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
What am I missing?
CentOS 6.4
python 3.4.3
An EOF error is being raised during your expect call. This means that the response received does not match SSH_NEWKEY, and reaches end of file within the timeout period. To catch this exception, you should change your except line to read:
i = child.expect( [ pexpect.TIMEOUT, SSH_NEWKEY, pexpect.EOF)
You can then make your if more robust:
if i == 1:
child.sendline('yes')
elif i == 0:
print "Timeout"
elif i == 2:
print "EOF"
print(child.before)
This doesn't solve the reason behind why you are on receiving a response with the expected string - it's hard to know without looking at more code but it's likely because you have the response slightly wrong. If you manually type in the SSH string, you should be able to see the response you can expect, and enter this response into your code.
You can also print child.before after your expect call, or print child.read() instead of your expect call to see what is being sent back as a response.