pool.apply_async not call target function, call main function after pyinstaller package - pyinstaller

I use Pool.async to execute a uninstall software function. If not package as a exe, it worked well. But when I package it as a exe, it not call target function but call main function. Here is my code.
if __name__ == '__main__':
print("main function called")
try:
# 使用文件锁,限制单进程
parent_folder = "C:\\Program Files (x86)\\ctyun\\eCloudDeskDRT\\sdk"
if not os.path.exists(parent_folder):
os.makedirs(parent_folder)
lock_file_name = 'eCloudDeskDRTSrv.lock'
lock_file_path = os.path.join(parent_folder, lock_file_name)
lock_file_fd = open(lock_file_path, 'w')
portalocker.lock(lock_file_fd, portalocker.LOCK_EX | portalocker.LOCK_NB)
lock_file_fd.writelines(str(os.getpid()))
lock_file_fd.flush()
except BaseException as e:
print("CloudDeskDRTSrv.exe have another instance running.")
sys.exit()
def uninstall_software_sub(software_name, uninstall_string, uninstall_process_name):
try:
log_utils.get_logger().info('uninstall_software_sub')
startupinfo = sp.STARTUPINFO()
startupinfo.dwFlags |= sp.STARTF_USESHOWWINDOW
popen = sp.Popen(uninstall_string, stdout=sp.PIPE, stdin=sp.PIPE, stderr=sp.STDOUT, startupinfo=startupinfo)
log_utils.get_logger().info(popen.pid)
b_cmd = uninstall_string.encode("UTF-8")
log_utils.get_logger().info("execute command %s", uninstall_string)
b_outs, errs = popen.communicate(b_cmd, timeout=None)
if uninstall_process_name:
while windows_utils.check_process_running(uninstall_process_name):
time.sleep(3)
else:
popen.wait()
log_utils.get_logger().info("wait finished")
outs = b_outs.decode('gbk', 'ignore')
if errs:
log_utils.get_logger().error(errs)
else:
log_utils.get_logger().info(outs)
except UnicodeDecodeError as error:
log_utils.get_logger().exception(error)
outs = ""
finally:
if not popen.poll():
popen.kill()
uninstall_software_info = get_uninstall_software_info(software_name)
uninstall_success = uninstall_software_info is None
return uninstall_success
def uninstall_software(software_name, uninstall_callback):
log_utils.get_logger().info("uninstall_software software_name=%s", software_name)
uninstall_info = get_uninstall_software_info(software_name)
uninstall_string = uninstall_info.uninstall_string
if software_name in uninstall_process.keys():
uninstall_process_name = uninstall_process.get(software_name)
log_utils.get_logger().info(os.getpid())
pool = Pool(1)
log_utils.get_logger().info('apply_async')
pool.apply_async(func=third_party_software_check.uninstall_software_sub,
args=(software_name, uninstall_string, uninstall_process_name),
callback=uninstall_callback)
unisntall_software_sub not execute, but main function is called when execute pool.apply_async. Here is the log ouput. Can somebody help me?

I found pyinstaller have support multiprocessing. But need add some command in source code. I add multiprocessing.freeze_support() below main function. It worked very well. Here is the source code:
if __name__ == '__main__':
print("main function called")
multiprocessing.freeze_support()
try:
# 使用文件锁,限制单进程
parent_folder = "C:\\Program Files (x86)\\ctyun\\eCloudDeskDRT\\sdk"
if not os.path.exists(parent_folder):
os.makedirs(parent_folder)
lock_file_name = 'eCloudDeskDRTSrv.lock'
lock_file_path = os.path.join(parent_folder, lock_file_name)
lock_file_fd = open(lock_file_path, 'w')
portalocker.lock(lock_file_fd, portalocker.LOCK_EX | portalocker.LOCK_NB)
lock_file_fd.writelines(str(os.getpid()))
lock_file_fd.flush()
except BaseException as e:
print("CloudDeskDRTSrv.exe have another instance running.")
sys.exit()

Related

Explicitly set dependency TaskFlowAPI

Suppose I have a dag as follows:
def my_dag():
#task
def task_a():
# Some code here
return
#task
def task_b():
# Some code here
return
task_a()
task_b()
my_dag_instance = my_dag()
I would like to set task_b downstream from task_a. That is, I want task_b to execute only if task_a has executed successfully. How can I do so? Note that task_a does not return anything.
I tried task_a >> task_b right before the last line of my code (my_dag_instance = my_dag()) yet got an error along the lines of " TaskDecorator >> TaskDecorator is invalid operation". Any advice?
Indeed, you can do that, inside the dag context, and with task instance instead of task method reference:
def my_dag():
#task
def task_a():
# Some code here
return
#task
def task_b():
# Some code here
return
task_a() >> task_b()
# or
# A = task_a()
# B = task_b()
# A >> B
my_dag_instance = my_dag()

Pytorch Problem: My jupyter stuck when num_workers > 0

This is a snippet of my code in PyTorch, my jupiter notebook stuck when I used num_workers > 0, I spent a lot on this problem without any answer. I do not have a GPU and I work only with a CPU.
class IndexedDataset(Dataset):
def __init__(self,data,targets, test=False):
self.dataset = data
if not test:
self.labels = targets.numpy()
self.mask = np.concatenate((np.zeros(NUM_LABELED), np.ones(NUM_UNLABELED)))
def __len__(self):
return len(self.dataset)
def __getitem__(self, idx):
image = self.dataset[idx]
return image, self.labels[idx]
def display(self, idx):
plt.imshow(self.dataset[idx], cmap='gray')
plt.show()
train_set = IndexedDataset(train_data, train_target, test = False)
test_set = IndexedDataset(test_data, test_target, test = True)
train_loader = DataLoader(train_set, batch_size=BATCH_SIZE, num_workers=2)
test_loader = DataLoader(test_set, batch_size=BATCH_SIZE, num_workers=2)
Any help, appreciated.
When num_workers is greater than 0, PyTorch uses multiple processes for data loading.
Jupyter notebooks have known issues with multiprocessing.
One way to resolve this is not to use Jupyter notebooks - just write a normal .py file and run it via command-line.
Or try use what's suggested here: Jupyter notebook never finishes processing using multiprocessing (Python 3).
Since jupyter Notebook doesn't support python multiprocessing, there are two thin libraries, you should install one of them as mentioned here 1 and 2.
I prefer to solve my problem in two ways without using any external libraries:
By converting my file from .ipynb format to .py format and run it in the terminal and I write my code in the main() function as follows:
...
...
train_set = IndexedDataset(train_data, train_target, test = False)
train_loader = DataLoader(train_set, batch_size=BATCH_SIZE, num_workers=4)
if `__name__ == '__main__'`:
for images,label in train_loader:
print(images.shape)
With multiprocessing library as follows:
In try.ipynb:
import multiprocessing as mp
import processing as ps
...
...
train_set = IndexedDataset(train_data, train_target, test = False)
train_loader = DataLoader(train_set, batch_size=BATCH_SIZE)
if __name__=="__main__":
p = mp.Pool(8)
r = p.map(ps.getShape,train_loader)
print(r)
p.close()
In processing.py file:
def getShape(data):
for i in data:
return i[0].shape

LLVMLITE Hello World Example Produces Wrong Output

The problem I am faced with regards to llvmlite is producing a simple hello world example.
I am unable to display the string global variable value in the function I created.
It appears to always print out the number one.
I have already tried to return the stringtype which produced a error.
from llvmlite import ir
i64 = ir.IntType(64)
i8 = ir.IntType(16)
hellostr = 'hello, world!'
stringtype = ir.ArrayType(i64, len(hellostr))
module = ir.Module( name="m_hello_example" )
hello = ir.GlobalVariable(module, stringtype, '.str4')
fn_int_to_int_type = ir.FunctionType(i64, [stringtype.as_pointer()] )
fn_hel = ir.Function( module, fn_int_to_int_type, name="fn_hel" )
fn_hel_block = fn_hel.append_basic_block( name="fn_hel_entry" )
builder = ir.IRBuilder(fn_hel_block )
# zero = builder.constant(i64, 0)
# const_1 = ir.Constant(stringtype,1);
# builder.ret(const_1)
const_1 = ir.Constant(i64,1);
# print(const_1)
builder.ret(const_1)
print( module )
I was expecting the output to print out the string 'hello, world!'.
Any help would be much appreciated.
Thanks.
It ended up that I was able to solve my problem with the following code:
import llvmlite.ir as ir
import llvmlite.binding as llvm
from ctypes import CFUNCTYPE
def main():
m = ir.Module()
func_ty = ir.FunctionType(ir.VoidType(), []) #defining printer function as type void
func = ir.Function(m, func_ty, name="printer") #define function as printer
builder = ir.IRBuilder(func.append_basic_block('entry')) #defining the entry point of the function printer
fmt = "%s\n\0" #in function printf allows for inserting arg in, next global_fmt statements allow for creating #"fstr" assignment
c_fmt = ir.Constant(ir.ArrayType(ir.IntType(8), len(fmt)),
bytearray(fmt.encode("utf8")))
global_fmt = ir.GlobalVariable(m, c_fmt.type, name="fstr")
global_fmt.linkage = 'internal'
global_fmt.global_constant = True
global_fmt.initializer = c_fmt
arg = "Hello, World!\0" #args will be passed into printf function.
c_str_val = ir.Constant(ir.ArrayType(ir.IntType(8), len(arg)),
bytearray(arg.encode("utf8"))) #creates the c_str_value as a constant
printf_ty = ir.FunctionType(ir.IntType(32), [], var_arg=True) #creation of the printf function begins here and specifies the passing of a argument
printf = ir.Function(m, printf_ty, name="printf")
c_str = builder.alloca(c_str_val.type) #creation of the allocation of the %".2" variable
builder.store(c_str_val, c_str) #store as defined on the next line below %".2"
voidptr_ty = ir.IntType(8).as_pointer()
fmt_arg = builder.bitcast(global_fmt, voidptr_ty) #creates the %".4" variable with the point pointing to the fstr
builder.call(printf, [fmt_arg, c_str]) #We are calling the prinf function with the fmt and arg and returning the value as defiend on the next line
builder.ret_void()
#Next lines are for calling llvm and returning the assembly.
llvm.initialize()
llvm.initialize_native_target()
llvm.initialize_native_asmprinter()
print(str(m)) #PRINTING OUT THE ASSEMBLY
llvm_module = llvm.parse_assembly(str(m)) #Parsing teh assembly
tm = llvm.Target.from_default_triple().create_target_machine() #creating the target machine
with llvm.create_mcjit_compiler(llvm_module, tm) as ee:
ee.finalize_object() #Making sure all modules owned by the execution engine are fully processed and usable for execution
fptr = ee.get_function_address("printer") #fptr will reference the printer function
py_func = CFUNCTYPE(None)(fptr)
py_func() #run the function printer
if __name__ == "__main__":
main()
It appears that I didn't correctly assign the variable and hence why I wasn't returning anything.

Pyparsing: ParseAction not called

On a simple grammar I am in the bad situation that one of my ParseActions is not called.
For me this is strange as parseActions of a base symbol ("logic_oper") and a derived symbol ("cmd_line") are called correctly. Just "pa_logic_cmd" is not called. You can see this on the output which is included at the end of the code.
As there is no exception on parsing the input string, I am assuming that the grammar is (basically) correct.
import io, sys
import pyparsing as pp
def diag(msg, t):
print("%s: %s" % (msg , str(t)) )
def pa_logic_oper(t): diag('logic_oper', t)
def pa_operand(t): diag('operand', t)
def pa_ident(t): diag('ident', t)
def pa_logic_cmd(t): diag('>>>>>> logic_cmd', t)
def pa_cmd_line(t): diag('cmd_line', t)
def make_grammar():
semi = pp.Literal(';')
ident = pp.Word(pp.alphas, pp.alphanums).setParseAction(pa_ident)
operand = (ident).setParseAction(pa_operand)
op_and = pp.Keyword('A')
op_or = pp.Keyword('O')
logic_oper = (( op_and | op_or) + pp.Optional(operand))
logic_oper.setParseAction(pa_logic_oper)
logic_cmd = logic_oper + pp.Suppress(semi)
logic_cmd.setParseAction(pa_logic_cmd)
cmd_line = (logic_cmd)
cmd_line.setParseAction(pa_cmd_line)
grammar = pp.OneOrMore(cmd_line) + pp.StringEnd()
return grammar
if __name__ == "__main__":
inp_str = '''
A param1;
O param2;
A ;
'''
grammar = make_grammar()
print( "pp-version:" + pp.__version__)
parse_res = grammar.parseString( inp_str )
'''USAGE/Output: python test_4.py
pp-version:2.0.3
operand: ['param1']
logic_oper: ['A', 'param1']
cmd_line: ['A', 'param1']
operand: ['param2']
logic_oper: ['O', 'param2']
cmd_line: ['O', 'param2']
logic_oper: ['A']
cmd_line: ['A']
'''
Can anybody give me a hint on this parseAction problem?
Thanks,
The problem is here:
cmd_line = (logic_cmd)
cmd_line.setParseAction(pa_cmd_line)
The first line assigns cmd_line to be the same expression as logic_cmd. You can verify by adding this line:
print("???", cmd_line is logic_cmd)
Then the second line calls setParseAction, which overwrites the parse action of logic_cmd, so the pa_logic_cmd will never get called.
Remove the second line, since you are already testing the calling of the parse action with pa_logic_cmd. You could change to using the addParseAction method instead, but to my mind that is an invalid test (adding 2 parse actions to the same pyparsing expression object).
Or, change the definition of cmd_line to:
cmd_line = pp.Group(logic_cmd)
Now you will have wrapped logic_cmd inside another expression, and you can then independently set and test the running of parse actions on the two different expressions.

Porting to Python3: PyPDF2 mergePage() gives TypeError

I'm using Python 3.4.2 and PyPDF2 1.24 (also using reportlab 3.1.44 in case that helps) on windows 7.
I recently upgraded from Python 2.7 to 3.4, and am in the process of porting my code. This code is used to create a blank pdf page with links embedded in it (using reportlab) and merge it (using PyPDF2) with an existing pdf page. I had an issue with reportlab in that saving the canvas used StringIO which needed to be changed to BytesIO, but after doing that I ran into this error:
Traceback (most recent call last):
File "C:\cms_software\pdf_replica\builder.py", line 401, in merge_pdf_files
input_page.mergePage(link_page)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2013, in mergePage
self.mergePage(page2)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2059, in mergePage
page2Content = PageObject._pushPopGS(page2Content, self.pdf)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 1973, in _pushPopGS
stream = ContentStream(contents, pdf)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2446, in __init
stream = BytesIO(b_(stream.getData()))
File "C:\Python34\lib\site-packages\PyPDF2\generic.py", line 826, in getData
decoded._data = filters.decodeStreamData(self)
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 326, in decodeStreamData
data = ASCII85Decode.decode(data)
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 264, in decode
data = [y for y in data if not (y in ' \n\r\t')]
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 264, in
data = [y for y in data if not (y in ' \n\r\t')]
TypeError: 'in <string>' requires string as left operand, not int
Here is the line and the line above where the traceback mentions:
link_page = self.make_pdf_link_page(pdf, size, margin, scale_factor, debug_article_links)
if link_page != None:
input_page.mergePage(link_page)
Here are the relevant parts of that make_pdf_link_page function:
packet = io.BytesIO()
can = canvas.Canvas(packet, pagesize=(size['width'], size['height']))
....# left out code here is just reportlab specifics for size and url stuff
can.linkURL(url, r1, thickness=1, color=colors.green)
can.rect(x1, y1, width, height, stroke=1, fill=0)
# create a new PDF with Reportlab that has the url link embedded
can.save()
packet.seek(0)
try:
new_pdf = PdfFileReader(packet)
except Exception as e:
logger.exception('e')
return None
return new_pdf.getPage(0)
I'm assuming it's a problem with using BytesIO, but I can't create the page with reportlab with StringIO. This is a critical feature that used to work perfectly with Python 2.7, so I'd appreciate any kind of feedback on this. Thanks!
UPDATE:
I've also tried changing from using BytesIO to just writing to a temp file, then merging. Unfortunately I got the same error.
Here is tempfile version:
import tempfile
temp_dir = tempfile.gettempdir()
temp_path = os.path.join(temp_dir, "tmp.pdf")
can = canvas.Canvas(temp_path, pagesize=(size['width'], size['height']))
....
can.showPage()
can.save()
try:
new_pdf = PdfFileReader(temp_path)
except Exception as e:
logger.exception('e')
return None
return new_pdf.getPage(0)
UPDATE:
I found an interesting bit of information on this. It seems if I comment out the can.rect and can.linkURL calls it will merge. So drawing anything on a page, then trying to merge it with my existing pdf is causing the error.
After digging in to PyPDF2 library code, I was able to find my own answer. For python 3 users, old libraries can be tricky. Even if they say they support python 3, they don't necessarily test everything. In this case, the problem was with the class ASCII85Decode in filters.py in PyPDF2. For python 3, this class needs to return bytes. I borrowed the code for this same type of function from pdfminer3k, which is a port for python 3 of pdfminer. If you exchange the ASCII85Decode() class for this code, it will work:
import struct
class ASCII85Decode(object):
def decode(data, decodeParms=None):
if isinstance(data, str):
data = data.encode('ascii')
n = b = 0
out = bytearray()
for c in data:
if ord('!') <= c and c <= ord('u'):
n += 1
b = b*85+(c-33)
if n == 5:
out += struct.pack(b'>L',b)
n = b = 0
elif c == ord('z'):
assert n == 0
out += b'\0\0\0\0'
elif c == ord('~'):
if n:
for _ in range(5-n):
b = b*85+84
out += struct.pack(b'>L',b)[:n-1]
break
return bytes(out)

Resources