I have this simple routine where I want to remove some dirs from $PATH:
test.py
import os
import shutil
# remove babel from PATH
bins = ["babel", "obabel"]
env_path = os.environ["PATH"].split(":")[:]
for abin in bins:
bpath = shutil.which(abin)
try:
rr = os.path.dirname(bpath) # <- mypy error
except TypeError:
rr = None
try:
env_path.remove(rr) # <- mypy error
except ValueError:
pass
os.environ["PATH"] = ":".join(env_path)
This does work, however mypy complains:
mypy --ignore-missing-imports test.py
test.py:10: error: Value of type variable "AnyStr" of "dirname" cannot be "Optional[str]"
test.py:14: error: Argument 1 to "remove" of "list" has incompatible type "Optional[str]"; expected "str"
Found 2 errors in 1 file (checked 1 source file)
I understand that bpath can be a str or None (hence the try), so I tried bpath: Optional[str] but that gave me others mypy errors.
Anyway, how to bring peace to mypy here?
Apparently, based on the suggestions from the comments, this seems to have worked:
for abin in bins:
bpath = shutil.which(abin)
if bpath:
rr = os.path.dirname(bpath)
else:
rr = ""
try:
env_path.remove(rr)
except ValueError:
pass
No complains from mypy.
Related
I am using "argparse" library in R for command line arguments.
# Create parser
parser = ArgumentParser(description='command line args')
# Add command line arguments
parser$add_argument("is_local", nargs='?', type="logical",
help="whether to use local or server path", default=FALSE)
parser$add_argument("alert", nargs='?', type="double",
help="alert threshold", default=0.99)
I am trying to call it on command line such as:
Rscript my_func.R TRUE 0.99
However boolean argument does not change. Any idea how to parse boolean argument in R?
Thanks!
I don't know R, but the description of this package says it's a wrapper for the Python argparse.
I would recommend changing these:
parser$add_argument("is_local", nargs='?', type="logical",
help="whether to use local or server path", default=FALSE)
parser$add_argument("alert", nargs='?', type="double",
help="alert threshold", default=0.99)
to
parser$add_argument("--local", action='store_true'),
help="whether to use local or server path")
parser$add_argument("--alert", type="double",
help="alert threshold", default=0.99)
which would be called with
Rscript my_func.R --local --alert 0.99
store_true is illustrated on the basic docs page, https://github.com/trevorld/r-argparse
If I read the R correctly, your is_local should be giving you a warning
"You almost certainly want to use action='store_true' or action='store_false' instead"
A store_true argument sets the attribute to TRUE if present, and the default FALSE if absent. It should be an optional (--) and not set the nargs.
(It is possible to have an argument that takes strings 'true' and 'false' (or any other pair in your native language) and converts them to logical values, but requires more coding.)
I made --alert a flagged argument as well, without the nargs. Its value will be the default if absent, and the convert the string to a double if provided. It could be a '?' positional, but while learning I think it's best to stick with optionals unless you want the argument to be required.
The R-argparse docs aren't very complete. You may need to refer to the Python docs, and experiment to get the translation right.
https://docs.python.org/3/library/argparse.html
Thanks for your time and help!
I would like to share my workaround for the problem as I find original argparse (action) usage a bit complicated.
# Convert str input to boolean
str2bool = function(input_str)
{
if(input_str == "0")
{
input_str = FALSE
}
else if(input_str == "1")
{
input_str = TRUE
}
return(input_str)
}
# Create parser
parser = ArgumentParser(description='command line args')
# Add command line arguments
parser$add_argument("is_local", nargs='?', type="character",
help="whether to use local or server path", default="1")
parser$add_argument("alert", nargs='?', type="double",
help="alert threshold", default=0.99)
# Parse arguments
args = parser$parse_args()
# Convert str arguments
args$is_local = str2bool(args$is_local)
# Call on CMD line
Rscript my_func.R 1 0.99 #equivalent Rscript my_func.R TRUE 0.99
Rscript my_func.R 0 0.99 #equivalent Rscript my_func.R FALSE 0.99
Below is the sample example given in the official docs of package argparse:
parser <- ArgumentParser(description='Process some integers')
parser$add_argument('integers', metavar='N', type="integer", nargs='+',
help='an integer for the accumulator')
parser$add_argument('--sum', dest='accumulate', action='store_const',
const='sum', default='max',
help='sum the integers (default: find the max)')
parser$print_help()
# default args for ArgumentParser()$parse_args are commandArgs(TRUE)
# which is what you'd want for an Rscript but not for interactive use
args <- parser$parse_args(c("--sum", "1", "2", "3"))
accumulate_fn <- get(args$accumulate)
print(accumulate_fn(args$integers))
Here is the link for the argparse pdf https://cran.r-project.org/web/packages/argparse/argparse.pdf
I hope it might help.
I would like to run this Python code from R:
>>> import nlmpy
>>> nlm = nlmpy.mpd(nRow=50, nCol=50, h=0.75)
>>> nlmpy.exportASCIIGrid("raster.asc", nlm)
Nlmpy is a Python package to build neutral landscape models. The example comes from the website
To run this Python code from R, I 'm trying to use the package rPithon. However, I obtain this error message:
if (pithon.available())
{
nRow <- 50
nCol <- 50
h <- 0.75
# this file contains the definition of function concat
pithon.load("C:/Users/Anaconda2/Lib/site-packages/nlmpy/nlmpy.py")
pithon.call( "mpd", nRow, nCol, h)
} else {
print("Unable to execute python")
}
Error in pithon.get("_r_call_return", instance.name = instname) :
Couldn't retrieve variable: Traceback (most recent call last):
File "C:/Users/Documents/R/win-library/3.3/rPithon/pythonwrapperscript.py", line 110, in <module>
reallyReallyLongAndUnnecessaryPrefix.data = json.dumps([eval(reallyReallyLongAndUnnecessaryPrefix.argData)])
File "C:\Users\ANACON~1\lib\json\__init__.py", line 244, in dumps
return _default_encoder.encode(obj)
File "C:\Users\ANACON~1\lib\json\encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "C:\Users\ANACON~1\lib\json\encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "C:\Users\ANACON~1\lib\json\encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: array([[ 0.36534654, 0.31962481, 0.44229946, ..., 0.11513079,
0.07156331, 0.00286971], [ 0.41534291, 0.41333479, 0.48118995, ..., 0.19203674,
0.04192771, 0.03679473], [ 0.5188
Is this error caused by a syntax issue in my code ? I work with the Anaconda 4.2.0 platform for Windows which uses the Python 2.7 version.
I haven't used the nlmpy package hence, I am not sure what would be your expected output. However, this code successfully communicates between R and Python.
There are two files,
nlmpyInR.R
command ="python"
path2script="path_to_your_pythoncode/nlmpyInPython.py"
nRow <-50
nCol <-50
h <- 0.75
# Build up args in a vector
args = c(nRow, nCol, h)
# Add path to script as first arg
allArgs = c(path2script, args)
Routput = system2(command, args=allArgs, stdout=TRUE)
#The command would be python nlmpyInPython.py 50 50 0.75
print(paste("The Output is:\n", Routput))
nlmpyInPython.py
import sys
import nlmpy
#Getting the arguments from the command line call
nRow = sys.argv[1]
nCol = sys.argv[2]
h = sys.argv[3]
nlm = nlmpy.mpd(nRow, nCol, h)
pyhtonOutput = nlmpy.exportASCIIGrid("raster.asc", nlm)
#Whatever you print will get stored in the R's output variable.
print pyhtonOutput
The cause of the error that you're getting is hinted at by the
"is not JSON serializable" line. Your R code calls the mpd
function with certain arguments, and that function itself will
execute correctly. The rPithon library will then try to send the
return value of the function back to R, and to do this it will try
to create a JSON object
that describes the return value.
This works well for integers, floating point values, arrays, etc,
but not every kind of Python object can be converted to such a
JSON representation. And because rPithon can't convert the return value
of mpd this way, an error is generated.
You can still use rPithon to call the mpd function though. The following
code creates a new Python function that performs two steps: first
it calls the mpd function with the specified parameters, and then it
exports the result to a file, of which the filename is also an argument.
Using rPithon, the new function is then called from R. Because myFunction doesn't return anything, representing the return value in JSON format will not be a problem.
library("rPithon")
pythonCode = paste("import nlmpy.nlmpy as nlmpy",
"",
"def myFunction(nRow, nCol, h, fileName):",
" nlm = nlmpy.mpd(nRow, nCol, h)",
" nlmpy.exportASCIIGrid(fileName, nlm)",
sep = "\n")
pithon.exec(pythonCode)
nRow <- 50
nCol <- 50
h <- 0.75
pithon.call("myFunction", nRow, nCol, h, "outputraster.asc")
Here, the Python code defined as an R string, and executed using
pithon.exec. You could also put that Python code in a separate file
and use pithon.load to process the code so that the myFunction
function is known.
My code:
for i in range( 3.3, 5 ):
print( i )
The above code have to print:
3.300000
4.300000
but the interpreter of Python 3.4.0 printed the following error:
TypeError: 'float' object cannot be interpreted as an integer
range() works with integers not floats, but you can build your own range generator which will do what you want:
def frange(start, stop, step=1):
i = start
while i < stop:
yield i
i += step
for i in frange(3.3, 5) will give you the desired result.
Note though, that frange will, unlike range but like xrange, return a generator rather than a list.
The following code works perfectly well if called by source command. It is part of a function where xgb.metric.label is a formal parameter that at run time can be valorized with values like "rmsle". bst.cv is a data.table.
lab = paste('test.',xgb.metric.label,'.mean',sep='')
bst.cv = xgboost::xgb.cv(param=param, data = data, label = y,
nfold = nfold, nrounds=cv.nround , folds = foldList,
feval = xgb.metric.fun , maximize = xgb.maximize, verbose=FALSE)
if (verbose) print(bst.cv)
early.stop = which(bst.cv[,get(lab)] == bst.cv[,min(get(lab))] )
On the contrary when I call such a code by R CMD check R-package --as-cran I got the following error Error in get(lab) : object 'test.rmsle.mean' not found. But from the standard output I can see perfectly well test.rmsle.mean exists.
>> xgb: cross-validation ...
train.rmsle.mean train.rmsle.std test.rmsle.mean test.rmsle.std
1: 1.982000 0.047535 1.971343 0.257277
2: 1.957587 0.047260 1.947059 0.255464
3: 1.933816 0.046985 1.923421 0.253643
4: 1.910654 0.046711 1.900397 0.251813
5: 1.888074 0.046436 1.877959 0.249975
---
2996: 0.267519 0.072264 0.307539 0.239068
2997: 0.267531 0.072254 0.307413 0.239195
2998: 0.267425 0.072160 0.307183 0.239163
2999: 0.267488 0.072168 0.307137 0.239798
3000: 0.267476 0.072201 0.307264 0.239906
I tried to call all the related libraries a line before the error, but without success.
library(data.table)
library(xgboost)
library(methods)
I also added these libraries in the imports section of DESCRIPTION file. Again, without success.
Imports:
parallel,
subselect,
plyr,
xgboost,
methods,
data.table
Why are you using get at all?
There is no need to incur the overhead....
which(bst.cv[[lab]] == min(bst.cv[[lab]]))
I'm using Python 3.4.2 and PyPDF2 1.24 (also using reportlab 3.1.44 in case that helps) on windows 7.
I recently upgraded from Python 2.7 to 3.4, and am in the process of porting my code. This code is used to create a blank pdf page with links embedded in it (using reportlab) and merge it (using PyPDF2) with an existing pdf page. I had an issue with reportlab in that saving the canvas used StringIO which needed to be changed to BytesIO, but after doing that I ran into this error:
Traceback (most recent call last):
File "C:\cms_software\pdf_replica\builder.py", line 401, in merge_pdf_files
input_page.mergePage(link_page)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2013, in mergePage
self.mergePage(page2)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2059, in mergePage
page2Content = PageObject._pushPopGS(page2Content, self.pdf)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 1973, in _pushPopGS
stream = ContentStream(contents, pdf)
File "C:\Python34\lib\site-packages\PyPDF2\pdf.py", line 2446, in __init
stream = BytesIO(b_(stream.getData()))
File "C:\Python34\lib\site-packages\PyPDF2\generic.py", line 826, in getData
decoded._data = filters.decodeStreamData(self)
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 326, in decodeStreamData
data = ASCII85Decode.decode(data)
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 264, in decode
data = [y for y in data if not (y in ' \n\r\t')]
File "C:\Python34\lib\site-packages\PyPDF2\filters.py", line 264, in
data = [y for y in data if not (y in ' \n\r\t')]
TypeError: 'in <string>' requires string as left operand, not int
Here is the line and the line above where the traceback mentions:
link_page = self.make_pdf_link_page(pdf, size, margin, scale_factor, debug_article_links)
if link_page != None:
input_page.mergePage(link_page)
Here are the relevant parts of that make_pdf_link_page function:
packet = io.BytesIO()
can = canvas.Canvas(packet, pagesize=(size['width'], size['height']))
....# left out code here is just reportlab specifics for size and url stuff
can.linkURL(url, r1, thickness=1, color=colors.green)
can.rect(x1, y1, width, height, stroke=1, fill=0)
# create a new PDF with Reportlab that has the url link embedded
can.save()
packet.seek(0)
try:
new_pdf = PdfFileReader(packet)
except Exception as e:
logger.exception('e')
return None
return new_pdf.getPage(0)
I'm assuming it's a problem with using BytesIO, but I can't create the page with reportlab with StringIO. This is a critical feature that used to work perfectly with Python 2.7, so I'd appreciate any kind of feedback on this. Thanks!
UPDATE:
I've also tried changing from using BytesIO to just writing to a temp file, then merging. Unfortunately I got the same error.
Here is tempfile version:
import tempfile
temp_dir = tempfile.gettempdir()
temp_path = os.path.join(temp_dir, "tmp.pdf")
can = canvas.Canvas(temp_path, pagesize=(size['width'], size['height']))
....
can.showPage()
can.save()
try:
new_pdf = PdfFileReader(temp_path)
except Exception as e:
logger.exception('e')
return None
return new_pdf.getPage(0)
UPDATE:
I found an interesting bit of information on this. It seems if I comment out the can.rect and can.linkURL calls it will merge. So drawing anything on a page, then trying to merge it with my existing pdf is causing the error.
After digging in to PyPDF2 library code, I was able to find my own answer. For python 3 users, old libraries can be tricky. Even if they say they support python 3, they don't necessarily test everything. In this case, the problem was with the class ASCII85Decode in filters.py in PyPDF2. For python 3, this class needs to return bytes. I borrowed the code for this same type of function from pdfminer3k, which is a port for python 3 of pdfminer. If you exchange the ASCII85Decode() class for this code, it will work:
import struct
class ASCII85Decode(object):
def decode(data, decodeParms=None):
if isinstance(data, str):
data = data.encode('ascii')
n = b = 0
out = bytearray()
for c in data:
if ord('!') <= c and c <= ord('u'):
n += 1
b = b*85+(c-33)
if n == 5:
out += struct.pack(b'>L',b)
n = b = 0
elif c == ord('z'):
assert n == 0
out += b'\0\0\0\0'
elif c == ord('~'):
if n:
for _ in range(5-n):
b = b*85+84
out += struct.pack(b'>L',b)[:n-1]
break
return bytes(out)