I am building a request containing a list of parameters, this is a list of endpoints that is read from a file. All these containing "/" characters.
First the file is read as:
pointRef = []
with open("myfolder/" + scope, 'r') as f:
for line in f:
pointRef.append(line.strip())
then passing
params = {'endDate': endDate, 'startDate': startDate, 'pointRef': pointRef}
and executing
r = requests.get(url=url_ranged_multiple, headers=headers, params=params)
and this gives error (I tried other requests by hand and they work), but I noticed that the final url request that is composed by "request.get" contains the "%2F" character instead of "/"
I wonder if this is the problem how can I correct it.
Many thanks in advance
Related
I have a function that I want to open .dat files with, to extract data from them, but the problem is I don't know how to turn that data back into a dictionary to store in a variable. Currently, the data in the files are stored like this: "{"x":0,"y":1}" (it uses up only one line of the file, which is just the normal structure of a dictionary).
Below is just the function where I open the .dat file and try to extract stuff from it.
def openData():
file = fd.askopenfile(filetypes=[("Data",".dat"),("All Files",".*")])
filepath = file.name
if file is None:
return
with open(filepath,"r") as f:
contents = dict(f.read())
print(contents["x"]) #let's say there is a key called "x" in that dictionary
This is the error that I get from it: (not because the key "x" is not in dict, trust me)
Exception in Tkinter callback
Traceback (most recent call last):
File "...\AppData\Local\Programs\Python\Python39\lib\tkinter\__init__.py", line 1892, in __call__
return self.func(*args)
File "...\PycharmProjects\[this project]\main.py", line 204, in openData
contents = dict(f.read())
ValueError: dictionary update sequence element #0 has length 1; 2 is required
Process finished with exit code 0
Update: I tried using json and it worked, thanks to #match
def openData():
file = fd.askopenfile(filetypes=[("Data",".dat"),("All Files",".*")])
filepath = file.name
if file is None:
return
with open(filepath,"r") as f:
contents = dict(json.load(f))
print(contents["x"])
You need to parse the data to get a data structure from a string, fortunately, Python provides a function for safely parsing Python data structures: ast.literal_eval(). E.g:
import ast
...
with open("/path/to/file", "r") as data:
dictionary = ast.literal_eval(data.read())
Reference stackoverflow
I'm trying to write a lambda that will return a .WAV file in chunks over HTTP. I've got my actual data in a byte slice (outputPayload [] byte) and am trying to pass it back. While the request seems to run, the response received is of a different length to what I expect and seems to be corrupted. Here's my code:
//Create the necessary headers
responseHeader := make(map[string]string)
responseHeader["Accept-Ranges"] = "bytes"
responseHeader["Content-Range"] = fmt.Sprintf("%s/%d", rangeRequired, fileSize)
responseHeader["Content-Type"] = fileType // this will be "audio/wav"
responseHeader["Content-Length"] = fmt.Sprintf("%d", returnedByteCount)
responseBody := string(outputPayload)
return events.APIGatewayProxyResponse{
StatusCode: http.StatusPartialContent,
Headers: responseHeader,
Body: responseBody,
}, nil
As a basic check, using more at the command line, the start of the original file looks like this:
RIFF$^?^C^#WAVEfmt ^P^#^#^#^A^#^B^#D<AC>^#^#^P<B1>^B^#^D^#^P^#data^#^?^C^#ESC^#^Y^#
While the downloaded file looks like this:
RIFF$^?^C^#WAVEfmt ^P^#^#^#^A^#^B^#D�^#^#^P�^B^#^D^#^P^#data^#^?^C^#ESC^#^Y^#
I'm guessing I have an encoding issue somewhere. My hunch is the string conversion is the problem, but that's the variable type I need for an APIGatewayProxyResponse "Body" component. How do I change my code output to ensure the payload matches the original file?
I'm trying to concat 2 variables Address and Payload. After that I want to send them with http to a server but I have 2 problems. When i try to concat the 2 variables with a delimiter ';' it doesn't work. Also sending the data of Payload or Address doesn't work. This is my code:
handle_rx(Gateway, #link{devaddr=DevAddr}=Link, #rxdata{port=Port, data= RxData }, RxQ)->
Data = base64:encode(RxData),
Devaddr = base64:encode(DevAddr),
TextAddr="Device address: ",
TextPayload="Payload: ",
Address = string:concat(TextAddr, Devaddr),
Payload = string:concat(TextPayload, Data),
Json=string:join([Address,Payload], "; "),
file:write_file("/tmp/foo.txt", io_lib:fwrite("~s.\n", [Json] )),
inets:start(),
ssl:start(),
httpc:request(post, {"http://192.168.0.121/apiv1/lorapacket/rx", [], "application/x-www-form-urlencoded", Address },[],[]),
ok;
handle_rx(_Gateway, _Link, RxData, _RxQ) ->
{error, {unexpected_data, RxData}}.
I have no errors that I can show you. When I write Address or Payload individually to the file it works but sending doesn't work...
Thank you for your help!
When i try to concat the 2 variables with a delimiter ';' it doesn't work.
5> string:join(["hello", <<"world">>], ";").
[104,101,108,108,111,59|<<"world">>]
6> string:join(["hello", "world"], ";").
"hello;world"
base64:encode() returns a binary, yet string:join() requires string arguments. You can do this:
7> string:join(["hello", binary_to_list(<<"world">>)], ";").
"hello;world"
Response to comment:
In erlang the string "abc" is equivalent to the list [97,98,99]. However, the binary syntax <<"abc">> is not equivalent to <<[97,98,99]>>, rather the binary syntax <<"abc">> is special short hand notation for the binary <<97, 98, 99>>.
Therefore, if you write:
Address = [97,98,99].
then the code:
Bin = <<Address>>.
after variable substitution becomes:
Bin = <<[97,98,99]>>.
and that isn't legal binary syntax.
If you need to convert a string/list contained in a variable, like Address, to a binary, you use list_to_binary(Address)--not <<Address>>.
In your code here:
Json = string:join([binary_to_list(<<Address>>),
binary_to_list(<<Payload>>)],
";").
Address and Payload were previously assigned the return value of string:concat(), which returns a string, so there is no reason to (attempt) to convert Address to a binary with <<Address>>, then immediately convert the binary back to a string with binary_to_list(). Instead, you would just write:
Json = string:join(Address, Payload, ";")
The problem with your original code is that you called string:concat() with a string as the first argument and a binary as the second argument--yet string:concat() takes two string arguments. You can use binary_to_list() to convert a binary to the string that you need for the second argument.
Sorry I'm new to Erlang
As with any language, you have to study the basics and write numerous toy examples before you can start writing code that actually does something.
You don't have to concatenate strings. It is called iolist and is one of best things in Erlang:
1> RxData = "Hello World!", DevAddr = "Earth",
1> Data = base64:encode(RxData), Devaddr = base64:encode(DevAddr),
1> TextAddr="Device address", TextPayload="Payload",
1> Json=["{'", TextAddr, "': '", Devaddr, "', '", TextPayload, "': '", Data, "'}"].
["{'","Device address","': '",<<"RWFydGg=">>,"', '",
"Payload","': '",<<"SGVsbG8gV29ybGQh">>,"'}"]
2> file:write_file("/tmp/foo.txt", Json).
ok
3> file:read_file("/tmp/foo.txt").
{ok,<<"{'Device address': 'RWFydGg=', 'Payload': 'SGVsbG8gV29ybGQh'}">>}
I've a json file with plone objects and there is one field of the objects giving me an error:
UnicodeDecodeError('ascii', '{"id":"aluminio-prata", "nome":"ALUM\xc3\x8dNIO PRATA", "num_demaos":0, "rendimento": 0.0, "unidade":"litros", "url":"", "particular":[], "profissional":[], "unidades":[]},', 36, 37, 'ordinal not in range(128)') (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'NoneType' object has no attribute 'getMethodAliases')
I already know witch field is, is the "title" from title = obj.pretty_title_or_id(), when I remove it from here its ok:
json += '{"id":"' + str(id) + '", "nome":"' + title + '", "num_demaos":' + str(num_demaos) + ', "rendimento": ' + str(rendimento) + ', "unidade":"' + str(unidade) + '", "url":"' + link_produto + '", "particular":' + arr_area_particular + ', "profissional":' + arr_area_profissional + ', "unidades":' + json_qtd + '},
but when I leave it I've got this error.
UnicodeDecodeError('ascii', '{"id":"aluminio-prata", "nome":"ALUM\xc3\x8dNIO PRATA", "num_demaos":0, "rendimento": 0.0, "unidade":"litros", "url":"", "particular":[], "profissional":[], "unidades":[]},', 36, 37, 'ordinal not in range(128)') (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'NoneType' object has no attribute 'getMethodAliases')
I'm going to assume that the error occurs when you're reading the JSON file.
Internally, Plone uses Python Unicode strings for nearly everything. If you read a string from a file, it will need to be decoded into Unicode before Plone can use it. If you give no instructions otherwise, Python will assume that the string was encoded as ASCII, and will attempt its Unicode conversion on that basis. It would be similar to writing:
unicode("ALUM\xc3\x8dNIO PRATA")
which will produce the same kind of error.
In fact, the string you're using was evidently encoded with the UTF-8 character set. That's evident from the "\xc3", and it also makes sense, because that's the character set Plone uses when it sends data to the outside world.
So, how do you fix this? You have to specify the character set that you wish to use when you convert to Unicode:
"ALUM\xc3\x8dNIO PRATA".decode('UTF8')
This gives you a Python Unicode string with no error.
So, after you've read your JSON file into a string (let's call it mystring), you will need to explicitly decode it by using mystring.decode('UTF8'). unicode(mystring, 'UTF8') is another form of the same operation.
As Steve already wrote do title.decode('utf8')
An Example illustrate the facts:
>>> u"Ä" == u"\xc4"
True # the native unicode char and escaped versions are the same
>>> "Ä" == u"\xc4"
False # the native unicode char is '\xc3\x84' in latin1
>>> "Ä".decode('utf8') == u"\xc4"
True # one can decode the string to get unicode
>>> "Ä" == "\xc4"
False # the native character and the escaped string are
# of course not equal ('\xc3\x84' != '\xc4').
I find this Thread very helpfull for Problems and Understanding with Encode/Decode of UTF-8.
I'm parsing HTML by inheriting HTMLParser, which is a class coming from the library html.parser. I'm making a web scraper. I have set "convert_charrefs" to true. The program downloads a page by doing "downloadPage(url)" and passes it to myParser (I think It will be better for you if I don't paste here all my code). When the parser finds the link I'm interested to (e.g Attività e procedimenti) from a web site, the program get the value of the attribute "href" and tries to download the page linked by href, by doing "downloadPage(href)", passes it to myParser and so on...
The code for downloadPage(href) is the following:
def getCharset(response):
str = response.info()["Content-type"]
if str:
end = re.search("charset=", str).span()[1]
if end:
return str[end:]
else:
return "ascii"
else:
return "ascii"
def downloadPage(url):
response = urllib.request.urlopen(url)
charset = getCharset(response)
return response.read().decode(charset)
Now, the problem is that certain link has some vowel stressed, such as "http://città.it/" (last url is faked). Not all links found in a web page are made of Unicode characters. So the following code sometimes raises UnicodeEncodeError:
urllib.request.urlopen(url)
I specify that I can't know at first glance how each link is composed
I have solved this problem in this way:
def fromIriToUri(iri):
myUri = []
iri = urlsplit(iri)
iri = list(iri)
for i in iri:
try:
i.encode("ascii")
myUri.append(i)
except UnicodeEncodeError:
myUri.append(urllib.parse.quote(i))
uri = urllib.parse.urlunsplit(myUri)
return uri