Backslashes are not correctly encoded - python-requests

I am at wits end.
My scenario is that I am using python requests to interact with Icinga2 API and I am trying to schedule a downtime. So I know how that is supposed to works and it works most of time. But unfortunately I am totally out of luck when the Icinga2 service I try to set a downtime to has name with a backslash in it.
My test environment:
Icinga2 2.9.0
Python 3.6.8 / Python 3.8.11
requests 2.27.0
Prerequisite: Create a host in Icinga. Create a service in Icinga with a "\"-character.
Python Code for Reproduction:
import requests
session = requests.Session()
session.hooks = {
"response": lambda r, *args, **kwargs: r.raise_for_status()
}
session.headers.update(
{
"Accept": "application/json",
"Content-Type": "application/json",
"cache-control": "no-cache",
}
)
session.auth = requests.auth.HTTPBasicAuth("user","password")
url = "https://icinga2-server.com:5665/v1/actions/schedule-downtime"
body = {
'comment': 'Downtime',
'author': ('icingaadmin',),
'start_time': 1605196800.0,
'filter': 'host.name==\"HOSTNAME\" && service.name==\"SERVICE\\NAME\"',
'end_time': 1605286800.0,
'fixed': True,
'type': 'Service'}
session.post(url, json=body, verify=False)
Result:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/.pyenv/versions/3.8.11/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/root/.pyenv/versions/3.8.11/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/root/.pyenv/versions/3.8.11/lib/python3.8/site-packages/requests/sessions.py", line 662, in send
r = dispatch_hook('response', hooks, r, **kwargs)
File "/root/.pyenv/versions/3.8.11/lib/python3.8/site-packages/requests/hooks.py", line 31, in dispatch_hook
_hook_data = hook(hook_data, **kwargs)
File "<stdin>", line 2, in <lambda>
File "/root/.pyenv/versions/3.8.11/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://icinga2-server.com:5665/v1/actions/schedule-downtime
I know very well that this error message indicates that Icinga2 could not find/match a service. But executing the command via curl is clearly working for me and I get a proper scheduled downtime!
CURL Request:
curl -k -s -u user:password -H 'Accept: application/json' -X POST 'https://icinga2-server.com:5665/v1/actions/schedule-downtime' -d '{"comment": "Downtime", "author": ["icingaadmin"], "start_time": 1605196800.0, "filter": "host.name==\"MSSQLSERVER\" && service.name==\"MSSQLSERVER\\\\INSTANCE2\"", "end_time": 1605286800.0, "fixed": true, "type": "Service"}'
CURL Answer (SUCCESS):
{"results":[{"code":200.0,"legacy_id":8.0,"name":"MSSQLSERVER!MSSQLSERVER\\INSTANCE2!137c9ef9-3150-4e57-ba0b-a22ddc6611d4","status":"Successfully scheduled downtime 'MSSQLSERVER!MSSQLSERVER\\INSTANCE2!137c9ef9-3150-4e57-ba0b-a22ddc6611d4' for object 'MSSQLSERVER!MSSQLSERVER\\INSTANCE2'."}]}
Alternate approaches that did not help:
session.post(url, data=json.dumps(body), verify=False)
string_body = json.dumps(body)
session.post(url, data=string_body, verify=False)

You can try adding an r to the front of the string after the filter key. Also, I think this closed parenthesis is unnecessary.
Python treats backslashes in strings as special characters, adding an 'r' before the string make it not treat them as special characters, and instead as backslashes
'filter': r'host.name=="HOSTNAME" && service.name=="SERVICE\NAME"'),
https://www.journaldev.com/23598/python-raw-string
body1 = {
'comment': 'Downtime',
'author': ('icingaadmin',),
'start_time': 1605196800.0,
'filter': 'host.name==\"HOSTNAME\" && service.name==\"SERVICE\\NAME\"',
'end_time': 1605286800.0,
'fixed': True,
'type': 'Service'}
body2 = {
'comment': 'Downtime',
'author': ('icingaadmin',),
'start_time': 1605196800.0,
'filter': r'host.name==\"HOSTNAME\" && service.name==\"SERVICE\\NAME\"',
'end_time': 1605286800.0,
'fixed': True,
'type': 'Service'}
body1==body2 -> False
body1['filter'] = 'host.name=="HOSTNAME" && service.name=="SERVICE\\NAME"'
body2['filter'] = 'host.name==\\"HOSTNAME\\" && service.name==\\"SERVICE\\\\NAME\\"'

Thank you very much for your support. I figured it out. I was able to successfully set a downtime with the following command to which Leo Wotzak beat me to.
body1 = {
'comment': 'Downtime',
'author': ('icingaadmin',),
'start_time': 1605196800.0,
'filter': 'host.name==\"HOSTNAME\" && service.name==\"SERVICE\\\\NAME\"',
'end_time': 1605286800.0,
'fixed': True,
'type': 'Service'}
I don't know why I didn't run a test with 4 backslashes earlier.
And to add some background to the situation. The initial problem was, that I was reading the output from available icinga services which gives me:
>>>> resp.json()['results'][1]['name']
'MSSQLSERVER!SERVICE\\NAME'
If I put that answer into the body it will fail because json converter does not convert 2 backslashes to 4 backslashes. Question is, what is the best way to tell json that 2 backslashes need to be 4 backslashes? My idea is to work with encoding:
svc_name = resp.json()['results'][1]['name'].encode("unicode-escape").decode("utf-8")
I don't know if that's the best way but it works.

Related

Failed when parsing body as json - call Rasa API with request.post()

I'm writing a function to call Rasa API for intent prediction. Here is my code:
def run_test():
url = "http://localhost:5005/model/parse"
obj = {"text": "What is your name?"}
response = requests.post(url, data=obj)
print(response.json())
I also start Rasa server with this command: rasa run -m models --enable-api --cors "*" --debug
And here is what I got from Rasa server terminal:
In the terminal that I excuted run_test(), I got this result:
{'version': '2.7.1', 'status': 'failure', 'message': 'An unexpected error occurred. Error: Failed when parsing body as json', 'reason': 'ParsingError', 'details': {}, 'help': None, 'code'
: 500}
Anybody help me to solve this problem? Thank you very much!
I found the solution:
Only need to use json.dumps() the object, because object in Python is different than object in json.
def run_test():
url = "http://localhost:5005/model/parse"
obj = {"text": "What is your name?"}
response = requests.post(url, data=json.dumps(obj))
print(response.json())

Yoututbe scraping by colab

I need to scrape car type video from YouTube by some tags like this list in Google Colab :
Abarth
AC
Acura
Adam
Adler
AEC
Aero
Aixam
Albion
SO i have tried these two code to find the video tag ( for example tag='Peugeot') in google colab:
!pip install youtube-search-python
from youtubesearchpython import SearchVideos
search = SearchVideos("NoCopyrightSounds", offset = 1, mode = "json", max_results = 20)
print(search.result())
and
!pip install youtube-dl
!echo '' > ford_video_list.txt
!chmod 755 ford_video_list.txt
!youtube-dl --match-title 'ford' --add-metadata --write-thumbnail --list-thumbnails --mark-watched --write-info-json 'ford_video_description_json.txt' --write-description 'ford_video_description.txt' --cookies='Search-youtube-url-file.txt' --ignore-errors --skip-download --get-url -f bestvideo+bestaudio/best --default-search "ytsearch2000:" "Ford Festiva" >> ford_video_list.txt
!echo '*****End of test 1 ******'
But by trying this code it don't showing any result:
import urllib.request
from bs4 import BeautifulSoup
textToSearch = 'python tutorials'
query = urllib.parse.quote(textToSearch)
url = "https://www.youtube.com/results?search_query=" + query
response = urllib.request.urlopen(url)
html = response.read()
soup = BeautifulSoup(html, 'html.parser')
for vid in soup.findAll(attrs={'class':'yt-uix-tile-link'}):
if not vid['href'].startswith("https://googleads.g.doubleclick.net/"):
print('https://www.youtube.com' + vid['href'])
So, i guess the class name is not correct!, and i asked here for debugging it.
Update:
I have made one google colab page (shown below) to test those codes ( also the code of youtube-dl showing this error:
https://colab.research.google.com/drive/1bZQ68gLggTQHCG_5fQQJJTICHA4K3HJ3?usp=sharing
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests
(caused by <HTTPError 429: 'Too Many Requests'>); please report this
issue on https://yt-dl.org/bug . Make sure you are using the latest
version; see https://yt-dl.org/update on how to update. Be sure to
call youtube-dl with the --verbose flag and include its complete
output.
I understand that error made because:
The google don't like too many request form one IP Address.
So tried to add these tags(--rm-cache-dir --force-ipv4 --verbose) to youtube-dl command as you can see below ( based of these reffrences 1 2 3):
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/lib/python3.6/dist-packages/youtube_dl/extractor/common.py", line 632, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python3.6/dist-packages/youtube_dl/YoutubeDL.py", line 2238, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 564, in error
result = self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 756, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
Thanks.
It has been working by changing the '"ytsearch2000:" "Ford Festiva"` :
to ' "ytsearch50":"Ford Festiva" as you can see below:
!pip install youtube-dl
# !youtube-dl --default-search gvsearch5:how to develop for android --no-playlist --write-info-json --write-annotation --write-thumbnail --write-sub --skip-download
!youtube-dl --match-title 'ford' "ytsearch50":"Ford Festiva"+"peugeot 405" --write-info-json --write-annotation --write-thumbnail --write-sub -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4'
and the problem was because of : and " location mistaking!
the entire code for scraping the video of some car type video from google could be seen here:
https://colab.research.google.com/github/CAR-Driving/yoloOnGoogleColab/blob/master/database_creating/Yoututbe_scraping_by_colab.ipynb

telegram bot keeps replying rather than executing the following code (webhook is used)

I am trying to host my telegram bot for a multiplayer game on GAE and web hook is used here. This is how my databasing part is like:
class Game(ndb.Model):
chat_id = ndb.IntegerProperty(required = True)
mission_num = ndb.IntegerProperty(default =1)
round_num = ndb.IntegerProperty(default =1)
class Player(ndb.Model):
user_id = ndb.IntegerProperty(required=True)
player_role = ndb.StringProperty (
choices = ['spy','resistance'])
The part of code under web hook handler:
if text.startswith('/'):
if text == '/start':
reply('Bot enabled')
setEnabled(chat_id, True)
elif text == '/stop':
reply('Bot disabled')
setEnabled(chat_id, False)
elif text == '/newgame':
if chat_type == 'group':
existing_game = Game.query (Game.chat_id == chat_id).get()
if existing_game:
reply ("game is alr intitiated liao")
else:
##create a new game here
#still stuck here
##========================##
#reply("keep replying this line")
##========================##
new_game = Game (
chat_id = chat_id,
id = chat_id
)
curr_game_key = new_game.put()
new_player = Player (
parent = curr_game_key,
user_id = fr_user_id,
id = fr_user_id)
new_player.put()
reply("waiting for more friends to join")
else:
reply('game must be conducted within a group chat! jio more friends!')
else:
reply('What command?')
else:
if getEnabled(chat_id):
reply('I got your message! (but I do not know how to answer)')
else:
logging.info('not enabled for chat_id {}'.format(chat_id))
The problem is that when I send '/newgame' in a group chat, nothing is sent back to me. If I uncomment the following line my bot keeps sending me "keep replying this line" like crazy.:
#reply("keep replying this line")
The reply function:
def reply(msg=None, img=None):
if msg:
resp = urllib2.urlopen(BASE_URL + 'sendMessage', urllib.urlencode({
'chat_id': str(chat_id),
'text': msg.encode('utf-8'),
'disable_web_page_preview': 'true',
'reply_to_message_id': str(message_id),
})).read()
elif img:
resp = multipart.post_multipart(BASE_URL + 'sendPhoto', [
('chat_id', str(chat_id)),
('reply_to_message_id', str(message_id)),
], [
('photo', 'image.jpg', img),
])
else:
logging.error('no msg or img specified')
resp = None
logging.info('send response:')
logging.info(resp)
The error:
Internal Server Error
The server has either erred or is incapable of performing the requested operation.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~orbitaltest2/1.393734187774164753/main.py", line 66, in get
self.response.write(json.dumps(json.load(urllib2.urlopen(BASE_URL + 'getUpdates'))))
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 409: Conflict
code for the handler that line 66 belongs to:
class GetUpdatesHandler(webapp2.RequestHandler):
def get(self):
urlfetch.set_default_fetch_deadline(60)
self.response.write(json.dumps(json.load(urllib2.urlopen(BASE_URL + 'getUpdates'))))
Totally newbie, any suggestion is appreciated!
You should check what status code & what content is returned by your webhook.
You have 2 options how to reply to use:
Call Telegram API
Return JSON as a response to webhook call
As you have not provided it source code of reply() it's hard to tell what exactly is wrong.
Anyway your webhook should return HTTP status code 200. If it does not Telegram treat it as an internal error and is trying to resent message to you. This is why you are getting repeated calls and it's "replying like crazy".
Most probably the call reply("keep replying this line") is successful but then something wrong and Telegram gets wrong reply.
Add try/except blocks and log exceptions.
Check your logs & put additional logging if needed. For example I'm logging HTTP response content from my webhook. It helps.

freebase not working in python

Im trying to run freebase using python on Ubuntu 12.10 the first time. here's what i did
import freebase
query = {
"id" : "/en/the_beatles",
"type" : "/music/artist",
"album" : [{
"name" : None,
"release_date" : None,
"track": {
"return" : "count"
},
"sort" : "release_date"
}]
}
freebase.mqlread(query)
Here's that error i got
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/freebase-1.0.8-py2.7.egg/freebase/api/session.py", line 597, in mqlread
r = self._httpreq_json(service, 'POST', form=dict(query=qstr))
File "/usr/local/lib/python2.7/dist-packages/freebase-1.0.8-py2.7.egg/freebase/api/session.py", line 420, in _httpreq_json
resp, body = self._httpreq(*args, **kws)
File "/usr/local/lib/python2.7/dist-packages/freebase-1.0.8-py2.7.egg/freebase/api/session.py", line 406, in _httpreq
return self._http_request(url, method, body, headers)
File "/usr/local/lib/python2.7/dist-packages/freebase-1.0.8-py2.7.egg/freebase/api/httpclients.py", line 66, in call
self.log.error('SOCKET FAILURE: %s', e.fp.read())
AttributeError: 'error' object has no attribute 'fp'
Could anyone help me resolve this?
Thansk in advance
If you're using the old Python client library it won't work because Google never migrated it to work with the new APIs. You'll need to use the standard Google APIs Python library and the discovery interface.
https://developers.google.com/api-client-library/python/start/get_started

Determine difference in http request between Python2 and Python3

I am attempting to use Python3 to send metrics to Hosted Graphite. The examples given on the site are Python2, and I have successfully ported the TCP and UDP examples to Python3 (despite my inexperience, and have submitted the examples so the docs may be updated), however I have been unable to get the HTTP method to work.
The Python2 example looks like this:
import urllib2, base64
url = "https://hostedgraphite.com/api/v1/sink"
api_key = "YOUR-API-KEY"
request = urllib2.Request(url, "foo 1.2")
request.add_header("Authorization", "Basic %s" % base64.encodestring(api_key).strip())
result = urllib2.urlopen(request)
This works successfully, returning a HTTP 200.
So far I have ported this much to Python3, and while I was (finally) able to get it to make a valid HTTP request (i.e. no syntax errors), the request fails, returning HTTP 400
import urllib.request, base64
url = "https://hostedgraphite.com/api/v1/sink"
api_key = b'YOUR-API-KEY'
metric = "testing.python3.http 1".encode('utf-8')
request = urllib.request.Request(url, metric)
request.add_header("Authorization", "Basic %s" % base64.encodestring(api_key).strip())
result = urllib.request.urlopen(request)
The full result is:
>>> result = urllib.request.urlopen(request)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 479, in open
response = meth(req, response)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 591, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 517, in error
return self._call_chain(*args)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 451, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python3/3.3.1/Frameworks/Python.framework/Versions/3.3/lib/python3.3/urllib/request.py", line 599, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
Is it obvious what I am doing wrong? Are there any suggestions on how I might capture and compare what the successful (python2) and failing (python3) requests are actually sending?
Don't mix Unicode strings and bytes:
>>> "abc %s" % b"def"
"abc b'def'"
You could construct the header as follows:
from base64 import b64encode
headers = {'Authorization': b'Basic ' + b64encode(api_key)}
A quick way to see the request is to change the host in the url to localhost:8888 and run before making the request:
$ nc -l 8888
You could also use wireshark to see the requests.

Resources