I have a Flask app which sends a request to a Graphite server to authenticate and redirect to it's dashboard (changed setting REMOTE_USER_AUTHENTICATION = True). The request is as follows:
url = 'https://graphite.localdomain/dashboard'
s = requests.Session()
r = s.get(url, auth=('userx', 'passwordx'),verify=False)
print r.cookies.__dict__
return (r.text, r.status_code, r.headers.items())
The authentication from the request to the graphite server is good, I get 200's for valid users, and 401's for invalid users.
"print r.cookies.__dict__" will output...
{'_now': 1429303134, '_policy': <cookielib.DefaultCookiePolicy instance
at 0x7f263ec2b638>, '_cookies': {'graphite.localdomain': {'/':
{'sessionid': Cookie(version=0, name='sessionid',
value='**********masked**********', port=None, port_specified=False,
domain='graphite.localdomain', domain_specified=False,
domain_initial_dot=False, path='/', path_specified=True, secure=False,
expires=1430512734, discard=False, comment=None, comment_url=None,
rest={'httponly': None}, rfc2109=False)}}}, '_cookies_lock': <_RLock
owner=None count=0>}
...which appears right because it looks identical to the one I get from logging in directly to Graphite. But, when I return the response object (Return a requests.Response object from Flask) the browser returns content encoding errors in both Chrome and FireFox. If I change that to something like...
return r.content
...the dashboard page appears, but it's missing everything because the CSS and JS resources are 404'ing.
I am obviously not understanding something, any help would be greatly appreciated.
Related
I follow stackoverflow for quite some time now. In most cases the problems I encountered were already mentioned and addressed by people before me. Now, I have an issue I have not found an applicable solution to yet. It may result from my humble understanding of the issue and not knowing what I actually am looking for, so I hope you can help me to at least better understand what happens. If additional info is required to make sense, please do not hesitate to ask.
Synopsis: One user of a program I built often (not always, interestingly) gets an InvalidAuthenticationToken error from the request python package when requesting calendar events with a token
generated by the msal package while none of the other users have any issues at all.
The situation is as follows:
I built a program for a small company which has to read out the events of some of its employees. I wrote it in python and used the msal and requests packages for the part of the interaction with MS Outlook:
import msal
import requests
class OutlookClient():
def __init__(self, client_id, authority):
# client_id and authority are the respective
# aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee style ids of the app I registered at azure.
self.app = msal.PublicClientApplication(
client_id = client_id,
client_credential = None,
authority = msal.authority.AuthorityBuilder(msal.authority.AZURE_PUBLIC,authority)
)
def getToken(self, username, pw):
# credentials of some dummy employee being authenticated to access
# the employees' calendars
self.auth = self.app.acquire_token_by_username_password(username,pw,
scopes=["Calendars.Read","Calendars.Read.Shared","People.Read"]
)
return
def getCalendar(self, agentCal, startDate, endDate):
# agentCal is the id of the employee in question obtained somewhere else.
graph_data = None
if 'access_token' in self.auth:
req = "https://graph.microsoft.com/v1.0/users/"+agentCal+"/calendar/calendarView"+\
"?startDateTime="+ startDate.strftime("%Y-%m-%dT02:00")+\
"&endDateTime="+ endDate.strftime("%Y-%m-%dT23:00")+\
graph_data = requests.get(req,
headers={'Authorization': 'Bearer ' + self.auth['access_token'], 'content-type': 'application/json'}
).json()
try:
return graph_data['value']
except KeyError:
return []
Currently, three employees are testing the program in the field. One of them faces a recurring error which neither of the other users nor I can reproduce. When getCalendar gets called the request gets answered as
graph_data =
{'error':
{'code': 'InvalidAuthenticationToken',
'message': 'Access token has expired or is not yet valid.',
'innerError':
{'date': '2022-10-27T05:56:39',
'request-id': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
'client-request-id': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
}
}
whereas all other users and the specific user also sometimes should get a list of events. The token, however, looked fine to me:
self.auth =
{'token_type': 'Bearer',
'scope': 'Calendars.Read Calendars.Read.Shared Calendars.ReadWrite Mail.ReadWrite Mail.Send openid People.Read profile User.Read email',
'expires_in': 4581,
'ext_expires_in': 4581,
'access_token': 'eyJ0eXAiOiJKV1Q...',
'refresh_token': '0.AREA...',
'id_token': 'eyJ0eXAiOiJKV1Q...',
'client_info': 'eyJ1aWQ...',
'id_token_claims': {...}
}
I have limited opportunity to identify the issue at the user's computer, unfortunately, as they are currently overwhelmed with work and therefore not very responsive. So, before I bother them and myself with many trial and error approaches I hoped you could share some ideas.
The problem persists, as I was told, even when the program is closed and restarted.
I let the program create a log-file which stores the relevant variables such as the token etc. to see if any pattern arises but everytime a token is generated independent of whether the request for the calendars is answered correctly or incorrectly.
I thought that maybe the program gets started and after some time the token expires but in the log-file it seems to still be valid.
Sorry, it was as expected and I initially just did not check the correct things.
Indeed, the token expired and I did not see it. One solution is to check whether a request gets answered properly and if not get a token by the refresh token
if 'error' in graph_data:
self.auth = self.app.acquire_token_by_refresh_token(\
self.auth['refresh_token'],scopes)
and request again.
I've deployed my Bot to Webchat, Skype and MS Teams.
In OnTurnAsync method I check, if the user input begins with bnr then I call a specific method.
if (turnContext.Activity.Text.ToLower().StartsWith("bnr"))
{
string msg = RequestHandler.BnrCaller(turnContext.Activity.Text);
await turnContext.SendActivityAsync(msg);
return;
}
It works fine with Skype and Webchat but with teams it does not work 100%, It works just sometimes.
[Edit]
I found out, that it does not work if I copied the input into the input field but when I type it, it works fine!
The Messages coming from the Teams are having a different structure.
The message text begins with the words <at>...</at>.
You need to delete this beginning in a Middleware component, etc.
Try to look at your incoming messages through NGROK at localhost:4040 in your web browser.
So, I found out the issue by dint of app-insights.
I've added a middleware, which logs the request body into app-insights and just realized, that when I copy/paste a message like: "Hello", it would be logged in app-insights as something like this "\r\n\n\rHello\r\n\n\r\n". When I type it, it would be logged just fine.
So I jsut remove these symboles from the request and it works!
I'm trying to scrap a webpage for data but came across the problem of needing to log in.
Connection.Response loginForm = Jsoup.connect("http://www.rapidnyc.net/users/google_login")
.method(Connection.Method.GET)
.execute();
Document document = Jsoup.connect("http://www.rapidnyc.net/users/google_login")
.data("Email", "testEmail")
.data("Passwd", "testPass")
.... //other form data
.cookies(loginForm.cookies())
.post();
This gives me the org.jsoup.HttpStatusException: HTTP error fetching URL. Status=400
I used chrome developer tool to look at the Form data being posted but nothing I post works.
1. Have you submitted ALL input fields? Including HIDDEN ones.
2. I see the website requires "captcha-box" authentication, which is to prevent web crawlers from logging in. I highly doubt you will be able to log in with your program.
I say the 400 status is coming from your program not being able to provide the value for "captcha" authentication.
This one has been baffling me for almost 24 hours now. For some reason on my linode server, tastypie isn't return a response to POSTs until it reaches the server's keepalive_timeout. It works perfectly for all GETs, PUTs, DELETEs and POSTs handled via override_urls (I'm using 0.9.11 for tastypie), but any standard POST (e.g. not handled override_urls) just sits there until the timeout limit is reached.
At first I thought it was a server config issue, but I'm now starting to suspect it's something to do with tastypie.
Relevant tastypie code:
class MissionResource(ModelResource):
game = fields.ToOneField(GameResource, 'game')
mission_bank = fields.ToOneField('goosechase.website.api.MissionBankResource', 'mission', null = True)
class Meta:
queryset = Mission.objects.all()
resource_name = "mission"
always_return_data = True
list_allowed_methods = ['get', 'post', 'put', 'delete', 'patch']
excludes = ['hint_file_name']
filtering = {
"game" : ('exact'),
}
authentication = ApiKeyAuthentication()
authorization = Authorization()
# Added for testing various configs
def override_urls(self):
return [
url(r"^(?P<resource_name>%s)/manually-handled-post/$" %
self._meta.resource_name,
self.wrap_view('generic_post')),
]
# Test method for manually handled posts
def manually_handled_post(self, request, **kwargs):
self._meta.authentication.is_authenticated(request)
self.is_authorized(request)
return HttpResponse()
For some reason, doing a post to the "manually-handled-post/" endpoint works perfectly, as the response comes back in milliseconds, but a typical post doesn't return until it hits the server timeout. The testing code I did:
import requests
import json
# NORMAL POST ATTEMPT
response = requests.get('http://www.example.com/api/web/mission/19/?format=json&api_key=API_KEY&username=USERNAME')
event = json.loads(response.content)
del event['id']
del event['resource_uri']
response = requests.post('http://www.example.com/api/web/mission/?format=json&api_key=API_KEY&username=USERNAME', data=json.dumps(event), headers={'content-type': 'application/json'})
# DELAY UNTIL SERVER keepalive_timeout IS REACHED, THEN <Response [201]> IS RETURNED
# OVERRIDEN POST ATTEMPT
response = requests.post('http://www.example.com/api/web/mission/manually-handled-post/?format=json&api_key=API_KEY&username=USERNAME', data=json.dumps({}), headers={'content-type': 'application/json'})
# INSTANTANEOUS RESPONSE WITH <Response [200]>
I checked that there's no save or obj_create hooks that are causing the delays. I've also verified that the same issues happen on multiple model resources.
Since overriden_urls work perfectly, I'm thinking that there's a bug in tastypie for POSTs, but I haven't found anything online and in the github issues.
Has anyone ran into this before and fixed it without reducing the keepalive_timeout to 0?
EDIT 1:
I started logging with timestamps to figure out where the slowdown occurred. Turns out there was no slowdown in the code, the delay was happening after the response was returned (e.g. return http.HttpCreated(location=location)). When I forced the POST to return a status code of 202 (via http.httpAccepted), it worked.
So now it seems the issue stems from 201 status codes on my server. I have no idea why this would be happening, so if anyone can point me in the right direction I'd really appreciate it.
I am trying to log onto a website using python and requests. I'm pretty sure I am logging on properly. The next part is I go to a different page and try to download a file from that page. However, in order to download the file you have to be logged in. When I go to download the file, however, it redirects me to the log-in menu saying I haven't logged in. I am stuck and don't know what to do! By the way, the website is grabcad.com, what I'm basically trying to do is press the download all button featured on such a page
http://grabcad.com/library/apple-ipod-touch-5th-gen-1
payload = {'member[email]': 'username', 'member[password]': 'pass'}
with requests.Session() as s:
rObject = s.post('http://www.grabcad.com/login', data=payload)
cookies = rObject.cookies
rObject = s.get('http://www.grabcad.com' + downloadUrl, cookies=cookies)
#download URL is something I obtain early and I know it's correct. It's the URL for when you press the downloadAll button
path = 'C:\\User\\Desktop\\filename
with open(path, 'wb') as f:
for chunk in rObject.iter_content():
f.write(chunk)
So I took an altogether different route to solve the problem, I simply used mechanize which is an automated browswer tool for python.
#how to use mechanize to log-in, specifically for grabcad
b.open('http://grabcad.com/login')
b.form = list(b.forms())[1]
control = b.form.find_control("member[email]")
control2 = b.form.find_control("member[password]")
control.value = 'username'
control2.value = 'pass'
b.submit()
#Download Part
path = 'C:\\User\\Desktop\\filename
b.retrieve('https://www.grabcad.com' + downloadUrl, path)
#downloadUrl is obtained earlier and is simply the URL for the download
How are you ensuring that you're logged in correctly? I would print out the html after sending that post request from the session object & ensure it isn't a login page or an invalid password page. Cookies are automatically persistent across requests made on the session object, so I believe that the initial login isn't successful (http://docs.python-requests.org/en/latest/user/advanced/#session-objects).
Personally, I would use selenium for this though.
I have correctly logged into grabcad with the following code:
import requests
s = requests.session()
payload = {'member[email]': 'yourEmail', 'member[password]': 'yourPassword'}
p = s.post('https://grabcad.com/login', data=payload) # Ensure you're posting to HTTPS