Urllib3 session (persist settings between requests) - python-requests

I wrote a Python script which makes several https requests one after the other by following redirections (its purpose is to login to AWS Cognito via a load balancer):
import requests
session = requests.session()
response = session.get(
f'https://<initial_url>',
allow_redirects=False,
verify=False,
)
response = session.get(
response.headers["Location"],
allow_redirects=False,
verify=False,
)
response = session.post(
response.headers["Location"],
allow_redirects=False,
verify=False,
data={
"_csrf": session.cookies["XSRF-TOKEN"],
"username": <user>,
"password": <password>,
},
)
This works as expected. I would like to use the urllib3 library instead of requests and I transformed the script as follows:
import urllib3
http = urllib3.PoolManager(cert_reqs="CERT_NONE")
response = session.request(
"GET",
f'https://<initial_url>',
redirect=False,
retries=False,
)
response = session.request(
"GET",
response.headers["Location"],
redirect=False,
retries=False,
)
csrf=<get the XSRF-TOKEN cookie from response.headers["Set-Cookie"]>
fields = {
"_csrf": csrf,
"username": <user>,
"password": <password>,
}
response = session.request(
"POST",
response.headers["Location"],
redirect=False,
retries=False,
fields=fields,
)
The GET requests work and the redirects are as expected, but the POST does not (I get an error from Cognito). Based on the documentation
https://docs.python-requests.org/en/latest/user/quickstart/
and
https://urllib3.readthedocs.io/en/latest/user-guide.html
I understand that the equivalent of requests/data is urllib3/fields, as both are used to form-encode data
In the first version of the code, I created a new requests.session() object before the POST request and used it for that and I got the same error as in the urrlib3 case, which led me to believe that urllib3.PoolManager() does not provide a session like requests.session() and that each request is made separately, which makes the POST request fail. Does anyone know if there is a way to have a session in urrlib3 ? I could not find anything in the documentation

Related

Python requests post: data and json

I used the following Python code to retrieve a web page behind a login page successfully for some years:
username = 'user'
password = 'pass'
login_url = 'https://company.com/login?url='
redirect_url = 'https://epaper.company.com/'
data = { 'email' : username, 'pass' : password }
initial_url = login_url + quote(redirect_url)
response = requests.post(initial_url, data=data)
Then something changed at company.com about 2 months ago, and the request returned status code 400. I tried changing the data parameter to json (response = requests.post(initial_url, json=data)) which gave me a 200 response telling me a wrong password was provided.
Any ideas what I could try to debug?
Thanks,
Jan
Update: I just tried using a requests session to retrieve the csrf_token from the login page (as suggested here), so now my code reads:
with requests.Session() as sess:
response = sess.get(login_url)
signin = BeautifulSoup(response._content, 'html.parser')
data['csrf_token'] = signin.find('input', {'name':'csrf_token'})['value']
response = sess.post(initial_url, data=data)
Unfortunately, the response is still 400 (and 200/wrong password with the json parameter).
First: When you send data=data, used {"Content-Type":"application/x-www-form-urlencoded"}; if you send json=data, in headers response should be used {"Content-Type":"application/json"}
Second: Perhaps redirects have been added. Try to add:
response = sess.post(url, data=data)
print("URL you expect", url)
print("Last request URL:", response.url)
Be sure to check:
print(sess.cookies.get_dict())
print(response.headers)
If you get an unexpected result when checking, change the code like this:
response = sess.post(url, data=data, allow_redirects=False)

Locust does response with 2xx but fails to gather request statistics

I am trying to do a local load testing with Locust. I got the test environment up and running and a local build is also working. I am trying to test the responses of a local path and the response I get in the terminal is correct. But the Locust UI and also the statistics after terminating the test give me 100% fail results.
For creating the locust code (I am pretty new to it) I use the postman content and adjusted it. This is the Code for Locust:
from locust import HttpLocust, TaskSet, task, between
import requests
url = "http://localhost:8080/registry/downloadCounter"
payload = "[\n {\n \"appName\": \"test-app\",\n \"appVersion\": \"1.6.0\"\n }\n]"
class MyTaskSet(TaskSet):
#task(2)
def index(self):
self.client.get("")
headers = {
'Content-Type': 'application/json',
'Accept':'application/json'
}
response = requests.request("POST", url, headers=headers, data = payload)
print(response.text.encode('utf8'))
class MyLocust(HttpLocust):
task_set = MyTaskSet
wait_time = between(2.0, 4.0)
For the Locust swarm I used just basic numbers:
Number of total users to simulate: 1
Hatch Rate: 3
Host: http://localhost:8080/registry/downloadCounter
I do not get any results there, the table stays blank. I guess it has something to do with the json format but I am not able to find the solution myself.
I also put a Screenshot of the Terminal response after termination in this post.
Thank you in advance for your help!
Best regards
This helped:
from locust import HttpLocust, TaskSet, task, between
import requests
url = "http://localhost:8080/registry/downloadCounter"
payload = "[\n {\n \"appName\": \"test-app\",\n \"appVersion\": \"1.6.0\"\n }\n]"
headers = {'Content-type':'application/json', 'Accept':'application/json'}
class MyTaskSet(TaskSet):
#task(2)
def index(self):
response = self.client.post(url = url, data = payload, headers=headers)
print(response.text.encode('utf8'))
print(response.status_code)
class MyLocust(HttpLocust):
task_set = MyTaskSet
wait_time = between(2.0, 4.0)
```

Sending headers to post request

I have this python code that does not work as expected.
import requests
import json
API_ENDPOINT = "https://lkokpdvhc4.execute-api.us-east-1.amazonaws.com/mycall"
data = {'mnumber':'9819838466'}
r = requests.post(url = API_ENDPOINT, data = json.dumps(data))
print (r.text)
This will return an error:
{"stackTrace": [["/var/task/index.py", 5, "handler", "return
mydic[code]"]], "errorType": "KeyError", "errorMessage": "''"}
When I test the API using Amazon console's gateway, I get the expected output (i.e. string like "mumbai"). It means this is client side issue. I have confirmed this by using "postman" as well that returns the same error as mentioned above. How do I send correct headers to post request?
You can create a dictionary with the headers such as
headers = {
"Authorization": "Bearer 12345",
"Content-Type": "application/json",
"key" : "value"
}
Then at the point of making the request pass it as a keyword argument to the request method i.e .post() or .get() or .put
This will be
response = requests.post(API_ENDPOINT, data=json.dumps(data), headers=headers)

Python Request Session JIRA REST post http 405

Using python requests session I can connect to JIRA and retrieve issue information ...
session = requests.Session()
headers = {"Authorization": "Basic %s" % bas64_val}
session.post(jira_rest_url, headers=headers)
jira = session.get(jira_srch_issue_url + select_fields)
# select_fields = the fields I want from the issue
Now I'm trying to post a payload via the JIRA API, using a fixed issue url e.g. "https://my_jira_server.com:1234/rest/api/latest/issue/KEY-9876"
Which should be a case of the following, given: https://developer.atlassian.com/jiradev/jira-apis/about-the-jira-rest-apis/jira-rest-api-tutorials/jira-rest-api-example-edit-issues
payload = { "update": {
"fixVersions": [ {"set": "release-2.139.0"} ]
}}
posted = session.post(jira_task_url, data=payload)
# returns <Response [405]>
# jira_task_url = https://my_jira_server.com:1234/rest/api/latest/issue/KEY-9876
But this doesn't appear to work! Looking into the http 405 response, suggests that my payload is not properly formatted! Which notably, is the not easiest thing to diagnose.
What am I doing wrong here? Any help on this would be much appreciated.
Please note, I am not looking to use the python jira module, I am using requests.session to manage several sessions for different systems i.e. JIRA, TeamCity, etc..
Found the solution! I had two problems:
1) The actual syntax structure should have been:
fix_version = { "update": { "fixVersions": [ {"set" : [{ "name" : "release-2.139.0" }]}]
2) To ensure the payload is actually presented as JSON, use json.dumps() which takes an object and produces a string (see here) AND set 'content-type' to 'application/json':
payload = json.dumps(fix_version)
app_json = { 'content-type': 'application/json' }
session.put(https://.../rest/api/latest/issue/KEY-9876, headers=app_json, data=payload)
Rather than trying to define the JSON manually!

Windows system credentials in Go HTTP NTLM requests

I am looking for the path of least resistance for doing NTLM authentication in a Go HTTP request using the system credentials of the Windows user calling the application.
In C#/.NET, I would be able to achieve this through
WebRequest request = WebRequest.Create(url);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = request.GetResponse();
Stream receiveStream = response.GetResponseStream();
and in Python, the equivalent result can be obtained through
import win32com.client
h = win32com.client.Dispatch('WinHTTP.WinHTTPRequest.5.1')
h.SetAutoLogonPolicy(0)
h.Open('GET', url, False)
h.Send()
but I have not been able to find any resources on how to do the same thing in Go. I could of course use a library for NTLM authentication and manually provide a username/password, but the goal here is to avoid ever putting those in.
After digging into it a bit further, it looks like go-ole can be utilized to make use of WinHTTPRequest in the same way as the Python example in the question. Ignoring all error catching,
package main
import (
"fmt"
ole "github.com/go-ole/go-ole"
"github.com/go-ole/go-ole/oleutil"
)
func main() {
ole.CoInitialize(0)
defer ole.CoUninitialize()
unknown, _ := oleutil.CreateObject("WinHTTP.WinHTTPRequest.5.1")
request, _ := unknown.QueryInterface(ole.IID_IDispatch)
oleutil.CallMethod(request, "SetAutoLogonPolicy", 0)
oleutil.CallMethod(request, "Open", "GET", "http://example.com", false)
oleutil.CallMethod(request, "Send")
resp := oleutil.MustGetProperty(request, "ResponseText")
fmt.Println(resp.ToString())
}

Resources