How to extract the original response header from a site before redirection using the requests module in python - python-requests

Question

def login_with_requests():
url = "https://url/login/"
login_data = {'csrfmiddlewaretoken':'', 'username':'username', 'password':'password'}
response = requests.get(url)
# print(response.headers)
response_cookies = response.cookies
print(csrf_token)
csrfmiddlewarepattern = re.compile(r'csrfmiddlewaretoken\W\svalue\W{2}([a-zA-Z0-9]+)\W')
matches = csrfmiddlewarepattern.finditer(response.text)
for match in matches:
csrfmiddlewaretoken = match.group(1)
# print(csrfmiddlewaretoken)
login_data['csrfmiddlewaretoken'] = csrfmiddlewaretoken
login_response = requests.post(url, cookies=response_cookies, data=login_data)
print(login_response.headers)
print(login_response.history)
I'm able to successfully login to a site using this code. The problem I have is that when I make a post request to the login site with the necessary parameters, although it is successful the site makes a redirection to the home page. Therefore I receive 2 response headers; the first one is the actual post response (status:302) to the login made containing a redirection header to the home page and the second one is the response containing data meant for the home page.
My problem is that the first response from the site contains a session-id token that I need before I can keep on interacting with the website. But the login_response.headers returns the final response headers which are meant for the request made to the redirected home page.
How can I extract the original response headers received from the site before the redirection as it contains the session-id token that I need for further interaction with the website?
I checked the login_response.history data, it seems to only return the status code for the previous request.

Answer 1

I found a solution, i thought i should share.
def login_with_requests_PaymentSite():
url = "<site.com/login/>"
login_data = {'csrfmiddlewaretoken':'', 'username':<username>, 'password':<password>}
response = requests.get(url)
csrf_token = response.cookies # Cookies returned from site for non-authenticated user.
# Extract csrfmiddlewaretoken that would be used to make a login post request.
csrfmiddlewarepattern = re.compile(r'csrfmiddlewaretoken\W\svalue\W{2}([a-zA-Z0-9]+)\W')
matches = csrfmiddlewarepattern.finditer(response.text)
for match in matches:
csrfmiddlewaretoken = match.group(1)
login_data['csrfmiddlewaretoken'] = csrfmiddlewaretoken # Save csrfmiddlewaretoken in the post login data.
# Start a session
session = requests.Session()
login_session = session.post(url, cookies=csrf_token, data=login_data) # Login to the site
sessionid_cookies = login_session.cookies # Set sessionid cookies that would be used for consecutive requests.
login_response_file = open('Login_Response_Paymentsite.html', 'w')
login_response_file.write(login_session.text)
login_response_file.close()
transaction_history_url = "<site.com/transactions/>"
transaction_history = requests.get(transaction_history_url, cookies=sessionid_cookies)
print("\n Result returned for the transactions page: ")
print(transaction_history.text)
userinfomation_url = "<site.com/userinformation/>"
userinformation = requests.get(userinfomation_url, cookies=sessionid_cookies)
print('\n Result returned for userinformation page: ')
print(userinformation.text)
To be able to make consecutive requests to a site after successful login using the requests module you have to make use of the requests.Session() method. This method helps you to store the session_id returned by the web application after successful login. If you make use of requests.post method instead you won't be able to retrieve the session_id. But using the requests.Session method stores the session_id automatically.
After making the post request;
login_session = session.post(url, cookies=csrf_token, data=login_data) # Login to the site You extract the session_id that would be used for consecutive requests with sessionid_cookies = login_session.cookies