When I try to send a request via python3.6 to some urls, it waits until Timeout exception is raised( ConnectionError: HTTPSConnectionPool(host={host}, port=443): Read timed out) . But when I try the same request via python2.7 it is successfully completed with status code: 200. Can you help me?
Version of Requests Package: 2.23.0
Sample Code:
import requests
url = "https://www.khaneyeshoma.ir/"
requests.get(url=url, timeout=10)
Thanks!
sometimes problem occurs cuz of using timeout parameter try :
requests.get(url=url,)
It think is because of the website you are trying to access. The request is correct, but it may need some extra headers.
If you try the request on other address it will work:
import requests
url = "https://www.google.com"
requests.get(url=url, timeout=10)
Response:
<Response [200]>
You can use urllib.request with postman header, and you won't need timeout anymore:
import urllib.request
url = "https://www.khaneyeshoma.ir/"
req = urllib.request.Request(
url,
data=None,
headers={
'User-Agent':"PostmanRuntime/7.6.0"
}
)
response = urllib.request.urlopen(req)
html = response.read()
print(html)
It is because of a whitespace between the header field-name(access-control-expose-headers) and colon. RFC 7230:
No whitespace is allowed between the header field-name and colon. In the past,
differences in the handling of such whitespace have led to security vulnerabilities
in request routing and response handling. A server MUST reject any received request
message that contains whitespace between a header field-name and colon with a
response code of 400 (Bad Request). A proxy MUST remove any such whitespace from a
response message before forwarding the message downstream.
Related
Good afternoon!
For many months I have not been able to generate a POST request to send the firmware.
My method:
Used fiddler to keep track of titles.
With disabled authorization, urllib3 sends the necessary files, but the fiddler could not see it
import urllib3
http = urllib3.PoolManager()
ip="10.254.255.105"
name_file="boot_v1.5_app_v2.27.8.fw"
uri = "/firmware/update_from_page.htm"
with open(name_file, 'rb') as fp:
file_data = fp.read()
send = http.request(
'POST',
'http://' + ip + uri,
fields={'new_firmware': (name_file, file_data),}
)
POSTMAN also worked even with enabled (digest) authorization. I found the correct header line ("Content-Type": "multipart/form-data; border=--------------------7e6a5e075c") needed to send a file with form function -data
Please help to form a request to send a file through the requests library. it is used in the rest of the code and supports the desired digital authorization.
Tried to use the requests library in different ways. I decided to reproduce the sending with disabled authorization based on the working code urllib3, but same errors.
"fields=" missing in the requests library replaced it with others and changed arguments with headers
send = requests.post(url='http://'+ip+uri,
# data={'new_firmware': file_data},
files={'new_firmware': file_data}
# verify=False,
)
date= {'new_firmware': file_data}or {'new_firmware': fp} and analogues - leads to freezing
files= {'new_firmware': (name_file, file_data),} and analogues = requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I noticed the urllib3 header in Wireshark
POST /firmware/update_from_page.htm HTTP/1.1
Host: 10.254.255.105
Accept-Encoding: identity
Content-Length: 524476
Content-Type: multipart/form-data; boundary=4cf5492caddccc65a2b42d8bc691c30f
User-Agent: python-urllib3/1.26.14
Maybe it sends a file in jason format, but so far it has not been possible to figure it out, and so far I have not found working options
I've read https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication and the Basic Authentication chapter from HTTP: The Definitive Guide.
I thought Proxy-Authenticate + Proxy-Authorization + status code 407 was essentially the same as WWW-Authenticate + Authorization + status code 401. I thought if the server responded WWW-Authenticate + 401 or Proxy-Authorization + 407, under both conditions, the browser would pop up an auth dialog, and then the browser would send the credentials with the Authorization or Proxy-Authorization header.
The "WWW-Authenticate combination headers" did work as expected, while the "Proxy combination headers" did not. For Proxy-Authorization + 407, I get ERR_UNEXPECTED_PROXY_AUTH in Chrome and get nothing happened in Firefox(No auth dialog popping up!).
Error in Chrome:
This site can’t be reached.
The webpage at http://localhost:5000/http_auth might be temporarily down or it may have moved permanently to a new web address.
ERR_UNEXPECTED_PROXY_AUTH
So what's the difference between these 2 sets of similar headers? When and where do I use Proxy-Authenticate? Practical exmaples that I can run would be much appreciated.
I am using Python with Flask for testing.
My serverside code:
WWW-Authenticate
#app.route('/www_auth')
def ha():
print("====request headers begin======")
print(request.headers)
print("====request headers end======")
if 'Authorization' in request.headers and request.headers['Authorization'] == 'Basic MTIzOjQ1Ng==':
return render_template('demo.html')
else:
resp = make_response(render_template('demo.html'), 401)
resp.headers['WWW-Authenticate'] = 'Basic realm="WWW-Authenticate required :)"'
return resp
Proxy-Authenticate
#app.route('/proxy_auth')
def haha():
print("====request headers begin======")
print(request.headers)
print("====request headers end======")
if 'Proxy-Authorization' in request.headers and request.headers['Proxy-Authorization'] == 'Basic MTIzOjQ1Ng==':
return render_template('demo.html')
else:
resp = make_response(render_template('demo.html'), 407)
resp.headers['Proxy-Authenticate'] = 'Basic realm="Proxy-Authenticate required :)"'
return resp
I did some tests and here's what I found. (I took a look at RFC and as usual it's too overwhelming :) )
The Proxy-Authenticate set of headers can indeed result in auth pop-up dialog too. But it is something that one must manually set in the client/browser at first. Specifically, for example in Firefox, it's related to the proxy setting.
The Proxy-Authenticate set of headers is used when you connects to a proxy which needs username and password.
Attention: You need to set the root path to your proxy function like this:
#app.route('/')
def haha():
#rest of the code
The workflow is:
-----------------------------------Step 1---------------------------------------------------->
client/browser <---Step 2, 407,Proxy-Authorization response header, required username and password----------- proxy
----Step 3, Proxy-Authorization request headers, contains credentials------------------------> --------> target website
----Subsequent requests, Proxy-Authorization request headers with credentials is still sent--> ---------> target website
In this case, the Proxy-Authorization(with credentials) will be sent automatically for each request.
If the server does not require authentication, then the client can visit the target website directly, and there's no Proxy-Authorization header in the request. (Most free http proxies that you find on the Web works in this way I think)
I also tried the WWW-Authenticate set of headers while I had set the proxy setting in Firefox. The result is that: Every time I visit a new website, I need to authenticate again. So obviously the WWW-Authenticate set of headers aren't meant to be used in this case.
Any other in-depth opinions/explanation would be appreciated. After all I merely did some tests and I want to know more.
there's next site - vten.ru
When I try to make GET request with Postman to it, I give in return status code 304 Not Modified.
Code on Phyton:
import requests
url = "http://vten.ru"
payload = ""
headers = {
'cache-control': "no-cache",
'Postman-Token': "29ae741a-1c31-4a52-b10e-4486cb0d6eb7"
}
response = requests.request("GET", url, data=payload, headers=headers)
print(response.text)
how can I get the page?
You presumably already have a version of the request cached, hence the "Not Modified" response indicating that the response hasn't changed since you last requested it.
EDIT:
Viewing that site/inspecting the network activity via Chrome shows that the document returned is actually http://m.vten.ru. You should try making your GET request to that URL instead.
You also need to add the Accept: text/html header to your request. That returns the page you want having just tested it locally.
I'm using axios in my react-native app, but I can't make any POST request in android simulator/device. It always return 400 Bad Request from the server.
I've tried to set Content-Type: application/json on headers but it didn't work as well.
on postman the request works as expected.
here's the request config object:
As you know, 400 Bad Request error means that the request you sent to the website server was somehow incorrect or corrupted and the server couldn't understand it.
I think there maybe problems on your request like as wrong URL or params or header format...
PS. Axios post methods syntax is as follows:
axios.post(url[, data[, config]])
You can test any axios method here and put it into your code base after it works.
I have seen previous posts about Jetty BadMessage: 400 No Host for HttpChannelOverHttp and I can confirm that I am able to repeat the problem.
I have a Jetty route in Camel Blueprint, which creates another request and forwards on to a Dropwizard service via Camel HTTP.
.process(new Processor() {
//Creates Object for request
}
.marshal(jsonFormat)
.convertBodyTo(String.class)
.setHeader(Exchange.HTTP_URI, simple(serviceEndpoint))
.setHeader(Exchange.HTTP_METHOD, constant(HttpMethod.POST))
.to(userviceEndpoint)
When this request executes, I see the following error on Dropwizard
WARN [2014-11-12 23:15:35,333] org.eclipse.jetty.http.HttpParser: BadMessage: 400 No Host for HttpChannelOverHttp#3aa99dd2{r=0,a=IDLE,uri=-}
This happens constantly, and this problem does not occur when I send a request to the DW service using SOAP-UI (using the serviceEndpoint URL).
Please if anyone has solved this problem, I would like to know how. Thank you.
Capture your network traffic, and post the HTTP request headers you are sending to Jetty.
Odds are that your HTTP client is not sending the Host: header (which is required on HTTP/1.1)
In my case, I was setting header with null value. After removing header having null value from request solved the issue.
Jetty Version: 9.3.8
I got this error when I was making a request with incorrectly formatted headers. So instead of having the header as "X_S_ID: ABC" I had "X_S_ID: ["X_S_ID":BLAH]". So the error sometimes may not literally mean you need to pass a Host header.
Fixing the headers fixed this. Print the exact request you are making and make sure all headers are correctly formatted.