Using the python standard library, is there a way to determine if a given web address should use HTTP or HTTPS? If you hit a site using HTTP://.com is there a standard error code that says hey dummy it should be 'HTTPS' not http?
Thank you
Did u make any sort of testing?
The short, prematural answer of your questions is:
Does not exist should use... it's your preference, or a server decision at all, because of redirects.
Some servers does allow only https, and when you call http does return 302 code.
So, if you goal is to load https from a given url, just try it with a fallback to normal http.
I've recommend you to send only HEAD requests, so you can recognize very fast if the https connection is being listening or not. I do not recommend you to check for port 443 (ssl) because sometimes people do not follow that rule and https protocol will ensure that you is under https and not under a fake 443 port.
A bit of code:
#!/usr/bin/env python
#! -*- coding: utf-8 -*-
from urlparse import urlparse
import httplib, sys
def check_url(url):
url = urlparse(url)
conn = httplib.HTTPConnection(url.netloc)
conn.request("HEAD", url.path)
if conn.getresponse():
return True
else:
return False
if __name__ == "__main__":
url = "http://httpbin.org"
url_https = "https://" + url.split("//")[1]
if check_url(url_https):
print "Nice, you can load it with https"
else:
if check_url(url):
print "https didn't load, but you can use http"
if check_url(url):
print "Nice, it does load with http too"
Related
I've got Fiddler to decrypt https traffic. which works just fine.
http://fiddler2.com/documentation/Configure-Fiddler/Tasks/DecryptHTTPS
But there are some requests that are shown as "Tunnel to" port 443. Also the protocol of these requests are shown as HTTP instead of HTTPS. I do not see the decrypted content of these requests.
on the response details pane, it's shown as below:
Can someone let me know how I can get fiddler to decrypt these tunnelled requests too. Also appreciate if you could provide me some details about what's going on in this case.
Cheers
As noted in the inspector description the line Tunnel to example.org:443 is the first line of a successful captured HTTPS call.
It is notes as HTTP because at that time the HTTPS channel has not been established yet.
If afterwards no further HTTPS requests to that server are shown this means that the program performing the request has not accepted the presented server certificate generated by Fiddler and therefore the HTTPS requests failed.
There are two possible explanations:
The program performing the requests does not trust the Fiddle root CA certificate.
For programs that don't use the Windows certificate store (e.g. Firefox) you have to export this root ca certificate from Fiddler and install it manually into the program's certificate trusted store (list of trusted root-CA certificates).
A second possibility is that the site is certificate pinned. For web browsers this can happen is the web site uses the HSTS configuration.
I want to redirect all HTTP traffic intercepted by mitmproxy to a particular HTTP server, regardless of where the HTTP traffic was destined too.
I know how to set an upstream proxy server for mitmserver, but in this case I don't want another proxy server, but a (destination) HTTP server instead.
Any ideas?
One way to do this would be inject a python script that overwrites the destination of every request. You add a -s script.py parameter to the mitmproxy/mitmdump command (or call master.addons.add(script.Script('script.py')) if you are using mitmproxy library) and add for example the following into your script:
from mitmproxy import http
def request(self, flow: http.HTTPFlow) -> None:
flow.request.host = 'google.com'
flow.request.path = '/'
... further customize request method, cookies, etc etc as needed
I've seen lot of software that tries to determine if current protocol is HTTP or HTTPS, mainly to output links and avoid the Mixed content error.
Usually the software checks some server variables (for example, $_SERVER['HTTP'] in PHP, see this question: PHP Get Site URL Protocol - http vs https).
This method may work, but fails for example when you have a reverse proxy that receives SSL traffic and requests content to a web server over HTTP (so when the software checks the HTTPS status it's off). Web server will response with HTTP links but content is actually server over HTTPS.
There's a simple solution for this: just use links without protocol: '//' instead of 'http://' or 'https://'.
So, my question is: is a better practice to detect current protocol (http or https) instead of just using default protocol for content links (CSS, JS, images, AJAX, etc)? If yes, why is this?
Using '//' works, but it means your resources must be available with http and https.
So you can simply use 'https://' so you are sure to always use the secure connection, and avoid mixed-content errors.
(Of course, the most secure option is to always use https, with a 301 redirect on http and HSTS)
I'm trying to reproduce an odd bug that we believe may be caused by our load balancers trying to check the status of our services, with requests using HTTP/0.9. The service is only configured to use HTTPS, so they are being sent as HTTP/0.9 over HTTPS.
I could use use telnet to send a HTTP/0.9 request, but we have to use HTTPS so that doesn't work. My usual go-to tool for this kind of thing is cURL, but it doesn't look like cURL supports sending 0.9 requests (for good reasons, I know).
What could I use to generate a HTTP/0.9 GET request over HTTPS?
You could use openssl. First establish the SSL connection:
$ openssl s_client -crlf -connect ip:port
CONNECTED
...
lots of output, certificate etc
And then send the request
GET /
[empty line]
Is there a very simple way to access the direct text/bytes of an http request in Python 3.x? Similar to what you would get out of Telnet or something like that. I'm looking for something I can set up to listen on a port, accept the request and directly read what comes across. It wouldn't define it was looking for POST or GET, etc., just the raw values:
Sample value:
GET /index.html/?=request HTTP/1.1
Host: www.example.com
User-Agent: Safari/4.0
The library I was looking for is here. The code in particular I was looking for is this:
import socketserver
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()