I'm using firebase-ui-web for authentication in my GAE app. There is an annoying issue recently where Safari is blocking 3rd party cookies and this breaks the login process.
The best solution (described here) seems to be implement a reverse-proxy config with nginx. Here are the details:
# reverse proxy for signin-helpers for popup/redirect sign in.
location /__/auth {
proxy_pass https://<project>.firebaseapp.com;
}
Is it possible to accomplish the same thing in GAE where we are not able to add nginx rules? I'm using Python3/Flask if it matters.
With some Googling, I came up with this:
#app.route('/<path:path>', methods=['GET', 'POST'])
def proxy(path):
url = f'https://www.example.com/{path}'
excluded_headers = ['content-encoding', 'content-length',
'transfer-encoding', 'connection']
if request.method == 'GET':
resp = requests.get(url)
elif request.method == 'POST':
resp = requests.post(url, data=request.form)
headers = [(name, value) for (name, value) in resp.raw.headers.items()
if name.lower() not in excluded_headers]
response = Response(resp.content, resp.status_code, headers)
return response
Though I'm not confident that the sources are good so feedback is welcome.
Related
Stuck on this problem where I received this error everytime making POST request to my actix-web server.
CORS header 'Access-Control-Allow-Origin' missing
my javascript (VueJs running on localhost:3000) :
let data = //some json data
let xhr = new XMLHttpRequest();
xhr.open("POST", "http://localhost:8080/abc");
xhr.setRequestHeader("Content-Type", "application/json");
xhr.onload = () => {
console.log(xhr.responseText);
}
xhr.send(JSON.stringify(data));
My Actix_Web server (running on localhost:8080) :
#[actix_web::main]
async fn main() {
HttpServer::new(move || {
let cors = Cors::default()
.allowed_origin("http://localhost:3000/")
.allowed_methods(vec!["GET", "POST"])
.allowed_header(actix_web::http::header::ACCEPT)
.allowed_header(actix_web::http::header::CONTENT_TYPE)
.max_age(3600);
App::new()
.wrap(cors)
.service(myfunc)
})
.bind(("0.0.0.0", 8080))
.unwrap()
.run()
.await
.unwrap();
}
my cargo.toml dependencies
[dependencies]
actix-web = "4"
actix-cors = "0.6.1"
...
Got any idea?
Okay, so I've done some testing. If you're writing a public API, you probably want to allow all origins. For that you may use the following code:
HttpServer::new(|| {
let cors = Cors::default().allow_any_origin().send_wildcard();
App::new().wrap(cors).service(greet)
})
If you're not writing a public API... well, I'm not sure what they want you to do. I've not figured out how to tell the library to send that header. I guess I will look at the code.
UPDATE:
So funny story, this is how you allow specific origins:
let cors = Cors::default()
.allowed_origin("localhost:3000")
.allowed_origin("localhost:2020");
BUT, and oh boy, is that but juicy. The Access-Control-Allow-Origin response header is only set when there is a Origin request header. That header is normally added by the browser in certain cases 1. So I did that (using the Developer tools in the browser). What did I get? "Origin is not allowed to make this request". I set my origin header to localhost:3000. Turns out, the arctix library simply discards that header if no protocol was provided... (e.g. http://) (I assume it discards it, if it deems its format invalid). That internally results in the header being the string "null". Which is, checks notes, not in the list of allowed origins.
And now the grand finale:
Your origin header needs to be set to (by either you or the browser): "http://localhost:3000".
Your configuration needs to include: .allowed_origin("http://localhost:3000").
After doing that, the server will happily echo back your origin header in the Access-Control-Allow-Origin header. And it will only send that one.
I've no idea if any of that is what the standard specifies (or not). I encourage you to read through it, and if it doesn't comply, please open an issue on GitHub. I would do it myself, but I'm done with programming for today.
Cheers!
using latest Requests library and Python 3.8.5, I can't seem to "disable" certificate checking on my API call. I understand the reasons not to disable, but I'd like this to work.
When i attempt to use "verify=True", the servers I connect to throw this error:
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))
When i attempt to use "verify=False", I get:
Error making PS request to [<redacted server name>] at URL https://<redacted server name/rest/v2/api_endpoint: Cannot set verify_mode to CERT_NONE when check_hostname is enabled.
I don't know how to also disable "check_hostname" as I haven't seen a way to do that with the requests library (which I plan to keep and use).
My code:
self.ps_server = server
self.ps_base_url = 'https://{}/rest/v2/'.format(self.ps_server)
url = self.ps_base_url + endpoint
response = None
try:
if req_type == 'POST':
response = requests.post(url, json=post_data, auth=(self.ps_username, self.ps_password), verify=self.verify, timeout=60)
return json.loads(response.text)
elif req_type == 'GET':
response = requests.get(url, auth=(self.ps_username, self.ps_password), verify=self.verify, timeout=60)
if response.status_code == 200:
return json.loads(response.text)
else:
logging.error("Error making PS request to [{}] at URL {} [{}]".format(server, url, response.status_code))
return {'status': 'error', 'trace': '{} - {}'.format(response.text, response.status_code)}
elif req_type == 'DELETE':
response = requests.delete(url, auth=(self.ps_username, self.ps_password), verify=self.verify, timeout=60)
return response.text
elif req_type == 'PUT':
response = requests.put(url, json=post_data, auth=(self.ps_username, self.ps_password), verify=self.verify, timeout=60)
return response.text
except Exception as e:
logging.error("Error making PS request to [{}] at URL {}: {}".format(server, url, e))
return {'status': 'error', 'trace': '{}'.format(e)}
Can someone shed some light on how I can disable check_hostname as well, so that I can test this without SSL checking?
If you have pip-system-certs, it monkey-patches requests as well. Here's a link to the code: https://gitlab.com/alelec/pip-system-certs/-/blob/master/pip_system_certs/wrapt_requests.py
After digging through requests and urllib3 source for awhile, this is the culprit in pip-system-certs:
ssl_context = ssl.create_default_context()
ssl_context.load_default_certs()
kwargs['ssl_context'] = ssl_context
That dict is used to grab an ssl_context later from a urllib3 connection pool but it has .check_hostname set to True on it.
As far as replacing the utility of the pip-system-certs package, I think forking it and making it only monkey-patch pip would be the right way forward. That or just adding --trusted-host args to any pip install commands.
EDIT:
Here's how it's normally initialized through requests (versions I'm using):
https://github.com/psf/requests/blob/v2.21.0/requests/adapters.py#L163
def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK, **pool_kwargs):
"""Initializes a urllib3 PoolManager.
This method should not be called from user code, and is only
exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param connections: The number of urllib3 connection pools to cache.
:param maxsize: The maximum number of connections to save in the pool.
:param block: Block when no free connections are available.
:param pool_kwargs: Extra keyword arguments used to initialize the Pool Manager.
"""
# save these values for pickling
self._pool_connections = connections
self._pool_maxsize = maxsize
self._pool_block = block
# NOTE: pool_kwargs doesn't have ssl_context in it
self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
block=block, strict=True, **pool_kwargs)
And here's how it's monkey-patched:
def init_poolmanager(self, *args, **kwargs):
import ssl
ssl_context = ssl.create_default_context()
ssl_context.load_default_certs()
kwargs['ssl_context'] = ssl_context
return super(SslContextHttpAdapter, self).init_poolmanager(*args, **kwargs)
I am in the process of scraping public data regarding metheorology for a project (data science), and in order to effectively do that I need to change the proxy used on my scrapy requests in the event of a 403 response code.
For this, I have defined a download middleware to handle such situation, which is as follows
class ProxyMiddleware(object):
def process_response(self, request, response, spider):
if response.status == 403:
f = open("Proxies.txt")
proxy = random_line(f) # Just returns a random line from the file with a valid structure ("http://IP:port")
new_request = Request(url=request.url)
new_request.meta['proxy'] = proxy
spider.logger.info("[Response 403] Changed proxy to %s" % proxy)
return new_request
return response
After properly adding the class to settings.py, I expected this middleware to deal with 403 responses by generating a new request with the new proxy, hence finishing in a 200 response. The observed behaviour is that it actually gets executed (I can see the Logger info about Changed proxy), but the new request does not seem to be made. Instead, I'm getting this:
2018-12-26 23:33:19 [bot_2] INFO: [Response] Changed proxy to https://154.65.93.126:53281
2018-12-26 23:33:26 [bot_2] INFO: [Response] Changed proxy to https://176.196.84.138:51336
... indefinitely with random proxies, which makes me think that I'm still retrieving 403 errors and the proxy is not changing.
Reading the documentation, regarding process_response, it states:
(...) If it returns a Request object, the middleware chain is halted and the returned request is rescheduled to be downloaded in the future. This is the same behavior as if a request is returned from process_request().
Is it possible that "in the future" is not "right after it is returned"? How should I do to change the proxy for all requests from that moment on?
Scrapy will drop duplicate requests to the same url by default, so that's probably what's happening on your spider. To check if this is your case you can set this settings:
DUPEFILTER_DEBUG=True
LOG_LEVEL='DEBUG'
To solve this you should add dont_filter=True:
new_request = Request(url=request.url, dont_filter=True)
Try this:
class ProxyMiddleware(object):
def process_response(self, request, response, spider):
if response.status == 403:
f = open("Proxies.txt")
proxy = random_line(f)
new_request = Request(url=request.url)
new_request.meta['proxy'] = proxy
spider.logger.info("[Response 403] Changed proxy to %s" % proxy)
return new_request
else:
return response
A better approach would be to use scrapy random proxies module instead:
'DOWNLOADER_MIDDLEWARES' : {
'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_proxies.middlewares.BanDetectionMiddleware': 620
},
I am looking for the path of least resistance for doing NTLM authentication in a Go HTTP request using the system credentials of the Windows user calling the application.
In C#/.NET, I would be able to achieve this through
WebRequest request = WebRequest.Create(url);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = request.GetResponse();
Stream receiveStream = response.GetResponseStream();
and in Python, the equivalent result can be obtained through
import win32com.client
h = win32com.client.Dispatch('WinHTTP.WinHTTPRequest.5.1')
h.SetAutoLogonPolicy(0)
h.Open('GET', url, False)
h.Send()
but I have not been able to find any resources on how to do the same thing in Go. I could of course use a library for NTLM authentication and manually provide a username/password, but the goal here is to avoid ever putting those in.
After digging into it a bit further, it looks like go-ole can be utilized to make use of WinHTTPRequest in the same way as the Python example in the question. Ignoring all error catching,
package main
import (
"fmt"
ole "github.com/go-ole/go-ole"
"github.com/go-ole/go-ole/oleutil"
)
func main() {
ole.CoInitialize(0)
defer ole.CoUninitialize()
unknown, _ := oleutil.CreateObject("WinHTTP.WinHTTPRequest.5.1")
request, _ := unknown.QueryInterface(ole.IID_IDispatch)
oleutil.CallMethod(request, "SetAutoLogonPolicy", 0)
oleutil.CallMethod(request, "Open", "GET", "http://example.com", false)
oleutil.CallMethod(request, "Send")
resp := oleutil.MustGetProperty(request, "ResponseText")
fmt.Println(resp.ToString())
}
I'm new to angular2 and to be fair I have very few knowledges which I try to fix, however I've run into some issues about cross site request, trying to access a service from another application but I have this issue whatever I try to do
XMLHttpRequest cannot load https://hr/Team/EditEmployeeInfo.aspx. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:54396' is therefore not allowed access. The response had HTTP status code 401.
This is my angular2 service and I've tried something like this
getUserHrtbProfile(userId): Promise<any> {
const headers = new Headers();
headers.append('Access-Control-Allow-Headers', 'Content-Type');
headers.append('Access-Control-Allow-Methods', 'GET, PUT, POST, DELET');
headers.append('Access-Control-Allow-Origin', '*');
var apiUri: string = "https://hrtb/Team/EditEmployeeInfo.aspx?emplid={0}&Menu=InfoEmployee&T=0".replace("{0}", userId);
return this.http.get(apiUri, headers).map(result => result.json()).toPromise();
}
and this is my component
this.bannerService.getUserHrtbProfile(this.userId).then(hrtbJson => {
this.hasHrtbAccess = hrtbJson.HasHrtbAccess;
this.hrtbProfileUrl = hrtbJson.HrtbProfileUrl;
}).catch(err => {
this.hasHrtbAccess = false;
});
I've search a solution on my problem but still could not find one that suits my need.
Angular 2 http request with Access-Control-Allow-Origin set to *
But most important, is this an angular2 problem that I need to solve? Or in fact as I've read it should have been handled by the team that exposes the API?
Thank you all.
You need to enable CORS on your API backend.
Only for testing purpose you could use this Chrome Extension to simulate CORS on your api backend:
https://chrome.google.com/webstore/detail/allow-control-allow-origi/nlfbmbojpeacfghkpbjhddihlkkiljbi
You are trying to make request on other domain, this is what you can not resolve here. try with making request at you backed code, this will resolve you issue.