I would like to know how to include a cert file when sending requests in HTTP.jl.
In Python, using Requests it would look like this,
requests.get(url, params=payload, verify=cert_file)
The documentation mentions SSL certs, but is unclear.
It really is poorly documented, and in similar cases I've had to look at the source code to
MbedTLS (within the site https://tls.mbed.org/), which is what the package HTTP.jl calls for certificates.
MbedTLS in turn looks for the systems's installed certificates, so if you install the certificate for your user, HTTP.jl should use it for https. I realize this may not help your specific need, which may require something like this (untested):
using HTTP, MbedTLS
conf = MbedTLS.SSLConfig(cert_file, key_file)
resp = HTTP.get("https://httpbin.org/ip", sslconfig=conf)
println(resp)
itself.
If you have to go back to the MbedTLS source as I did, I suggest you look at the example at https://github.com/JuliaLang/MbedTLS.jl and the source at
https://github.com/JuliaLang/MbedTLS.jl/blob/master/src/MbedTLS.jl,
especially the function SSLConfig(cert_file, key_file) on line 103.
Related
I need to make a request to an internal web service, and need to provide a custom SSL certificate chain.
In python + requests, I would set the REQUESTS_CA_BUNDLE environment variable to the path of the bundle, /etc/ssl/certs/ca-bundle.crt. What is the equivalent with Julia's HTTP.jl? It doesn't seem to be picking up the bundle automatically.
HTTP.jl uses MbedTLS to process certificates, so I wonder if your Julia install somehow is missing that library. You might try installing MbedTLS directly for you platform and see where it looks for certificates by default.
According to the docs you can pass an sslconfig object to the call. You can supply the certificate to this object:
Untested
using HTTP, MbedTLS
conf = MbedTLS.SSLConfig(cert_file, key_file)
resp = HTTP.get("https://httpbin.org/ip", sslconfig=conf)
println(resp)
I want to make a bot that would send some post requests and other http requests. I want to ask you what is the easiest language to do this. The language that requires the least of writing, maybe not to install any modules, and also the most simple one.
Python has an awesome module called requests. A simple post request:
import requests
payload = {'key':'value'}
r = requests.post('http://httpbin.org/post', data = payload)
maybe not to install any modules, and also the most simple one.
It's so simple to install a Python module like requests, just this:
> pip install requests
require(data.table)
require(httr)
url = "http://www.dropbox.com/s/0brabdf53lc37i/data.csv?dl=1"
request <- GET(url)
Loading required package: data.table
Loading required package: httr
Error in curl::curl_fetch_memory(url, handle = handle) :
Couldn't resolve host name
Calls: GET ... request_fetch -> request_fetch.write_memory -> -> .Call
Execution halted
What gives? The URL works fine in my browser and others have had success downloading dropbox files this way...
I have an answer to your question, after a lot of searching around. When I noticed that R was changing the protocol of your DropBox URL from http to https, I became suspicious that you might have a certificate problem. As this SO post mentions, this seems to precisely be the case. Try using this code:
require(data.table)
require(httr)
cafile <- system.file("CurlSSL", "cacert.pem", package = "RCurl")
url = "http://www.dropbox.com/s/0brabdf53lc37i/data.csv?dl=1"
request <- GET(url, config(cainfo = cafile))
What is happening here:
The cert file cacert.pem contains a list of trusted certificates, issued from a CA (Certificate Authority). When DropBox sends R its public SSL certificate, R will search through this list of trusted certificates to see if it can find it. If it can, it will allow the SSL handshake to complete, and your data will be downloaded.
The reason why you are having this problem but many who read your question do not have it, is that you likely never configured your curl settings in R.
The "can't resolve" message is occurring because dropbox is turning your http request into an https request (not all services can follow redirects) and (most likely) because your download protocol can't handle secure http and because the url doesn't request the raw data...
All problems are best fixed by moving to the format the host (dropbox) will direct you too, ie., https, and then switching your code (if necessary) to use the new protocol, and correcting the url to tell dropbox to serve the raw file (not the ?dl=1 suffix you're using, but ?raw=1
So:
switch the url to secure
"https://www.dropbox.com/s/0brabdf53lc37i/data.csv?dl=1"
Switch the request to raw
"http://www.dropbox.com/s/0brabdf53lc37i/data.csv?raw=1"
Test that in a browser - would work if this wasn't a bad link.
Open using R functions like url() that can handle secure transport (example of this is at this answer
I write a bunch of scripts in javascript and want to switch to dart or at least start using it. I do have one question though: i know js doesn't support x-domain requests in the browser, but what about running a dart application/script from the server, is it still the same? can this even be done?
basically, since i have no access to the web server to which i query, cross domain ability is a big necessity.
It sounds like you might be asking about writing a server side command line script which can make requests to an HTTP server. Though the wording of question isn't totally clear to me. (The answers above are about browser based Dart scripts.)
This is possible with Dart. There are no cross origin restrictions in this case.
See the HttpClient class.
Or you can use the http package on pub.
I recommend using the http package, as it provides a simpler high-level interface.
Here is an example using the http package:
import 'dart:io';
import 'package:http/http.dart' as http;
main() {
http.read("http://google.com").then((content) {
print(content);
});
}
You'll need to update your pubspec.yaml file to add the following dependencies:
name: Http Example
dependencies:
http: any
pathos: any
(Actually, you should only need to include http, but I think the http package is missing the pathos dependency in it's pubspec.yaml file.)
I couldn't find the pretty documentation for http, but there is some doc comments in the source file.
Cross domain security is built into the browser, so is neither a feature of Dart or JavaScript.
Useful for testing: you can pass flags into chrome that will disable this security feature. See here: http://joshuamcginnis.com/2011/02/28/how-to-disable-same-origin-policy-in-chrome/
If you want to do a GET request, then you can use Dart JavaScript interop, see this section in this article: http://www.dartlang.org/articles/json-web-service/#note-on-cors
If you want to POST requests on the other hand, you'll run into problems unless the target server supports CORS. More details here: https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS
Edit: The correct approach would be to have your client-side code communicate with a server you own, and have that server communicate with the third-party server.
You can enable cross-origin requests on the server by setting the Access-Control-Allow-Origin header on the http response. The value is either * to allow any origin to access the resource, but it is definitely safer to specify valid origins.
I have some software which makes a request to a specific URL in internet and I want it to receive my custom response. Is there any software tool for that on Windows? Also it would be nice if I could map a regexp instead of specific URL
Found the solution myself:
Set the domain of the URL to point to 127.0.0.1 in windows hosts file
Install nginx and set it up to show your file for the request response to which you're willing to modify and proxy all other requests to the original server
You could consider writing a test and mocking out the http response with your custom response.
I could give an example using C# and rhino mocks but it's not clear which platform you are working with.
You can:
Try to enject your dll into the process and replace functions like (HttpSendRequest, HttpQueryInfo,...) with your oun versions.
Try to use something like WinPCap (http://www.winpcap.org/).
Fiddler (www.fiddler2.com) has an AutoResponder feature which does exactly that.