I am trying to make an Http request which needs to go through a proxy. I saw an answer referring how to do it with create-elm-app, but I am not using this and was wondering if there is an easy method of doing this? I have tried with the Via header (left down below to see the use I am going for), but that probably has nothing to do with it.
Do I need to use some kind of behind the scene trickery or is there a native way of doing it?
fetchUiElements : Cmd Msg
fetchUiElements =
let
baseUrl = "https://example.com/"
responseDecoder =
Json.Decode.field "data" Json.Decode.string
httpBody =
Http.jsonBody <|
Json.Encode.object
[ ("", Json.Encode.string "")
, ("", Json.Encode.string "")
]
in
Http.request
{ method = "POST"
, headers = [ Http.header "Via" "http://proxy.com:8888"]
, url = baseUrl ++ "getUiElements"
, body = httpBody
, timeout = Nothing
, tracker = Nothing
, expect = Http.expectJson FetchUiElements responseDecoder
}
Edit: For further clarification I want the same behaviour as this curl
curl -x "http://proxy:8888" -v https://endpoint/getstuff
Related
I'm using lua-http for HTTP requests in my Lua script. I'm trying to find a way to send data as a POST request, similar to the -d option of curl.
I've tried new_from_uri:set_body() but I don't think I'm doing it correctly.
request = require "http.request"
headers, stream = assert(request.new_from_uri("https://example.org"):set_body("body text"))
headers, stream = assert(request.new_from_uri("https://example.org"):go())
body = assert(stream:get_body_as_string())
if headers:get ":status" ~= "200" then
error(body)
end
Could someone show me how to do this properly?
I've decided to use luasocket for this instead. Here is the code I'm using:
http = require "socket.http"
body = "body text"
respbody = {
result, respcode, respheaders, respstatus = http.request {
method = "POST",
url = "https://example.org",
source = ltn12.source.string(body),
headers = {
["content-type"] = "application/json", -- change if you're not sending JSON
["content-length"] = tostring(#body)
},
sink = ltn12.sink.table(respbody)
}
respbody = table.concat(respbody)
I am writing a http proxy which needs to call DumpRequest. For some reason, the client does not offer "Host" header while the server needs it. I then do this:
if req.Header.Get("host") == "" {
req.Header.Set("Host", "www.domain.com:80")
}
data, _ := httputil.DumpRequest(req, true)
conn.Write(data)
The problem is after I set "host", data still does not has it. I digged into source code of DumpRequest, found this:
var reqWriteExcludeHeaderDump = map[string]bool{
"Host": true, // not in Header map anyway
"Transfer-Encoding": true,
"Trailer": true,
}
Why is Host "not in Header map anyway"?
Short answer
I guess you should use Header.Add() instead of Header.Set()
I didn't try it. So if my answer is wrong, be free to figure it out.
Reference
From source code , It said that.
It replaces any existing values associated with key
But in you case, you don't have that K existing. So it behaves as we expected.
If you use Set() method, it's cool.
DumpRequest should not be used to implement a proxy - see doc:
It should only be used by servers to debug client requests. The returned
representation is an approximation only ...
If you are trying to change the origin host value of the request while it goes through your proxy, you do not need to change the header of host. You can do the following:
if req.Host == "" {
req.Host = "www.domain.com"
}
data, _ := httputil.DumpRequest(req, true)
conn.Write(data)
Playground
https://play.golang.org/p/f-qaZRC0RMO
Using python requests session I can connect to JIRA and retrieve issue information ...
session = requests.Session()
headers = {"Authorization": "Basic %s" % bas64_val}
session.post(jira_rest_url, headers=headers)
jira = session.get(jira_srch_issue_url + select_fields)
# select_fields = the fields I want from the issue
Now I'm trying to post a payload via the JIRA API, using a fixed issue url e.g. "https://my_jira_server.com:1234/rest/api/latest/issue/KEY-9876"
Which should be a case of the following, given: https://developer.atlassian.com/jiradev/jira-apis/about-the-jira-rest-apis/jira-rest-api-tutorials/jira-rest-api-example-edit-issues
payload = { "update": {
"fixVersions": [ {"set": "release-2.139.0"} ]
}}
posted = session.post(jira_task_url, data=payload)
# returns <Response [405]>
# jira_task_url = https://my_jira_server.com:1234/rest/api/latest/issue/KEY-9876
But this doesn't appear to work! Looking into the http 405 response, suggests that my payload is not properly formatted! Which notably, is the not easiest thing to diagnose.
What am I doing wrong here? Any help on this would be much appreciated.
Please note, I am not looking to use the python jira module, I am using requests.session to manage several sessions for different systems i.e. JIRA, TeamCity, etc..
Found the solution! I had two problems:
1) The actual syntax structure should have been:
fix_version = { "update": { "fixVersions": [ {"set" : [{ "name" : "release-2.139.0" }]}]
2) To ensure the payload is actually presented as JSON, use json.dumps() which takes an object and produces a string (see here) AND set 'content-type' to 'application/json':
payload = json.dumps(fix_version)
app_json = { 'content-type': 'application/json' }
session.put(https://.../rest/api/latest/issue/KEY-9876, headers=app_json, data=payload)
Rather than trying to define the JSON manually!
I am trying to convert a cURL command to a Google Apps Script PUT request.
I came up with the below google script but it does not seem to work. When I change the URL of both of these to RequestBin in order to probe what the server side is getting, RequestBin shows that the raw source of both of these come through as exactly the same. Yet, only the cURL command is able to perform the desired action (update a file on github) when I change it back to the github URL.
curl -i -X PUT -H 'Authorization: token yadayda' -d '{"path":"mygeojson.json","message":"Initial Commit","committer":{"name":"Sidd","email":"siddsubra#gmail.com"},"content":"bXkgbmV3IGZpbGUgY29udGVudHM=","sha":"0d5a690c8fad5e605a6e8766295d9d459d65de42","branch":"master"}' https://api.github.com/repos/teku45/GeoJSON-Update/contents/mygeojson.json
cURL Command
And this is the Google Script
function mainscript() {
var sha = getSHA();
var authenticationToken = "yadayada";
var url = "http://api.github.com/repos/teku45/GeoJSON-Update/contents/mygeojson.json";
var headers = {
"Authorization" : "token " + authenticationToken,
};
var payload = {
"path": "mygeojson.json",
"message": "Initial Commit",
"committer":{
"name": "Sidd",
"email": "siddsubra#gmail.com"
},
"content": "bXkgbmV3IGZpbGUgY29udGVudHM=",
"sha": sha,
"branch": "master"
};
var options = {
"headers" : headers,
"method" : "PUT",
"payload" : JSON.stringify(payload)
};
Logger.log(options);
Logger.log(UrlFetchApp.fetch(url, options));
}
Google Apps Script
I recently did a script that call a request to a API from scrapy cloud, and had a dificulty to convert curl to App Script request too.
I noticed that payload doesn't worked when i used JSON.stringify, so try remove it from payload:
"payload" : payload
Other observation was the need to use Utilities.base64Encode in authentication token:
headers = {"Authorization":"Basic " + Utilities.base64Encode(authenticationToken + ":" + "")}
See this sample:
url = "https://app.scrapinghub.com/api/run.json"
headers = {"Authorization":"Basic " + Utilities.base64Encode("myToken" + ":" + "")}
payload = {"project":"327107", "spider":"pmp"}
options = {"headers":headers, "payload":payload}
response = UrlFetchApp.fetch(url, options)
data = JSON.parse(response.getContentText())
I'm trying to use the Groovy HTTPBuilder library to delete some data from Firebase via a HTTP DELETE request. If I use curl, the following works
curl -X DELETE https://my.firebase.io/users/bob.json?auth=my-secret
Using the RESTClient class from HTTPBuilder works if I use it like this:
def client = new RESTClient('https://my.firebase.io/users/bob.json?auth=my-secret')
def response = client.delete(requestContentType: ContentType.ANY)
However, when I tried breaking down the URL into it's constituent parts, it doesn't work
def client = new RESTClient('https://my.firebase.io')
def response = client.delete(
requestContentType: ContentType.ANY,
path: '/users/bob.json',
query: [auth: 'my-secret']
)
I also tried using the HTTPBuilder class instead of RESTClient
def http = new HTTPBuilder('https://my.firebase.io')
// perform a POST request, expecting TEXT response
http.request(Method.DELETE, ContentType.ANY) {
uri.path = '/users/bob.json'
uri.query = [auth: 'my-secret']
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
}
}
But this also didn't work. Surely there's a more elegant approach than stuffing everything into a single string?
There's an example of using HttpURLClient in the tests to do a delete, which in its simplest form looks like:
def http = new HttpURLClient(url:'https://some/path/')
resp = http.request(method:DELETE, contentType:JSON, path: "destroy/somewhere.json")
def json = resp.data
assert json.id != null
assert resp.statusLine.statusCode == 200
Your example is very close to the test for the delete in a HTTPBuilder.
A few differences I see are:
Your path is absolute and not relative
Your http url path doesn't end with trailing slash
You're using content type ANY where test uses JSON. Does the target need the content type to be correct? (Probably not as you're not setting it in curl example unless it's doing some voodoo on your behalf)
Alternatively you could use apache's HttpDelete but requires more boiler plate. For a HTTP connection this is some code I've got that works. You'll have to fix it for HTTPS though.
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
HttpClient client = createClient()
def url = new URL("http", host, Integer.parseInt(port), "/dyn/admin/nucleus$component/")
HttpDelete delete = new HttpDelete(url.toURI())
// if you have any basic auth, you can plug it in here
def auth="USER:PASS"
delete.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
// convert a data map to NVPs
def data = [:]
List<NameValuePair> nvps = new ArrayList<NameValuePair>(data.size())
data.each { name, value ->
nvps.add(new BasicNameValuePair(name, value))
}
delete.setEntity(new UrlEncodedFormEntity(nvps))
HttpResponse response = client.execute(delete)
def status = response.statusLine.statusCode
def content = response.entity.content
I adopted the code above from a POST version, but the principle is the same.