I am looking for the path of least resistance for doing NTLM authentication in a Go HTTP request using the system credentials of the Windows user calling the application.
In C#/.NET, I would be able to achieve this through
WebRequest request = WebRequest.Create(url);
request.Credentials = CredentialCache.DefaultCredentials;
WebResponse response = request.GetResponse();
Stream receiveStream = response.GetResponseStream();
and in Python, the equivalent result can be obtained through
import win32com.client
h = win32com.client.Dispatch('WinHTTP.WinHTTPRequest.5.1')
h.SetAutoLogonPolicy(0)
h.Open('GET', url, False)
h.Send()
but I have not been able to find any resources on how to do the same thing in Go. I could of course use a library for NTLM authentication and manually provide a username/password, but the goal here is to avoid ever putting those in.
After digging into it a bit further, it looks like go-ole can be utilized to make use of WinHTTPRequest in the same way as the Python example in the question. Ignoring all error catching,
package main
import (
"fmt"
ole "github.com/go-ole/go-ole"
"github.com/go-ole/go-ole/oleutil"
)
func main() {
ole.CoInitialize(0)
defer ole.CoUninitialize()
unknown, _ := oleutil.CreateObject("WinHTTP.WinHTTPRequest.5.1")
request, _ := unknown.QueryInterface(ole.IID_IDispatch)
oleutil.CallMethod(request, "SetAutoLogonPolicy", 0)
oleutil.CallMethod(request, "Open", "GET", "http://example.com", false)
oleutil.CallMethod(request, "Send")
resp := oleutil.MustGetProperty(request, "ResponseText")
fmt.Println(resp.ToString())
}
Related
I wrote a Python script which makes several https requests one after the other by following redirections (its purpose is to login to AWS Cognito via a load balancer):
import requests
session = requests.session()
response = session.get(
f'https://<initial_url>',
allow_redirects=False,
verify=False,
)
response = session.get(
response.headers["Location"],
allow_redirects=False,
verify=False,
)
response = session.post(
response.headers["Location"],
allow_redirects=False,
verify=False,
data={
"_csrf": session.cookies["XSRF-TOKEN"],
"username": <user>,
"password": <password>,
},
)
This works as expected. I would like to use the urllib3 library instead of requests and I transformed the script as follows:
import urllib3
http = urllib3.PoolManager(cert_reqs="CERT_NONE")
response = session.request(
"GET",
f'https://<initial_url>',
redirect=False,
retries=False,
)
response = session.request(
"GET",
response.headers["Location"],
redirect=False,
retries=False,
)
csrf=<get the XSRF-TOKEN cookie from response.headers["Set-Cookie"]>
fields = {
"_csrf": csrf,
"username": <user>,
"password": <password>,
}
response = session.request(
"POST",
response.headers["Location"],
redirect=False,
retries=False,
fields=fields,
)
The GET requests work and the redirects are as expected, but the POST does not (I get an error from Cognito). Based on the documentation
https://docs.python-requests.org/en/latest/user/quickstart/
and
https://urllib3.readthedocs.io/en/latest/user-guide.html
I understand that the equivalent of requests/data is urllib3/fields, as both are used to form-encode data
In the first version of the code, I created a new requests.session() object before the POST request and used it for that and I got the same error as in the urrlib3 case, which led me to believe that urllib3.PoolManager() does not provide a session like requests.session() and that each request is made separately, which makes the POST request fail. Does anyone know if there is a way to have a session in urrlib3 ? I could not find anything in the documentation
I'm trying to use the Groovy HTTPBuilder library to delete some data from Firebase via a HTTP DELETE request. If I use curl, the following works
curl -X DELETE https://my.firebase.io/users/bob.json?auth=my-secret
Using the RESTClient class from HTTPBuilder works if I use it like this:
def client = new RESTClient('https://my.firebase.io/users/bob.json?auth=my-secret')
def response = client.delete(requestContentType: ContentType.ANY)
However, when I tried breaking down the URL into it's constituent parts, it doesn't work
def client = new RESTClient('https://my.firebase.io')
def response = client.delete(
requestContentType: ContentType.ANY,
path: '/users/bob.json',
query: [auth: 'my-secret']
)
I also tried using the HTTPBuilder class instead of RESTClient
def http = new HTTPBuilder('https://my.firebase.io')
// perform a POST request, expecting TEXT response
http.request(Method.DELETE, ContentType.ANY) {
uri.path = '/users/bob.json'
uri.query = [auth: 'my-secret']
// response handler for a success response code
response.success = { resp, reader ->
println "response status: ${resp.statusLine}"
}
}
But this also didn't work. Surely there's a more elegant approach than stuffing everything into a single string?
There's an example of using HttpURLClient in the tests to do a delete, which in its simplest form looks like:
def http = new HttpURLClient(url:'https://some/path/')
resp = http.request(method:DELETE, contentType:JSON, path: "destroy/somewhere.json")
def json = resp.data
assert json.id != null
assert resp.statusLine.statusCode == 200
Your example is very close to the test for the delete in a HTTPBuilder.
A few differences I see are:
Your path is absolute and not relative
Your http url path doesn't end with trailing slash
You're using content type ANY where test uses JSON. Does the target need the content type to be correct? (Probably not as you're not setting it in curl example unless it's doing some voodoo on your behalf)
Alternatively you could use apache's HttpDelete but requires more boiler plate. For a HTTP connection this is some code I've got that works. You'll have to fix it for HTTPS though.
def createClient() {
HttpParams params = new BasicHttpParams()
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1)
HttpProtocolParams.setContentCharset(params, "UTF-8")
params.setBooleanParameter(ClientPNames.HANDLE_REDIRECTS, true)
SchemeRegistry registry = new SchemeRegistry()
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80))
ClientConnectionManager ccm = new PoolingClientConnectionManager(registry)
HttpConnectionParams.setConnectionTimeout(params, 8000)
HttpConnectionParams.setSoTimeout(params, 5400000)
HttpClient client = new DefaultHttpClient(ccm, params)
return client
}
HttpClient client = createClient()
def url = new URL("http", host, Integer.parseInt(port), "/dyn/admin/nucleus$component/")
HttpDelete delete = new HttpDelete(url.toURI())
// if you have any basic auth, you can plug it in here
def auth="USER:PASS"
delete.setHeader("Authorization", "Basic ${auth.getBytes().encodeBase64().toString()}")
// convert a data map to NVPs
def data = [:]
List<NameValuePair> nvps = new ArrayList<NameValuePair>(data.size())
data.each { name, value ->
nvps.add(new BasicNameValuePair(name, value))
}
delete.setEntity(new UrlEncodedFormEntity(nvps))
HttpResponse response = client.execute(delete)
def status = response.statusLine.statusCode
def content = response.entity.content
I adopted the code above from a POST version, but the principle is the same.
I am totally new to WebApi and WebRequests and other things.
After hours of googling, finally, I managed to do POST using C# and HttpWebRequest.
When I do HttpWebRequest in debug mode using Visual Studio I do not get any exceptions.
My app work as I accept , I get data to webApi server and also get back data.
To be sure how my app communicate with WebApi server I start Fiddler Web Debugger.
During the POST to WebApi, Fiddler chace 401 errors
{"Message":"Authorization has been denied for this request."}
Steping step by step in debuger I fund that following lines of code doing 401 error
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(url);
wr.Credentials = new NetworkCredential(username, password);
wr.Method = "POST";
wr.ContentType = "application/json";
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(body);
wr.ContentLength = byteArray.Length;
using (System.IO.Stream dataStream = wr.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length); //After this line of code Fidler Chace HTTP 401
}
Later in code when I do wr.GetResponse() I do get status 200OK.
My questions are :
Do I need to redesign my code to avoid this error in Fiddler ?
Is there other methods to fill HttpWebRequest whit jsonSting beside using GetRequestStream() ?
If your service is enabled with Windows Authentcation, then in Fiddler, you can select the option to automatically authenticate using your logged on credentials by going here:
Composer tab -> Options tab -> Automatically Authenticate
Also, why not use HttpClient from System.Net.Http?...It has a much better and easy programming model...example:
HttpClientHandler handler = new HttpClientHandler();
handler.UseDefaultCredentials = true;
HttpClient client = new HttpClient(handler);
client.BaseAddress = new Uri("http://localhost:9095/");
HttpResponseMessage response = client.PostAsJsonAsync<Customer>("api/values", cust).Result;
I am trying to connect to a web service that uses Kerberos Authentication to authorize the user, but all I get is a 401 unauthorized everytime I try to make the request. Below is the code that I am using. Thanks in advance for any help you can provide!
public XPathNavigator GSASearch(string url, string searchString)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url + searchString);
request.CookieContainer = new CookieContainer();
request.Credentials = CredentialCache.DefaultCredentials;
request.ContentType = "text/xml";
request.Method = "POST";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream receiveStream = response.GetResponseStream();
XPathDocument doc = new XPathDocument(receiveStream);
return doc.CreateNavigator();
}
EDIT: I feel I should explain a bit more what I am attempting to do. I have been tasked with providing a new interface for my company's Google Search Appliance. I am using an ASP.NET page, which does some things like choose a Collection depending on where a user is located, etc. and then sends the appropriate search string the the GSA. This was all working well until they decided to turn authentication on, and now I can't get any results (I either get a 401 unauthorized, or a message stating that 'Data at the root level is invalid'). If I take the search string and provide it directly to the GSA, it authenticates fine, and displays the results, I just can't seem to get it through the HttpWebRequest.
EDIT 2: I did a little more looking (ran the request through Fiddler) and it looks like the request is only attempting Negotiate and not Kerberos. I set the credentials to use Kerberos explicitly as below, but it didn't help...
public XPathNavigator GSASearch(string url, string searchString)
{
CredentialCache credCache = new CredentialCache();
credCache.Add(new Uri(url), "Kerberos", CredentialCache.DefaultNetworkCredentials);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url + searchString);
request.CookieContainer = new CookieContainer();
request.PreAuthenticate = true;
request.Credentials = credCache;
request.ContentType = "text/xml";
request.Method = "POST";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream receiveStream = response.GetResponseStream();
//StreamReader readStream = new StreamReader(receiveStream);
XPathDocument doc = new XPathDocument(receiveStream);
return doc.CreateNavigator();
}
EDIT 3: Ok, looking closer again, the CredentialCache.DefaultCredentials doesn't appear to have my network credentials in it...
1) Have you done a wireshark trace of a successful session to the GSA using the browser? Does that work?
2) If #1 works, what is the WWW-Authenticate header that is sent by the GSA on the first unauthenticated request?
3) Is the machine on which the ASPX app is running a part of the same AD domain that the GSA is in? AFAIK this is probably required for a successful auth.
4) Next, since it is the ASPX app that is doing the request, you cannot use the DefaultCredentials because you actually need the credentials of a user that is trusted by the GSA. For this you should either create a special user account for the app that is talking to the GSA, or have each user be a trusted user on the GSA and have the ASPX page authenticate the user first, then pass those credentials to the GDA using Delegation. For this you will also have to make the server running the ASPX app trusted for delegation.
In my opinion, you should first model your code into a console app that you run, and debug. Then port it to ASPX page. That way you will be able to know if the failure is due to the host (ASPX vs console) or something else.
I'am using the WebRequest class in .net and POST data to a server which is responding with a Response.
The wierd thing is that its working when I started fiddler to analyze my network traffic, but without fiddler it isn't.
So i started to analyze the package which is sent to and from my computer with WireShark. With in this program its simple to follow the TCP-stream. So when I had fiddler on, I can see the correct Request-header/body is sent, and gets the Response-header/body. The strange part is when i dont use fiddler the Request-header is sent, then i´ve got the Response-header/body, and finally the request-body in the end of the TCP-stream.
Here is my code i've been elaborating:
string lcUrl = "http://XX.XX.XXX.XX";
// *** Establish the request
HttpWebRequest loHttp = (HttpWebRequest)WebRequest.Create(lcUrl);
string lcPostData = testdata;
loHttp.Method = "POST";
byte [] lbPostBuffer = System.Text.Encoding.GetEncoding(1252).GetBytes(lcPostData);
loHttp.ContentLength = lbPostBuffer.Length;
loHttp.Credentials = CredentialCache.DefaultCredentials;
//loHttp.SendChunked = true;
loHttp.ServicePoint.Expect100Continue = false;
Stream loPostData = loHttp.GetRequestStream();
loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length);
loPostData.Close();
HttpWebResponse loWebResponse = (HttpWebResponse)loHttp.GetResponse();
Encoding enc = System.Text.Encoding.GetEncoding(1252);
StreamReader loResponseStream = new StreamReader(loWebResponse.GetResponseStream(), enc);
string lcHtml = loResponseStream.ReadToEnd();
loWebResponse.Close();
loResponseStream.Close();
Please use following code. Seems that you have problems with time when underlying stream are send to remote server.
string lcUrl = "http://XX.XX.XXX.XX";
// *** Establish the request
HttpWebRequest loHttp = (HttpWebRequest)WebRequest.Create(lcUrl);
string lcPostData = testdata;
loHttp.Method = "POST";
byte[] lbPostBuffer = System.Text.Encoding.GetEncoding(1252).GetBytes(lcPostData);
loHttp.ContentLength = lbPostBuffer.Length;
loHttp.Credentials = CredentialCache.DefaultCredentials;
//loHttp.SendChunked = true;
loHttp.ServicePoint.Expect100Continue = false;
using (Stream loPostData = loHttp.GetRequestStream())
{
loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length);
}
string lcHtml;
using (HttpWebResponse loWebResponse = (HttpWebResponse)loHttp.GetResponse())
{
Encoding enc = System.Text.Encoding.GetEncoding(1252);
using (StreamReader loResponseStream = new StreamReader(loWebResponse.GetResponseStream(), enc))
{
lcHtml = loResponseStream.ReadToEnd();
}
}
// Perform processing of data here....
Also I could suggest you add following code in the app.config file for your application. This is helps when server returns response that not conforms with way how .NET handle HTTP request.
<configuration>
<system.net>
<settings>
<httpWebRequest
useUnsafeHeaderParsing="true"
/>
</settings>
</system.net>
</configuration>
I have a suspicion that the client is waiting for the "HTTP/1.1 100 continue" response from the server. This is how it works. When you are posting data to the server, sometimes the server might not be ready to accept the data just yet. For eg, it wants to authenticate the client first.
So, when you send a POST request, the client just sends the request headers, with an "Expect: 100-continue" appended.
POST /url HTTP/1.1
Server: Server-name/fqdn
Content-Length: 100
Expect: 100-continue
If the server is ready to receive the data it responds with:
HTTP/1.1 100 continue
Server: server-name/fqdn
Now, the client can send the data.
However if the server is not ready to receive the data, and wants to authenticate the client, it will respond with a different status code.
If you post your wireshark trace to pastebin.com I can verify, but I suspect this is what is happening.
The reason you dont see this in fiddler might be that fiddler is using HttpListener to listen to HTTP request, and HTTP listener hides the intermediate response like 100-continue from the app (in this case fiddler).