HTTP NTLM authentication - http

I am trying to consume an API which requires NTLM authentication.
This curl command works fine:
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --ntlm -u user:password -d '{ "key1": "100",     "key2": "1"}' http://some/api/v/12
Now I am trying to do the same from a Go program:
package main
import (
"bytes"
"fmt"
"net/url"
"net/http"
"io/ioutil"
"log"
"github.com/Azure/go-ntlmssp"
)
func main() {
url_ := "http://some/api/v/12"
client := &http.Client{
Transport: ntlmssp.Negotiator{
RoundTripper:&http.Transport{},
},
}
data := url.Values{}
data.Set("key1", "100")
data.Set("key2", "1")
b := bytes.NewBufferString(data.Encode())
req, err := http.NewRequest("POST", url_, b)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
req.SetBasicAuth("user", "password")
resp, err := client.Do(req)
if err != nil {
fmt.Printf("Error : %s", err)
} else {
responseData, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
responseString := string(responseData)
fmt.Println(responseString)
resp.Body.Close()
}
}
When I execute this program I receive an "invalid credentials" error which I normally receive when I don't include "--ntlm" flag in the curl command.
Can you please me give me a hint how can I accomplish this task with Go?
Update
printing the request from the curl command:
* About to connect() to www.xxx.xxx.com port xx (#0)
* Trying xxx.xxx.x.xxx...
* Connected to www.xxx.xxx.com (xxx.xxx.x.xx) port xx (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* Server auth using NTLM with user 'user'
> POST /some/api/v2 HTTP/1.1
> Authorization: NTLM xxxxx (44 cahracters)
> User-Agent: curl/7.29.0
> Host: www.xxx.xxx.com
> Content-Type: application/json
> Accept: application/json
> Content-Length: 0
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< WWW-Authenticate: NTLM xxxxx (312 characters)
< Date: Thu, xx Aug xxxx xx:xx:xx xxx
< Content-Length: 341
<
* Ignoring the response-body
* Connection
* Issue another request to this URL: 'http://some/api/v2'
* Found bundle for host www.xxx.xxx.com: 0x0000
* Re-using existing connection!
* Connected to www.xxx.xxx.com (xxx.xxx.x.xx) port xx (#0)
* Server auth using NTLM with user 'user'
> POST /api/v2 HTTP/1.1
> Authorization: NTLM xxx (176 characters)
> User-Agent: curl/7.29.0
> Host: www.xxx.xxx.com
> Content-Type: application/json
> Accept: application/json
> Content-Length: 39
>
* upload completely sent off: 39 out of 39 bytes
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: application/json; charset=utf-8
< Expires: -1
< Server: Microsoft-IIS/7.5
< X-AspNet-Version: 4.0.30319
< Persistent-Auth: true
< X-Powered-By: ASP.NET
< Date: Thu, 08 Aug 2019 06:49:41 GMT
< Content-Length: 1235

NTLM needs a fully qualified Domain\Username login. Email or simple username does not work. So for the username part, it has to look like this:
MYDOMAIN\[username]
where [username] is the actual windows user.

Related

Why do I get error ("status":405,"error":"Method Not Allowed") upon sending data to Thingsboard device via command line

I was just sending some data to my device in thingsboard by writing this command on command line
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/devices/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
but I get this error message at the last
{"timestamp":"2020-01-16T13:09:05.031+0000","status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","path":"/devices/api/v1/IG4Dxxxxxxxxxxxxxxxxs6CCQM/telemetry"}* Connection #0 to host demo.thingsboard.io left intact
what's it I am not doing right?
All the efforts are appreciated
Following are the documentation and whole error message
Thingsboard
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 104.196.24.70:80...
* TCP_NODELAY set
* Connected to demo.thingsboard.io (104.196.24.70) port 80 (#0)
> POST /devices/api/v1/IGxxxxxxxs6CCQM/telemetry HTTP/1.1
> Host: demo.thingsboard.io
> User-Agent: curl/7.65.1
> Accept: */*
> Content-Type:application/json
> Content-Length: 19
>
* upload completely sent off: 19 out of 19 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 405
< Allow: GET, HEAD
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Content-Type: application/json;charset=UTF-8
< Content-Language: en
< Transfer-Encoding: chunked
< Date: Thu, 16 Jan 2020 13:09:04 GMT
<
{"timestamp":"2020-01-16T13:09:05.031+0000","status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","path":"/devices/api/v1/IGxxxxxxxxxxxs6CCQM/telemetry"}* Connection #0 to host demo.thingsboard.io left intact
You need to replace 'device' with 'api' in your URL,in case you are using access tokens, hence, instead of
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/devices/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
You need to use
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"

How enable gzip compression middleware in go-chi

How can I enable gzip compression using the gzip middleware of the go-chi framework?
Try using the example shown here:
https://github.com/go-chi/chi/issues/204
but when I check with curl, I get this:
$ curl -H "Accept-Encoding: gzip" -I http://127.0.0.1:3333
HTTP/1.1 405 Method Not Allowed
Date: Sat, 31 Aug 2019 19:06:39 GMT
I tried the code "hello world":
package main
import (
"net/http"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
)
func main() {
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.Logger)
//r.Use(middleware.DefaultCompress) //using this produces the same result
r.Use(middleware.Compress(5, "gzip"))
r.Get("/", Hello)
http.ListenAndServe(":3333", r)
}
func Hello(w http.ResponseWriter, r *http.Request){
w.Header().Set("Content-Type", "text/html") //according to the documentation this must be here to enable gzip
w.Write([]byte("hello world\n"))
}
but when I try to verify with curl, the result is the same
$ curl -H "Accept-Encoding: gzip" -I http://127.0.0.1:3333
HTTP/1.1 405 Method Not Allowed
Date: Sat, 31 Aug 2019 19:06:39 GMT
what's going on?
The other answers are outdated now. I had to solve this myself, so here what I found out.
Your error is here:
r.Use(middleware.Compress(5, "gzip"))
The second argument ("types") refers to the content types that the compression will be applied to. For example: "text/html", "application/json", etc
Just add a list of the content-types you want to compress, or remove the argument altogether:
func main() {
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.Logger)
r.Use(middleware.Compress(5))
r.Get("/", Hello)
http.ListenAndServe(":3333", r)
}
This will compress all the content-types defined in the default list from middleware.Compress:
var defaultCompressibleContentTypes = []string{
"text/html",
"text/css",
"text/plain",
"text/javascript",
"application/javascript",
"application/x-javascript",
"application/json",
"application/atom+xml",
"application/rss+xml",
"image/svg+xml",
}
Good luck!
r.Use(middleware.DefaultCompress) has now been marked as DEPRECATED.
To enable compression you need to create a compressor, and use its handler.
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.Logger)
compressor := middleware.NewCompressor(flate.DefaultCompression)
r.Use(compressor.Handler())
r.Get("/", Hello)
http.ListenAndServe(":3333", r)
The flate package must be imported as compress/flate.
Use the commented middleware.DefaultCompress and a normal GET request.
package main
import (
"net/http"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
)
func main() {
r := chi.NewRouter()
r.Use(middleware.RequestID)
r.Use(middleware.Logger)
r.Use(middleware.DefaultCompress)
r.Get("/", Hello)
http.ListenAndServe(":3333", r)
}
func Hello(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
w.Write([]byte("hello world\n"))
}
Try with curl:
$ curl -v http://localhost:3333 --compressed
* Rebuilt URL to: http://localhost:3333/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3333 (#0)
> GET / HTTP/1.1
> Host: localhost:3333
> User-Agent: curl/7.58.0
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Content-Encoding: gzip
< Content-Type: text/html
< Date: Sat, 31 Aug 2019 23:37:52 GMT
< Content-Length: 36
<
hello world
* Connection #0 to host localhost left intact
Or HTTPie:
$ http :3333
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 36
Content-Type: text/html
Date: Sat, 31 Aug 2019 23:38:31 GMT
hello world

Unused headers in curl request

I'm trying to emulate a curl -X GET with Go, but the server I'm contacting has authentication. I've followed several sites that recommend me to use r.Header.Add(), but I can't get my curl call to work.
My curl call that actually returns something:
curl -X GET https://myserver.com/test/anothertest -H 'x-access-token: a1b2c3d4'
My code that doesn't return the expected JSON object:
func get(api string, headers map[string]string, dataStruct interface{}) (data interface{}, err error) {
req, _ := http.NewRequest("GET", api, nil)
for k, v := range headers {
req.Header[k] = []string{v}
}
currentHeader, _ := httputil.DumpRequestOut(req, true)
fmt.Println(string(currentHeader))
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return
}
defer resp.Body.Close()
data = dataStruct
err = getJson(api, &data)
if err != nil {
return
}
return
}
func main() {
// args := parse_args(Options{})
args := Options{Api: "my_server",
Headers: map[string]string{
"x-access-token": "a1b2c3d4"}}
body, _ := get(args.Api, args.Headers, HistoryDecoder{})
fmt.Println(body)
}
Returns:
GET /v1pre3/users/current HTTP/1.1
Host: my_server
User-Agent: Go-http-client/1.1
X-Access-Token: a1b2c3d4
Accept-Encoding: gzip
map[ResponseStatus:map[Message:Please ensure that valid credentials are being provided for this API call (This request requires authentication but none were provided)] Notifications:[map[Type:error Item:Please ensure that valid credentials are being provided for this API call (This request requires authentication but none were provided)]]]
Could anyone please tell me what I'm doing wrong?
edit adding curl -v response
* About to connect() to my_server port 443 (#0)
* Trying 54.210.110.53...
* Connected to my_server (54.210.110.53) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.cloud-test.com
* start date: Feb 28 00:00:00 2018 GMT
* expire date: Mar 28 12:00:00 2019 GMT
* common name: *.cloud-test.com
* issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
> GET /v1pre3/users/current HTTP/1.1
> User-Agent: curl/7.29.0
> Host: my_server
> Accept: */*
> x-access-token: a1b2c3d4
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, must-revalidate
< Content-Type: application/json
< Date: Tue, 30 Oct 2018 15:49:00 GMT
< Expires: 0
< Pragma: no-cache
< Server:
< x-capabilities: audit
< X-Content-Type-Options: nosniff
< X-Request-ID: 2018.10.30.15.49.00.9zHjdeQ-ZEmg2IvzTymnxQ
< transfer-encoding: chunked
< Connection: keep-alive
<
* Connection #0 to host my_server left intact
After a lot of rewriting and debugging, I figured out that getJson(api, &data) was also making a http.Get call but without headers. Since that took place after the request we were discussing, it bypassed all testing and printing.

Request Issue Failed with INVALID_ARGUMENT on Google Cloud Vision API

when executing, with a key that has worked in the past but that I haven't used for a few weeks, the following cURL
curl -v -k -s -H "Content-Type: application/json" https://vision.googleapis.com/v1/images:annotate?key=MyKey --data-binary #a.json
where a.json is
{"requests": [{"image": {"content": "SUkqADwmAAD////8gYEoGct1VHdGU..."}, "features": [{"type": "TEXT_DETECTION", "maxResults": 1}]}]}
returns
* Trying 173.194.205.239...
* Connected to vision.googleapis.com (173.194.205.239) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.googleapis.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
> POST /v1/images:annotate?key=MyKey HTTP/1.1
> Host: vision.googleapis.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 13454
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Vary: X-Origin
< Vary: Referer
< Content-Type: application/json; charset=UTF-8
< Date: Wed, 10 Feb 2016 18:02:12 GMT
< Server: ESF
< Cache-Control: private
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Alternate-Protocol: 443:quic,p=1
< Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
{
"error": {
"code": 400,
"message": "Request Issue Failed.",
"status": "INVALID_ARGUMENT"
}
}
* Connection #0 to host vision.googleapis.com left intact
Version 1 of the Google Cloud Vision API (beta) does not permit TIFF [1]. Here is a list of the currently supported formats:
JPEG
PNG8
PNG24
GIF
Animated GIF (first frame only)
BMP
WEBP
RAW
ICO
[1] https://cloud.google.com/vision/docs/image-best-practices#image_types
Turns out this is because I was sending base64 encoded tif images. Works fine for PNGs. Pretty sure I was told this in the docs.

cURL authorization on ASP.NET website

Can anyone help me to figure out what am I doing wrong?
What do we have:
A website where I need to log in via cURL.
Credentials of two accounts that have to be logged in.
What did I get
I managed to log in only one account via cURL, but can't log in the second one. However I can log in with both of them via browser.
What did I try
this one works fine
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login1&tbPassword=password1&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
this one doesn't
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login2&tbPassword=password2&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
The only difference is in the logins and passwords.
I also tried to use separate cookie files for each account.
Here are the logs
Working account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 59
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 59 out of 59 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 150950
<
<!-- Here goes the html of the page with user's account -->
Problem account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 56
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 56 out of 56 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< Location: /error.aspx?aspxerrorpath=/main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
<
* Ignoring the response-body
* Connection #0 to host (nil) left intact
* Issue another request to this URL: 'http://online.tmtr.ru/error.aspx?aspxerrorpath=/main.aspx'
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /error.aspx?aspxerrorpath=/main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 405
<
<!-- Here goes the html of the page with error -->
As I mentioned before, I can successfully log in with both of the accounts via the browser.
How can I figure out why the server rejects one of the accounts via curl but doesn't via browser?
I also tried to send request via Postman extention for the Chrome browser and it works just fine too.
It is because, when you do login using the second account, it is using the cookie from the previously saved cookie. And when server sees a valid cookie, it just redirect you into a different location (or something else decided by the server).
To overcome this, just flush the cookie file before the second curl.
echo ''> ~/cookie.txt
I've contacted with the owner of that site and it turned out that there is a custom validation of the "accept language" header. I didn't send this header at all. I don't know why some requests passed validation and some not, but after I added this header to my requests all gone well :)

Resources