when executing, with a key that has worked in the past but that I haven't used for a few weeks, the following cURL
curl -v -k -s -H "Content-Type: application/json" https://vision.googleapis.com/v1/images:annotate?key=MyKey --data-binary #a.json
where a.json is
{"requests": [{"image": {"content": "SUkqADwmAAD////8gYEoGct1VHdGU..."}, "features": [{"type": "TEXT_DETECTION", "maxResults": 1}]}]}
returns
* Trying 173.194.205.239...
* Connected to vision.googleapis.com (173.194.205.239) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.googleapis.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
> POST /v1/images:annotate?key=MyKey HTTP/1.1
> Host: vision.googleapis.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 13454
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Vary: X-Origin
< Vary: Referer
< Content-Type: application/json; charset=UTF-8
< Date: Wed, 10 Feb 2016 18:02:12 GMT
< Server: ESF
< Cache-Control: private
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Alternate-Protocol: 443:quic,p=1
< Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
{
"error": {
"code": 400,
"message": "Request Issue Failed.",
"status": "INVALID_ARGUMENT"
}
}
* Connection #0 to host vision.googleapis.com left intact
Version 1 of the Google Cloud Vision API (beta) does not permit TIFF [1]. Here is a list of the currently supported formats:
JPEG
PNG8
PNG24
GIF
Animated GIF (first frame only)
BMP
WEBP
RAW
ICO
[1] https://cloud.google.com/vision/docs/image-best-practices#image_types
Turns out this is because I was sending base64 encoded tif images. Works fine for PNGs. Pretty sure I was told this in the docs.
Related
I have a web app hosted at A.B.C.D:5601 and when I try curl A.B.C.D:5601 it doesn't print out anything on the screen however when I open the same link using a browser it does open the webapp. But it gets forwarded to A.B.C.D:5601/foo/bar and it opens fine.
Here is the output of curl -v
* Trying A.B.C.D:5601...
* TCP_NODELAY set
* Connected to A.B.C.D port 5601 (#0)
> GET / HTTP/1.1
> Host: A.B.C.D:5601
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /spaces/enter
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 0
< Date: Tue, 08 Jun 2021 03:01:11 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
* Connection #0 to host A.B.C.D left intact
Why is curl not giving me the response back?
Curl is telling you you are being redirected to /spaces/enter.
You can tell Curl to automatically follow redirects:
curl -vL [url]
I have a Java client, that is making a POST call to the v1/graphql endpoint of a Hasura server (v1.3.3)
I'm making the HTTP call using the Square okhttp3 library (v4.9.1). The data transfer is happening over HTTP1.1, using chunked transfer-encoding.
The client is failing with the following error:
Caused by: java.net.ProtocolException: unexpected end of stream
at okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.kt:415) ~[okhttp-4.9.1.jar:?]
at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:276) ~[okhttp-4.9.1.jar:?]
at okio.RealBufferedSource.read(RealBufferedSource.kt:189) ~[okio-jvm-2.8.0.jar:?]
at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:197) ~[okio-jvm-2.8.0.jar:?]
at okio.InflaterSource.refill(InflaterSource.kt:112) ~[okio-jvm-2.8.0.jar:?]
at okio.InflaterSource.readOrInflate(InflaterSource.kt:76) ~[okio-jvm-2.8.0.jar:?]
at okio.InflaterSource.read(InflaterSource.kt:49) ~[okio-jvm-2.8.0.jar:?]
at okio.GzipSource.read(GzipSource.kt:69) ~[okio-jvm-2.8.0.jar:?]
at okio.Buffer.writeAll(Buffer.kt:1642) ~[okio-jvm-2.8.0.jar:?]
at okio.RealBufferedSource.readString(RealBufferedSource.kt:95) ~[okio-jvm-2.8.0.jar:?]
at okhttp3.ResponseBody.string(ResponseBody.kt:187) ~[okhttp-4.9.1.jar:?]
Request Headers:
INFO: Content-Type: application/json; charset=utf-8
INFO: Content-Length: 1928
INFO: Host: localhost:10191
INFO: Connection: Keep-Alive
INFO: Accept-Encoding: gzip
INFO: User-Agent: okhttp/4.9.1
Response headers:
INFO: Transfer-Encoding: chunked
INFO: Date: Tue, 27 Apr 2021 12:06:39 GMT
INFO: Server: Warp/3.3.10
INFO: x-request-id: d019408e-e2e3-4583-bcd6-050d4a496b11
INFO: Content-Type: application/json; charset=utf-8
INFO: Content-Encoding: gzip
This is the client code used for the making the POST call:
private static final MediaType MEDIA_TYPE_JSON = MediaType.parse("application/json; charset=utf-8");
private static OkHttpClient okHttpClient = new OkHttpClient.Builder()
.connectTimeout(30, TimeUnit.SECONDS)
.writeTimeout(5, TimeUnit.MINUTES)
.readTimeout(5, TimeUnit.MINUTES)
.addNetworkInterceptor(loggingInterceptor)
.build();
public GenericHttpResponse httpPost(String url, String textBody, GenericHttpMediaType genericMediaType) throws HttpClientException {
RequestBody body = RequestBody.create(okHttpMediaType, textBody);
Request postRequest = new Request.Builder().url(url).post(body).build();
Call postCall = okHttpClient.newCall(okHttpRequest);
Response postResponse = postCall.execute();
return GenericHttpResponse
.builder()
.body(okHttpResponse.body().string())
.headers(okHttpResponse.headers().toMultimap())
.code(okHttpResponse.code())
.build();
}
This failure is only happening for large response sizes. As per the server logs, the response size (after gzip encoding) is around 52MB, but the call is still failing. This same code has been working fine for response sizes around 10-15MB.
I tried replicating the same issue through a simple cURL call, but that ran successfully:
curl -v -s --request POST 'http://<hasura_endpoint>/v1/graphql' \
--header 'Content-Type: application/json' \
--header 'Accept-Encoding: gzip, deflate, br' \
--data-raw '...'
* Trying ::1...
* TCP_NODELAY set
* Connected to <host> (::1) port <port> (#0)
> POST /v1/graphql HTTP/1.1
> Host: <host>:<port>
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/json
> Accept-Encoding: gzip, deflate, br
> Content-Length: 1840
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
} [1840 bytes data]
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Date: Tue, 27 Apr 2021 11:59:24 GMT
< Server: Warp/3.3.10
< x-request-id: 27e3ff3f-8b95-4328-a1bc-a5492e68f995
< Content-Type: application/json; charset=utf-8
< Content-Encoding: gzip
<
{ [6 bytes data]
* Connection #0 to host <host> left intact
* Closing connection 0
So I'm assuming that this error is specific to the Java client.
Based on suggestions provided in similar posts, I tried the following other approaches:
Adding a Connection: close header to the request
Sending Transfer-Encoding: gzip header in the request
Setting the retryOnConnectionFailure for the OkHttp client to true
But none of these approaches were able to resolve the issue.
So, my questions are:
What could be the underlying cause for this issue? Since I'm using chunked transfer encoding here, I suppose it's not due to an incorrect content-length header passed in the response.
What are the approaches I can try for debugging this further?
Would really appreciate any insights on this. Thank you.
I was just sending some data to my device in thingsboard by writing this command on command line
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/devices/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
but I get this error message at the last
{"timestamp":"2020-01-16T13:09:05.031+0000","status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","path":"/devices/api/v1/IG4Dxxxxxxxxxxxxxxxxs6CCQM/telemetry"}* Connection #0 to host demo.thingsboard.io left intact
what's it I am not doing right?
All the efforts are appreciated
Following are the documentation and whole error message
Thingsboard
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 104.196.24.70:80...
* TCP_NODELAY set
* Connected to demo.thingsboard.io (104.196.24.70) port 80 (#0)
> POST /devices/api/v1/IGxxxxxxxs6CCQM/telemetry HTTP/1.1
> Host: demo.thingsboard.io
> User-Agent: curl/7.65.1
> Accept: */*
> Content-Type:application/json
> Content-Length: 19
>
* upload completely sent off: 19 out of 19 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 405
< Allow: GET, HEAD
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Content-Type: application/json;charset=UTF-8
< Content-Language: en
< Transfer-Encoding: chunked
< Date: Thu, 16 Jan 2020 13:09:04 GMT
<
{"timestamp":"2020-01-16T13:09:05.031+0000","status":405,"error":"Method Not Allowed","message":"Request method 'POST' not supported","path":"/devices/api/v1/IGxxxxxxxxxxxs6CCQM/telemetry"}* Connection #0 to host demo.thingsboard.io left intact
You need to replace 'device' with 'api' in your URL,in case you are using access tokens, hence, instead of
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/devices/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
You need to use
curl -v -X POST -d "{\"temperature\": 25}" http://demo.thingsboard.io/api/v1/IG4XXXXXXXXXCQM/telemetry
--header "Content-Type:application/json"
I've been working on an R interface to a HTTP API with digest authentication and I've been running into a problem wherein the request will work absolutely fine on my non-Windows OSs, but I always get 401 status when running exactly the same code on Windows.
I'm currently trying to do it with RCurl, but the same thing was happening with httr when I tried that.
Also the API is unfotunately proprietary so I've had to change all the URLs, sorry.
On my non-Windows OSs I get the following behaviour:
rprompt> getURL('http://demo.someapi.net/some/url', userpwd="demo:demo", httpauth=1L, verbose=TRUE)
* Trying 195.224.16.34...
* Connected to demo.someapi.net (195.224.16.34) port 443 (#0)
* TLS 1.0 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: *.someapi.net
* Server certificate: RapidSSL SHA256 CA - G3
* Server certificate: GeoTrust Global CA
> GET /some/url HTTP/1.1
Host: demo.someapi.net
Accept: */*
< HTTP/1.1 401 Unauthorized
< Cache-Control: no-cache
< Content-Type: text/html
< Server: Microsoft-IIS/7.5
< WWW-Authenticate: Digest realm="company.product", nonce="MTEvMDcvMjAxNiAwODo0NTozMw", opaque="0000000000000000", stale=false, algorithm=MD5, qop="auth"
< X-Powered-By: ASP.NET
< Date: Mon, 11 Jul 2016 08:44:33 GMT
< Content-Length: 1293
<
* Ignoring the response-body
* Connection #0 to host demo.someapi.net left intact
* Issue another request to this URL: 'https://demo.someapi.net/some/url'
* Found bundle for host demo.someapi.net: 0x7f96c8d55af0
* Re-using existing connection! (#0) with host demo.someapi.net
* Connected to demo.someapi.net (195.224.16.34) port 443 (#0)
* Server auth using Digest with user 'demo'
> GET /some/url HTTP/1.1
Host: demo.someapi.net
Authorization: Digest username="demo", realm="company.product", nonce="MTEvMDcvMjAxNiAwODo0NTozMw", uri="/some/url", cnonce="YjRkMDQxYmM4MDFkYTMxOWZhNTViNGNmYTM5YzQyNGI=", nc=00000001, qop=auth, response="5d9643d083b2380f12d71855a98ceac3", opaque="0000000000000000", algorithm="MD5"
Accept: */*
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Length: 981
< Content-Type: application/json; charset=utf-8
< Server: Microsoft-IIS/7.5
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Date: Mon, 11 Jul 2016 08:44:33 GMT
<
* Connection #0 to host demo.someapi.net left intact
and evertything works exactly as we expect it to. On Windows however we get this:
rprompt> getURL('http://demo.someapi.net/some/url', userpwd="demo:demo", httpauth=1L, verbose=TRUE)
* Trying 195.224.16.34...
* Connected to demo.someapi.net (195.224.16.34) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: C:/Users/username/Documents/R/win-library/3.3/RCurl/etc/ca-bundle.crt
CApath: none
* SSL connection using TLSv1.0 / ECDHE-RSA-AES256-SHA
* Server certificate:
* subject: OU=GT56411961; OU=See www.rapidssl.com/resources/cps (c)15; OU=Domain Control Validated - RapidSSL(R); CN=*.someapi.net
* start date: 2015-01-26 09:31:11 GMT
* expire date: 2018-03-28 16:30:51 GMT
* subjectAltName: demo.someapi.net matched
* issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
* SSL certificate verify ok.
> GET /some/url HTTP/1.1
Host: demo.someapi.net
Accept: */*
< HTTP/1.1 401 Unauthorized
< Cache-Control: no-cache
< Content-Type: text/html
< Server: Microsoft-IIS/7.5
< WWW-Authenticate: Digest realm="company.product", nonce="MTEvMDcvMjAxNiAwODo1MjowOA", opaque="0000000000000000", stale=false, algorithm=MD5, qop="auth"
< X-Powered-By: ASP.NET
< Date: Mon, 11 Jul 2016 08:51:07 GMT
< Content-Length: 1293
<
* Ignoring the response-body
* Connection #0 to host demo.someapi.net left intact
* Issue another request to this URL: 'https://demo.someapi.net/some/url'
* Found bundle for host demo.someapi.net: 0xaa60b80
* Re-using existing connection! (#0) with host demo.someapi.net
* Connected to demo.someapi.net (195.224.16.34) port 443 (#0)
* Server auth using Digest with user 'demo'
> GET /some/url HTTP/1.1
Authorization: Digest username="demo",realm="",nonce="MTEvMDcvMjAxNiAwODo1MjowOA",uri="/some/url",cnonce="553f542ddef0e3c265e50539297bad81",nc=00000001,algorithm=MD5,response="1ec58793bb1d8142f09af112b905fa36",qop="auth",opaque="0000000000000000"
Host: demo.someapi.net
Accept: */*
< HTTP/1.1 401 Unauthorized
< Cache-Control: no-cache
< Content-Type: text/html
< Server: Microsoft-IIS/7.5
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="company.product", nonce="MTEvMDcvMjAxNiAwODo1MjowOA", opaque="0000000000000000", stale=false, algorithm=MD5, qop="auth"
< X-Powered-By: ASP.NET
< Date: Mon, 11 Jul 2016 08:51:07 GMT
< Content-Length: 1293
<
* Connection #0 to host demo.someapi.net left intact
which just returns a 401 landing page HTML.
The issue seems to be that the realm field is empty, but I have no idea how to fix this or even how to work around it.
It should be noted that both .NET's webclient and Python's requests library handles things fine, but unfortunately this has to be done in R.
I'm happy to use any R packages that are needed to help solve this.
Thanks.
For anyone else who ends up with a similar problem, you can get around it by using httr and using your own handles.
make.request <- function (url, user, pass) {
handle <- httr::handle(url)
response <- GET(url=NULL, authenticate(user, pass, type="digest"), handle=handle)
# error checking and stuff...
}
I'm not sure what the issue is with RCurl though.
Can anyone help me to figure out what am I doing wrong?
What do we have:
A website where I need to log in via cURL.
Credentials of two accounts that have to be logged in.
What did I get
I managed to log in only one account via cURL, but can't log in the second one. However I can log in with both of them via browser.
What did I try
this one works fine
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login1&tbPassword=password1&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
this one doesn't
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login2&tbPassword=password2&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
The only difference is in the logins and passwords.
I also tried to use separate cookie files for each account.
Here are the logs
Working account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 59
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 59 out of 59 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 150950
<
<!-- Here goes the html of the page with user's account -->
Problem account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 56
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 56 out of 56 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< Location: /error.aspx?aspxerrorpath=/main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
<
* Ignoring the response-body
* Connection #0 to host (nil) left intact
* Issue another request to this URL: 'http://online.tmtr.ru/error.aspx?aspxerrorpath=/main.aspx'
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /error.aspx?aspxerrorpath=/main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 405
<
<!-- Here goes the html of the page with error -->
As I mentioned before, I can successfully log in with both of the accounts via the browser.
How can I figure out why the server rejects one of the accounts via curl but doesn't via browser?
I also tried to send request via Postman extention for the Chrome browser and it works just fine too.
It is because, when you do login using the second account, it is using the cookie from the previously saved cookie. And when server sees a valid cookie, it just redirect you into a different location (or something else decided by the server).
To overcome this, just flush the cookie file before the second curl.
echo ''> ~/cookie.txt
I've contacted with the owner of that site and it turned out that there is a custom validation of the "accept language" header. I didn't send this header at all. I don't know why some requests passed validation and some not, but after I added this header to my requests all gone well :)