I have a web app hosted at A.B.C.D:5601 and when I try curl A.B.C.D:5601 it doesn't print out anything on the screen however when I open the same link using a browser it does open the webapp. But it gets forwarded to A.B.C.D:5601/foo/bar and it opens fine.
Here is the output of curl -v
* Trying A.B.C.D:5601...
* TCP_NODELAY set
* Connected to A.B.C.D port 5601 (#0)
> GET / HTTP/1.1
> Host: A.B.C.D:5601
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /spaces/enter
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 0
< Date: Tue, 08 Jun 2021 03:01:11 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
* Connection #0 to host A.B.C.D left intact
Why is curl not giving me the response back?
Curl is telling you you are being redirected to /spaces/enter.
You can tell Curl to automatically follow redirects:
curl -vL [url]
Related
I've noticed a whole bunch of Referer links like t.co/oPQO7Xdz in my access_log files, but no other URL shorteners ever show up. Why?
The URL shorteners never show up because HTTP 301 Moved Permanently et al redirects in HTTP are not designed to influence the Referer HTTP Request Header (apparently, not even if the header is blank, potentially due to the fact that it'll cause inconsistency in behaviour otherwise).
However, Twitter does not issue 301 Moved Permanently redirects with its t.co service if it sees what it deems is a popular desktop or mobile User-Agent. As the redirect is thus done outside of the HTTP stack, the Referer field in the subsequent brand-new HTTP Request would then be composed to include the prior HTML page that was responsible for the redirection, causing a t.co entry to appear in access_log.
% curl -v -A"iPhone;" t.co/oPQO7Xdz
* About to connect() to t.co port 80 (#0)
* Trying 104.244.42.5...
* connected
* Connected to t.co (104.244.42.5) port 80 (#0)
> GET /oPQO7Xdz HTTP/1.1
> User-Agent: iPhone;
> Host: t.co
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: private,max-age=300
< content-length: 258
< content-security-policy: referrer always;
< content-type: text/html; charset=utf-8
< date: Thu, 07 Jul 2016 05:24:16 GMT
< expires: Thu, 07 Jul 2016 05:29:16 GMT
< server: tsa_o
< set-cookie: muc=1f43e292-e319-4818-ba81-f12d16e5b629; Expires=Tue, 19 Jun 2018 05:24:16 UTC; Domain=t.co
< x-connection-hash: 0dc5a2a6a7e83ac2d7fb207eb0cedf84
< x-response-time: 115
< x-xss-protection: 1; mode=block
<
* Connection #0 to host t.co left intact
<head><meta name="referrer" content="always"><noscript><META http-equiv="refresh" content="0;URL=http://mdoc.su/n/curl"></noscript><title>http://mdoc.su/n/curl</title></head><script>window.opener = null; location.replace("http:\/\/mdoc.su\/n\/curl")</script>* Closing connection #0
Compare this to what were to occur otherwise (and which is the only way that most other URL shorteners redirect, which would preserve whichever Referer appears when the HTTP request first hits the HTTP stack of the browser):
% curl -v t.co/oPQO7Xdz
* About to connect() to t.co port 80 (#0)
* Trying 104.244.42.69...
* connected
* Connected to t.co (104.244.42.69) port 80 (#0)
> GET /oPQO7Xdz HTTP/1.1
> User-Agent: curl/7.26.0
> Host: t.co
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< cache-control: private,max-age=300
< content-length: 0
< date: Thu, 07 Jul 2016 05:24:40 GMT
< expires: Thu, 07 Jul 2016 05:29:40 GMT
< location: http://mdoc.su/n/curl
< server: tsa_o
< set-cookie: muc=2c727b50-311f-4043-9861-9f703996a8a8; Expires=Tue, 19 Jun 2018 05:24:40 UTC; Domain=t.co
< x-connection-hash: 5583cc49ddbcefe8fac9ba392ca868fd
< x-response-time: 103
<
* Connection #0 to host t.co left intact
* Closing connection #0
when executing, with a key that has worked in the past but that I haven't used for a few weeks, the following cURL
curl -v -k -s -H "Content-Type: application/json" https://vision.googleapis.com/v1/images:annotate?key=MyKey --data-binary #a.json
where a.json is
{"requests": [{"image": {"content": "SUkqADwmAAD////8gYEoGct1VHdGU..."}, "features": [{"type": "TEXT_DETECTION", "maxResults": 1}]}]}
returns
* Trying 173.194.205.239...
* Connected to vision.googleapis.com (173.194.205.239) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.googleapis.com
* Server certificate: Google Internet Authority G2
* Server certificate: GeoTrust Global CA
> POST /v1/images:annotate?key=MyKey HTTP/1.1
> Host: vision.googleapis.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 13454
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Vary: X-Origin
< Vary: Referer
< Content-Type: application/json; charset=UTF-8
< Date: Wed, 10 Feb 2016 18:02:12 GMT
< Server: ESF
< Cache-Control: private
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Alternate-Protocol: 443:quic,p=1
< Alt-Svc: quic=":443"; ma=604800; v="30,29,28,27,26,25"
< Accept-Ranges: none
< Vary: Origin,Accept-Encoding
< Transfer-Encoding: chunked
<
{
"error": {
"code": 400,
"message": "Request Issue Failed.",
"status": "INVALID_ARGUMENT"
}
}
* Connection #0 to host vision.googleapis.com left intact
Version 1 of the Google Cloud Vision API (beta) does not permit TIFF [1]. Here is a list of the currently supported formats:
JPEG
PNG8
PNG24
GIF
Animated GIF (first frame only)
BMP
WEBP
RAW
ICO
[1] https://cloud.google.com/vision/docs/image-best-practices#image_types
Turns out this is because I was sending base64 encoded tif images. Works fine for PNGs. Pretty sure I was told this in the docs.
Can anyone help me to figure out what am I doing wrong?
What do we have:
A website where I need to log in via cURL.
Credentials of two accounts that have to be logged in.
What did I get
I managed to log in only one account via cURL, but can't log in the second one. However I can log in with both of them via browser.
What did I try
this one works fine
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login1&tbPassword=password1&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
this one doesn't
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login2&tbPassword=password2&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
The only difference is in the logins and passwords.
I also tried to use separate cookie files for each account.
Here are the logs
Working account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 59
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 59 out of 59 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 150950
<
<!-- Here goes the html of the page with user's account -->
Problem account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 56
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 56 out of 56 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< Location: /error.aspx?aspxerrorpath=/main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
<
* Ignoring the response-body
* Connection #0 to host (nil) left intact
* Issue another request to this URL: 'http://online.tmtr.ru/error.aspx?aspxerrorpath=/main.aspx'
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /error.aspx?aspxerrorpath=/main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 405
<
<!-- Here goes the html of the page with error -->
As I mentioned before, I can successfully log in with both of the accounts via the browser.
How can I figure out why the server rejects one of the accounts via curl but doesn't via browser?
I also tried to send request via Postman extention for the Chrome browser and it works just fine too.
It is because, when you do login using the second account, it is using the cookie from the previously saved cookie. And when server sees a valid cookie, it just redirect you into a different location (or something else decided by the server).
To overcome this, just flush the cookie file before the second curl.
echo ''> ~/cookie.txt
I've contacted with the owner of that site and it turned out that there is a custom validation of the "accept language" header. I didn't send this header at all. I don't know why some requests passed validation and some not, but after I added this header to my requests all gone well :)
Open the opengraph debugger page and try it with this url: http://www.jetradar.com/?marker=12345 - I get this as a result: http://screencloud.net/v/rHlW - for some reason it tries this obviously wrong redirect url "http://www.jetradar.com/,%20http://www.jetradar.com/" though ask the server via curl - you get nothing suspicious.
$ curl -v http://www.jetradar.com/?marker=12345
* Adding handle: conn: 0x20e9b20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x20e9b20) send_pipe: 1, recv_pipe: 0
* About to connect() to www.jetradar.com port 80 (#0)
* Trying 5.10.84.53...
* Connected to www.jetradar.com (5.10.84.53) port 80 (#0)
> GET /?marker=12345 HTTP/1.1
> User-Agent: curl/7.32.0
> Host: www.jetradar.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
* Server nginx/1.5.12 is not blacklisted
< Server: nginx/1.5.12
< Date: Fri, 07 Nov 2014 12:43:03 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Status: 301 Moved Permanently
< Location: http://www.jetradar.com/
< X-UA-Compatible: IE=Edge,chrome=1
< Set-Cookie: marker=12345; path=/; expires=Sun, 07-Dec-2014 12:43:03 GMT
< X-Request-Id: ac8d046ea03191f637cbdf8d9129a1a0
< X-Runtime: 0.011112
< X-Rack-Cache: miss
< X-Powered-By: Phusion Passenger 4.0.19
< Location: http://www.jetradar.com/
< X-Page-Speed: 1.8.31.4-4009
< Cache-Control: max-age=0, no-cache
<
* Connection #0 to host www.jetradar.com left intact
<html><head/><body>You are being redirected.</body></html>
Any ideas?
Hello from jetradar team to stackoverflow community:)
Consider the following hyperlink:
<a href="http://www.cs.rutgers.edu/∼shklar/">
What HTTP/1.0 request will get submitted by the browser?
What HTTP/1.1 request will get submitted by the browser?
Will these requests change if the browser is configured to
contact an HTTP proxy? If yes, how?
While you could use tcpdump to dump the actual network traffic, curl is surely more handy to test the HTTP conversation from the command line.
An HTTP/1.0 request:
curl -v -0 http://www.cs.rutgers.edu/∼shklar/
* About to connect() to www.cs.rutgers.edu port 80 (#0)
* Trying 128.6.4.24...
* connected
* Connected to www.cs.rutgers.edu (128.6.4.24) port 80 (#0)
> GET /∼shklar/ HTTP/1.0
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.cs.rutgers.edu
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Wed, 31 Oct 2012 17:57:31 GMT
< Server: Apache/1.3.26 (Unix)
< Content-Type: text/html; charset=iso-8859-1
< Connection: close
An HTTP/1.1 request:
curl -v http://www.cs.rutgers.edu/∼shklar/
* About to connect() to www.cs.rutgers.edu port 80 (#0)
* Trying 128.6.4.24...
* connected
* Connected to www.cs.rutgers.edu (128.6.4.24) port 80 (#0)
> GET /∼shklar/ HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.cs.rutgers.edu
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Wed, 31 Oct 2012 17:59:47 GMT
< Server: Apache/1.3.26 (Unix)
< Content-Type: text/html; charset=iso-8859-1
< Transfer-Encoding: chunked
Use the -x (or --proxy) <[protocol://][user#password]proxyhost[:port]> switch to use a proxy and see the results.
More about curl here: http://curl.haxx.se/docs/manpage.html