Open the opengraph debugger page and try it with this url: http://www.jetradar.com/?marker=12345 - I get this as a result: http://screencloud.net/v/rHlW - for some reason it tries this obviously wrong redirect url "http://www.jetradar.com/,%20http://www.jetradar.com/" though ask the server via curl - you get nothing suspicious.
$ curl -v http://www.jetradar.com/?marker=12345
* Adding handle: conn: 0x20e9b20
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x20e9b20) send_pipe: 1, recv_pipe: 0
* About to connect() to www.jetradar.com port 80 (#0)
* Trying 5.10.84.53...
* Connected to www.jetradar.com (5.10.84.53) port 80 (#0)
> GET /?marker=12345 HTTP/1.1
> User-Agent: curl/7.32.0
> Host: www.jetradar.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
* Server nginx/1.5.12 is not blacklisted
< Server: nginx/1.5.12
< Date: Fri, 07 Nov 2014 12:43:03 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Status: 301 Moved Permanently
< Location: http://www.jetradar.com/
< X-UA-Compatible: IE=Edge,chrome=1
< Set-Cookie: marker=12345; path=/; expires=Sun, 07-Dec-2014 12:43:03 GMT
< X-Request-Id: ac8d046ea03191f637cbdf8d9129a1a0
< X-Runtime: 0.011112
< X-Rack-Cache: miss
< X-Powered-By: Phusion Passenger 4.0.19
< Location: http://www.jetradar.com/
< X-Page-Speed: 1.8.31.4-4009
< Cache-Control: max-age=0, no-cache
<
* Connection #0 to host www.jetradar.com left intact
<html><head/><body>You are being redirected.</body></html>
Any ideas?
Hello from jetradar team to stackoverflow community:)
Related
I have a web app hosted at A.B.C.D:5601 and when I try curl A.B.C.D:5601 it doesn't print out anything on the screen however when I open the same link using a browser it does open the webapp. But it gets forwarded to A.B.C.D:5601/foo/bar and it opens fine.
Here is the output of curl -v
* Trying A.B.C.D:5601...
* TCP_NODELAY set
* Connected to A.B.C.D port 5601 (#0)
> GET / HTTP/1.1
> Host: A.B.C.D:5601
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< location: /spaces/enter
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 0
< Date: Tue, 08 Jun 2021 03:01:11 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
* Connection #0 to host A.B.C.D left intact
Why is curl not giving me the response back?
Curl is telling you you are being redirected to /spaces/enter.
You can tell Curl to automatically follow redirects:
curl -vL [url]
Hello I did run curl command and equal telnet command but response is different.
I try to get HTTP HEADER response code from a web server.
curl returns correct response, page indeed doesn't exist. But telnet with equal request return different response.
Please see CURL code:
$ curl -vI "https://www.abcanchorage.org/not-exist-page-con2uiu28itact"
* About to connect() to www.abcanchorage.org port 443 (#0)
* Trying 185.230.63.96...
* Connected to www.abcanchorage.org (185.230.63.96) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=abcanchorage.org
* start date: Nov 09 17:32:52 2020 GMT
* expire date: Feb 07 17:32:52 2021 GMT
* common name: abcanchorage.org
* issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> HEAD /not-exist-page-con2uiu28itact HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.abcanchorage.org
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Date: Mon, 14 Dec 2020 14:06:25 GMT
Date: Mon, 14 Dec 2020 14:06:25 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Connection: keep-alive
Connection: keep-alive
< content-language: en
content-language: en
< x-wix-request-id: 1607954785.82658627446080423994
x-wix-request-id: 1607954785.82658627446080423994
< Age: 0
Age: 0
< Server-Timing: cache;desc=miss, varnish;desc=miss, dc;desc=84
Server-Timing: cache;desc=miss, varnish;desc=miss, dc;desc=84
< X-Seen-By: r5KTLwzxoi1C+SXup0UeuQ==,sHU62EDOGnH2FBkJkG/Wx8EeXWsWdHrhlvbxtlynkVg1leqePNkwQ1H/U0dj1nMN,2d58ifebGbosy5xc+FRalthBvkD9LHf+OIny1MrcDwiq6+5o8xu+S2UUMqtKcebjUMQY1HiUPm50af1ZAsGkCw==,2UNV7KOq4oGjA5+PKsX47Gzh5saLoQp8TIRIohc0Wac=,m0j2EEknGIVUW/liY8BLLuvhI/meCohDY7RevwAJ7JU=,w4q8mm9FnmU4emOs6psVXTk8VV0IpMOWw3zRTVWAPzKTzRA6xkSHdTdM1EufzDIPWIHlCalF7YnfvOr2cMPpyw==,8OhaUUQpIrZVCQED4XmuQ40tjCf+ijnQjDvVBuKrqCUYXyC7l2oN0jRBomMsk5tniLmOBCJX9PwPq0FFNfh5cw==
X-Seen-By: r5KTLwzxoi1C+SXup0UeuQ==,sHU62EDOGnH2FBkJkG/Wx8EeXWsWdHrhlvbxtlynkVg1leqePNkwQ1H/U0dj1nMN,2d58ifebGbosy5xc+FRalthBvkD9LHf+OIny1MrcDwiq6+5o8xu+S2UUMqtKcebjUMQY1HiUPm50af1ZAsGkCw==,2UNV7KOq4oGjA5+PKsX47Gzh5saLoQp8TIRIohc0Wac=,m0j2EEknGIVUW/liY8BLLuvhI/meCohDY7RevwAJ7JU=,w4q8mm9FnmU4emOs6psVXTk8VV0IpMOWw3zRTVWAPzKTzRA6xkSHdTdM1EufzDIPWIHlCalF7YnfvOr2cMPpyw==,8OhaUUQpIrZVCQED4XmuQ40tjCf+ijnQjDvVBuKrqCUYXyC7l2oN0jRBomMsk5tniLmOBCJX9PwPq0FFNfh5cw==
< Cache-Control: no-cache
Cache-Control: no-cache
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
<
* Connection #0 to host www.abcanchorage.org left intact
Please see TELNET code:
$ telnet www.abcanchorage.org 80
Trying 35.246.6.109...
Connected to www.abcanchorage.org.
Escape character is '^]'.
HEAD /not-exist-page-con2uiu28itact HTTP/1.1
User-Agent: curl/7.29.0
Host: www.abcanchorage.org
Accept: */*
HTTP/1.1 301 Moved Permanently
Date: Mon, 14 Dec 2020 14:08:50 GMT
Content-Length: 0
Connection: keep-alive
location: https://www.abcanchorage.org/not-exist-page-con2uiu28itact
Age: 1496
Server-Timing: cache;desc=hit, varnish;desc=hit, dc;desc=euw2
X-Seen-By: sHU62EDOGnH2FBkJkG/Wx8EeXWsWdHrhlvbxtlynkVgmNySqidgeEPHXBvm3U9iS,2d58ifebGbosy5xc+FRalhEF/w8i5SZb0AgDKvjhMuBVlHGeDiRYk3btR6xZR3zaGgqFbFMYwiXnFojPwdof6LPb61OjwxlOgfM3AWuO4IQ=,2UNV7KOq4oGjA5+PKsX47LZ7Kls+1whC/C/a0aUIqJE=
Cache-Control: no-cache
Expires: -1
X-Wix-Request-Id: 1607954930.2811039255969122439
Server: Pepyaka/1.19.0
^]quit
telnet> quit
Connection closed.
I try to host some wordpress with an nginx proxy above:
This is my docker-compose.yml
proxy:
image: jwilder/nginx-proxy:0.4.0
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
# SSL Certificates
- /etc/letsencrypt/live/mysupersite.com/cert.pem:/etc/nginx/certs/site.com.crt:ro
- /etc/letsencrypt/live/mysupersite.com/privkey.pem:/etc/nginx/certs/site.com.key:ro
wordpress:
image: wordpress:4.8.1-php5.6-apache
expose:
- 80
volumes:
- /var/www/html:/var/www/html
restart: always
environment:
- WORDPRESS_DB_HOST=mysql
- WORDPRESS_DB_USER=dev
- WORDPRESS_DB_PASSWORD=passwd
- WORDPRESS_DB_NAME=db
- VIRTUAL_HOST=mysupersite.com,www.mysupersite.com
Everything seems to start well. I have 2 running containers.
But I'm not able to visit my website on https://mysupersite.com in my browser.
What part am I missing? Do I need extra configuration somewhere?
I see this error in my browser:
The page isn’t redirecting properly
output o cur -L -v https://www.mysupersite.com is the following:
* Connection #0 to host www.mysupersite.com left intact
* Issue another request to this URL: 'https://www.mysupersite.com/'
* Found bundle for host www.mysupersite.com: xx [can pipeline]
* Re-using existing connection! (#0) with host www.mysupersite.com
* Connected to www.mysupersite.com (IP) port 443 (#0)
> GET / HTTP/1.1
> Host: www.mysupersite.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.9.12
< Date: Tue, 22 Aug 2017 07:33:22 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< X-Powered-By: PHP/5.6.31
< Set-Cookie: PHPSESSID=xxx; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-Pingback: http://www.mysupersite.com/xmlrpc.php
< Location: https://www.mysupersite.com/
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host www.mysupersite.com left intact
* Issue another request to this URL: 'https://www.mysupersite.com/'
* Found bundle for host www.mysupersite.com: xxxx [can pipeline]
* Re-using existing connection! (#0) with host www.mysupersite.com
* Connected to www.mysupersite.com (IP) port 443 (#0)
> GET / HTTP/1.1
> Host: www.mysupersite.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.9.12
< Date: Tue, 22 Aug 2017 07:33:23 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< X-Powered-By: PHP/5.6.31
< Set-Cookie: PHPSESSID=xxxx; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-Pingback: http://www.mysupersite.com/xmlrpc.php
< Location: https://www.mysupersite.com/
< Strict-Transport-Security: max-age=31536000
<
* Connection #0 to host www.mysupersite.com left intact
* Maximum (50) redirects followed
curl: (47) Maximum (50) redirects followed
I've noticed a whole bunch of Referer links like t.co/oPQO7Xdz in my access_log files, but no other URL shorteners ever show up. Why?
The URL shorteners never show up because HTTP 301 Moved Permanently et al redirects in HTTP are not designed to influence the Referer HTTP Request Header (apparently, not even if the header is blank, potentially due to the fact that it'll cause inconsistency in behaviour otherwise).
However, Twitter does not issue 301 Moved Permanently redirects with its t.co service if it sees what it deems is a popular desktop or mobile User-Agent. As the redirect is thus done outside of the HTTP stack, the Referer field in the subsequent brand-new HTTP Request would then be composed to include the prior HTML page that was responsible for the redirection, causing a t.co entry to appear in access_log.
% curl -v -A"iPhone;" t.co/oPQO7Xdz
* About to connect() to t.co port 80 (#0)
* Trying 104.244.42.5...
* connected
* Connected to t.co (104.244.42.5) port 80 (#0)
> GET /oPQO7Xdz HTTP/1.1
> User-Agent: iPhone;
> Host: t.co
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: private,max-age=300
< content-length: 258
< content-security-policy: referrer always;
< content-type: text/html; charset=utf-8
< date: Thu, 07 Jul 2016 05:24:16 GMT
< expires: Thu, 07 Jul 2016 05:29:16 GMT
< server: tsa_o
< set-cookie: muc=1f43e292-e319-4818-ba81-f12d16e5b629; Expires=Tue, 19 Jun 2018 05:24:16 UTC; Domain=t.co
< x-connection-hash: 0dc5a2a6a7e83ac2d7fb207eb0cedf84
< x-response-time: 115
< x-xss-protection: 1; mode=block
<
* Connection #0 to host t.co left intact
<head><meta name="referrer" content="always"><noscript><META http-equiv="refresh" content="0;URL=http://mdoc.su/n/curl"></noscript><title>http://mdoc.su/n/curl</title></head><script>window.opener = null; location.replace("http:\/\/mdoc.su\/n\/curl")</script>* Closing connection #0
Compare this to what were to occur otherwise (and which is the only way that most other URL shorteners redirect, which would preserve whichever Referer appears when the HTTP request first hits the HTTP stack of the browser):
% curl -v t.co/oPQO7Xdz
* About to connect() to t.co port 80 (#0)
* Trying 104.244.42.69...
* connected
* Connected to t.co (104.244.42.69) port 80 (#0)
> GET /oPQO7Xdz HTTP/1.1
> User-Agent: curl/7.26.0
> Host: t.co
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< cache-control: private,max-age=300
< content-length: 0
< date: Thu, 07 Jul 2016 05:24:40 GMT
< expires: Thu, 07 Jul 2016 05:29:40 GMT
< location: http://mdoc.su/n/curl
< server: tsa_o
< set-cookie: muc=2c727b50-311f-4043-9861-9f703996a8a8; Expires=Tue, 19 Jun 2018 05:24:40 UTC; Domain=t.co
< x-connection-hash: 5583cc49ddbcefe8fac9ba392ca868fd
< x-response-time: 103
<
* Connection #0 to host t.co left intact
* Closing connection #0
Can anyone help me to figure out what am I doing wrong?
What do we have:
A website where I need to log in via cURL.
Credentials of two accounts that have to be logged in.
What did I get
I managed to log in only one account via cURL, but can't log in the second one. However I can log in with both of them via browser.
What did I try
this one works fine
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login1&tbPassword=password1&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
this one doesn't
curl --verbose --location -b ~/cookie.txt -c ~/cookie.txt
--data "tbLogin=login2&tbPassword=password2&btSubmit=Войти"
http://online.tmtr.ru/login.aspx
The only difference is in the logins and passwords.
I also tried to use separate cookie files for each account.
Here are the logs
Working account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 59
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 59 out of 59 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:49:39 GMT
< Content-Length: 150950
<
<!-- Here goes the html of the page with user's account -->
Problem account:
* About to connect() to online.tmtr.ru port 80 (#0)
* Trying 109.73.3.134...
* connected
* Connected to online.tmtr.ru (109.73.3.134) port 80 (#0)
> POST /login.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
> Content-Length: 56
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 56 out of 56 bytes
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
* Replaced cookie _some_cookie_data_
< Set-Cookie: _some_cookie_data_
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 129
<
* Ignoring the response-body
* Connection #0 to host online.tmtr.ru left intact
* Issue another request to this URL: 'http://online.tmtr.ru/main.aspx'
* Violate RFC 2616/10.3.3 and switch from POST to GET
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 302 Found
< Cache-Control: private
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< Location: /error.aspx?aspxerrorpath=/main.aspx
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
<
* Ignoring the response-body
* Connection #0 to host (nil) left intact
* Issue another request to this URL: 'http://online.tmtr.ru/error.aspx?aspxerrorpath=/main.aspx'
* Re-using existing connection! (#0) with host (nil)
* Connected to (nil) (109.73.3.134) port 80 (#0)
> GET /error.aspx?aspxerrorpath=/main.aspx HTTP/1.1
> User-Agent: curl/7.26.0
> Host: online.tmtr.ru
> Accept: */*
> Cookie: _some_cookie_data_
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 2.0.50727
< X-Powered-By: ASP.NET
< Date: Wed, 02 Dec 2015 13:45:10 GMT
< Content-Length: 405
<
<!-- Here goes the html of the page with error -->
As I mentioned before, I can successfully log in with both of the accounts via the browser.
How can I figure out why the server rejects one of the accounts via curl but doesn't via browser?
I also tried to send request via Postman extention for the Chrome browser and it works just fine too.
It is because, when you do login using the second account, it is using the cookie from the previously saved cookie. And when server sees a valid cookie, it just redirect you into a different location (or something else decided by the server).
To overcome this, just flush the cookie file before the second curl.
echo ''> ~/cookie.txt
I've contacted with the owner of that site and it turned out that there is a custom validation of the "accept language" header. I didn't send this header at all. I don't know why some requests passed validation and some not, but after I added this header to my requests all gone well :)