I am trying to authenticate my local Windows 10 machine to a web service running inside a docker container. To be more specific, this container is running Hadoop services and a MIT Kerberos KDC. I have installed MIT Kerberos for Windows on my local machine and have successfully gotten a ticket hadoop/quickstart.cloudera#CLOUDERA from the KDC. When I authenticate with the same principal inside my container and call this command: curl -i --negotiate -u : "http://quickstart.cloudera:50070/webhdfs/v1/?op=GETFILESTATUS" I get a valid response. However, the same command run from my Windows machine returns this error:
Error 403 GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
Can anyone familiar with SSPI/GSSAPI give me some insight on what the problem here could be?
I have the environment variable KRB5CCNAME=<path to ccache> set correctly. This is the cURL info from my Windows command prompt:
curl 7.52.1 (x86_64-w64-mingw32) libcurl/7.52.1 WinSSL zlib/1.2.8 WinIDN libssh2/1.7.0_DEV
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz
Let me know if you need any more information about my environment setup.
Update:
Here is the full HTTP response from my Windows machine when I run the command with the -v flag added for verbosity.
* timeout on name lookup is not supported
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to quickstart.cloudera (127.0.0.1) port 50070 (#0)
> GET /webhdfs/v1/?op=GETFILESTATUS HTTP/1.1
> Host: quickstart.cloudera:50070
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 401 Authentication required
HTTP/1.1 401 Authentication required
< Cache-Control: must-revalidate,no-cache,no-store
Cache-Control: must-revalidate,no-cache,no-store
< Date: Fri, 10 Feb 2017 19:41:27 GMT
Date: Fri, 10 Feb 2017 19:41:27 GMT
< Pragma: no-cache
Pragma: no-cache
< Date: Fri, 10 Feb 2017 19:41:27 GMT
Date: Fri, 10 Feb 2017 19:41:27 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
< WWW-Authenticate: Negotiate
WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 1404
Content-Length: 1404
< Server: Jetty(6.1.26.cloudera.4)
Server: Jetty(6.1.26.cloudera.4)
<
* Ignoring the response-body
* Curl_http_done: called premature == 0
* Connection #0 to host quickstart.cloudera left intact
* Issue another request to this URL: 'http://quickstart.cloudera:50070/webhdfs/v1/?op=GETFILESTATUS'
* Found bundle for host quickstart.cloudera: 0x817220 [can pipeline]
* Re-using existing connection! (#0) with host quickstart.cloudera
* Connected to quickstart.cloudera (127.0.0.1) port 50070 (#0)
* Server auth using Negotiate with user ''
> GET /webhdfs/v1/?op=GETFILESTATUS HTTP/1.1
> Host: quickstart.cloudera:50070
> Authorization: Negotiate TlRMTVNTUAABAAAAt4II4gAAAAAAAAAAAAAAAAAAAAAKAFopAAAADw==
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 403 GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
HTTP/1.1 403 GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)
< Cache-Control: must-revalidate,no-cache,no-store
Cache-Control: must-revalidate,no-cache,no-store
< Date: Fri, 10 Feb 2017 19:41:27 GMT
Date: Fri, 10 Feb 2017 19:41:27 GMT
< Pragma: no-cache
Pragma: no-cache
< Date: Fri, 10 Feb 2017 19:41:27 GMT
Date: Fri, 10 Feb 2017 19:41:27 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 1546
Content-Length: 1546
< Server: Jetty(6.1.26.cloudera.4)
Server: Jetty(6.1.26.cloudera.4)
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/. Reason:
<pre> GSSException: Defective token detected (Mechanism level: GSSHeader did not find the right tag)</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
</body>
</html>
* Curl_http_done: called premature == 0
* Closing connection 0
In contrast, this is the response from running the same command inside my container:
* About to connect() to quickstart.cloudera port 50070 (#0)
* Trying 172.18.0.2... connected
* Connected to quickstart.cloudera (172.18.0.2) port 50070 (#0)
> GET /webhdfs/v1/?op=GETFILESTATUS HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: quickstart.cloudera:50070
> Accept: */*
>
< HTTP/1.1 401 Authentication required
HTTP/1.1 401 Authentication required
< Cache-Control: must-revalidate,no-cache,no-store
Cache-Control: must-revalidate,no-cache,no-store
< Date: Fri, 10 Feb 2017 21:15:39 GMT
Date: Fri, 10 Feb 2017 21:15:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Date: Fri, 10 Feb 2017 21:15:39 GMT
Date: Fri, 10 Feb 2017 21:15:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
< WWW-Authenticate: Negotiate
WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 1404
Content-Length: 1404
< Server: Jetty(6.1.26.cloudera.4)
Server: Jetty(6.1.26.cloudera.4)
<
* Ignoring the response-body
* Connection #0 to host quickstart.cloudera left intact
* Issue another request to this URL: 'http://quickstart.cloudera:50070/webhdfs/v1/?op=GETFILESTATUS'
* Re-using existing connection! (#0) with host quickstart.cloudera
* Connected to quickstart.cloudera (172.18.0.2) port 50070 (#0)
* Server auth using GSS-Negotiate with user ''
> GET /webhdfs/v1/?op=GETFILESTATUS HTTP/1.1
> Authorization: Negotiate YIICkQYJKoZIhvcSAQICAQBuggKAMIICfKADAgEFoQMCAQ6iBwMFAAAAAACjggGBYYIBfTCCAXmgAwIBBaEKGwhDTE9VREVSQaImMCSgAwIBA6EdMBsbBEhUVFAbE3F1aWNrc3RhcnQuY2xvdWRlcmGjggE8MIIBOKADAgESoQMCAQKiggEqBIIBJnWbXp9WlAVk0nGIqD7T25On1+OCPzXDX/aoH01FTjJbEwrYj4cMML7Tf6jaKDIANEh57kTJOvPknL3CWHI1c3LeNpt1Ir8H2M3Zvk91HpbWXzv5WJTeUOK9L6zTaKFEs/dKQgD7VzmHKDJtMyVKQWLLVU8JuyKAV6iM4FvxfZ+WDF8QCk7pxwjgX1OT7jv9jR28MPpsIweqUYYnneJxVTsxgmsHdOvj5wpMGy9RA9R8jtR+Wh3l5r3a3zcUTmGwAqY+NXhBkviSTw+DgitnipYh5tXBRhNqGfk86qWAdodGgL+SdkwwGsq91PyYQiMCLXjWx90aBOEFeZLDyqBaXlMIZ3TT3urUQEuB206+8KNw1n6N6+u+ZY4QT7NJyVZqHbnOR5V5maSB4TCB3qADAgESooHWBIHTyaBY3PormkycaX9uf/lLe6ISYnItZikJqslAGpJVnla2HXYvhFjqn5yr8td1pw3zzdnDEZx3a9EylIrRQD5IoIvHCzd0mlJhHFj4xxISM5hxlMiL8DewMjGsVcDveqpHw1SyxIsrEPOhe62HuRXS7c1Z9kYkP6KldzyAJOttOVYCuL36hOxwEFqJtbWk1/f9gfTdzmxQmEASM3/wsj2Q/WYCZY/hazDIz6dmHsyla/F6NXGK0BwRnHHBCqSHe7GdWmBDNjHiuo6R0/YvqTl5Uvf0Rw==
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: quickstart.cloudera:50070
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Fri, 10 Feb 2017 21:15:39 GMT
Expires: Fri, 10 Feb 2017 21:15:39 GMT
< Date: Fri, 10 Feb 2017 21:15:39 GMT
Date: Fri, 10 Feb 2017 21:15:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Expires: Fri, 10 Feb 2017 21:15:39 GMT
Expires: Fri, 10 Feb 2017 21:15:39 GMT
< Date: Fri, 10 Feb 2017 21:15:39 GMT
Date: Fri, 10 Feb 2017 21:15:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: application/json
Content-Type: application/json
< Set-Cookie: hadoop.auth="u=hadoop&p=hadoop/quickstart.cloudera#CLOUDERA&t=kerberos&e=1486797339425&s=BqBHGJ+/FxxeSR0ayBXHOrfPkwU="; Path=/; HttpOnly
Set-Cookie: hadoop.auth="u=hadoop&p=hadoop/quickstart.cloudera#CLOUDERA&t=kerberos&e=1486797339425&s=BqBHGJ+/FxxeSR0ayBXHOrfPkwU="; Path=/; HttpOnly
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Server: Jetty(6.1.26.cloudera.4)
Server: Jetty(6.1.26.cloudera.4)
<
* Connection #0 to host quickstart.cloudera left intact
* Closing connection #0
{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":5,"fileId":16385,"group":"supergroup","length":0,"modificationTime":1459909590753,"owner":"hdfs","pathSuffix":"","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}
The answer is very easy in this case. Curl on Windows is compiled with SSPI. When SSPI is requested to perform SPNEGO, it tries Kerberos, which fails here. Likely "Server not found in database" (use Wireshark) and then falls back to NTLM. It sends a raw NTLM token to your JGSS-backed server which rejects the token because
This is not a SPNEGO wrapped token, but a raw NTLM token
Java does not support NTLM
Here is sample code how to intercept this and respond with a meaningful message. Raise an issue with Hadoop.
Moreover, your Curl version 7.19.7 on Linux is extremely old and unsecure, you should upgrade immediately AND the SPNEGO Authenticator on Jetty is broken because it does not respond with a context-completion token. To sum up, the entire authentication should not be trusted because it is faulty. See RFC 7546.
Related
I am getting an issue uploading to shinyapps.io. I had issues with the firewall before so I am wondering if maybe there are leftover issues there but I can't find anything that would indicate that in the error message.
Preparing to deploy application...DONE
Uploading bundle for application: 4222252...Error in force(code) : Could not upload file.
Calls: <Anonymous> -> withStatus -> force
Execution halted
The application is a very simple shinyapp that I used to test. There's really only a dummy data.table and a linechart created with the billboarder package. Nothing crazy going on there.
I tried to log it and this is what I got:
* Connection 2 seems to be dead!
* Closing connection 2
* schannel: shutting down SSL/TLS connection with api.shinyapps.io port 443
* Connection 3 seems to be dead!
* Closing connection 3
* schannel: shutting down SSL/TLS connection with api.shinyapps.io port 443
* Found bundle for host api.shinyapps.io: 0x1fd5bd908e0 [can pipeline]
* Trying 18.204.73.112...
* TCP_NODELAY set
* Connected to api.shinyapps.io (18.204.73.112) port 443 (#5)
> GET /v1/applications/?filter=account_id:932539&count=100&offset=0 HTTP/1.1
Host: api.shinyapps.io
User-Agent: rsconnect/0.8.17
Accept: */*
Accept-Encoding: deflate, gzip
Date: Thu, 03 Jun 2021 08:06:19 GMT
X-Auth-Token: EB5EF5C5A9A0A1AD8021108F4973EA4F
X-Auth-Signature: NDA5YzIwY2E3YTMxMTQ1Y2U4YzYwZjdhMTA2ODM2NzJmOWE1MDExNTI3NmFiODRmNzFkMjUzZmJlZDA3YWRkZg==; version=1
X-Content-Checksum: d41d8cd98f00b204e9800998ecf8427e
< HTTP/1.1 200 OK
< Date: Thu, 03 Jun 2021 08:06:20 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 5879
< Connection: keep-alive
< Etag: "36941c41b5fad4ebdec0fb9ed5ef1154e9a52e79"
< Server: TornadoServer/4.5.3
<
* Connection #5 to host api.shinyapps.io left intact
* Found bundle for host api.shinyapps.io: 0x1fd5bd908e0 [can pipeline]
* Hostname api.shinyapps.io was found in DNS cache
* Trying 18.204.73.112...
* TCP_NODELAY set
* Connected to api.shinyapps.io (18.204.73.112) port 443 (#6)
> GET /v1/applications/?filter=account_id:932539&count=100&offset=0 HTTP/1.1
Host: api.shinyapps.io
User-Agent: rsconnect/0.8.17
Accept: */*
Accept-Encoding: deflate, gzip
Date: Thu, 03 Jun 2021 08:06:31 GMT
X-Auth-Token: EB5EF5C5A9A0A1AD8021108F4973EA4F
X-Auth-Signature: MjdiYWUzZWYzODIzNmM5NzMxMGZhOGM5MTQ4Y2VjMTFlYjdlODFkMWRhNDU4ZGFkNjNmMGQ5YmVlMTllZGQ5Ng==; version=1
X-Content-Checksum: d41d8cd98f00b204e9800998ecf8427e
< HTTP/1.1 200 OK
< Date: Thu, 03 Jun 2021 08:06:32 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 5879
< Connection: keep-alive
< Etag: "36941c41b5fad4ebdec0fb9ed5ef1154e9a52e79"
< Server: TornadoServer/4.5.3
<
* Connection #6 to host api.shinyapps.io left intact
* Connection 4 seems to be dead!
* Closing connection 4
* schannel: shutting down SSL/TLS connection with api.shinyapps.io port 443
* Found bundle for host api.shinyapps.io: 0x1fd5bd908e0 [can pipeline]
* Trying 3.231.196.102...
* TCP_NODELAY set
* Connected to api.shinyapps.io (3.231.196.102) port 443 (#7)
> GET /v1/applications/?filter=account_id:932539&count=100&offset=0 HTTP/1.1
Host: api.shinyapps.io
User-Agent: rsconnect/0.8.17
Accept: */*
Accept-Encoding: deflate, gzip
Date: Thu, 03 Jun 2021 08:07:34 GMT
X-Auth-Token: EB5EF5C5A9A0A1AD8021108F4973EA4F
X-Auth-Signature: YjMzMzNlMDQ0MTEwZmZmYjQxOTMxMzA1N2JkMzc4ZmM0MTA5ZGE4ZmE5NGFmYmY5MGE0ZWZmNjAzOTFmZjc5Yg==; version=1
X-Content-Checksum: d41d8cd98f00b204e9800998ecf8427e
* schannel: failed to decrypt data, need more data
< HTTP/1.1 200 OK
< Date: Thu, 03 Jun 2021 08:07:36 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 5879
< Connection: keep-alive
< Etag: "36941c41b5fad4ebdec0fb9ed5ef1154e9a52e79"
< Server: TornadoServer/4.5.3
<
* Connection #7 to host api.shinyapps.io left intact
* Found bundle for host api.shinyapps.io: 0x1fd5bd908e0 [can pipeline]
* Hostname in DNS cache was stale, zapped
* Trying 3.231.196.102...
* TCP_NODELAY set
* Connected to api.shinyapps.io (3.231.196.102) port 443 (#8)
> GET /v1/applications/?filter=account_id:932539&count=100&offset=0 HTTP/1.1
Host: api.shinyapps.io
User-Agent: rsconnect/0.8.17
Accept: */*
Accept-Encoding: deflate, gzip
Date: Thu, 03 Jun 2021 08:09:51 GMT
X-Auth-Token: EB5EF5C5A9A0A1AD8021108F4973EA4F
X-Auth-Signature: MDhlNzE0MDFhMmJhNjU3NTA2Y2UyOTk3ZWY2OWQ1ZDgyNjMyMWRhYjcyOTNhOTMyMDFkMGY4YzIwZmJiZWM1MQ==; version=1
X-Content-Checksum: d41d8cd98f00b204e9800998ecf8427e
< HTTP/1.1 200 OK
< Date: Thu, 03 Jun 2021 08:09:51 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 5879
< Connection: keep-alive
< Etag: "36941c41b5fad4ebdec0fb9ed5ef1154e9a
GET /v1/applications/?filter=account_id:932539&count=100&offset=0 870ms
If anyone has any idea what's happening here I would love to hear it!
Hello I did run curl command and equal telnet command but response is different.
I try to get HTTP HEADER response code from a web server.
curl returns correct response, page indeed doesn't exist. But telnet with equal request return different response.
Please see CURL code:
$ curl -vI "https://www.abcanchorage.org/not-exist-page-con2uiu28itact"
* About to connect() to www.abcanchorage.org port 443 (#0)
* Trying 185.230.63.96...
* Connected to www.abcanchorage.org (185.230.63.96) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=abcanchorage.org
* start date: Nov 09 17:32:52 2020 GMT
* expire date: Feb 07 17:32:52 2021 GMT
* common name: abcanchorage.org
* issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> HEAD /not-exist-page-con2uiu28itact HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.abcanchorage.org
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Date: Mon, 14 Dec 2020 14:06:25 GMT
Date: Mon, 14 Dec 2020 14:06:25 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Connection: keep-alive
Connection: keep-alive
< content-language: en
content-language: en
< x-wix-request-id: 1607954785.82658627446080423994
x-wix-request-id: 1607954785.82658627446080423994
< Age: 0
Age: 0
< Server-Timing: cache;desc=miss, varnish;desc=miss, dc;desc=84
Server-Timing: cache;desc=miss, varnish;desc=miss, dc;desc=84
< X-Seen-By: r5KTLwzxoi1C+SXup0UeuQ==,sHU62EDOGnH2FBkJkG/Wx8EeXWsWdHrhlvbxtlynkVg1leqePNkwQ1H/U0dj1nMN,2d58ifebGbosy5xc+FRalthBvkD9LHf+OIny1MrcDwiq6+5o8xu+S2UUMqtKcebjUMQY1HiUPm50af1ZAsGkCw==,2UNV7KOq4oGjA5+PKsX47Gzh5saLoQp8TIRIohc0Wac=,m0j2EEknGIVUW/liY8BLLuvhI/meCohDY7RevwAJ7JU=,w4q8mm9FnmU4emOs6psVXTk8VV0IpMOWw3zRTVWAPzKTzRA6xkSHdTdM1EufzDIPWIHlCalF7YnfvOr2cMPpyw==,8OhaUUQpIrZVCQED4XmuQ40tjCf+ijnQjDvVBuKrqCUYXyC7l2oN0jRBomMsk5tniLmOBCJX9PwPq0FFNfh5cw==
X-Seen-By: r5KTLwzxoi1C+SXup0UeuQ==,sHU62EDOGnH2FBkJkG/Wx8EeXWsWdHrhlvbxtlynkVg1leqePNkwQ1H/U0dj1nMN,2d58ifebGbosy5xc+FRalthBvkD9LHf+OIny1MrcDwiq6+5o8xu+S2UUMqtKcebjUMQY1HiUPm50af1ZAsGkCw==,2UNV7KOq4oGjA5+PKsX47Gzh5saLoQp8TIRIohc0Wac=,m0j2EEknGIVUW/liY8BLLuvhI/meCohDY7RevwAJ7JU=,w4q8mm9FnmU4emOs6psVXTk8VV0IpMOWw3zRTVWAPzKTzRA6xkSHdTdM1EufzDIPWIHlCalF7YnfvOr2cMPpyw==,8OhaUUQpIrZVCQED4XmuQ40tjCf+ijnQjDvVBuKrqCUYXyC7l2oN0jRBomMsk5tniLmOBCJX9PwPq0FFNfh5cw==
< Cache-Control: no-cache
Cache-Control: no-cache
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
<
* Connection #0 to host www.abcanchorage.org left intact
Please see TELNET code:
$ telnet www.abcanchorage.org 80
Trying 35.246.6.109...
Connected to www.abcanchorage.org.
Escape character is '^]'.
HEAD /not-exist-page-con2uiu28itact HTTP/1.1
User-Agent: curl/7.29.0
Host: www.abcanchorage.org
Accept: */*
HTTP/1.1 301 Moved Permanently
Date: Mon, 14 Dec 2020 14:08:50 GMT
Content-Length: 0
Connection: keep-alive
location: https://www.abcanchorage.org/not-exist-page-con2uiu28itact
Age: 1496
Server-Timing: cache;desc=hit, varnish;desc=hit, dc;desc=euw2
X-Seen-By: sHU62EDOGnH2FBkJkG/Wx8EeXWsWdHrhlvbxtlynkVgmNySqidgeEPHXBvm3U9iS,2d58ifebGbosy5xc+FRalhEF/w8i5SZb0AgDKvjhMuBVlHGeDiRYk3btR6xZR3zaGgqFbFMYwiXnFojPwdof6LPb61OjwxlOgfM3AWuO4IQ=,2UNV7KOq4oGjA5+PKsX47LZ7Kls+1whC/C/a0aUIqJE=
Cache-Control: no-cache
Expires: -1
X-Wix-Request-Id: 1607954930.2811039255969122439
Server: Pepyaka/1.19.0
^]quit
telnet> quit
Connection closed.
I've noticed a whole bunch of Referer links like t.co/oPQO7Xdz in my access_log files, but no other URL shorteners ever show up. Why?
The URL shorteners never show up because HTTP 301 Moved Permanently et al redirects in HTTP are not designed to influence the Referer HTTP Request Header (apparently, not even if the header is blank, potentially due to the fact that it'll cause inconsistency in behaviour otherwise).
However, Twitter does not issue 301 Moved Permanently redirects with its t.co service if it sees what it deems is a popular desktop or mobile User-Agent. As the redirect is thus done outside of the HTTP stack, the Referer field in the subsequent brand-new HTTP Request would then be composed to include the prior HTML page that was responsible for the redirection, causing a t.co entry to appear in access_log.
% curl -v -A"iPhone;" t.co/oPQO7Xdz
* About to connect() to t.co port 80 (#0)
* Trying 104.244.42.5...
* connected
* Connected to t.co (104.244.42.5) port 80 (#0)
> GET /oPQO7Xdz HTTP/1.1
> User-Agent: iPhone;
> Host: t.co
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: private,max-age=300
< content-length: 258
< content-security-policy: referrer always;
< content-type: text/html; charset=utf-8
< date: Thu, 07 Jul 2016 05:24:16 GMT
< expires: Thu, 07 Jul 2016 05:29:16 GMT
< server: tsa_o
< set-cookie: muc=1f43e292-e319-4818-ba81-f12d16e5b629; Expires=Tue, 19 Jun 2018 05:24:16 UTC; Domain=t.co
< x-connection-hash: 0dc5a2a6a7e83ac2d7fb207eb0cedf84
< x-response-time: 115
< x-xss-protection: 1; mode=block
<
* Connection #0 to host t.co left intact
<head><meta name="referrer" content="always"><noscript><META http-equiv="refresh" content="0;URL=http://mdoc.su/n/curl"></noscript><title>http://mdoc.su/n/curl</title></head><script>window.opener = null; location.replace("http:\/\/mdoc.su\/n\/curl")</script>* Closing connection #0
Compare this to what were to occur otherwise (and which is the only way that most other URL shorteners redirect, which would preserve whichever Referer appears when the HTTP request first hits the HTTP stack of the browser):
% curl -v t.co/oPQO7Xdz
* About to connect() to t.co port 80 (#0)
* Trying 104.244.42.69...
* connected
* Connected to t.co (104.244.42.69) port 80 (#0)
> GET /oPQO7Xdz HTTP/1.1
> User-Agent: curl/7.26.0
> Host: t.co
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< cache-control: private,max-age=300
< content-length: 0
< date: Thu, 07 Jul 2016 05:24:40 GMT
< expires: Thu, 07 Jul 2016 05:29:40 GMT
< location: http://mdoc.su/n/curl
< server: tsa_o
< set-cookie: muc=2c727b50-311f-4043-9861-9f703996a8a8; Expires=Tue, 19 Jun 2018 05:24:40 UTC; Domain=t.co
< x-connection-hash: 5583cc49ddbcefe8fac9ba392ca868fd
< x-response-time: 103
<
* Connection #0 to host t.co left intact
* Closing connection #0
I will be sending simple data to a web server, most probably just as a GET request which will be:
Latitude
Longitude
Speed
Height
deviceId
I could do this via SSH I suppose if it uses less data - but I cant see this being much.
So if i send this in myurl.com/parsedata.php?id=1&.... etc, how much data will it send require per request? the page will not load anything and it will do it over cURL or a similar protocol.
So, typically how much data will be in this per single request?
Thanks.
You can simply run curl from the command line with -v option for verbose output and see the data which goes to the server and back. Here is what I get when I do a curl on this question's URL:
curl http://stackoverflow.com/questions/31789124/how-much-data-is-in-one-curl-request -v
* Hostname was NOT found in DNS cache
* Trying 104.16.103.85...
* Connected to stackoverflow.com (104.16.103.85) port 80 (#0)
> GET /questions/31789124/how-much-data-is-in-one-curl-request HTTP/1.1
> User-Agent: curl/7.35.0
> Host: stackoverflow.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 04 Aug 2015 08:51:53 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=d89a0800ec97e30cad4dedad83d88e8dd1438678312; expires=Wed, 03-Aug-16 08:51:52 GMT; path=/; domain=.stackoverflow.com; HttpOnly
< Cache-Control: public, no-cache="Set-Cookie", max-age=60
< Cf-Railgun: 5107f6033d 18.60 0.037768 0030 4701
< Expires: Tue, 04 Aug 2015 08:52:53 GMT
< Last-Modified: Tue, 04 Aug 2015 08:51:53 GMT
< Vary: *
< X-Frame-Options: SAMEORIGIN
< X-Request-Guid: 3f5aca8a-2715-4fcf-b68b-f71f85a620e0
< Set-Cookie: prov=fcd7db27-9d37-4235-afb6-5c3d76dc03f5; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
* Server cloudflare-nginx is not blacklisted
< Server: cloudflare-nginx
< CF-RAY: 210905e0082d03b8-SIN
<
<!DOCTYPE html>
<html itemscope itemtype="http://schema.org/QAPage">
... html content follows here ...
All the > lines are what is sent to the server as a request (as plain text over a socket connection) and the < lines are what is received back.
I have the following get:
#Get
public String represent(Variant variant) throws ResourceException
{
String text = "returntext";
text+="\r\n";
return text;
}
The response from invoking this service:
CFG - HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Type: text/plain;charset=UTF-8
Date: Mon, 29 Jul 2013 19:59:37 GMT
Server: Restlet-Framework/2.0.9
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Length: 118
Connection: keep-alive
How do I change the connection header value to close ?
I think this maybe a restlet bug.
Whether the server closes the connection or not, depends on whether the client request asks for the connection to close or not.
Here is a sample Server code:
import org.restlet.data.Form;
import org.restlet.data.MediaType;
import org.restlet.data.Parameter;
import org.restlet.resource.Get;
import org.restlet.resource.ServerResource;
import org.restlet.util.Series;
public class TestRestlet extends ServerResource {
#Get
public String getImpl(){
return "Sample Response Text\r\n";
}
}
And here is what i got on linux commmand line (using only telnet):
[Please note that the last line in request-header in each request is followed by 2 newlines] [To avoid any confusion, some of the requests do not contain request-body.]
[root#mylinuxserver]# telnet 172.16.101.34 6060
Trying 172.16.101.34...
Connected to win7comp01 (172.16.101.34).
Escape character is '^]'.
GET /TestRestlet HTTP/1.1
Host: 172.16.101.34:6060
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=C2E77F4D0437E525A0FC66498EF09E8B; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:25:44 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Sample Response Text
GET /TestRestlet HTTP/1.1
Host: 172.16.101.34:6060
Connection: Keep-Alive
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=1873DE26443F5DF62379B895AEA0F004; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:25:48 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Sample Response Text
GET /TestRestlet HTTP/1.1
Host: 172.16.101.34:6060
Connection: close
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=43EC7C9AACC6C0CEF6FAC8F608B1D79C; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:25:57 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Connection: close
Sample Response Text
Connection closed by foreign host.
[root#mylinuxserver]# telnet 172.16.101.34 6060
Trying 172.16.101.34...
Connected to win7comp01 (172.16.101.34).
Escape character is '^]'.
GET /TestRestlet HTTP/1.0
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=2C879A91F2501DD9D3B39EF50C3F46CA; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:26:09 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Connection: close
Sample Response Text
Connection closed by foreign host.
[root#mylinuxserver]# telnet 172.16.101.34 6060
Trying 172.16.101.34...
Connected to win7comp01 (172.16.101.34).
Escape character is '^]'.
GET /TestRestlet
Sample Response Text
Connection closed by foreign host.
[root#mylinuxserver]#
In the above examples, several types of HTTP connections are made.
The response to the 1st request:
GET /TestRestlet HTTP/1.1
Host: 172.16.101.34:6060
[Note: the line Host: 172.16.101.34:6060 is followed by 2 \r\n: \r\n\r\n]
is:
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=C2E77F4D0437E525A0FC66498EF09E8B; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:25:44 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Sample Response Text
The connection is not closed yet, and we send another request:
GET /TestRestlet HTTP/1.1
Host: 172.16.101.34:6060
Connection: Keep-Alive
That gets the response:
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=1873DE26443F5DF62379B895AEA0F004; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:25:48 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Still the connection is not closed.
However on the 3rd request:
GET /TestRestlet HTTP/1.1
Host: 172.16.101.34:6060
Connection: close
The connection is closed, because the request contains Connection: close header.
You can see the telnet exits after displaying a message: Connection closed by foreign host.
There are 2 more sample request-response in the above given examples:
1.An HTTP 1.0 request:
GET /TestRestlet HTTP/1.0
With response:
HTTP/1.1 200 OK
Set-Cookie: JSESSIONID=2C879A91F2501DD9D3B39EF50C3F46CA; Path=/hotelSoft
Date: Wed, 31 Jul 2013 08:26:09 GMT
Accept-Ranges: bytes
Server: Restlet-Framework/2.0.15
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
Content-Type: application/json;charset=UTF-8
Content-Length: 22
Connection: close
Sample Response Text
And the telnet exits after displaying: Connection closed by foreign host.
2.A request without HTTP version mentioned:
GET /TestRestlet
And response is: (Without headers)
Sample Response Text
And the telnet exits with a message: Connection closed by foreign host.
Conclusion:
Whatever is your client / client-program , make it send an HTTP-1.0 request , or a HTTP-1.1 request with Connection: close header.
In Java, you achieve this by:
import java.net.HttpURLConnection;
import java.net.URL;
.
.
.
HttpURLConnection httpURLConnection = (HttpURLConnection) new URL("http://....").openConnection();
httpURLConnection.setRequestProperty("Connection","close");
// rest of the code here....
Also check if a statement like this:
httpURLConnection.disconnect();
can help you disconnect the connection.