postman http get request with authentification fails with 401 reply - http

I have an IOT device (PV inverter). As with many of these there is an official API mostly for data retrival and some settings. However, given the right credentials (admin account) you can configure significantly more in the webinterface.
I would like to be able to do this from my home automation server directly (via node red finally). So I tried to have a look at the communication between the browser and the inverter via Wireshark.
I found some GET an POST calls, and wanted to start with the replication of a GET call via node-red or Postman first.
However, no matter what I tried so far I only get 401 replies.
Seemingly, I'm not able to setup the message with proper authentificaion. In wireshark the Postman and the original GET request are very close.
Here is the original (followed by a "200 OK" response):
Internet Protocol Version 4, Src: 192.168.0.64, Dst: 192.168.0.5
Transmission Control Protocol, Src Port: 56183, Dst Port: 80, Seq: 1, Ack: 1, Len: 601
Source Port: 56183
Destination Port: 80
[Stream index: 1]
[Conversation completeness: Incomplete (28)]
[TCP Segment Len: 601]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 2463465501
[Next Sequence Number: 602 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 1894190984
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
[TCP Flags: ·······AP···]
Window: 512
[Calculated window size: 512]
[Window size scaling factor: -1 (unknown)]
Checksum: 0x8409 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (601 bytes)
Hypertext Transfer Protocol
GET /commands/StandbyState HTTP/1.1\r\n
Host: 192.168.0.5\r\n
Connection: keep-alive\r\n
Accept: application/json, text/plain, /\r\n
Authorization: Digest username="technician", realm="Webinterface area",
nonce="63af2777:24350f8b8a09fb90b82b6ac480d325cc", uri="/commands/StandbyState", response="a09ac5fe504563040d0ff8acfd68653e", qop=auth, nc=00000022, cnonce="NaN"\r\n
username="technician"
realm="Webinterface area"
nonce="63af2777:24350f8b8a09fb90b82b6ac480d325cc"
uri="/commands/StandbyState"
response="a09ac5fe504563040d0ff8acfd68653e"
qop=auth
nc=00000022
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/108.0.0.0 Safari/537.36\r\n
Referer: http://192.168.0.5/\r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\r\n
\r\n
[Full request URI: http://192.168.0.5/commands/StandbyState]
and here is the captured Postman packet
Internet Protocol Version 4, Src: 192.168.0.64, Dst: 192.168.0.5
Transmission Control Protocol, Src Port: 43404, Dst Port: 80, Seq: 1, Ack: 1, Len: 471
Source Port: 43404
Destination Port: 80
[Stream index: 5]
[Conversation completeness: Incomplete, DATA (15)]
[TCP Segment Len: 471]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 990556558
[Next Sequence Number: 472 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 2175082347
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
[TCP Flags: ·······AP···]
Window: 513
[Calculated window size: 131328]
[Window size scaling factor: 256]
Checksum: 0x8387 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (471 bytes)
Hypertext Transfer Protocol
GET /commands/StandbyState HTTP/1.1\r\n
[truncated]Authorization: Digest username="technician", realm="Webinterface area", nonce="63af2777:24350f8b8a09fb90b82b6ac480d325cc", uri="/commands/StandbyState", algorithm="MD5", qop=auth, nc=00000022, cnonce="NaN", response="a09ac5fe5
username="technician"
realm="Webinterface area"
nonce="63af2777:24350f8b8a09fb90b82b6ac480d325cc"
uri="/commands/StandbyState"
algorithm="MD5"
qop=auth
nc=00000022
cnonce="NaN"
User-Agent: PostmanRuntime/7.30.0\r\n
Accept: /\r\n
Postman-Token: e5d8ee2c-37fb-49ae-aa37-1cf05bfe8608\r\n
Host: 192.168.0.5\r\n
Accept-Encoding: gzip, deflate, br\r\n
Connection: keep-alive\r\n
\r\n
[Full request URI: http://192.168.0.5/commands/StandbyState]
[HTTP request 1/1]
Why does this not work?

Related

Strange firefox bug triggers on reload

I noticed a bug in an old version of firefox, that was shipped with my os.
These were the sympthoms:
Guile web server failed to serve the request when data was reposted.
I came up with a minimal example to show the problem.
Steps to reproduce:
start the server script
load localhost:8080
select the test.csv file for upload, and upload it
hit the refresh button in the browser
answer yes to resend post data dialog.
test.scm:
(use-modules (web server)
(rnrs bytevectors))
(define (handler request body)
(if body
(display (utf8->string body)))
(values '((content-type . (text/html)))
(string-append "<html><body>"
"<form action=\"do\" method=\"POST\" enctype=\"multipart/form-dat\
a\">"
"<input type=\"file\" name=\"x\">"
"<input type=\"submit\">")))
(run-server handler)
test.csv:
a,b
Expected result: no error displayed on the console.
Actual result:
-----------------------------18912432064747206221264673165
Content-Disposition: form-data; name="x"; filename="test.csv"
Content-Type: text/csv
In ice-9/boot-9.scm:
841:4 4 (with-throw-handler _ _ _)
In web/server/http.scm:
127:28 3 (_)
In web/request.scm:
205:31 2 (read-request #<closed: file 5559bbcb82a0> _)
In web/http.scm:
1141:6 1 (read-request-line _)
In ice-9/boot-9.scm:
752:25 0 (dispatch-exception _ _ _)
Bad request: Bad Request-Line: "a,b"
What am I doing wrong here?
Some additional information:
on a whireshark capture it turns out, that the following is sent on resend:
POST /do HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: hu-HU,hu;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:8080/do
Content-Type: multipart/form-data; boundary=---------------------------121791188820701943592108452984
Content-Length: 150
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
-----------------------------121791188820701943592108452984
Content-Disposition: form-data; name="x"; filename="test.csv"
Content-Type: text/csv
POST /do HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: hu-HU,hu;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:8080/do
Content-Type: multipart/form-data; boundary=---------------------------121791188820701943592108452984
Content-Length: 150
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
-----------------------------121791188820701943592108452984
Content-Disposition: form-data; name="x"; filename="test.csv"
Content-Type: text/csv
a,b
-----------------------------121791188820701943592108452984--
I will check the http spec if it has anything to say about this. The first http request is partial, followed by a well formed request.
UPDATE:
It turned out that guile webserver threw the error completely legitimately.
Answering my own question:
This is actually this firfox bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1434553
Fixed in firefox62.

Content length limitation in http requests to pod ip and port

Env:
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
km12-01 Ready master 26d v1.13.0 10.42.140.154 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.6.2
km12-02 Ready master 26d v1.13.0 10.42.104.113 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.6.2
km12-03 Ready master 26d v1.13.0 10.42.177.142 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.6.2
prod-k8s-node002 Ready node 25d v1.13.0 10.42.78.21 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.3.2
prod-k8s-tmpnode005 Ready node 24d v1.13.0 10.42.177.75 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.3.2
calico v3.3.1
What Happen:
I have a deployment, with 2 pods scheduled on prod-k8s-node002 and prod-k8s-tmpnode005. Just like this:
# kubectl -n gitlab-prod get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
api-monkey-76489bd8c9-7gbkt 1/1 Running 0 35m 192.244.199.37 prod-k8s-node002 <none> <none>
api-monkey-76489bd8c9-w2zrs 1/1 Running 0 55m 192.244.124.240 prod-k8s-tmpnode005 <none> <none>
Now I curl each pod from a master node, say it is km12-01:
# # wait 0ms, response a json string, which has a property named 'data', it is a string, which is '1' repeated by 1000 times
# curl -v '192.244.124.240:3000/health/test?ms=0&content=1&repeat=1000'
* Trying 192.244.124.240...
* Connected to 192.244.124.240 (192.244.124.240) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=1000 HTTP/1.1
> Host: 192.244.124.240:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 1011
< ETag: W/"3f3-oUOV2TWikka+Y8l16Cqo/Q"
< Date: Sat, 08 Dec 2018 20:12:26 GMT
< Connection: keep-alive
<
* Connection #0 to host 192.244.124.240 left intact
{"data":"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"}
# curl -v '192.244.199.37:3000/health/test?ms=0&content=1&repeat=1000'
* Trying 192.244.199.37...
* Connected to 192.244.199.37 (192.244.199.37) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=1000 HTTP/1.1
> Host: 192.244.199.37:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 1011
< ETag: W/"3f3-oUOV2TWikka+Y8l16Cqo/Q"
< Date: Sat, 08 Dec 2018 20:15:16 GMT
< Connection: keep-alive
<
* Connection #0 to host 192.244.199.37 left intact
{"data":"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"}
Good, both work.
So if I make a longer response body (2000 bytes), what will happen?
# curl -v '192.244.124.240:3000/health/test?ms=0&content=1&repeat=2000'
* Trying 192.244.124.240...
* Connected to 192.244.124.240 (192.244.124.240) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=2000 HTTP/1.1
> Host: 192.244.124.240:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Unfortunately, the connection was hanging on and then reset by peer after couple minutes. But it works on the host of the requested pod.
Curl a pod from its host:
# curl -v '192.244.124.240:3000/health/test?ms=0&content=1&repeat=2000'
* Trying 192.244.124.240...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.244.124.240 (192.244.124.240) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=2000 HTTP/1.1
> Host: 192.244.124.240:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
{"data":"11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"}< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 2011
< ETag: W/"7db-ViIL+fXpsfh/YwmlqDHSsQ"
< Date: Sat, 08 Dec 2018 20:22:55 GMT
< Connection: keep-alive
<
{ [2011 bytes data]
100 2011 100 2011 0 0 440k 0 --:--:-- --:--:-- --:--:-- 490k
* Connection #0 to host 192.244.124.240 left intact
The other one is in the same situation.
Summary:
I tried many times, and found a interesting and strange thing: If I request a pod from any other node (except the host of the pod), the response body cannot be longer than 1140 bytes. Otherwise, the connection will hang on.
My problem:
How it happen? And how can I break this limitation?
This is the kubeadm-1.12.0 initialization file:
# cat kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
controlPlaneEndpoint: 10.42.79.210:6443
kind: ClusterConfiguration
kubernetesVersion: v1.12.0
networking:
podSubnet: 192.244.0.0/16
serviceSubnet: 192.96.0.0/16
apiServerCertSANs:
- 10.42.140.154
- 10.42.104.113
- 10.42.177.142
- km12-01
- km12-02
- km12-03
- 127.0.0.1
- localhost
- 10.42.79.210
etcd:
external:
endpoints:
- https://10.42.140.154:2379
- https://10.42.104.113:2379
- https://10.42.177.142:2379
caFile: /etc/kubernetes/ssl/ca.pem
certFile: /etc/etcd/ssl/etcd.pem
keyFile: /etc/etcd/ssl/etcd-key.pem
dataDir: /var/lib/etcd
clusterDNS:
- 192.96.0.2
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
We use an external etcd cluster, and the underlying network uses IPVS rather than iptables. We did an upgrade from 1.12.0 to 1.13.0.
# kubeadm upgrade apply 1.13.0
You can use this to test, and we're the ones that came up after the upgrade. #Adam Otto
There are some IPVS errors as follows:
kube-proxy logs
Seems like to be an MTU issue.

TikaJAXRS PUT from Python client

Apache Tika should be accessible from Python program via HTTP, but I can't get it to work.
I am using this command to run the server (with and without the two options at the end):
java -jar tika-server-1.17.jar --port 5677 -enableUnsecureFeatures -enableFileUrl
And it works fine with curl:
curl -v -T /tmp/tmpsojwBN http://localhost:5677/tika
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5677 (#0)
> PUT /tika HTTP/1.1
> Host: localhost:5677
> User-Agent: curl/7.47.0
> Accept: */*
> Accept-Encoding: gzip, deflate
> Content-Length: 418074
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Sat, 07 Apr 2018 12:28:41 GMT
< Transfer-Encoding: chunked
< Server: Jetty(8.y.z-SNAPSHOT)
But when I try something like (tried different combinations for headers, here I recreated same headers as python-tika client uses):
with tempfile.NamedTemporaryFile() as tmp_file:
download_file(url, tmp_file)
payload = open(tmp_file.name, 'rb')
headers = {
'Accept': 'application/json',
'Content-Disposition': 'attachment; filename={}'.format(
os.path.basename(tmp_file.name))}
response = requests.put(TIKA_ENDPOINT_URL + '/tika', payload,
headers=headers,
verify=False)
I've tried to use payload as well as fileUrl - with the same result of WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable and java stack trace on the server. Full trace:
WARN javax.ws.rs.ClientErrorException: HTTP 406 Not Acceptable
at org.apache.cxf.jaxrs.utils.SpecExceptions.toHttpException(SpecExceptions.java:117)
at org.apache.cxf.jaxrs.utils.ExceptionUtils.toHttpException(ExceptionUtils.java:173)
at org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:542)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:177)
at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:77)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:261)
at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:76)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
I've also tried to compare ( with nc -l localhost 5677 | less) what is so different with two requests (payload abbreviated):
From curl:
PUT /tika HTTP/1.1
Host: localhost:5677
User-Agent: curl/7.47.0
Accept: */*
Content-Length: 418074
Expect: 100-continue
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
From Python requests library:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-type: application/pdf
Content-Length: 246176
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
The question is, what is the correct way to call Tika server from Python?
I've also tried python tika library in client-only mode and using tika-app via jnius. With tika client, as well as using tika-app.jar with pyjnius, I only freezes (call never returns) when I use them in a celery worker. At the same, pyjnius / tika-app and tika-python script both work nicely in a script: I have not figured out what is wrong inside celery worker. I guess, something to do with threading and/or initialization in wrong place. But that is a topic for another question.
And here is what tika-python requests:
PUT /tika HTTP/1.1
Host: localhost:5677
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-requests/2.13.0
Content-Disposition: attachment; filename=tmpb3YkTq
Content-Length: 183234
%PDF-1.4
%<D3><EB><E9><E1>
1 0 obj
<</Creator (Chromium)
And now it seems like this is some kind of a problem with tika server:
$ tika-python --verbose --server 'localhost' --port 5677 parse all /tmp/tmpb3YkTq
2018-04-08 09:44:11,555 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7f0b688eb1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:44:13 GMT', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
['./tmpb3YkTq_meta.json']
Cf:
$ tika-python --verbose --server 'localhost' --port 5677 parse text /tmp/tmpb3YkTq
2018-04-08 09:43:38,326 [MainThread ] [INFO ] Writing ./tmpb3YkTq_meta.json
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Request headers: ', {'Accept': 'application/json', 'Content-Disposition': 'attachment; filename=tmpb3YkTq'})
(<open file '<stderr>', mode 'w' at 0x7fc3eee4a1e0>, 'Response headers: ', {'Date': 'Sun, 08 Apr 2018 06:43:38 GMT', 'Content-Length': '0', 'Server': 'Jetty(8.y.z-SNAPSHOT)'})
2018-04-08 09:43:38,409 [MainThread ] [WARNI] Tika server returned status: 406
['./tmpb3YkTq_meta.json']

What is wrong with my GET request?

Sorry to bother with something that should be easy.
I have this HTTP GET request:
GET /ip HTTP/1.1
Host: httpbin.org
Connection: close
Accept: */*
User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)
When I send this request via my ESP8266 it returns a 404 error:
HTTP/1.1 404 Not Found
Date: Fri, 04 Sep 2015 16:34:46 GMT
Server: Apache
Content-Length: 1363
X-Frame-Options: deny
Connection: close
Content-Type: text/html
But when I (and you) go to http://httpbin.org/ip it works perfectly!
What is wrong?
DETAILS
I construct my request in Lua:
conn:on("connection", function(conn, payload)
print('\nConnected')
req = "GET /ip"
.." HTTP/1.1\r\n"
.."Host: httpbin.org\r\n"
.."Connection: close\r\n"
.."Accept: */*\r\n"
.."User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n"
.."\r\n"
print(req)
conn:send(req)
end)
And if I use another host (given is this example) it works:
conn:on("connection", function(conn, payload)
print('\nConnected')
conn:send("GET /esp8266/test.php?"
.."T="..(tmr.now()-Tstart)
.."&heap="..node.heap()
.." HTTP/1.1\r\n"
.."Host: benlo.com\r\n"
.."Connection: close\r\n"
.."Accept: */*\r\n"
.."User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n"
.."\r\n")
end)
Are you actually connecting to httpbin.org ? Or somewhere else?
I just tried issuing your request by typing it into telnet, and it worked for me. But the server responding was nginx, whereas your example shows apache.
$ telnet httpbin.org 80
Trying 54.175.219.8...
Connected to httpbin.org.
Escape character is '^]'.
GET /ip HTTP/1.1
Host: httpbin.org
Connection: close
Accept: */*
User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 07 Oct 2015 06:08:40 GMT
Content-Type: application/json
Content-Length: 32
Connection: close
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
{
"origin": "124.149.55.34"
}
Connection closed by foreign host.
When I try another request with some other URI to force a 404 response, I see this:
HTTP/1.1 404 NOT FOUND
Server: nginx
Date: Wed, 07 Oct 2015 06:12:21 GMT
Content-Type: text/html
Content-Length: 233
Connection: close
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Which again is nothing like the response you said you got from httpbin.org.
http.get("http://httpbin.org/ip", nil, function(code, data)
if (code < 0) then
print("HTTP request failed")
else
print(code, data)
end
end)
http.post('http://httpbin.org/post',
'Content-Type: application/json\r\n',
'{"hello":"world"}',
function(code, data)
if (code < 0) then
print("HTTP request failed")
else
print(code, data)
end
end)
see the reference here.
It's your request line the server doesn't like.
This will do the job:
GET http://httpbin.org/ip HTTP/1.1
Host: httpbin.org

Async responses with Aleph aren't being received over IPv4 but are with IPv6

I'm trying to get server-sent events set up in Clojure with Aleph, but it's just not working over IPv4. Everything is fine if I connect over IPv6. This occurs both on Linux and MacOS. I've got a full example of what I'm talking about on GitHub.
I don't think I'm doing anything particularly fancy. The whole code is up on GitHub, but essentially my program is:
(def my-channel (permanent-channel))
(defroutes app-routes
(GET "/events" []
{:headers {"Content-Type" "text/event-stream"}
:body my-channel}))
(def app
(handler/site app-routes))
(start-server (wrap-ring-handler app) {:port 3000}))
However, when I connect to 127.0.0.1:3000, I can see curl sending the request headers, but it just hangs, never printing the response headers:
$ curl -vvv http://127.0.0.1:3000/events
* About to connect() to 127.0.0.1 port 3000 (#0)
* Trying 127.0.0.1...
* Adding handle: conn: 0x7f920a004400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f920a004400) send_pipe: 1, recv_pipe: 0
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /events HTTP/1.1
> User-Agent: curl/7.30.0
> Host: 127.0.0.1:3000
> Accept: */*
If I connect over IPv6 the response comes right away, and events that I enqueue in the channel get sent correctly:
$ curl -vvv http://localhost:3000/events
* Adding handle: conn: 0x7f943c001a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f943c001a00) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 3000 (#0)
* Trying ::1...
* Connected to localhost (::1) port 3000 (#0)
> GET /events HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:3000
> Accept: */*
>
< HTTP/1.1 200 OK
* Server aleph/0.3.0 is not blacklisted
< Server: aleph/0.3.0
< Date: Tue, 15 Apr 2014 12:27:05 GMT
< Connection: keep-alive
< Content-Type: text/event-stream
< Transfer-Encoding: chunked
I have also reproduced this behaviour in Chrome. In both the IPv4 and IPv6 cases, tcpdump shows that the response headers are going over the wire.
This behaviour occurs both with lein run and an uberjar. It also occurs if I execute the uberjar with -Djava.net.preferIPv4Stack=true.
How do I get my application to behave the same over IPv4 as over IPv6?

Resources