Env:
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
km12-01 Ready master 26d v1.13.0 10.42.140.154 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.6.2
km12-02 Ready master 26d v1.13.0 10.42.104.113 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.6.2
km12-03 Ready master 26d v1.13.0 10.42.177.142 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.6.2
prod-k8s-node002 Ready node 25d v1.13.0 10.42.78.21 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.3.2
prod-k8s-tmpnode005 Ready node 24d v1.13.0 10.42.177.75 <none> Ubuntu 16.04.5 LTS 4.17.0-041700-generic docker://17.3.2
calico v3.3.1
What Happen:
I have a deployment, with 2 pods scheduled on prod-k8s-node002 and prod-k8s-tmpnode005. Just like this:
# kubectl -n gitlab-prod get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
api-monkey-76489bd8c9-7gbkt 1/1 Running 0 35m 192.244.199.37 prod-k8s-node002 <none> <none>
api-monkey-76489bd8c9-w2zrs 1/1 Running 0 55m 192.244.124.240 prod-k8s-tmpnode005 <none> <none>
Now I curl each pod from a master node, say it is km12-01:
# # wait 0ms, response a json string, which has a property named 'data', it is a string, which is '1' repeated by 1000 times
# curl -v '192.244.124.240:3000/health/test?ms=0&content=1&repeat=1000'
* Trying 192.244.124.240...
* Connected to 192.244.124.240 (192.244.124.240) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=1000 HTTP/1.1
> Host: 192.244.124.240:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 1011
< ETag: W/"3f3-oUOV2TWikka+Y8l16Cqo/Q"
< Date: Sat, 08 Dec 2018 20:12:26 GMT
< Connection: keep-alive
<
* Connection #0 to host 192.244.124.240 left intact
{"data":"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"}
# curl -v '192.244.199.37:3000/health/test?ms=0&content=1&repeat=1000'
* Trying 192.244.199.37...
* Connected to 192.244.199.37 (192.244.199.37) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=1000 HTTP/1.1
> Host: 192.244.199.37:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 1011
< ETag: W/"3f3-oUOV2TWikka+Y8l16Cqo/Q"
< Date: Sat, 08 Dec 2018 20:15:16 GMT
< Connection: keep-alive
<
* Connection #0 to host 192.244.199.37 left intact
{"data":"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"}
Good, both work.
So if I make a longer response body (2000 bytes), what will happen?
# curl -v '192.244.124.240:3000/health/test?ms=0&content=1&repeat=2000'
* Trying 192.244.124.240...
* Connected to 192.244.124.240 (192.244.124.240) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=2000 HTTP/1.1
> Host: 192.244.124.240:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Unfortunately, the connection was hanging on and then reset by peer after couple minutes. But it works on the host of the requested pod.
Curl a pod from its host:
# curl -v '192.244.124.240:3000/health/test?ms=0&content=1&repeat=2000'
* Trying 192.244.124.240...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.244.124.240 (192.244.124.240) port 3000 (#0)
> GET /health/test?ms=0&content=1&repeat=2000 HTTP/1.1
> Host: 192.244.124.240:3000
> User-Agent: curl/7.47.0
> Accept: */*
>
{"data":"11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111"}< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Content-Type: application/json; charset=utf-8
< Content-Length: 2011
< ETag: W/"7db-ViIL+fXpsfh/YwmlqDHSsQ"
< Date: Sat, 08 Dec 2018 20:22:55 GMT
< Connection: keep-alive
<
{ [2011 bytes data]
100 2011 100 2011 0 0 440k 0 --:--:-- --:--:-- --:--:-- 490k
* Connection #0 to host 192.244.124.240 left intact
The other one is in the same situation.
Summary:
I tried many times, and found a interesting and strange thing: If I request a pod from any other node (except the host of the pod), the response body cannot be longer than 1140 bytes. Otherwise, the connection will hang on.
My problem:
How it happen? And how can I break this limitation?
This is the kubeadm-1.12.0 initialization file：
# cat kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1alpha3
controlPlaneEndpoint: 10.42.79.210:6443
kind: ClusterConfiguration
kubernetesVersion: v1.12.0
networking:
podSubnet: 192.244.0.0/16
serviceSubnet: 192.96.0.0/16
apiServerCertSANs:
- 10.42.140.154
- 10.42.104.113
- 10.42.177.142
- km12-01
- km12-02
- km12-03
- 127.0.0.1
- localhost
- 10.42.79.210
etcd:
external:
endpoints:
- https://10.42.140.154:2379
- https://10.42.104.113:2379
- https://10.42.177.142:2379
caFile: /etc/kubernetes/ssl/ca.pem
certFile: /etc/etcd/ssl/etcd.pem
keyFile: /etc/etcd/ssl/etcd-key.pem
dataDir: /var/lib/etcd
clusterDNS:
- 192.96.0.2
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
We use an external etcd cluster, and the underlying network uses IPVS rather than iptables. We did an upgrade from 1.12.0 to 1.13.0.
# kubeadm upgrade apply 1.13.0
You can use this to test, and we're the ones that came up after the upgrade. #Adam Otto
There are some IPVS errors as follows：
kube-proxy logs
Seems like to be an MTU issue.
Related
I've installeld Riak 2 on a Centos 6 box and I've managed to boot Riak but now get this error when I try to test it:
curl -v http://<hostname>:8098/types/default/props
* About to connect() to <hostname> port 8098 (#0)
* Trying <hostname>... connected
* Connected to <hostname> (<hostname>) port 8098 (#0)
> GET /types/default/props HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <hostname>:8098
> Accept: */*
>
< HTTP/1.1 404 Object Not Found
< Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)
< Date: Fri, 21 Aug 2015 01:02:48 GMT
< Content-Type: text/html
< Content-Length: 193
<
* Connection #0 to host <hostname> left intact
* Closing connection #0
<HTML><HEAD><TITLE>404 Not Found</TITLE></HEAD><BODY><H1>Not Found</H1>The requested document was not found on this server.<P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></BODY></HTML>[root#riak1 ~]#
Also I did modify my apps.config and vm like so:
{pb, [ {"", 8087 } ]}
{http, [ {"", 8098 } ]},
-name riak#
What does this mean and how do I fix it?
I'm using phpari with Asterisk 13 and trying to record a bridge (mixing type).
In my code:
$this->phpariObject->bridges()->bridge_start_recording($bridgeID, "debug", "wav");
It returns:
array(4) {
["name"]=>
string(5) "debug"
["format"]=>
string(3) "wav"
["state"]=>
string(6) "queued"
["target_uri"]=>
string(15) "bridge:5:1:503"
}
When and I stop and save with
$this->phpariObject->recordings()->recordings_live_stop_n_store("debug");
It returns FALSE.
I debug with
curl -v -u xxxx:xxxx -X POST "http://localhost:8088/ari/recordings/live/debug/stop"
Result:
* About to connect() to localhost port 8088 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8088 (#0)
* Server auth using Basic with user 'xxxxx'
> POST /ari/recordings/live/debug/stop HTTP/1.1
> Authorization: Basic xxxxxxx
> User-Agent: curl/7.19.7 (xxxxx) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8088
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: Asterisk/13.2.0
< Date: Thu, 19 Feb 2015 11:58:18 GMT
< Cache-Control: no-cache, no-store
< Content-type: application/json
< Content-Length: 38
<
{
"message": "Recording not found"
* Connection #0 to host localhost left intact
* Closing connection #0
}
Asterisk CLI verbose 5 trace: http://pastebin.com/QZXnpXVA
So, I've solved the problem.
It was a simple write permission problem.
Asterisk user couldn't write on /var/spool/asterisk/recording because it was owned by root.
Changing the ownership to the asterisk user solved it.
I detected this problem by looking at the Asterisk CLI trace again:
-- x=0, open writing: /var/spool/asterisk/recording/debug format: sln, (nil)
This (nil) indicates that the file could not be written, so I checked the folder and saw where the problem was.
I'm trying to get server-sent events set up in Clojure with Aleph, but it's just not working over IPv4. Everything is fine if I connect over IPv6. This occurs both on Linux and MacOS. I've got a full example of what I'm talking about on GitHub.
I don't think I'm doing anything particularly fancy. The whole code is up on GitHub, but essentially my program is:
(def my-channel (permanent-channel))
(defroutes app-routes
(GET "/events" []
{:headers {"Content-Type" "text/event-stream"}
:body my-channel}))
(def app
(handler/site app-routes))
(start-server (wrap-ring-handler app) {:port 3000}))
However, when I connect to 127.0.0.1:3000, I can see curl sending the request headers, but it just hangs, never printing the response headers:
$ curl -vvv http://127.0.0.1:3000/events
* About to connect() to 127.0.0.1 port 3000 (#0)
* Trying 127.0.0.1...
* Adding handle: conn: 0x7f920a004400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f920a004400) send_pipe: 1, recv_pipe: 0
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /events HTTP/1.1
> User-Agent: curl/7.30.0
> Host: 127.0.0.1:3000
> Accept: */*
If I connect over IPv6 the response comes right away, and events that I enqueue in the channel get sent correctly:
$ curl -vvv http://localhost:3000/events
* Adding handle: conn: 0x7f943c001a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f943c001a00) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 3000 (#0)
* Trying ::1...
* Connected to localhost (::1) port 3000 (#0)
> GET /events HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:3000
> Accept: */*
>
< HTTP/1.1 200 OK
* Server aleph/0.3.0 is not blacklisted
< Server: aleph/0.3.0
< Date: Tue, 15 Apr 2014 12:27:05 GMT
< Connection: keep-alive
< Content-Type: text/event-stream
< Transfer-Encoding: chunked
I have also reproduced this behaviour in Chrome. In both the IPv4 and IPv6 cases, tcpdump shows that the response headers are going over the wire.
This behaviour occurs both with lein run and an uberjar. It also occurs if I execute the uberjar with -Djava.net.preferIPv4Stack=true.
How do I get my application to behave the same over IPv4 as over IPv6?
I have installed OpenStack Swift service on machine 1 (Centos 6) by following instructions given on http://docs.openstack.org/developer/swift/development_saio.html.
I am able to retrieve token using curl on the same environment.
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://machine-1IP:8099/auth/v1.0
* About to connect() to machine-1 ip port 8099 (#0)
* Trying machine-1ip... connected
* Connected to machine-1 port 8099 (#0)
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: machine-1IP:8099
> Accept: */*
> X-Storage-User: test:tester
> X-Storage-Pass: testing
>
< HTTP/1.1 200 OK
< X-Storage-Url: http://machine-1IP:8099/v1/AUTH_test
< X-Auth-Token: AUTH_tk1dc541c4be23424a92477b13deaa2fac
< Content-Type: text/html; charset=UTF-8
< X-Storage-Token: AUTH_tk1dc541c4be23424a92477b13deaa2fac
< Content-Length: 0
< Date: Thu, 18 Jul 2013 11:45:34 GMT
If I try to access it from another machine (Machine 2).
I am getting host is not reachable exception. I am able to ping machine-1 from machine-2.
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://machine-1IP:port/auth/v1.0
* Adding handle: conn: 0xa7b670
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0xa7b670) send_pipe: 1, recv_pipe: 0
* Could not resolve host: test
* Closing connection 0
curl: (6) Could not resolve host: test
* Adding handle: conn: 0xa7c1b0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 1 (0xa7c1b0) send_pipe: 1, recv_pipe: 0
* Could not resolve host: testing'
* Closing connection 1
curl: (6) Could not resolve host: testing'
* About to connect() to machine-1 IP port 8099 (#2)
* Trying machine-1 IP...
* Adding handle: conn: 0xa7c1b0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 2 (0xa7c1b0) send_pipe: 1, recv_pipe: 0
* Host unreachable
* Failed connect to machine-1IP:8099; Host unreachable
* Closing connection 2
curl: (7) Failed connect to Machine-1 IP:8099; Host unreachable
Please suggest why Swift REST services are not accessible from remote machine? Did I missed out anything in configurations?
Host unreachable means Machine 2 can not access to Machine 1. It's not related to Openstack/Swift. Check your router settings and /etc/hosts.
* Could not resolve host: testing'
* Failed connect to machine-1IP:8099; Host unreachable
Consider the following hyperlink:
<a href="http://www.cs.rutgers.edu/∼shklar/">
What HTTP/1.0 request will get submitted by the browser?
What HTTP/1.1 request will get submitted by the browser?
Will these requests change if the browser is configured to
contact an HTTP proxy? If yes, how?
While you could use tcpdump to dump the actual network traffic, curl is surely more handy to test the HTTP conversation from the command line.
An HTTP/1.0 request:
curl -v -0 http://www.cs.rutgers.edu/∼shklar/
* About to connect() to www.cs.rutgers.edu port 80 (#0)
* Trying 128.6.4.24...
* connected
* Connected to www.cs.rutgers.edu (128.6.4.24) port 80 (#0)
> GET /∼shklar/ HTTP/1.0
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.cs.rutgers.edu
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Wed, 31 Oct 2012 17:57:31 GMT
< Server: Apache/1.3.26 (Unix)
< Content-Type: text/html; charset=iso-8859-1
< Connection: close
An HTTP/1.1 request:
curl -v http://www.cs.rutgers.edu/∼shklar/
* About to connect() to www.cs.rutgers.edu port 80 (#0)
* Trying 128.6.4.24...
* connected
* Connected to www.cs.rutgers.edu (128.6.4.24) port 80 (#0)
> GET /∼shklar/ HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.cs.rutgers.edu
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Wed, 31 Oct 2012 17:59:47 GMT
< Server: Apache/1.3.26 (Unix)
< Content-Type: text/html; charset=iso-8859-1
< Transfer-Encoding: chunked
Use the -x (or --proxy) <[protocol://][user#password]proxyhost[:port]> switch to use a proxy and see the results.
More about curl here: http://curl.haxx.se/docs/manpage.html