I have four single node installs for IBM Cloudant. All IBM Cloudant instances are installed on RHEL 6.5. Three of the four installs are working just fine. On the fourth, I am having issues with dashboard.html, haproxy, and connecting to databases. The following are the details from my debugging effort:
I used the same install instructions on all 4 machines.
I also verified that all RPMS are at same levels.
I made sure /etc/hosts, /etc/resolv.conf, and /etc/sysconfig/network are all set correctly.
I disabled iptables for some of my tests, with not luck.
haproxy is set to run on port 10080
nginx is set to run on 5657
from the end of a weatherreport run.
['cloudant#prdpcrdlp01.w3-969.ibm.com'] [warning] Cluster member cloudant#localhost is not connected to this node. Please check whether it is down.
From haproxy.log 500's for all of my database from dashboard.html
Mar 2 12:41:38 localhost.localdomain haproxy[26792]: 9.72.190.182:49510 [02/Mar/2016:12:41:38.166] dbfarm dbfarm/prdpcrdlp01.w3-969.ibm.com 181/0/0/2/183 500 312 - - ---- 5/5/0/1/0 0/0 "GET /stats HTTP/1.1"
Mar 2 12:41:38 localhost.localdomain haproxy[26792]: 9.72.190.182:49516 [02/Mar/2016:12:41:34.963] dbfarm dbfarm/prdpcrdlp01.w3-969.ibm.com 3417/0/0/2/3419 500 312 - - ---- 5/5/0/1/0 0/0 "GET /_replicator HTTP/1.1"
Mar 2 12:41:38 localhost.localdomain haproxy[26792]: 9.72.190.182:49517 [02/Mar/2016:12:41:34.964] dbfarm dbfarm/prdpcrdlp01.w3-969.ibm.com 3425/0/0/3/3428 500 312 - - ---- 5/5/1/2/0 0/0 "GET /metrics HTTP/1.1"
Mar 2 12:41:38 localhost.localdomain haproxy[26792]: 9.72.190.182:49518 [02/Mar/2016:12:41:34.968] dbfarm dbfarm/prdpcrdlp01.w3-969.ibm.com 3422/0/0/3/3425 500 312 - - ---- 5/5/0/1/0 0/0 "GET /ray HTTP/1.1"
Mar 2 12:41:38 localhost.localdomain haproxy[26792]: 9.72.190.182:49515 [02/Mar/2016:12:41:34.925] dbfarm dbfarm/prdpcrdlp01.w3-969.ibm.com 3726/0/0/2/3728 500 312 - - ---- 5/5/0/1/0 0/0 "GET /test2 HTTP/1.1"
from cloudant.log
2016-03-02 12:55:52.245 [error] cloudant#prdpcrdlp01.w3-969.ibm.com <0.10284.0> Missing IOQ stats db:
2016-03-02 12:56:04.066 [error] cloudant#prdpcrdlp01.w3-969.ibm.com <0.10127.0> httpd 500 error response:
{"error":"nodedown","reason":"progress not possible"}
from firebug...
I see 500's when attempting to access all db's
Example: http://prdpcrdlp01.w3-969.ibm.com:10080/test2"
I have performed a clean install of IBM Cloudant twice and the issue persists.
I would guess the system has had multiple node names over its lifetime, i.e. previously it was brought up with the node name cloudant#localhost and now it has the node name cloudant#prdpcrdlp01.w3-969.ibm.com. Any databases created when the node name was cloudant#localhost will therefore be unavailable now.
What does the output of curl -X GET http://prdpcrdlp01.w3-969.ibm.com:10080/_membership look like?
solved, the DELETE command worked, then I just needed to rerun "configure.sh -D" Which deleted all of the db's and recreated them.
Related
In order to make changes on the web site I closed the cache then when I tried to open it it gave an error. I saw solutions to that problem on the web such as using htp{}.. etc. But it didn't work as well. It gives an error like;
it cannot be reached at that website or declined to connect.
And this is not a problem caused just by the website. It gives the same error in the server that has portals which means I received the same error notification in different apps too.
At dokku there are two different nginx files. One of them I get
"server" directive is not allowed here
and the other one I get
"proxy_cache_path" directive is not allowed here
but I think If I can solve one of them system would work again and when I try to open the cache I get "proxy_cache_path" error
In the same time, inside of the /home/dokku/myapp/nginx files
include /home/dokku/myapp/nginx.conf.d/*.conf;`error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 /400-error.html;`location /400-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;`part. ``} `
However in the /home/dokku/myapp/nginx.conf.d/*.conf; named file there is no file such as I mentioned before is exist
when I run systemctl status nginx command I got these errors:
Starting A high performance web server and a reverse proxy server...
nginx[26504]: nginx: [emerg] "proxy_cache_path" directive is not allowed here in /home/dokku/myapp/nginx.conf:93
nginx[26504]: nginx: configuration file /etc/nginx/nginx.conf test failed
systemd[1]: nginx.service: Control process exited, code=exited status=1
systemd[1]: Failed to start A high performance web server and a reverse proxy server.
systemd[1]: nginx.service: Unit entered failed state.
systemd[1]: nginx.service: Failed with result 'exit-code'.
I tried dokku plugin:uninstall nginx-cache command bur error not resolved
Reloading nginx
nginx: [emerg] "proxy_cache_path" directive is not allowed here in /home/dokku/myapp/nginx.conf:93
nginx: configuration file /etc/nginx/nginx.conf test failed
validate_nginx failed.
contents of /home/dokku/muhasebe/nginx.conf below...
and nginx file 93.row:
proxy_cache_path /tmp/nginx levels:1-2 keys_zone=dokku:10m inactive=60m use_temp_path=off;
( also tmp file is not have nginx file )
The proxy_cache_path directive is only allowed within the http context. It is likely that the plugin version you are using didn't correctly apply the proxy_cache config.
After removing the plugin, you will want to call dokku nginx:build-config to rebuild your configuration without the invalid calls.
Been trying to share the local site using a temporary url https://xxxxx.ngrok.io running with Laravel Valet. When I run Valet version 1.1.22:
valet --version
Laravel Valet version 1.1.22
securely the ngrok url leads to a connection refused. When it is unsecure it leads to connection refused as well (404 only with valet running on the secondary machine as it wouldn't be found there). Either that or a DNS resolving issue as I mention later on.
Locally on my wifi network and on the PC is works just fine. Access logs show me this:
127.0.0.1 - [03/Oct/2016:08:57:06 +0300] "POST /server.php?doing_wp_cron=1475474226.5450510978698730468750 HTTP/1.1" 200 0
127.0.0.1 - [03/Oct/2016:08:57:07 +0300] "POST /server.php HTTP/1.1" 200 47
127.0.0.1 - [03/Oct/2016:08:59:09 +0300] "POST /server.php?doing_wp_cron=1475474348.8563120365142822265625 HTTP/1.1" 200 0
127.0.0.1 - [03/Oct/2016:08:59:10 +0300] "POST /server.php HTTP/1.1" 200 47
Still do not see an error related to refused connection in this log at ~/.valet/Log/access.log . Error logs show old errors, not related to this issue. Ngrook window in terminal shows 301 Moved permanently on the two loads I just tried. Ngrok status site http://localhost:4040/status showed me:
GET / HTTP/1.1
Host: site.dev
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.8 (KHTML, like Gecko) Version/9.1.3 Safari/601.7.8
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us
X-Forwarded-For: xx.xx.xx.xxx
X-Forwarded-Proto: https
X-Original-Host: xxxxxx.ngrok.io
and then the redirect:
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Location: http://mysite.dev/
Server: Caddy
Status: 301 Moved Permanently
X-Powered-By: PHP/7.0.11
X-Ua-Compatible: IE=edge
Date: Mon, 03 Oct 2016 06:14:10 GMT
Content-Length: 0
Caddyfile here for completion (generated by Valet):
import /Users/jasper/.valet/Caddy/*
:80 {
fastcgi / 127.0.0.1:9000 php {
index server.php
}
rewrite {
to /server.php?{query}
}
log /Users/jasper/.valet/Log/access.log {
rotate {
size 10
age 3
keep 1
}
}
errors {
log /Users/jasper/.valet/Log/error.log {
size 10
age 3
keep 1
}
}
}
Ngrok is running too (added after tld domain was changed to .localhost):
ps aux | grep ngrok
jasper 1260 0.0 0.2 556735952 28692 s001 S+ 10:23AM 1:27.14 /Users/jasper/.composer/vendor/laravel/valet/bin/ngrok http -host-header=rewrite site.localhost:80
root 1254 0.0 0.1 2463108 8964 s001 S+ 10:23AM 0:00.01 sudo -u jasper /Users/jasper/.composer/vendor/laravel/valet/bin/ngrok http -host-header=rewrite site.localhost:80
jasper 3557 0.0 0.0 2432804 2096 s000 S+ 2:36PM 0:00.00 grep ngrok
So it does hit the Caddy Server and the ngrok status does show that. But it then does a redirect which translates into a connection refused or DNS resolution problemns for the browsers.. So what is the issue here?
In the end I realized WordPress was creating an extra redirect using its permalink structure. So when you turn off permalinks you can share your Laravel Valet WordPress site to the outside world using Ngrok. Not the perfect solution, but at least one that works and that allows you to show your work in progress to clients running it on your local machine.
I have a question. We use nginx+uwsgi stack and see many errors like:
Aug 30 00:00:55 imfmce-va-81-2 uwsgi: Tue Aug 30 00:00:55 2016 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /provisioning/user/f205970b-6a9f-42b5-830f-c2bec9967b32 (ip 10.216.153.254) !!!
I understand that error occurs when client close connection before reading response or by uwsgi_read_timeout, but I don't understand why in access log I cannt see any error, nginx just log 200 OK:
Aug 30 00:00:55 imfmce-va-81-2 provisioning: active [ 55544 ] 10.216.153.254 Sync-Wopi-SyncLocksTask hostpilot 73af65e4-5984-4b2c-baf4-c88cf8385898 - ECDHE-RSA-AES256-GCM-SHA384 GET /provisioning/user/f205970b-6a9f-42b5-830f-c2bec9967b32 - 0,0,1,0 200 - 1 OK - 321 515 844
We use next format log line:
log_format ss_log_format "active\t[ \$pid ]\t\$remote_addr\t\$http_user_agent\t\$upstream_http_x_user_identity\t\$http_x_client_id\t\$http_x_request_id\t\$ssl_cipher\t\$request_method\t\$uri\t\$args\t\$upstream_http_x_durations\t\$status\t\$upstream_status\t\$http_x_error_code\t\$connection_requests\t\$request_completion\t\$content_length\t\$request_length\t\$body_bytes_sent\t\$bytes_sent";
I would like you to understand that,
we don't need to fix this error, we just need have right access logs.
I'm serving two websites through HAProxy and Varnish. There's a wiki site and a wordpress site. The wiki site works continuously and without problem. However the Wordpress site continuously shows a 504 error each time you reload the page.
If I spoof the wordpress site in my hosts file by using the IP of the varnish server instead of HAProxy the site comes back and starts working fine. It's only when wordpress is on haproxy that the site 504's.
I'd like to know how to turn on debug logging for HAProxy and also maybe get some help solving this problem.
This is all that I see in the logs for haproxy:
Apr 3 20:29:18 lb1.example.com haproxy[18501]: 52.21.231.226:52845 [03/Apr/2016:20:29:15.318] varnish-cluster varnish-cluster/varnish1 0/0/0/2786/2786 200 626 - - --NR 2/2/1/1/0 0/0 "HEAD / HTTP/1.1"
Apr 3 20:29:28 lb1.example.com haproxy[18501]: 61.174.10.22:18645 [03/Apr/2016:20:29:09.522] varnish-cluster varnish-cluster/varnish1 0/0/0/18206/19039 404 101736 - - --VN 0/0/0/0/0 0/0 "GET /groups/ HTTP/1.0"
Apr 3 20:29:34 lb1.example.com haproxy[18501]: 61.174.10.22:26372 [03/Apr/2016:20:29:31.045] varnish-cluster varnish-cluster/varnish1 0/0/0/3048/3048 301 549 - - --VN 0/0/0/0/0 0/0 "GET /members/pzwkathi09454/activity HTTP/1.0"
Apr 3 20:29:54 lb1.example.com haproxy[18501]: 61.174.10.22:27761 [03/Apr/2016:20:29:34.879] varnish-cluster varnish-cluster/varnish1 0/0/0/-1/20003 504 194 - - sHVN 0/0/0/0/0 0/0 "GET /activity/ HTTP/1.0"
And this is my config:
global
log 127.0.0.1 local2 debug
user root
group root
defaults
log global
retries 2
timeout connect 12000
timeout server 20000
timeout client 20000
listen varnish-cluster 0.0.0.0:80
mode http
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:secret
balance roundrobin
option http-server-close
timeout http-keep-alive 3000
option forwardfor
option httplog
cookie PHPSESSID prefix
server varnish1 xx.xx.xx.xx:80 cookie s1 check
listen mysql-master-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-master-1 xx.xx.xx.xx:3306 check
server mysql-master-2 xx.xx.xx.xx:3306 check
I'd appreciate any advice you'd have in solving the 504 error with HAProxy!
I'm having a problem on a Wordpress instance I need to show to a client.
I'm using free gear. The app is not intended for production yet. Everything was working fine, I made a child-theme. Have been working on this page for about a month. Since yesterday morning I'm getting: Service Temporarily Unavailable when I try to access the page. It isn't the first time this happens, but it never lasted more then a few hours. Now it has been almost 48 hours. I need to show her the demo but I can't make it work.
Here is the output of the tail command:
C:\Users\Joao Paulo\Projetos\GibbInsurance\sources\demo>rhc tail demo
DL is deprecated, please use Fiddle
==> app-root/logs/mysql.log <==
140820 22:03:29 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: ready for connec
tions.
Version: '5.5.37' socket: '/var/lib/openshift/539c92755973caa1f000044c/mysql//s
ocket/mysql.sock' port: 3306 MySQL Community Server (GPL)
140823 18:05:36 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: Normal shutdown
140823 18:05:36 [Note] Event Scheduler: Purging the queue. 0 events
140823 18:05:36 InnoDB: Starting shutdown...
140823 18:05:39 InnoDB: Shutdown completed; log sequence number 9866622
140823 18:05:39 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: Shutdown complet
e
chown: changing ownership of `/var/lib/openshift/539c92755973caa1f000044c/mysql/
/stdout.err': Operation not permitted
140823 18:05:39 mysqld_safe mysqld from pid file /var/lib/openshift/539c92755973
caa1f000044c/mysql/pid/mysql.pid ended
140823 18:05:39 mysqld_safe mysqld from pid file /var/lib/openshift/539c92755973
caa1f000044c/mysql/pid/mysql.pid ended
==> app-root/logs/php.log <==
10.6.135.27 - - [23/Aug/2014:16:10:37 -0400] "HEAD / HTTP/1.1" 200 - "-" "Ruby"
10.6.135.27 - - [23/Aug/2014:16:10:37 -0400] "HEAD / HTTP/1.1" 200 - "-" "Ruby"
10.6.135.27 - - [23/Aug/2014:17:10:38 -0400] "POST /wp-cron.php?doing_wp_cron=14
08828238.7940719127655029296875 HTTP/1.1" 200 - "-" "WordPress/3.9.2; http://dem
o-gibbinsurance.rhcloud.com"
10.6.135.27 - - [23/Aug/2014:17:10:38 -0400] "HEAD / HTTP/1.1" 200 - "-" "Ruby"
10.6.135.27 - - [23/Aug/2014:17:10:39 -0400] "HEAD / HTTP/1.1" 200 - "-" "Ruby"
- - - [23/Aug/2014:17:10:41 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15
(Red Hat) (internal dummy connection)"
- - - [23/Aug/2014:18:05:41 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15
(Red Hat) (internal dummy connection)"
- - - [23/Aug/2014:18:05:41 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15
(Red Hat) (internal dummy connection)"
- - - [23/Aug/2014:18:05:41 -0400] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15
(Red Hat) (internal dummy connection)"
[Sat Aug 23 18:05:41 2014] [notice] caught SIGWINCH, shutting down gracefully
Interrupted
Terminate batch job (Y/N)? Y
When I try to restart the server, this is what I'm getting:
C:\Users\Joao Paulo\Projetos\GibbInsurance\sources\demo>rhc app restart -a demo
DL is deprecated, please use Fiddle
Failed to execute: 'control restart' for
/var/lib/openshift/539c92755973caa1f000044c/mysql
Failed to execute: 'control restart' for
/var/lib/openshift/539c92755973caa1f000044c/php
I appreciate any help.
Thank's a lot!
Try doing a force-stop on your application, and then a start and see if that helps. You should also try checking your quota and make sure that you are not out of disk space on your gear.