I have problems using MariaDB within a Docker Swarm using an nfs share. The database suddenly stops accepting new connections after fdatasync() failed. This happens randomly. Aftera few hours or after a few days. If I remove the service and start it again, everything ist running fine. The service seems not to repair itself. But I think this error should not even occur, even if the service should heal itself. I run the database as a persistence layer for the nextcloud app.
This is my docker-compose file:
version: '3.3'
services:
nextcloud_db:
image: mariadb:10.7.4
#container_name: nextcloud-db
command:
- "--transaction-isolation=READ-COMMITTED"
- "--log-bin=ROW"
- "--innodb_read_only_compressed=OFF"
- "--character-set-server=utf8mb4"
- "--collation-server=utf8mb4_unicode_ci"
#- "--innodb-rollback-on-timeout=ON" # Tested this but did not help
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: on-failure
labels:
- traefik.enable=false
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=myrootpassword
- MYSQL_PASSWORD=mymysqlpassword
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_INITDB_SKIP_TZINFO=1
networks:
- nextcloud
### other services for running nextcloud ###
volumes:
db:
driver_opts:
type: "nfs"
o: "addr=<storage-server-ip>,nolock,soft,rw"
device: ":/mnt/storage/nextcloud/db"
networks:
traefik-public:
external: true
nextcloud:
driver: overlay
# driver_opts:
# encrypted: "true"
These are the logs from the moment the db died:
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 19:51:17 4671 [ERROR] [FATAL] InnoDB: fdatasync() returned 5
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 220629 19:51:17 [ERROR] mysqld got signal 6 ;
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | This could be because you hit a bug. It is also possible that this binary
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | or one of the libraries it was linked against is corrupt, improperly built,
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | or misconfigured. This error can also be caused by malfunctioning hardware.
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | To report this bug, see https://mariadb.com/kb/en/reporting-bugs
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | We will try our best to scrape up some info that will hopefully help
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | diagnose the problem, but since we have already crashed,
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | something is definitely wrong and this may fail.
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Server version: 10.7.4-MariaDB-1:10.7.4+maria~focal-log
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | key_buffer_size=134217728
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | read_buffer_size=131072
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | max_used_connections=10
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | max_threads=153
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | thread_count=11
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | It is possible that mysqld could use up to
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467995 K bytes of memory
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Hope that's ok; if not, decrease some variables in the equation.
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Thread pointer: 0x55d81db99108
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Attempting backtrace. You can use the following information to find out
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | where mysqld died. If you see no messages after this, something went
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | terribly wrong...
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | stack_bottom = 0x7fcf10137d98 thread_stack 0x49000
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | mariadbd(my_print_stacktrace+0x32)[0x55d81b24de52]
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | mariadbd(handle_fatal_signal+0x485)[0x55d81ad282b5]
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 21:49:49 4673 [Warning] Aborted connection 4673 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 21:49:49 4672 [Warning] Aborted connection 4672 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 21:49:49 4674 [Warning] Aborted connection 4674 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 22:16:02 4676 [Warning] Aborted connection 4676 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 22:18:13 4678 [Warning] Aborted connection 4678 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 22:24:46 4679 [Warning] Aborted connection 4679 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-07-01 21:49:02 7148 [Warning] Aborted connection 7148 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
I found no other logs related to the isse.
Anyone has a clue what's going on here?
Maybe the NFS share is unavailable for a few seconds and so the database has problems reading/writing? Is it possible to self-heal the mariadb service after this error occurs? There are no other problems as long as the database service is running. I can upload and delete files etc. So it is not a permissions issue on the nfs share.
Further MariaDB metrics:
https://jpst.it/2TX-F
Host system info:
Docker node VM with Ubuntu:
Ubuntu 20.04.4 LTS
2 vCPUs
8 GB RAM
160 GB SSD System-Storage (Raid 10)
MySQL does not support mounting NFS to initialize data
Related
I have a keycloak server deployed with docker behind a nginx reverse proxy.
In dev mode all is working fine, but since I am trying to deploy it in prod I can acces the app but when I want to config my server from the admin console I'm stuck in this page :
Docker-compose
version: '3'
services:
postgresql:
image: postgres:latest
command: postgres -c 'max_connections=200'
networks:
- keycloak-net
- postgresql-net
restart: always
environment:
- POSTGRES_USER=keycloak
- POSTGRES_PASSWORD=password
- POSTGRES_DB=keycloak
volumes:
- ./keycloak/postgresql:/var/lib/postgresql/data
keycloak:
image: quay.io/keycloak/keycloak:20.0.3
networks:
- proxy-net
- keycloak-net
restart: always
command: start --spi-truststore-file-file /etc/x509/https/truststore.jks --spi-truststore-file-password password --spi-truststore-file-hostname-verification-policy ANY
environment:
- KC_HOSTNAME=keycloak
- KC_HOSTNAME_STRICT=false
- KC_PROXY=edge
- KC_HOSTNAME_STRICT_HTTPS=false
- KC_DB=postgres
- KC_DB_URL_HOST=postgresql
- KC_DB_URL_DATABASE=keycloak
- KC_DB_PASSWORD=password
- KC_DB_USERNAME=keycloak
- KC_DB_SCHEMA=public
- KEYCLOAK_ADMIN=admin
- KEYCLOAK_ADMIN_PASSWORD=password
depends_on:
- postgresql
volumes:
- ./keycloak/stores:/etc/x509/https
networks:
proxy-net:
external: true
keycloak-net:
driver: bridge
postgresql-net:
driver: bridge
Keycloak logs on startup
keycloak_1 | Next time you run the server, just run:
keycloak_1 |
keycloak_1 | kc.sh start --optimized --spi-truststore-file-file=/etc/x509/https/truststore.jks --spi-truststore-file-password=password --spi-truststore-file-hostname-verification-policy=ANY
keycloak_1 |
keycloak_1 | 2023-01-19 18:15:53,562 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: true
keycloak_1 | 2023-01-19 18:15:57,778 WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak_1 | 2023-01-19 18:16:00,549 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
keycloak_1 | 2023-01-19 18:16:00,555 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
keycloak_1 | 2023-01-19 18:16:00,658 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak_1 | 2023-01-19 18:16:00,742 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak_1 | 2023-01-19 18:16:01,352 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
keycloak_1 | 2023-01-19 18:16:01,673 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
keycloak_1 | 2023-01-19 18:16:01,675 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
keycloak_1 | 2023-01-19 18:16:01,866 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
keycloak_1 | 2023-01-19 18:16:01,869 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 212.99KB
keycloak_1 | 2023-01-19 18:16:01,870 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
keycloak_1 | 2023-01-19 18:16:01,872 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 212.99KB
keycloak_1 | 2023-01-19 18:16:03,900 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) 4b41e052f453-15754: no members discovered after 2003 ms: creating cluster as coordinator
keycloak_1 | 2023-01-19 18:16:03,933 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [4b41e052f453-15754|0] (1) [4b41e052f453-15754]
keycloak_1 | 2023-01-19 18:16:03,947 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `4b41e052f453-15754`, physical addresses are `[192.168.16.3:54114]`
keycloak_1 | 2023-01-19 18:16:05,038 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: 4b41e052f453-15754, Site name: null
keycloak_1 | 2023-01-19 18:16:06,893 INFO [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 17.483s. Listening on: http://0.0.0.0:8080
keycloak_1 | 2023-01-19 18:16:06,897 INFO [io.quarkus] (main) Profile prod activated.
keycloak_1 | 2023-01-19 18:16:06,899 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
keycloak_1 | 2023-01-19 18:16:07,033 ERROR [org.keycloak.services] (main) KC-SERVICES0010: Failed to add user 'admin' to realm 'master': user with username exists
Note that the error at the end append beacause my db has already an admin.
I have the same issue when strating from an empty db.
nginx.conf
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name myserver.fr;
ssl_certificate /etc/nginx/ssl/live/myserver.fr/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/live/myserver.fr/privkey.pem;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
location / {
proxy_pass http://keycloak:8080;
}
}
Network console web browser
Thanks in advance for your help !
I have tried a lot of configuration, I saw a lot of time that adding KC_HOSTNAME_STRICT_HTTPS=false on env variable is the solution. I tried but it does not work..
With the help of #stdunbar I was able to solve this.
I had set KC_HOSTNAME to keycloak which is the hostname of the container in my docker network.
To solve my problem I set the KC_HOSTNAME to the real hostname accessible from browser : keycloak.mydomain.fr
At the startup of ActiveMQ we're facing this issue :
2022-07-18 14:02:48,968 | WARN | Transport Connection to: tcp://127.0.0.1:58090 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58090#61616
2022-07-18 14:02:48,972 | WARN | Transport Connection to: tcp://127.0.0.1:58094 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58094#61616
2022-07-18 14:02:48,976 | WARN | Transport Connection to: tcp://127.0.0.1:58098 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58098#61616
2022-07-18 14:02:48,983 | WARN | Transport Connection to: tcp://127.0.0.1:58102 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58102#61616
2022-07-18 14:02:48,994 | WARN | Transport Connection to: tcp://127.0.0.1:58106 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58106#61616
2022-07-18 14:02:49,009 | WARN | Transport Connection to: tcp://127.0.0.1:58114 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58114#61616
2022-07-18 14:02:49,009 | WARN | Transport Connection to: tcp://127.0.0.1:58110 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///127.0.0.1:58110#61616
We thought that they are related to disk space as it was over 90%, but even after expanding the disk space we're still facing the same issue!
OS: Ubuntu
Update:
we're using the PIM solution contentserv from Atlassian
These are WARN messages that have no impact on message flow. These messages indicate some clients are disconnecting without fully cleaning up resources and calling .close(). Check with your app teams' code to make sure they are not leaking connections, or not properly closing resources.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
Welcome,
Im trying to install Wordpress on kubernetes. I installed chart and I typed:
"helm install projectname-wordpress bitnami/wordpress --set allowOverrideNone=true"
but even it giving output its not working and I cant login into.
When I typed "kubectl describe pods"
I get output like this:
Name: projectname-wordpress-785d4c4c84-xzt6m
Namespace: default
Priority: 0
Node: skalowalne-node-73a107/59.813.226.646
Start Time: Fri, 28 May 2021 02:00:35 +0200
Labels: app.kubernetes.io/instance=projectname-wordpress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=wordpress
helm.sh/chart=wordpress-11.0.10
pod-template-hash=785d4c4c84
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/projectname-wordpress-785d4c4c84
Containers:
wordpress:
Container ID:
Image: docker.io/bitnami/wordpress:5.7.2-debian-10-r9
Image ID:
Ports: 8080/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 300m
memory: 512Mi
Liveness: http-get http://:http/wp-admin/install.php delay=120s timeout=5s period=10s #success=1 #failure=6
Readiness: http-get http://:http/wp-login.php delay=30s timeout=5s period=10s #success=1 #failure=6
Environment:
ALLOW_EMPTY_PASSWORD: yes
MARIADB_HOST: projectname-wordpress-mariadb
MARIADB_PORT_NUMBER: 3306
WORDPRESS_DATABASE_NAME: bitnami_wordpress
WORDPRESS_DATABASE_USER: bn_wordpress
WORDPRESS_DATABASE_PASSWORD: <set to the key 'mariadb-password' in secret 'projectname-wordpress-mariadb'> Optional: false
WORDPRESS_USERNAME: user
WORDPRESS_PASSWORD: <set to the key 'wordpress-password' in secret 'projectname-wordpress'> Optional: false
WORDPRESS_EMAIL: user#example.com
WORDPRESS_FIRST_NAME: FirstName
WORDPRESS_LAST_NAME: LastName
WORDPRESS_HTACCESS_OVERRIDE_NONE: no
WORDPRESS_ENABLE_HTACCESS_PERSISTENCE: no
WORDPRESS_BLOG_NAME: User's Blog!
WORDPRESS_SKIP_BOOTSTRAP: no
WORDPRESS_TABLE_PREFIX: wp_
WORDPRESS_SCHEME: http
WORDPRESS_EXTRA_WP_CONFIG_CONTENT:
WORDPRESS_AUTO_UPDATE_LEVEL: none
WORDPRESS_PLUGINS: none
Mounts:
/bitnami/wordpress from wordpress-data (rw,path="wordpress")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mxtw7 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
wordpress-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: projectname-wordpress
ReadOnly: false
default-token-mxtw7:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mxtw7
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 15m (x181 over 9h) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data default-token-mxtw7]: timed out waiting for the condition
Warning FailedMount 3m49s (x58 over 8h) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[default-token-mxtw7 wordpress-data]: timed out waiting for the condition
Warning FailedAttachVolume 2m40s (x139 over 9h) attachdetach-controller AttachVolume.Attach failed for volume "ovh-managed-kubernetes-do2ymc-pvc-80079ec2-e6f9-4210-852e-04fa286f714c" : attachdetachment timeout for volume 3b160677-40e8-4170-9cc3-cdd58e230942
Name: projectname-wordpress-mariadb-0
Namespace: default
Priority: 0
Node: skalowalne-node-f1da93/59.83.226.180
Start Time: Fri, 28 May 2021 02:00:27 +0200
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=projectname-wordpress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mariadb
controller-revision-hash=projectname-wordpress-mariadb-85d4cb8f7
helm.sh/chart=mariadb-9.3.11
statefulset.kubernetes.io/pod-name=projectname-wordpress-mariadb-0
Annotations: checksum/configuration: 878384c0d68b5abc46d5d5d719a9e83aa911941710552c3dfcebd48203ce5d9f
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/projectname-wordpress-mariadb
Containers:
mariadb:
Container ID:
Image: docker.io/bitnami/mariadb:10.5.10-debian-10-r0
Image ID:
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: exec [/bin/bash -ec password_aux="${MARIADB_ROOT_PASSWORD:-}"
if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=120s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -ec password_aux="${MARIADB_ROOT_PASSWORD:-}"
if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
BITNAMI_DEBUG: false
MARIADB_ROOT_PASSWORD: <set to the key 'mariadb-root-password' in secret 'projectname-wordpress-mariadb'> Optional: false
MARIADB_USER: bn_wordpress
MARIADB_PASSWORD: <set to the key 'mariadb-password' in secret 'projectname-wordpress-mariadb'> Optional: false
MARIADB_DATABASE: bitnami_wordpress
Mounts:
/bitnami/mariadb from data (rw)
/opt/bitnami/mariadb/conf/my.cnf from config (rw,path="my.cnf")
/var/run/secrets/kubernetes.io/serviceaccount from projectname-wordpress-mariadb-token-92mm2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-projectname-wordpress-mariadb-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: projectname-wordpress-mariadb
Optional: false
projectname-wordpress-mariadb-token-92mm2:
Type: Secret (a volume populated by a Secret)
SecretName: projectname-wordpress-mariadb-token-92mm2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 19m (x41 over 8h) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[config projectname-wordpress-mariadb-token-92mm2 data]: timed out waiting for the condition
Warning FailedMount 9m51s (x36 over 8h) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[projectname-wordpress-mariadb-token-92mm2 data config]: timed out waiting for the condition
Warning FailedMount 5m21s (x161 over 9h) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data config projectname-wordpress-mariadb-token-92mm2]: timed out waiting for the condition
Warning FailedAttachVolume 2m48s (x139 over 9h) attachdetach-controller AttachVolume.Attach failed for volume "ovh-managed-kubernetes-do2ymc-pvc-fad9b535-f6d5-4e71-9e47-3a555936c546" : attachdetachment timeout for volume d96dbb2d-2200-48bd-940d-74dc0c3b5128
UPDATE: I dont have firewall enabled on cloud machine. Im using ovh services.
What should I do to make it working?
Failed events looks exacly like this just after try to deploy wordpress:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5m19s default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Warning FailedScheduling 5m19s default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 5m12s default-scheduler Successfully assigned default/projectname-wordpress-5466b7b45c-rzx9h to standard-node-fe7236
Warning FailedMount 3m10s kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[default-token-mxtw7 wordpress-data]: timed out waiting for the condition
Warning FailedAttachVolume 72s (x2 over 3m13s) attachdetach-controller AttachVolume.Attach failed for volume "ovh-managed-kubernetes-do2ymc-pvc-3e3686eb-6cf5-4697-99b0-0689bbd7d0a9" : attachdetachment timeout for volume f8b78a8d-f0d8-4dcb-bcae-ec84fb7d82e4
Warning FailedMount 56s kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data default-token-mxtw7]: timed out waiting for the condition
Logs from first pod
mariadb 16:49:02.01 mariadb 16:49:02.01 Welcome to the Bitnami mariadb
container mariadb 16:49:02.01 Subscribe to project updates by watching
https://github.com/bitnami/bitnami-docker-mariadb mariadb 16:49:02.02
Submit issues and feature requests at
https://github.com/bitnami/bitnami-docker-mariadb/issues mariadb
16:49:02.02 Send us your feedback at containers#bitnami.com mariadb
16:49:02.02 mariadb 16:49:02.02 INFO ==> ** Starting MariaDB setup **
mariadb 16:49:02.07 INFO ==> Validating settings in MYSQL_/MARIADB_
env vars mariadb 16:49:02.07 INFO ==> Initializing mariadb database
mariadb 16:49:02.09 INFO ==> Using persisted data mariadb 16:49:02.10
INFO ==> Running mysql_upgrade mariadb 16:49:02.10 INFO ==> Starting
mariadb in background mariadb 16:49:03.14 INFO ==> Stopping mariadb
Logs from second pod
Welcome to the Bitnami wordpress container Subscribe to project
updates by watching
https://github.com/bitnami/bitnami-docker-wordpress Submit issues and
feature requests at
https://github.com/bitnami/bitnami-docker-wordpress/issues Send us
your feedback at containers#bitnami.com
WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes.
For safety reasons, do not use this flag in a production environment.
nami INFO Initializing apache nami INFO apache successfully
initialized nami INFO Initializing php nami INFO php
successfully initialized nami INFO Initializing mysql-client nami
INFO mysql-client successfully initialized nami INFO Initializing
wordpress wordpre INFO ==> Preparing Varnish environment wordpre INFO
==> Preparing Apache environment wordpre INFO ==> Preparing PHP environment mysql-c INFO Trying to connect to MySQL server Error
executing 'postInstallation': Failed to connect to
student-mariadb:3306 after 36 tries
I reset kubernates and same code started to work. I made everything same so I dont know how. Thank you guys for support.
Can someone please help me why this call is no longer working in Apache Karaf 3.0.2. I verified that it was working in version 3.0.1. All instances are up and running, but I am unable to connect to one of my instances directly from the command line.
su - karaf -c " client -h localhost -a 8101 -u karaf -r 50 -d 2 \" instance:connect -u karaf -p karaf test1 \\\" feature:repo-list \\\" \" "
Logging in as karaf
455 [sshd-SshClient[bea319b]-nio2-thread-1] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, b6:f6:d6:3f:8b:2f:ad:a4:0f:3f:3d:c3:7b:96:fd:ae] presented unverified {} key: {}
Connecting to host localhost on port 8103
Connecting to unknown server. Automatically adding to known hosts.
Storing the server key in known_hosts.
Error executing command: Authentication failed
The call is part of an automated process and I cannot connect to a specific instance directly. Is there any specific configuration required, that was not necessary in 3.0.1?
UPDATE #1:
I have added the verbose option... Does it give you any hints what to do?
client -v -h localhost -a 8101 -u karaf -r 50 -d 2 " instance:connect -u karaf test1 \" feature:repo-list \" "
39 [main] INFO org.apache.sshd.common.util.SecurityUtils - BouncyCastle not registered, using the default JCE provider
Logging in as karaf
367 [sshd-SshClient[bea319b]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Client session created
380 [main] INFO org.apache.sshd.client.session.ClientSessionImpl - Start flagging packets as pending until key exchange is done
383 [sshd-SshClient[bea319b]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Server version string: SSH-2.0-SSHD-CORE-0.12.0
384 [sshd-SshClient[bea319b]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Kex: server->client [aes128-ctr, hmac-sha1, none] {} {}
384 [sshd-SshClient[bea319b]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Kex: client->server [aes128-ctr, hmac-sha1, none] {} {}
444 [sshd-SshClient[bea319b]-nio2-thread-1] WARN org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier - Server at [localhost/127.0.0.1:8101, DSA, 22:8b:f8:9d:bc:c6:40:d8:fe:52:aa:90:c0:f2:70:ec] presented unverified {} key: {}
457 [sshd-SshClient[bea319b]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientSessionImpl - Dequeing pending packets
524 [sshd-SshClient[bea319b]-nio2-thread-1] INFO org.apache.sshd.client.session.ClientUserAuthServiceNew - Received SSH_MSG_USERAUTH_FAILURE
568 [sshd-SshClient[bea319b]-nio2-thread-2] INFO org.apache.sshd.client.session.ClientUserAuthServiceNew - Received SSH_MSG_USERAUTH_SUCCESS
Connecting to host localhost on port 8102
Error executing command: Authentication failed
UPDATE #2:
I switched the logger to DEBUG and I found this exception:
2015-01-15 11:28:48,920 | DEBUG | 5]-nio2-thread-1 | ClientSessionImpl | 28 - org.apache.sshd.core - 0.12.0 | Received SSH_MSG_SERVICE_ACCEPT
2015-01-15 11:28:48,920 | INFO | 5]-nio2-thread-1 | ClientUserAuthServiceNew | 28 - org.apache.sshd.core - 0.12.0 | Received SSH_MSG_USERAUTH_FAILURE
2015-01-15 11:28:48,920 | DEBUG | 5]-nio2-thread-1 | ClientUserAuthServiceNew | 28 - org.apache.sshd.core - 0.12.0 | Authentications that can continue: keyboard-interactive, password, publickey
2015-01-15 11:28:48,922 | DEBUG | 5]-nio2-thread-1 | Nio2Session | 28 - org.apache.sshd.core - 0.12.0 | Caught exception, now calling handler
2015-01-15 11:28:48,922 | WARN | 5]-nio2-thread-1 | ClientSessionImpl | 28 - org.apache.sshd.core - 0.12.0 | Exception caught
java.lang.IllegalStateException: No SSH_AUTH_SOCK environment variable set
at org.apache.karaf.shell.ssh.KarafAgentFactory.createClient(KarafAgentFactory.java:71)
at org.apache.sshd.client.auth.UserAuthPublicKey.init(UserAuthPublicKey.java:78)
at org.apache.sshd.client.session.ClientUserAuthServiceNew.tryNext(ClientUserAuthServiceNew.java:212)
at org.apache.sshd.client.session.ClientUserAuthServiceNew.processUserAuth(ClientUserAuthServiceNew.java:178)
at org.apache.sshd.client.session.ClientUserAuthServiceNew.process(ClientUserAuthServiceNew.java:131)
at org.apache.sshd.client.session.ClientUserAuthService.process(ClientUserAuthService.java:80)
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:399)
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:295)
at org.apache.sshd.client.session.ClientSessionImpl.handleMessage(ClientSessionImpl.java:256)
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:731)
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:277)
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:187)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:173)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[28:org.apache.sshd.core:0.12.0]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.7.0_65]
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.7.0_65]
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:275)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:296)[:1.7.0_65]
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:407)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:173)[28:org.apache.sshd.core:0.12.0]
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:189)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:173)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[28:org.apache.sshd.core:0.12.0]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.7.0_65]
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.7.0_65]
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:275)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:296)[:1.7.0_65]
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:407)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:173)[28:org.apache.sshd.core:0.12.0]
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:189)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:173)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[28:org.apache.sshd.core:0.12.0]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.7.0_65]
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.7.0_65]
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:275)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:296)[:1.7.0_65]
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:407)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:173)[28:org.apache.sshd.core:0.12.0]
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:189)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:173)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[28:org.apache.sshd.core:0.12.0]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.7.0_65]
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)[:1.7.0_65]
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:275)[:1.7.0_65]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:296)[:1.7.0_65]
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:407)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2Session.startReading(Nio2Session.java:173)[28:org.apache.sshd.core:0.12.0]
at org.apache.sshd.common.io.nio2.Nio2Connector$1.onCompleted(Nio2Connector.java:53)[28:org.apache.sshd.core:0.12.0]
at org.apache.sshd.common.io.nio2.Nio2Connector$1.onCompleted(Nio2Connector.java:46)[28:org.apache.sshd.core:0.12.0]
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_65]
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)[28:org.apache.sshd.core:0.12.0]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)[:1.7.0_65]
at sun.nio.ch.Invoker$2.run(Invoker.java:218)[:1.7.0_65]
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)[:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_65]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_65]
I'm using Centos 6.5 x86_64 to setup Openstack Havana and all services work well. But when I've rebooted the operating system, I've founded that the nova service does not work properly, the following error triggered:
nova flavor-list
ERROR: [Errno 111] Connection refused
Reviewing the log files in / var / log / nova gives the following error:
2014-03-24 12:24:04.293 6275 INFO nova.osapi_compute.wsgi.server [-] (6275) wsgi starting up
2014-03-24 12:24:04.297 6267 CRITICAL nova [-] [Errno 98] Address already in use
2014-03-24 12:24:04.412 6275 INFO nova.openstack.common.service [-] Parent process has died unexpectedly, exiting
2014-03-24 12:24:04.412 6274 INFO nova.openstack.common.service [-] Parent process has died unexpectedly, exiting
2014-03-24 12:24:04.412 6275 INFO nova.wsgi [-] Stopping WSGI server.
2014-03-24 12:24:04.412 6274 INFO nova.wsgi [-] Stopping WSGI server.
The state of my OpenStack server
nova-manage service list
Binary Host Zone Status State Updated_At
nova-cert controller internal enabled :-) 2014-03-24 14:28:03
nova-consoleauth controller internal enabled :-) 2014-03-24 14:28:01
nova-scheduler controller internal enabled :-) 2014-03-24 14:28:00
nova-conductor controller internal enabled :-) 2014-03-24 14:27:59
nova-compute controller nova enabled :-) 2014-03-24 14:28:06
nova-network controller internal enabled :-) 2014-03-24 14:27:58
keystone service-list
+----------------------------------+----------+----------+---------------------------+
| id | name | type | description |
+----------------------------------+----------+----------+---------------------------+
| 7ce108d652ee48d7897127045a371795 | cinder | volume | Cinder Volume Service |
| 9452b875328f4763b7766eb533bd75c4 | cinderv2 | volumev2 | Cinder Volume Service V2 |
| e9607d1a308140298f8364fd2a0e62a8 | glance | image | Glance Image Service |
| b7ac07f69e2e41f684d6470c69db4781 | keystone | identity | Keystone Identity Service |
| cbdfa73329094d7d94c7464b9bf0ef7d | nova | compute | Nova Compute service |
+----------------------------------+----------+----------+---------------------------+
ps -ef | grep "nova-api"
nova 2522 1 0 11:22 ? 00:00:00 /usr/bin/python /usr/bin/nova-api-metadata --logfile /var/log/nova/metadata-api.log
root 11909 6217 0 15:11 pts/1 00:00:01 gedit nova-api.log
root 12644 3832 0 15:31 pts/0 00:00:00 grep nova-api
netstat -napo | grep 877
tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 2522/python off (0.00/0/0)
Any pointers would be extremely helpful.
Thanks
firstly, i strongly recommend you to find or ask for answer on ask.openstack.org
then from what you described, it may caused by: you've enabled nova-api-metadata and nova-api service in the same time.
from the default configuration we know that: ['ec2', 'osapi_compute', 'metadata'] are enabled, see https://github.com/openstack/nova/blob/stable/havana/nova/service.py#L55
so it will start each service one by one when nova-api service is called, see https://github.com/openstack/nova/blob/stable/havana/nova/cmd/api.py#L45
since nova-api-metadata service is running, which cause the 8775 port is used, then one service launched by nova-api will die and since this exception is not caught, then the other two will die too, then you get what you see in the log
If what I've assumed is right, please cancel the nova-api-metadata service and use nova-api service only, which means 'chkconfig openstack-nova-api-metadata off; chkconfig openstack-nova-api on', i'm not sure about the specific service name on your system, but should be something like that, correct it if i'm wrong
Connection refused is a common error encountered everytime. One of the case is keystone is refusing the connection for the nova service.
make sure SERVICE_PASSWORD for nova and quantum are same while creating the keystone services.Go to quantum and nova config files and verify the SERVICE_PASSWORD are same.
Njoy!!