Airflow MySql Aborted connection - airflow

We're seeing a lot of connection warnings as well with Airflow 1.10.9 and Mariadb: 10.4.15. I saw this post - https://issues.apache.org/jira/browse/AIRFLOW-4848 but it doesn't give any specific answers.
2021-10-11 16:48:03 63580 [Warning] Aborted connection 63580 to db: 'airflow' user: 'blabla' host: 'airflow-scheduler-prod-64d6cf8879-gb4lz.airflow-scheduler-pr' (Got an error reading communication packets)
2021-10-11 16:48:03 63578 [Warning] Aborted connection 63578 to db: 'airflow' user: 'blabla' host: 'airflow-scheduler-prod-64d6cf8879-gb4lz.airflow-scheduler-pr' (Got an error reading communication packets)
2021-10-11 16:48:03 63581 [Warning] Aborted connection 63581 to db: 'airflow' user: 'blabla' host: 'airflow-scheduler-prod-64d6cf8879-gb4lz.airflow-scheduler-pr' (Got an error reading communication packets)

Related

Problems using MariaDB in Docker Swarm with NFS

I have problems using MariaDB within a Docker Swarm using an nfs share. The database suddenly stops accepting new connections after fdatasync() failed. This happens randomly. Aftera few hours or after a few days. If I remove the service and start it again, everything ist running fine. The service seems not to repair itself. But I think this error should not even occur, even if the service should heal itself. I run the database as a persistence layer for the nextcloud app.
This is my docker-compose file:
version: '3.3'
services:
nextcloud_db:
image: mariadb:10.7.4
#container_name: nextcloud-db
command:
- "--transaction-isolation=READ-COMMITTED"
- "--log-bin=ROW"
- "--innodb_read_only_compressed=OFF"
- "--character-set-server=utf8mb4"
- "--collation-server=utf8mb4_unicode_ci"
#- "--innodb-rollback-on-timeout=ON" # Tested this but did not help
deploy:
replicas: 1
update_config:
parallelism: 1
delay: 10s
restart_policy:
condition: on-failure
labels:
- traefik.enable=false
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
- db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=myrootpassword
- MYSQL_PASSWORD=mymysqlpassword
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_INITDB_SKIP_TZINFO=1
networks:
- nextcloud
### other services for running nextcloud ###
volumes:
db:
driver_opts:
type: "nfs"
o: "addr=<storage-server-ip>,nolock,soft,rw"
device: ":/mnt/storage/nextcloud/db"
networks:
traefik-public:
external: true
nextcloud:
driver: overlay
# driver_opts:
# encrypted: "true"
These are the logs from the moment the db died:
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 19:51:17 4671 [ERROR] [FATAL] InnoDB: fdatasync() returned 5
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 220629 19:51:17 [ERROR] mysqld got signal 6 ;
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | This could be because you hit a bug. It is also possible that this binary
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | or one of the libraries it was linked against is corrupt, improperly built,
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | or misconfigured. This error can also be caused by malfunctioning hardware.
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | To report this bug, see https://mariadb.com/kb/en/reporting-bugs
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | We will try our best to scrape up some info that will hopefully help
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | diagnose the problem, but since we have already crashed,
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | something is definitely wrong and this may fail.
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Server version: 10.7.4-MariaDB-1:10.7.4+maria~focal-log
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | key_buffer_size=134217728
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | read_buffer_size=131072
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | max_used_connections=10
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | max_threads=153
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | thread_count=11
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | It is possible that mysqld could use up to
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467995 K bytes of memory
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Hope that's ok; if not, decrease some variables in the equation.
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 |
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Thread pointer: 0x55d81db99108
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | Attempting backtrace. You can use the following information to find out
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | where mysqld died. If you see no messages after this, something went
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | terribly wrong...
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | stack_bottom = 0x7fcf10137d98 thread_stack 0x49000
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | mariadbd(my_print_stacktrace+0x32)[0x55d81b24de52]
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | mariadbd(handle_fatal_signal+0x485)[0x55d81ad282b5]
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 21:49:49 4673 [Warning] Aborted connection 4673 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 21:49:49 4672 [Warning] Aborted connection 4672 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 21:49:49 4674 [Warning] Aborted connection 4674 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 22:16:02 4676 [Warning] Aborted connection 4676 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 22:18:13 4678 [Warning] Aborted connection 4678 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-06-29 22:24:46 4679 [Warning] Aborted connection 4679 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
nc_nextcloud_db.1.1mfx9xkwd1sd#v220210169548138574 | 2022-07-01 21:49:02 7148 [Warning] Aborted connection 7148 to db: 'nextcloud' user: 'nextcloud' host: '10.0.7.189' (Got an error reading communication packets)
I found no other logs related to the isse.
Anyone has a clue what's going on here?
Maybe the NFS share is unavailable for a few seconds and so the database has problems reading/writing? Is it possible to self-heal the mariadb service after this error occurs? There are no other problems as long as the database service is running. I can upload and delete files etc. So it is not a permissions issue on the nfs share.
Further MariaDB metrics:
https://jpst.it/2TX-F
Host system info:
Docker node VM with Ubuntu:
Ubuntu 20.04.4 LTS
2 vCPUs
8 GB RAM
160 GB SSD System-Storage (Raid 10)
MySQL does not support mounting NFS to initialize data

Innodb cluster - add instance error - the server is not configured properly

I am trying to add instance to my innodb cluster. I turned off my firewall on both hosts (local vms).
This is the output of the cluster.addInstance('root#innodb-2:3306'); command:
ERROR: Unable to start Group Replication for instance 'innodb-2:3306'.
The MySQL error_log contains the following messages:
2021-10-15 19:13:31.922188 [System] [MY-013587] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2021-10-15 19:13:47.112664 [System] [MY-010597] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2021-10-15 19:14:13.496929 [Error] [MY-011735] Plugin group_replication reported: '[GCS] Error on opening a connection to innodb-1:33061 on local port: 33061.'
2021-10-15 19:14:23.498488 [Error] [MY-011735] Plugin group_replication reported: '[GCS] Error on opening a connection to innodb-1:33061 on local port: 33061.'
2021-10-15 19:14:33.508595 [Error] [MY-011735] Plugin group_replication reported: '[GCS] Error on opening a connection to innodb-1:33061 on local port: 33061.'
2021-10-15 19:14:43.520173 [Error] [MY-011735] Plugin group_replication reported: '[GCS] Error on opening a connection to innodb-1:33061 on local port: 33061.'
2021-10-15 19:14:47.373813 [Error] [MY-011640] Plugin group_replication reported: 'Timeout on wait for view after joining group'
2021-10-15 19:14:47.373884 [Error] [MY-011735] Plugin group_replication reported: '[GCS] The member is already leaving or joining a group.'
Cluster.addInstance: Group Replication failed to start: MySQL Error 3092 (HY000): innodb-2:3306: The server is not configured properly to be an active member of the group. Please see more details on error log. (RuntimeError)
I've done dba.configureInstance('root#innodb-2:3306') beforehand.
What version of MySQL and MySQL Shell are you using?
This seems like a connectivity issue, please ensure the allowlist is set if needed: https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-innodb-cluster-securing.html#create-allowlist-servers

PHPMailer: Error number 2. "Error notice: stream_socket_client()

I am getting the following error (using the WP Mail SMTP Plugin).
Any idea how to resolve this?
(All ports, passwords, etc are correct)
Thanks
Versions:
WordPress: 4.9.1
WordPress MS: No
PHP: 5.6.31-4+wpengine10
WP Mail SMTP: 1.2.2
Params:
Mailer: smtp
Constants: No
ErrorInfo: SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting
Host: smtp.hosts.co.uk
Port: 465
SMTPSecure: ssl
SMTPAutoTLS: bool(false)
SMTPAuth: bool(true)
Server:
OpenSSL: Yes
SMTP Debug:
2018-01-15 15:54:32 Connection: opening to ssl://smtp.hosts.co.uk:465, timeout=300, options=array ()
2018-01-15 15:54:39 Connection: Failed to connect to server. Error number 2. "Error notice: stream_socket_client(): unable to connect to ssl://smtp.hosts.co.uk:465 (Connection timed out)
2018-01-15 15:54:39 SMTP ERROR: Failed to connect to server: Connection timed out (110)
2018-01-15 15:54:39 SMTP connect() failed. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting
It looks like your ISP is blocking outbound SMTP. How to diagnose this is covered in the troubleshooting guide linked from the error message.
I can get to your host on that port, so it's definitely an issue with your network, not your code. You should ask your ISP or read their support docs about how they want you to send email.

Realm - Failed to connect to endpoint ':::9080': Connection refused

I am getting the below error when starting the Realm object server. I am running the server on Mac.
Realm Mobile Platform version is 1.8.1
2017-07-08T09:48:06.362Z - info: Logging to console at level 'info'.
2017-07-08T09:48:06.438Z - info: Realm Object Server sync engine listening on 127.0.0.1:27800.
2017-07-08T09:48:06.484Z - info: permission: Seed permission-Realms
2017-07-08T09:48:06.496Z - info: Realm Object Server web server listening on 127.0.0.1:27080.
2017-07-08T09:48:06.498Z - info: http proxy listening on :::9080.
2017-07-08T09:48:06.503Z - info: client: Opening Realm file: /Users/vkuppusamy/Documents/realm-mobile-platform/realm-object-server/object-server/root_dir/internal_data/auth.realm
2017-07-08T09:48:06.503Z - info: client: Connection[1]: Session[1]: Starting session for '/Users/vkuppusamy/Documents/realm-mobile-platform/realm-object-server/object-server/root_dir/internal_data/auth.realm'
2017-07-08T09:48:06.503Z - info: client: Connection[1]: Resolving ':::9080'
2017-07-08T09:48:06.503Z - info: client: Connection[1]: Connecting to endpoint ':::9080' (1/1)
2017-07-08T09:48:06.503Z - error: client: Connection[1]: Failed to connect to endpoint ':::9080': Connection refused
2017-07-08T09:48:06.503Z - error: client: Connection[1]: Failed to connect to ':::9080': All endpoints failed
2017-07-08T09:48:06.504Z - info: client: Opening Realm file: /Users/vkuppusamy/Documents/realm-mobile-platform/realm-object-server/object-server/realm-object-server/listener/__admin.realm
2017-07-08T09:48:06.504Z - info: client: Connection[2]: Session[2]: Starting session for '/Users/vkuppusamy/Documents/realm-mobile-platform/realm-object-server/object-server/realm-object-server/listener/_
You can ignore that - it's likely due to clients trying to connect before the server has initialized completely.

phpmailer network is unreachable

Some of the times trying to send email through php web application I'm getting "smtp connection failed", and some other times it works perfectly without throwing any exceptions.
The app is under the same configuration as always ( "smtp.gmail.com", port: 587, protocol: TLS ). I beleive that the error it's triggered by a timeout as the gmail server does not response. I've noticed this irregular behaviour since last week or so. Any clues?
This is the response with "SMTPDebug = 3" enabled:
2017-02-02 16:19:36 Connection: opening to smtp.gmail.com:587, timeout=300, options=array (
)
2017-02-02 16:20:18 SMTP ERROR: Failed to connect to server: Network is unreachable (101)
2017-02-02 16:20:18 Connection: opening to smtp.gmail.com:587, timeout=300, options=array (
)
2017-02-02 16:21:02 SMTP ERROR: Failed to connect to server: Network is unreachable (101)
I've also tried adding "tls://" to the host address smtp.gmail.com, as suggested in other tread with no success...
thanks in advance

Resources