Airflow webserver not starting - airflow

I have installed airflow via github source. I have configured airflow with mysql metadb with local executer. When I tried to start my webserver it couldn't able to start.
install.sh
mkdir -p ~/home
export AIRFLOW_HOME=~/airflow
cd $AIRFLOW_HOME
virtualenv env
source env/bin/activate
mkdir -p /usr/local/src/
cd /usr/local/src/
git clone https://github.com/apache/incubator-airflow.git
cd incubator-airflow
git checkout tags/1.8.2
pip install -e .
pip install -e .[hive]
pip install -e .[gcp_api]
pip install -e .[mysql]
pip install -e .[password]
pip install -e .[celery]
airflow.cfg:
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /root/airflow
dags_folder = /root/airflow/dags
base_log_folder = /root/airflow/logs
encrypt_s3_logs = False
executor = LocalExecutor
sql_alchemy_conn = mysql://root:*****#localhost/airflow
when I tried to start my webserver using it shows ttou signal handling and existing worker.
airflow webserver -p 8080
[2017-11-20 04:05:30,642] {__init__.py:57} INFO - Using executor LocalExecutor
[2017-11-20 04:05:30,723] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-11-20 04:05:30,756] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:05:31,437] [3079] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
Running the Gunicorn Server with:
Workers: 8 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2017-11-20 04:05:32,074] {__init__.py:57} INFO - Using executor LocalExecutor
[2017-11-20 04:05:32,153] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-11-20 04:05:32,184] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2017-11-20 04:05:32 +0000] [3087] [INFO] Starting gunicorn 19.3.0
[2017-11-20 04:05:32 +0000] [3087] [INFO] Listening at: http://0.0.0.0:8080 (3087)
[2017-11-20 04:05:32 +0000] [3087] [INFO] Using worker: sync
[2017-11-20 04:05:32 +0000] [3098] [INFO] Booting worker with pid: 3098
[2017-11-20 04:05:32 +0000] [3099] [INFO] Booting worker with pid: 3099
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:05:32 +0000] [3100] [INFO] Booting worker with pid: 3100
[2017-11-20 04:05:32 +0000] [3101] [INFO] Booting worker with pid: 3101
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:05:32 +0000] [3102] [INFO] Booting worker with pid: 3102
[2017-11-20 04:05:32 +0000] [3103] [INFO] Booting worker with pid: 3103
[2017-11-20 04:05:32 +0000] [3104] [INFO] Booting worker with pid: 3104
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:05:32 +0000] [3105] [INFO] Booting worker with pid: 3105
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:05:33,198] [3099] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:33,312] [3098] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:33,538] [3100] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:33,863] [3101] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:33,963] [3102] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:33,987] [3104] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:34,062] [3105] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:05:34,162] [3103] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:06:05 +0000] [3087] [INFO] Handling signal: ttin
[2017-11-20 04:06:05 +0000] [3121] [INFO] Booting worker with pid: 3121
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:06:05,426] [3121] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:06:06 +0000] [3087] [INFO] Handling signal: ttou
[2017-11-20 04:06:06 +0000] [3098] [INFO] Worker exiting (pid: 3098)
[2017-11-20 04:06:36 +0000] [3087] [INFO] Handling signal: ttin
[2017-11-20 04:06:36 +0000] [3136] [INFO] Booting worker with pid: 3136
/root/env/local/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
.format(x=modname), ExtDeprecationWarning
[2017-11-20 04:06:36,818] [3136] {models.py:167} INFO - Filling up the DagBag from /root/airflow/dags
[2017-11-20 04:06:37 +0000] [3087] [INFO] Handling signal: ttou
[2017-11-20 04:06:37 +0000] [3099] [INFO] Worker exiting (pid: 3099)
[2017-11-20 04:07:07 +0000] [3087] [INFO] Handling signal: ttin
[2017-11-20 04:07:07 +0000] [3144] [INFO] Booting worker with pid: 3144

Your webserver is running fine. Workers are regularly "refreshed" at an interval set by worker_refresh_interval so that they pickup on new or updated DAGs. When this happens, you'll see the signal ttin (increase processes by one) always followed byttou (decrease processes by one), where a new worker is added before the oldest worker is removed.

Related

Wordpress on docker this site cant be reached

So I have setup wordpress on docker this is my yml file configuration:
version: '3.8'
services:
wordpress:
build:
context: .
depends_on:
mysql:
condition: service_healthy
environment:
- WORDPRESS_DEBUG=true
- WORDPRESS_DB_HOST=mysql
- WORDPRESS_DB_USER=wordpress
- WORDPRESS_DB_PASSWORD=password
volumes:
- .:/var/www/html/wp-content/plugins/plugin
ports:
- "8082:80"
mysql:
image: mysql:5.7
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_DATABASE=wordpress
- MYSQL_USER=wordpress
- MYSQL_PASSWORD=password
ports:
- "3308:3306"
healthcheck:
test: mysqladmin ping -h 127.0.0.1 -u $$MYSQL_USER --password=$$MYSQL_PASSWORD
interval: 1s
timeout: 3s
retries: 30
And this is my Dockerfile:
FROM wordpress:latest
RUN curl -o /usr/local/bin/wp https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar \
&& chmod +x /usr/local/bin/wp
ADD ./bin/ /
RUN chmod +x /*.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["apache2-foreground"]
When I run docker-compose up and go to localhost:8082 I get this This site can’t be reachedlocalhost refused to connect.
Inside wordpress docker logs I see this:
/usr/local/bin/docker-entrypoint.sh: line 99: exec: php-fpm: not found
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.19.0.3. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.19.0.3. Set the 'ServerName' directive globally to suppress this message
[Mon Oct 17 09:22:49.514781 2022] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.54 (Debian) PHP/7.4.32 configured -- resuming normal operations
[Mon Oct 17 09:22:49.514849 2022] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
172.19.0.1 - - [17/Oct/2022:09:23:42 +0000] "-" 408 0 "-" "-"
Cant find a way to fix it. Would appreciate some help.

Running nginx on gcp with docker-compose

I would like to run (to test if it works) nginx on my gcp vm instance with docker-compose.
My docker-compose.yml looks like:
version: '3'
services:
nginx:
image: nginx:latest
container_name: webserver
restart: unless-stopped
ports:
- 8080:8080
On my gcp I allow for http and https trafics.
When I start docker-compose I get:
Starting webserver ... done
Attaching to webserver
webserver | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
webserver | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
webserver | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
webserver | 10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
webserver | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
webserver | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
webserver | /docker-entrypoint.sh: Configuration complete; ready for start up
webserver | 2022/04/24 11:48:14 [notice] 1#1: using the "epoll" event method
webserver | 2022/04/24 11:48:14 [notice] 1#1: nginx/1.21.6
webserver | 2022/04/24 11:48:14 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
webserver | 2022/04/24 11:48:14 [notice] 1#1: OS: Linux 4.19.0-20-cloud-amd64
webserver | 2022/04/24 11:48:14 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
webserver | 2022/04/24 11:48:14 [notice] 1#1: start worker processes
webserver | 2022/04/24 11:48:14 [notice] 1#1: start worker process 26
And when I visit my vm's external ip: EXTERNAL_IP:8080 in web browser I don't get any information about nginx installed. What else do I need to make sure nginx works properly?
EDIT: Problem solved. From weird reson, nginx wasn't starting on 8080 port but on 49154

Nginx is not proxying Gunicorn

I'm trying to deploy my Flask project using Gunicorn and Nginx but I'm still struggling.
/etc/systemd/system/gunicorn3.service
[Unit] Description=Gunicorn service After=network.target
[Service]
User=www-data
Group=adm
WorkingDirectory=/home/project/
Environment="PATH=/home/project/env/bin"
ExecStart=/home/project/env/bin/gunicorn --workers 3 --bind unix:cima.sock -m 007 run:app
[Install] WantedBy=multi-user.target
Then I check if there's any problem, everything seems OK
>>sudo systemctl status gunicorn3
gunicorn3.service - Gunicorn service
Loaded: loaded (/etc/systemd/system/gunicorn3.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-01-17 20:25:01 UTC; 18min ago
Main PID: 18451 (gunicorn)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/gunicorn3.service
├─18451 /home/acelerathon_cima_grupo_3/chatbot-grupo-3/chatbot-cima/chatbot-cima-back/cimaenv/bin/python3 /home/acelerathon_cima_grupo_3/chatbo
├─18453 /home/acelerathon_cima_grupo_3/chatbot-grupo-3/chatbot-cima/chatbot-cima-back/cimaenv/bin/python3 /home/acelerathon_cima_grupo_3/chatbo
├─18454 /home/acelerathon_cima_grupo_3/chatbot-grupo-3/chatbot-cima/chatbot-cima-back/cimaenv/bin/python3 /home/acelerathon_cima_grupo_3/chatbo
└─18456 /home/acelerathon_cima_grupo_3/chatbot-grupo-3/chatbot-cima/chatbot-cima-back/cimaenv/bin/python3 /home/acelerathon_cima_grupo_3/chatbo
Jan 17 20:25:01 acelerathon-cima-grupo-3 systemd[1]: Started Gunicorn service.
Jan 17 20:25:01 acelerathon-cima-grupo-3 gunicorn[18451]: [2020-01-17 20:25:01 +0000] [18451] [INFO] Starting gunicorn 19.9.0
Jan 17 20:25:01 acelerathon-cima-grupo-3 gunicorn[18451]: [2020-01-17 20:25:01 +0000] [18451] [INFO] Listening at: unix:cima.sock (18451)
Jan 17 20:25:01 acelerathon-cima-grupo-3 gunicorn[18451]: [2020-01-17 20:25:01 +0000] [18451] [INFO] Using worker: sync
Jan 17 20:25:01 acelerathon-cima-grupo-3 gunicorn[18451]: [2020-01-17 20:25:01 +0000] [18453] [INFO] Booting worker with pid: 18453
Jan 17 20:25:01 acelerathon-cima-grupo-3 gunicorn[18451]: [2020-01-17 20:25:01 +0000] [18454] [INFO] Booting worker with pid: 18454
Jan 17 20:25:01 acelerathon-cima-grupo-3 gunicorn[18451]: [2020-01-17 20:25:01 +0000] [18456] [INFO] Booting worker with pid: 18456
/etc/nginx/sites-enabled/project
server {
listen 80;
server_name 127.0.0.1;
location / {
include proxy_params;
proxy_pass http://unix:/home/acelerathon_cima_grupo_3/chatbot-grupo-3/chatbot-cima/chatbot-cima-back/cima.sock;
}
}
And then I checked for errors.
sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
So both services are OK, now when I access to http://localhost I see the Nginx Home Page, not my Flask app home page
I'm not sure why is not proxying
Add localhost to server_name directive in Nginx. Nginx use this directive to match server block for each request and in your case localhost doesn't match 127.0.0.1 so it's server by default server.
May be this example will help to understand:
server {
listen 127.0.0.1:80;
server_name 127.0.0.1;
return 200 "At 127.0.0.1\n";
}
server {
listen 127.0.0.1:80;
server_name localhost;
return 200 "At localhost\n";
}
> curl -4 127.0.0.1:80
At 127.0.0.1
> curl -4 localhost:80
At localhost

nginx proxy_pass does not take affect

I want to deploy an flask app and followed a tutorial to get this done using nginx.
As the tutorial states I do as follows:
sudo nano /etc/nginx/sites-available/app
this file contains:
server {
listen 80;
server_name server_domain_or_IP;
location / {
include proxy_params;
proxy_pass http://unix:/home/pi/Desktop/python_scripts/internetdisplay/app.sock;
}
}
A systemd Unit service was created and is succesfully running. This created the app.sock file in the 'internetdisplay' directory. Systemctl status app.service results:
● app.service - Gunicorn instance to serve myproject
Loaded: loaded (/etc/systemd/system/app.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2019-11-10 21:16:49 CET; 16h ago
Main PID: 438 (gunicorn)
Tasks: 4 (limit: 2200)
Memory: 46.4M
CGroup: /system.slice/app.service
├─438 /usr/bin/python2 /usr/bin/gunicorn --workers 3 --bind unix:app.sock -m 007 wsgi:app
├─679 /usr/bin/python2 /usr/bin/gunicorn --workers 3 --bind unix:app.sock -m 007 wsgi:app
├─681 /usr/bin/python2 /usr/bin/gunicorn --workers 3 --bind unix:app.sock -m 007 wsgi:app
└─682 /usr/bin/python2 /usr/bin/gunicorn --workers 3 --bind unix:app.sock -m 007 wsgi:app
Nov 10 21:16:49 raspberrypi systemd[1]: Started Gunicorn instance to serve myproject.
Nov 10 21:16:57 raspberrypi gunicorn[438]: [2019-11-10 21:16:57 +0000] [438] [INFO] Starting gunicorn 19.9.0
Nov 10 21:16:57 raspberrypi gunicorn[438]: [2019-11-10 21:16:57 +0000] [438] [INFO] Listening at: unix:app.sock (438)
Nov 10 21:16:57 raspberrypi gunicorn[438]: [2019-11-10 21:16:57 +0000] [438] [INFO] Using worker: sync
Nov 10 21:16:57 raspberrypi gunicorn[438]: [2019-11-10 21:16:57 +0000] [679] [INFO] Booting worker with pid: 679
Nov 10 21:16:57 raspberrypi gunicorn[438]: [2019-11-10 21:16:57 +0000] [681] [INFO] Booting worker with pid: 681
Nov 10 21:16:57 raspberrypi gunicorn[438]: [2019-11-10 21:16:57 +0000] [682] [INFO] Booting worker with pid: 682
Then I link to sites-enabled and restart nginx:
sudo ln -s /etc/nginx/sites-available/app /etc/nginx/sites-enabled
sudo systemctl restart nginx
But surfing to http://localhost leads to an "this site can't be reached" error
It sounds like your location block is not set up correctly to find your resourses.
I assume that this is not the location of your unix socket:
/home/tasnuva/work/deployment/src/app.sock
Check the following:
systemd unit file is creating a socket in the expected location
the daemon is indeed running and the socket file exists
your nginx config is pointing to the correct socket file.
If none of this tells you anything, please update your question with appropriate error log entries.

Serving API via Flask / Gunicorn / Nginx: Connection refused

I'm having trouble getting gunicorn and Nginx to work together and allow me to offer a simple API via flask:
Locally, running gunicorn and getting responses from the server works fine:
gunicorn wsgi:app (start server)
[2019-06-11 23:12:48 +0000] [14615] [INFO] Starting gunicorn 19.9.0
[2019-06-11 23:12:48 +0000] [14615] [INFO] Listening at: http://127.0.0.1:8000 (14615)
[2019-06-11 23:12:48 +0000] [14615] [INFO] Using worker: sync
[2019-06-11 23:12:48 +0000] [14619] [INFO] Booting worker with pid: 14619
curl http://127.0.0.1:8000/predict (client call server for prediction)
output: "SERVER WORKS"
The problem arises when I try to use Nginx as well.
/etc/systemd/system/app.service
[Unit]
Description=Gunicorn instance to serve app
After=network.target
[Service]
User=root
Group=www-data
WorkingDirectory=/root/server
ExecStart=/usr/local/bin/gunicorn --bind unix:app.sock -m 007 wsgi:app
[Install]
WantedBy=multi-user.target
/etc/nginx/sites-available/app
server {
listen 80;
server_name [SERVER_IP_ADDRESS];
location / {
include proxy_params;
proxy_pass http://unix:/root/server/app.sock;
}
}
The status of my systemd looks fine:
systemctl status app
● app.service - Gunicorn instance to serve app
Loaded: loaded (/etc/systemd/system/app.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-06-11 23:24:07 UTC; 1s ago
Main PID: 14664 (gunicorn)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/app.service
├─14664 /usr/bin/python /usr/local/bin/gunicorn --bind unix:app.sock -m 007 wsgi:app
└─14681 /usr/bin/python /usr/local/bin/gunicorn --bind unix:app.sock -m 007 wsgi:app
systemd[1]: Started Gunicorn instance to serve app.
gunicorn[14664]: [2019-06-11 23:24:07 +0000] [14664] [INFO] Starting gunicorn 19.9.0
gunicorn[14664]: [2019-06-11 23:24:07 +0000] [14664] [INFO] Listening at: unix:app.sock (14664)
gunicorn[14664]: [2019-06-11 23:24:07 +0000] [14664] [INFO] Using worker: sync
gunicorn[14664]: [2019-06-11 23:24:07 +0000] [14681] [INFO] Booting worker with pid: 14681
When I make a request to the server, I have trouble connecting:
curl http://[SERVER_IP_ADDRESS]:80/predict
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>
EDIT:
I tried removing server_name [SERVER_IP_ADDRESS]; from /etc/nginx/sites-available/app. I now receive 'Welcome to nginx!' at http://SERVER_IP_ADDRESS, and '404 Not Found' at http://SERVER_IP_ADDRESS/predict
FYI, my flask app only has one route, which is '/predict'
It looks like you don't have Port 80 open, so here's a quick iptables command to do so:
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT

Resources