I launch a airflow webserver command in my local machine to start an airflow instance on port 8081. The server starts, however the pŕompt constantly shows some warning messages, as a loop. No error message appears, but the server doesn't works. Those are the messages:
/usr/local/lib/python3.8/dist-packages/airflow/configuration.py:361 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
/usr/local/lib/python3.8/dist-packages/airflow/configuration.py:361 DeprecationWarning: The dag_concurrency option in [core] has been renamed to max_active_tasks_per_dag - the old setting has been used, but please update your config.
/usr/local/lib/python3.8/dist-packages/airflow/configuration.py:361 DeprecationWarning: The processor_poll_interval option in [scheduler] has been renamed to scheduler_idle_sleep_time - the old setting has been used, but please update your config.
[2022-06-13 15:11:57,355] {manager.py:779} WARNING - No user yet created, use flask fab command to do it.
[2022-06-13 15:12:01,925] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create User
[2022-06-13 15:12:19 +0000] [1117638] [INFO] Handling signal: ttou
[2022-06-13 15:12:19 +0000] [1120256] [INFO] Worker exiting (pid: 1120256)
[2022-06-13 15:12:19 +0000] [1117638] [WARNING] Worker with pid 1120256 was terminated due to signal 15
[2022-06-13 15:12:22 +0000] [1117638] [INFO] Handling signal: ttin
[2022-06-13 15:12:22 +0000] [1121568] [INFO] Booting worker with pid: 1121568
Do you know what can could be happening?
Thank you in advance!
Related
I'm new to airflow, really appreciate any help for the following problem.
I tried to run the airflow webserver on my laptop.
Theoratically, I set the start_time=datetime.now(), it should be run successfully when I manually ran the dag on the webserver,but it changed overtime, it had been either queued or successful.
Sometimes it was successful(but runtime is 00:00:00, and obviously my dag hadn't been ran), and sometimes it was just queued.
Here's the code in my DAG:
from datetime import datetime
from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator
def get_var():
#a=Variable.get('abc')
print('abd')
with DAG(dag_id='test_var',start_date=datetime.now()) as dag:
task1=PythonOperator(task_id='var',python_callable=get_var)
However, every time when I check the Graph bar in the airflow webUI, it shows up as the below picture:
I'm not sure if it matters with the way I initialize airflow, I follow the below steps:
airflow webserver -p 8080
airflow db init
--- These two steps worked, yet the third step ---
airflow scheduler
[2022-10-31 09:46:45,562] {scheduler_job.py:701} INFO - Starting the scheduler
[2022-10-31 09:46:45,562] {scheduler_job.py:706} INFO - Processing each file at most -1 times
[2022-10-31 09:46:45,565] {executor_loader.py:107} INFO - Loaded executor: SequentialExecutor
[2022-10-31 09:46:45,569] {manager.py:163} INFO - Launched DagFileProcessorManager with pid: 13315
[2022-10-31 09:46:45,570] {scheduler_job.py:1381} INFO - Resetting orphaned tasks for active dag runs
[2022-10-31 09:46:46,169] {settings.py:58} INFO - Configured default timezone Timezone('UTC')
[2022-10-31T09:46:46.172+0800] {manager.py:409} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2) when using sqlite. So we set parallelism to 1.
[2022-10-31 09:46:46 +0800] [13314] [INFO] Starting gunicorn 20.1.0
[2022-10-31 09:46:46 +0800] [13314] [ERROR] Connection in use: ('::', 8793)
[2022-10-31 09:46:46 +0800] [13314] [ERROR] Retrying in 1 second.
[2022-10-31 09:46:47 +0800] [13314] [ERROR] Connection in use: ('::', 8793)
[2022-10-31 09:46:47 +0800] [13314] [ERROR] Retrying in 1 second.
[2022-10-31 09:46:48 +0800] [13314] [ERROR] Connection in use: ('::', 8793)
[2022-10-31 09:46:48 +0800] [13314] [ERROR] Retrying in 1 second.
[2022-10-31 09:46:49 +0800] [13314] [ERROR] Connection in use: ('::', 8793)
[2022-10-31 09:46:49 +0800] [13314] [ERROR] Retrying in 1 second.
[2022-10-31 09:46:50 +0800] [13314] [ERROR] Connection in use: ('::', 8793)
[2022-10-31 09:46:50 +0800] [13314] [ERROR] Retrying in 1 second.
[2022-10-31 09:46:51 +0800] [13314] [ERROR] Can't connect to ('::', 8793)
It turned out like this.
Does this have anything to do with my DAG operation on the webUI?
Thanks for your time and help!
I tried to search to another stackflow post about `[ERROR] Can't connect to ('::', 8793), but they only discussed about the webserver stuff, and also I'm not sure if the reason that my dag couldn't work is because of airflow scheduler
Airflow Scheduler service needs to be running to get the Airflow DAGs to work. In your case, it seems another service is already using port 8793 (default for scheduler) which is probably why Airflow Scheduler service is unable to start.
Consider the following:
Use Executor = LocalExecutor
Running Airflow Scheduler in the Background first with the command: airflow scheduler &
Use something like this for defining your start_date
start_date = datetime.now(local_tz) - timedelta(1)
I would also recommend using with Docker & Docker-compose to isolate your environment and all python dependency issues associated with running Airflow directly on your laptop.
I found this great article. You can also refer to Airflow's official documentation around this.
I met an issue that my task in a tag never got pick up by workers for some reason.
When I look at the task details:
All dependencies are met but the task instance is not running. In most
cases this just means that the task will probably be scheduled soon
unless:
- The scheduler is down or under heavy load
If this task instance does not start soon please contact your Airflow
administrator for assistance.
I checked the scheduler, no errors in the log, also restarted it a few times.
I also checked the airflow websever log, only notice this:
22/11/2018 12:10:39[2018-11-22 01:10:39,747] {{cli.py:644}} DEBUG - [5
/ 5] killing 1 workers 22/11/2018 12:10:39[2018-11-22 01:10:39 +0000]
[43] [INFO] Handling signal: ttou 22/11/2018 12:10:39[2018-11-22
01:10:39 +0000] [348] [INFO] Worker exiting (pid: 348)
Not sure what happens, it worked fine before.
Airflow version 1.9.0, never change the version, only playing around some of the config: min_file_process_interval and dag_dir_list_interval (but I put it back to default when encounter this issue)
I do notice that this happens when I am playing around with some of the airflow config and rebuild our docker airflow image, then I revert it back to the original version, which used to work. Then the problem solved.
I also notice one error occurred (but not always captured) in my celery workers when I use the newly built image:
Unrecoverable error: AttributeError("'float' object has no attribute 'items'",)
So find that it is related to the latest redis release (Celery will use redis), you can find more details.
I am having trouble deploying my own shiny apps to an AWS server. I followed these instructions to a tee. I was able to install shiny and see the example applications working on my server. However when I try to launch my own app, I see the error messages:
"An error has occurred
The application failed to start.
The application exited during initialization."
According to many forums, for example this one, I need to adjust the permissions in the /etc/shiny-server/shiny-server.conf file from run_as shiny; to run_as myusername shiny. I tried this, but it didn't work.
My log looks like this:
cat /var/log/shiny-server.log
[2018-05-15T08:25:18.770] [INFO] shiny-server - Shiny Server v1.5.7.907 (Node.js v8.10.0)
[2018-05-15T08:25:18.772] [INFO] shiny-server - Using config file "/etc/shiny-server/shiny-server.conf"
[2018-05-15T08:25:18.827] [WARN] shiny-server - Running as root unnecessarily is a security risk! You could be running more securely as non-root.
[2018-05-15T08:25:18.831] [INFO] shiny-server - Starting listener on 0.0.0.0:3838
[2018-05-15T08:25:32.474] [INFO] shiny-server - Created bookmark state directory: /var/lib/shiny-server/bookmarks
[2018-05-15T08:25:32.475] [INFO] shiny-server - Created user bookmark state directory: /var/lib/shiny-server/bookmarks/shiny
[2018-05-16T22:31:45.674] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T22:40:21.378] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T22:40:54.257] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T22:42:02.326] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T22:42:14.474] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T22:44:18.246] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T22:56:41.494] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:00:17.853] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:09:32.642] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:15:39.846] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:16:04.410] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:23:36.596] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:26:43.907] [INFO] shiny-server - Stopping listener on 0.0.0.0:3838
[2018-05-16T23:26:43.907] [INFO] shiny-server - Shutting down worker processes (with notification)
[2018-05-16T23:26:56.234] [INFO] shiny-server - Shiny Server v1.5.7.907 (Node.js v8.10.0)
[2018-05-16T23:26:56.237] [INFO] shiny-server - Using config file "/etc/shiny-server/shiny-server.conf"
[2018-05-16T23:26:56.295] [WARN] shiny-server - Running as root unnecessarily is a security risk! You could be running more securely as non-root.
[2018-05-16T23:26:56.298] [INFO] shiny-server - Starting listener on 0.0.0.0:3838
[2018-05-16T23:27:00.502] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:30:22.786] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-16T23:31:25.808] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
[2018-05-17T00:35:40.060] [INFO] shiny-server - Error getting worker: Error: The application exited during initialization.
Also when I type shiny-server I see:
Using config file "/etc/shiny-server/shiny-server.conf"
[2018-05-17T01:17:12.659] [ERROR] shiny-server - Error loading config: The user 'ubuntu' does not have permissions to run applications as one of the users in 'shiny'. Please restart shiny-server as one of the users in 'shiny'. (/etc/shiny-server/shiny-server.conf:2:1)
Config file:
cat /etc/shiny-server/shiny-server.conf
# Instruct Shiny Server to run applications as the user "shiny"
run_as ubuntu;
# Define a server that listens on port 3838
server {
listen 3838;
# Define a location at the base URL
location / {
# Host the directory of Shiny Apps stored in this directory
site_dir /srv/shiny-server;
# Log all Shiny output to files in this directory
log_dir /var/log/shiny-server;
# When a user visits the base URL rather than a particular application,
# an index of the applications available in this directory will be shown.
directory_index on;
}
}
sanitize_errors off;
Any help would be greatly appreciated. I feel like I am so close to finally launching my app!
As your error says the user ubuntu isn't allowed to start shiny apps. Make sure to login as sudo, try something below. if that doesn't work check that the shiny app is installed globally with sudo access
sudo -i
sudo systemctl restart shiny-server
or try stopping and starting it
sudo systemctl stop shiny-server
sudo systemctl start shiny-server
Im running ubuntu on my server btw
I'm trying to set up multiple servers that look like:
Client Request ----> Nginx (Reverse-Proxy / Load-Balancer)
|
/|\
| | `-> App. Server I. 10.128.xxx.yy1:8080 # Our example
| `--> App. Server II. 10.128.xxx.yy2:8080
`----> ..
I understand that I need to put the App servers (Gunicorn in this case) behind an Nginx Proxy, but how do I set up the App servers by themselves?
I'm trying to set up the App server with systemd, and my configuration looks like:
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=kyle
Group=www-data
WorkingDirectory=/home/kyle/do_app_one
ExecStart=/home/kyle/do_app_one/venv/bin/gunicorn --workers 3 --bind unix:/home/kyle/do_app_one/do_app_one.sock do_app_one.wsgi:application
[Install]
WantedBy=multi-user.target
I know the socket is being created because I can see it:
but I can't access the Gunicorn server by itself when I hit the IP address, with or without the :8000 port attached to it. Without the systemd configuration, I can access the site if I do:
gunicorn --bind 0.0.0.0:8000 myproject.wsgi:application
but I want to do this the right way with an init system like systemd, and I don't think I'm supposed to be binding it directly to a port because I've read it's less efficient/secure than using a socket. Unless binding to a port is the only way, then I guess that's what I have to do.
Every tutorial I see says I need an Nginx server in front of my Gunicorn server, but I already have an Nginx server in front of them. Do I need another one in front of each server such that it looks like:
Client Request ----> Nginx (Reverse-Proxy / Load-Balancer)
|
/|\
| | `-> Nginx + App. Server I. 10.128.xxx.yy1:8080 # Our example
| `--> Nginx + App. Server II. 10.128.xxx.yy2:8080
`----> ..
If Nginx is an HTTP server, and Gunicorn is an HTTP server, why would I need another Nginx server in front of each App Server? It seems redundant.
And if I don't need another Nginx server in front of each Gunicorn server, how do I set up the Gunicorn server with systemd such that it can stand alone?
Edit:
I was curious as to why the binding to a physical port was working, but the socket wasn't, so I ran gunicorn status and got errors:
kyle#ubuntu-512mb-tor1-01-app:~/do_app_one$ . venv/bin/activate
(venv) kyle#ubuntu-512mb-tor1-01-app:~/do_app_one$ gunicorn status
[2016-12-03 20:19:49 +0000] [11050] [INFO] Starting gunicorn 19.6.0
[2016-12-03 20:19:49 +0000] [11050] [INFO] Listening at: http://127.0.0.1:8000 (11050)
[2016-12-03 20:19:49 +0000] [11050] [INFO] Using worker: sync
[2016-12-03 20:19:49 +0000] [11053] [INFO] Booting worker with pid: 11053
[2016-12-03 20:19:49 +0000] [11053] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
worker.init_process()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process
self.load_wsgi()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/kyle/do_app_one/venv/lib/python3.5/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
ImportError: No module named 'status'
[2016-12-03 20:19:49 +0000] [11053] [INFO] Worker exiting (pid: 11053)
[2016-12-03 20:19:49 +0000] [11050] [INFO] Shutting down: Master
[2016-12-03 20:19:49 +0000] [11050] [INFO] Reason: Worker failed to boot.
Still not sure how to fix the problem though.
The right answer is to just bind Gunicorn to a port instead of a unix socket. I'm not too sure about the details, but unix sockets can only be used within a local network according to:
https://unix.stackexchange.com/questions/91774/performance-of-unix-sockets-vs-tcp-ports
So when I changed the gunicorn.service file ExecStart line to:
ExecStart=/home/kyle/do_app_one/venv/bin/gunicorn --workers 3 --bind 0.0.0.0:8000 do_app_one.wsgi:application
I was able to access the server by itself, and connect it to my Nginx server that was on a different IP.
I currently using ejabberd_http_auth mod to authenticate a user by external http api. However after I set up the following configuration in ejabberd.yml. I got Unknown option 'auth_opts' error.
I have already installed that plugin in ejabberd using command prompt and I have disable register mod.
Configuration:
auth_method: http
auth_opts:
host: "http://localhost:8080"
connection_pool_size: 10
connection_opts: []
basic_auth: ""
path_prefix: "/test/auth/"
Error Message:
2015-12-15 00:16:16.268 [error] <0.37.0>#ejabberd_config:validate_opts:794 unknown option 'auth_opts' will be likely ignored
2015-12-15 00:16:16.366 [info] <0.37.0>#cyrsasl_digest:start:60 FQDN used to check DIGEST-MD5 SASL authentication: MY_SERVER
2015-12-15 00:16:16.367 [info] <0.37.0>#ejabberd_app:add_windows_nameservers:195 Adding machine's DNS IPs to Erlang system:
[]
2015-12-15 00:16:16.373 [error] <0.36.0> CRASH REPORT Process <0.36.0> with 0 neighbours exited with reason: call to undefined function ejabberd_auth_http:start(<<"localhost">>) in application_master:init/4 line 133
2015-12-15 00:16:16.373 [info] <0.7.0> Application ejabberd exited with reason: call to undefined function ejabberd_auth_http:start(<<"localhost">>)
Thanks a lot.
The unknown option is not the problem here.
The problem is on that line:
2015-12-15 00:16:16.373 [info] <0.7.0> Application ejabberd exited with reason: call to undefined function ejabberd_auth_http:start(<<"localhost">>)
It means ejabberd_auth_http.beam is not in your Erlang path. It means it is either not installed or placed outside Erlang VM path.