Issue: while jobs are running with airflow admin id logs in the Linux directory are getting created with readable permissions
but when we create a new user "other than admin role" and with "User/Op role" and when the user is trying to trigger the dag the logs are getting created with different permissions and the user is not able to monitor the logs from the server (but still he can access from UI)
I tried with
modifying "venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py"
[root#server logs]# ls -la
total 12
drwx------ 2 airflow airflow 23 Feb 15 03:39 2022-01-15T00:00:00+00:00
under this directory
-rw-rw-rw-. 1 airflow airflow 23 Feb 15 03:39 7 17:35 1.log
but the issue is still persists.. directory is getting created with " drwx------" permissions
Could any one please suggest what can be done
Related
I have a Synology Diskstation DS216se running DSM 6.2.3-25426. I've installed MariaDB 10, Web Station, PHP 7.2, and myPhpAdmin, but when I open it at http://diskstation/phpMyAdmin/ I get this error message
"Sorry, the page you are looking for is not found."
I'm using an nginx server in Web Station, and the error log at /var/log/nginx/error.log contains multiple entries like the following
*621 open() "/var/services/web/phpMyAdmin/js/vendor/jquery/jquery.debounce-1.0.5.js" failed (13: Permission denied)
The file, and all other files with permission denied entries in the logs, exist in the /var/services/web/phpMyAdmin/ directory - what permissions need to be granted to the directory for this to succeed?
I hit this as well. I managed to recover, but it effectively amounts to hard clearing any evidence of prior installs of Web Station, PHP 7.2, phpMyAdmin, and any other web related services. Then manually ripping out some bad directories with broken symlinks/permissions.
My hypothesis is that I tried to install adminer prior to this and - having not done any set up for Web Station et. al. - it put the filesystem in a bad state.
I am not willing to try installing adminer again to test this hypothesis.
What I did to fix this:
Backup what you need (e.g., any personal web site).
SSH into your diskstation. Please be aware of what you are doing and keep in mind the big picture. Don't go deleting random things.
Uninstall Web Station, PHP 7.2, Apache, phpMyAdmin, etc. Anything that Web Station would ultimately be inclined to read and serve up.
Verify that /var/services/web doesn't contain anything you care about, and delete it (sudo rm -rf /var/services/web).
Verify that /volume1/web doesn't contain anything you care about, and delete everything inside it (sudo rm -rf /var/services/web). You may need to chmod permissions for this - I ended up leaving the web directory itself intact, but nothing inside.
Reboot. Mount any encrypted disks, etc.
Check that /var/services/web now shows it is symlinked to /volume1/web, e.g. sudo readlink -e /var/services/web.
Also check permissions for /volume1/web, e.g. ls -al /volume1. It should be owned by root:root and have permissive (777) bits.
Install Web Station, PHP 7.2, and phpMyAdmin in that order.
After this, I could open phpMyAdmin and be served its log in screen.
Debugging notes:
For me, when I SSH in I see in the logs similar issues:
2020/12/17 10:36:35 [error] 32658#32658: *1028 "/var/services/web/phpMyAdmin/index.php" is forbidden (13: Permission denied),
ps says that the nginx workers run as the http user (uid=1023(http) gid=1023(http) groups=1023(http)).
The directory /var/services/web/ appears to be owned by root, both group and user:
# ls -al /var/services/web/
total 424
drwxr-xr-x 3 root root 4096 Dec 17 10:29 .
drwxr-xr-x 3 root root 4096 Dec 17 10:22 ..
-rw-r--r-- 1 root root 27959 Apr 13 2016 adminer.css
-rw-r--r-- 1 root root 82 Apr 13 2016 .htaccess
-rw-r--r-- 1 root root 387223 Apr 13 2016 index.php
drwxr-xr-x 10 root root 4096 Dec 17 10:29 phpMyAdmin
It's not clear to me how Web Station's nginx is intended to work at all given the mismatch - perhaps some set of actions I took prior caused it to decide to install with bad ownership.
I decided to leave everything owned by root, but changed group permissions so that http can access:
# chown -R root:http /var/services/web/
# chmod -R 775 /var/services/web/
This got past the initial error, but revealed a new one:
"/usr/syno/synoman/phpMyAdmin/index.cgi" is not found (2: No such file or directory)
Indeed, there was no trace of phpMyAdmin anywhere in that directory. Evidence of a bad install.
I decided to uninstall anything web related: phpMyAdmin, PHP 7, Apache (happened to be installed), nginx, and Web Station. Once I did, I still had two files in /var/services/web: adminer.css index.php.
I had tried adminer prior to this. In /var/services, there were symlinks to specific volume locations, e.g.:
# ls -al /var/services/
total 12
drwxr-xr-x 3 root root 4096 Dec 17 10:22 .
drwxr-xr-x 17 root root 4096 Dec 17 10:21 ..
lrwxrwxrwx 1 root root 18 Jan 20 2020 download -> /volume1/#download
lrwxrwxrwx+ 1 root root 14 Dec 17 10:22 homes -> /volume1/homes
lrwxrwxrwx 1 root root 24 Jan 20 2020 pgsql -> /volume1/#database/pgsql
lrwxrwxrwx 1 root root 13 Dec 17 10:22 tmp -> /volume1/#tmp
lrwxrwxrwx 1 root root 13 Dec 17 10:22 web
Interestingly, web was not symlinked. I fully deleted /var/services/web.
Looking over at /volume1, I do see a /volume1/web, again fully owned by root but with extremely constrained permission:
d---------+ 1 root root 52 Dec 17 10:14 web
There are only a few things in here, which look related to a blank install of Web Station. I fully deleted everything within /volume1/web, but left it as is. With everything maximally cleaned I rebooted.
Upon boot, /var/services/web was now symlinked to /volume1/web, which now also had useful permission bits (777), and owned by root:root. Maybe this was done by some boot recover process, who knows. (I still have nothing web related installed at this point.)
I installed Web Station, then PHP 7.2, then phpMyAdmin.
I had the same issue when accessing my server via
<name>.local/phpMyAdmin/
It worked when I accessed it via
<local ip>/phpMyAdmin/
I'm running a DAG that runs once per day. It starts with 9 concurrently running tasks that all do the same thing - each is basically polling S3 to see if that tasks's designated 1 file exists. Each task is the same code in Airflow and is put into the structure in the same way. I have 1 of these tasks, which, on random days, fails to "begin" - it won't enter the running stage. It just sits as queued . When it does this, here's what its log says
*** Log file isn't local.
*** Fetching here: http://:8793/log/my.dag.name./my_airflow_task/2020-03-14T07:00:00
*** Failed to fetch log file from worker.
*** Reading remote logs...
Could not read logs from s3://mybucket/airflow/logs/my.dag.name./my_airflow_task/2020-03-14T07:00:00
Why does this only happen on random days? All similar questions I've seen point to this error happening consistently, and once overcome, no longer continues. To "trick" this task into "running" I manually touch whatever the name of the log file is supposed to be, and then it changes to running.
So the issue appears that it had to do with the system's ownership rules regarding the folder the logs for that particular task wrote to. I used a CI tool to ship the new task_3 when I updated my Airflow's Python code to the production environment, so the task was created that way. When I peaked for log directory ownership, I noticed this for the tasks:
# inside/airflow/log/dir:
drwxrwxr-x 2 root root 4096 Mar 25 14:53 task_3 # is the offending task
drwxrwxr-x 2 airflow airflow 20480 Mar 25 00:00 task_2
drwxrwxr-x 2 airflow airflow 20480 Mar 25 15:54 task_1
So, I think what was going on, was that randomly, Airflow couldn't get the permission to write the log file, thus it wouldn't start the rest of the task. When I applied the appropriate chown command using something like sudo chown -R airflow:airflow task_3 . Ever since I changed this, the issue has disappeared.
I'm running airflow 1.10.3, on Red Hat Linux. I'm using a LocalExecutor, and the webserver and scheduler are both started via systemd.
The log files being generated by the scheduler are world-readable (i.e. mode "-rw-rw-rw-"). The log directories being created are "drwxrwxrwx".
This fails the security scans my organisation has in place. I need to be able to restrict the permissions on these files.
The umask in /etc/profile is 077. I've also added UMask=0007 to both the systemd unit files for the services. However, although this seems to be working for the logs in the dags/logs/scheduler/ directory, it is not affecting the DAG run logs.
[root#server logs]# ls -la s3_dag_test/
total 4
drwxrwxrwx. 4 airflow airflow 54 Aug 7 17:35 .
drwxrwx---. 46 airflow airflow 4096 Aug 7 20:00 ..
drwxrwxrwx. 5 airflow airflow 126 Aug 7 17:37 bash_test
drwxrwxrwx. 5 airflow airflow 126 Aug 7 17:29 check_s3_for_file_in_s3
[root#server logs]# ls -la s3_dag_test/bash_test/2019-08-07T17\:29\:27.988953+00\:00/
total 12
drwxrwxrwx. 2 airflow airflow 19 Aug 7 17:35 .
drwxrwxrwx. 5 airflow airflow 126 Aug 7 17:37 ..
-rw-rw-rw-. 1 airflow airflow 8241 Aug 7 17:35 1.log
This is probably too late to be a helpful answer for you, but I had the exact same issue. My organization raised the permissions of the Airflow log directories as a security finding. I likewise checked the umask, to no avail.
I did manage to find this:
https://anoopkm.wordpress.com/2020/03/26/world-readable-airflow-dag-logs-issue/
In a nutshell, it looks like Airflow hard-codes the permissions used for creating files and folders.
I edited this Python file: venv/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py and changed lines 242 and 247 to use the 0o770 and 0o660 instead of 0o777 and 0o666 for creating folders and files, respectively. Then I manually triggered a DAG and checked the folder permissions. The newest log folder no longer had global rwx permissions.
Can you let us know how airflow is installed as normal user or root user
I hava a systemd service "myservice" running as user www-data. I want to start/stop/restart this service by a http request. The problem is, that I get an error because of missing authentication:
Failed to restart myservice.service: Interactive authentication required.
See system logs and 'systemctl status myservice.service' for details.
First I request a php script, calling restartService.sh:
$exec_command = "../restartService.sh";
restartService.sh is only one command:
systemctl restart myservice.service
Owner and group of restartService.sh are ww-data:
-rwxr-xr-x 1 www-data www-data 41 Aug 24 12:42 restartService.sh
Any suggestions for solutions?
You can use sudo configuration to give the www-data user permission to run systemctl restart yourservice without giving it permission to run any other commands as root.
I was trying to start Phabricator's Notification Server, but experienced the following error:
/phabricator/phabricator/bin/aphlict start
[2015-11-16 18:41:08] EXCEPTION: (FilesystemException) Requested path '/var/tmp/aphlict/pid' is not writable. at [<phutil>/src/filesystem/Filesystem.php:1081]
arcanist(head=master, ref.master=9dd6eafb5254), phabricator(head=master, ref.master=50d158a8c4d9), phutil(head=master, ref.master=e9ed72483a14)
#0 Filesystem::assertWritable(string) called at [<phutil>/src/filesystem/Filesystem.php:73]
#1 Filesystem::assertWritableFile(string) called at [<phutil>/src/filesystem/Filesystem.php:89]
#2 Filesystem::writeFile(string, string) called at [<phabricator>/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:140]
#3 PhabricatorAphlictManagementWorkflow::willLaunch() called at [<phabricator>/src/applications/aphlict/management/PhabricatorAphlictManagementWorkflow.php:249]
#4 PhabricatorAphlictManagementWorkflow::executeStartCommand() called at [<phabricator>/src/applications/aphlict/management/PhabricatorAphlictManagementStartWorkflow.php:15]
#5 PhabricatorAphlictManagementStartWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:406]
#6 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:301]
#7 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/support/aphlict/server/aphlict_launcher.php:23]
The directory in question seems to be writable:
ls -l /var/tmp/aphlict
total 4
drwxr-xr-x 2 root root 4096 Nov 16 13:40 pid
If it matters, I'm running all operations as non-'root' on Ubuntu 14.04 LTS system.
I have just figured out this. As I said in the recent update, I was trying to start notification server as non-'root'. Looking again at permissions of the /var/tmp/aphlict/pid folder, the problem suddenly became crystal clear and trivial.
ls -l /var/tmp/aphlict
total 4
drwxr-xr-x 2 root root 4096 Nov 16 13:40 pid
Therefore, all that needed to be done to fix the problem is to make the directory writable for everyone (I hope that this approach does not create a potential security issue):
chmod go+w /var/tmp/aphlict/pid
su MY_NON_ROOT_USER_NAME -c './bin/aphlict start'
Aphlict Server started.
Problem solved. By the way, for the Notification Server to work properly, do I need to open port 22281, in addition to already opened 22280? (Please answer in comments. Thank you!)