I am using Airflow 1.10.5. Can't seem to find complete documentation or sample on how to setup remote logging using Elasticsearch. I saw airflow documentation about logging, but it wasn't helpful. I am trying to write the airflow (not task) logs to ES.
As far as I understand the docs, the ES log handler can only read from ES. You would have to setup your logging to print into a file, then use something like filebeat to post the file content to ES and Airflow can then read them back...
https://airflow.readthedocs.io/en/stable/howto/write-logs.html#writing-logs-to-elasticsearch
Writing Logs to Elasticsearch
Airflow can be configured to read task
logs from Elasticsearch and optionally write logs to stdout in
standard or json format. These logs can later be collected and
forwarded to the Elasticsearch cluster using tools like fluentd,
logstash or others.
I was able to achieve using [filebeat][1] shipper.
Input config section in filebeat.yml
</snip>
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /path/to/logs/*.log
</snip>
Output config section in filebeat.yml
<snip>
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
username: "elastic"
password: "changeme"
</snip>
Good doc to read especially about airflow --> ES.
I am writing a playbook, which dynamically enters the network servers provided by user at prompt in host file.Further i want to create a task which will ssh on the server one by one and run show version command to fetch the OS of those network devices.
As of now, I am getting the below error:
1."Unable to automatically determine host network os. Please manually configure ansible_network_os value for this host"
I dont want to define the OS beforehand, rather want the playbook to do it.
I tried entering some configurations in host file as below:
[device]
192.1xx.xxx.xx
[all:vars]
ansible_connection=network_cli
ansible_user=user
ansible_ssh_pass=password
ssh_args = -o GSSAPIAuthentication=no
The below plsybook is taking device name from the user and adding it to the file, further it need to ssh on those servers and fetch the OS details:
---
- name: add host dynamically
hosts: localhost
gather_facts: no
vars:
Server: null_val1
vars_prompt:
- name: "Server"
prompt: "Please enter the non-reporting server/IP"
private: no
default: null_val1
pre_tasks:
- name: Save variable
set_fact:
Server: "{{Server}}"
- name: Copying servers to host file
hosts: localhost
become: true
tasks:
- name: copying variable value
lineinfile:
path: "{{playbook_dir}}/hosts.txt"
line: "{{Server}}"
insertafter: '^\[device\]'
state: present
- name: Main execution task
hosts: device
gather_facts: true
tasks:
- name: Run command on devices
cli_command:
command: show version
register: result
- name: display result
debug:
var: result.stdout_lines
The actual result should show me the OS of network device(s) entered in host file by running show version command remotely.
Composer is failing a task due to it not being able to read a log file, it's complaining about incorrect encoding.
Here's the log that appears in the UI:
*** Unable to read remote log from gs://bucket/logs/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log
*** 'ascii' codec can't decode byte 0xc2 in position 6986: ordinal not in range(128)
*** Log file does not exist: /home/airflow/gcs/logs/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log
*** Fetching from: http://airflow-worker-68dc66c9db-x945n:8793/log/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-68dc66c9db-x945n', port=8793): Max retries exceeded with url: /log/campaign_exceptions_0_0_1/merge_campaign_exceptions/2019-08-03T10:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1c9ff19d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
I try viewing the file in the google cloud console and it also throws an error:
Failed to load
Tracking Number: 8075820889980640204
But I am able to download the file via gsutil.
When I view the file, it seems to have text overriding other text.
I can't show the entire file but it looks like this:
--------------------------------------------------------------------------------
Starting attempt 1 of 1
--------------------------------------------------------------------------------
#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
[2019-08-04 10:01:23,313] {models.py:1569} INFO - Executing <Task(BigQueryOperator): merge_campaign_exceptions> on 2019-08-03T10:00:00+00:00#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
[2019-08-04 10:01:23,314] {base_task_runner.py:124} INFO - Running: ['bash', '-c', u'airflow run __campaign_exceptions_0_0_1 merge_campaign_exceptions 2019-08-03T10:00:00+00:00 --job_id 22767 --pool _bq_pool --raw -sd DAGS_FOLDER//-campaign-exceptions.py --cfg_path /tmp/tmpyBIVgT']#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
[2019-08-04 10:01:24,658] {base_task_runner.py:107} INFO - Job 22767: Subtask merge_campaign_exceptions [2019-08-04 10:01:24,658] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800#-#{"task-id": "merge_campaign_exceptions", "execution-date": "2019-08-03T10:00:00+00:00", "workflow": "__campaign_exceptions_0_0_1"}
Where the #-#{} pieces seems to be "on top of" the typical log.
I faced the same problem. In my case the problem was that I removed the google_gcloud_default connection that was being used to retrieve the logs.
Check the configuration and look for the connection name.
[core]
remote_log_conn_id = google_cloud_default
Then check the credentials used for that connection name has the right permissions to access the GCS bucket.
I'm having a similar problem with viewing logs in GCP Cloud Composer. It doesn't appear to be preventing the failing DAG task from running though. What it looks like is a permissions error between the GKE and Storage Bucket where the log files are kept.
You can still view the logs by going into your cluster's storage bucket in the same directory as your /dags folder where you should also see a logs/ folder.
Your helm chart should setup global env:
- name: AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT
value: "google-cloud-platform://"
Then, you should deploy a Dockerfile with root account only (not airflow account), additionaly, you set up your helm uid, gid as:
uid: 50000 #airflow user
gid: 50000 #airflow group
Then upgrade helm chart with new config
*** Unable to read remote log from gs://bucket
1)Found the solution after assigning the roles to the service account
2)The SA key(json or txt) to be added and configured to the connection in the
remote_log_conn_id = google_cloud_default
3)restart the scheduler and webserver of the airflow
4)restart the dags on the airflow
you can find the logs on the GCS bucket where its configured
I'm trying to use CodeDeploy to deploy an application onto EC2 but I am facing the following error
Duplicate permission setting instructions for /var/www/html/storage/framework
My appspec.yml is below
version: 0.0
os: linux
files:
- source: /
destination: /var/www/html
permissions:
- object: /var/www/html
owner: apache
group: apache
mode: 644
except:
- storage/*
type:
- directory
- object: /var/www/html/storage
owner: apache
group: apache
mode: 777
type:
- directory
I've tried various formats for except including
Explicitly listing relative paths
except:
- storage
- storage/app
- storage/logs
- storage/framework
- storage/framework/views
- storage/framework/cache
- storage/framework/sessions
Using a wildcard
except:
- storage/*
Using just the folder name
except:
- storage
None of which seem to resolve the issue.
Similar questions
AWS CodeDeploy Duplicate permission
Duplicate permission setting instructions
The except option needs to be specified as an array. (Use [] not a nested list)
You can see it in the permissions examples in the appspec reference guide (scroll down a little): http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref.html#app-spec-ref-permissions
Attempts to use a sqlite file database in the app/cache/tests directory fail. This is determined by clearing the dev MySQl database and populating the test environment database via the console. Test database being populated was confirmed by an external Sqlite Manager. [Tests performed without the sqlite configuration pass.]
codeception.yml:
actor: Tester
paths:
tests: tests
log: tests/_output
data: tests/_data
support: tests/_support
envs: tests/_envs
settings:
bootstrap: _bootstrap.php
colors: false
memory_limit: 1024M
extensions:
enabled:
- Codeception\Extension\RunFailed
modules:
config:
- Db:
dsn: 'sqlite:./app/cache/test/test.sqlite'
user: ''
password: ''
dump: tests/_data/test.sql
populate: true
cleanup: false
reconnect: true
Edit:
In an Ubuntu VM, adding the Symfony2 configuration to acceptance.yml allows for partial success - the test uses the Sqlite db but does not rewrite from the specified dump (test.sql). In Windows adding the Symfony configuration makes no difference.
acceptance.yml:
class_name: AcceptanceTester
modules:
enabled:
- PhpBrowser:
url: http://vol
- \Helper\Acceptance
# adding these lines enables Ubuntu to use sqlite db
- Symfony2:
app_path: ./app
environment: test
You haven't enabled Db module in acceptance.yml
modules:
enabled:
- Db
- PhpBrowser:
url: http://vol
- \Helper\Acceptance
Also don't add PhpBrowser and Symfony2 at the same time,
only one can be used.