I would like airflow to use the default local ~./aws credentials when running locally and when ran in EMR it must take those credentials.
Currently, I am exporting them as AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID . I have two environments. therefore I am extracting the account number and based on that it will take the appropriate credentials
os.environ["AWS_ACCESS_KEY_ID"] = "A"
os.environ["AWS_SECRET_ACCESS_KEY"] = "s"
account_id = boto3.client('sts').get_caller_identity().get('Account')
if account_id=="2222222":
path="prd-datahub"
elif account_id=="111111111":
path="datahub"
tz = pytz.timezone('US/Central')
DAG_ID = os.path.basename(__file__).replace(".py", "")
os.environ['AWS_DEFAULT_REGION'] = 'us-west-1'
When I execute locally it says "botocore. credentials not found". How do I give this local AWS credentials path?
Related
When my application is up, telegraf works fine and collects data related to jolokia since my application opens the port 11722 that telegraf uses to get the metrics. But then, when my application is down, telegraf starts to get errors since it can't connect to Jolokia. My telegraf version is 1.5.3 and this is a Production environment, so I don't have much flexibility to change the version. Is there a way to collect the jolokia metrics just when my application is up and running?
I've tried to create a script to check if jolokia was running and use with a tag that then I could use with my agent, but this didn't work:
[[inputs.exec]]
commands = ["sh /local/1/home/svcegctp/telegraf/inputs/scripts/check_jolokia.sh"]
timeout = "1s"
data_format = "influx"
name_override = "jvm_status"
[inputs.exec.tags]
running = "true"
(...)
[[inputs.jolokia2_agent]]
# Add agents URLs to query
urls = ["http://localhost:11722/jolokia"]
[inputs.jolokia2_agent.tags]
running = "true"
This is my script:
check_jolokia.sh
#!/bin/bash
if curl -s -u <username>:<password> http://localhost:11722/jolokia/version >/dev/null 2>&1; then
echo "jvm_status running=true"
else
echo "jvm_status running=false"
fi
I connect to snowflake via both odbc and snowsql. For whatever reason, when querying via odbc and looking at running queries in the history tab, the size is set to Large. When querying via SnowSQL the size is set to small.
I did some searching for odbc and snowsql config but could not tell if this is something I set within the UI or via the connection settings in either client.
Can I set size via a connection setting for either odbc or SnowSQL? If yes, how?
E.g. my current sanitized connection settings:
SnowSQL:
Config file:
[connections]
authenticator = SNOWFLAKE_JWT
private_key_path = /root/.snowsql/rsa_key.p8
[variables]
#Loads these variables on startup
#Can be used in SnowSql as select $example_variable
example_variable=27
[options]
# If set to false auto-completion will not occur interactive mode.
auto_completion = True
# going to use start and end date variables from environment in the sql script
variable_substitution = True
# main log file location. The file includes the log from SnowSQL main
# executable.
log_file = /root/.snowsql/snowsql_rt.log
# bootstrap log file location. The file includes the log from SnowSQL bootstrap
# executable.
log_bootstrap_file = /root/.snowsql/log_bootstrap
# Default log level. Possible values: "CRITICAL", "ERROR", "WARNING", "INFO"
# and "DEBUG".
log_level = DEBUG
# Timing of sql statments and table rendering.
timing = True
# Table format. Possible values: psql, plain, simple, grid, fancy_grid, pipe,
# orgtbl, rst, mediawiki, html, latex, latex_booktabs, tsv.
# Recommended: psql, fancy_grid and grid.
output_format = psql
# Keybindings: Possible values: emacs, vi.
# Emacs mode: Ctrl-A is home, Ctrl-E is end. All emacs keybindings are available in the REPL.
# When Vi mode is enabled you can use modal editing features offered by Vi in the REPL.
key_bindings = emacs
# OCSP Fail Open Mode.
# The only OCSP scenario which will lead to connection failure would be OCSP response with a
# revoked status. Any other errors or in the OCSP module will not raise an error.
# ocsp_fail_open = True
# Enable temporary credential file for Linux users
# For Linux users, since there are no OS-key-store, an unsecure temporary credential for SSO can be enabled by this option. The default value for this option is False.
# client_store_temporary_credential = True
# Repository Base URL
# The endpoint to download the SnowSQL main module.
repository_base_url = https://sfc-repo.snowflakecomputing.com/snowsql
I use this config when connecting with the following:
snowsql -f ${INPUT_QUERY_FILE} \
-o quiet=true \
-o friendly=false \
-o header=true \
-o output_format=csv \
-o output_file=output_data/data.csv \
--accountname=${INPUT_ACCOUNT_NAME} \
--username=${INPUT_USER_NAME} \
--dbname=${INPUT_DBNAME} \
--private-key-path=/root/.snowsql/rsa_key.p8 \
--config /config
This runs as expected and returns the query result. But it uses a small setting when querying Snowflake. Is there a way to adjust this, e.g. large?
And for odbc:
[snowflake]
Description=SnowflakeDB
Driver=SnowflakeDSIIDriver
Locale=en-US
SERVER=<ourorg>.us-east-1.snowflakecomputing.com
PORT=443
SSL=on
ACCOUNT=<ourorg>.us-east-1
# change to your snowflake user_name
UID=MY_NAME
# change to /home/<your-home>/keys/rsa_key.p8
PRIV_KEY_FILE=/home/my_name/keys/rsa_key.p8
# change "blahblah' to the passphrase you set for your private key in step 1.c
PRIV_KEY_FILE_PWD=blahblah
AUTHENTICATOR=SNOWFLAKE_JWT
And then I connect with (R)
sfconn <- DBI::dbConnect(odbc::odbc(), dsn = 'snowflake', warehouse = 'DATA_SCIENCE_WH_L',
database = 'ourorg', role='DATA_SCIENCE_FULL')
This connects fine and I'm able to query Snowflake using odbc client. The query is run with the 'large' settings. Is it possible to adjust this?
By 'size' do you mean the virtual warehouse size? Are you specifying the warehouse when connecting via SnowSQL or defaulting to one that is sized as small? I see on the ODBC line that you are using a warehouse name of DATA_SCIENCE_WH_L while you are not listing a warehouse (or role) on the SnowSQL connection string.
I am trying to deploy plumber API on DigitalOcean with plumberDeploy package. I created DigitalOcean account and already added public SSH key to it. I have private key on my machine.
Now I run:
id <- plumberDeploy::do_provision(example = FALSE,
region = getOption("do_region", "sfo3"))
And get back:
THIS ACTION COSTS YOU MONEY!
Provisioning a new server for which you will get a bill from DigitalOcean.
Using default ssh keys: work_R_laptop
NB: This costs $0.00744 / hour until you droplet_delete() it
Waiting for create .........................
New server key: some_long_key
Error: Authentication with ssh server failed
But in DigitalOcean I can see that droplet is created.
Next I run:
plumberDeploy::do_deploy_api(id, "date", "digital_o_test/", 8000, docs = TRUE)
but get back:
Error in as.droplet(droplet) : object 'id' not found
Does this happen because SSH authentication error? I am trying to go through documentation, but I am still confused how to authenticate from R session. I can authenticate using Bash and OpenSSH.
Found that in airflow.cfg there is possibility to 'Mount DAGs via Git':
# Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
git_repo = https://username#bitbucket.org/repo.git
git_branch = master
git_subpath =
# Use git_user and git_password for user authentication or git_ssh_key_secret_name and git_ssh_key_secret_key
# for SSH authentication
git_user = username
git_password = pass
git_sync_root = /git
git_sync_dest = repo
# Mount point of the volume if git-sync is being used.
# i.e. /home/user/airflow/dags
git_dags_folder_mount_point = /home/user/airflow/dags
Was unable to find any documentation on this.
Do I understand it correctly, that I am able to deploy my DAGs to Airflow production server using Git repo by just pushing to the branch specified?
Couldn't get that operating, would appreciate any thoughts.
The problem is pretty simple. I need to limit airflow web users to see and execute only certain DAGs and tasks.
If possible, I'd prefer not to use Kerberos nor OAuth.
The Multi-tenancy option seems like an option to go, but couldn't make it work the way I expect.
My current setup:
added airflow web users test and ikar via Web Authentication / Password
my unix username is ikar with a home in /home/ikar
no test unix user
airflow 1.8.2 is installed in /home/ikar/airflow
added two DAGs with one task:
one with owner set to ikar
one with owner set to test
cat airflow.cfg:
[core]
# The home folder for airflow, default is ~/airflow
airflow_home = /home/ikar/airflow
# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository
# This path must be absolute
dags_folder = /home/ikar/airflow-test/dags
# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = /home/ikar/airflow/logs
# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
# must supply a remote location URL (starting with either 's3://...' or
# 'gs://...') and an Airflow connection id that provides access to the storage
# location.
remote_base_log_folder =
remote_log_conn_id =
# Use server-side encryption for logs stored in S3
encrypt_s3_logs = False
# DEPRECATED option for remote log storage, use remote_base_log_folder instead!
s3_log_folder =
# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = SequentialExecutor
# The SqlAlchemy connection string to the metadata database.
# SqlAlchemy supports many different database engine, more information
# their website
sql_alchemy_conn = sqlite:////home/ikar/airflow/airflow.db
# The SqlAlchemy pool size is the maximum number of database connections
# in the pool.
sql_alchemy_pool_size = 5
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
# not apply to sqlite.
sql_alchemy_pool_recycle = 3600
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
# The number of task instances allowed to run concurrently by the scheduler
dag_concurrency = 16
# Are DAGs paused by default at creation
dags_are_paused_at_creation = True
# When not using pools, tasks are run in the "default pool",
# whose size is guided by this config element
non_pooled_task_slot_count = 128
# The maximum number of active DAG runs per DAG
max_active_runs_per_dag = 16
# Whether to load the examples that ship with Airflow. It's good to
# get started, but you probably want to set this to False in a production
# environment
load_examples = False
# Where your Airflow plugins are stored
plugins_folder = /home/ikar/airflow/plugins
# Secret key to save connection passwords in the db
fernet_key = cryptography_not_found_storing_passwords_in_plain_text
# Whether to disable pickling dags
donot_pickle = False
# How long before timing out a python file import while filling the DagBag
dagbag_import_timeout = 30
# The class to use for running task instances in a subprocess
task_runner = BashTaskRunner
# If set, tasks without a `run_as_user` argument will be run with this user
# Can be used to de-elevate a sudo user running Airflow when executing tasks
default_impersonation =
# What security module to use (for example kerberos):
security =
# Turn unit test mode on (overwrites many configuration options with test
# values at runtime)
unit_test_mode = False
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8888
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
[operators]
# The default owner assigned to each new operator, unless
# provided explicitly or passed via `default_args`
default_owner = Airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
[webserver]
# The base url of your website as airflow cannot guess what domain or
# cname you are using. This is used in automated emails that
# airflow sends to point links to the right web server
base_url = http://localhost:8888
# The ip specified when starting the web server
web_server_host = 0.0.0.0
# The port on which to run the web server
web_server_port = 8888
# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_cert =
web_server_ssl_key =
# Number of seconds the gunicorn webserver waits before timing out on a worker
web_server_worker_timeout = 120
# Number of workers to refresh at a time. When set to 0, worker refresh is
# disabled. When nonzero, airflow periodically refreshes webserver workers by
# bringing up new ones and killing old ones.
worker_refresh_batch_size = 1
# Number of seconds to wait before refreshing a batch of workers.
worker_refresh_interval = 30
# Secret key used to run your flask app
secret_key = temporary_key
# Number of workers to run the Gunicorn web server
workers = 4
# The worker class gunicorn should use. Choices include
# sync (default), eventlet, gevent
worker_class = sync
# Log files for the gunicorn webserver. '-' means log to stderr.
access_logfile = -
error_logfile = -
# Expose the configuration file in the web server
expose_config = False
# Set to true to turn on authentication:
# http://pythonhosted.org/airflow/security.html#web-authentication
authenticate = True
auth_backend = airflow.contrib.auth.backends.password_auth
# Filter the list of dags by owner name (requires authentication to be enabled)
filter_by_owner = True
# Filtering mode. Choices include user (default) and ldapgroup.
# Ldap group filtering requires using the ldap backend
#
# Note that the ldap server needs the "memberOf" overlay to be set up
# in order to user the ldapgroup mode.
owner_mode = user
# Default DAG orientation. Valid values are:
# LR (Left->Right), TB (Top->Bottom), RL (Right->Left), BT (Bottom->Top)
dag_orientation = LR
# Puts the webserver in demonstration mode; blurs the names of Operators for
# privacy.
demo_mode = False
# The amount of time (in secs) webserver will wait for the initial handshake
# while fetching logs from another worker machine
log_fetch_timeout_sec = 5
# By default, the webserver shows paused DAGs. Flip this to hide paused
# DAGs by default
hide_paused_dags_by_default = False
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
# smtp_user = airflow
# smtp_password = airflow
smtp_port = 25
smtp_mail_from = airflow#airflow.com
[celery]
# This section only applies if you are using the CeleryExecutor in
# [core] section above
# The app name that will be used by celery
celery_app_name = airflow.executors.celery_executor
# The concurrency that will be used when starting workers with the
# "airflow worker" command. This defines the number of task instances that
# a worker will take, so size up your workers based on the resources on
# your worker box and the nature of your tasks
celeryd_concurrency = 4
# When you start an airflow worker, airflow starts a tiny web server
# subprocess to serve the workers local log files to the airflow main
# web server, who then builds pages and sends them to users. This defines
# the port on which the logs are served. It needs to be unused, and open
# visible from the main web server to connect into the workers.
worker_log_server_port = 8793
# The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally
# a sqlalchemy database. Refer to the Celery documentation for more
# information.
broker_url = sqla+mysql://airflow:airflow#localhost:3306/airflow
# Another key Celery setting
celery_result_backend = db+mysql://airflow:airflow#localhost:3306/airflow
# Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start
# it `airflow flower`. This defines the IP that Celery Flower runs on
flower_host = 0.0.0.0
# This defines the port that Celery Flower runs on
flower_port = 5555
# Default queue that tasks get assigned to and that worker listens on.
default_queue = default
[scheduler]
# Task instances listen for external kill signal (when you clear tasks
# from the CLI or the UI), this defines the frequency at which they should
# listen (in seconds).
job_heartbeat_sec = 5
# The schedule constantly tries to trigger new tasks (look at the
# scheduler section in the docs for more information). This defines
# how often the scheduler should run (in seconds).
scheduler_heartbeat_sec = 5
# after how much time should the scheduler terminate in seconds
# -1 indicates to run continuously (see also num_runs)
run_duration = -1
# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
dag_dir_list_interval = 300
# How often should stats be printed to the logs
print_stats_interval = 30
child_process_log_directory = /home/ikar/airflow/logs/scheduler
# Local task jobs periodically heartbeat to the DB. If the job has
# not heartbeat in this many seconds, the scheduler will mark the
# associated task instance as failed and will re-schedule the task.
scheduler_zombie_task_threshold = 300
# Turn off scheduler catchup by setting this to False.
# Default behavior is unchanged and
# Command Line Backfills still work, but the scheduler
# will not do scheduler catchup if this is False,
# however it can be set on a per DAG basis in the
# DAG definition (catchup)
catchup_by_default = False
# Statsd (https://github.com/etsy/statsd) integration settings
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
# The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run. However, airflow will never
# use more threads than the number of cpu cores available.
max_threads = 2
authenticate = False
[mesos]
# Mesos master address which MesosExecutor will connect to.
master = localhost:5050
# The framework name which Airflow scheduler will register itself as on mesos
framework_name = Airflow
# Number of cpu cores required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_cpu = 1
# Memory in MB required for running one task instance using
# 'airflow run <dag_id> <task_id> <execution_date> --local -p <pickle_id>'
# command on a mesos slave
task_memory = 256
# Enable framework checkpointing for mesos
# See http://mesos.apache.org/documentation/latest/slave-recovery/
checkpoint = False
# Failover timeout in milliseconds.
# When checkpointing is enabled and this option is set, Mesos waits
# until the configured timeout for
# the MesosExecutor framework to re-register after a failover. Mesos
# shuts down running tasks if the
# MesosExecutor framework fails to re-register within this timeframe.
# failover_timeout = 604800
# Enable framework authentication for mesos
# See http://mesos.apache.org/documentation/latest/configuration/
authenticate = False
# Mesos credentials, if authentication is enabled
# default_principal = admin
# default_secret = admin
[kerberos]
ccache = /tmp/airflow_krb5_ccache
# gets augmented with fqdn
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = airflow.keytab
[github_enterprise]
api_rev = v3
[admin]
# UI to hide sensitive variable fields when set to True
hide_sensitive_variable_fields = True
I'd expect that test user will only see DAG with the owner set to test but both users can see and execute both DAGs.
Couldn't find any detailed documentation on how to setup the user restrictions for airflow DAGs.
Can anyone help? Am I missing something?
this error is solved in airflow 1.10.5, you can check the changelog
https://github.com/apache/airflow/blob/6fb821713fe0185b4af3a17d008a04a7669a3a64/CHANGELOG.txt#L932