Trying to test artifactory-resource by running through the example pipeline.
groups:
- name: all
jobs:
- set-pipeline
- trigger-when-new-file-is-added-to-artifactory
jobs:
- name: set-pipeline
serial: true
plan:
- in_parallel:
- get: ea-terraform-module-aws-rds
trigger: true
- set_pipeline: deploying-rds-instance-from-jfrog-artifact
file: ea-terraform-module-aws-rds/examples/concourse/ea-terraform-module-aws-rds.yml
- name: trigger-when-new-file-is-added-to-artifactory
plan:
- get: ea-rds-jfrog-repo
- task: use-new-file
config:
platform: linux
image_resource:
type: docker-image
source:
repository: ubuntu
inputs:
- name: ea-rds-jfrog-repo
run:
path: cat
args:
- "./ea-rds-jfrog-repo/ea-terraform-module-aws-rds*.zip"
resource_types:
- name: artifactory
type: docker-image
source:
repository: pivotalservices/artifactory-resource
resources:
- name: ea-rds-jfrog-repo
type: artifactory
check_every: 1m
source:
endpoint: https://xxx.jfrog.io/artifactory
repository: "ea-terraform-module-aws-rds-1.4.0.zip"
regex: "ea-terraform-module-aws-rds-(?<version>.*).zip"
username: ${JF_USER}
password: ${JF_PASSWORD}
- name: ea-terraform-module-aws-rds
type: git
source:
private_key: ((github_private_key))
uri: git#github.com:xxx/xxx
branch: SAAS-27134
Concourse Error: pipeline path -> deploying-rds-isntance-from-jfrog-artifact/ea-rds-jfrog-repo
enter image description here
Repo on JFrog Artifactory
enter image description here
tried adding a version parameter
The following error indicates that the concourse resource is making a call to the Artifactory API, but instead of receiving a JSON structure, it gets a null response instead. The resource passes the null response to the jq utility that assumes an iterable object.
So why does it get a null response from the API?
It looks like, at minimum, the repository: in the ea-rds-jfrog-repo resource definition is incorrect.
Based on the second snapshot:
I'm going to guess it should be set to
repository: "/ea-terraform-module-aws-rds/terraform-module/aws"
I recommend using the https://github.com/spring-io/artifactory-resource which is in active development, instead of the unmaintained one from pivotalservices
I want to have Linux patches schedule and restart the minions if updates are success. I have created state for OS update process and sending event to event bus. Then I have created reactors to listen event tag and reboot server if tag success but reactor not react to anything and no action.
# /srv/salt/os/updates/linuxupdate.sls
uptodate:
pkg.uptodate:
- refresh: True
event-completed:
event.send:
- name: 'salt/minion-update/success'
- require:
- pkg: uptodate
event-failed:
event.send:
- name: 'salt/minion-update/failed'
- onfail:
- pkg: uptodate
# /etc/salt/master.d/reactor.conf
reactor:
- 'salt/minion-update/success':
- /srv/reactor/reboot.sls
# /srv/reactor/reboot.sls
reboot_server:
local.state.sls:
- tgt: {{ data['id'] }}
- arg:
- os.updates.reboot-server
Background
I have a YAML file like this on a web server. I am trying to read it and make user accounts in the file with an Ansible playbook.
users:
- number: 20210001
name: Aoki Alice
id: alice
- number: 20210002
name: Bob Bryant
id: bob
- number: 20210003
name: Charlie Cox
id: charlie
What I tried
To confirm how to read a downloaded YAML file dynamically with include_vars, I had written a playbook like this:
- name: Add users from list
hosts: workstation
tasks:
- name: Download yaml
get_url:
url: http://fqdn.of.webserver/path/to/yaml.yml
dest: "/tmp/tmp.yml"
notify:
- Read yaml
- List usernames
handlers:
- name: Read yaml
include_vars:
file: /tmp/tmp.yml
name: userlist
- name: List usernames
debug:
var: "{{ item }}"
loop: "{{ userlist.users }}"
Problem
In the handler Read yaml, I got the following error message. On the target machine (workstation.example.com), /tmp/tmp.yml is downloaded correctly.
RUNNING HANDLER [Read yaml] *****
fatal: [workstation.example.com]: FAILED! => {"ansible facts": {"userlist": []},
"ansible included var files": [], "changed": false, "message": "Could not find o
r access '/tmp/tmp. yml' on the Ansible Controller.\nIf you are using a module a
nd expect the file to exist on the remote, see the remote src option"}
Question
How can I get a YAML file with HTTP and use it as a variable with include_vars?
Another option would be to use the uri module to retrieve the value into an Ansible variable, then the from_yaml filter to parse it.
Something like:
- name: Add users from list
hosts: workstation
tasks:
- name: Download YAML userlist
uri:
url: http://fqdn.of.webserver/path/to/yaml.yml
return_content: yes
register: downloaded_yaml
- name: Decode YAML userlist
set_fact:
userlist: "{{ downloaded_yaml.content | from_yaml }}"
Note that uri works on the Ansible Controller, while get_url works on the target host (or on the host specified in delegate_to); depending on your network configuration, you may need to use different proxy settings or firewall rules to permit the download.
The include_vars task looks for files on the local (control) host, but you've downloaded the file to /tmp/tmp.yml on the remote host. There are a number of ways of getting this to work.
Perhaps the easiest is just running the download task on the control machine instead (note the use of delegate_to):
tasks:
- name: Download yaml
delegate_to: localhost
get_url:
url: http://fqdn.of.webserver/path/to/yaml.yml
dest: "/tmp/tmp.yml"
notify:
- Read yaml
- List usernames
This will download the file to /tmp/tmp.yml on the local system, where it will be available to include_vars. For example, if I run this playbook (which grabs YAML content from an example gist I just created)...
- hosts: target
gather_facts: false
tasks:
- name: Download yaml
delegate_to: localhost
get_url:
url: https://gist.githubusercontent.com/larsks/70d8ac27399cb51fde150902482acf2e/raw/676a1d17bcfc01b1a947f7f87e807125df5910c1/example.yaml
dest: "/tmp/tmp.yml"
notify:
- Read yaml
- List usernames
handlers:
- name: Read yaml
include_vars:
file: /tmp/tmp.yml
name: userlist
- name: List usernames
debug:
var: item
loop: "{{ userlist.users }}"
...it produces the following output:
RUNNING HANDLER [Read yaml] ******************************************************************
ok: [target]
RUNNING HANDLER [List usernames] *************************************************************
ok: [target] => (item=bob) => {
"ansible_loop_var": "item",
"item": "bob"
}
ok: [target] => (item=alice) => {
"ansible_loop_var": "item",
"item": "alice"
}
ok: [target] => (item=mallory) => {
"ansible_loop_var": "item",
"item": "mallory"
}
Side note: based on what I see in your playbook, I'm not sure you want
to be using notify and handlers here. If you run your playbook a
second time, nothing will happen because the file /tmp/tmp.yml
already exists, so the handlers won't get called.
With #Larsks 's answer, I made this playbook that works correctly in my environment:
- name: Download users list
hosts: 127.0.0.1
connection: local
become: no
tasks:
- name: Download yaml
get_url:
url: http://fqdn.of.webserver/path/to/yaml/users.yml
dest: ./users.yml
- name: Add users from list
hosts: workstation
tasks:
- name: Read yaml
include_vars:
file: users.yml
- name: List usernames
debug:
msg: "{{ item.id }}"
loop: "{{ users }}"
Point
Run get_url on the control host
As #Larsks said, you have to run the get_url module on the control host rather than the target host.
Add become: no to the task run on the control host
Without "become: no", you will get the following error message:
TASK [Gathering Facts] ******************************************************
fatal: [127.0.0.1]: FAILED! => {"ansible_facts": {}, "changed": false, "msg":
"The following modules failed to execute: setup\n setup: MODULE FAILURE\nSee
stdout/stderr for the exact error\n"}
Use connection: local rather than local_action
If you use local_action rather than connection: local like this:
- name: test get_url
hosts: workstation
tasks:
- name: Download yaml
local_action:
module: get_url
url: http://fqdn.of.webserver/path/to/yaml/users.yml
dest: ./users.yml
- name: Read yaml
include_vars:
file: users.yml
- name: output remote yaml
debug:
msg: "{{ item.id }}"
loop: "{{ users }}"
You will get the following error message:
TASK [Download yaml] ********************************************************
fatal: [workstation.example.com]: FAILED! => {"changed": false, "module_stde
rr": "sudo: a password is required\n", "module_stdout":"", "msg":"MODULE FAIL
URE\nSee stdout/stderr for the exact error", "rc": 1}
get_url stores a file on the control host
In this situation, the get_url module stores users.yml on the control host (in the current directory). So you have to delete the users.yml if you don't want to leave it.
I was setting up Argo in my k8s cluster in Argo namespace.
I also Installed MinIO as an Artifact repository (https://github.com/argoproj/argo-workflows/blob/master/docs/configure-artifact-repository.md).
I am configuring a workflow which tries to access that Non-Default Artifact Repository as:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artifact-passing-
spec:
entrypoint: artifact-example
templates:
- name: artifact-example
steps:
- - name: generate-artifact
template: whalesay
- - name: consume-artifact
template: print-message
arguments:
artifacts:
# bind message to the hello-art artifact
# generated by the generate-artifact step
- name: message
from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"
- name: whalesay
container:
image: docker/whalesay:latest
command: [sh, -c]
args: ["cowsay hello world | tee /tmp/hello_world.txt"]
outputs:
artifacts:
# generate hello-art artifact from /tmp/hello_world.txt
# artifacts can be directories as well as files
- name: hello-art
path: /tmp/hello_world.txt
s3:
endpoint: argo-artifacts-minio.argo:9000
bucket: my-bucket
key: /my-output-artifact.tgz
accessKeySecret:
name: argo-artifacts-minio
key: accesskey
secretKeySecret:
name: argo-artifacts-minio
key: secretkey
- name: print-message
inputs:
artifacts:
# unpack the message input artifact
# and put it at /tmp/message
- name: message
path: /tmp/message
s3:
endpoint: argo-artifacts-minio.argo:9000
bucket: my-bucket
accessKeySecret:
name: argo-artifacts-minio
key: accesskey
secretKeySecret:
name: argo-artifacts-minio
key: secretkey
container:
image: alpine:latest
command: [sh, -c]
args: ["cat /tmp/message"]
I created the workflow in argo namespace by:
argo submit --watch artifact-passing-nondefault-new.yaml -n argo
But the workflow fails with an error:
STEP PODNAME DURATION MESSAGE
✖ artifact-passing-z9g64 child 'artifact-passing-z9g64-150231068' failed
└---⚠ generate-artifact artifact-passing-z9g64-150231068 12s failed to save outputs: Get https://argo-artifacts-minio.argo:9000/my-bucket/?location=: http: server gave HTTP response to HTTPS client
Can someone help me to solve this error?
Since the minio setup runs without TLS configured, the workflow should specify that it should connect to an insecure artifact repository.
Including a field insecure: true in the s3 definition section of the workflow solves the issue.
s3:
endpoint: argo-artifacts-minio.argo:9000
insecure: true
bucket: my-bucket
key: /my-output-artifact.tgz
accessKeySecret:
name: argo-artifacts-minio
key: accesskey
secretKeySecret:
name: argo-artifacts-minio
key: secretkey
im playing around with saltstack and wanted to use the apache-formula from github/saltstack-formulas.
my pillar looks like following:
top.sls
base:
'ubuntu-xenial-salt':
- systems.ubuntu-xenial-salt
systems/ubuntu-xenial-salt.sls
include:
- setups.apache.prod
apache:
sites:
ubuntu-salt-xenial:
enabled: True
template_file: salt://apache/vhosts/standard.tmpl
template_engine: jinja
interface: '*'
port: '80'
exclude_listen_directive: True # Do not add a Listen directive in httpd.conf
ServerName: ubuntu-salt-xenial
ServerAlias: ubuntu-salt-xenial
ServerAdmin: minion#ubuntu-salt-xenial.com
LogLevel: debug
ErrorLog: /var/log/apache2/example.com-error.log
CustomLog: /var/log/apache2/example.com-access.log
DocumentRoot: /var/www/ubuntu-salt-xenial/
Directory:
default:
Options: -Indexes +FollowSymLinks
Require: all granted
AllowOverride: None
setups/apache/prod.sls
include:
- applications.apache
# ``apache`` formula configuration:
apache:
register-site:
# any name as an array index, and you can duplicate this section
UNIQUE_VALUE_HERE:
name: 'PROD'
path: 'salt://path/to/sites-available/conf/file'
state: 'enabled'
# Optional - use managed file as Jinja Template
#template: true
#defaults:
# custom_var: "default value"
modules:
enabled: # List modules to enable
- rewrite
- ssl
disabled: # List modules to disable
- ldap
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
keepalive: 'On'
security:
# can be Full | OS | Minimal | Minor | Major | Prod
# where Full conveys the most information, and Prod the least.
ServerTokens: Prod
# ``apache.mod_remoteip`` formula additional configuration:
mod_remoteip:
RemoteIPHeader: X-Forwarded-For
RemoteIPTrustedProxy:
- 10.0.8.0/24
- 127.0.0.1
# ``apache.mod_security`` formula additional configuration:
mod_security:
crs_install: True
# If not set, default distro's configuration is installed as is
manage_config: True
sec_rule_engine: 'On'
sec_request_body_access: 'On'
sec_request_body_limit: '14000000'
sec_request_body_no_files_limit: '114002'
sec_request_body_in_memory_limit: '114002'
sec_request_body_limit_action: 'Reject'
sec_pcre_match_limit: '15000'
sec_pcre_match_limit_recursion: '15000'
sec_debug_log_level: '3'
rules:
enabled:
modsecurity_crs_10_setup.conf:
rule_set: ''
enabled: True
modsecurity_crs_20_protocol_violations.conf:
rule_set: 'base_rules'
enabled: False
custom_rule_files:
# any name as an array index, and you can duplicate this section
UNIQUE_VALUE_HERE:
file: 'PROD'
path: 'salt://path/to/modsecurity/custom/file'
enabled: True
applications/apache.sls
apache:
lookup:
version: '2.4'
default_charset: 'UTF-8'
global:
AllowEncodedSlashes: 'On'
name_virtual_hosts:
- interface: '*'
port: 80
- interface: '*'
port: 443
Using this pillar configuration and calling highstate for my minion ubuntu-xenial-salt run without any error, however the setup is not the same as declared in the pillar:
for example:
the enabled rewrite module is not there.
the virtual-host config is not as setup in the pillar.
everything seems to be pretty much standard configuration from example.pillar.
When i call
salt 'ubuntu-xenial-salt' pillar.data
i get the pillar data just like i modified it... i cant understand what is happening...
ubuntu-xenial-salt:
----------
apache:
----------
keepalive:
On
lookup:
----------
default_charset:
UTF-8
global:
----------
AllowEncodedSlashes:
On
name_virtual_hosts:
|_
----------
interface:
*
port:
80
|_
----------
interface:
*
port:
443
version:
2.4
mod_remoteip:
----------
RemoteIPHeader:
X-Forwarded-For
RemoteIPTrustedProxy:
- 10.0.8.0/24
- 127.0.0.1
mod_security:
----------
crs_install:
True
custom_rule_files:
----------
UNIQUE_VALUE_HERE:
----------
enabled:
True
file:
PROD
path:
salt://path/to/modsecurity/custom/file
manage_config:
True
rules:
----------
enabled:
None
modsecurity_crs_10_setup.conf:
----------
enabled:
True
rule_set:
modsecurity_crs_20_protocol_violations.conf:
----------
enabled:
False
rule_set:
base_rules
sec_debug_log_level:
3
sec_pcre_match_limit:
15000
sec_pcre_match_limit_recursion:
15000
sec_request_body_access:
On
sec_request_body_in_memory_limit:
114002
sec_request_body_limit:
14000000
sec_request_body_limit_action:
Reject
sec_request_body_no_files_limit:
114002
sec_rule_engine:
On
modules:
----------
disabled:
- ldap
enabled:
- ssl
- rewrite
register-site:
----------
UNIQUE_VALUE_HERE:
----------
name:
PROD
path:
salt://path/to/sites-available/conf/file
state:
enabled
security:
----------
ServerTokens:
Prod
sites:
----------
ubuntu-salt-xenial:
----------
CustomLog:
/var/log/apache2/example.com-access.log
Directory:
----------
default:
----------
AllowOverride:
None
Options:
-Indexes +FollowSymLinks
Require:
all granted
DocumentRoot:
/var/www/ubuntu-salt-xenial/
ErrorLog:
/var/log/apache2/example.com-error.log
LogLevel:
debug
ServerAdmin:
minion#ubuntu-salt-xenial.com
ServerAlias:
ubuntu-salt-xenial
ServerName:
ubuntu-salt-xenial
enabled:
True
exclude_listen_directive:
True
interface:
*
port:
80
template_engine:
jinja
template_file:
salt://apache/vhosts/standard.tmpl
Do someone knows what's happening here and can help me get it running?