I have a bunch of packages in a private debian repository. Following salt documentation (http://docs.saltstack.com/en/latest/ref/states/all/salt.states.pkgrepo.html), in a salt state I defined a pkgerepo entry like this:
my-private-repo:
pkgrepo.managed:
- hmanname: My Deb
- name: deb <url>....
- dist: my-repo
- require_in:
- pkg: pkg1
- pkg: pkg2
- pkg: ...
and in each pkg definition added the refresh: True stanza:
pkg1:
pkg:
- latest
- fromrepo: my-repo
- refresh: True
Now, it works in the sense that I get an "apt-get update" before installing (upgrading) each package, but there are quite a few of them (around 20) and I get an update for each one. Is there a way to have apt update just once after the repo state has been tested?
Helices and Antstud answers put me in the right direction. Anyway in the end I found out some interesting things that might be helpful for others:
"refresh: True" is useless with pkg.latest, seems like 'latest' implies "refresh: True"
What's stated in SaltStack doc seems not to apply (at last with version 2014.7.1)
require_in:
Set this to a list of pkg.installed or pkg.latest to trigger the running of apt-get update prior to attempting to install these packages. Setting a require in the pkg will not work for this.
I just added
- require:
- pkgrepo: my_repo
to my pkg definition and it's working (making includes less of a mess).
I believe you can just install multiple packages with a single state by using pkgs:. It works for me, even with a custom repository:
install packages:
pkg:
- latest
- fromrepo: my-repo
- refresh: True
- pkgs:
- pkg1
- pkg2
...
You can try to define pkg list in the pillars for every minion and than get the list in the state.
install packages:
pkg:
- latest
- fromrepo: my-repo
- refresh: True
- pkgs:
{% for pkg in pillar.get('packages', {}).items() %}
{{pkg}}
{% endfor %}
Related
I have set up a GitHub action to run the rcmdcheck::rcmdcheck(args = "--no-manual", error_on = "error") function on my R package every time there is a pull request, but it's taking over 15 minutes each time to download and install all of the packages, then it needs to run through the unit tests, which are also taking a while to finish.
So, I want to try and cache the required packages and dependencies using the GitHub cache action. However, I can't figure out what needs to change from the R example provided.
Here is the github action yaml which I want to cache my dependencies on:
name: R
on:
push:
branches: [ "main", "dev" ]
pull_request:
branches: [ "main", "dev" ]
permissions:
contents: read
jobs:
build:
runs-on: macos-latest
strategy:
matrix:
r-version: ['4.1.1']
env:
GITHUB_PAT: ${{secrets.REPO_KEY}}
steps:
- uses: actions/checkout#v3
- name: Set up R ${{ matrix.r-version }}
uses: r-lib/actions/setup-r#f57f1301a053485946083d7a45022b278929a78a
with:
r-version: ${{ matrix.r-version }}
- name: Install dependencies
run: |
install.packages(c("remotes", "rcmdcheck"), type = "binary", dependencies = TRUE)
remotes::install_deps(dependencies = TRUE, type = "binary")
shell: Rscript {0}
- name: Check
run: rcmdcheck::rcmdcheck(args = "--no-manual", error_on = "error")
shell: Rscript {0}
The idea is that once the code is run for the first time, so long as none of the required packages, dependencies or OS changes, then it can re-use the dependencies from last time and skip a costly part of the run time.
**Further information:** The repository this runs on is private, and requires some packages which aren't on CRAN, which adds further complexity. Some are other private repo's and the other is stored here https://inla.r-inla-download.org/R/stable/src/contrib/INLA_21.02.23.tar.gz
I have tried to use the setup-r-dependencies functionality for this, and while it works just fine on another package, it doesn't seem to work for this package. (We were unable to resolve this despite posting a bounty, see my previous post)
My sls file looks like this:
init.sls
include:
- .packages
- .user_and_group
packages.sls
monitoring_packages:
pkg.installed:
- pkgs:
- git
user_and_group.sls
monitoring__group:
group.present:
- name: myuser
For some strange reason the state monitoring__group from the include "user_and_group" get executed before installing git.
Question
How can I tell salt to install the packages first?
init.sls (unchanged)
include:
- .packages
- .user_and_group
packages.sls (unchanged)
monitoring_packages:
pkg.installed:
- pkgs:
- git
user_and_group.sls (added require)
monitoring__group:
group.present:
- name: myuser
require:
- sls: packages
Docs
I found the answer here: https://docs.saltstack.com/en/latest/ref/states/requisites.html#require-an-entire-sls-file
As of Salt 0.16.0, it is possible to require an entire sls file.
One question remains
This solves my problem. But one question remains: Why did salt execute the first version (see question) not in the top-to-bottom order? If you know it, please leave a comment.
I'm new to Saltstack and I'm just trying to do some simple installs on a subset of minions. I want to include Environments so I have my file roots as:
file_roots:
base:
- /srv/salt/base
dev:
- /srv/salt/dev
qa:
- /srv/salt/qa
stage:
- /srv/salt/stage
prod:
- /srv/salt/prod
I set up the git backend:
fileserver_backend:
- git
- roots
I'm using gitfs set as:
gitfs_remotes:
- https://github.com/saltstack-formulas/postgres-formula
- https://github.com/saltstack-formulas/apache-formula
- https://github.com/saltstack-formulas/memcached-formula
- https://github.com/saltstack-formulas/redis-formula
So I have the master set up and I add top.sls to /srv/salt/stage with
include:
- apache
stage:
'stage01*':
- apache
But I get an error when I execute
salt -l debug \* state.highstate test=True
Error
stage01.example.net:
Data failed to compile:
----------
No matching sls found for 'apache' in env 'stage'
I've tried many ways and the master just can't seem to find the apache formula I configured for it.
I found the answer and it was sitting in the Saltstack docs the whole time.
First you will need to fork the current repository such as postgres-formula.
Depending on the environment create a branch of the same name in your newly create fork of the repo.
So for example I wanted to use postgres in my stage environment. So it wouldn't work until I created a branch named stage ined my forked repo of postgres-formula then it worked like a charm.
Can someone point me towards a working example where packrat is used with AppVeyor to build an R package? Searching through Google and GitHub, I can't find any packrat-enable package that uses AppVeyor.
Does the appveyor.yml file need to change? Are there some settings I need to add through the AppVeyor website?
I have a very minimal package (testthat is the only dependency) that broke AppVeyor builds. Here is the code frozen for that commit. Here is the AppVeyor log.
(If this SO question sounds familiar, I'm about to ask a similar question for Travis-CI.)
Yes, the solution here is similar to the same question for Travis-CI.
Here's an example of an appveyor.yml file that will enable you to use packrat packages in your package:
# DO NOT CHANGE the "init" and "install" sections below
# Download script file from GitHub
init:
ps: |
$ErrorActionPreference = "Stop"
Invoke-WebRequest http://raw.github.com/krlmlr/r-appveyor/master/scripts/appveyor-tool.ps1 -OutFile "..\appveyor-tool.ps1"
Import-Module '..\appveyor-tool.ps1'
install:
ps: Bootstrap
# Adapt as necessary starting from here
environment:
global:
WARNINGS_ARE_ERRORS: 0
USE_RTOOLS: true
build_script:
- R -e "0" --args --bootstrap-packrat
test_script:
- travis-tool.sh run_tests
on_failure:
- 7z a failure.zip *.Rcheck\*
- appveyor PushArtifact failure.zip
artifacts:
- path: '*.Rcheck\**\*.log'
name: Logs
- path: '*.Rcheck\**\*.out'
name: Logs
- path: '*.Rcheck\**\*.fail'
name: Logs
- path: '*.Rcheck\**\*.Rout'
name: Logs
- path: '\*_*.tar.gz'
name: Bits
- path: '\*_*.zip'
name: Bits
The important parts that differ from the template are:
environment:
global:
WARNINGS_ARE_ERRORS: 0
USE_RTOOLS: true
build_script:
- R -e "0" --args --bootstrap-packrat
This enables Rtools for the build, and loads the packrat packages.
It's also important to note that we are excluding - travis-tool.sh install_deps because that would cause the packages you depend on to be downloaded from CRAN, rather than built from your packrat directory.
Here's an example of an appveyor build for a simple R package where this is working: https://ci.appveyor.com/project/benmarwick/javaonappveyortest/build/1.0.21
I have two state files:
php/init.sls
php-fpm.sls
php/init.sls installs the package php53u
I'm trying to get php-fpm.sls to require php as that is how I declare it inside php/init.sls, however it only works if I require the pkg: php53u and not sls: php
Contents of php/init.sh:
php:
pkg:
- name: php53u
- installed
Contents of php-fpm.sls (using pkg where it works):
include:
- nginx
- php
php-fpm:
pkg:
- name: php53u-fpm
- installed
service:
- running
- enable: True
- require:
- pkg: php53u-fpm
- pkg: php53u
extend:
nginx:
file:
- name: /etc/nginx/php-fpm
- source: salt://nginx/src/etc/nginx/php-fpm
- managed
- template: jinja
(note that this has extra stuff about nginx that currently isn't a require though it should be)
You are correctly including the php sls file. You just need to set your require like this:
- pkg: php
Here's an example that should work:
php-fpm:
pkg:
- name: php53u-fpm
- installed
- require:
- pkg: php
service:
- running
- enable: True
- watch:
- pkg: php53u-fpm
- pkg: php
Notice that I also added the require to the pkg stanza to make sure that the php package is installed before the php53u-fpm package.
I also changed the "require" under the service stanza to a "watch". A "watch" acts as a "require", but also will restart the service if the "watched" pkg changes. So if there's a package upgrade it will restart the service automatically.
It's perfectly normal, in the require list, you need to use ID Declaration (http://docs.saltstack.com/ref/states/highstate.html#term-id-declaration).
The module php/init.sls contains only one ID Declaration : {pkg: php53u}. ID Declaration is composed of state name (pkg) and component name (php53u).
You can't require an entire module in salt, but in the last versions, modules will be executed in order of declaration, so the php module should be executed before the php-fm one. Salt will respect require requisite references in any cases.
Vocabulary in salt-stack could be a little difficult for beginners, here is a page which can helps you : http://docs.saltstack.com/ref/states/highstate.html.