Env variable not found by R in Azure Pipeline - r

I'm trying to convert my Travis-based tests to Azure Pipelines, and I'm running into a problem where R doesn't seem to pick up an environment variable.
Here is the yaml:
trigger:
- master
variables:
- group: GH
- name: R_LIBS_USER
value: '$(Agent.BuildDirectory)/R/library'
- name: containerImage
value: rocker/rstudio:latest
pool:
vmImage: 'ubuntu-latest'
container: rocker/rstudio:latest
steps:
- script: |
echo 'options(repos = "https://cloud.r-project.org")' > ~/.Rprofile
mkdir -p ${R_LIBS_USER}
displayName: 'Setup R library directory'
- bash: |
Rscript -e "install.packages(c('remotes', 'rcmdcheck'))" <-- problem here
Rscript -e "remotes::install_deps(dependencies=TRUE)"
displayName: 'Installing package dependencies'
- bash: |
Rscript -e "rcmdcheck::rcmdcheck(args = '--no-manual', error_on = 'error', check_dir = 'check')"
displayName: 'Checking package'
The problem is in the bash step where I install the remotes and rcmdcheck packages, and then run install_deps. For some reason R doesn't see the R_LIBS_USER variable, which should expand to something like /__w/1/R/library. Because of this, it tries to install to the site-wide location /usr/local/lib/R/site-library, which fails because this directory is not user-writeable.
What's happening here?

For whatever reason, R just can't see the environment variable R_LIBS_USER. What finally worked for me was editing the ~/.Rprofile config file to include the user library directory:
steps:
- script: |
echo 'options(repos = "https://cloud.r-project.org")' > ~/.Rprofile
echo ".libPaths(c('$R_LIBS_USER', .libPaths()))" >> ~/.Rprofile
mkdir -p $(R_LIBS_USER)
displayName: 'Setup R library directory'
Now I just need to figure out how to avoid reinstalling several megabytes of dependencies on each run....

Related

Error using GitHub actions with R markdown

I'm new to using GitHub actions, so please be gentle with me! I'm trying to automate a script that I'm currently running using Windows Task Scheduler. I've written the yml code below and its stored (privately) on GitHub at jeremy-horne/Client/.github/workflows/
The RMD code is 900 lines long and works when run in R studio. It is stored on GitHub at jeremy-horne/Client
`
name: my_project
on:
schedule:
- cron: "18 17 */1 * *" # Every day at 18:17
jobs:
render-rmarkdown:
runs-on: ubuntu-latest
# optionally use a convenient Ubuntu LTS + DVC + CML image
#container: docker://dvcorg/cml:latest
steps:
- uses: actions/checkout#v3
- name: Setup R
uses: r-lib/actions/setup-r#v2
#- name: Setup Pandoc
# uses: r-lib/actions/setup-pandoc#v2
- name: Setup dependencies
run: |
R -e 'install.packages("renv")'
R -e 'renv::restore()'
- name: Identify and email articles
#env:
# repo_token: ${{ secrets.GITHUB_TOKEN }}
run: |
Rscript -e 'rmarkdown::render(input = "My_Markdown_Code.Rmd")'
if [[ "$(git status --porcelain)" != "" ]]; then
git config --local user.name "$GITHUB_ACTOR"
git config --local user.email "$GITHUB_ACTOR#users.noreply.github.com"
git add *
git commit -m "Auto update Report"
git push origin
fi
`
The error I get says "render-rmarkdown
there is no package called ‘rmarkdown’"
Any help much appreciated!

How to change GitHub Actions OS from MacOs to Windows or Ubuntu?

I'm currently using the macOs os version for my Github Actions but it's very consuming on my minutes thresholds as it has a 10x cost on the monthly limit
I would like to switch to Ubuntu or Windows but I'm not quite sure about how to change my YML file.
Here is my actual YML file of one of my workflow:
# Hourly scraping
name: amazonR Polite Price Checker
# Controls when the action will run.
on:
schedule:
- cron: '0 0 * * 1,4'
jobs:
autoscrape:
# The type of runner that the job will run on
runs-on: macos-latest
# Load repo and install R
steps:
- uses: actions/checkout#master
- uses: r-lib/actions/setup-r#master
# Set-up R
- name: Install packages
run: |
R -e 'install.packages("tidyverse")'
R -e 'install.packages("tibble")'
R -e 'install.packages("openxlsx")'
R -e 'install.packages("gdata")'
R -e 'install.packages("lubridate")'
R -e 'install.packages("rvest")'
R -e 'install.packages("stringr")'
R -e 'install.packages("dplyr")'
R -e 'install.packages("purrr")'
R -e 'install.packages("plyr")'
R -e 'install.packages("polite")'
R -e 'install.packages("xml2")'
R -e 'install.packages("gt")'
R -e 'install.packages("blastula")'
# Run R script
- name: Scrape with the polite package
env:
EMAIL_SENDER: ${{ secrets.EMAIL_SENDER }}
EMAIL_PASSWORD: ${{ secrets.EMAIL_PASSWORD }}
EMAIL_RECIPIENT: ${{ secrets.EMAIL_RECIPIENT }}
EMAIL_CC_1: ${{ secrets.EMAIL_CC_1 }}
EMAIL_CC_2: ${{ secrets.EMAIL_CC_2 }}
EMAIL_CC_3: ${{ secrets.EMAIL_CC_3 }}
run: Rscript polite_price_checker.R
# Add new files in data folder, commit along with other modified files, push
- name: Commit files
run: |
git config --local user.name github-actions
git config --local user.email "actions#github.com"
git add polite_price/*
git commit -am "GH ACTION Autorun POLITE PRICE $(date)"
git push origin main --force
env:
REPO_KEY: ${{secrets.GITHUB_TOKEN}}
username: github-actions
The runner is configured by the runs-on for each job.
It can be a GitHub hosted runner or a self-hosted runner according to your context.
Examples with GitHub hosted runner:
jobs:
job1:
runs-on: ubuntu-latest
steps:
[ ... ]
job2:
runs-on: macos-latest
steps:
[ ... ]
job3:
runs-on: windows-latest
steps:
[ ... ]
The available GitHub runner images (with the pre-installed tools) can be found here for more details.
Be aware that each runner use a specific shell by default, for example ubuntu and macos use bash, where windows use powershell.
Therefore, in your case, you just need to update the runs-on: macos-latest to runs-on: ubuntu-latest or runs-on: windows-latest (supposing you want to use the latest, and not a specific version).
Note that you can also use matrix to run the same job on different runners if necessary (without duplicating code).

Gitlab CI fails for r package

I am building an R Package on GitLab and I am trying to get the GitLab CI to work, the issues are
devtools::check fails if there is an error, warning or note. I only want it to fail on errors
Deploy pkgdown to GitLab pages, it doesn't seem to work?
Below is the .gitlab-ci-yml I am using. I used the R package template from R Studio to test it.
# .gitlab-ci.yml
image: methodsconsultants/r-packaging
variables:
DOCKER_DRIVER: overlay2
PKGNAME: "test"
R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/$PKGNAME.Rcheck"
cache:
paths:
- $R_LIBS_USER
- vendor/apt
stages:
- build
- check
- test
- pages
before_script:
- mkdir -p vendor/apt
- apt-get --allow-releaseinfo-change update -qq
- apt-get remove -y libgcc-8-dev
- apt-get -o dir::cache::archives="vendor/apt" install -y libcairo2-dev -qq
buildbinary:
stage: build
script:
- r -e 'devtools::build(binary = TRUE)'
checkErrors:
stage: check
script:
- r -e 'if (!identical(devtools::check(document = FALSE, args = "--no-tests")[["errors"]], character(0))) stop("Check with Errors")'
unittests:
stage: test
script:
- r -e 'if (any(as.data.frame(devtools::test())[["failed"]] > 0)) stop("Some tests failed.")'
pages:
script:
- R -e "pkgdown::build_site()"
artifacts:
paths:
- public
only:
- master
Note
Everything works locally, devtools::check() produces warnings and notes but no errors. pkgdown builds fine. tests pass fine.
Gitlab CI is working, the build step and test stages pass fine, then fails on devtools::check() error message
I tried gitlab pages by removing the check() phrase, the pipeline finishes fine but under setting > Pages I can't see anything?
Worked it out:
devtools::check seems to raise an error on warnings on gitlab CI BUT not locally? Don't understand it, but you can set it explicitly which works devtools::check(error_on = "error")
GitLab pages I think needs to be in a deploy stage? Anyway it works now :) I did change the output folder of pkgdown to public to match gitlab pages (shown below)
Hope this helps anyone who stumbles upon this!
# .gitlab-ci.yml
image: rocker/tidyverse
variables:
DOCKER_DRIVER: overlay2
PKGNAME: "test"
R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/$PKGNAME.Rcheck"
cache:
paths:
- $R_LIBS_USER
- vendor/apt
stages:
- build
- test
- deploy
before_script:
- R -e 'devtools::install_deps(dep = T)'
buildbinary:
stage: build
script:
- R -e 'devtools::check(vignettes = FALSE, error_on = "error")'
unittests:
stage: test
script:
- r -e 'if (any(as.data.frame(devtools::test())[["failed"]] > 0)) stop("Some tests failed.")'
pages:
stage: deploy
script:
- R -e "install.packages('pkgdown')"
- R -e "pkgdown::build_site()"
artifacts:
paths:
- public
expire_in: 1 day
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
# _pkgdown.yml
url: ~
template:
bootstrap: 5
destination: public/

Github Action is not Running on Push, only on schedule

Based on this article, I am trying to generate a repo that runs everytime y push, but also every 4 or 8 hours. as seen by the action history, only one of them was successful, and only on a Mac Machine:
But my goal is to run it on at ubuntu-latest, windows-latest and macos-latest to check the code in all OS. One final goal is to make it so that the message of the scheduled pushs is updated in xxx date. This is my latest code EDITED on sept 23 which can be seen in this repo:
name: render readme
on:
push:
paths:
- README.Rmd
schedule:
- cron: 0 */4 * * *
jobs:
render:
runs-on: macOS-latest
steps:
- uses: actions/checkout#v2
- uses: r-lib/actions/setup-r#v1
- uses: r-lib/actions/setup-pandoc#v1
- name: "[Custom block] [macOS] Install spatial libraries"
if: runner.os == 'macOS'
run: |
# conflicts with gfortran from r-lib/actions when linking gcc
rm '/usr/local/bin/gfortran'
brew install pkg-config gdal proj geos
- name: "[Stage] [macOS] Install libgit2"
if: runner.os == 'macOS'
run: brew install libgit2
- name: Install packages
run: Rscript -e 'install.packages(c("rmarkdown", "pacman", "rgeos"))'
- name: install rgdal
run: Rscript -e 'install.packages("rgdal", configure.args = c("--with-proj-lib=/usr/local/lib/", "--with-proj-include=/usr/local/include/"))'
- name: Render README
run: Rscript -e 'rmarkdown::render("README.Rmd", output_format = "md_document")'
- name: Commit results
run: |
git add README.md man/figures/README-*
git commit README.md -am 'Re-build README.md' || echo "No changes to commit"
git push origin main || echo "No changes to commit"
Now it runs and it actually gives me a tick, but the md is not updated, within the runs in the task called commit results we can see the following error
Run git add README.md man/figures/README-*
git add README.md man/figures/README-*
git commit README.md -am 'Re-build README.md' || echo "No changes to commit"
git push origin main || echo "No changes to commit"
shell: /bin/bash -e {0}
env:
R_LIBS_USER: /Users/runner/work/_temp/Library
TZ: UTC
_R_CHECK_SYSTEM_CLOCK_: FALSE
NOT_CRAN: true
fatal: paths 'README.md ...' with -a does not make sense
No changes to commit
Everything up-to-date
So the commit or push is not working, thank you #krzysztof-madej, you catched one mistake which was the branch name, but I still can make the results update the README, even though it runs smoothly
Your issue seems to be with git, not github actions. The -a arg will stage all files, so adding README.md seems to make the command invalid. Try changing:
git commit README.md -am 'Re-build README.md' || echo "No changes to commit"
to
git commit -am 'Re-build README.md' || echo "No changes to commit"
Objective: To run it on at ubuntu-latest, windows-latest and macos-latest to check the code in all OS:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- ubuntu-latest
- macos-latest
- windows-latest

SonarQube with Travis not showing issues on dot net core 2 project

I have a simple dotnet core 2.0 project with a simple issue which is failing SonarLint with an unused variable issue.
The code is stored in a public github repository (here). A travis job (here) runs and has the SonarQube plugin and should post to SonarCloud (here).
The problem I have is that this issue is not being picked up by the analysis and published as an issue. I obviously have something set up incorrectly but I dont know what.
My .travis.yml is below
language: csharp
dist: xenial
sudo: required
mono: none
dotnet: 2.0.0
solution: Dibware.Salon.sln
addons:
sonarcloud:
organization: "dibley1973-github" # the key of the org you chose at step #3
token:
secure: $SONAR_TOKEN
branches:
only:
- master
before_script:
- chmod +x build.sh
- chmod +x run-tests.sh
script:
- ./build.sh
- ./run-tests.sh
- sonar-scanner
My sonar-project.properties file is below
# Project identification
sonar.projectKey=Core:Dibware.Salon
sonar.projectVersion=1.0.0.0
sonar.projectName=Dibware.Salon
# Info required for SonarQube
sonar.sources=./Domain
sonar.language=cs
sonar.sourceEncoding=UTF-8
C# Settings
sonar.dotnet.visualstudio.solution=Dibware.Salon.sln
# MSBuild
sonar.dotnet.buildConfiguration=Release
sonar.dotnet.buildPlatform=Any CPU
# StyleCop
sonar.stylecop.mode=
# SCM
sonar.scm.enabled=false
In the travis log I do have:
INFO: 27 files to be analyzed
WARN: Shallow clone detected, no blame information will be provided. You can convert to non-shallow with 'git fetch --unshallow'.
INFO: 0/27 files analyzed
WARN: Missing blame information for the following files:
WARN: *
.
<lots of files>
.
WARN: This may lead to missing/broken features in SonarQube
INFO: Calculating CPD for 0 files
INFO: CPD calculation finished
INFO: Analysis report generated in 216ms, dir size=381 KB
INFO: Analysis report compressed in 56ms, zip size=89 KB
INFO: Analysis report uploaded in 340ms
INFO: ANALYSIS SUCCESSFUL, you can browse https://sonarcloud.io/dashboard?id=Core%3ADibware.Salon
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=AWo0YQeAUanQDuOXxh79
INFO: Analysis total time: 11.484 s
Is this what is affecting the analysis? If so how do I resolve it? If not what else is stopping the analysis of the files, please?
EDIT:
I can see the following in the log, but it still does not get picked up by SoanrQube..
Chair.cs(17,17): warning CS0219: The variable 'a' is assigned but its value is never used
Edit 2:
I managed to getthe analzed number to go up, see below...
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=6ms
INFO: SCM provider for this project is: git
INFO: 27 files to be analyzed
INFO: 27/27 files analyzed
INFO: Calculating CPD for 0 files
... using the following in my .travis.yml
install:
- git fetch --unshallow --tags
That came from here:
https://stackoverflow.com/a/47441734/254215
Ok, I am not out of the wood yet, but am getting some analysis using the following .travis.yml
language: csharp
dist: xenial
sudo: required
mono: none
dotnet: 2.1.300
solution: Dibware.Salon.sln
addons:
sonarcloud:
organization: "dibley1973-github" # the key of the org you chose at step #3
token:
secure: $SONAR_TOKEN
branches:
only:
- master
install:
- dotnet tool install --global dotnet-sonarscanner
- git fetch --unshallow --tags
before_script:
- export PATH="$PATH:$HOME/.dotnet/tools"
- chmod +x build.sh
- chmod +x run-tests.sh
script:
- dotnet sonarscanner begin /k:"Core:Dibware.Salon" /d:sonar.login="$SONAR_TOKEN" /d:sonar.exclusions="**/bin/**/*,**/obj/**/*" /d:sonar.cs.opencover.reportsPaths="lcov.opencover.xml" || true
- ./build.sh
- ./run-tests.sh
- dotnet sonarscanner end /d:sonar.login="$SONAR_TOKEN" || true
In the end the travis.yml file I used which worked is this:
language: csharp
dist: xenial
sudo: required
mono: none
dotnet: 2.1.300
solution: Dibware.Salon.sln
addons:
sonarcloud:
organization: "dibley1973-github" # the key of the org you chose at step #3
token:
secure: $SONAR_TOKEN
branches:
only:
- master
install:
- dotnet tool install --global dotnet-sonarscanner
- git fetch --unshallow --tags
before_script:
- export PATH="$PATH:$HOME/.dotnet/tools"
- chmod +x build.sh
- chmod +x run-tests.sh
script:
- dotnet sonarscanner begin /k:"Core:Dibware.Salon" /d:sonar.login="$SONAR_TOKEN" /d:sonar.cs.opencover.reportsPaths="**/coverage.opencover.xml" /d:sonar.exclusions="**/bin/**/*,**/obj/**/*,**/Dibware.Salon.Web/**/*" || true
- ./build.sh
- ./run-tests.sh
- dotnet sonarscanner end /d:sonar.login="$SONAR_TOKEN" || true
The build file is this.
#!/usr/bin/env bash
dotnet restore
dotnet clean -c Release
dotnet build Dibware.Salon.sln -c Release
The test is this.
# Run the tests and collate code coverage results
dotnet test -c Release --no-build --no-restore Domain/SharedKernel/Dibware.Salon.Domain.SharedKernel.UnitTests/Dibware.Salon.Domain.SharedKernel.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutputFormat=opencover
I did not use a sonar-project.properties file.
HTH someone, one day

Resources